net-cpp

Large leak for each request

Bug #1423765 reported by Michi Henning
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical System Image
Fix Released
High
Unassigned
Canonical System Image ww11-ota
net-cpp
Fix Released
Critical
Thomas Voß
net-cpp (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

When I run the http_client_test under valgrind, I'm seeing a large leak for each request. After the 11 request made by the test, valgrind reports that 2.7 MB of memory have gone missing. Note that this appears to be a different leak than the one in bug #1419620.

Excerpt from valgrind:

==6870== 2,014 (72 direct, 1,942 indirect) bytes in 1 blocks are definitely lost in loss record 129 of 143
==6870== at 0x4C2CC90: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6870== by 0x6267E96: curl_share_init (in /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0)
==6870== by 0x4EA0614: curl::shared::Handle::Private::Private() (shared.cpp:42)
==6870== by 0x4EA05B5: curl::shared::Handle::Handle() (shared.cpp:57)
==6870== by 0x4E80E35: curl::easy::Handle::Private::Private() (easy.cpp:137)
==6870== by 0x4E7F501: curl::easy::Handle::Handle() (easy.cpp:221)
==6870== by 0x4E4E1A5: core::net::http::impl::curl::Client::put(core::net::http::Request::Configuration const&, std::istream&, unsigned long) (client.cpp:183)
==6870== by 0x43CDDD: HttpClient_put_request_for_existing_resource_succeeds_Test::TestBody() (http_client_test.cpp:469)
==6870== by 0x47D111: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078)
==6870== by 0x47852C: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114)
==6870== by 0x45F3C0: testing::Test::Run() (gtest.cc:2151)
==6870== by 0x45FBCA: testing::TestInfo::Run() (gtest.cc:2326)
==6870==
==6870== 2,748,793 (17,920 direct, 2,730,873 indirect) bytes in 35 blocks are definitely lost in loss record 143 of 143
==6870== at 0x4C2B100: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6870== by 0x4E833FA: curl::multi::Handle::Handle() (multi.cpp:257)
==6870== by 0x4E4D5A8: core::net::http::impl::curl::Client::Client() (client.cpp:39)
==6870== by 0x4E771EB: _ZN9__gnu_cxx13new_allocatorIN4core3net4http4impl4curl6ClientEE9constructIS6_IEEEvPT_DpOT0_ (in /home/michi/src/net-cpp/build/src/libnet-cpp.so.1.1.0)
==6870== by 0x4E71DDA: _ZNSt16allocator_traitsISaIN4core3net4http4impl4curl6ClientEEE12_S_constructIS5_IEEENSt9enable_ifIXsrSt6__and_IINS7_18__construct_helperIT_IDpT0_EE4typeEEE5valueEvE4typeERS6_PSC_DpOSD_ (alloc_traits.h:253)
==6870== by 0x4E6DCDE: _ZNSt16allocator_traitsISaIN4core3net4http4impl4curl6ClientEEE9constructIS5_IEEEDTcl12_S_constructfp_fp0_spcl7forwardIT0_Efp1_EEERS6_PT_DpOS9_ (alloc_traits.h:399)
==6870== by 0x4E6A7BD: std::_Sp_counted_ptr_inplace<core::net::http::impl::curl::Client, std::allocator<core::net::http::impl::curl::Client>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<>(std::allocator<core::net::http::impl::curl::Client>) (shared_ptr_base.h:515)
==6870== by 0x4E66C9B: _ZN9__gnu_cxx13new_allocatorISt23_Sp_counted_ptr_inplaceIN4core3net4http4impl4curl6ClientESaIS7_ELNS_12_Lock_policyE2EEE9constructISA_IKS8_EEEvPT_DpOT0_ (in /home/michi/src/net-cpp/build/src/libnet-cpp.so.1.1.0)
==6870== by 0x4E637BD: _ZNSt16allocator_traitsISaISt23_Sp_counted_ptr_inplaceIN4core3net4http4impl4curl6ClientESaIS6_ELN9__gnu_cxx12_Lock_policyE2EEEE12_S_constructISA_IKS7_EEENSt9enable_ifIXsrSt6__and_IINSC_18__construct_helperIT_IDpT0_EE4typeEEE5valueEvE4typeERSB_PSI_DpOSJ_ (alloc_traits.h:253)
==6870== by 0x4E60CC1: _ZNSt16allocator_traitsISaISt23_Sp_counted_ptr_inplaceIN4core3net4http4impl4curl6ClientESaIS6_ELN9__gnu_cxx12_Lock_policyE2EEEE9constructISA_IKS7_EEEDTcl12_S_constructfp_fp0_spcl7forwardIT0_Efp1_EEERSB_PT_DpOSF_ (alloc_traits.h:399)
==6870== by 0x4E5DDA2: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<core::net::http::impl::curl::Client, std::allocator<core::net::http::impl::curl::Client>>(std::_Sp_make_shared_tag, core::net::http::impl::curl::Client*, std::allocator<core::net::http::impl::curl::Client> const&) (shared_ptr_base.h:619)
==6870== by 0x4E599AB: std::__shared_ptr<core::net::http::impl::curl::Client, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<core::net::http::impl::curl::Client>>(std::_Sp_make_shared_tag, std::allocator<core::net::http::impl::curl::Client> const&) (shared_ptr_base.h:1090)
==6870==
==6870== LEAK SUMMARY:
==6870== definitely lost: 18,640 bytes in 45 blocks
==6870== indirectly lost: 2,750,293 bytes in 36,500 blocks
==6870== possibly lost: 0 bytes in 0 blocks
==6870== still reachable: 64 bytes in 2 blocks
==6870== suppressed: 0 bytes in 0 blocks
==6870== Reachable blocks (those to which a pointer was found) are not shown.
==6870== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6870==
==6870== For counts of detected and suppressed errors, rerun with: -v
==6870== ERROR SUMMARY: 11 errors from 11 contexts (suppressed: 0 from 0)

Related branches

lp://staging/~thomas-voss/net-cpp/fix-1423765
Michi Henning (community): Approve
PS Jenkins bot: Needs Fixing (continuous-integration)
Diff: 484 lines (+149/-91)
4 files modified
debian/control (+0/-1)
src/core/net/http/impl/curl/easy.cpp (+19/-9)
src/core/net/http/impl/curl/easy.h (+38/-12)
src/core/net/http/impl/curl/multi.cpp (+92/-69)
lp://staging/ubuntu/vivid-proposed/net-cpp
Thomas Voß (thomas-voss)
Changed in net-cpp:
importance: Undecided → Critical
status: New → In Progress
assignee: nobody → Thomas Voß (thomas-voss)
Revision history for this message
Thomas Voß (thomas-voss) wrote :
Download full text (5.1 KiB)

With the MP attached to this bug, the leaks are gone, except for a spurious one on shutdown that can be attributed to CURL. Quoting from the original MP discussion here:

Running the load test in a loop, after maybe a hundred iterations or so, I got a valgrind complaint:

==29504==
==29504== HEAP SUMMARY:
==29504== in use at exit: 161,360 bytes in 1,769 blocks
==29504== total heap usage: 325,601 allocs, 323,832 frees, 54,917,164 bytes allocated
==29504==
==29504== 161,296 (73 direct, 161,223 indirect) bytes in 1 blocks are definitely lost in loss record 64 of 64
==29504== at 0x4C2B100: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==29504== by 0x4E8A699: boost::asio::detail::thread_info_base::allocate(boost::asio::detail::thread_info_base*, unsigned long) (thread_info_base.hpp:60)
==29504== by 0x4E8A7A8: boost::asio::asio_handler_allocate(unsigned long, ...) (handler_alloc_hook.ipp:50)
==29504== by 0x4E883DE: void* boost_asio_handler_alloc_helpers::allocate<curl::multi::Handle::Private::Timeout::Private::async_wait_for(std::weak_ptr<curl::multi::Handle::Private> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&)::{lambda(boost::system::error_code const&)#1}>(unsigned long, curl::multi::Handle::Private::Timeout::Private::async_wait_for(std::weak_ptr<curl::multi::Handle::Private> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&)::{lambda(boost::system::error_code const&)#1}&) (handler_alloc_helpers.hpp:37)
==29504== by 0x4E87F56: void boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::async_wait<curl::multi::Handle::Private::Timeout::Private::async_wait_for(std::weak_ptr<curl::multi::Handle::Private> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&)::{lambda(boost::system::error_code const&)#1}>(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, curl::multi::Handle::Private::Timeout::Private::async_wait_for(std::weak_ptr<curl::multi::Handle::Private> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&)::{lambda(boost::system::error_code const&)#1}&) (deadline_timer_service.hpp:185)
==29504== by 0x4E87CA1: boost::asio::async_result<boost::asio::handler_type<curl::multi::Handle::Private::Timeout::Private::async_wait_for(std::weak_ptr<curl::multi::Handle::Private> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&)::{lambda(boost::system::error_code const&)#1}, void (boost::system::error_code)>::type>::type boost::asio::deadline_timer_service<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> >::async_wait<curl::multi::Handle::Private::Timeout::Private::async_wait_for(std::weak_ptr<curl::multi::Handle::Private> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&)::{lambda(boost::system::error_code const&)#1}>(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::asio::handler_type&&) (deadline_timer_service.hpp:149)
==29504== by 0x4E87B93: boost::asio::async_result<boost::asio::handler_type<curl::mult...

Read more...

Revision history for this message
Pat McGowan (pat-mcgowan) wrote :

Be good to land this fix if possible as its a large memory leak

Changed in canonical-devices-system-image:
importance: Undecided → High
milestone: none → ww13-ota
status: New → In Progress
Pat McGowan (pat-mcgowan)
Changed in canonical-devices-system-image:
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package net-cpp - 1.1.0+15.04.20150305-0ubuntu1

---------------
net-cpp (1.1.0+15.04.20150305-0ubuntu1) vivid; urgency=medium

  [ thomas-voss ]
  * Make sure that Multi::Private instances are correctly cleaned up by
    only handing out weak_ptr's to it. (LP: #1419620, #1423765)
 -- CI Train Bot <email address hidden> Thu, 05 Mar 2015 12:08:09 +0000

Changed in net-cpp (Ubuntu):
status: New → Fix Released
Michi Henning (michihenning)
Changed in net-cpp:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.