Last Friday I decided to use Valgrind to check the memory usage in opkg and discovered that it was leaking quite a substantial amount of memory.
Over the last two and a half days I’ve been fixing all the memory leaks exhibited by running my libopkg_test application. I have reduced the directly lost and indirectly lost memory to 0 bytes. The reachable memory at exit has been reduced to 29kB (down from ~13MB), with the remaining blocks belonging to libgpgme and libcurl which does not appear to be freed by their
respective de-initialisation routines.
Having had such a detailed look at the code, I am fairly certain further optimisation work is possible in terms of both memory usage and speed.
Some interesting statistics:
- I’ve add some 65 additional free and deinit statements
- … of which 52 are additional free() calls alone
- 18 files changed, mostly in libopkg, but also one in libbb
- There where a total of 184 lines inserted and 25 lines deleted in the full diff
I used Valgrind’s memcheck tool to help identify the leaks and massif to visualise the memory usage. It seems the Valgrind authors have remove the postscript output from massif in favour of an ascii chart. I’d be interested if anyone knows a way to display the massif output in a more graphical (non-ascii) way.
N.B. The scales on the two graphs are different, so be sure to read the axis when comparing!
Before the cleanup…
ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 67 from 1)
malloc/free: in use at exit: 14,808,479 bytes in 577,152 blocks.
malloc/free: 1,145,825 allocs, 568,673 frees, 1,444,779,097 bytes allocated.
For counts of detected errors, rerun with: -v
searching for pointers to 577,152 not-freed blocks.
checked 12,358,812 bytes.
LEAK SUMMARY:
definitely lost: 611,971 bytes in 40,899 blocks.
indirectly lost: 1,425,560 bytes in 83,798 blocks.
possibly lost: 183 bytes in 7 blocks.
still reachable: 12,770,765 bytes in 452,448 blocks.
suppressed: 0 bytes in 0 blocks.
MB
22.73^ #
| @#
| @ @# : .
| ,@@ @# : : :. : . .. :::
| , @@@ @# : :,: :: : : :: :::
| ,@ @@@ @# : :@: :: : : :: :::
| , @@ @@@ @# : :@: :: : : :: :::
| @@ @@ @@@ @# : :@: :: : : :: :::
| , .@@ @@ @@@ @# : :@: :: : : :: :::
| @@. . :@@ @@ @@@ @# : :@: :: : : :: :::
| .@@@: : :@@ @@ @@@ @# : :@: :: : : :: :::
| . :@@@: : ..@ .:: :@@ @@ @@@ @# : :@: :: : : :: :::
| @: :@@@: ::. . ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
| @@: :@@@: ::: : ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
| , @@: :@@@: ::: : ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
| .@ @@: :@@@: ::: : ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
| : :@ @@: :@@@: ::: : ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
| .: :@ @@: :@@@: ::: : ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
| :: :@ @@: :@@@: ::: : ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
| : :: :@ @@: :@@@: ::: : ::@ : : :::: :@@ @@ @@@ @# : :@: :: : : :: :::
0 +----------------------------------------------------------------------->MB
0 73.83
Number of snapshots: 53
Detailed snapshots: [6, 7, 8, 11, 12, 13, 21, 30, 31, 32, 33, 34, 35, 36, 37, 38 (peak), 41]
The two peaks in the graph represent the parsing of the repository data into a hash table. In my test application, the repository data is updated from the feeds, so the package lists are re-parsed. Previously, the main de-initialisation procedure for opkg (which, I hasten to add, was inherited from ipkg) did not free the hash table of packages properly, which meant every time the package list was refreshed, the entire hash table of packages would be leaked. Had a user refreshed the package list several times in one session, it’s likely the device would very quickly have run out of memory.
After the cleanup…
ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 65 from 1)
malloc/free: in use at exit: 29,064 bytes in 2,013 blocks.
malloc/free: 1,145,726 allocs, 1,143,713 frees, 1,443,951,146 bytes allocated.
For counts of detected errors, rerun with: -v
searching for pointers to 2,013 not-freed blocks.
checked 390,804 bytes.
LEAK SUMMARY:
definitely lost: 0 bytes in 0 blocks.
possibly lost: 0 bytes in 0 blocks.
still reachable: 29,064 bytes in 2,013 blocks.
suppressed: 0 bytes in 0 blocks.
Reachable blocks (those to which a pointer was found) are not shown.
To see them, rerun with: --leak-check=full --show-reachable=yes
MB
13.04^ # .
| # :
| :# ::
| .:# @ ::: :
| ::# @ .::: :.
| @::# @ . .. :::: :: .. ,. ,:
| :@::# @: .:@:: : ::::: :@ :::@: .:@:.
| .:@::# @: .::@:::::. .::::: :@. :::@::::::@::
| ,::@::# @: :::@:::::: .:::::: :@: :::@::::::@:::
| @::@::# @: :::@::::::. ::::::: :@: :::@::::::@::.
| ,@::@::# @: :::@::::::: .::::::: :@: :::@::::::@:::
| ,@@::@::# @: :::@:::::::. .:::::::: :@: :::@::::::@::::
| @@@::@::# @: :::@:::::::: ::::::::: :@: :::@::::::@::::.
| .@@@::@::# @: :::@:::::::: ::::::::: :@: :::@::::::@:::::
| :@@@::@::# @: :::@::::::::: . ::::::::: :@: :::@::::::@::::::
| :@@@::@::# @: :::@:::::::::. : ::::::::: :@: :::@::::::@:::::.
| ::@@@::@::# @: :::@:::::::::: : ::::::::: :@: :::@::::::@::::::,
| .::@@@::@::# @: :::@:::::::::: :: ::::::::: :@: :::@::::::@::::::@
| :::@@@::@::# @: :::@::::::::::: .:: ::::::::: :@: :::@::::::@::::::@:
| ,:::@@@::@::# @: :::@:::::::::::. ::: ::::::::: :@: :::@::::::@::::::@.
0 +----------------------------------------------------------------------->MB
0 94.44
Number of snapshots: 76
Detailed snapshots: [1, 5, 6, 7, 10, 13 (peak), 14, 19, 32, 47, 53, 63, 73]
gj 🙂