Asterisk memtesting reloaded


It was about time to re-test up to date Asterisk versions for memory leakage. As these tests take a rather long time, i will update this post when the 1.4 and 1.6 versions have gone through the test.
The test procedure is the same as in my previous test, incoming SIP calls that send a ringing, wait a little and hang up (no call is ever connected, no RTP is flowing).

Asterisk 1.8.4-rc2

Since my last test, things have improved a lot! Asterisk does not consume 200m+ right from the start and it does not leak that much any more. However it still leaks a bit. After 32M+ calls i stopped the test.

Memory consumption before the test: virt 513m  16m 5940 S

Memory consumption after the test: virt 712m res 192m 6432 S

Asterisk 1.6.2.18-rc1

After pushing 37M+ calls Asterisk stopped processing calls because it ran out of RTP ports (it was using the default RTP port range from 10000 to 20000). “netstat -l -n -p  | grep asterisk -c” shows that 10009 ports are in use by asterisk. Calls fail with:

[Mar 27 12:42:03] ERROR[25983] rtp.c: No RTP ports remaining. Can’t setup media stream for this call.
[Mar 27 12:42:03] WARNING[25983] chan_sip.c: Unable to create RTP audio session: Address already in use

Memory consumption before the test: 476m  14m 5412 S

Memory consumption after the test: 742m 197m 5868 S

Asterisk 1.4.41-rc1

After pushing 10M+ calls Asterisk is processing calls very slowly. The machine has a high load because it is constantly swapping pages to and from disk. Retransmissions start to fail:

[Mar 30 10:58:50] WARNING[10667]: chan_sip.c:2070 retrans_pkt: Maximum retries exceeded on transmission a3621613-d54e-122e-43b7-001aa0314ced for seqno 10401673 (Critical Response) — See doc/sip-retransmit.txt.

Memory consumption before the test: 392m  12m 4396 S

Memory consumption after the test: 1472m 358m 1324 S

Asterisk 1.2.40

During the test Asterisk was complaining about “avoiding initial deadlocks” and “avoiding deadlocks” a lot, but it did not deadlock or go into the famous 100% cpu loop. It just works. :-)

After processing 65M+ calls,  I decided to stop the test as i could not see an increasing memory consumption. The 1.2 branch of Asterisk keeps its place as my favourite branch (when it comes to stability).

Memory consumption before the test:  276m  11m 3464 S

Memory consumption after the test:  460m  79m 3768 S S


2 thoughts on “Asterisk memtesting reloaded

  • Kevin P. Fleming

    Well, I’m glad to see that Asterisk 1.8.x is handling your stress test better than the previous versions, although the problems you ran into with 1.4.x and 1.6.x are certainly bugs that should be fixed.

    However, I’m not sure you can classify this behavior as a ‘memory leak’. Do you have snapshots of the memory usage of the process throughout the test, after the peak simultaneous call volume was reached? If the memory usage continues to climb even if the simultaneous call volume stays the same (or drops), then that’s a leak. If the memory usage grows with the call volume and then stays relatively static as long as the call volume doesn’t increase… that’s not a leak, it’s just normal behavior. When Asterisk requests memory from the C library’s heap, the heap allocator will request memory from the kernel, and the process’ memory size will grow. If Asterisk later releases that memory, it is released back to the heap, but *never* released back to the kernel… and thus the kernel’s view of the process’ memory size stays the same, even though most of the memory is unused.

    After your test is complete, and Asterisk is idle, it’s completely possible for the kernel to still think it is using 50% more memory than when Asterisk was started, even though that memory is not being actively used. In order to determine whether there is a leak or not, you’d have to inspect the amount of *heap* memory that is actually in use at that point, not how large the heap is.

  • kapejod Post author

    Currently i only monitor the memory usage “by hand”, i should probably use a tool for making some nice graphs, any suggestions?

    What i am observing with 1.8 is that the memory usage is increasing slowly while the number of channels (and call setups) is constant. The other side is a number verification dialer which is configured to use 1000 channels and 100 call attempts per second.

    I totally agree that a higher memory usage after the test does not automatically qualify as a memory leak. I do see this for my sofia-sip-based number verification dialer, too. The memory usage after the test almost stays at the level of the peak call volume, but so far it pushed 500M+ calls without the memory usage increasing.

Comments are closed.