ThreadPool allocation of virtual memory

[jthomas]

Hello,

I have been experimenting with POCO C++ Libraries v1.3.2. I like it very much. The design is very clean, and it works well. Thank you for publishing POCO as open source.

I experimented first with the XML and SQLite integration with good success.

Now I am experimenting with ThreadPool, FastMutex, and NotificationQueue. This is a separate test program without the use of XML or SQLite. It just creates some number of threads and the threads then send each other messages as fast as possible so I can benchmark the performance.

This thread test program also runs successfully, but I noticed that if I use 'pmap' under FC8 Linux to inspect the runtime memory footprint, each thread has reserved 10 MB of memory. With 64 threads, this results in a 660 MB allocation of virtual memory:


b6106000 10240K rw--- [ anon ] <-- repeated for each thread
b6b06000 4K ----- [ anon ] <-- repeated for each thread
b6b07000 10240K rw--- [ anon ]
b7507000 4K ----- [ anon ]
b7508000 10252K rw--- [ anon ]
b7f1e000 4K rw--- [ anon ]
bfcd7000 84K rw--- [ stack ]
total 660092K

The 'top' utility confirms the memory map. Note '644m' (MB) of virtual memory, while the physical memory is only 1964 KB:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4008 jthomas 20 0 644m 1964 1468 S 99.9 0.4 0:43.81 queue

The allocation occurs at startup, and does not increase over time, so this does not feel like a memory leak. The 10 MB per thread allocation occurs for each potential thread in the ThreadPool size, even if a Runnable class is not started for each thread.

Is this large allocation of virtual memory, 10 MB per thread, expected? Is there any API function I can use to tune this?

Thanks.

Jim

[lewislp]

> Is this large allocation of virtual memory, 10 MB per thread, expected? Is there any API function I can use to tune this?

8-10 MB seems to be the current default for Linux. There's no API yet, but I just submitted a patch to provide (get|set)StackSize on Thread. I presume this API could be extended to ThreadPool as well.

Larry

[alex]

> > Is this large allocation of virtual memory, 10 MB per thread, expected? Is there any API function I can use to tune this?

> 8-10 MB seems to be the current default for Linux. There's no API yet, but I just submitted a patch to provide (get|set)StackSize on Thread. I presume this API could be extended to ThreadPool as well.

This has just been freshly submitted in the repository trunk. I just wrote and ran some basic tests. I'd be curious to hear how it works in a real world scenario.

Alex

[ryppka]

Hi,


> There's no API yet, but I just submitted a patch to provide (get|set)StackSize on Thread. I presume this API could be extended to ThreadPool? as well.

Definitely so. Is there some hope it'll be in trunk any time soon? ;)

TIA

Serge

[alex]

> Definitely so. Is there some hope it'll be in trunk any time soon? ;)

Thread::set/getStackSize() has been in the trunk for almost a week now. I'll look into the ThreadPool as soon as I get a chance.

Alex

[alex]

> > Definitely so. Is there some hope it'll be in trunk any time soon? ;)

> Thread::set/getStackSize() has been in the trunk for almost a week now. I'll look into the ThreadPool as soon as I get a chance.

A global macro has been introduced (in Config.h):

Code: Select all


#define POCO_THREAD_STACK_SIZE 0


The default value (zero) results in OS default thread stack sizes.
Change the value to globally affect the stack size for threads. Change
of this macro value affects Thread, ))ThreadPool(( and default thread
pool. Note that stack sizes are usually rounded up to the nearest page
size multiple.

Alex