Lighttpd 1.4.19 Memory Leak

[remi@…]

I was attempting to upgrade lighttpd from 1.4.18 to 1.4.19 (in order to pull in the security fixes) for our site, but it never made it through testing. Specifically, when I use apachebench to throw a lot of traffic at it, lighttpd 1.4.19 grows in an unbounded way. It appears that there is a memory leak in the server. After approximately one million hits, the lighttpd process had grown to be more than 600MB in size.

I have created two identical server images -- one with lighttpd 1.4.19 and one with lighttpd 1.4.18 plus the DOS patch lighttpd-1.4.x_high_load_dos.patch. When I send identical traffic to each of them, I see the following results.

lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) before traffic: 4276 / 2140
lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) after traffic: 
lighttpd 1.4.19 memory usage (DRS/RSS) before traffic: 4267 / 2132
lighttpd 1.4.19 memory usage (DRS/RSS) after traffic: 227863 / 226004

The URLs requested were all very small files (most smaller than 1,024 bytes; all smaller than 4,096 bytes) that lighttpd is supposed to send using x-sendfile. Basically the request comes in and is sent through to an FCGI backend which responds with an x-sendfile location that lighttpd then sends.

I will attempt to respond to this ticket with more valgrind information as I garner it.

[anonymous]

Oops. Typo.

lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) after traffic: 

Should be:

lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) after traffic: 2668 / 2860

[remi@…]

Alright, I ran lighttpd within valgrind (needed to disable "server.max-fds" in order to make it work), and got the attached file from sending 13,000 requests at the server.

[anonymous]

Annoyingly, Trac refuses the attachment as spam. I will e-mail it to jan@… .

[oscar@…]

Hullo

I am hit by this as well. Right now I have a script that shoots lighty when it reaches the 50MB mark.

Can I help in any way?

[anonymous]

We are also experiencing this. I just upgraded our server to 1.4.19 using the FreeBSD ports system a week ago. The main lighttpd process appears to add 1M of resident use every few minutes.

The server uses php 5.2.6 fastcgi to serve a lot of X-sendfile files and dynamically generated pages. Static images and cs/jss are a small part of the load. 30-50 reqs/second baseline load.

FreeBSD 6-2-RELEASE-p11.

Has there been any motion on this?

[anonymous]

Here's the outline of a test scenario I just created that seems to point to sendfile handling as the problem:

stop lighty start lighty

lighty RES size ~3M

100k reqs for php script that generates 60k text output

RES size 4.8M

another 100k reqs

RES size 4.852k (unchanged)

stop lighty start lighty

RES size ~ 3M

100k reqs for static jpg

RES size ~ 3.5M

another 100k reqs

RES size 3.508M (unchanged)

stop lighty start lighty

RES size ~ 3M

100 k reqs for same static jpg sent using x-sendfile from php

RES size 35M

another 100k reqs

RES size 67M

I can upload the actual scripts but you see the problem: getting a jpg 100k times is fine, getting that SAME jpg 100k times with sendfile results in massive growth of the resident size of the process.

The FreeBSD port has a patch to fix a lighty sendfile bug when a connection dies. I compiled a version without this patch, and the memory leak behavior is still there.

That leads me to believe it's not port specific, but it still might be FreeBSD specific since sendfile is handled from within network_freebsd_sendfile.c.

I hope this jogs some developer's mind or sheds a little light... something about memory cleanup after sendfile changed with the version change from 1.4.18 to 1.4.19.

[anonymous]

I realized my earlier post might have been slightly incorrect, since even the static output is done with sendfile, the issue is really the handler for the X-LIGHTTPD-sendfile header set from within php.

With that in mind, there is an extremely suspicious diff between the 1.4.18 and 1.4.19 versions of mod_fastcgi.c

On line 2532 of the 1.4.18 version, it checks for the header, does 3 lines of work and the test block ends.

In the 1.4.19 version, the test block starts on line 2541. It inits a variable with this line within the test block scope: data_string *dcls = data_string_init(); There are some new buffer copies (to set the Content-length header it looks like)

Most suspiciously, the dcls variable is used as an lval once again and then freed:

dcls = (data_string*) array_replace(con->response.headers, (data_unset *)dcls); if (dcls) dcls->free((data_unset*)dcls);

(line 2555)

I am not a lighty developer but is there any chance this block has the leak in it?

[stbuehler]

Fixed in [2143]