Lighttpd

A backtrace of the crash would be really useful; and if it dies under heavy load i suspect it is #1562.

I tried to get a backtrace but the binary wasn't compiled with any debug stuff, so i recompiled it from source (1.4.18.tar.gz), with the '-g' option, and then i got something. Hope this help a little bit more. See attachment.

-- gimre

Hm sh... I don't think that backtrace helps us - the code referenced is well tested and should not fail itself, so i think the memory got corrupted by something else and lighty crashed later.

I still think it could be related to #1562 (out of bounds array access), so if you want you can try the pre-release of 1.4.19: http://blog.lighttpd.net/

Perhaps it would be good to know what is in the buffer (b=0x9ab5448), with the following gdb commands you could have looked at that: (too late now, i know. but perhaps next time, if it crashes at the same place)

frame 2
print *b

Replying to stbuehler:

Perhaps it would be good to know what is in the buffer (b=0x9ab5448), with the following gdb commands you could have looked at that: (too late now, i know. but perhaps next time, if it crashes at the same place) {{{
frame 2
print *b
}}}

I actually didn't quit gdb, was thinking maybe you need some more info, so here it is:

(gdb) frame 2
#2  0x0805c4f7 in buffer_copy_string_buffer (b=0x9ab5448, src=0x9c14060) at buffer.c:163
163             return buffer_copy_string_len(b, src->ptr, src->used - 1);
(gdb) print *b
$2 = {ptr = 0x0, used = 0, size = 64}

Before the crash(es) i noticed a lot of connections to port 80, like 600-700 (or more) simultaneous connections in various states (as per the netstat -tnp command).

Anything else i can help with in this gdb session?

Crazy...^^

Okay. It could be that your assert() function doesn't work and malloc returned 0 - but i don't think that is the case as it doesn't look like your server was out of ram.

So it looks like someone set ptr to NULL but left size = 64; the buffer routine thinks there is enough space and just tries to reuse that, without checking ptr for zero (if someone corrupts the buffer struct that wouldn't help in every case anyway, it could just be an invalid non-zero pointer).

Do you use any 3rd-party modules? I didn't find something suspicious happening with con->physical.basedir (that is the buffer) in the upstream source.

So, still the possibility with #1562 remains ;-)

And i think you can close your gdb session, i cannot think of something else useful.

Nope, i'm not using any fancy modules, in fact only mod_access and mod_accesslog, no virtualhost, just the basic setup, one docroot.

In netstat output however i did see a lot for connections from the same IP address, that would explain the high connection count, so i'm just going to try mod_evasive, and limit the concurrent connections a bit, see how it works out.

After that, i'll still try the pre-release without the limit.

I'll update this ticket if i find something.

-- gimre

like 600-700 (or more) simultaneous connections

What does /server-status say?
And what is your max FDs value?
If it's the default (1024), you probably ran into http://trac.lighttpd.net/trac/ticket/1562

Replying to Olaf van der Spek:

What does /server-status say?
And what is your max FDs value?
If it's the default (1024), you probably ran into http://trac.lighttpd.net/trac/ticket/1562

Yes, it's the default. I didn't mess with it. Maybe i should increment it? That won't resolve the problem, only delay it, right? I'm not sure i understand completely what that #1562 bug is all about.

I just enabled server-status to see what it says. For now the connections are increasing steadily, most of them in (W)rite state, serving files.

(btw, i have problems submitting comments to this ticket, see http://trac.edgewall.org/ticket/6975)

Maybe i should increment it? That won't resolve the problem, only delay it, right?

Yes, you should. ;)
It should solve the problem, as there is only a max connections, which is also 1024 by default. IF you triple max FDs, you'll hit max connections and not max FDs, so you won't hit #1562.

It seems increasing the server.max-fds to 3072 helped, no crashes for 3 days. I'll keep monitoring it, and i'll definitely will upgrade to 1.4.19 when it's ready.

Thanks for all the help.

-- gimre

1.4.19 has been released already.