Lighttpd

I experience this problem too but using lighttpd newest from branch-1.4, a cert from CAcert.org and with both newest Konqueror and newest Mozilla Firefox in Linux. I too get SSL handshake failures about every 10 minutes and "white pages" everyonce in a while ... :(

-- philipp

Hello all,

Same problem for me...
Someone has found a solution or an alternative ?

Regards

Jean

-- jean

I been getting:

$time$: (network_openssl.c.115) SSL (error): 5 0 0 Success

$time$: (connections.c.529) connection closed: write failed on fd $num$

with lighttpd 1.4.11 (only tested with firefox 2). Noticed that if I return a content-type, the errors go away. If your getting this with files (and not dynamic pages that allows a content-type header) add an entry to the mime-types in lightty's config and it should stop happening.

-- deadram at gmail

Well, the comment by darix is not very helpful.

I get this:

2007-06-07 08:13:01: (connections.c.279) SSL: 1 error:1408C095:SSL routines:SSL3_GET_FINISHED:digest check failed
2007-06-07 08:13:01: (connections.c.279) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure

using lighttpd 1.4.15 built against OpenSSL 0.9.7j on a self-signed cert on a ppc platform but everything is ok on an x86 with the same client and same certificate, so the explanation that someone is trying to use non-ssl connection on an ssl port is plain wrong.

My bad (should have previewed) - repost of errors and config file:

Feb 18 18:23:58 lighttpd[579]: (connections.c.281) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure 
Feb 18 18:24:52 lighttpd[579]: (connections.c.281) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure

server.port = 80

$SERVER["socket"] == ":80" {
        $HTTP["host"] =~ "([^:]*)(:.*)?" {
                url.redirect = ( "/admin/(.*)" => "https://%1/$1" )
        }
}

$SERVER["socket"] == "192.168.1.1:443" {
        ssl.engine      =       "enable" 
        ssl.pemfile     =       "/etc/lighttpd_cert1.pem" 
        ssl.ca-file     =       "/www/lighttpd_ca.crt" 
}

I'm havving the exact same problem

2008-03-25 15:40:44: (connections.c.279) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure

I was using 1.4.18 so I tried upgrading to 1.4.19, but I still have the same problem. A friend can reproduce this problem each time with ie6 and I have reports of customers having the problem too. Maybe it's a bug in an old version of ie6 tough because I can't reproduce it here. But I will have to disable ssl for now because of this.

Also, probably not related (should I open another bug report ?) on konqueror I have :

An error occurred while loading https://www.mydomain.ca:
Could not connect to host www.mydomain.ca: SSL negotiation failed.

-- pelletiermaxime

Sorry for my last comment, the problem isn't related to lighttpd (for konqueror).

But, I still have a ton of ssl errors and ALL of them are from internet explorer 6 and i'm at a loss to explain how it happens. You said it's because "someone tries to use non ssl connections on your ssl port", but i'm using the little trick from http://trac.lighttpd.net/trac/wiki/HowToRedirectHttpToHttps to redirect and it's working fine for 95% of people. I wish I could reproduce it with my ie6, but I can't so I can't give you exact information, but this make the site unusable for a lot of my customers (working at the gouvernements/etc and that can't use a real browser).
Here's all the ssl error I see in my log file :

2008-04-04 09:10:26: (connections.c.262) SSL: -1 5 104 Connection reset by peer
2008-04-04 10:32:18: (connections.c.280) SSL: 1 error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
2008-04-04 15:02:35: (connections.c.280) SSL: 1 error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000)
2008-04-04 15:31:21: (connections.c.280) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure
2008-04-04 15:43:58: (connections.c.280) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure

This uis using 1.4.19 with patch from bug #285.

-- pelletiermaxime

This in the server logs:

Jul  3 08:28:24 lighttpd[1741]: (connections.c.281) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure 
Jul  3 08:28:24 lighttpd[1741]: (connections.c.281) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure 
Jul  3 08:28:26 lighttpd[1740]: (connections.c.281) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure 
Jul  3 08:28:27 lighttpd[1740]: (connections.c.281) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure 
Jul  3 08:28:28 lighttpd[1741]: (connections.c.281) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure 
Jul  3 08:28:29 lighttpd[1741]: (connections.c.281) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure 
Jul  3 08:28:29 lighttpd[1740]: (connections.c.281) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure 

Why is it sometimes SSL23_READ and sometimes SSL3_READ_BYTES? (Since they're both emitting the same error, I don't know if it matters.)

-- paul

Perhaps other webservers just ignore that error...

As we just use the openssl library for ssl communication i don't see how it could be our fault that the ssl handshake fails and no other error gets logged.

There have been some bugs where the connection wasn't closed which should be fixed now; so it would be nice if you could test with the current svn version of lighttpd-1.4.x

Checked out svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x/ (r2279) and built. Same stream of errors:

Aug  4 06:58:29 lighttpd[1477]: (connections.c.280) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure 
Aug  4 06:58:33 lighttpd[1478]: (connections.c.280) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure 
Aug  4 06:58:35 lighttpd[1477]: (connections.c.280) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure 
Aug  4 06:58:36 lighttpd[1478]: (connections.c.280) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure 
Aug  4 06:58:40 lighttpd[1478]: (connections.c.280) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure 

They're reasonably constant from IE6/IE7, and very rare with Firefox (I'll see a few a day with much testing.) Something else sloppy in IE's notorious SSL? Some other setting similar to max-keep-alive-requests to set?

-- paul

Are you using server.max-workers > 1?

Replying to stbuehler:

Are you using server.max-workers > 1?

Yes:

server.max-worker = 2

Changed it to 1, same errors.

-- paul

The SSL23_READ handshake failure is easy to trigger:

echo -n | telnet localhost 443

So i guess we will just ignore handshake failures in the future (perhaps with a config option?); i think they only indicate a connection close.

... and I can crash my computer by pressing the power button. Doesn't mean anything. ;-)

The fact that the same error is incidentally caused by http connections to the https port was already discussed and discounted in this case.

Not sure what else to offer or do at this point.

-- paul

You got that wrong; i didn't do a http connection to the https port. I just didn't send anything. As a matter of fact i could do that with curl too.

curl -E somestupidfile https://localhost/

That file shouldn't contain a real openssl key of course; curl will first open the connection and realize afterwards the file isn't valid and just close the connection again.

And even the openssl example server would print an error for this behaviour, so i just cannot take that error seriously.

And i just triggered the SSL3_READ_BYTES by doing curl https://example.com/ and pressing immediately Control-C.

Apache probably doesn't have this problem as they do the socket io themselves (if i read the code correctly).

Please, for the love of all that's holy, roll back r2291. The errors are not 'noise' - especially in cases where a client certificate is in use those are real errors that need to show through at runtime. Classifying them as such says little for the security-mindedness of the project.

It looks entirely like we might have a poorly behaved client - too bad it's ~90% of the clients out there, otherwise we could relegate it to the obscurity it so richly deserves. Have done a bit of researching into why Apache has settings to help with unclean socket close (which sounds like what we have here) and see if there's anything useful there. Will continue that line of inquiry.

-- paul

Replying to stbuehler:

No.

Breathe, big guy. Breathe...

Angry code is rarely good code, and the slapdash 'I'm supposed to report errors but I'm going to decide to eat some random set' is prime angry coding.. Sleep on it, and if you still feel that hiding that level of error by default is a good idea, sleep on it some more.

Tell the openssl devs to give me a way to be able to separate errors from "client just closed the connection", and we can talk about that again.

I am not interested in errors the client shows anyway (like "i don't trust you") and stupid connection closes.

The errors I was referring to is the case of using client-side certificates - this isn't the 'stupid' client saying 'I don't trust you' this is the server saying I don't trust the client. It's to authenticate the client end of the connection as well as the server end. The code that is in to attempt to cloak this issue will hide errors that are a direct indication of a tamper or attack. Thus my plea to remove the code.

Apart from that, i have nothing to say to you anymore. If you reopen the bug again without getting to the point in a bug report and without confirming the use of a current version, i will just delete your comments. I really have better things to do.

I'm not sure what else I can tell you about the bug: I've identified the clients, the fact that under a bit of load they hang. Which clients, how they're accessing the server, what versions of the server have been tested, have followed requests to try different versions and settings. By all measures there was some good ol' fashioned debugging going on.

Unfortunately I'm faced with having a reproducible issue here, that is actually killing some of our QA people because their browser hangs and the server just sits there not sending any packets for up to a couple of minutes. Yes, it is observed almost exclusively with a couple of different IE versions. I have seen it happen with both static and fcgi content, and have sunk multiple days into trying to narrow the issue down further. The errors in the log are the only indication on the server side that there is a problem. Perhaps that got lost in the noise of people just asking about stray errors in the log.

I'm currently suspending other projects to try to build a reproducible test case. Do you have a preference on whether I focus my efforts on the latest release or the current svn head?

As a last resort, if you are absolutely not interested in further investigation despite having every shred of information I can provide (and a proven willingness to experiment and follow instructions) tell me to piss off and I will. ;-)

-- paul

@paul:

I´m running current code from svn, that error shows from time to time and i know its from opera in my case, so there might be some other strange settings in your config if you get this on IE/FX too.

Nevertheless you still can see and investigate further on this by using debug.log-ssl-noise, for others its just annoying to get this on every opera req. I never had this with IE/FX and stuff.

So for many people its nice to have this turned off, for you there´s a debug-option. So we are not hiding stuff, we just move them to an debug-option.

Replying to nitrox:

So for many people its nice to have this turned off, for you there´s a debug-option. So we are not hiding stuff, we just move them to an debug-option.

Having a way to turn off known-spurious errors is one thing (a good thing, used judiciously). This (as you point out) is simply a browser-specific hack, which should not be the default behavior. If you have an ill-behaved client, then you should be able to set the option to turn off the error messages.

We are not discussing the bug anymore, so please feel free to join #lighttpd on irc.freenode.net and go ahead discussing this stuff there. For this "bug" its all said, its noise and we don´t want that in the logs.

paul:

You never wrote before that the clients still hang. You just said that the error.log still shows the handshake failure.

And i hope you now start to see what the problem is: You see an ssl error, and think it is important information - but it isn't. The handshake failure is probably just a connection close, nothing else (as no other error is logged).

So again: there is absolutely no use for a normal server admin in this ssl noise. Everyone could trigger it just to spam the log file. It only may help, if you try to find an error with a specific client.

I already showed how to trigger the handshake failure, here the others:
- tlsv1 alert unknown ca (SSL_R_TLSV1_ALERT_UNKNOWN_CA):

# either self signed or not trusted cert
curl -I https://localhost/
# or just use an invalid cert path
curl -I --capath /xxx https://localhost/

# same as "tlsv1 alert unknown ca", just with ssl3
curl -3 -I https://localhost/

And the last point: this are all errors the client is fully aware of (the clients sends them to the server to indicate why it didn't want to connect). It just doesn't make sense to log them on the server without knowing the client. You have no chance fixing/finding a problem based on this information alone.

And now your problem: your connection hangs, and you don't know why; you think it is ssl related as you saw the handshaking error, but it doesn't have to (as i already pointed out, it probably is just the "unexpected" connection close after a timeout. but we don't log them without ssl, so why should we do with ssl?).

Of course, if you cannot easily reproduce it, it is hard to track down; but you could use some other debugging options, strace and network sniffing to find out in which state it hangs and who is supposed to read/send data.

Btw:
- i wasn't angry while i coded this. but your comments were really annyoing. and i talked with others about the problem. and slept already a night over it.
- There is no client certificate verification (and i don't think the hidden errors are related to that; but if the openssl guys used the same error for server and client certificate verification i really don't care).
- It is not a browser specific hack, although some browsers trigger it more often than others.

And as nitrox already said: irc is a better medium to chat than a bug tracker.

Replying to stbuehler:

paul:

You never wrote before that the clients still hang. You just said that the error.log still shows the handshake failure.

Yes I did. 7 weeks ago. (Comment 13. I'll wait while you go re-read it.)

And i hope you now start to see what the problem is: You see an ssl error, and think it is important information - but it isn't. The handshake failure is probably just a connection close, nothing else (as no other error is logged).

I hope you now start to see the problem: I'm chasing what looks like the browser/server interaction hanging.

I have been following every request, answering every question. So we're at an impasse. (You're welcome to go back through the history here and point out any unanswered questions or unfulfilled requests.)

And now your problem: your connection hangs, and you don't know why; you think it is ssl related as you saw the handshaking error

... and because turning off SSL (simply tweaking the URL) lets us run for hours without a single hiccup. You do the math.

And as nitrox already said: irc is a better medium to chat than a bug tracker.

And lose the great history of the discussion? Nah.. ;-) What started as a simple asynchronous debugging exercise devolved into a fun little flame-fest. Whee! IRC at this point probably little more than a waste of all of our time.. ;-)

No worries though (hoping to tie off the conversation on a positive note), I'll see what else I can research from this end. It's infuriatingly tied to a few builds of IE on specific machines (IE6 on one machine has consistent failures, on another it doesnt..) If I come up with anything concrete, will let you know. (Part of my living is getting hosed by this, so I'm rather obligated to solve it, just as I'm obligated to share anything I find.)

-- paul

Replying to paul@brivo.com:

Replying to stbuehler:

paul:

You never wrote before that the clients still hang. You just said that the error.log still shows the handshake failure.

Yes I did. 7 weeks ago. (Comment 13. I'll wait while you go re-read it.)

So you didn't test if it still hangs with the current svn version. comment from 2 weeks ago (16). You only said that you still get the handshake errors in the log (17).

Is it a good enough reason to flame people if they don't see such things?

Btw: I didn't say that there is no bug lighty. I just don't think the ssl handshake failures got something to do with it apart from the fact that the connection times out. So actually the only information you got from it is "connection closed", not "ssl horribly fails".

And i have a log of my irc discussions :) all the passwords and...

If this one specific build of IE fails very often, it shouldn't be too hard to sniff the conversation; iirc there is way do decode the ssl packets if you have the server ssl key.

You could also try to setup an openssl tunnel to lighttpd (http://www.stunnel.org/) instead of using ssl in lighttpd (downside is that all stunnel connection have localhost as remote ip).

Not that anyone will ever read this closed bug... I had a problem very much like this and wanted to share my findings. We have an application that runs SSL + webdav that involves uploading a very large number of files ranging from 10s of bytes to GBs. We see the same long periods of unresponsiveness reported here when running SSL that disappears when turning it off. During the long periods of seeming inactivity, the lighttpd process was deleting a bazillion /var/tmp/lighttpd-upload-xxxxxx tmp files. The problem may be related to this issue:

http://redmine.lighttpd.net/issues/show/1070

...which is still unfixed in 1.14.x. I'm testing this patch now to see if it makes any difference. My next step will be to hack in some kind of chunked delete queuing to minimize the amount of time lighttpd spends waiting on the kernel to come back from a delete call when the I/O wait is high. Hope this helps someone.