Ticket #217 (closed defect: fixed)

Opened 3 years ago

Last modified 1 year ago

https transfer of large files fail with SSL error

Reported by: m@marsorange.com Assigned to: jan
Priority: normal Milestone:
Component: core Version: 1.3.15
Severity: critical Keywords: openssl openbsd
Cc: Blocking:
Need Feedback:

Description

I recently compiled & installed Lighttpd-1.3.16 (ssl) on an OpenBSD 3.7 server Resin.

When servicing requests via HTTPS, some large (greater than 16KB) files get truncated at 16KB of data. When I say "some large files", I mean that some files work & some don't. I do not see a pattern to which files get truncated, but it's always the same ones.

All of the JPEGS linked from this HTTPS page fail 16KB into the transfer. When using standard HTTP the images will download entirely.

Here the State & Request Handling log during one of these failures: 

2005-08-21 16:45:44: (connections.c.1167) state at start 8 req-end 
2005-08-21 16:45:44: (connections.c.1195) state for fd 8 req-end 
2005-08-21 16:45:44: (connections.c.1221) state for fd 8 handle-req 
2005-08-21 16:45:44: (response.c.1019) -- splitting Request-URI 
2005-08-21 16:45:44: (response.c.1020) Request-URI  :  /photo/ws2003/images/11-FiresideMusicians.jpg 
2005-08-21 16:45:44: (response.c.1021) URI-scheme   :  https 
2005-08-21 16:45:44: (response.c.1022) URI-authority:  marsorange.com 
2005-08-21 16:45:44: (response.c.1023) URI-path     :  /photo/ws2003/images/11-FiresideMusicians.jpg 
2005-08-21 16:45:44: (response.c.1024) URI-query    :   
2005-08-21 16:45:44: (response.c.1068) -- sanitising URI 
2005-08-21 16:45:44: (response.c.1069) URI-path     :  /photo/ws2003/images/11-FiresideMusicians.jpg 
2005-08-21 16:45:44: (response.c.1167) -- logical -> physical 
2005-08-21 16:45:44: (response.c.1168) Doc-Root     : /home2/mars/typo/public/ 
2005-08-21 16:45:44: (response.c.1169) Rel-Path     : /photo/ws2003/images/11-FiresideMusicians.jpg 
2005-08-21 16:45:44: (response.c.1170) Path         : /home2/mars/typo/public/photo/ws2003/images/11-FiresideMusicians.jpg 
2005-08-21 16:45:44: (response.c.1171) Server-Name  : marsorange.com 
2005-08-21 16:45:44: (response.c.1187) -- handling physical path 
2005-08-21 16:45:44: (response.c.1188) Path         : /home2/mars/typo/public/photo/ws2003/images/11-FiresideMusicians.jpg 
2005-08-21 16:45:44: (response.c.1325) -- file found 
2005-08-21 16:45:44: (response.c.1326) Path         : /home2/mars/typo/public/photo/ws2003/images/11-FiresideMusicians.jpg 
2005-08-21 16:45:44: (connections.c.1303) state for fd 8 resp-start 
2005-08-21 16:45:44: (connections.c.1427) state for fd 8 write 
2005-08-21 16:45:44: (connections.c.1533) state at exit: 8 write 
2005-08-21 16:45:44: (network_openssl.c.168) SSL: 1 -1 error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry 
2005-08-21 16:45:44: (connections.c.530) connection closed: write failed on fd 8 
2005-08-21 16:45:44: (connections.c.1167) state at start 8 error 
2005-08-21 16:45:44: (connections.c.1506) shutdown for fd 8 
2005-08-21 16:45:44: (connections.c.1381) state for fd 8 close 
2005-08-21 16:45:44: (connections.c.1410) connection closed for fd 8 
2005-08-21 16:45:44: (connections.c.1370) state for fd 8 connect 
2005-08-21 16:45:44: (connections.c.1533) state at exit: 8 connect

Thinking that there may be a problem between Lighttpd's SSL support & the server's OpenSSL 0.9.7, I recompiled Lighttpd with a completely new version of OpenSSL 0.9.8. The same error occurs.

Attachments

Change History

08/21/2005 11:00:43 PM changed by nikns@secure.lv

I have the same on my OpenBSD, there is debug. btw sometimes it is possible to download 16,32,80kb. Files can't be downloaded fully through SSL on i386 arch the same with lighttpd 1.4.0.

# lighttpd -v lighttpd-1.3.15 (ssl) - a light and fast webserver Build-Date: Jul 24 2005 00:33:30 # uname -a OpenBSD obsd.secure.lv 3.7 GENERIC#0 i386

# wget -dt 1 https://localhost/bubu DEBUG output created by Wget 1.8.2 on openbsd3.7.

--02:24:15-- https://localhost/bubu

=> `bubu'

Resolving localhost... done. Caching localhost => ::1 127.0.0.1 Connecting to localhost[::1]:443... Closing fd 4 failed: Connection refused. Connecting to localhost[127.0.0.1]:443... connected. Created socket 4. Releasing 0x3c01d830 (new refcount 1). ---request begin--- GET /bubu HTTP/1.0 User-Agent: Wget/1.8.2 Host: localhost Accept: */* Connection: Keep-Alive

---request end--- HTTP request sent, awaiting response... HTTP/1.0 200 OK Connection: keep-alive Date: Sat, 23 Jul 2005 23:24:15 GMT Last-Modified: Sat, 23 Jul 2005 23:19:03 GMT Content-Length: 20971520 ETag: "-1879875376" Accept-Ranges: bytes Content-Type: application/octet-stream Server: lighttpd/1.3.15

Found localhost in host_name_addresses_map (0x3c01d830) Registered fd 4 for persistent reuse. Length: 20,971,520 [application/octet-stream]

0% [ ] 16,384 3.12M/s ETA 00:06

02:24:15 (3.12 MB/s) - Connection closed at byte 16384. Giving up.

# ls -al | grep bubu -rw-r--r-- 1 root wheel 16384 Jul 24 02:24 bubu # ls -alh | grep bubu -rw-r--r-- 1 root wheel 16.0K Jul 24 02:24 bubu

# tail lighttpd.error.log 2005-07-24 02:19:45: (network_openssl.c.168) SSL: 1 -1 error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry 2005-07-24 02:19:45: (connections.c.530) connection closed: write failed on fd 7 2005-07-24 02:19:49: (network_openssl.c.168) SSL: 1 -1 error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry 2005-07-24 02:19:49: (connections.c.530) connection closed: write failed on fd 7 2005-07-24 02:19:56: (network_openssl.c.168) SSL: 1 -1 error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry 2005-07-24 02:19:56: (connections.c.530) connection closed: write failed on fd 7 2005-07-24 02:20:20: (network_openssl.c.168) SSL: 1 -1 error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry 2005-07-24 02:20:20: (connections.c.530) connection closed: write failed on fd 6 2005-07-24 02:20:34: (network_openssl.c.168) SSL: 1 -1 error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry 2005-07-24 02:20:34: (connections.c.530) connection closed: write failed on fd 6

# echo "GET /bubu HTTP/1.0\n" | openssl s_client -crlf -debug -connect localhost:443 > s_client.log 2>&1

http://secure.lv/~nikns/workaround/lighttpd/s_client.log

08/22/2005 12:25:47 AM changed by m@marsorange.com

To clarify: The amount of data downloaded before the premature connection closure is 16,384-bytes, which is exactly 16KB. (Isn't that the payload size of an individual HTTPS packet?)

08/22/2005 09:14:40 AM changed by jan

  • status changed from new to assigned.

I want to supply a fix for this in 1.4.1 but can't reproduce it good enough. Please come to IRC (irc.freenode.net) into the channel #lighttpd and ping me (weigon).

08/22/2005 10:46:08 AM changed by jan

  • status changed from assigned to closed.
  • resolution set to fixed.

fixed in changeset [607].

we can't use mmap() with SSL_write as we don't have guarantee that after a SSL_ERROR_WANT_WRITE we get the same mmap()ed address as start pointer. On OpenBSD the start-address is completly random.

The fix is the use a constant local buffer where the content is fetch into first. As the buffer is allocate only once, the address stays the same all the time.

Add/Change #217 (https transfer of large files fail with SSL error)




Change Properties
Action