Ticket #1548 (new defect)

Opened 6 months ago

Last modified 5 months ago

range requests not working

Reported by: oscar@netpromote.hu Assigned to: jan
Priority: high Milestone: 1.5.0
Component: core Version: 1.5.x-svn
Severity: normal Keywords: range-request, byte range request
Cc: Blocking:
Need Feedback: 0

Description

Test case I use:

wget -t1 --debug http://site.com/content/movies/pretty_big.wmv --header="Range:bytes=100-200" -O wget.out

Result: wget.out remains empty.

---request begin---
GET /content/movies/pretty_big.wmv HTTP/1.0
User-Agent: Mozilla/5.0 (compatible; Panther)
Accept: */*
Host: site.com
Connection: Keep-Alive
Range: bytes=100-200
X-Panther: Validate

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.0 206 Partial Content
Cache-Control: max-age=31536000
Cache-Control: max-age=31536000
Cache-Control: private
X-Powered-By: PHP/5.2.0-8+etch10
Pragma: public
Expires: 0
Content-Type: application/force-download
Content-Disposition: attachment; filename="pretty_big.wmv";
Content-Transfer-Encoding: binary
ETag: "1615866504"
Accept-Ranges: bytes
Last-Modified: Fri, 14 Dec 2007 12:53:52 GMT
Content-Range: bytes 100-200/2553951618
Content-Length: 101
Connection: keep-alive
Date: Fri, 01 Feb 2008 08:57:00 GMT
Server: lighttpd/1.5.0

---response end---
206 Partial Content
Registered socket 5 for persistent reuse.
Disabling further reuse of socket 5.
Closed fd 5
Giving up.

strace: attached

Nothing I can see in the logs. The requests are rewritten to go through a small php script to facilitate authentication. I tried the test with large files, small files, with authorizer and without, network-backends linux-sendfile and gthread-aio. Same result.

The code was checked out of trunk on 2008_01_22 I added a patch from this site that enables setting the basis of etag generation. I also use this build in clusters. I'll attach that patch too.

I enabled request debugging and will attach the output of that as well.

Logs have been sanitized to change paths and camouflage host names and such. I did not intentionally remove debugging info. If the logs look consistent I am happy to provide the originals. (in a less open channel)

If you need more information don't hesitate to contact me.

Thanks in advance for looking at the ticket.

Attachments

light.trace (9.6 kB) - added by anonymous on 02/01/2008 09:28:52 AM.
strace from a byte-ranged get request
actualized_etags.patch (5.5 kB) - added by anonymous on 02/01/2008 09:29:44 AM.
etags patch I applied
lighty.log (3.7 kB) - added by anonymous on 02/01/2008 09:30:30 AM.
debug log output

Change History

02/01/2008 09:28:52 AM changed by anonymous

  • attachment light.trace added.

strace from a byte-ranged get request

02/01/2008 09:29:44 AM changed by anonymous

  • attachment actualized_etags.patch added.

etags patch I applied

02/01/2008 09:30:30 AM changed by anonymous

  • attachment lighty.log added.

debug log output

02/01/2008 11:18:31 AM changed by oscar@garbled.netpromote.hu

huh... I just tested with a 1.4.18 server. I don't ever get anything in the wget.out file.

I wonder if my test is broken...

02/01/2008 11:26:39 AM changed by anonymous

  • priority changed from normal to low.

ughh. I did the test using curl and I am getting the partial transfers ok. Let's put this on hold while I figure out what's going on.

02/01/2008 02:32:34 PM changed by oscar@garbled.netpromote.hu

  • priority changed from low to normal.

Found one possible culprit:

doing the test the following way:

--header "Range:bytes=2147483650-2553951618"

fails. Any address range starting above 2G fails!

This time however I found some noise in the log:

network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1381: (trace) (network-subsys sent us a fatal-error)
network_gthread_aio.c.186: (error) reading file failed: 22 (Invalid argument)
network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1381: (trace) (network-subsys sent us a fatal-error)
network_gthread_aio.c.186: (error) reading file failed: 22 (Invalid argument)
network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1381: (trace) (network-subsys sent us a fatal-error)

Good news is: If I switch to linux-sendfile backend the problem disappears. gthread-aio didn't appreciate this particular task.

So I have a workaround, how can I help creating a fix?

02/11/2008 04:23:52 PM changed by njahnke@gmail.com

confirmed this is a > 2gb issue on my debian etch box w/ libaio.so.1.0.1, kernel 2.6.18-4-486, lighty 1.5.0 r1992:

network_gthread_aio.c.186: (error) reading file failed: 22 (Invalid argument)
network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1370: (trace) (network-subsys sent us a fatal-error)

when requesting range beyond 2 gig.

this is high priority for me as i need aio to avoid constant i/o bottleneck ...

02/12/2008 05:51:33 AM changed by grenola

edit network_gthread_aio.c

change occurrence of pread() to pread64()

recompile

02/12/2008 05:54:25 AM changed by njahnke@gmail.com

  • priority changed from normal to high.

thanks grenola. i've confirmed this works on my debian box. please integrate into svn (we're too lazy) :P


Add/Change #1548 (range requests not working)




Change Properties