lighty vs. apt-get - problem with pipelining

there can be keep-alive limits in the server side, and the client never know when it reach the limit. the client should retry if the pipelined requests is failed. even the 1st request in the pipeline should be try 2 times, according to the http rfc

We were also seeing this issue when using APT with lighttpd 1.4.13 over a fast LAN. I did some test runs with trace and tcpdumps, and found the following strace:

open("/srv/ubuntu/pool/main/p/parted/parted_1.7.1-2.1ubuntu3_i386.deb", O_RDONLY|O_LARGEFILE
) = 10
fcntl64(10, F_SETFD, FD_CLOEXEC)        = 0
sendfile64(8, 10, [0], 55200)           = -1 EAGAIN (Resource temporarily unavailable)
setsockopt(8, SOL_TCP, TCP_CORK, [0], 4) = 0
write(3, "66.230.200.243 apt.wikimedia.org - [06/Dec/2006:00:36:34 +0000] \"GET /ubuntu/pool
/main/p/parted/parted_1.7.1-2.1ubuntu3_i386.deb HTTP/1.1\" 200 0 \"-\" \"Ubuntu APT-HTTP/1.3
\"\n", 171) = 171
close(10)                               = 0
shutdown(8, 1 /* send */)               = 0

lighty receives a EAGAIN on sendfile() (buffer full?) and then shuts down the sending to the socket and logs a 200 OK?!?

The following code in network_linux_sendfile.c seems responsible for this:

            if (-1 == (r = sendfile(fd, c->file.fd, &offset, toSend))) {
                    switch (errno) {
                    case EAGAIN:
                    case EINTR:
                            r = 0;
                            break;
                    case EPIPE:
                    case ECONNRESET:
                            return -2;
                    default:
                            log_error_write(srv, __FILE__, __LINE__, "ssd",
                                            "sendfile failed:", strerror(errno), fd);
                            return -1;
                    }
            }
            if (r == 0) {
                    /* We got an event to write but we wrote nothing
                     *
                     * - the file shrinked -> error
                     * - the remote side closed inbetween -> remote-close */
                    if (HANDLER_ERROR == stat_cache_get_entry(srv, con, c->file.name, &sce)) {
                        /* file is gone ? */
                        return -1;
                    }
                    if (offset > sce->st.st_size) {
                        /* file shrinked, close the connection */
                        return -1;
                    }
                    return -2;
            }

If EAGAIN, r is set to 0. In the following code block, it's assumed that either the source file has shrunk (not the case) or the remote end must have closed the connection (not the case either). The latter seems strange to me - shouldn't ECONNRESET be returned for that?

EAGAIN likely means that some buffer is full, in which case this code returns -2 which makes lighty close down the connection.

I disabled the return -2, which seems to fix this issue. However, there seems to be another bug occurring with APT...

...and the other issue was simply a too low server.max-keep-alive-requests, which is not set to 128 by default as the manual used to say, but to 16.

lighttpd correctie closed the connection after 16 requests with Connection: close, but apparently APT doesn't handle that correctly.

Hi,

we have the same problems with debian.netcologne.de and deb.grml.org where I tried lighttpd. A fix would be really appreciated.

Thanks Alex

a patch for the EAGAIN was applied in [2072]