lighty 1.4.13 + php/fastcgi hangs after a short time of running

It seemed that lighty failed to connect to php-fastcgi. I saw

switch(fcgi_write_request(srv, hctx)) { case HANDLER_ERROR:

: : :

if (hctx->reconnects < 5) {

fcgi_reconnect(srv, hctx); joblist_append(srv, con); /* in case we come from the event-handler */

return HANDLER_WAIT_FOR_FD;

} else {

fcgi_connection_close(srv, hctx);

buffer_reset(con->physical.path); con->mode = DIRECT; con->http_status = 500; joblist_append(srv, con); /* in case we come from the event-handler */

return HANDLER_FINISHED;

}

I closed zlib output compression in php.ini, after a restart, lighty runs perfectly for about TWO hours w/o any problems.

After running 6hrs, lighty halted again w/o php's zlib output compression.

I did a tail -f access.log | grep ' 500 ' trace. When lighty halted, there would be tons of '500's in the access log and *FEW* normal replies (200,206,301,304, and even 404, etc.).

It seems the communication between lighty and php/fastcgi has some trouble.

I added extra error logs in mod_fastcgi.c where it sends 500 to the client, and in lighty.error.log I finally found:

[ honeymoon period ]
2006-10-22 01:30:48: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 01:30:49: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 01:30:58: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 01:31:05: (mod_fastcgi.c.3005) ndx==-1  
[ tragedy period ]
[ I restarted lighty then ]
[ honeymoon period ]
2006-10-22 10:15:51: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 10:15:52: (mod_fastcgi.c.3005) ndx==-1 
[ tragedy period again ]

So, the problem perhaps happens at:

        if (hctx->host == NULL) {
                size_t k;
                int ndx, used = -1;
                   
                /* get best server */
                for (k = 0, ndx = -1; k < hctx->ext->used; k++) {
                        host = hctx->ext->hosts[k];
                   
                        /* we should have at least one proc that can do something */
                        if (host->active_procs == 0) continue;
                   
                        if (used == -1 || host->load < used) {
                                used = host->load;

                                ndx = k;
                        }
                }       

                /* found a server */
                if (ndx == -1) {
                        /* all hosts are down */

                        fcgi_connection_close(srv, hctx);

                        con->http_status = 500;
                        con->mode = DIRECT;

log_error_write(srv,__FILE__,__LINE__,"ss","ndx==-1");
     
                        return HANDLER_FINISHED;
                }

Maybe lighty cannot find a fcgi host to work because

                        if (host->active_procs == 0) continue;

All hosts are not activated in lighty's eyes?

I've tried the reconnect thresold from 5 to 50000 and disable_time from 60 to 1.

Failed again. The error log reads:

2006-10-22 16:15:25: (mod_fastcgi.c.1705) connect failed: Connection refused on unix:/tmp/php1.socket-0
2006-10-22 16:15:25: (mod_fastcgi.c.2817) backend died, we disable it for a 5 seconds and send the request to another backend instead: reconnects: 0 load: 1447
2006-10-22 16:15:25: (mod_fastcgi.c.3040) FCGI_STATE_INIT FCGI_STATE_CONNECT_DELAYED, active_procs--
2006-10-22 16:15:25: (mod_fastcgi.c.3048) HANDLER_ERROR, reconnects<50000 -2078608690
2006-10-22 16:15:25: (mod_fastcgi.c.3006) ndx==-1             
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3436) all handlers for  /w.php on .php are down.
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL    
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL    
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL

Is there a backup mechanism to re-create fastcgi servers when current servers are dead?

During testing, do not enable kern.timecounter.hardware=TSC, TSC on SMP is a little unreliable and quirky. Also, paste your dmesg so we can see what the kernel thinks.