Improving the Throughput of File Transfers

Jan has blogged about the development of async io and threaded stat() at

http://blog.lighttpd.net/articles/tag/aio

vmstat

<pre> $ vmstat 5 </pre>

If you are using a threaded io-backend (e.g. gthread-aio) and see that the b column reaches the value of server.max-read-threads you know that now all threads are waiting for the disk to respond.

iostat

<pre> $ iostat -x 5 </pre>

r/s is the number of requests per second which are handled by the disk. Due to the nature of hard-disks there is a upper limit you can get here:

<pre> max-rps = (disk-rpm / 60) * 2

((7200 rot/min) / (60 s/min)) * 2 seek/rot = 120 * 2 seek/rot = 240 seek/s </pre>

For each request handle by the disk it has to do a seek. In the best case the next seek is just around the corner, in the worst case it just passed over the block to read and has to do a full rotation to get to the place again. On average this means it has to do half a rotation for a seek or 2 seeks per rotation.

If you have more disks the max-rps sums up IF your data is well distributed over all disks.

util% of 100% means that the disk is either at the seek-limit (see r/s) or the throughput limit.