Saturday, January 29, 2011

Equivalent of `nice` for filesystem IO?

When I'm running an rsync backup job which involves copying over large files, the machine running the backup (both Mac desktops and Linux servers) grinds to a halt and the load average goes through the roof.

I've tried:

  • niceing the rsync process (doesn't help - the bottleneck is the disk)
  • On Linux, reniceing a kjournald (helps, but seems like a hack and doesn't work on the Mac)
  • Using the --bwlimit rsync flag (helps, but it means all the transfers are slow - even when they don't need to be)

So, is there any way I can "nice" rsync's IO so the machines are useable while the backup is running?

PS: I'm aware of the dangers of rsync on the Mac… But I've used BackupBouncer to verify my backups, though, and they seem OK.

  • It looks like the setpriority API on Mac OS X is supposed to be able to alter IO scheduling (see http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/setpriority.2.html). I don't have any MacOS handy so I can't test that nice actually changes the IO priority.

    On the Linux side, ionice is what you're looking for.

    David Wolever : Ah, thanks. Here's what the `setpriority` manpage says: "When setting a thread into background state the scheduling priority is set to lowest value, disk and network IO are throttled." -- which seems to indicate that disk/network IO are only throttled when something is niced down to 20 (the "background state").
    David Wolever : Some quick testing also seems to confirm this - when the backup script was only niced to 10 (ie, `nice backup`), I noticed a slowdown... But when it's niced down to 20 (ie, `nice -n 20 backup`), I don't notice any slowdown.
    David Wolever : (and by "noticed a slowdown" I mean "noticed that interactive programs running on the computer didn't respond as quickly as they normally do")
    Evan Anderson : It's a little disappointing that you only get throttling at a priority of 20, though.
    Avery Payne : +1, Nice answer. Always wondered about twiddling IO.
  • On linux you can use ionice http://linux.die.net/man/1/ionice

0 comments:

Post a Comment