Fixes for robots.txt handling (bug introduced by previous commit).