add missing pages and reorder entries
authorWolfram Schneider <wosch@indexdata.dk>
Wed, 21 Apr 2010 12:20:22 +0000 (14:20 +0200)
committerWolfram Schneider <wosch@indexdata.dk>
Wed, 21 Apr 2010 12:20:22 +0000 (14:20 +0200)
Google has crawled more than 10.900 pages from
irspy.indexdata.com - how stupid!

web/htdocs/robots.txt

index beef7ee..d19bcff 100644 (file)
@@ -1,8 +1,11 @@
 User-agent: *
-Disallow: /full.html
-Disallow: /all.html
+Disallow: /admin/
 Disallow: /add.html
-Disallow: /upload.html
+Disallow: /all.html
+Disallow: /doc.html
+Disallow: /find.html
+Disallow: /full.html
 Disallow: /raw.html
-Disallow: /admin/
+Disallow: /stats.html
+Disallow: /upload.html