From: Wolfram Schneider <wosch@indexdata.dk>
Date: Wed, 21 Apr 2010 12:20:22 +0000 (+0200)
Subject: add missing pages and reorder entries
X-Git-Tag: CPAN-v1.02~119
X-Git-Url: http://git.indexdata.com/?p=irspy-moved-to-github.git;a=commitdiff_plain;h=45ba11c57aa05265267660fb344851d4fa9f1f35;hp=15dce7cdea43a6606ddf025aa236d429d7ff2910

add missing pages and reorder entries

Google has crawled more than 10.900 pages from
irspy.indexdata.com - how stupid!
---

diff --git a/web/htdocs/robots.txt b/web/htdocs/robots.txt
index beef7ee..d19bcff 100644
--- a/web/htdocs/robots.txt
+++ b/web/htdocs/robots.txt
@@ -1,8 +1,11 @@
 User-agent: *
-Disallow: /full.html
-Disallow: /all.html
+Disallow: /admin/
 Disallow: /add.html
-Disallow: /upload.html
+Disallow: /all.html
+Disallow: /doc.html
+Disallow: /find.html
+Disallow: /full.html
 Disallow: /raw.html
-Disallow: /admin/
+Disallow: /stats.html
+Disallow: /upload.html