User-agent: * Disallow: /nlwin User-agent: Googlebot Disallow: */misc/* Disallow: */private/* Disallow: *.tmp Disallow: *.tmp User-agent: MSNbot Crawl-delay: 5 # See: http://www.robotstxt.org/wc/norobots-rfc.html # # This table illustrates some examples: # # Record Path URL path Matches # /tmp /tmp yes # /tmp /tmp.html yes # /tmp /tmp/a.html yes # /tmp/ /tmp no # /tmp/ /tmp/ yes # /tmp/ /tmp/a.html yes # # /a%3cd.html /a%3cd.html yes # /a%3Cd.html /a%3cd.html yes # /a%3cd.html /a%3Cd.html yes # /a%3Cd.html /a%3Cd.html yes # # /a%2fb.html /a%2fb.html yes # /a%2fb.html /a/b.html no # /a/b.html /a%2fb.html no # /a/b.html /a/b.html yes # # /%7ejoe/index.html /~joe/index.html yes # /~joe/index.html /%7Ejoe/index.html yes