The Lemur Web Crawler

Opt-Out Form

If you wish to exclude a page, a portion of a web site, or an entire web site from our crawl, please fill in the form below and then press the Submit button. An email will be sent automatically to confirm that we received your request. A followup email will be sent after the the crawler is updated to exclude your page(s).

Please enter your page URL or domain name in the following format:

http://www.example.com/
To exclude an entire site
http://sensitive.example.com/
To just exclude everything below "sensitive.example.com", but allow other parts of example.com to be crawled.
http://www.example.com/sensitive.html
To exclude a single sensitive web page from the crawl

After you make an opt out request, the crawler will be instructed to exclude the specified page(s) from future crawling activity, and to discard any page(s) that it has collected already. Please note that the opt-out process is not immediate.

Although we try to process requests quickly, it may take some time before the crawler is informed. During that time, your site may continue to be accessed.