URL is Crawlable
Sitemap: https://nerdbot.com/sitemap_index.xml
User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /?s=*
Disallow: /search/
Disallow: /page/*/?s=
# Block AI crawlers from search entirely
User-agent: GPTBot
Disallow: /?s=
Disallow: /search/
User-agent: Applebot
Disallow: /?s=
Disallow: /search/
User-agent: bingbot
Disallow: /?s=
Disallow: /search/
robots.txt
Lines in robots.txt
Page tested on 7th February 2026 at 00:29
What Else Could We Find on This Page?
| Page Authority: | 0 | View Report |
|---|---|---|
| Page Speed: | 1/100 Mobile / 94/100 Desktop | View Report |
| Page Language: | View Report | |
| Meta Description: | 157 characters | View Report |