URL is Crawlable
# You know, for robots
#
# [email protected]
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: */format:thumbnail
Disallow: /download/
Disallow: /*/download?
Disallow: /couchmode/
Disallow: /groups/*/forum/*
Disallow: /musicstore/preview
Disallow: /musicstore/download
Disallow: /forgot_password
Disallow: /help/forgot_login
Disallow: /help/verify_email
Disallow: /help/zendesk_sso
Disallow: /help/sso
Disallow: /user/verify
Disallow: /search
Disallow: /_*
Allow: /_rv/viewer
Allow: /_next/viewer
Allow: /_rv/gtm
Allow: /_next/gtm
Allow: /_global_notifications
Allow: /_rv/jwt
Allow: /_next/jwt
Allow: /_rv/title
Allow: /_next/title
Crawl-delay: 2
User-agent: SeznamBot
Request-rate: 1/10s
Disallow: */format:thumbnail
Disallow: /download/
Disallow: /*/download?
Disallow: /couchmode/
Disallow: /musicstore/preview
Disallow: /musicstore/download
Disallow: /forgot_password
Disallow: /help/forgot_login
Disallow: /help/verify_email
Disallow: /help/zendesk_sso
Disallow: /help/sso
Disallow: /api/
Disallow: /user/verify
Disallow: /search/
Disallow: /search?
Disallow: /search\?
Disallow: /_*
User-agent: msnbot
Crawl-delay: 5
User-agent: Bingbot
Crawl-delay: 5
User-agent: AdIdxBot
Crawl-delay: 5
User-agent: BingPreview
Crawl-delay: 5
User-agent: Yandex
Crawl-delay: 2
# Block Open AI
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
# Block Google (Gemini)
User-agent: Google-Extended
Disallow: /
# Block Claude
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Claudebot
Disallow: /
# Block CommonCrawl
User-agent: CCBot
Disallow: /
# Block Diffbot
User-agent: Diffbot
Disallow: /
# Block Meta (Facebook)
User-agent: FacebookBot
Disallow: /
# Block ByteDance
User-agent: Bytespider
Disallow: /
# Block Webz.io
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
# Block ImagesiftBot
User-agent: ImagesiftBot
Disallow: /
# Block Meltwater
User-agent: Meltwater
Disallow: /
# Block Seekr
User-agent: Seekr
Disallow: /
robots.txt
Lines in robots.txt
Page tested on 5th November 2024 at 19:22
What Else Could We Find on This Page?
| Page Authority: | 0 | View Report |
|---|---|---|
| Page Speed: | 85/100 Mobile / 74/100 Desktop | View Report |
| Page Language: | en | View Report |
| Meta Description: | 158 characters | View Report |