Skip to main content
Loading...

https://socialgem.net/read-blog/29907

URL is Crawlable
User-agent: *
Disallow: /assets
Disallow: /cache
Disallow: /sources
Disallow: /api
Disallow: /script_backups
Disallow: /updates
Disallow: /install
Disallow: /admincp
Disallow: /admin-panel
Disallow: /ajax_loading.php
Disallow: /api.php
Disallow: /xml
Disallow: /system_status.php
Disallow: /nodejs
Disallow: /requests.php

User-agent: amazon-QBusiness # Amazon Q Web Crawler
Disallow: / # disallow access to any pages
User-agent: AhrefsBot/7.0
Disallow: /
User-agent: DotBot/1.2
Disallow: /
User-agent: SemrushBot/7~bl
Disallow: /
robots.txt
Lines in robots.txt

Page tested on 19th July 2025 at 11:04