C.M. Posted September 26, 2021 Share Posted September 26, 2021 В robots.txt: # Alexa: https://support.alexa.com/hc/en-us/articles/200450194-Alexa-s-Web-and-Site-Audit-Crawlers User-agent: ia_archiver Disallow: / # archive.org User-agent: archive.org_bot Disallow: / # Ahrefs: https://ahrefs.com/robot User-agent: AhrefsBot Disallow: / # MOZ: https://moz.com/help/moz-procedures/crawlers/rogerbot User-agent: rogerbot Disallow: / # MOZ: https://moz.com/help/moz-procedures/crawlers/dotbot User-agent: dotbot Disallow: / # DataForSeo https://dataforseo.com/dataforseo-bot User-agent: DataForSeoBot Disallow: / # Semrush: https://www.semrush.com/bot/ User-agent: SemrushBot Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: SemrushBot-CT Disallow: / User-agent: SemrushBot-BM Disallow: / User-agent: SplitSignalBot Disallow: / # Majestic: https://mj12bot.com/ User-agent: MJ12bot Disallow: / # SerpStat: https://serpstatbot.com/ User-agent: serpstatbot Disallow: / # MegaIndex: https://ru.megaindex.com/blog/seo-bot-detection User-agent: MegaIndexBot Disallow: / # SEO-PowerSuite-bot: https://www.link-assistant.com/seo-workflow/site-audit.html User-agent: SEO-PowerSuite-bot Disallow: / User-agent: * Disallow: Link to comment Share on other sites More sharing options...
Recommended Posts