Jump to content
Akinix
Sign in to follow this  
C.M.

How to block Ahrefs, Semrush, Serpstat, Majestic SEO, MegaIndex, and similar bots for competitive intelligence

Recommended Posts

In robots.txt:

# Alexa: https://support.alexa.com/hc/en-us/articles/200450194-Alexa-s-Web-and-Site-Audit-Crawlers
User-agent: ia_archiver
Disallow: /

# archive.org
User-agent: archive.org_bot
Disallow: /

# Ahrefs: https://ahrefs.com/robot
User-agent: AhrefsBot
Disallow: /

# MOZ: https://moz.com/help/moz-procedures/crawlers/rogerbot
User-agent: rogerbot
Disallow: /

# MOZ: https://moz.com/help/moz-procedures/crawlers/dotbot
User-agent: dotbot
Disallow: /

# DataForSeo https://dataforseo.com/dataforseo-bot
User-agent: DataForSeoBot
Disallow: /

# Semrush: https://www.semrush.com/bot/
User-agent: SemrushBot
Disallow: /

User-agent: SiteAuditBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-SI
Disallow: /

User-agent: SemrushBot-SWA
Disallow: /

User-agent: SemrushBot-CT
Disallow: /

User-agent: SemrushBot-BM
Disallow: /

User-agent: SplitSignalBot
Disallow: /

# Majestic: https://mj12bot.com/
User-agent: MJ12bot
Disallow: /

# SerpStat: https://serpstatbot.com/
User-agent: serpstatbot
Disallow: /

# MegaIndex: https://ru.megaindex.com/blog/seo-bot-detection
User-agent: MegaIndexBot
Disallow: /

# SEO-PowerSuite-bot: https://www.link-assistant.com/seo-workflow/site-audit.html
User-agent: SEO-PowerSuite-bot
Disallow: /

User-agent: *
Disallow:

Share this post


Link to post
Share on other sites
Sign in to follow this  

×
×
  • Create New...