Jump to content
Akinix
Sign in to follow this  
C.M.

How to block Ahrefs, Semrush, Serpstat, Majestic SEO, MegaIndex, and similar bots for competitive intelligence

Recommended Posts

In robots.txt:

# Alexa: https://support.alexa.com/hc/en-us/articles/200450194-Alexa-s-Web-and-Site-Audit-Crawlers
User-agent: ia_archiver
Disallow: /

# Ahrefs: https://ahrefs.com/robot
User-agent: AhrefsBot
Disallow: /

# MOZ: https://moz.com/help/moz-procedures/crawlers/rogerbot
User-agent: rogerbot
Disallow: /

# MOZ: https://moz.com/help/moz-procedures/crawlers/dotbot
User-agent: dotbot
Disallow: /

# Semrush: https://www.semrush.com/bot/
User-agent: SemrushBot
Disallow: /

User-agent: SiteAuditBot
Disallow: /

User-agent: SemrushBot-BA
Disallow: /

User-agent: SemrushBot-SI
Disallow: /

User-agent: SemrushBot-SWA
Disallow: /

User-agent: SemrushBot-CT
Disallow: /

User-agent: SemrushBot-BM
Disallow: /

User-agent: SplitSignalBot
Disallow: /

# Majestic: https://mj12bot.com/
User-agent: MJ12bot
Disallow: /

# SerpStat: https://serpstatbot.com/
User-agent: serpstatbot
Disallow: /

# MegaIndex: https://ru.megaindex.com/blog/seo-bot-detection
User-agent: MegaIndexBot
Disallow: /

# SEO-PowerSuite-bot: https://www.link-assistant.com/seo-workflow/site-audit.html
User-agent: SEO-PowerSuite-bot
Disallow: /

User-agent: *
Disallow:

Please note: comments are provided for convenience, but some robots do not handle robots.txt files with comments correctly. Therefore, it is advisable to remove them for use in production.

Edited by C.M.

Share this post


Link to post
Share on other sites
Sign in to follow this  

×
×
  • Create New...