robots.txt is a simple text file that gives instructions to automated crawlers. It can allow or disallow specific bots from specific paths, and it's often where sites accidentally block search or AI crawlers they actually want to let in.
Because AI answer engines rely on their own crawlers (GPTBot, ClaudeBot, PerplexityBot), a robots.txt that blocks them - deliberately or not - can make you invisible in AI answers even when your SEO is otherwise strong.
Further reading
Related terms
Want to see how your page measures up on this and more?