Home › Guides › How Do You Let AI Search Bots Read Your Contractor Website? robots.txt Guide

How Do You Let AI Search Bots Read Your Contractor Website?

By Ansh Singh · Updated November 8, 2026 · 9 min read

Short answer: To let AI search bots read a contractor website, make sure its robots.txt file does not block OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot or Bingbot. They find sites for ChatGPT, Perplexity, Claude and Google. Most small sites already allow them, but check, because one stray line can shut them out.

What is robots.txt?

Robots.txt is a plain text file on your website that tells crawlers which pages they may visit. A crawler is a program that reads web pages so a search or AI system can learn what is on them.

Google describes it as a file that tells crawlers which URLs they can access on your site. Its main use is to manage crawler traffic. You can see yours by typing your domain and /robots.txt into a browser, for example yoursite.com/robots.txt.

Two points to keep straight:

• Robots.txt is a request. Google's page says the instructions cannot enforce crawler behavior. Well-run crawlers follow it. Bad ones do not.
• Robots.txt is not how you hide a page from Google. Google says it is not a mechanism for keeping a page out of Google. For that, use a noindex tag or a password.

Why does this matter for a contractor?

When someone asks ChatGPT "who is a good roofer in Galesburg, Illinois (an example town)," the tool needs to find sites that answer. If its search bot is blocked from your site, you give it nothing to read.

OpenAI says plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. They can still show up as plain links.

This block usually happens by accident. A website builder or a security plugin adds a rule that shuts out all unknown bots. A developer left "Disallow: /" on from when the site was being built. Nobody notices, because the site still works fine for people.

If you want the full picture of how these tools pick a business, read our guide on how to get recommended by ChatGPT as a contractor. This page covers only the technical gate.

Which AI bots exist, and what does each do?

There are two kinds that matter. Search bots find pages so a tool can show them in answers. Training bots collect pages to teach a model. Each company runs both and keeps them separate.

Perplexity says PerplexityBot is not used to crawl content for AI foundation models. It is a search bot only.

Leave Bingbot allowed too, since Bing is a search engine other tools and people use. Our guide on Bing Places for contractors covers the listing side.

Should you block the training bots?

For most contractors, no. But it is your call, and each choice stands alone.

Here is the honest trade. Search bots decide whether you can show up in answers today. Training bots decide whether your pages go into the material a model learns from. Blocking a training bot does not remove you from search answers. OpenAI says each robots.txt setting is independent, and that blocking GPTBot signals your content should not be used for training.

Our view, which is an opinion and not a rule: a plumber or roofer who wants to be known to AI tools has little to gain by blocking training bots. Your pages are public advertising for your business. If you run a paid course or members-only material, block the training bots on that part of the site.

Google-Extended deserves its own note. Google says it controls whether crawled content may be used to train Gemini models, and that it does not affect inclusion in Google Search nor is it a ranking signal. So blocking it does not hurt your normal Google rankings.

How do you set up robots.txt for AI search bots?

Step 1: Open your current file

Type yoursite.com/robots.txt in a browser. If you see a "not found" message, you have no file, which means every bot is allowed by default. If you see lines, read them.

Step 2: Look for the bad lines

The danger signs are these:

• Disallow: / under User-agent: *. That blocks everything for everyone.
• A list of AI bots each followed by Disallow: /. Some security tools add these by default.
• A Disallow line for pages you want found, such as /services/.

Step 3: Write the file you want

Here is a safe default for a contractor. It allows every crawler and keeps private areas out.

``` User-agent: * Allow: / Disallow: /wp-admin/

User-agent: OAI-SearchBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: Claude-SearchBot Allow: /

Sitemap: https://www.yoursite.com/sitemap.xml ```

The first block covers all bots. The named blocks make your intent clear for the main AI search bots. Change /wp-admin/ to match your system, or delete that line.

If you want to block one training bot, add a block for it:

`` User-agent: GPTBot Disallow: / ``

Anthropic gives the same style of rule for ClaudeBot, and notes it will honor it. Put each rule in its own block, with a blank line between blocks.

Step 4: Put it at the top level

The file must be named robots.txt and sit at the root of your domain. A file at /files/robots.txt does nothing. Anthropic's help page says to put it in your top-level directory and to repeat it for each subdomain.

If you use a builder such as Wix or Squarespace, you may not be able to edit the file. Google notes that sites on some systems may not be able to edit robots.txt directly and may get a search settings page instead. Look there for a switch that says something like "let search engines index this site," and make sure it is on.

Step 5: Wait, then test

OpenAI says changes to search-related robots.txt settings can take about 24 hours to take effect. Perplexity gives the same time, up to 24 hours. Check the file in a browser again the next day. Then ask ChatGPT or Perplexity a question about your trade and town, and see whether your site shows up. Do not expect an instant jump. Finding a site and choosing to recommend it are two different steps. Our guide on how to check if ChatGPT recommends your business shows a simple way to test.

What about bots that ignore robots.txt?

Two of the bots above are special. They visit a page because a real person asked a question.

OpenAI says ChatGPT-User visits are started by users, so robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt because a person requested the fetch.

For you this means: you cannot use robots.txt to stop a person from asking an AI tool about your page. And you do not need to. If someone pastes your link into ChatGPT, you want it to open.

If you need to block traffic from the actual bad actors, robots.txt is the wrong tool. That job belongs to your host or a firewall. Anthropic also advises against blocking its bots by IP address, since that can stop them from reading your robots.txt and may not reliably opt you out.

Does robots.txt control Google's AI Overviews?

Google's page on AI features says you do not need any special files or markup to appear in them. Google's own crawling rules say Googlebot controls how your pages are read for Search and its features, so the first gate is simple: can Googlebot read your pages.

That is why Googlebot should never be blocked on a contractor site. Google-Extended is a separate switch, and it is not the one that decides AI Overviews placement for your rankings.

What else should you check at the same time?

Robots.txt is one gate among several. While you are in the settings, check these:

• A sitemap. A sitemap is a list of your pages. The line Sitemap: ... in the file points to it. Our guide on Google Search Console for contractors shows how to submit it.
• No noindex tag. A noindex tag on a page tells search engines to leave it out. Builders sometimes leave it on while a site is in draft mode.
• Service pages that load as text. If your services sit inside images or scripts, bots may see an empty page.
• Clean labels. See schema markup for contractors for the labels that help machines read your facts.

Different trades face the same gate. A plumber's guide and an HVAC guide both start with this same check. If you want us to handle the file and the rest, see a free demo site.

Common questions

How do I allow ChatGPT to crawl my website?

Make sure your robots.txt does not block OAI-SearchBot. That is the bot OpenAI uses to find sites for ChatGPT search. If you have no robots.txt file, it is allowed by default. If you add one, include a block that says User-agent: OAI-SearchBot and Allow: /.

Is it safe to allow GPTBot?

It is safe in the sense that it only reads public pages. GPTBot collects content that may train OpenAI's models. Blocking it signals you do not want that, and blocking it does not remove you from ChatGPT search. Decide based on how you feel about training, not on rankings.

Will robots.txt hide my pages from Google?

No. Google says robots.txt is not a way to keep a page out of Google, and a blocked address can still appear in results without a description. Use a noindex tag or a password for pages that must stay private.

How long until a robots.txt change works?

OpenAI and Perplexity both say about 24 hours for their search bots. Check the file in your browser after saving it, and again the next day.

Do I need a robots.txt file at all?

No. With no file, bots may crawl everything. A file is useful when you want to keep a login area or admin pages out, or to point bots to your sitemap. If your builder makes one for you, read it once to be sure it is not blocking the wrong things.

Sources

• Google Search Central: Introduction to robots.txt (2026)
• Google Search Central: Google's common crawlers (2026)
• Google Search Central: AI features and your website (2026)
• OpenAI: Overview of OpenAI crawlers (2026)
• Perplexity: Perplexity crawlers (2026)
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (2026)