## **🤖 What the warning means**

Search Atlas reports this warning when your **robots.txt** file appears to block crawlers used by AI services, such as large language model (LLM) crawlers. The warning does not automatically mean your website is broken or that search engine visibility is affected.

A rule such as **User-agent: \*** followed by **Disallow: /** blocks every crawler from the entire website, including AI crawlers. Rules that block only selected folders, checkout pages, cart pages, internal search results, or URL parameters may be intentional and are not necessarily problems.

## **🔍 Check which rule triggered the warning**

1. Open your website's robots.txt file by visiting **https://your-domain.com/robots.txt**.
2. Look for **User-agent: \*** and any broad **Disallow** rules.
3. Check whether the rule blocks the entire site with **Disallow: /**, or only specific paths.
4. Review whether named AI crawlers, such as GPTBot, ClaudeBot, Google-Extended, or other AI user agents, are blocked.

Do not remove rules automatically. E-commerce protections, private areas, duplicate URL controls, and parameter-based disallows can be valid. The key concern is whether important public pages are unintentionally blocked from AI crawlers.

## **🛠️ What to change**

If you want AI services to access your public content, ask your website platform whether it can remove or narrow the wildcard block. A safer setup usually allows public content while continuing to block private areas, checkout pages, account pages, and internal search URLs.

If the entire site is intentionally private, under construction, or restricted, you may leave the rule in place. In that case, the warning is informational and no change is required.

## **🏗️ Guidance for closed website builders**

Website builders often generate robots.txt automatically. You may not have access to edit the file directly, so do not follow instructions that refer to uploading robots.txt to a hosting root directory or changing it through WordPress plugins such as Yoast or Rank Math.

- **Simvoly:** Check the site's SEO, visibility, and indexing settings. If Simvoly does not provide a robots.txt editor or crawler-specific controls, send Simvoly support the exact robots.txt URL and ask whether the wildcard rule can be narrowed to allow AI crawlers on public pages.
- **Wix:** Review the site's SEO settings and robots.txt customization options available in the Wix dashboard. If the relevant rule is platform-controlled, ask Wix whether it can be changed for your site.
- **Squarespace:** Review the site's SEO and page visibility settings. Squarespace may manage robots.txt automatically, so contact Squarespace through its available support process if a platform-generated rule blocks public content.
- **Shopify:** Review the store's robots.txt customization options and theme or app settings. Keep protections for checkout, customer accounts, carts, and internal search, and change only broad rules that block public content unnecessarily.

Platform features can change. If you cannot find a relevant setting, ask the builder to confirm whether the robots.txt file is editable and whether its wildcard rules are intentional.

## **💬 What to tell your platform provider**

Send the provider your robots.txt URL and ask: **“Does this file block AI crawlers from all public pages? Can you allow AI crawlers while keeping private, checkout, account, cart, and internal search URLs blocked?”**

After any change, review the updated robots.txt file and rerun the Site Audit. Changes may take time to appear in crawler results.

## **✅ When no action is needed**

You may not need to make changes when the rules block only non-public or low-value areas, when you intentionally restrict AI access, or when the builder confirms the rules are required for its platform. Search Atlas cannot override a robots.txt file controlled by your website provider.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type **human teammate** to be instantly connected with a member of our team.