🗺️ Fix Topical Map and Crawl Settings Errors

🔍 Overview

When creating a topical map in Search Atlas, you may run into two common issues: a URL validation error that greys out the input field, or OTTO only returning results for one page even though your site has many more. Both issues are usually caused by incorrect URL formatting or misconfigured crawl settings. This article walks you through how to fix each problem.

⚠️ Issue 1: URL Field Shows a Validation Error and Is Greyed Out

This error appears when the URL entered does not match the expected format. The field becomes uneditable until the issue is resolved.

Common causes:

• Missing or incorrect protocol (e.g., typing www.example.com instead of https://www.example.com)
• A trailing space before or after the URL
• Special characters or unsupported symbols in the URL
• Entering a URL path instead of the root domain (e.g., https://example.com/blog instead of https://example.com)

How to fix it:

1. Go to Left sidebar → Content → Topical Maps.
2. Click to create a new topical map.
3. Clear the URL field completely.
4. Re-enter your URL in the correct format: https://www.yourdomain.com — include the protocol and avoid any trailing slashes or spaces.
5. Wait for the validation indicator to confirm the URL is accepted before proceeding.

📄 Issue 2: OTTO Only Shows Results for One Page

If OTTO is crawling your site but only returning data for a single page, your Pages to Crawl limit or project URL is likely limiting the number of pages analysed.

Common causes:

• Pages to Crawl is set to 1 (or a very low number), meaning only the root URL is crawled
• The project was set up on the wrong URL (a single page or subdirectory instead of the root domain)
• The sitemap has not been submitted or is not accessible
• The website blocks crawlers via robots.txt rules (only relevant if Crawl Criteria is set to Respect robots.txt in your crawl settings)

How to fix it:

1. Navigate to Left sidebar → OTTO SEO → SEO Automation (URL: /seo-automation-v3).
2. Open your project and click the Autopilot / Smart Crawl card to open the OTTO SEO Settings modal.
3. Increase Pages to Crawl so it covers the number of pages your site contains (adding a small buffer).
4. Ensure the crawl scope is set to your full root domain, not a specific subfolder or page.
5. If you have a sitemap, confirm the URL is entered correctly in the sitemap field (e.g., https://www.yourdomain.com/sitemap.xml).
6. Save your settings and re-run the crawl.

Once the crawl completes, return to Left sidebar → Content → Topical Maps and generate or refresh your topical map. You should now see data across all crawled pages.

✅ Best Practices for Topical Map Setup

• Always use the full root domain with https:// when entering your URL.
• Set Pages to Crawl high enough to cover your full site (with a small buffer).
• Submit an XML sitemap to help OTTO discover all pages faster.
• Check your robots.txt file if Crawl Criteria is set to Respect robots.txt — otherwise OTTO ignores it.
• Re-run the crawl after any major site changes before generating a new topical map.

💬 Still Need Help?

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.