🧭 Overview

OTTO, Search Atlas's OTTO SEO automation engine, crawls your site to discover and apply on-page optimizations. By default, OTTO will process all pages it can reach within your project. If you want to prevent OTTO from crawling or optimizing certain pages — such as Spanish-language pages, staging URLs, or low-priority sections — you do this through the URL Exclusion conditions in your project's OTTO SEO Settings inside Search Atlas, not through robots.txt or noindex tags.

This article explains how to set up OTTO-specific exclusions, clarifies whether OTTO changes are permanently written to your site, and covers how OTTO's reporting and integration features work.

⚠️ OTTO Exclusions vs. robots.txt — Key Difference

A common point of confusion is the difference between site-wide crawl rules and OTTO-specific exclusions.

• robots.txt / noindex tags — These are server-level directives that instruct all crawlers (Google, Bing, OTTO, etc.) how to behave. Editing robots.txt does not exclusively affect OTTO and is not the correct method for managing what OTTO processes.
• URL Exclusion conditions — These are exclusion rules configured entirely within Search Atlas. They tell OTTO which URLs to skip during its crawl and optimization passes, without touching your site's server configuration or affecting other crawlers.

Always use URL Exclusion conditions when you want to control what OTTO specifically crawls or optimizes.

🚫 How to Exclude Pages from OTTO Crawl

1. In the left sidebar, click OTTO SEO and open SEO Automation (All Sites view at /seo-automation-v3), then open your project.
2. At the top of the project overview, click the Autopilot / Smart Crawl card to open the OTTO SEO Settings modal. You can also reach it from the All Sites table by clicking the Smart Crawl status text on your site's card.
3. Scroll to URL Exclusion conditions and click + Add condition.
4. Set the condition to Exclude all URLs that Contains (or Start With) the text you want to skip — for example, Contains /es/ excludes all Spanish-language pages under a language subdirectory.
5. Click Update Settings.

When your exclusion rules are saved with Update Settings, OTTO will skip all matching URLs during its next crawl cycle. Any optimizations OTTO had already applied to those pages before the exclusion was added will remain in place unless you manually revert them.

🔄 Dynamic vs. Permanent Changes

A frequent question is whether OTTO's optimizations are written permanently to your site.

• Dynamic changes — Some OTTO optimizations are applied dynamically and are managed through Search Atlas. If you disconnect OTTO or remove a rule, those changes can be rolled back without a permanent edit to your CMS or server files.
• Permanent changes — Other optimizations (such as those pushed directly to your CMS) are written to your site and persist independently of OTTO. These require manual intervention to revert.

If you are unsure whether a specific OTTO action is dynamic or permanent, contact our support team before making changes.

📊 Reporting and Integrations

OTTO provides reporting on the optimizations it has applied across your project. You can review which pages have been processed, which changes are active, and track the impact of OTTO's work over time. OTTO also integrates with your CMS, allowing certain optimizations to be pushed directly to your site from within Search Atlas.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.