🔍 Overview
Search Atlas needs to access your website in order to crawl pages, run audits, and power OTTO SEO. If your staging environment is protected by a Vercel deployment passcode, Search Atlas cannot crawl it — the passcode blocks all automated crawlers, including ours. This article explains how to set up a crawlable staging environment that stays hidden from Google.
⚠️ Why a Vercel Passcode Blocks Search Atlas
Vercel's password protection (available on Pro and Enterprise plans) requires a browser-based authentication step. Because Search Atlas sends automated HTTP requests rather than using a browser session, it cannot pass this challenge. The crawler receives a 401 Unauthorized or redirect response and cannot proceed past the login wall.
The solution is to remove the passcode gate for your staging subdomain and rely on noindex directives to prevent Google from indexing it instead.
🛠️ Recommended Setup: Open Staging + noindex
Follow these steps to give Search Atlas full crawl access while keeping your staging content out of Google's index.
-
Remove the Vercel passcode from your staging deployment or create a dedicated staging subdomain (for example, staging.yourdomain.com) that has no password protection.
-
Add a noindex meta tag to every page on the staging environment. Place the following tag inside the
<head>of your page template:<meta name="robots" content="noindex, nofollow"> -
Block the staging subdomain in your staging robots.txt by adding the following rules to staging.yourdomain.com/robots.txt:
- User-agent: *
- Disallow: /
Using both a noindex tag and a robots.txt Disallow provides two layers of protection against accidental indexing.
-
Verify the staging URL is not submitted to Google Search Console. Do not add your staging subdomain as a property in GSC, and do not include it in any production sitemap.
-
Add your staging site to Search Atlas. Go to Left sidebar → OTTO SEO, create a new project, and enter your staging subdomain URL. Search Atlas will now be able to crawl the site freely.
📊 Confirming Search Atlas Can Crawl Your Staging Site
After completing the setup above, run a manual crawl to confirm everything is working:
- Go to Left sidebar → OTTO SEO → Overview.
- Select your staging project and click Scan.
- Once the scan completes, navigate to Left sidebar → OTTO SEO → All Audits to review discovered pages and issues.
- Use Left sidebar → OTTO SEO → Page Explorer to browse individual URLs and confirm your key staging pages are being indexed by Search Atlas (not by Google).
If the crawl returns zero pages or shows access errors, double-check that the Vercel passcode has been fully removed for that deployment and that no IP allowlist or firewall rule is blocking external crawlers.
💡 Additional Tips
- Use environment variables in your Vercel project to conditionally inject the noindex meta tag only when VERCEL_ENV equals preview or development. This prevents the noindex tag from accidentally appearing on your production build.
- Separate projects in Search Atlas: Keep your production site and your staging site as separate OTTO SEO projects so audits and recommendations do not mix between environments.
- Vercel preview deployments (the auto-generated .vercel.app URLs) are already set to noindex by Vercel by default on most plans, so you can point Search Atlas directly at those URLs without additional configuration — just make sure password protection is off.
- If you use Vercel's GitHub integration for continuous deployment, remember that each new preview deployment generates a new URL. Update your Search Atlas project URL whenever you want to scan a specific deployment.
🔒 Quick Reference: Staging Checklist
- Vercel passcode removed from staging deployment
- noindex + nofollow meta tag added to all staging pages
- robots.txt on staging subdomain set to Disallow: /
- Staging subdomain not added to Google Search Console
- Staging subdomain not included in production sitemap
- Search Atlas OTTO SEO project created for staging URL
- Manual scan completed successfully in Search Atlas
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.