Robots.txt Builder
Create a standards-compliant robots.txt file without touching a line of code. Choose your platform, decide what crawlers can see, and copy the result straight to your site root.
Configure
Your robots.txt
Place this file at your domain root so it is reachable at https://yoursite.com/robots.txt.
What is a robots.txt file?
robots.txt is a plain text file that lives at the root of your website and tells search-engine crawlers which parts of your site they may or may not request. It is one of the first files Google looks for when it visits your domain.
Used well, it keeps crawlers away from admin areas, duplicate pages and internal search results — focusing their budget on the pages you actually want ranked. Used carelessly, a single wrong line can hide your entire site from Google.
Common mistakes to avoid
- Blocking everything by accident —
Disallow: /tells bots to ignore your whole site. - Blocking CSS or JS — Google needs these to render and rank your pages properly.
- Relying on it for privacy — robots.txt is public and does not secure private pages; use authentication for that.
- Forgetting the sitemap line — pointing crawlers to your sitemap speeds up discovery.
Where do I upload the robots.txt file?
It must sit in the top-level (root) directory of your domain so it resolves at https://yoursite.com/robots.txt. On most CMS platforms you can upload it via FTP, a file manager, or a dedicated SEO plugin.
Does robots.txt remove a page from Google?
No. Disallowing a URL stops crawling but the page can still appear in results if other sites link to it. To remove a page from the index, use a noindex meta tag or the Search Console removal tool instead.
Not sure which rules your site needs?
Technical mistakes in robots.txt are one of the most common causes of lost rankings. Our technical SEO team can audit your crawl setup — get in touch.
Want your whole technical setup checked?
Robots.txt is just one piece. Get a full technical SEO audit of your Adelaide website.
Request a Technical Audit