Welcome!

Restrict or block AI access to the content of an Odoo website

• Go to Website → Configuration → Settings
• Scroll to the SEO section and click Edit robots.txt

Add rules that specifically disallow the main AI training crawlers. A typical 2026 configuration looks like this (Optionally add Cloudflare AI bot protection for stronger enforcement.):

# Block major AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Optional: also block AI search/retrieval bots if you want zero AI visibility
User-agent: OAI-SearchBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

# Keep normal search engines allowed
User-agent: *
Allow: /

keeps crawlers away from Backend & authentication areas:

Disallow: /web/
Disallow: /web/login
Disallow: /web/database/
Disallow: /web/session/

Customer Portal (very important as these contains private customer data)

Disallow: /my/
Disallow: /my/home
Disallow: /my/account
Disallow: /my/orders
Disallow: /my/invoices
Disallow: /my/quotes

Forum (if you have the Forum app installed)

Disallow: /forum/

(or leave it allowed if you want the forum content indexed as public knowledge)

E-commerce transactional pages (if the Shop app is active)

Disallow: /shop/cart
Disallow: /shop/checkout
Disallow: /shop/payment
Disallow: /shop/confirmation
Disallow: /shop/confirm_order

Forms & interactive pages

Disallow: /contactus
Disallow: /newsletter
Disallow: /website/form/
Disallow: /mailing/

Other common utility / system pages

Disallow: /website/info
Disallow: /slides/   (if you use eLearning and don’t want it public)
Disallow: /event/    (registration pages, if sensitive)
Disallow: /jobs/     (job application forms)
Disallow: /survey/

Practical example (good starting point)

User-agent: *

# Backend & login
Disallow: /web/
Disallow: /web/login
Disallow: /web/database/

# Portal (private customer area)
Disallow: /my/

# Forms
Disallow: /contactus
Disallow: /newsletter
Disallow: /website/form/

# Shop transactional (if applicable)
Disallow: /shop/cart
Disallow: /shop/checkout
Disallow: /shop/payment

# Forum (optional – remove this line if you want forum content indexed)
Disallow: /forum/

# AI crawlers (from previous discussion)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# ... add other AI bots as needed

Tips

• Check the exact URLs on your site (open the pages in a browser) because some paths can vary with custom configuration or installed modules.
• After editing, visit https://yourdomain.com/robots.txt to verify the rules.

• For pages that should never appear in search results at all, also set them to noindex in the page SEO properties (stronger than robots.txt alone).

  • Go to Website → Site → Properties and turn Off the indexed switch option

    • This automatically adds the meta tag <meta name="robots" content="noindex"

  • noindex is stronger than Disallow in robots.txt for preventing a page from appearing in search results. Best practice is often to use both:

  • Disallow in robots.txt (to reduce crawling)
  • noindex via the Indexed toggle (to prevent indexing)