SEO: Indexability Rules
Control which content search engines can index and how they display it, with per-bot rules and advanced meta robots directives.
Overview
Indexability rules let you control how search engine crawlers interact with your content. You can allow or block indexing per content type (posts, pages, categories, tags, authors) and target specific bots individually. Advanced directives give you fine-grained control over snippets, image previews, and content expiration.
Accessing indexability settings
- Go to SEO in the sidebar and select your site.
- Click the Config tab.
- The indexability rules table shows your content types and their current directives.
How rules work
Each rule combines a content type with a search engine bot and a set of directives. For example, you might allow Googlebot to index your posts but block Baiduspider from indexing author archives.
Supported bots
You can create rules that target all bots or specific ones: Googlebot, Bingbot, Yandexbot, Baiduspider, DuckDuckBot, Slurp (Yahoo), Facebot (Facebook), and AI crawlers such as GPTBot, ClaudeBot, CohereBot, and others.
Basic directives
- Index: Allow the bot to add the page to its search index.
- Follow: Allow the bot to follow links on the page.
- NoIndex: Prevent the page from appearing in search results.
- NoFollow: Prevent the bot from following links on the page.
Advanced directives
Click the advanced settings button on any rule to access additional meta robots attributes:
- nosnippet: Hide the text snippet in search results. Useful for preventing Google from showing excerpts of legal or confidential pages.
- indexifembedded: Allow indexing only when the page is embedded in another page (via iframe).
- max-snippet: Set a character limit for the text snippet shown in results.
- max-image-preview: Control the maximum image preview size (none, standard, or large).
- max-video-preview: Set the maximum duration (in seconds) for video previews.
- notranslate: Prevent Google from offering to translate the page.
- noimageindex: Prevent search engines from indexing images on the page.
- unavailable-after: Set an expiration date after which the page should be removed from search results.
IndexNow integration
IndexNow lets you notify search engines immediately when content is published or updated. When enabled, the plugin pings the IndexNow API each time you publish, update, or delete a post. Supported engines include Bing, Yandex, and others that implement the IndexNow protocol.
To enable IndexNow, toggle it on in the Config tab. RakuWP generates and manages the verification key automatically.
How rules are applied
The plugin reads the indexability rules from the panel and outputs the corresponding <meta name="robots"> tags in the <head> of each page. Rules are evaluated in order of specificity: a rule targeting a specific bot overrides a general rule for the same content type.