Home > AI > Backend > Wordpress >

Search engine crawl

When this box is checked, WordPress adds this line to your website’s header.

<meta name='robots' content='noindex,follow' />

WordPress also modifies your site’s robots.txt file and add these lines to it:

User-agent: *
Disallow: /

These lines ask robots (web crawlers) not to index your pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect this, some page or random image from your site may get indexed.

Leave a Reply