Why Your Drupal robots.txt Changes Aren't Showing Up

You've installed the robotstxt module, made your changes in the Drupal admin at /admin/config/search/robotstxt, cleared cache, run cron — and the live robots.txt file still showing old content. Before you go down a rabbit hole and here are the most common culprits.

1. Composer Scaffold Is Regenerating the File on Every Deploy

If your project uses robots.txt (which most Composer-based Drupal projects do), robots.txt is one of the scaffold files it copies into your webroot on every composer install. This means even if you delete it manually, it comes back on the next deploy, silently overriding the robotstxt module every time.

The fix is to tell the scaffold plugin to skip robots.txt entirely. In your composer.json, add a false entry for the file under file-mapping:

"drupal-scaffold": {
   "file-mapping": {
       "[web-root]/robots.txt": false
   },
   "locations": {
       "web-root": "web/"
   }
}

If your project already has a file-mapping section, just add the robots.txt line alongside existing entries:

"drupal-scaffold": {
   "file-mapping": {
       "[profile-root]/.editorconfig": false,
       "[profile-root]/.gitattributes": false,
       "[web-root]/sites/default/default.services.yml": {
           "mode": "replace",
           "overwrite": false,
           "path": "docroot/core/assets/scaffold/files/default.services.yml"
       },
       "[web-root]/robots.txt": false
   },
   "locations": {
       "web-root": "docroot/"
   }
}

Setting a file to false is a first-class scaffold feature — it tells the plugin to skip that file completely on install and update, on every environment.

Once the composer.json change is in place, delete the existing file from your webroot (no git rm needed if it's not committed to your repo):

rm docroot/robots.txt

After the next composer install — locally or on deploy — the file won't be regenerated, and the robotstxt module will have a clear path to serve it dynamically.

2. A Static File Is Overriding the Module

If you're not using Composer scaffold, or the file appeared some other way, a physical robots.txt in your webroot will still take precedence. Apache and NGINX both serve static files before passing requests to Drupal's front controller, so the module never gets a chance to run.

ls -la /path/to/drupal/webroot/robots.txt

If the file exists and isn't being managed by scaffold, just delete it and run drush cr.

3. Drupal's Cache Hasn't Actually Cleared

The admin UI cache clear and cron aren't always enough, especially on complex setups. Use Drush for a full rebuild:

drush cr

If you're on a multi-site or have a custom cache setup, also check whether your robots.txt path has its own caching configuration in your settings.php or services.yml.

4. Your Hosting Platform Has Its Own Caching Layer

Pantheon and Acquia both cache aggressively at the platform level, and that caching can serve a stale robots.txt even after a Drupal cache clear.

On Pantheon: Check the Global CDN cache. You may need to do a full cache purge from the Pantheon dashboard, or use Terminus:

terminus env:clear-cache your-site.env

On Acquia: Check Varnish. Acquia's Varnish layer may be serving a cached version. Purge via the Cloud UI or trigger a full cache clear from the environment dashboard.

5. A CDN or Reverse Proxy Is Caching the File

If you're running Cloudflare or another CDN in front of your site, it may have cached robots.txt with a long TTL. Check your caching rules — robots.txt is often explicitly cached by CDN configurations. You can manually purge a single URL in Cloudflare: Caching → Cache Purge → Custom Purge → enter your robots.txt URL.

Quick Diagnostic Checklist

Run through these in order before diving deeper:

1. Check composer.json — is robots.txt excluded from scaffold file-mapping?
2. ls -la docroot/robots.txt — does a static file exist in the webroot?
3. drush cr — full cache rebuild
4. curl -I https://yoursite.com/robots.txt — check response headers for cache hits (X-Cache: HIT, Age, CF-Cache-Status)
5. Check platform-level cache (Pantheon/Acquia dashboard)
6. Check CDN purge (Cloudflare or equivalent)
7. Review .htaccess or NGINX config for robots.txt rules

In most cases, the answer is step one or step two. The robotstxt module works — it just needs a clear path to do its job.