Monday 9 October 2017

Robots.txt

User-agent: *

Allow: /wp-admin/admin-ajax.php
Allow: /wp-includes/*.js
Allow: /wp-includes/*.css
Disallow: /wp-admin/
Disallow: /tag/
Sitemap: http://www.scuhs.edu/sitemap_index.xml

Pattern-matching

  • * is a wildcard that represents any sequence of characters
  • $  matches the end of the URL