# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /r/ Disallow: /404 Disallow: /not-logged