# robots.txt file # Version 2.2 - 10/26/06 # updated 7/10/2009 # - added block of exclusions for google # BASIC robots.txt file to reduce crawler load. # Must be named "robots.txt" and be in the root folder. # Only Google, MSN, Yahoo, and Google-Image are allowed. # Certain files excluded. # Yahoo-Slurp held back to 1 download every 45 seconds # Add ponyfish for RSS 2/15/2008 #The following was added by http://www.alegent.com/smm_googleadmin.cfm User-Agent: Medseek-gsa-crawler Disallow: Disallow: /blank Disallow: /BLANK Disallow: /print Disallow: /PRINT Disallow: /secure Disallow: /images # For (dev) google mini User-Agent: msdevgsa Disallow: Disallow: /blank Disallow: /BLANK Disallow: /print Disallow: /PRINT Disallow: /secure Disallow: /images # Throttle Yahoo Slurp User-agent: Slurp Crawl-delay: 45 # Googlebot Yahoo MSN User-agent: Googlebot User-agent: Slurp User-agent: msnbot User-agent: Ponyfish User-agent: * Disallow: Disallow: /blank Disallow: /BLANK Disallow: /print Disallow: /PRINT Disallow: /secure Disallow: /images User-agent: Googlebot Disallow: /?src= Disallow: /body.cfm/ Disallow: /Defa Disallow: /Qabo Disallow: /Adam/ Disallow: /Body Disallow: /body.cfm.cfm # All robots User-agent: * Disallow /body.cfm?id=3866&action=locationdetail* Disallow /body.cfm?id=3866&action=locationdetail* Disallow *.wmv Disallow /body.cfm?id=139&action=list* Disallow /body.cfm?id=139&action=list* # Disallow: / # Google Images User-agent: Googlebot-Image Disallow: /