Ohio

Not read yet: Ohio

Where we would read it: NOT FETCHED, BY THEIR INSTRUCTION. codes.ohio.gov/robots.txt is twenty-six bytes and reads, in full and on a single line: 'User-agent: * Disallow: /'. Written that way it is ambiguous -- a strict parser sees one malformed User-agent line and no rule at all, which would let us read the whole site. We take the other reading. When a publisher's robots file contains a blanket refusal in any form a person would recognise, the refusal is what it means, and a parser technicality is not permission. Ohio's constitution index and its nineteen article links are recorded here so nobody has to crawl the site again to find out we may not read it.