California

Not read yet: California

Where we would read it: NOT FETCHED, BY THEIR INSTRUCTION. leginfo.legislature.ca.gov/robots.txt, read 2026-09-21, carries a group 'User-agent: *' whose rule is 'Disallow: /'. It disallows everything for every agent, while also publishing a sitemap -- the two sit oddly together, and the explicit refusal is the one we obey. We do not rename our crawler to get around it: the declared identity is the whole point of having one, and a refusal we answer by pretending to be somebody else is not a refusal we honoured.