It seems that OpenAI is scraping [certificate transparency] logs

Summary

lol. I minted a new TLS cert and it seems that OpenAI is scraping CT logs for what I assume are things to scrape from, based on the near instant response from this: Dec 12 20:43:04 xxxx xxx[719]: l=debug m="http request" pkg=http httpaccess= handler=(nomatch) method=get url=/robots.txt host=autoconfig.benjojo.uk duration="162.176µs" statuscode=404 proto=http/2.0 remoteaddr=74.7.175.182:38242 tlsinfo=tls1.3 useragent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; robots.txt; +https://openai.com/searchbot" referrr= size=19 cid=19b14416d95 @benjojowp-login.php bots have been doing that for years so I'd be surprised if OpenAI didn't @wolf480pl yeah and I guess it's a non terrible way of "seeding" a "search engine" @benjojowhat if CT logs contained hash(domain, nonce) instead of containing the domain in plain, and the nonce was part of the CT inclusion proof? @wolf480pl the point of certificate transparency logs is so that outside observers can do the double-checking of the CAs certificate and policy in full, if you mess with any part of this, the entire system becomes deeply exploitable and difficult to end to end verify @benjojooh, duh I need to be able to find who's issuing carts for my domain and I'm guessing some people look at all certs issued by CAs and verify certain criteria that may require knowing the domains... it's kinda sad that it provides domain enumeration, but I guess putting addng zero-knowledge proofs to the mix would've been too complex @wolf480pl tbh domain's are not really that secret, and if you depended on that then something was very wrong. You can work around a lot of this stuff by "just" using wildcard certs instead @benjojobut then why bother with NSEC3... @wolf480pl tbh I would argue why bother with DNSSEC (outside of extremely marginal situations), but NSEC3 even more @benjojo It's interesting to watch web server logs to see what things p...