Here's a thought experiment.
You're an HTML developer. A client comes to you.
They signed a "consent agreement" with a state Attorney General; and it involved putting up a public access Web site. Now. How would you structure it so you could (a) put up the web site while (b) making sure no one could find it with Google? And without using the obvious HTML tags for no web crawler indexing?
Let's just say COMPANY-X makes brake pads, and were being sued over a chemical byproduct.
They signed off on the judge's decree to take various cleanup measures and report them to the state environmental agency, and also, to make them public by placing them on a Web site.
But the company just might want to keep the connection between the cleanup and its corporate Web sites, hmmm, "indistinct." So for starters, the company sets up a domain name with no clear connection to the parents.
The consent decree just says this: "PUBLIC INFORMATION: COMPANY-X agrees to develop within 45 days (of signing this agreement) a website and make available to the public the progress of implementation of this (decree) by posting all reports and work plans submitted to the Department in portable document format (PDF) within 10 business days of their timely submission to the Department."
The company's legal eagles have already noted that the judge didn't say "as part of the COMPANY-X corporate web site, using its domain name."
The company Webmaster does not put a press release about the consent agreement on the corporate web pages; and does not provide links from the corporate pages to the new web site, which has its own cryptic domain.
That much is pretty easy to come up with.
Now: how do you keep the resulting Web page, with its PDF of the consent decree; and a growing chronological archive of required documents, invisible to Google?
I'm curious to see how closely Kossacks' answers correspond with what I have observed.