TL;DR
A healthcare practice can make different decisions about search discovery and model training. The available controls depend on the platform, and some cover more than training. Write down the desired use of public content before asking a developer to change crawler rules. Private patient systems need actual access controls regardless of those preferences.
Replace “allow AI” with a specific decision
Suppose a practice wants people to find its public service pages through AI search, but its owner does not want those pages used to train foundation models. “Block AI bots” is too broad a change request. It could work against the discovery goal without establishing the protection the owner intended.
There are at least three activities to distinguish: finding pages for search, using content to develop a model, and retrieving a page during a person's interaction with an assistant. A platform may offer different controls for these activities, or group several uses under one control. Product names alone will not tell you which.
The following comparison covers documented Google and OpenAI behavior checked on October 17, 2026. It is not a universal rule for every AI company.
| Public-content use | Documented control or agent | Decision to discuss with the implementer |
|---|---|---|
| Google Search, including its AI features | Googlebot crawling controls; separate preview/indexing controls | What should remain discoverable, and what content may be shown? |
| Specified Gemini training and grounding uses | Google-Extended product token | Is the practice comfortable allowing both uses covered by this token? |
| ChatGPT search discovery | OAI-SearchBot | Should public practice pages be available to this search crawler? |
| OpenAI foundation-model training | GPTBot | What training preference should be expressed independently of search? |
Google identifies Googlebot as the access control for Search, including AI features. Its AI features documentation also describes preview controls. Those controls can affect what Search displays; they are not a way to promise citations while withholding the information needed to support them.
The important Google-specific qualification is that Google-Extended is not a training-only switch. It also covers specified grounding uses in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search. Grounding means supplying retrieved information to a model when it responds. Review the exact covered products in Google's crawler documentation before approving the tradeoff.
OpenAI documents independent OAI-SearchBot and GPTBot settings. It also distinguishes ChatGPT-User, used for certain user-initiated visits, for which robots.txt rules may not apply. These distinctions appear in OpenAI's crawler documentation. They should remain separate entries in the change request.
Work through a decision brief
Complete this brief with the practice's actual preferences before requesting a configuration change:
| Decision field | What the owner and implementer should record |
|---|---|
| Public content in scope | Approved service, location and administrative URL areas |
| Desired discovery | Named search products in which the practice wants those pages to remain eligible |
| Training preference | A separate allow or decline decision for each documented training use |
| Grouped uses | Whether the owner accepts the training and grounding uses covered together by Google-Extended |
| Excluded systems | Patient portal, internal files and staff-only tools protected independently of crawler preferences |
| Unresolved choice | The specific use that needs a decision, with its implementation left unchanged until resolved |
If an owner asks for “no Gemini training” without considering grounding, return that grouped-use choice for clarification. The developer can explain the independently documented OpenAI controls while the owner resolves the Google tradeoff. There is no need to guess what “block AI” was supposed to mean.
For another platform, add another row only after reading its current primary documentation. Do not copy a familiar token and assume the same purpose or behavior.
Ask for implementation evidence
Before approving a change, request the exact affected domains and URL areas, the existing rule, the proposed rule and its intended effect. Include any hosting or security layer that could override the expected access. Avoid sharing sensitive logs in an ordinary marketing document.
Have the implementer test both sides of the decision: intended public access still works, and the new preference is expressed correctly. Retain the previous configuration and an authorized rollback plan. A test that only proves a block exists misses accidental loss of public discovery. Use the public-page crawlability checks to investigate access and rendering after an authorized change.
Treat crawler preferences separately from confidentiality. Google's robots.txt introduction explains that robots.txt is not a mechanism for keeping a page out of Google and is not supported identically by all crawlers. Do not put private information on an otherwise public page and rely on crawler rules to protect it.
What should the agency report afterward?
Ask for the decision, the implemented rule and the technical verification. An observed crawl or a correctly configured preference does not prove that a platform will recommend the practice. Likewise, a missing citation does not by itself prove that a crawler was blocked.
Review the policy when the practice changes its goals or the platform changes its documented uses. Keep the owner decision attached to the technical configuration so a later website vendor does not “fix” an intentional choice without understanding it.
Frequently asked questions
Does blocking a training crawler hide private records?
No. Crawler preferences do not replace authentication and other access controls. Keep private systems protected independently of public-content decisions.
Can we make the same choice for every platform?
Only after checking each platform’s current documented controls and covered uses. Record the decision separately where the controls or purposes differ. An optional llms.txt guide serves a different purpose and does not set these permissions. The llms.txt proposal distinguishes the guide from crawler-access preferences.