TL;DR
Build a small, stable set of questions around the practice’s actual services, locations and visitor tasks. Separate branded questions from unbranded discovery. Record exact wording and context, score distinct observations, and keep new experimental questions outside the comparison group until you deliberately revise the method.
Choose the decisions the questions should support
A question set should help identify useful work. It might test whether an assistant gives the correct office information, understands an available service or points to the right booking path. “Make our visibility score higher” is not enough to define a useful sample.
Start with approved public business facts. Do not put patient records, personal health histories or private operational information into test prompts. This is a business-discovery exercise, not a test of clinical advice.
Build a simple question matrix
Use these three tasks and six prompt templates to draft the panel. Replace [practice], [location] and [service] with approved public facts before collection. The brackets are required inputs, not observations or invented practice details.
| Task | Branded prompt | Unbranded prompt |
|---|---|---|
| Identify a practice | “Where in [location] is [practice]?” | “Which practices in [location] provide [service]?” |
| Understand the process | “What does [practice] say about arranging [service]?” | “What appointment-request instructions do practices offering [service] in [location] publish?” |
| Find the next step | “Where is [practice]’s public appointment-request page?” | “Show public appointment-request pages for practices offering [service] in [location].” |
The branded prompts test a known name; the unbranded prompts leave discovery open. Neither asks the assistant to endorse the practice. Review the completed wording for clarity, then save it exactly and give each prompt a permanent ID. A clinical reviewer should check any wording that moves beyond public administrative information.
Declare the collection protocol before seeing results
Before collection, choose and record the repetitions per prompt, review window and platforms based on actual review capacity. Planned observations per platform equal the number of prompts multiplied by the agreed repetitions. Leave those inputs blank until the real protocol is approved; a manageable operating sample is not necessarily statistically representative.
Use the same recorded platform/mode, language and visible location settings. Start without earlier practice-specific conversation, paste the exact prompt and retain every response. Record the date and visible model/version information when available; if a setting cannot be controlled, say so. Do not retry a completed answer until it improves.
A tool failure stays in the record as missing. Report completed observations and missing attempts separately; do not classify missing attempts as negative mentions. Matching the protocol improves interpretability but cannot make variable answers identical.
Make changes explicit
Keep exploratory prompts outside the fixed panel. When an office is added, start a new question-set version with the approved location wording and identify which comparisons remain valid. Do not blend new-office observations into the old panel and present the difference as pure performance improvement.
A platform’s retrieval phrases are separate evidence. Bing describes grounding queries as sampled phrases used by AI to retrieve cited content. Bing’s AI Performance explanation supports keeping those phrases distinct from the exact user prompts in your panel.
Score observable facts separately
For each completed observation, record whether the practice was mentioned, whether its website was cited, whether key business facts were correct and whether the action destination was accurate. Use “not assessed” when the answer does not address a field.
Attach evidence to factual-error findings. A wrong address can be compared with the approved record; an unfavorable opinion needs different treatment. Avoid combining all categories into a single score before the team understands what each category means.
If a saved answer names the practice correctly but cites an obsolete office page, record the mention and source problem separately. Calling the entire answer either a success or a failure would hide a useful distinction.
Frequently asked questions
How many questions do we need?
Enough to cover the agreed decisions without exceeding your review capacity. A small documented sample is more useful than a large unreviewed export, but it cannot establish market-wide visibility.
Should we repeat prompts until we get a good answer?
No. Decide the sampling approach in advance and retain the results. Selectively saving favorable answers makes later comparisons misleading.