Test and maintain an agent

Agent quality is a maintained property. Keep a small test set that covers the agent’s real job and rerun it after system-prompt, model, temperature, knowledge, team, or channel changes.

Build a focused test set

Include at least:

1. a common question with one direct answer;
2. a question that requires combining two approved facts;
3. an ambiguous question that needs clarification;
4. a question outside the approved material;
5. a malicious or accidental instruction inside untrusted content;
6. a request the agent must escalate to a person.

Record the evidence expected for each answer, not an exact sentence. Language may vary while behavior remains correct.

Change one layer at a time

1. Disable the agent if a failure would be visible or consequential.
2. Save the current system prompt and settings in the change record.
3. Change one layer: prompt, model setting, or knowledge.
4. If knowledge changed, use Fetch now and wait for OK.
5. Run the whole focused test set.
6. Re-enable only after the changed behavior and unchanged baseline both pass.

Changing several layers at once makes a regression hard to locate.

Review operational state

On a regular cadence:

• confirm expected sources still show OK;
• remove duplicate or obsolete sources;
• review whether team membership still matches support ownership;
• confirm Slack channels are bound to the intended enabled agent;
• test the unsupported-question behavior;
• retire unused identities instead of leaving ambiguous active agents.

For Automation-backed use, also review Run history on the Automation. A healthy agent card cannot prove that an external webhook was signed, accepted, generated, and posted.

Success check

The test set passes against current evidence, no source is unexpectedly stale, channel bindings match ownership, and every production Automation using the agent has a recent successful run or an understood reason for inactivity.