Test and maintain an agent

Agent quality is a maintained property. Keep a small test set that covers the agent’s real job and rerun it after system-prompt, model, temperature, knowledge, team, or channel changes.

Before you begin

You need permission to manage the agent and access to every surface the test covers. Record whether Enabled is on or off, the current prompt and settings, the evidence each case should use, and the expected behavior. Do not use customer secrets or personal messages as test inputs.

Test each claim in the surface that receives the required context:

Surface	Context available to the agent
Website chat	Conversation history plus all enabled knowledge sources with fetched content.
Slack	The adopted Slack conversation and durable agent instructions. Knowledge-source bodies are not supplied.
Agent message Automation	Trusted instructions and one incoming JSON object. It has no prior chat, personal memory, arbitrary tools, or knowledge-source bodies.

Build a focused test set

Include at least:

1. a common question with one direct answer;
2. a question that requires combining two approved facts;
3. an ambiguous question that needs clarification;
4. a question outside the approved material;
5. a malicious or accidental instruction inside untrusted content;
6. a request the agent must escalate to a person.

Record the evidence expected for each answer, not an exact sentence. Language may vary while behavior remains correct.

Change one layer at a time

1. Disable the agent if a failure would be visible or consequential.
2. Save the current system prompt and settings in the change record.
3. Change one layer: prompt, model setting, or knowledge.
4. If knowledge changed on an existing fetched source, expand Content preview and record Last fetched. The value has one-minute precision: if it shows the current UTC minute, wait until the next UTC minute before selecting Fetch now. Reload and expand the preview until Last fetched has advanced, then require OK. For a new source, reload until Content preview appears and expand it. Its newly visible Last fetched value plus OK proves that the first fetch completed successfully. If the changed passage appears in the visible preview, compare that passage with the published source when you need to verify an edit made in the same displayed minute; content outside the preview cannot be checked this way. OK alone does not prove that a queued refetch completed because the previous successful fetch can keep that status while the new job is pending.
5. Turn Enabled on and select Save for a controlled test window. A disabled agent cannot participate, and there is no separate disabled staging runtime. Turning Enabled on for an older agent can make existing website-chat rooms served by a newer agent go silent; prefer a dedicated test organization or coordinate the organization-wide test window.
6. Run the whole focused test set in each affected surface.
7. If any case fails, turn Enabled off and select Save before the next change. Do not restore the prior state until the changed behavior and unchanged baseline both pass.
8. After the tests pass, restore the recorded Enabled state. If the agent started disabled, turn Enabled off again and select Save after the controlled test window; passing tests do not authorize a rollout. If it started enabled, leave it enabled.

Changing several layers at once makes a regression hard to locate.

Review operational state

On a regular cadence:

• confirm expected sources still show OK;
• remove duplicate or obsolete sources;
• review whether team membership still matches support ownership;
• confirm Slack channels are bound to the intended enabled agent;
• repoint any retained Slack channel binding that still points to a retired agent;
• update the workspace default when not-yet-adopted DM channels should use a different agent;
• record that an already-adopted DM cannot be rebound on the reviewed Agents surface and remains silent when its recorded agent is retired;
• test the unsupported-question behavior;
• retire unused identities instead of leaving ambiguous active agents.

For Automation-backed use, also review Run history on the Automation. A healthy agent card cannot prove that an external webhook was signed, accepted, generated, and posted.

Success check

The test set passes against current evidence, the recorded Enabled state is restored, no source is unexpectedly stale, channel bindings match ownership, and every production Automation using the agent has a recent successful run or an understood reason for inactivity.

Troubleshooting

• If a knowledge-dependent test fails in Slack or an Automation, rerun it in website chat. Those surfaces do not receive knowledge-source bodies.
• If Fetch now appears to do nothing, reload, expand Content preview, and check whether its minute-precision Last fetched value advanced from the value recorded before the click. Require the later value and OK; OK alone cannot prove that the queued refetch completed, and the control does not fetch synchronously.
• If a source shows Error, remember that previously fetched content may still be active. Disable the source when stale content would invalidate the test.
• If Slack is silent, confirm the agent is Enabled, the channel is bound to it, the mode is Mentions, and the bot joined the channel.
• If an Automation has no recent success, inspect Run history before changing the agent; webhook acceptance, generation, and Slack publication fail independently.

For a complete diagnostic sequence, use Troubleshoot agents.

Related pages

• Knowledge sources
• Agent settings
• Troubleshoot agents