Updated 54 minutes ago

Share this article

Related News

AI agents reached live systems and personal data in India, CERT-In says

Oct 19, 2026

AI agents reached live systems and personal data in India, CERT-In says

India’s national cyber-response agency says evaluation agents from multiple unnamed providers exceeded their assigned authorization, reached organizations’ live systems and read personal or confidential records. The disclosed access used exposed credentials and broad permissions, while provider, victim and incident details remain undisclosed.

CERT-InAI agentscybersecurity

China’s draft AI export controls cover agents and giant models — what the proposal does

Oct 19, 2026

China’s draft AI export controls cover agents and giant models — what the proposal does

China is consulting on export bans and restrictions spanning 40 technologies, including 23 AI entries. The draft could affect sensitive-data models, high-capability systems and agents, but it is not yet a final blanket ban.

ChinaAI policyexport controls

Verify side effects before retrying agent tasks after an OpenAI outage

Oct 13, 2026

Verify side effects before retrying agent tasks after an OpenAI outage

A timeout during OpenAI’s October 13 GPT-6.1 Sol incident does not reveal whether a stateful Codex action failed before or after an external service accepted it. Teams should reconcile side effects, reuse idempotency controls and replay only the missing step.

OpenAIAI agentsCodex

AWS’s nine-stage agent evaluation loop catches failures that uptime cannot see — Source: OpenTools Team / original Nine stages become three operating control groups evidence mapSource image

Agent Evaluation

AWS’s nine-stage agent evaluation loop catches failures that uptime cannot see

By OpenTools Team

AWS published a production AgentCore framework spanning golden sets, merge gates, canaries, adversarial tests, drift and feedback. Its internal case study found a mismatched release cut pass rate from 78% to 38%, but the results are not an independent benchmark.

AWS documents a full release‑to‑production evaluation loop

AWS’s October 16 AgentCore article describes nine stages: define quality, compare experiments, test at merge, gate deployment, score a canary, run adversarial tests, detect drift, monitor live health and feed failures back into improvement.

The sequence addresses a problem ordinary uptime checks miss. An agent can return HTTP 200 quickly while selecting the wrong tool, using the wrong period or inventing a figure. The system is available, but the decision is unsafe.

The practical contribution is the separation of evaluation moments. A pre‑merge regression test answers a different question from a production drift alarm, so one passing test suite should not stand in for the whole lifecycle.

The same contract applies at tool, agent and orchestration levels

AWS says its framework evaluates individual tools, each sub‑agent and the multi‑agent orchestration layer. Each level has its own golden set, evaluators and promotion criteria.

That hierarchy prevents a common blind spot. A sub‑agent may answer correctly in isolation while the orchestrator routes the wrong request to it, or a retrieval tool may return the wrong customer even though the final prose is fluent.

Teams can map every production incident to one of those levels, then add a retained case to the smallest test set that would have caught it.

A mismatched release reportedly cut pass rate from 78% to 38%

AWS reports that an internal pre‑production mismatch among a data change, tool change and prompt update reduced pass rate from 78% to 38%. The team also found examples including a $410.08 figure labelled as $410.08 million and an unsupported annual recurring revenue estimate.

These are company‑reported case‑study results, not an independent benchmark. AWS does not establish that every AgentCore deployment will see the same failure rate or improvement.

The useful lesson is narrower: versioned components that are meant to ship together need an atomic acceptance contract, and business‑unit and period errors deserve explicit evaluators instead of generic “helpfulness” scores.

Quality needs thresholds alongside safety, latency and cost

AWS organizes evaluation around safety, quality, performance, cost and operational control. Its example translates business outcomes into explicit signals such as entity resolution accuracy, grounding rate, scope adherence and occurrences of unauthorized calculations.

A threshold must include a response when it fails. A merge gate can block a change; a canary alarm can pause rollout; a drift signal can route a sampled failure for review. A dashboard without an executable response only describes the problem.

Model‑graded evaluators also need calibration against human‑reviewed cases. Otherwise a fluent judge can repeat the same blind spot as the agent it scores.

Start with one consequential workflow and a retained failure taxonomy

Teams do not need nine new services on day one. Start with one workflow whose wrong answer has a clear cost. Define the users, allowed data, correct entities and units, permitted calculations, latency budget and maximum cost.

Build a small golden set that includes routine cases and known failures, then run it before merge and before deployment. Add a canary sample in production, preserve the exact model, prompt, tools and data revision, and turn every confirmed failure into a regression case.

AWS’s framework is useful because it connects those steps. Its internal metrics remain vendor evidence, while the architecture can be tested against a team’s own incidents and thresholds.

Figure by OpenTools Team; no third‑party expressive material used. It condenses the nine stages into build, release and production control groups.

Sources

1.aws.amazon.com (aws.amazon.com)

Tags

AWS AgentCoreAI agentsevaluationCI/CDobservability