EU AI Act readiness starts in your eval sandbox: what can your eval agent reach?
On October 15, 2026, India's CERT-In issued advisory CIAD-2026-0049 on weaknesses in AI evaluation environments. The pattern it names is specific: an eval agent is handed real secrets, a permissive role and an open config, and the sandbox everyone assumed was isolated turns out to be wired into production.
EvalScope is Ingenious AI's scanner for exactly this surface. It reads the repo files and configs you declare and reports eval-agent reachability: exposed secrets, permissive roles and open configs, each with a severity and a fix. The audit below applies the same three classes to your setup, in plain questions.
Honest scope, read this first: this page checks declared setup, not live systems. EvalScope scans the files and configs you point it at; it does not probe your production environment, run your evals, or certify EU AI Act compliance. The CERT-In advisory is summarized as published; the compliance duty for your specific system depends on its risk class and role, and that call belongs to your own legal review.
The EU AI Act angle: the advisory, as published
CERT-In is India's national incident response authority, and CIAD-2026-0049 is aimed at a target most security programs skip: the evaluation environment. Evals run agents against realistic tasks, and realistic tasks tempt teams to hand over realistic access. The advisory's three weakness classes are the result: secrets sitting in eval configs, roles that allow far more than the eval needs, and configuration files that leave the eval surface open.
The timing matters for European duties too. The EU AI Act's transparency obligations have applied since August 2, 2026, and the grace period for marking AI-generated content ends on December 2, 2026. An eval agent that can reach production data is not just a security finding; it is a governance gap in the system your documentation describes.
The advisory: CERT-In CIAD-2026-0049, issued October 15, 2026, covers weaknesses in AI evaluation environments.
Weakness class 1: exposed secrets: real API keys and credentials present in eval configs, fixtures or prompts.
Weakness class 2: permissive roles: the eval agent's identity can do more than the evaluation requires, up to production writes.
Weakness class 3: open configs: eval endpoints, buckets and dashboards reachable without the controls production would demand.
The EU clock: EU AI Act transparency obligations apply from August 2, 2026; the content-marking grace period ends December 2, 2026.
The demand signal: ‘eu ai act’ draws 6,600 US searches a month and ‘ai governance’ 3,200 (SearchEngineLand, 2026): the compliance audience is already looking.
3weakness classes named in CERT-In advisory CIAD-2026-0049: secrets, roles, configs
Oct 15, 2026the date CERT-In published the eval-environment advisory
Aug 2, 2026EU AI Act transparency obligations took effect
Dec 2, 2026the EU AI Act content-marking grace period ends
Why does an eval sandbox become a production risk?
Because evals reward realism. Teams copy production configs into the eval environment, hand the agent production-shaped credentials, and leave both in place after the run. The agent under test then holds a working path into real systems, and the first outsider to find that path inherits it.
Audit your eval setup: what can the agent under test reach?
Answer for the eval environment as it exists today, not as it was designed. Everything runs in your browser; nothing is sent anywhere.
Close the eval-to-production path: the order matters
Map the route: list every way the eval environment can reach production: network paths, shared buckets, shared queues, shared credentials. If a path exists on paper, treat it as open.
Rotate before you tidy: any real secret that has lived in an eval config is already exposed. Rotate it first; cleaning the file afterwards does not un-expose the old key.
Scope the role to the eval: the agent under test needs the dataset and nothing else. Read-only, one environment, expiring with the run.
Scan the declared files: run a reachability scan over the repo files and configs your eval actually loads, and fix findings by severity. This is the scan EvalScope automates.
Keep evidence: keep the scan output with your EU AI Act documentation. When an auditor asks how you know the eval surface is contained, a dated scan beats a memory.
EvalScope, by Ingenious AI, scans your declared repo files and configs for eval-agent reachability and reports exposed secrets, permissive roles and open configs with severity and fixes, so the three advisory classes get checked on every change instead of once. See how EvalScope works.
FROM INGENIOUS AI
EvalScope is built for exactly this: a reachability scan over the files and configs your eval loads, with findings your compliance file can cite.
1. What did CERT-In advisory CIAD-2026-0049 cover?
2. Which of these is one of the advisory's three weakness classes?
3. When did the EU AI Act transparency obligations take effect?
4. Why do real production keys end up in eval environments?
5. What is the first fix when a real secret has lived in an eval config?
Frequently asked questions
What is eval-agent reachability?
Eval-agent reachability is the set of systems, data and actions an agent under evaluation can actually touch from inside the eval environment. It is defined by the eval's network routes, the secrets present in its configs and the permissions on its role. EvalScope measures it from your declared files, so the answer is evidence, not opinion.
What is CERT-In advisory CIAD-2026-0049?
CIAD-2026-0049 is an advisory from CERT-In, India's national incident response authority, published on October 15, 2026. It names three weakness classes in AI evaluation environments: exposed secrets in eval configs, permissive roles granted to eval agents, and open configuration surfaces. The advisory is the reason this check exists.
Does the EU AI Act cover evaluation setups?
The EU AI Act regulates AI systems by risk class and role, and its transparency obligations have applied since August 2, 2026. An eval environment that can reach production data undermines the governance story your documentation tells. Whether a specific duty applies to your setup is a legal call; the reachability facts underneath it are measurable.
How does EvalScope scan an eval setup?
EvalScope reads the repo files and configs you declare: the files your eval actually loads. It reports exposed secrets, permissive roles and open configs as reachability findings, each with a severity and a concrete fix. It does not probe live production systems, and it does not run your evals for you.
How often should eval reachability be checked?
Check it whenever the eval environment changes: new datasets, new tools for the agent, new infrastructure, or a copied production config. Eval setups drift toward realism over time, and drift is exactly how production secrets and routes arrive. A scan per change keeps the evidence current for audits.
Sources: CERT-In advisory CIAD-2026-0049 (October 15, 2026) · EU AI Act transparency timeline (obligations from August 2, 2026; content-marking grace period to December 2, 2026) · Ingenious AI unicorn audit of EvalScope (October 23, 2026).