Eval-to-Production Exposure Cheat Sheet — agentic ai evals, measured against what they can reach
— Ingenious AI security checks.
Direct answer: Eval-to-production exposure is the gap between what an evaluation agent is supposed to touch and what it can actually reach — production credentials, data, networks and wallets that leak across the eval boundary. This cheat sheet is for teams evaluating agentic ai before release. EvalScope scans that boundary for you (Free single-file scan, Pro $49/mo, Team $149/mo), and its free scan scores one repo file or config to start.
Key facts
Exposure is about reach, not intent: an eval agent with a production key in its environment is exposed even if the eval never calls it.
5 families cover the boundary: credentials, data, network, tools and configuration drift between eval and production.
Findings carry severity and fixes: an exposure report that cannot rank what to fix first is a list, not a control.
What is eval-to-production exposure in agentic ai?
Eval-to-production exposure is everything an evaluation agent could reach beyond its sandbox: the production credentials sitting in a shared environment file, the real customer data copied in as a fixture, the network path from the eval runner to live systems. Evals run experimental prompts against powerful agents, which makes the boundary between eval and production the worst place for shared anything.
Credential exposure: production API keys, tokens or connection strings present in eval environments, configs or fixtures.
Data exposure: real user data used as eval input, or eval outputs written back into production stores.
Network exposure: eval runners that can reach production endpoints, internal services or cloud metadata addresses.
Tool exposure: eval agents holding write-capable or payment-capable tools 'just for realism'.
Configuration drift: eval and production configs that differ silently, so the eval certifies a system that is not the one shipping.
How is exposure scored?
Reach first: can the eval agent actually get to the asset from where it runs? Unreachable beats theoretical.
Privilege second: read-only reach scores below write reach; anything that can spend, delete or send scores highest.
Blast radius third: one test account is a finding; the full production database is an incident waiting for a prompt.
Every finding ships with a fix: scope the key, scrub the fixture, segment the network, pin the config — in severity order.
How do you close the boundary?
Separate credentials completely: eval gets its own keys, scoped to eval resources, rotatable independently.
Sanitise fixtures: synthetic or scrubbed data with the same shape as production, never raw exports.
Segment the network: eval runners cannot route to production or metadata endpoints by default.
Diff the configs: eval and production configuration are compared mechanically, and every difference is intentional and documented.
Re-scan on change: every new eval, tool or data source re-opens the boundary question.
It is the set of production assets an evaluation agent can actually reach — credentials in its environment, real data in its fixtures, production endpoints on its network, write-capable tools in its harness. Exposure is measured by reach and privilege, not by what the eval was intended to do.
Why are agentic ai evals a special risk?
Because evals deliberately push agents with adversarial, malformed and exploratory inputs — exactly the conditions under which an agent tries unexpected actions. If the eval environment shares production credentials, data or network reach, a routine eval can touch live systems. The boundary has to be engineered, not assumed.
What does an exposure report include?
For each finding: what the eval agent can reach, the severity based on reach, privilege and blast radius, and the concrete fix — scope this key, scrub that fixture, segment that route. EvalScope's report scores the boundary so the worst exposure is unambiguously first in the fix queue.
What does EvalScope cost?
EvalScope starts with a free single-file scan — paste one repo file or config and get its exposure report. Pro is $49 per month for whole-repo scanning and Team is $149 per month. The free scan uses the same scoring, so one file shows you the standard the full scan applies everywhere.