sentinel-eval
An open-source local testing tool for quantifying authorization vulnerabilities and leakage risks in vector databases within RAG and agent memory scenarios. It supports deterministic offline testing.
Timeline (last 90 days)
sentinel-eval is a local testing tool, licensed under Apache-2.0, for quantifying authorization leakage risk in Agent memory scenarios.
In sentinel-eval's tests, the original RAG scenario had a leakage rate of 0% (18 public chunks retrieved, 0 restricted chunks exposed); in the Agent memory scenario, 2 of 3 synthetic tasks leaked, and the cross-boundary workload leaked 6/18 canary markers.
sentinel-eval's test suite includes 23 local tests using mock vectors, completing in seconds, deterministic and requiring no API keys or network calls.
The reproduction commands for sentinel-eval are: git clone https://github.com/harishbalasubramanian/sentinel-eval, then run uv run sentinel-eval live --framework chroma.
All Facts (4)
sentinel-eval is a local testing tool, licensed under Apache-2.0, for quantifying authorization leakage risk in Agent memory scenarios.
50%UnverifiedIn sentinel-eval's tests, the original RAG scenario had a leakage rate of 0% (18 public chunks retrieved, 0 restricted chunks exposed); in the Agent memory scenario, 2 of 3 synthetic tasks leaked, and the cross-boundary workload leaked 6/18 canary markers.
50%Unverifiedsentinel-eval's test suite includes 23 local tests using mock vectors, completing in seconds, deterministic and requiring no API keys or network calls.
50%UnverifiedThe reproduction commands for sentinel-eval are: git clone https://github.com/harishbalasubramanian/sentinel-eval, then run uv run sentinel-eval live --framework chroma.
50%