QASecClaw
A software assurance framework based on an agent architecture that combines language models with deterministic tools such as static analyzers and verifiers, with agents coordinating evidence collection and reasoning.
Timeline (last 90 days)
The study "Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance" explores through five experiments the reliability boundaries of multi-agent collaboration and verifier feedback in software assurance.
On eight related development warnings, increasing the context count from 1 to 50 led to a 49-fold increase in token usage but did not improve the rate of clearing insecure code.
In tests with two sets of 120 evidence packages each, the model rejected 70% to 100% of the applicable counterexamples, while checks constructed against specific scopes correctly classified all evidence packages.
The study recommends adopting a deterministic evidence admission strategy in software assurance testing environments, rather than blindly scaling up the number of agents or the context size.
In high-risk domains such as software assurance, the reliability of deterministic tools remains the cornerstone, and language models are better suited as an auxiliary role rather than a replacement.
Validator feedback can improve accuracy but may also introduce new errors, so an evidence admission mechanism needs to be established to ensure that only reliable evidence enters the model's decision-making step.
All Facts (6)
The study "Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance" explores through five experiments the reliability boundaries of multi-agent collaboration and verifier feedback in software assurance.
50%UnverifiedOn eight related development warnings, increasing the context count from 1 to 50 led to a 49-fold increase in token usage but did not improve the rate of clearing insecure code.
50%UnverifiedIn tests with two sets of 120 evidence packages each, the model rejected 70% to 100% of the applicable counterexamples, while checks constructed against specific scopes correctly classified all evidence packages.
50%UnverifiedThe study recommends adopting a deterministic evidence admission strategy in software assurance testing environments, rather than blindly scaling up the number of agents or the context size.
50%UnverifiedIn high-risk domains such as software assurance, the reliability of deterministic tools remains the cornerstone, and language models are better suited as an auxiliary role rather than a replacement.
50%UnverifiedValidator feedback can improve accuracy but may also introduce new errors, so an evidence admission mechanism needs to be established to ensure that only reliable evidence enters the model's decision-making step.
50%