Secret Scanning
An automated security technique used to scan code repositories for leaked sensitive information such as API keys, tokens, and credentials. It forms an important component of code security management practices.
Timeline (last 90 days)
The GitHub team shared their hands-on experience evaluating large language models in a real-world secret scanning scenario
The secret scanning task requires the model to accurately identify leaked API keys, tokens, and credentials from vast amounts of code
In secret scanning scenarios, both false positives and false negatives carry extremely high costs: false negatives may lead to data breaches, while false positives overwhelm security teams with large volumes of invalid alerts
The GitHub team emphasizes that the evaluation process must set metric thresholds in conjunction with specific business objectives, as there is no universal answer for the balance point in secret scanning and it must be calibrated repeatedly based on real data
All Facts (4)
The secret scanning task requires the model to accurately identify leaked API keys, tokens, and credentials from vast amounts of code
50%UnverifiedIn secret scanning scenarios, both false positives and false negatives carry extremely high costs: false negatives may lead to data breaches, while false positives overwhelm security teams with large volumes of invalid alerts
50%UnverifiedThe GitHub team emphasizes that the evaluation process must set metric thresholds in conjunction with specific business objectives, as there is no universal answer for the balance point in secret scanning and it must be calibrated repeatedly based on real data
50%UnverifiedThe GitHub team shared their hands-on experience evaluating large language models in a real-world secret scanning scenario
50%