DEVOURED
Brain the Size of a Planet: Are LLMs Thonking too Hard?
Extensive testing of 26 LLM configurations shows that higher reasoning effort and newer models frequently fail to outperform smaller models at security bug triage.
What: Researcher Parsia Yasini ran 2,080 test iterations using GPT-5.4/5.5 and Claude-4.6/4.7/4.8 on two security vulnerabilities, finding that function-level input significantly increases success rates over file-level input, where models often struggle.
Why it matters: This reveals that for security static analysis, prompt engineering and 'reasoning effort' settings often induce 'gaslighting' or excessive token usage without improving detection accuracy, especially when models are overwhelmed by context.
Takeaway: When using LLMs for security analysis, pass individual functions instead of entire source files to dramatically improve detection success.
Deep dive
- Tested 26 combinations of GPT-5.x and Claude-4.x models against two vulnerabilities: openbsd-sack and freebsd-nfs-vuln.
- Success rate for vulnerability detection was 70.8% at the function level, but dropped to 1.7% when passing the entire source file.
- 'High' or 'Extra High' reasoning effort often resulted in lower performance compared to 'Medium' settings.
- GPT-5.4/5.5 generally outperformed Claude equivalents in full vulnerability chain resolution.
- Claude models were the only ones to consistently reference CVEs in their analysis.
- Total cost for the full experimental series was approximately $9,200.
- An 'LLM Council' triage approach (majority voting) achieved 86.2% unanimity, making it an effective way to handle multiple AI outputs.
Decoder
- Triage: The process of determining the priority or relevance of security vulnerabilities.
- Content Filtering: An automated safety mechanism that blocks an LLM from responding to a prompt deemed sensitive or dangerous.
- Context Window: The maximum amount of information (in tokens) an LLM can consider in a single request.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi