Local LLM Selection Guide: Practical Applications for Cybersecurity Purple Teams

Introduction: When Security Teams Need Local LLMs

As the capabilities of large language models (LLMs) rapidly advance, an increasing number of cybersecurity professionals are exploring ways to integrate them into their daily workflows. Recently, a security engineer on Reddit posed a highly representative question: how to locally deploy LLMs suitable for Purple Team operations and SOC L2/L3 analysis?

A Purple Team is a collaborative exercise model in cybersecurity that combines the functions of Red Teams (attack simulation) and Blue Teams (defensive detection). Unlike traditional red-blue adversarial exercises, Purple Teams emphasize real-time collaboration—after the Red Team executes an attack technique, the Blue Team immediately validates whether detection capabilities are effective, and both sides work together to optimize defensive strategies. Within the SOC (Security Operations Center) tiered system, L1 handles initial alert triage and filtering, L2 conducts in-depth investigation and correlation analysis, and L3 deals with advanced threat hunting, reverse engineering, and complex incident response. L2/L3 analysts typically need to understand the full attack chain, write detection rules, and analyze malicious samples—tasks that LLMs can help accelerate.

This user's hardware configuration is quite impressive—128GB DDR5 RAM, a high-end i9 processor—though the GPU only has 8GB of VRAM. Their core requirements are very clear: they want a model capable of generating payloads targeting specific technology stacks, analyzing logs and alerts, understanding context and quickly extracting key data, ultimately serving to speed up report writing and uncover security insights they might have missed.

Behind this requirement lies a deeper industry issue: in security scenarios, do we actually need "censored" or "uncensored" models? This article will analyze the topic through the lenses of hardware constraints, model selection, and practical considerations.

Reddit original post discussion

Hardware Reality: VRAM Is the True Bottleneck for Local Inference

Large RAM ≠ Ability to Run Large Models

Many people are easily misled by the 128GB RAM figure, thinking they can effortlessly run ultra-large parameter models. However, in local inference scenarios, GPU VRAM is the decisive factor. 8GB of VRAM means that with pure GPU loading, you can only comfortably run 7B–8B class quantized models (such as Q4 quantization).

Here it's important to understand the principles behind quantization: quantization is a technique that compresses model weights from high-precision floating-point numbers (such as FP16, FP32) to lower-precision representations (such as INT4, INT8). With Q4 quantization, for example, each weight uses only 4 bits of storage, reducing VRAM usage by approximately 75% compared to FP16's 16 bits. A 7B parameter model requires about 14GB of VRAM in FP16, but only about 4–5GB after Q4 quantization. GGUF is a quantized model file format defined by the llama.cpp project that supports flexible mixed-precision and CPU/GPU layered loading. While quantization introduces some precision loss, at Q4/Q5 levels the impact on actual task performance is typically limited—especially for security analysis scenarios that don't require the highest quality language generation.

For Purple Team and SOC analysis tasks that demand a certain level of reasoning capability, this limitation cannot be ignored. The good news is that through CPU + GPU hybrid inference (such as llama.cpp's offloading mechanism), combined with ample system memory, users can absolutely run 30B or even larger parameter models—just at noticeably reduced speeds.

The core idea of hybrid inference is to distribute different layers of the Transformer model across the GPU and CPU. llama.cpp uses the --n-gpu-layers parameter to control how many layers are loaded into GPU VRAM, with remaining layers computed by the CPU in system memory. Layers processed by the GPU benefit from parallel computation acceleration, while CPU-processed layers depend on memory bandwidth and are significantly slower than GPU processing. DDR5 memory bandwidth is typically 50–80 GB/s, while modern GPU memory bandwidth can reach 300–1000 GB/s, so layers placed on the CPU become the primary inference speed bottleneck. The advantage of 128GB DDR5 is the ability to fully load all weights of a 70B-class model into memory—even if the speed is slow (approximately 2–5 tokens/s), it remains usable for non-real-time tasks like report writing and log analysis.

Censored vs. Uncensored Models: A Critical Choice for Security Scenarios

Why This Issue Is Particularly Prominent in Security

The original post title specifically noted "da valutare se uncensored o no" (need to evaluate whether to use uncensored models), which precisely highlights the pain point for security practitioners.

Mainstream commercial models (such as GPT, Claude) and most open-source models have built-in safety alignment mechanisms. Safety alignment refers to using techniques like RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) to train models to refuse harmful instructions. When you request "generate a payload for a specific technique," these models often refuse to respond or give highly vague answers. This mechanism protects ordinary users from harm in general scenarios, but for legally authorized penetration testing and Red Team exercises, it's a genuine efficiency barrier.

The Value and Risks of Uncensored Models

Uncensored or "de-aligned" fine-tuned models (such as the Dolphin series, certain abliterated versions) are more willing to cooperate in generating payloads and explaining exploitation principles. De-alignment (also called abliteration or uncensoring) is the process of removing safety restrictions through fine-tuning. Common methods include continued training on refusal-free datasets and directly modifying the model's internal "refusal direction" vectors through representation engineering. The Dolphin series, maintained by Eric Hartford, removes aligned responses from training data during fine-tuning, making the model behave as a pure instruction follower.

For Purple Team work, this means:

Advantages: In authorized environments, it can directly output usable technical content, reducing the time cost of repeatedly "bypassing" prompt restrictions.
Risks: Output quality from uncensored models varies widely—they may generate outdated, incorrect, or even dangerous code that requires strict review by professionals.

It must be emphasized that using uncensored models for unauthorized attack activities is illegal in most jurisdictions.

Core Principle: Regardless of which model you choose, the greatest value of local deployment is that data never leaves the premises—sensitive logs, alerts, and internal information won't be uploaded to third parties, which is crucial for SOC environments with strict compliance requirements.

Specific Model Recommendations: From General Reasoning to Security-Specific

Candidate Models for Reasoning and Analysis

Based on the current open-source ecosystem and this user's requirement for "a certain level of reasoning capability," the following models are worth evaluating:

Qwen2.5 Series (14B/32B): Balanced capabilities across Chinese, English, and code; excellent instruction following; suitable for log analysis and report generation.
DeepSeek-R1 Distilled Versions: Strong chain-of-thought reasoning capabilities; suitable for alert triage scenarios requiring "understanding context and drawing conclusions."
Llama 3.1 (8B/70B): Mature ecosystem with abundant fine-tuned versions; 8B can run within VRAM; 70B requires hybrid inference.

Security-Specific Fine-Tuned Models

WhiteRabbitNeo: Specifically fine-tuned for cybersecurity scenarios; more accommodating for payload generation and vulnerability analysis tasks; a strong candidate for Purple Team work.
Dolphin (based on Mistral/Qwen): De-censored processing combined with capable base models; suitable for authorized penetration testing.

A Combined Strategy Is More Practical Than a Single Model

In practice, a single model can rarely handle all tasks effectively. A division-of-labor approach is recommended: use models with strong reasoning capabilities for log analysis and contextual understanding, use security-specific/uncensored models for payload generation, and use general-purpose models to assist with report writing.

Key Recommendations for Practical Implementation

Build a RAG Knowledge Base to Improve Accuracy

Relying solely on the model's "memory" is far from sufficient. It's recommended to build a Retrieval-Augmented Generation (RAG) knowledge base for common MITRE ATT&CK techniques, internal asset information, and historical alert cases, allowing the model to reference the most current and relevant contextual data when answering.

RAG works by retrieving the most relevant document fragments from an external knowledge base before the model generates its answer, injecting them as context into the prompt. In security scenarios, this means building a vector database from the hundreds of technique descriptions in the MITRE ATT&CK framework, CVE vulnerability details, internal SOC handbooks, historical incident reports, and more. MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) is the world's most widely used knowledge base of cyber attack behaviors, systematically categorized by tactics (such as Initial Access, Lateral Movement, Data Exfiltration) and techniques, currently containing over 200 attack techniques and nearly 700 sub-techniques. Injecting ATT&CK knowledge into a local LLM's RAG system enables the model to automatically correlate attack stages and recommend response measures when analyzing alerts.

Commonly used local RAG tech stacks include ChromaDB or Milvus as vector databases, sentence-transformers for generating embedding vectors, paired with LangChain or LlamaIndex for orchestrating retrieval and generation workflows. This approach effectively compensates for the model's training data timeliness limitations, especially for rapidly evolving threat intelligence.

Toolchain Integration, Not Isolated Use

Embed local models into existing SIEM/SOAR workflows, using APIs to let them automatically parse logs and generate preliminary analysis conclusions, with humans making final decisions. This is the correct approach to "speeding up the reporting phase."

SIEM (Security Information and Event Management) systems like Splunk, Elastic Security, and QRadar are responsible for collecting and correlating security logs and alerts from across the network. SOAR (Security Orchestration, Automation, and Response) platforms like Palo Alto XSOAR and Splunk SOAR automate response processes—for example, automatically blocking IPs, isolating hosts, and notifying analysts when high-severity alerts are received. Integrating local LLMs via API into these toolchains enables automated preliminary alert triage: after SIEM generates an alert, a SOAR playbook calls the local model API to perform natural language analysis on the alert context, outputting initial assessments and suggested remediation actions for analysts to simply review and confirm. This pattern can multiply L1-stage alert classification efficiency several times over, freeing L2/L3 analysts to focus on truly complex incidents.

Always Keep Humans in the Loop

No matter how powerful the model, the security domain cannot go fully automated. Model-generated payloads must be verified in isolated environments, and analysis conclusions must be reviewed by L2/L3 analysts. AI is a force multiplier, not a replacement.

Conclusion

This Reddit user's question actually represents a common exploration shared by many security teams today: finding the optimal landing point for local LLMs between compliance, efficiency, and capability. The answer isn't some "silver bullet" model, but rather a systems engineering approach of realistic model selection based on hardware constraints + division of labor + toolchain integration + human-in-the-loop. For an 8GB VRAM configuration, starting with 14B–32B quantized models paired with a RAG knowledge base offers a stable path that balances capability with feasibility.