Red Team AI is now building safer, smarter models for tomorrow

Editor’s note: Louis will be leading an editorial roundtable discussion on this topic this month at VB Transform. Register today by calling

AI model are under attack. The attackers’ cyber defenses are outpaced by their tradecraft. 77% of enterprises have already been hit by adversarial models attacks, and 41% of these attacks use prompt injections and poisoning data.

In order to reverse this trend, we must rethink the way security is integrated into models today. DevOps teams must shift from a reactive defense approach to continuous adversarial tests at every step.

The core of the DevOps process is red teaming.

To protect large language models across DevOps cycles, red teaming must be a key component. Continuous adversarial testing should be integrated into all phases of the Software Development Life Cycle.

Adopting a more integrative approach to DevSecOps fundamentals is becoming necessary to mitigate the growing risks of prompt injections, data poisoning and the exposure of sensitive data. Severe attacks like these are becoming more prevalent, occurring from model design through deployment, making ongoing monitoring essential.

Microsoft’s recent guidance on planning Red teaming for large languages models (LLMs),as well as their applications, provides a valuable method for starting an integrated process. NIST’s AI Risk Management Framework (19459066) reinforces this by highlighting the need for a proactive, lifecycle-long, adversarial testing approach and risk mitigation. Microsoft’s recent red-teaming of over 100 generative AI models underscores the importance of integrating automated threat detection and expert oversight throughout model creation. As regulatory frameworks such as the EU AI Act mandate rigorous adversarial tests, integrating continual red teaming ensures compliance.

Openai’s Approach to red teaming integrates red teams from early design to deployment, confirming the importance of consistent, preemptive testing for LLM development.

Why traditional cybersecurity approaches fail against AI

Traditional and long-standing cybersecurity approaches fail against AI-driven attacks because they are fundamentally distinct from conventional threats. Red teaming techniques must be updated as adversaries’ tactics surpass traditional methods. Here are some examples of the various types of tradecraft that can be used to attack AI models during DevOps cycles as well as once they have been deployed in the wild.

• : Adversaries inject corrupted training data into the sets to cause models to learn incorrectly, creating persistent inaccuracies or operational errors until the adversaries discover them. This undermines the trust in AI-driven decision making.
• Adversaries use carefully crafted input changes to evade detection systems. They do this by exploiting the inherent limitations in static rules and pattern based security controls.
• Model Inversion:Systematic queries on AI models allow adversaries to extract confidential data, potentially exposing proprietary or sensitive training data. This creates ongoing privacy risks.
• Injection Prompt: Adversaries create inputs that are specifically designed to trick generative AI, resulting in harmful or unauthorized outcomes.
• Dual Use Frontier Risks: (19459072) In a recent paper, Researchers from Australia have developed a framework for assessing and managing dual-use hazards of AI foundation models . The Center for Long-Term Cybersecurity, University of California, Berkeleyhighlights that advanced AI models lower barriers, enabling even non-experts carry out sophisticated cyberattacks or chemical threats, as well as other complex exploits. This fundamentally changes the global threat landscape, and increases risk exposure.

Integrated Machine Learning Operations, or MLOps, further compound these threats, vulnerabilities, and risks. The interconnectedness of LLM and the broader AI pipelines magnifies attack surfaces, which requires improvements in red teams.

Cybersecurity experts are increasingly adopting continual adversarial testing in order to counter these emerging AI-based threats. Structured red-team exercise are essential to uncover hidden vulnerabilities, and close security gaps prior to attackers exploiting them.

How AI Leaders Stay Ahead of Attackers with Red Teaming

The use of AI by adversaries to create new forms of tradecraft, which defy traditional cyber defenses, continues to increase. Their goal is exploiting as many vulnerabilities as possible.

Industry players, including major AI companies have responded by embedding sophisticated and systematic red-teaming at the core their AI security. Instead of treating red-teaming as an occasional test, they deploy continuous adversarial tests by combining expert insights, disciplined automated, and iterative “human-in-the middle” evaluations to uncover threats and reduce them before attackers could exploit them proactively.

They use rigorous methodologies to identify weaknesses in their models and harden them against real-world adversarial situations.

Anthropic relies heavily on human insight to support its ongoing red-teaming method. By integrating human-in the-loop evaluations and automated adversarial attack, the company proactively identifies weaknesses and continuously refines the reliability and accuracy of its models.

• Meta increases AI model security by automating adversarial testing. The Multi-round Automated Red-Teaming (MART), which is a system that generates adversarial prompts in an iterative fashion, reveals hidden vulnerabilities and narrows attack vectors for large AI deployments. Microsoft’s red-teaming is based on interdisciplinary collaboration. Microsoft’s Python Risk Identification Toolkit (PyRIT) combines cybersecurity expertise, advanced analytics, and a human-in-the middle validation to accelerate vulnerability detection, and provide detailed, actionable information to strengthen model resilience. OpenAI leverages global security expertise to strengthen AI defenses on a large scale. OpenAI, by combining the insights of external security experts with automated adversarial assessments and rigorous human validation processes, proactively addresses sophisticated threats. It targets misinformation and prompt injection vulnerabilities to maintain robust performance.

AI leaders understand that staying ahead of attackers requires continuous and proactive vigilance. These industry leaders have set the standard for resilient and trusted AI by integrating structured human oversight, disciplined automaton, and iterative refining into their red-teaming strategies.

As attacks on LLMs and AI models continue to evolve rapidly, DevOps and DevSecOps teams must coordinate their efforts to address the challenge of enhancing AI security. VentureBeat is finding the following five high-impact strategies security leaders can implement right away:

1. Integrate security early (Anthropic, OpenAI)
   Build adversarial testing directly into the initial model design and throughout the entire lifecycle. Catching vulnerabilities early reduces risks, disruptions and future costs.

• Deploy adaptive, real-time monitoring (Microsoft)
  Static defenses can’t protect AI systems from advanced threats. Leverage continuous AI-driven tools like CyberAlly to detect and respond to subtle anomalies quickly, minimizing the exploitation window.

• Balance automation with human judgment (Meta, Microsoft)
  Pure automation misses nuance; manual testing alone won’t scale. Combine automated adversarial testing and vulnerability scans with expert human analysis to ensure precise, actionable insights.

• Regularly engage external red teams (OpenAI)
  Internal teams develop blind spots. Periodic external evaluations reveal hidden vulnerabilities, independently validate your defenses and drive continuous improvement.

• Maintain dynamic threat intelligence (Meta, Microsoft, OpenAI)
  Attackers constantly evolve tactics. Continuously integrate real-time threat intelligence, automated analysis and expert insights to update and strengthen your defensive posture proactively.

Taken together, these strategies ensure DevOps workflows remain resilient and secure while staying ahead of evolving adversarial threats.

The use of red teams is no longer optional.

AI-based threats are too sophisticated and prevalent to rely on traditional reactive cybersecurity methods. To stay ahead of the curve, organizations must continuously and proactive embed adversarial tests into every stage model development. Leading AI providers have shown that innovation and robust security can coexist by balancing automation and human expertise, and adapting their defenses dynamically.

Red teams aren’t just for defending AI models. It’s all about building trust, resilience and confidence in an increasingly AI-driven future.

Join me at Transform 2025

I’ll be hosting two cybersecurity-focused r oundtables at VentureBeat’s Transform 2025 will be held on June 24-25, at Fort Mason in San Francisco. Register to join the discussion.

I will be presenting a session on red teams, AI Red Teaming & Adversarial Testis a comprehensive guide to testing and strengthening AI-driven cyber security solutions against sophisticated adversarial attacks.