I watched a small team
of artificial intelligence agents try to hack into my brand-new vibe-coded site a few weeks ago.
The AI agent, developed by startup RunSybil worked together to probe and identify weak points on my website. A orchestrator agent called Sybil oversees a number of specialized agents, powered by a combination custom language models and off the shelf APIs.
While conventional vulnerability scanners look for specific problems, Sybil can operate at a high level, using artificial intelligence to find weaknesses. It could, for instance, determine that a guest has privileged access – something a regular scan might miss – and use this information to build an assault. Ariel Herbert-Voss is the CEO and cofounder at RunSybil. She says that AI models with increasing capabilities will revolutionize offensive and defensive cybersecurity. Herbert-Voss said, “I’d argue that we are definitely on the verge of a technology boom in terms of capabilities which both bad and good actors could take advantage of.” “Our mission is building the next generation offensive security testing to help everyone keep up.”
Sybil targeted a website I created using Claude Code recently to help me sort new AI research papers. The site, which is called Arxiv slurper ( ) is a backend that searches through abstracts of papers for words such as “novel”‘first’,’surprising’ and other technical terms. It’s still a work-in-progress, but I was impressed by how easy it was for me to put together something useful, even though I had to fix some bugs and configuration issues manually.
The problem with a vibe-coded website is that you can’t be sure what security vulnerabilities you have introduced. When I talked to Herbert-Voss regarding Sybil, I asked if Sybil could test my site for vulnerabilities. Sybil found no vulnerabilities on my site, thankfully, and only because it is so basic.
Herbert Voss says that most vulnerabilities are the result of complex functionality such as forms, plugins and cryptographic features. We watched as the agents tried probing Herbert-Voss owns a dummy website that has known vulnerabilities. Sybil created a map of how the application is accessed and probed for weaknesses by manipulating parameters. It then tested edge cases and chained findings together, testing hypotheses and escalating it until it broke something meaningful. In this case it did identify ways of hacking the site. Sybil, Herbert-Voss claims, runs thousands of processes in parallel. It doesn’t miss any details and doesn’t stop. “The result is a machine that acts like an experienced attacker, but with machine precision and scale,” says Herbert-Voss.
According to Lujo Bauer, computer scientist at Carnegie Mellon University who specializes in AI/computer security, “AI-powered pen tests are a promising direction which can have significant benefits for protecting systems.” Bauer coauthored a recent article. A study that was conducted with researchers from CMU, Anthropic and other AI companies explores the promise AI penetration testing. The researchers found the most advanced commercial models were unable to perform network attacks, but they developed a system which set high-level goals like scanning a networking or infecting an host. This allowed them to perform penetration testing.
Sarah Guo is an investor and founder of investment firm Conviction, which is supporting RunSybil. She says that it is rare to come across people who are both knowledgeable about AI and cybersecurity. Guo says that RunSybil will make the type of security assessment that big companies perform periodically available to everyone, and on a continual basis. She says that “they can do baseline penetration tests with models and tools use continuously.” “So, you’ll have a constant view of what it looks like to be attacked.”
As attackers develop their AI strategies, the techniques being developed by RunSybil will become even more necessary. Bauer, from CMU, says that we have to assume attackers already use AI to their advantage. “Developing pen-testing software that uses it is both responsible as well as likely necessary to balance out the increasing risk of an attack.
Herbert Voss would be a good choice to help, as he was the original security researcher at OpenAI. Herbert-Voss explains that he has built a lot of crazy tools, including new prototypes of polymorphic spyware, spearphishing infrastructure and reverse engineering tools. I was worried that we wouldn’t have a way to deal with the situation when everyone, including the bad guys, has access to language models.
The AI Lab newsletteris a version of Will Knight’s AI Lab Newsletter. Here you can read previous newsletters .
