Delve into Darkreading’s curated compilation of emerging enterprises at the forefront of constructing technology and solutions aimed at evaluating the risk landscape of AI systems and machine learning models. As the realm of artificial intelligence (AI) extends beyond the boundaries of research and ventures into enterprise applications, cybersecurity leaders find themselves tasked with overseeing a novel layer of risk within their technological infrastructure. The role of CISOs now extends to encompass the incorporation of strategies that thwart fresh attack vectors, including adversarial AI assaults like model inversion attacks and data poisoning. Yet, this paints only a fraction of the overall risk spectrum.

The growing significance of resilience, reliability, and trust-related concerns—such as model fragility, AI bias, and interpretability—has added layers of complexity to the risk management paradigm.

In parallel, the AI landscape introduces heightened complexities to software supply chain dynamics, as open-source AI models and training data have become foundational for system development. Within this context, a promising opportunity emerges for new vendors, advisors, and consultants to step in and assist organizations in navigating these intricate risk terrains.

Envisioning the Future: Pioneers in AI Risk and Resilience
In this evolving domain, a collection of early frontrunners has emerged, showcasing their prowess in the realm of AI risk and resilience technology and services. As this niche continues to burgeon, these entities stand as a testament to the potential growth and transformation within the AI risk management landscape.

1. HiddenLayer
   HiddenLayer, a frontrunner in machine learning detection and response (MLDR), offers an innovative platform. It monitors machine learning (ML) algorithms’ inputs and outputs to spot adversarial ML attack techniques, aligned with MITRE ATLAS. With $6 million seed funding, it won RSA Conference 2023 Sandbox Innovation and gained early momentum.
   Founded: 2022 Founders: Chris Sestito, James Ballard, Tanner Burns
2. CalypsoAI
   CalypsoAI now focuses on safeguarding high-risk domains like defense, finance, and pharmaceuticals from the evolving wave of large language model (LLM) innovations. Their core technology, CalypsoAI Moderator, ensures data loss prevention, intellectual property protection, and guards against malicious LLM usage. Additionally, the company continues to support VESPR Validate, an ML model validation tool for stress-testing critical applications.
   Founded: 2018 Founder: Neil Serebryany
3. Robust Intelligence
   Robust Intelligence is a frontrunner in the burgeoning field. Its focus is developing advanced technology for stress-testing and fortifying AI/ML models in real-world deployment. At the core of the innovation is a platform featuring an AI firewall for real-time protection against undesired outputs. Continuous validation checks ensure adherence to predeployment AI risk standards and ongoing monitoring for anomalies. Going beyond cybersecurity, Robust Intelligence addresses ethical and operational risks that impact model trustworthiness. The company’s groundbreaking AI Risk Database aims to become the go-to community platform, akin to a “VirusTotal for AI.”
   Founded: 2019 Founders: Alexander Rilee, Eric Balkanski, Kojin Oshiba, Yaron Singer
4. Protect AI
   Protect AI is swiftly making waves in the AI risk management realm. Renowned cybersecurity expert Diana Kelley has joined as CISO, bringing her extensive security leadership background from IBM and Microsoft. Protect AI’s early offerings include AI Radar, providing vital visibility into ML attack surfaces and detecting risks, with a unique feature of generating a dynamic ML Bill of Materials. NB Defense, a complimentary Jupyter Notebooks scanning tool, focuses on early-stage ML experimentation vulnerabilities.
   Founded: 2022 Founder: Ian Swanso
5. Troj.AI
   Hailing from Canada, TrojAI, a seed-stage venture, is dedicated to crafting an arsenal of solutions for enterprises. Their focus encompasses model stress testing, model risk auditing, LLM protection, and AI firewall services. In the previous year, TrojAI secured $2.4 million in seed funding, with Flying Fish Ventures leading the investment.
   Founded: 2019 Founder: James Stewart
6. Adversa AI
   Adversa AI takes a distinctive path by focusing on AI assurance and assessment services rather than a product-centered approach. This self-funded venture is carving its niche in the consulting realm, offering education to clients’ technical and leadership teams about AI risks, spearheading AI strategy development, and executing an array of threat modeling and ethical hacking assessments within enterprise AI/ML ecosystems. Adversa is rapidly gaining recognition due to its dynamic AI threat research efforts, led by the Adversa AI Red Team. This team has notably published comprehensive research on various AI jailbreaks, attacks, and vulnerabilities over the past years.
   Founded: 2021 Founder: Alex Polyakov
7. DeepKeep
   DeepKeep brings forth MLProtect, a potent platform designed to bolster risk management and safeguard AI endeavors. Catering to data scientists, MLOps, and cybersecurity teams, MLProtect automates model pen tests, enhances AI security policies, promotes model explainability, and offers real-time production monitoring. Underpinning DeepKeep’s vision is an investment of approximately $5.45 million in seed funds from Awz Ventures. DeepKeep is further propelled by a two-year partnership with the Awz X-Seed Hub, an early-stage accelerator.
   Founded: 2021 Founder: Rony Ohayon
8. Cranium
   Emerging from KPMG Studio, Cranium carving a niche as an AI risk specialist. With a core focus on aiding enterprises in navigating dynamic compliance obligations, their AI Card technology streamlines compliance mapping to frameworks like NIST AI RMF and the EU AI Act. It further enhances risk assessment through visualizations, offering decision-makers and auditors a clear perspective on an organization’s model portfolio risk.
   Founded: 2022 Founders: Felix Knoll, Jonathan Dambrot, Paul Spicer

Source: Darkreading