News2min read

Project Deal: A Security Test of Autonomous AI Agents in a Marketplace

Anthropic conducted an experiment where Claude agents autonomously negotiated and executed trades for employees, revealing critical security dynamics regarding Tool Call permissions, adversarial prompting against autonomous systems, and the impact of model capability on negotiation outcomes.

High relevanceAgent SecurityLLM SecuritySecure AI Development

Project Deal was a live experiment at Anthropic to observe how AI models function as independent trading entities. Employees were assigned custom Claude agents with specific budgets ($100) that operated within an internal Slack-based marketplace without human intervention or post-trade sign-off.

The system tested the boundaries of Tool Call Permissions, where agents could list items, make offers, and execute trades based on data gathered from initial interviews. Security engineers actively engaged in Red Teaming exercises against these running agents to identify potential vectors for Prompt Injection or Jailbreaks. The experiment compared two model configurations: a frontier model (Claude Opus 4.5) and a smaller, less capable model (Claude Haiku 4.5).

Results showed that while the system successfully facilitated over $4,000 in transactions via autonomous agents, there were significant security implications regarding untrusted data sources and adversarial inputs. Participants represented by weaker models achieved objectively better financial outcomes but remained unaware of this disadvantage due to a lack of transparency about model capabilities.

Why it matters

This experiment is highly relevant for AI-security practitioners because it demonstrates the practical risks associated with granting autonomous agents broad **Tool Call** permissions in uncontrolled environments. The active engagement in **Red Teaming** against live trading agents highlights how easily adversarial prompts can influence decision-making when humans are absent from the loop. Furthermore, the disparity in outcomes based on model size underscores the need for rigorous evaluation of agent capabilities before deployment to high-stakes commercial interactions.

Takeaways

  • Autonomous AI agents require strict **Guardrails** and limited **Tool Permissions**, as demonstrated by the uncontrolled nature of this marketplace experiment.
  • **Red Teaming** against live trading agents is essential to identify vulnerabilities in how models interpret instructions from external data sources.
  • The disparity between model capabilities (Opus vs. Haiku) suggests that weaker models may be more susceptible to manipulation or less effective at reasoning through complex negotiations.

Sources