firsthire.md

Every benchmark we could find for agents doing advertising and business work

59 benchmarks and evaluations, from ad auctions and analytics to running a simulated company. For each: who made it, what the agent does, and how it is graded.

What we did not find: an independent benchmark where you connect your own agent and it takes real actions inside an ad account. That is the one we are building.

Compiled in November 2026 from public papers and pages. We list what each benchmark tests, not model scores: many published scores are vendor-reported or change between versions, so check the source. Something missing or wrong? This list will be corrected.

Advertising 28

  • AD-Bench

    Tencent-affiliated authors · 2026-02 (v2 2026-06) · Answers questions

    Offline replay of expert tool trajectories, tools re-executed on current data

    Graded by LLM judge (Gemini-3.1-Pro) plus trajectory coverage

  • AdCraft

    System1 · 2023-06 · Takes actions

    Gymnasium sim, non-stationary, sparse

    Graded by Profit-type reward

  • AdPilot / AdBench (KDD 2026)

    Tencent Advertising

    Autonomous ad delivery as MDP; 38 scenes, 7,600 instances. Learned simulator (95% feedback accuracy), 500k trajectories

    Graded by Business score (Pass@k) + LLM-judge reasoning score

  • arXiv 2609.22194 ("A Pinch of SFT, A Dash of RL")

    9 authors, affiliation not shown on page

    Enterprise advertiser-analytics agents, 8 advertiser skills; GPT-OSS 120B vs frontier control. "Production-mirroring beta APIs"

    Graded by Paired deltas, SME audit

  • AucArena

    Chen et al. · 2023-10 · Takes actions

    Simulated multi-item auctions

    Graded by Utility vs heuristics/humans

  • AuctionNet

    Alibaba, Peking Univ · 2024-12 (NeurIPS 2024) · Takes actions

    Simulated GSP auctions

    Graded by Outcome (value under constraints)

  • Autobidding Arena

    Avito-affiliated authors · 2025-10 · Takes actions

    Open-source industry bidding env

    Graded by Efficiency, cost, pacing RMSE

  • BAT

    Avito · 2025-05 (WWW 2025) · Takes actions

    Log-driven synthetic-augmented simulator

    Graded by Pacing uniformity, CPC constraint

  • Centific RL Environments-as-a-Service

    Centific (data/RL vendor)

    19 enterprise envs incl. Marketing; ad-creative env (21 scenarios, 13 tools per search snippet). "Safe connector replicas"

    Graded by "Verifier-ready" reward; no published scores

  • e-dialog hands-on: Claude + Meta Ads MCP

    e-dialog (agency)

    Read-only account analysis. Live Meta account

    Graded by Compared with existing client reports

  • Hawky.ai open benchmark v2

    Hawky.ai (performance-marketing AI vendor)

    494 knowledge/reasoning questions on Meta and Google Ads. Q&A, no account

    Graded by MC exact-match + rubric by Gemini 2.5 Flash

  • KP-Agent

    Academic · 2025-10 · Takes actions

    0.5M records, one pharma advertiser on Meituan

    Graded by Cumulative profit

  • LBM

    Academic · 2026-03 · Takes actions

    Not stated

    Graded by n/v

  • Marin Software LLM scorecard

    Marin (ad-mgmt vendor), Feb 2025

    4 LLMs, 3 marketing scenarios. Chat prompts

    Graded by 0-10 per scenario

  • Mark-Bench

    Optimizely · 2026-09 (vendor) · Answers questions

    Static scenarios

    Graded by Multi-LLM rubric judges

  • Marketing Benchmarks: Google Ads Optimization

    Unnamed small outfit; sells custom benchmarks, post-training, "Pulse" monitoring

    21 Google Ads tasks, 3 levels. Simulated Google Ads account, 28 tools, goal-only prompt

    Graded by LLM judge vs preloaded answers; required/bonus steps

  • Marketing Benchmarks: Influencer

    same

    14 tasks, simulated TikTok ecosystem. 21 tools, 120 creators, 1,200 videos, 250 competitor ads, 10 data files

    Graded by Hybrid programmatic + LLM

  • Marketing Benchmarks: SEO

    same

    32 tasks, simulated outdoor e-commerce site. 34 tools (22 read, 12 write), 15 data files

    Graded by Hybrid: programmatic tool-use checks + LLM rubric

  • MonBench

    Microsoft authors · 2026 (ACL Findings) · Answers questions

    Static production-derived data

    Graded by Multiple choice

  • OMS

    Chen, Wang, Takamatsu · 2025-07 · Actions and answers

    Real campaigns plus unnamed benchmarks

    Graded by Impressions, clicks, conversions, human eval

  • Optmyzr Claude vs ChatGPT via MCP

    Optmyzr (PPC software vendor)

    5 tests on same Optmyzr MCP account. Live data via vendor MCP

    Graded by None (qualitative)

  • PM-AGI

    Hawky.ai · 2026 (vendor) · Answers questions

    Static Q&A on Meta/Google Ads

    Graded by LLM judge (Gemini 2.5 Flash) plus exact match

  • RTBAgent

    Academic/industry · 2025-02 (WWW 2025) · Takes actions

    Offline ad dataset

    Graded by Profit

  • Ryze AI comparisons (Claude/ChatGPT/Gemini on Google Ads)

    Ryze AI (vendor)

    Claims 8-week, 3-account test; another page reports different winner. Live accounts (claimed)

    Graded by Vs control campaigns (claimed)

  • SimAB

    Academic/industry · 2026-03 · Answers questions

    Persona agents judge variants

    Graded by Agreement with real A/B

  • Synter "AI Agent Benchmarks"

    Synter (ad-automation vendor)

    AI-managed vs human-managed Google/Meta campaigns. 500 real campaigns, Jan-Dec 2025; 120 launches

    Graded by Outcome metrics (CTR, CPA, ROAS), no rubric, no significance tests

  • WordStream AI accuracy study

    WordStream (PPC software vendor), Jul 2025

    45 PPC Q&A across 5 AI tools. Q&A

    Graded by Accuracy review of 225 responses

  • Xnurta Retail Media AI Benchmark

    Xnurta (Amazon-ads software vendor)

    91 Amazon advertiser questions, portfolio analysis to budget pacing. "Live Amazon advertiser workflows" (no sandbox described)

    Graded by 5 dimensions; Claude Opus 4.7 as LLM judge; 9 human raters; accuracy (G3) as multiplier

Running a business 9

  • AgenticPay

    Liu, Gu, Song · 2026-02 · Takes actions

    Negotiation sim

    Graded by Feasibility, efficiency, welfare

  • CoffeeBench

    Sakana-affiliated · 2026-06 · Takes actions

    6-firm, 90-day multi-agent economy

    Graded by Net income

  • E-Commerce Bench

    Qwen team · 2026-08 · Takes actions

    Simulated, Taobao/Tmall-grounded, 365 days

    Graded by 7 dimensions incl. year-end assets, fraud avoidance

  • EcoGym

    Zhou et al. · 2026-02 · Takes actions

    3 simulated economies

    Graded by Net worth, income, DAU

  • Market-Bench

    SJTU · 2026-04 (ACL 2026) · Takes actions

    20-agent retail supply chain sim

    Graded by Profit, risk-adjusted return, semantic

  • Project Vend

    Anthropic + Andon · 2025-06 · Takes actions

    Live real shop

    Graded by Profit, qualitative

  • Vending-Bench

    Andon Labs · 2025-02 · Takes actions

    Simulated

    Graded by Final balance

  • Vending-Bench 2 / Arena

    Andon Labs · 2025-11 · Takes actions

    Simulated, 365 days

    Graded by Final bank balance

  • YC-Bench

    Collinear-linked authors · 2026-04 · Takes actions

    Simulated startup, 1 year

    Graded by Final funds

CRM, service and enterprise work 14

  • APEX-Agents

    Mercor · 2026-01 · Actions and answers

    33 file-based "worlds"

    Graded by Rubrics, gold outputs

  • CRMArena

    Salesforce · 2024-11 (NAACL 2025) · Actions and answers

    Live Salesforce org sandbox

    Graded by Task success

  • CRMArena-Pro

    Salesforce · 2025-05 (TMLR 2026-02) · Actions and answers

    Salesforce sandbox, synthetic data

    Graded by Task success, confidentiality

  • EnterpriseOps-Gym

    Academic · 2026-03 · Takes actions

    Docker, 164 tables, 512 tools

    Graded by Success rate

  • GDPval

    OpenAI · 2025-10 · Answers questions

    Deliverables

    Graded by Expert pairwise win-or-tie

  • MCP-Universe

    Salesforce · 2025-08 · Takes actions

    11 real MCP servers

    Graded by Execution-based

  • Remote Labor Index

    Scale / CAIS · 2025-10 · Answers questions

    Freelance projects

    Graded by Human acceptance as commissioned work

  • tau-bench

    Sierra / Princeton · 2024-06 (ICLR 2025) · Takes actions

    Simulated user + API DB

    Graded by End-state DB diff, pass^k

  • tau-Knowledge (tau-Banking)

    Sierra · 2026-03 · Takes actions

    ~700 interlinked docs

    Graded by Policy-compliant state change

  • tau2-bench

    Sierra / Princeton · 2025-06 · Takes actions

    Dual-control telecom

    Graded by State, pass^k

  • TheAgentCompany

    CMU · 2024-12 (v3 2025-09) · Takes actions

    Self-hosted company sim

    Graded by State checks plus LLM rubrics

  • Toolathlon

    HKUST / CMU et al. · 2025-10 · Takes actions

    32 apps, 604 tools

    Graded by Execution scripts

  • WorkArena++

    ServiceNow · 2024-07 (NeurIPS 2024) · Takes actions

    Live ServiceNow

    Graded by Validators

  • Zapier AutomationBench

    Zapier

    Cross-app business workflows incl. Marketing domain. CRM + inbox + calendar state

    Graded by Deterministic end-state checks, no LLM judge

Data and spreadsheets 3

  • DABstep

    Adyen + Hugging Face · 2025-06 · Answers questions

    Data analysis on payments data

    Graded by Exact factoid

  • Spider 2.0

    XLANG · 2024-11 (ICLR 2025 oral) · Actions and answers

    BigQuery, Snowflake, DBT

    Graded by Execution scripts

  • SpreadsheetBench 2

    Zhu et al. · 2026-06 · Takes actions

    Excel workflows

    Graded by Task accuracy

Shopping and the web 5

  • ECom-Bench

    Xiaoduo AI / SJTU · 2025-07 (EMNLP 2025) · Takes actions

    Simulated customers, MCP tools

    Graded by pass^3

  • ShoppingBench

    Alibaba-affiliated · 2025-08 (AAAI 2026 oral) · Takes actions

    2.5M-product sandbox

    Graded by n/v

  • TRAP

    Oxford et al. · 2025-12 (ICML 2026) · Takes actions

    Website clones

    Graded by Attack success

  • WebArena

    CMU · 2023-07 · Takes actions

    Self-hosted sites incl. e-commerce

    Graded by Functional correctness

  • WebMall

    Mannheim · 2025-08 (SIGIR 2026) · Takes actions

    4 offline shops

    Graded by Task completion