Model benchmarks
Every model below ran the real Maddie agent against the same scenarios, with a simulated lead on the other end. These are the numbers we use to decide what runs for our clients.
Each model runs the exact conversation agent every Maddie tenant runs: the same tools, the same operating framework, the same live context and the same safeguards. Only the model changes.
Messaging, memory, follow-ups and the team's Slack are stood in by an in-memory sandbox. A fixed model plays the recipient from a hidden brief, so conversations unfold without a human and no real customer is ever texted.
Every scenario traces to something that went wrong or needed a rule in production: a message ignored mid-turn, a follow-up fired at 2 AM, a price quoted that should not have been, a link sent before the first reply.
Deterministic checks decide pass or fail from what the agent actually did. A separate judge model then rates the texts for how a sharp human teammate would read them. The judge never overrides a failed check.
70% weighted pass rate across scenarios, 30% judge quality. A model that gets the mechanics right but texts like a bot does not top the board.
How long a turn took, as the recipient would feel it: the median wait for a reply. Faster matters once a model clears the bar.
Client tools and CRMs are off in every run, so the board measures the conversation, not integrations. One run per model is shown; repeats are averaged.
Suite fd96c0d972cb: 116 scenarios across 16 families and 5 fictional businesses. Judge: anthropic/claude-sonnet-5. Simulated recipient: anthropic/claude-sonnet-5.
The owner says she handled the lead by phone herself
Northwind Remodeling · medium
The homeowner asks whether the 41k job can be done for 38
Northwind Remodeling · medium
A no-show returns asking about the next workshop and her refund question
Meridian Trading Academy · medium
Jamie reports the prospect's call has been moved to 4 PM
Apex Metabolic Coaching · easy
A booked prospect texts that he needs to cancel his call
Apex Metabolic Coaching · medium
The day-before confirmation comes due while the prospect is mid-conversation
Apex Metabolic Coaching · medium
A booked prospect might need to push his call, then offers Friday afternoon
Apex Metabolic Coaching · hard
A prospect repeats yesterday's cancel request the day before his call
Apex Metabolic Coaching · medium
A buyer offers two windows for a showing
Lakeside Realty Group · medium
The prospect has not watched the video and asks whether to push the call
Apex Metabolic Coaching · easy
The homeowner declines both windows and says next week afternoons
Northwind Remodeling · easy
The strategist never joined the Zoom and the prospect is done waiting
Apex Metabolic Coaching · medium
A reply to the homeowner fails to send on the spot
Northwind Remodeling · medium
A gentle reminder comes back refused: the lead has opted out
Evergreen Planning · easy
A long scripted pitch is due for a lead who has never replied
Evergreen Planning · medium
The team says a new lead's guide email bounced and passes along her address
Evergreen Planning · easy
The agent's last text fails because the line is sending too fast
Meridian Trading Academy · medium
A text to a booked prospect fails for the third time
Apex Metabolic Coaching · medium
The connection drops mid-send and nobody knows if the address went out
Lakeside Realty Group · hard
At 11 PM an advisor asks for a team update tomorrow morning
Meridian Trading Academy · easy
Three hours later the buyer asks whether there is any word on the showing
Lakeside Realty Group · easy
A student is locked out of the course before tonight's live Q and A
Meridian Trading Academy · medium
The prospect is on the Zoom and asks whether anyone is coming
Apex Metabolic Coaching · medium
An unknown number texts in after the home show asking about bathrooms
Northwind Remodeling · easy
The team asks for a guide link to go to a buyer who has never replied
Lakeside Realty Group · medium
A fresh Zillow inquiry arrives with a name and a property
Lakeside Realty Group · medium
A new booking arrives with the prospect's name, call time and application
Apex Metabolic Coaching · medium
A website estimate request arrives with a phone number and no name
Northwind Remodeling · medium
The day-one pitch comes due right after the lead says the guide never arrived
Evergreen Planning · medium
A go-live reminder comes due for a registrant silent for six days
Meridian Trading Academy · hard
A lead signs off with 'Thank you dear, talk soon'
Evergreen Planning · easy
Forty minutes after an unanswered pitch, with the call six days away
Apex Metabolic Coaching · medium
A reminder comes due after the prospect already said yes
Apex Metabolic Coaching · easy
On Saturday evening a registrant asks to be texted Monday at 11
Meridian Trading Academy · hard
A buyer goes quiet after being offered a showing
Lakeside Realty Group · hard
The Monday status update Priya asked for comes due again
Lakeside Realty Group · medium
The homeowner asks for a 6 or 6:30 PM walkthrough the company does not do
Northwind Remodeling · medium
An older lead asks the agent to just call her instead of texting
Evergreen Planning · medium
The next morning, nobody has confirmed either showing window
Lakeside Realty Group · medium
The homeowner picks Monday afternoon for his walkthrough
Northwind Remodeling · easy
The homeowner asks whether the measurements she emailed arrived
Northwind Remodeling · medium
A prospect shares his bloodwork and medication and asks if the program can fix it
Apex Metabolic Coaching · easy
A lead without a printer asks for the guide in the mail
Evergreen Planning · hard
Two hours after a confirmed call, nobody has said whether it happened
Apex Metabolic Coaching · medium
A prospect asks about payment plans, then about a guarantee
Apex Metabolic Coaching · medium
A prospect demands the price and cites a number from Reddit
Apex Metabolic Coaching · hard
A buyer asks the HOA fees and how long the listing has been on the market
Lakeside Realty Group · medium
A registrant asks if three modules in two weeks still qualifies for the refund
Meridian Trading Academy · medium
A homeowner asks how long a kitchen takes and how long the walkthrough takes
Northwind Remodeling · easy
An attendee says the website promises 30 days, not the 14 she was told
Meridian Trading Academy · medium
The owner says she emailed the homeowner the financing terms herself
Northwind Remodeling · hard
A new registrant arrives alongside a boilerplate 'no further action needed' notice
Meridian Trading Academy · hard
The coordinator reports the prospect is 'Confirmation Pending' after a phone call
Apex Metabolic Coaching · medium
The owner is mid-conversation with the homeowner from the dashboard
Northwind Remodeling · medium
The prospect answers Jamie's rescheduling text with two logistics questions
Apex Metabolic Coaching · hard
Priya texted the buyer herself this morning, and the agent's reminder comes due
Lakeside Realty Group · medium
The registrant changes her question while the agent is typing
Meridian Trading Academy · hard
A 'never mind' lands while the agent is typing the answer
Meridian Trading Academy · medium
Mid-reply, the buyer says she has her own agent now
Lakeside Realty Group · hard
Two team updates land while the agent is answering a quick question
Apex Metabolic Coaching · hard
'Two quick things' arrives, and the typing bubble is still going
Northwind Remodeling · medium
A homeowner asks for before and after pictures of past kitchens
Northwind Remodeling · easy
A prospect says nothing survives his travel schedule
Apex Metabolic Coaching · medium
A homeowner sends a video of her kitchen instead of picking a time
Northwind Remodeling · medium
Ninety minutes before his call, a prospect asks for the Zoom link again
Apex Metabolic Coaching · medium
Two days later, the homeowner asks which day they landed on
Northwind Remodeling · medium
The team passes along a resubmitted form with new details for a lead
Lakeside Realty Group · medium
A buyer who mentioned his pre-approval yesterday asks when they can see the house
Lakeside Realty Group · medium
A buyer confirms his pre-approval and adds his time zone and his dog's name
Lakeside Realty Group · hard
The prospect corrects his goal from 20 pounds to 35 and mentions Denver
Apex Metabolic Coaching · medium
A booked prospect says yes to the opener and asks what time his call is
Apex Metabolic Coaching · easy
A lead's first reply: she already has a prepaid plan
Evergreen Planning · easy
A lead texts that her husband passed away on Sunday
Evergreen Planning · hard
A buyer decides to use her cousin, who is a realtor
Lakeside Realty Group · medium
A quiet buyer asks to be checked on in a few weeks
Lakeside Realty Group · medium
A warm lead sends two logistics questions three minutes apart
Northwind Remodeling · easy
The prospect answers the opener with 'sure send them over'
Apex Metabolic Coaching · medium
An inline 'yes please' points at a two-day-old message
Lakeside Realty Group · medium
The prospect watched the case study and asks for more like it
Apex Metabolic Coaching · medium
The lead's whole reply to 'did the guide come through?' is 'No'
Evergreen Planning · medium
The prospect answers the light close with 'Okay 👍'
Apex Metabolic Coaching · easy
Money is tight with a kid in college, but he will still take the call
Apex Metabolic Coaching · medium
An older lead on a green-bubble thread closes with 'Ok thanks.'
Evergreen Planning · medium
A booked prospect's buddy is a trainer, so he will skip the call
Apex Metabolic Coaching · hard
A student ten days in asks to cancel and get a refund
Meridian Trading Academy · medium
The prospect taps Like on the day-before confirmation
Apex Metabolic Coaching · easy
The number turns out to be another company's marketing line
Evergreen Planning · hard
The team has paused the conversation, and the prospect texts a question
Apex Metabolic Coaching · medium
Two minutes after the opener, the prospect replies 'Stop'
Apex Metabolic Coaching · easy
The team confirmed the call by phone, and the day-before reminder comes due
Apex Metabolic Coaching · hard
A booked prospect asks to be emailed instead of texted
Apex Metabolic Coaching · medium
The owner settled a complaint by phone, and the homeowner asks about Monday
Northwind Remodeling · hard
A reminder comes due while the team has the conversation paused
Meridian Trading Academy · medium
A quiet buyer tells the agent to stop texting
Lakeside Realty Group · easy
'Wrong number', then a team invite for the same number a day later
Lakeside Realty Group · medium
A reminder comes due at 9:30 PM for a prospect known to be on Eastern time
Apex Metabolic Coaching · hard
At 10 PM a homeowner says she will talk to her husband tonight
Northwind Remodeling · medium
At 11:30 PM the homeowner asks to lock in Friday morning
Northwind Remodeling · easy
A reminder comes due at 11:40 PM for a buyer whose time zone is unknown
Lakeside Realty Group · hard
A workshop registration arrives at 1 AM from a Seattle number
Meridian Trading Academy · medium
A buyer relocating from Seattle asks for 'Thursday around 4'
Lakeside Realty Group · medium
A prospect will be in Sacramento tomorrow and wants his 9 AM call later
Apex Metabolic Coaching · medium
A booked prospect asks how long 'this sales call' is, then pushes back twice
Apex Metabolic Coaching · medium
'Are you a bot?' as a joke, then sincerely, then anger about the price
Apex Metabolic Coaching · medium
A one-word texter's workshop-day reminder comes due
Meridian Trading Academy · medium
Three quick logistics messages from a homeowner picking a day
Northwind Remodeling · easy
Three short replies from a buyer who was just greeted by name
Lakeside Realty Group · easy
A registrant does not want to sit through a 75 minute sales pitch
Meridian Trading Academy · medium
A 72 year old asks exactly what a planning session covers
Evergreen Planning · easy
'No free lunch': a registrant asks how the company really makes money
Meridian Trading Academy · medium
An admin's campaign text is due for a registrant who has never replied
Meridian Trading Academy · medium
'Please cancel.' then, six seconds later, 'actually next week'
Apex Metabolic Coaching · hard
A lead asks to be texted back in about 45 minutes
Evergreen Planning · easy
A quick question lands while the agent is doing housekeeping
Meridian Trading Academy · medium
An older lead on SMS confirms Tuesday and asks who calls whom
Evergreen Planning · medium
The team confirms Saturday's showing ten minutes after the buyer cancelled it
Lakeside Realty Group · medium