Building an agentic AI helpdesk from zero
Building an agentic AI helpdesk from zero
How design insights turned a DevOps platform into a customer support AI agent that drove $780K in ARR in 14 months.
Impact
$780K
ARR in 14 months,
from zero
780K
14 months
90%
human approval rate
on AI drafted emails
90%
Approval rate
60%
less agent workload
with QueryPal
80%
tickets resolved
at first contact
80%
tckts resolved
Role
Founding Designer
Product Design Lead
Team
CEO, VP of Marketing, LLM Engineers
Product Design Lead
Timeline
2024 - 2025, 14 months
2018 - 9 months
Outcome
$780K ARR
AI Cloud Software
QueryPal / O > 1 AI Startup
QueryPal / O > 1 AI Startup
My role
I WAS THE FOUNDING AND ONLY DESIGNER
I owned every design surface the company shipped: product UX, bot configuration UX, admin and analytics dashboard, brand identity, the GTM marketing site, and the prompt and eval system governing agent behavior.
I architected the admin object model from the ground up, designed the three surface component system shared across Slack, email, and the admin dashboard, and worked with the LLM engineers on the tone constraints governing every customer facing draft. No PM, no researcher, no second designer.
How we got here (Story arc)
TWO YEARS OF BUILDING THE WRONG PRODUCT TAUGHT US THE MARKET
QueryPal started as an observability platform designed to help SRE teams make sense of metrics, logs, and traces. But as we introduced AI, we saw a fundamental shift: people no longer needed another dashboard to interpret their data. They wanted to ask a question and get an answer.
That insight changed our direction.
Support teams had the same problem: hours spent answering repetitive questions that could be resolved from information already sitting across the organization. Unlike SRE teams, they had a clear business case for automation and a budget line to pay for it.
The product evolved from observability → AI-powered answers → customer support automation.
ACT 1
DevOps Observability
I designed an RCA tool unifying metrics, logs, and events into one interface. As the models matured, the dashboard started competing with the intelligence users actually wanted. The lesson: visualization wasn’t the product. Fast answers were.
Unified metrics, logs, and events into a single RCA interface.
The lesson: Users didn't want visual dashboards—they wanted fast answers.
ACT 2
QueryPal Slackbot
We shifted from dashboards to native chat. I designed QueryPal’s Slackbot so teams could resolve issues where they already worked, with no tab switching. The catch: high usage didn't mean high willingness to pay.
Brought issue resolution directly into native Slack workflows. The catch: High daily usage didn’t turn into paying customers.
ACT 3
AI Email Drafts for CX
We pivoted into customer support, generating on-brand email drafts from existing company knowledge. Agents reviewed and resolved tickets in minutes instead of hours.
The breakthrough: clear ROI, a clear buyer, and product-market fit.
Turned internal knowledge into instant, on-brand support replies.
The win: Clear buyer, measurable ROI, and true PMF.
ACT 1: Devops observability
VIEW THE BEFORE DASHBOARD
I designed the RCA dashboard with a senior DevOps engineer: metrics, logs and events in one view.
ACT 2
We didn’t pivot to an AI Slackbot by accident. CtrlStack showed us that dashboards explained problems, but decisions happened in Slack.
ACT 2
I led GTM with our VP of Marketing. Ads, Whitepapers, and test marketing sites to check which ICPs signed up the most.
ACT 3
The Slackbot validated the behavior, but testing interest across marketing personas clarified the buyer. CX leaders responded the strongest.
ACT 1: Devops observability
VIEW THE BEFORE DASHBOARD
I designed the RCA dashboard with a senior DevOps engineer: metrics, logs and events in one view.
ACT 2
We didn’t pivot to an AI Slackbot by accident. CtrlStack showed us that dashboards explained problems, but decisions happened in Slack.
ACT 2
I led GTM with our VP of Marketing. Ads, Whitepapers, and test marketing sites to check which ICPs signed up the most.
ACT 3
The Slackbot validated the behavior, but testing interest across marketing personas clarified the buyer. CX leaders responded the strongest.
Object mapping
Mapped the admin model before drawing a screen
Card sorting set the object structure first: accounts, integrations, automations, billing. Screens followed the model.
Admin dashboard - Onboarding
Three steps to a working bot
Connect a source, invite the team, set the tone. Three tasks, one screen, progress visible.
Design System
One component set across Slack, email, and admin
Integration pickers, tone controls, and approval states built as shared primitives. Three surfaces, one vocabulary, no redraws.
Trust and Adoption
CONFIDENCE METERS EARNED AGENT TRUST IN AI DRAFTS
A user study audit showed agents rating AI drafts poorly and rewriting good answers. Pilot accounts were at risk.
BEFORE - Every draft looked the same whether the AI was certain or guessing. Agents reviewed all of them from scratch.
AFTER - Score on every ticket, plain language reasons, and labeled knowledge sources inside the agent's existing help desk.
THE AUDIT
Agents could not verify the AI
My user study audit of pilot teams traced low ratings to missing evidence, not wrong answers. Agents rewrote good drafts because nothing showed why to trust them.
THE BUILD
A confidence layer in the help desk
I drove a score on every draft, the reasons behind it, and sources labeled live, matched, referenced, or verified. Approve, edit, and deny stay with the agent on every ticket.
THE RESULT
Ratings recovered and accounts stayed
Queue level scores let agents approve high confidence drafts fast and focus on the low ones. Approval reached 90% across seven pilot accounts.
AI CONFIDENCE
92%
HIGH CONFIDENCE
AI is confident. Ready to send.
Exact match in license portal
Real time account data, current today
No ambiguity in the question
Clear, documented plan policy
KNOWLEDGE SOURCES
License Portal
live
Enterprise Plan Policy
matched
Toolbox Pricing
referenced
Account CRM Data
verified
HOW THE SCORE MAPS TO ACTION
Three tiers, three agent actions. Attention goes where the AI is least sure.
85% AND UP
HIGH CONFIDENCE
Clear policy, recent data. Cited answer with a verified label.
Accept
60 TO 85%
MEDIUM CONFIDENCE
Some ambiguity or older data. Answer with a caveat and a prompt to review.
Edit
UNDER 60%
LOW CONFIDENCE
Missing data or an edge case. States uncertainty and routes to an expert.
Dismiss
68%
of AI drafts approved by agents with no edits at all
80%
tickets resolved at
first contact
7
pilot accounts running confidence meters in their help desk
Interactive prototype. Toggle Before, Cited, After, and Final.
Analytics Dashboard
MAKING THE AGENT'S VALUE VISIBLE
Support leaders do not renew on ticket counts. They renew on proof the agent is absorbing work their team would otherwise do.
What I got wrong
I DESIGNED TRUST LAST AND IT SHOULD HAVE COME FIRST
Confidence meters were a rescue, not a plan. We shipped AI drafts with no evidence attached, and agents told us so in their ratings.
THE MISTAKE
I treated accuracy as the trust problem
I assumed better answers would earn approval on their own. The first release gave agents a draft and nothing to check it against.
WHAT IT COST
Low ratings and pilots close to leaving
Agents rewrote drafts that were already right and scored the AI poorly. We had to win back accounts that should never have been at risk.
WHAT I DO NOW
Evidence ships with the first AI output
Sources, a confidence signal, and an easy way to reject go into release one. I test for trust in the first study, not after the ratings come in.
Outcomes
QUERYPAL WENT FROM CONCEPT TO PAID PILOTS IN 14 MONTHS
$780K
Grew from 0 to $780K ARR with design embedded across product, brand, and GTM
90%
human approval on AI drafted replies across seven pilot accounts.
60%
reduction in tier 1 support workload, freeing agents for cases that needed a human.
780K
14 months
90%
Approval rate
80%
tckts resolved
Testimonials
WHAT THE PILOT TEAMS SAID
"We completely removed Freddy AI in the first week after implementing QueryPal."
- Customer Support Lead, JetBrains
"Our agents rely on QueryPal daily & we’ve never handled peak season this smoothly"
- Operations Lead, SimplyBenefits
"We completely removed Freddy AI in the first week after implementing QueryPal."
- Customer Support Lead, JetBrains
"Our agents rely on QueryPal daily & we’ve never handled peak season this smoothly"
- Operations Lead, SimplyBenefits
EXPLORE MORE
Building an agentic AI helpdesk from zero
Building an agentic AI helpdesk from zero
How design insights turned a DevOps platform into a customer support AI agent that drove $780K in ARR in 14 months.
Impact
$780K
ARR in 14 months,
from zero
780K
14 months
90%
human approval rate
on AI drafted emails
90%
Approval rate
60%
less agent workload
with QueryPal
80%
tickets resolved
at first contact
80%
tckts resolved
Role
Founding Designer
Product Design Lead
Team
CEO, VP of Marketing, LLM Engineers
Product Design Lead
Timeline
2024 - 2025, 14 months
2018 - 9 months
Outcome
$780K ARR
AI Cloud Software
QueryPal / O > 1 AI Startup
QueryPal / O > 1 AI Startup
My role
I WAS THE FOUNDING AND ONLY DESIGNER
I owned every design surface the company shipped: product UX, bot configuration UX, admin and analytics dashboard, brand identity, the GTM marketing site, and the prompt and eval system governing agent behavior.
I architected the admin object model from the ground up, designed the three surface component system shared across Slack, email, and the admin dashboard, and worked with the LLM engineers on the tone constraints governing every customer facing draft. No PM, no researcher, no second designer.
How we got here (Story arc)
TWO YEARS OF BUILDING THE WRONG PRODUCT TAUGHT US THE MARKET
QueryPal started as an observability platform designed to help SRE teams make sense of metrics, logs, and traces. But as we introduced AI, we saw a fundamental shift: people no longer needed another dashboard to interpret their data. They wanted to ask a question and get an answer.
That insight changed our direction.
Support teams had the same problem: hours spent answering repetitive questions that could be resolved from information already sitting across the organization. Unlike SRE teams, they had a clear business case for automation and a budget line to pay for it.
The product evolved from observability → AI-powered answers → customer support automation.
ACT 1
DevOps Observability
I designed an RCA tool unifying metrics, logs, and events into one interface. As the models matured, the dashboard started competing with the intelligence users actually wanted. The lesson: visualization wasn’t the product. Fast answers were.
Unified metrics, logs, and events into a single RCA interface.
The lesson: Users didn't want visual dashboards—they wanted fast answers.
ACT 2
QueryPal Slackbot
We shifted from dashboards to native chat. I designed QueryPal’s Slackbot so teams could resolve issues where they already worked, with no tab switching. The catch: high usage didn't mean high willingness to pay.
Brought issue resolution directly into native Slack workflows. The catch: High daily usage didn’t turn into paying customers.
ACT 3
AI Email Drafts for CX
We pivoted into customer support, generating on-brand email drafts from existing company knowledge. Agents reviewed and resolved tickets in minutes instead of hours.
The breakthrough: clear ROI, a clear buyer, and product-market fit.
Turned internal knowledge into instant, on-brand support replies.
The win: Clear buyer, measurable ROI, and true PMF.
ACT 1: Devops observability
VIEW THE BEFORE DASHBOARD
I designed the RCA dashboard with a senior DevOps engineer: metrics, logs and events in one view.
ACT 2
We didn’t pivot to an AI Slackbot by accident. CtrlStack showed us that dashboards explained problems, but decisions happened in Slack.
ACT 2
I led GTM with our VP of Marketing. Ads, Whitepapers, and test marketing sites to check which ICPs signed up the most.
ACT 3
The Slackbot validated the behavior, but testing interest across marketing personas clarified the buyer. CX leaders responded the strongest.
ACT 1: Devops observability
VIEW THE BEFORE DASHBOARD
I designed the RCA dashboard with a senior DevOps engineer: metrics, logs and events in one view.
ACT 2
We didn’t pivot to an AI Slackbot by accident. CtrlStack showed us that dashboards explained problems, but decisions happened in Slack.
ACT 2
I led GTM with our VP of Marketing. Ads, Whitepapers, and test marketing sites to check which ICPs signed up the most.
ACT 3
The Slackbot validated the behavior, but testing interest across marketing personas clarified the buyer. CX leaders responded the strongest.
Object mapping
Mapped the admin model before drawing a screen
Card sorting set the object structure first: accounts, integrations, automations, billing. Screens followed the model.
Admin dashboard - Onboarding
Three steps to a working bot
Connect a source, invite the team, set the tone. Three tasks, one screen, progress visible.
Design System
One component set across Slack, email, and admin
Integration pickers, tone controls, and approval states built as shared primitives. Three surfaces, one vocabulary, no redraws.
Trust and Adoption
CONFIDENCE METERS EARNED AGENT TRUST IN AI DRAFTS
A user study audit showed agents rating AI drafts poorly and rewriting good answers. Pilot accounts were at risk.
BEFORE - Every draft looked the same whether the AI was certain or guessing. Agents reviewed all of them from scratch.
AFTER - Score on every ticket, plain language reasons, and labeled knowledge sources inside the agent's existing help desk.
THE AUDIT
Agents could not verify the AI
My user study audit of pilot teams traced low ratings to missing evidence, not wrong answers. Agents rewrote good drafts because nothing showed why to trust them.
THE BUILD
A confidence layer in the help desk
I drove a score on every draft, the reasons behind it, and sources labeled live, matched, referenced, or verified. Approve, edit, and deny stay with the agent on every ticket.
THE RESULT
Ratings recovered and accounts stayed
Queue level scores let agents approve high confidence drafts fast and focus on the low ones. Approval reached 90% across seven pilot accounts.
AI CONFIDENCE
92%
HIGH CONFIDENCE
AI is confident. Ready to send.
Exact match in license portal
Real time account data, current today
No ambiguity in the question
Clear, documented plan policy
KNOWLEDGE SOURCES
License Portal
live
Enterprise Plan Policy
matched
Toolbox Pricing
referenced
Account CRM Data
verified
HOW THE SCORE MAPS TO ACTION
Three tiers, three agent actions. Attention goes where the AI is least sure.
85% AND UP
HIGH CONFIDENCE
Clear policy, recent data. Cited answer with a verified label.
Accept
60 TO 85%
MEDIUM CONFIDENCE
Some ambiguity or older data. Answer with a caveat and a prompt to review.
Edit
UNDER 60%
LOW CONFIDENCE
Missing data or an edge case. States uncertainty and routes to an expert.
Dismiss
68%
of AI drafts approved by agents with no edits at all
80%
tickets resolved at
first contact
7
pilot accounts running confidence meters in their help desk
Interactive prototype. Toggle Before, Cited, After, and Final.
Analytics Dashboard
MAKING THE AGENT'S VALUE VISIBLE
Support leaders do not renew on ticket counts. They renew on proof the agent is absorbing work their team would otherwise do.
What I got wrong
I DESIGNED TRUST LAST AND IT SHOULD HAVE COME FIRST
Confidence meters were a rescue, not a plan. We shipped AI drafts with no evidence attached, and agents told us so in their ratings.
THE MISTAKE
I treated accuracy as the trust problem
I assumed better answers would earn approval on their own. The first release gave agents a draft and nothing to check it against.
WHAT IT COST
Low ratings and pilots close to leaving
Agents rewrote drafts that were already right and scored the AI poorly. We had to win back accounts that should never have been at risk.
WHAT I DO NOW
Evidence ships with the first AI output
Sources, a confidence signal, and an easy way to reject go into release one. I test for trust in the first study, not after the ratings come in.
Outcomes
QUERYPAL WENT FROM CONCEPT TO PAID PILOTS IN 14 MONTHS
$780K
Grew from 0 to $780K ARR with design embedded across product, brand, and GTM
90%
human approval on AI drafted replies across seven pilot accounts.
60%
reduction in tier 1 support workload, freeing agents for cases that needed a human.
780K
14 months
90%
Approval rate
80%
tckts resolved
Testimonials
WHAT THE PILOT TEAMS SAID
"We completely removed Freddy AI in the first week after implementing QueryPal."
- Customer Support Lead, JetBrains
"Our agents rely on QueryPal daily & we’ve never handled peak season this smoothly"
- Operations Lead, SimplyBenefits
"We completely removed Freddy AI in the first week after implementing QueryPal."
- Customer Support Lead, JetBrains
"Our agents rely on QueryPal daily & we’ve never handled peak season this smoothly"
- Operations Lead, SimplyBenefits
EXPLORE MORE
Building an agentic AI helpdesk from zero
Building an agentic AI helpdesk from zero
How design insights turned a DevOps platform into a customer support AI agent that drove $780K in ARR in 14 months.
Impact
$780K
ARR in 14 months,
from zero
780K
14 months
90%
human approval rate
on AI drafted emails
90%
Approval rate
60%
less agent workload
with QueryPal
80%
tickets resolved
at first contact
80%
tckts resolved
Role
Founding Designer
Product Design Lead
Team
CEO, VP of Marketing, LLM Engineers
Product Design Lead
Timeline
2024 - 2025, 14 months
2018 - 9 months
Outcome
$780K ARR
AI Cloud Software
QueryPal / O > 1 AI Startup
QueryPal / O > 1 AI Startup
My role
I WAS THE FOUNDING AND ONLY DESIGNER
I owned every design surface the company shipped: product UX, bot configuration UX, admin and analytics dashboard, brand identity, the GTM marketing site, and the prompt and eval system governing agent behavior.
I architected the admin object model from the ground up, designed the three surface component system shared across Slack, email, and the admin dashboard, and worked with the LLM engineers on the tone constraints governing every customer facing draft. No PM, no researcher, no second designer.
How we got here (Story arc)
TWO YEARS OF BUILDING THE WRONG PRODUCT TAUGHT US THE MARKET
QueryPal started as an observability platform designed to help SRE teams make sense of metrics, logs, and traces. But as we introduced AI, we saw a fundamental shift: people no longer needed another dashboard to interpret their data. They wanted to ask a question and get an answer.
That insight changed our direction.
Support teams had the same problem: hours spent answering repetitive questions that could be resolved from information already sitting across the organization. Unlike SRE teams, they had a clear business case for automation and a budget line to pay for it.
The product evolved from observability → AI-powered answers → customer support automation.
ACT 1
DevOps Observability
I designed an RCA tool unifying metrics, logs, and events into one interface. As the models matured, the dashboard started competing with the intelligence users actually wanted. The lesson: visualization wasn’t the product. Fast answers were.
Unified metrics, logs, and events into a single RCA interface.
The lesson: Users didn't want visual dashboards—they wanted fast answers.
ACT 2
QueryPal Slackbot
We shifted from dashboards to native chat. I designed QueryPal’s Slackbot so teams could resolve issues where they already worked, with no tab switching. The catch: high usage didn't mean high willingness to pay.
Brought issue resolution directly into native Slack workflows. The catch: High daily usage didn’t turn into paying customers.
ACT 3
AI Email Drafts for CX
We pivoted into customer support, generating on-brand email drafts from existing company knowledge. Agents reviewed and resolved tickets in minutes instead of hours.
The breakthrough: clear ROI, a clear buyer, and product-market fit.
Turned internal knowledge into instant, on-brand support replies.
The win: Clear buyer, measurable ROI, and true PMF.
ACT 1: Devops observability
VIEW THE BEFORE DASHBOARD
I designed the RCA dashboard with a senior DevOps engineer: metrics, logs and events in one view.
ACT 2
We didn’t pivot to an AI Slackbot by accident. CtrlStack showed us that dashboards explained problems, but decisions happened in Slack.
ACT 2
I led GTM with our VP of Marketing. Ads, Whitepapers, and test marketing sites to check which ICPs signed up the most.
ACT 3
The Slackbot validated the behavior, but testing interest across marketing personas clarified the buyer. CX leaders responded the strongest.
ACT 1: Devops observability
VIEW THE BEFORE DASHBOARD
I designed the RCA dashboard with a senior DevOps engineer: metrics, logs and events in one view.
ACT 2
We didn’t pivot to an AI Slackbot by accident. CtrlStack showed us that dashboards explained problems, but decisions happened in Slack.
ACT 2
I led GTM with our VP of Marketing. Ads, Whitepapers, and test marketing sites to check which ICPs signed up the most.
ACT 3
The Slackbot validated the behavior, but testing interest across marketing personas clarified the buyer. CX leaders responded the strongest.
Object mapping
Mapped the admin model before drawing a screen
Card sorting set the object structure first: accounts, integrations, automations, billing. Screens followed the model.
Admin dashboard - Onboarding
Three steps to a working bot
Connect a source, invite the team, set the tone. Three tasks, one screen, progress visible.
Design System
One component set across Slack, email, and admin
Integration pickers, tone controls, and approval states built as shared primitives. Three surfaces, one vocabulary, no redraws.
Trust and Adoption
CONFIDENCE METERS EARNED AGENT TRUST IN AI DRAFTS
A user study audit showed agents rating AI drafts poorly and rewriting good answers. Pilot accounts were at risk.
BEFORE - Every draft looked the same whether the AI was certain or guessing. Agents reviewed all of them from scratch.
AFTER - Score on every ticket, plain language reasons, and labeled knowledge sources inside the agent's existing help desk.
THE AUDIT
Agents could not verify the AI
My user study audit of pilot teams traced low ratings to missing evidence, not wrong answers. Agents rewrote good drafts because nothing showed why to trust them.
THE BUILD
A confidence layer in the help desk
I drove a score on every draft, the reasons behind it, and sources labeled live, matched, referenced, or verified. Approve, edit, and deny stay with the agent on every ticket.
THE RESULT
Ratings recovered and accounts stayed
Queue level scores let agents approve high confidence drafts fast and focus on the low ones. Approval reached 90% across seven pilot accounts.
AI CONFIDENCE
92%
HIGH CONFIDENCE
AI is confident. Ready to send.
Exact match in license portal
Real time account data, current today
No ambiguity in the question
Clear, documented plan policy
KNOWLEDGE SOURCES
License Portal
live
Enterprise Plan Policy
matched
Toolbox Pricing
referenced
Account CRM Data
verified
HOW THE SCORE MAPS TO ACTION
Three tiers, three agent actions. Attention goes where the AI is least sure.
85% AND UP
HIGH CONFIDENCE
Clear policy, recent data. Cited answer with a verified label.
Accept
60 TO 85%
MEDIUM CONFIDENCE
Some ambiguity or older data. Answer with a caveat and a prompt to review.
Edit
UNDER 60%
LOW CONFIDENCE
Missing data or an edge case. States uncertainty and routes to an expert.
Dismiss
68%
of AI drafts approved by agents with no edits at all
80%
tickets resolved at
first contact
7
pilot accounts running confidence meters in their help desk
Interactive prototype. Toggle Before, Cited, After, and Final.
Analytics Dashboard
MAKING THE AGENT'S VALUE VISIBLE
Support leaders do not renew on ticket counts. They renew on proof the agent is absorbing work their team would otherwise do.
What I got wrong
I DESIGNED TRUST LAST AND IT SHOULD HAVE COME FIRST
Confidence meters were a rescue, not a plan. We shipped AI drafts with no evidence attached, and agents told us so in their ratings.
THE MISTAKE
I treated accuracy as the trust problem
I assumed better answers would earn approval on their own. The first release gave agents a draft and nothing to check it against.
WHAT IT COST
Low ratings and pilots close to leaving
Agents rewrote drafts that were already right and scored the AI poorly. We had to win back accounts that should never have been at risk.
WHAT I DO NOW
Evidence ships with the first AI output
Sources, a confidence signal, and an easy way to reject go into release one. I test for trust in the first study, not after the ratings come in.
Outcomes
QUERYPAL WENT FROM CONCEPT TO PAID PILOTS IN 14 MONTHS
$780K
Grew from 0 to $780K ARR with design embedded across product, brand, and GTM
90%
human approval on AI drafted replies across seven pilot accounts.
60%
reduction in tier 1 support workload, freeing agents for cases that needed a human.
780K
14 months
90%
Approval rate
80%
tckts resolved
Testimonials
WHAT THE PILOT TEAMS SAID
"We completely removed Freddy AI in the first week after implementing QueryPal."
- Customer Support Lead, JetBrains
"Our agents rely on QueryPal daily & we’ve never handled peak season this smoothly"
- Operations Lead, SimplyBenefits
"We completely removed Freddy AI in the first week after implementing QueryPal."
- Customer Support Lead, JetBrains
"Our agents rely on QueryPal daily & we’ve never handled peak season this smoothly"
- Operations Lead, SimplyBenefits