Measures of Success: Our Top 10 Chatbot Evaluation Metrics

Written by Amy Wallace  on   Dec 23, 2021

As with any large project or undertaking, metrics are key in order to gauge performance. Are your current processes adequate enough, or is there room for improvement? Metrics are important to track for all aspects of a business, from sales, marketing and financial performance, to overall customer service. When it comes to improving the automation processes of customer service, chatbot evaluation metrics are also critical to track. They’re critical to track not only for the sake of measuring progress, but also for knowing how it’s impacting your customer experience.

Here, we break down the top 10 chatbot evaluation metrics to have on your radar. 

These metrics fall under two categories: measuring your users and measuring the conversions that take place. As chatbots are in place to help your customers, a first-rate user experience is the ultimate aim. So, it makes sense to start by looking at your user base. What is the overall user experience like? How are they interacting with the chatbot? As all significantly impact the user experience, it is important to analyze:

  1. Conversation Length: It is one thing that users interact with the chatbot, but whether they remain engaged is another part of the story. What is the length of interaction between your chatbot and its users? If increased efficiency is one of your goals, this may be a key metric on which to concentrate. This metric varies from case to case, and is dependent upon the situation. For marketing chatbots focused on user engagement, longer conversation time would be viewed as positive. Yet for support chatbots aimed at resolving issues and queries in a timely manner, seeing a decrease in conversation length over time is favorable.  
  2. Goal Completion Rate (GCR): How often are users receiving the adequate information that they need? This metric tracks the percentage of those users who reach the goal that the chatbot was designed to accomplish, whether that may be scheduling an appointment, resolving an issue or clicking on a CTA button1
  3. Bounce Rate: Similar to users navigating away after viewing only one page on a website, this metric indicates how useful customers found the chatbot to be. Who is leaving, at what point in the conversation are they dropping off, and why? If a customer ends a session prematurely upon finding it not of use to them, for instance, this would constitute a high bounce rate.  
  4. New Users: Growth in new users is a strong indication that your chatbot is engaging and performing well. During the COVID-19 pandemic, for instance, airline WestJet saw a 45X spike in new users and customer requests, with a slew of questions regarding cancellations and flying restrictions. Leveraging Netomi’s AI-powered chatbot, Juliet, the company was able to scale its support team and deflect tens of thousands of calls from human agents. 
  5. Return Users: Aside from new users, are there customers enjoying it enough and finding it useful enough to come back?
  6. Customer Satisfaction: How useful or helpful was the chatbot, in the eyes of the customer? Sent immediately following an interaction, customer satisfaction (CSAT) surveys allow you to understand if you’re meeting customer expectations, enabling you to pinpoint where things go wrong in the customer journey and optimize the experience as necessary. Such customer service tools allow customers to feel appreciated and that their voices and concerns are being heard.
  7. Escalation Rate: There are some situations in which a chatbot will need to escalate a conversation to a human agent. This could be for several reasons: it does not understand a user’s input, has not been trained on a specific topic or a user explicitly asks for a human agent. If experiencing a high escalation rate, it might be worth conducting additional training and increasing the chatbot’s scope to deliver a boost in performance. 
  8. Out of Scope Accuracy or True Negative Rate: How can you improve the accuracy of your chatbot? In conversational AI, accuracy relates to how often the chatbot provides the correct response. Out-of-scope accuracy, on the other hand, recognizes when the chatbot understands which topics it is not trained on and follows the appropriate behavior (i.e., escalating to a human agent or directing to another channel) rather than guessing a response.
  9. Message Volume: How many lines of chat go between the user and the chatbot? Examining the volume of messages sent back and forth helps you to determine how many questions your chatbot needs to be asked before it can provide users with the necessary information2. In customer service, you want to reduce effort for your customers, so you can use this metric to identify topics that have high message volume and try to simplify the flow as much as possible. 
  10. Number of Conversations: When a user is successfully serviced by a chatbot without any human intervention taking place, this is considered to be a completed chatbot conversation. Looking at the end result – how many queries were resolved successfully – is key. 

Related content: Hear how support teams benefit from smart automation in this webinar featuring Mike McCarron from Gladly.

Chatbot Evaluation Metrics: Pick a Keystone Metric

While the above metrics are simply suggestions, and tracking all of them is not necessary, it is up to you to choose the one that matters the most, based on your business goals. This list of metrics also provides an overview of what changes you can make to the parts of the chatbot experience that might require some further finessing. Tracking various metrics, and continuing to monitor them over time, will help to paint a comprehensive picture of overall chatbot success. 

Curious to dive into the world of chatbots? Join us for a free chatbot consultation to see Netomi in action. We’ll show you the out-of-the-box tools you can use to measure and optimize your AI chatbot over time, how you can scale your support across each and every channel, and more! 



What Is a Virtual Contact Center? Virtual Customer Service Explained

Written by Can Ozdoruk  on   Dec 2, 2021

A virtual contact center is a digital hub where agents respond to incoming phone calls, emails, social media requests, and other customer service tickets remotely. By definition, virtual contact centers do not need to exist in a centralized, physical location. This is a change from more traditional, brick-and-mortar call centers in which agents work from the same location.  

While traditional, brick-and-mortar call centers used to be prevalent, increasingly companies are turning to virtual contact centers. The tools used by virtual call centers are in the cloud allowing agents to work from home, different offices, even different time zones. Virtual call centers, or VCCs, offer many benefits in both operational efficiency and the customer experience

As a result of the COVID-19 pandemic, many companies that had not already done so have moved to virtual contact centers. While many companies struggled initially to set up new operations that didn’t rely on on-premise technology and strict policies, the pandemic forced changes. And now today, about 80% of call center agents are working from home1. This trend is likely to be permanent, especially as the industry grapples with a labor shortage and workers increasingly consider flexible work environments when taking a job. In one study, 58% of people say they want to be full-time remote employees post-pandemic2

With virtual contact centers the future of customer service, we’re diving deep into everything you need to know about call centers – from the benefits of taking a contact center virtual to how to run a successful virtual call center. 

How Have Call Centers Changed Over the Years?

In 1957, the first call center, Life Circulation Co, was launched by Time Magazine to increase subscriptions. While this was more outbound marketing, it had agents working side-by-side in a centralized location (this would later become a major telemarketing firm). 

In the 1960s, switchboards became common which enabled a receptionist to connect calls to the right person. In the 1970s and 80s, new technology like Private Automatic Branch Exchanges and Interactive Voice Response (IVR) brought call centers mainstream with more and more companies using them, but still mostly for sales and outbound purposes.  It wasn’t until toll-free 1-800 numbers became prominent that the inbound call center agent came to life. 

Virtual customer service contact center over the years

In the 1990s, the internet created a huge shift in customer service. Voice over Internet Protocol (VoIP) and channels like email emerged as support channels, turning traditional “call centers” into “contact centers” as agents were having customer interactions over many channels. These virtual customer contact centers relied heavily on on-premise technology and symptoms3

Today, many contact centers are virtual with a remote and distributed workforce leveraging flexible, cloud-based software solutions to provide omnichannel support to customers. Platforms like Zendesk, Freshworks, Gladly, Salesforce and Khoros enable teams to have the same powerful tools from home offices or distributed offices. With flexible CRM integrations, a cloud contact center solution can improve customer experiences, enable accurate forecasting, and provide better workforce management than ever before.

What Are the Benefits of a Virtual Contact Center?

Virtual contact centers offer even more benefits today than ever before. This is because of a few key reasons. 

First, the workforce is distributed. Hiring a team of agents in one place is not required, and the talent pool becomes that much bigger. This is critical as there are currently about 25% fewer agents than pre-pandemic. And agent attrition is among the highest of any industry (about 45%). Nearly 60% of people say that if they are not able to work remotely, they would “‘absolutely’ look for a new job. With agent turnover already high due to other factors, offering agents flexible work environments is essential.

Having virtual office hours also enables companies to provide 24/7 support. Customer service agents can have shifts during their regular business hours and companies can have coverage across different regions.

Another benefit to virtual contact centers is scalability. The overhead costs associated with desks, equipment and office space with adding agent headcount are eliminated. Lastly, many companies find that remote workers are actually more productive. Research has shown that $600 billion a year is lost to workplace distractions. And interestingly, remote workers are 35-40% more productive than their in-office counterparts2.

Join us for an honest conversation about how support teams have adapted in response to the pandemic.

How Do You Successfully Run a Virtual Call Center?

When setting up a virtual customer service, there are a few key things to keep in mind to ensure a positive agent and customer experience. 

  • Use a best-in-class agent desk platform: Tools like Zendesk and Gladly centralize everything and make agents’ jobs much easier.  These platforms pull in information from other back-end systems, provide a single view of the customer and even make recommended responses to make agents more efficient.
  • Leverage AI and automation: Every virtual contact center should use AI and automation to offload repeatable work from agents and reduce resolution time. AI lets teams scale up and down in seconds, adding a level of elasticity to a remote team to manage both unexpected and anticipated spikes in ticket volume. AI also allows teams to offer support 24/7, not just within office hours, and every day, including weekends and holidays.  Advanced AI chatbots can skip humans entirely by automatically resolving simple questions via live chat (return policies, resetting passwords, upgrading a plane seat, etc.). 
  • Create a team atmosphere and reward individuals and teams: Create comradery amongst your dispersed team by having daily or weekly check-ins or virtual coffee breaks, rewarding great work such as agents with great CSAT scores or celebrating milestones like work anniversaries. A virtual team can still be a connected team.
  • Analyze and optimize: Transparency is essential. Keep a pulse on how your agents are working, trends in tickets resolved and productivity levels, and how this changes over time. Agent desk platforms come with a suite of analytics that can uncover opportunities to improve both customer support and agent performance.
  • Use intelligent routing: Reduce stress with omnichannel routing. Distribute tickets based on things like the sentiment, the length of resolution and complexity of a query. This can help ensure some customer service representatives aren’t getting the hardest tickets or the most irate customers again and again. 

Virtual contact centers are here. The workforce is now virtual. To combat the labor shortage and provide a great customer experience, having at least a semi-virtual contact center will be key. Can we discuss how to use AI to ensure consistency and provide scalability with a virtual contact center? Get in touch today