Bot CSAT vs. Human CSAT: Why They Should Be Measured Separately

As many CX leaders know, CSAT scores are only one small piece of the pie when it comes to evaluating how your customer experience is performing. Still, many CX leaders are required to report these metrics up to their C-suite executives. When you throw new automation efforts into the mix, it gets a lot more complicated.

Historically, an increase in automated customer service (CS) has meant a decrease in CSAT. Some statistics show that customers prefer a human interaction by 75% over automated CS. But with the digital experience emerging as the de facto brand experience, automated customer service is essential for scaling operations, especially when you consider that a large percentage of initial brand interactions can be resolved with automation and self-service—without the need to escalate to a human agent.

This automation-first approach means brands can use CS agents wisely for urgent or high-value interactions, instead of impeding productivity with simple password resets and other routine issues. Automation is also essential to scaling a business digitally without hiring tens or hundreds of new employees or diluting the brand experience via outsourcing.

The problem with measuring CSAT without nuance is that scores for consumers that interact with human vs. virtual agents are taken as one and the same. Using the right CX tools to differentiate bot CSAT and human CSAT enables brands to have a better understanding of CX performance and set/report on more accurate KPIs, which makes it easier to make the case for automation to key C-suite stakeholders.

In this article, we’ll discuss the importance of introducing nuance into CSAT measurement and offer tips on how to measure and optimize CSAT to account for automation. Let’s dive in.

The dangers of combining bot and human CSAT scores

When brands are introducing automation for the first time, CX leaders don’t know how to level set expectations with senior leadership on what impact automation will have on CSAT scores, or even how to measure success for automation (spoiler alert: it’s not CSAT). This confusion can shatter confidence in new systems when CSAT numbers dip after rollout.

The truth is, CSAT is a human measurement. It’s not really a relevant metric for bot conversations as it doesn’t take into consideration the massive scale that automation serves and the objectives of automated conversations. It also ignores the dimensions and attributes of good automated conversations in favor of subjective customer sentiment.

So there’s no way to hold bot interactions to the same standard as human interactions, nor should you! It’s not an apples-to-apples comparison. In a human interaction, a customer may not get the answer they’re looking for, but the agent was kind and accommodating and the customer may feel bad giving a poor rating.

Some customers are inherently unhappy when they encounter automation, even if it resolves their issue. On the flip side, if automation removes the easy wins and agents are left with only the more difficult customer problems, average CSAT could drop and Average Handle Time (AHT) could increase. Lumping bot CSAT scores with human CSAT scores results in reports that are unreliable and lack the nuance necessary for targeted improvement.

The importance of measuring bot and human CSAT separately

Despite the potential resulting dips in CSAT, initiatives for increased automation in customer support are more essential now than ever. Decreasing the reliance on human CS for low-touch brand interactions is key for cutting costs and increasing revenue. Not to mention employee happiness and productivity.

The most important aspect of deploying automation is understanding its true impact on CSAT; and to do that, it’s essential to measure bot and human CSAT scores separately. Doing this gives CX teams a more accurate representation of how different CX efforts are doing.

And separate measurement helps you know where to make adjustments. For example, if human CSAT scores are dipping, you know you’ll need to work on that, and similarly if bot CSAT is dipping, you’ll know to make necessary adjustments to the system to optimize it to help more customers. By identifying aspects of CS such as “Where are handoffs happening?” you can start seeing areas to improve the bot’s abilities.

If you measure CSAT scores individually, you can report more accurate KPIs to senior leadership and give a more nuanced interpretation of how automation efforts are impacting CSAT overall. While many CS automation platforms lack nuance in their CSAT measurement capabilities, leading conversational AI platforms make it easy to separate bot and human CSAT scores for easy analysis and reporting.

Final takeaways

CSAT at its best should measure a very high-altitude, generalized sense of consumer satisfaction with your brand. CSAT was never intended to measure such low-altitude, singular interactions such as a web chat, especially with conversational AI. There are far too many variables that affect it—even something like what mood the customer was in when they left a rating.

Instead, focus on creating an overall better experience for customers with things like automated CS that’s scalable, available 24/7, asynchronous, multilingual, and personalized. And keep in mind that the best CS is a blended CS; a bot and human hybrid approach that provides customers with the best of both worlds.

By understanding when and how to measure CSAT, what it’s good for and what it’s not, CX teams can communicate the nuance of these metrics to high-level stakeholders. Creating differentiation points between success metrics for bot and human interactions is key to understanding what’s working, and what needs to be improved to build the best customer experience overall.