Building Customer Empathy in Engineering Teams

How to build better products, one customer at a time.

In December, I wrote about the session by Frank Slootman at FC Build on How to Build a Snowflake, and one of the things that he said really struck a chord. He talked about how it’s imperative that engineers build empathy with customers, are sensitized to their pain, which helps them to understand their use cases very well. He also talked about relentlessly working on the customer NPS - and not just the score, but also the narrative behind the NPS — understanding what is really moving the needle for the customer.

I've been privileged to see all of this in action at Cohesity. Cohesity had a near-impossible NPS score of 100 last year, but more importantly, it stayed in the high 90s for the longest time. Customer Obsession is one of the core values of the company and but what's really admirable is that it’s not just something for the culture pages on the website - it’s actually lived every day all the way from the CEO to each engineer. If you ask around, almost everybody would have a ton of war stories to talk about. All of this is possible through not just product excellence but also some amazing collaboration from support to engineering. What I wanted to do in this post was to touch upon some of the learnings on how to enable engineering teams to stay laser-focused on customer obsession. This is written in the context of a B2B software product - though some learnings may be applicable elsewhere too.

What do customers care about?

The advantage of working in hi-tech is that you deal with sophisticated buyers and end-users. They stay abreast with the latest technology and have a really good eye for where the world is going. In my own interactions with customers, I've been surprised time and again with how they challenge and get the best out of engineers. Sometimes, it means that customers have to explain to us how to walk the fine line between "I just want this to work" and "I want to be on the bleeding edge on this capability".

They can anticipate some of the challenges they might see, and want assurance that the partner they pick is committed to making them successful, not just interested in delivering on a contract. They want to make sure that the partner takes their needs seriously, empathizes with their situation, and finds resolution (or sometimes works with them to explain why something is not possible!).

It's amazing how many times the vendor goes to customers with a negative answer and instead of being angry they are actually understanding when provided with a logically reasoned explanation - in fact, it makes them trust the vendor more. This synergistic relationship becomes even more powerful when the customer themselves stretches the product in new and myriad ways, some that we as engineers may not even have thought of.

All of us want to work with such customers and earn their trust :-) So, how do we get started with this journey?

It all starts with Culture.

A perfect NPS score can not become a reality unless the whole organization is whole-heartedly committed to the customer, and it is a value that has been deliberately and thoughtfully woven throughout the cultural fabric. If a customer goes through a difficult phase, will the whole company rally together to address their needs, or will departments work in territorial fiefdoms apathetic to this customer's challenges? If such situations can be diffused quickly, the customer will not blink twice before making major investments into the vendor's (sometimes unproven) technology.

While this is something that would be easy in a small startup, rallying everybody in an organization of thousands of people is no easy task! In most companies, only long-time insiders and early engineers with an end-to-end understanding of the systems would manage to do something like this, but a sign of success is when relative newcomers are able to get organizational resources aligned on the customer issues. Many companies achieve this through rotation on-call programs but often they are limited to the silo-ed understanding of their own components. Ensuring that new engineers understand not just their own component but how it fits into the overall stack and how the layers interact only comes by seeing real customer deployments in unique real-world environments. Such knowledge needs to be celebrated and rewarded.

Yes, this is disruptive at times - since high severity cases demand a lot of attention from engineers buried in deep work - but at the same time, having a better understanding of real customer issues helps them build better products. When each engineer understands customer's pain they automatically optimize for the most customer-centric solution - and isn't that all of our goals?

Of course, the Product needs to be a Rockstar...

It really helps if you are selling next-generation technology, which is vastly superior to erstwhile solutions. If the core product provides a step-function improvement in customer value, customers are far more willing to work through nitty-gritty details. No product is complete and big customers are ever demanding - even in the largest software companies like Microsoft and AWS, some of the largest enterprise customers (like big banks) can negotiate extended support or longer sunset period for products. What really helps a lot is if the customer sees a strong commitment across all levels on excellence, product quality, and supportability.

The product is not just the core technology - in fact, once the core technology is in place, the real hard work begins - building something that will stand the test of the real world. A lot of thought has to go into making sure the product interacts well with the external environment and is resilient to any changes therein, and a lot of tools have to be built to ensure that should a problem arise, the product can either deal with it (resilience), alert the right people (monitoring), or provide the right information and context for debugging purposes (observability). One can not undermine the level of effort that goes into this engineering - especially when these ideas meet the reality of today's large geographically distributed engineering teams.

...and should Continuously Improve.

One of the ways in which great engineering cultures distinguish themselves is through their focus on continuous improvement. We all see issues, but doing the right retrospective with a positive learning attitude (how could we have done better and how do we avoid this in the future?) is extremely important. Also important is not to stop with surface-level answers but to use the five-whys framework to get to the root of the problem. It's also one of the hardest things to implement as the organization grows. Dashboards and tools are our friends but so are scorecards that help us track improvements and flow the information all over the teams - see this example from Hashicorp. The tools are the easy part, the culture is always harder :)

When the Rubber Meets the Road, you gotta speed things up...

Companies invest a lot in quality so that no issues are actually seen in real customer environments. However, shit happens. How you deal when an issue arises and turn the customer sentiment around is what differentiates an average NPS score from a truly legendary one. The folks in first-line support teams need to do an outstanding job of ensuring that a knowledgeable professional is immediately available to answer a customer query.

However, If it’s a "tricky" situation, it's also super critical that we can get the "right" help to the customer really quickly - which may mean having the right engineer available to look at the issue. In most cases, what frustrates the customer is finding somebody who either talks down to them, doesn't empathize with their situation, or hides their own lack of knowledge in platitudinous recommendations (turn off your computer, unplug, wait ten seconds, and plug it back in).

...while keeping the pathways clean

However, this is easier said than done. We all know how hard is it to navigate the labyrinth of component ownership - and finding help from the right person - quickly. Given the high bar on quality, in most cases, customers run into issues due to a complex set of unanticipated interactions under unique environments (a "Disturbance in the Force" if you will) - and it takes Jedis across multiple areas to come together to resolve them. A lot of investment needs to be made in tooling and collaboration infrastructure to make sure that all of this real-time collaboration is very well oiled, the right people can huddle together at the right time, keeping ownership information clean and current and ensuring there's positive hand-off when passing the baton between different teams. Customer sentiment also needs to be managed - customers want to make sure their issues are being taken seriously - which means engineering leadership may need to stay involved and reassure the customer in case of high severity situations.

The same communication and collaboration pathways also have to disseminate learnings and new understanding, thumb rules, tips, and tricks quickly across both engineering and support teams. This way once an issue has been seen the organization no longer spends all the extra bandwidth to root-cause a similar issue twice. This is harder than it looks and it needs both people, processes, and (search) tools to come together. Early on in my career, I worked on a search tool for “stack traces” at a large tech company - and I can only understand the impact now with one more decade of mistakes and learnings - at the time it had just felt like a cool application of TF-IDF indexing.

The amount of pixie dust required to keep these "collaboration pathways" clean is mind-boggling!

This still doesn't get you to a perfect 100

These are just table stakes in helping customers get value out of the product, but there are three additional things that I believe delivers true customer delight

1. White-Glove Service. There are instances where you need to provide high-touch assistance to customers which is, of course, very expensive. This may be a dedicated account manager or special attention during an escalation. How and when to provide this (and when to disengage) is something that needs deep planning and commitment across so many parts of the organization.

2. Metrics-Driven Approach. Ultimately, winning organizations back their decisions with data, whether it comes to measuring product stability, sentiment, or planning improvements to the product. This also means tracking your MTTRs closely, looking over your defect escape rate with a microscope, and SLA violations of promised resolution times. This is where a lot of investment in tooling comes into play.

3. Using Next-Generation Technology. The best engineering organizations also invest in next-generation technology and AI-based bots to get help to customers automatically and triaging any issues to get to the right team. It not only helps stay ahead of the game but also delights customers faster.

In conclusion, delivering customer delight is something that's not limited to one team, one product attribute, or one single process. A lot of things have to fall in place, and a lot of people contribute to making it happen. If customer obsession is ingrained in the cultural fabric of the company, it makes achieving the goal a lot easier. Maniacal customer obsession is a lofty goal that many aim for, but very few are able to live and practice day-to-day.