In this episode of McKinsey on Building Products, a podcast dedicated to exploring software product management and engineering, McKinsey partner Rikki Singh speaks with Inbal Shani, chief product officer (CPO) at cloud communications company Twilio. Topics of discussion included leading with cross-functional empathy, anchoring product development in customer value, and executing effective change management during the transition into a new era of AI. This interview took place in August 2024. An abridged version of their conversation follows.
Leading with cross-functional empathy
Rikki Singh: Inbal, tell us about your background and how it has shaped your product philosophy.
Inbal Shani: I have been with Twilio for four months, and I am excited about the journey ahead. I have many years of experience in both machine learning and AI. I started my career as an aerospace engineer working on navigation and control systems. I did my master’s in mechanical engineering with a focus on control systems, and I was using what was at the time called “genetic algorithms.” Really, it was the beginning of machine learning. Ever since then, I have spent my time across the stack, from building robots to working in the cloud.
The biggest thing that has shaped my thinking is my experience as a systems engineer early in my career. It helped me develop end-to-end thinking—how to solve a big problem by dissecting it into smaller, accessible problems.
My time at Amazon and AWS shaped my product thinking because customer obsession became part of my DNA and my continued focus. Last, learning how to think big picture helped me to focus not only on the details but also on how everything comes together.
Rikki Singh: You have had multiple CPO experiences. How do you define the role of a chief product officer, and what makes CPOs successful?
Inbal Shani: Across the industry, there is no single definition for a CPO—they can play different roles. For example, some are more heavily focused on the product, while others are focused on managing R&D. In my time at GitHub, I also managed product marketing and strategy. It really depends on the company, but CPO is a versatile role.
Renee Niemi, resident CPO of Products That Count, said, “A CPO is the only seat in the C-suite, next to the CEO, that touches every single function of the organization.” That is really the role of a CPO, and that is the biggest difference between a CPO and a senior vice president [SVP] of product. The SVP of product will define the focus of the product, the product strategy, what the team needs to build, and the product vision. A CPO needs to think about the entire product development life cycle, from ideation to when a product is in the hands of customers, and make sure that it is working well. CPOs also work closely with marketing and sales teams to make sure the product is landing the way it needs to.
Through that lens, one success measurement for a CPO is collaboration. How well are you able to establish collaboration with your peers, and how well do you help the company connect the dots?
Most important, the success of a CPO is based on the success of their products. So what does success look like? Do customers love the product? Do they adopt the product? When the product hits the market, does it have the right quality? How does that product translate to a go-to-market strategy? You can build the most successful product in the market, but if you are unable to position it right, encourage customers to use it, or enable the sales team to sell it, then you have failed to do your job as a CPO.
Rikki Singh: What is the most effective training ground for this role?
Inbal Shani: Through my different roles, I spent a lot of time building an overarching thinking to understand all the functions, all the stakeholders, and all the customers that I need to interact with. It is important to spend time with customers, the sales team, the marketing team, analysts, and investors to figure out what the market needs. Understanding the entire end-to-end experience is what gave me the best tools to do my role.
Building AI-led products
Rikki Singh: How is your role changing as AI becomes more core to products?
Inbal Shani: AI has created a need for a specific skill set. For me, AI has forced me to use holistic thinking. It is important to understand all the aspects of a product, and a lot of that is the user experience. Today, AI is at the forefront of productivity, automation, and consumer delight and engagement. Historically, especially in the SaaS [software-as-a-service] world, product managers would think more about the back end and the data but less about the user experience. AI flips a lot of that thinking. You need to know when to use AI and for what because it is expensive. You need to ask if it is really giving the customers the ROI they expect from the tool.
AI also forces product managers to understand data. If your data is not where it needs to be, then AI is not an effective tool. Privacy and security become more important, too. AI has increased fraud, spam, phishing, and voice theft. For product managers, it is important to think about preventing vulnerabilities in the code.
Rikki Singh: What AI-centric product have you built that required you to think about these aspects?
Inbal Shani: Six or seven years ago, when I was working for Microsoft, we were building a conversational agent for digital customer support, focusing on the consumer world. At that time, customers did not want to chat with a chatbot. They wanted to talk to a human. But the volume of tickets that needed support made it impossible for employees to give customers the best service, so we needed to create a conversational agent.
The technology was quite new, and it was not easy to integrate—it did not feel like a human. We spent a lot of time trying to understand the customer because this service was also a change for them. We wanted to have most traffic come through the chatbot because it could solve problems, troubleshoot, suggest actions, and provide documentation. We had to figure out how to take customers through that change management phase so they could move past their initial inclination to pick up the phone or send an email.
That AI-centric project was centered on customer engagement. We asked, how can we get customers more involved and more excited to have a conversation via a chatbot versus a human being? How close can we get that chatbot to sounding and looking like a human without being one? How can we take all the information that came from a chat and summarize it if an issue needs to be escalated to an agent?
That project required us to focus on the customer and work backwards to understand the change management that the technology required.
Rikki Singh: How did you involve other functions? Did you have to engage the product marketing team and others to learn what they have done in the past or how they have seen behavior shift?
Inbal Shani: We engaged product marketing and the team that was building that foundational layer of AI, collaborating with them to improve the models based on customer feedback. We also worked closely with the support agents because they were the ones at the end of the line catching all the conversations the chatbot could not solve. Through them, we could figure out how we were improving customer support engagements and making the chatbot more visible.
Rikki Singh: You mentioned that one of the key aspects of being successful as a CPO and building products is figuring out the metrics for success. Do success metrics of AI-centric products look different? Have they evolved?
Inbal Shani: In the past few years, there has been a shift to measuring outcomes more than traffic. Measuring traffic is not enough in the world of AI, and yet outcomes are difficult to define. I think measuring outcomes is complementary to metrics we have always used, such as adoption, engagement, churn, and time to value. But we are shifting from measuring speed to focusing on time to value, and that is a much better representation of return on investment.
We used to think that reducing time to production was enough, but the bar for customer expectations has become much higher. Now companies think more about the value customers are getting and the effort required to get that value. These additional measures help determine the success of a product. At the same time, you need to measure the things that already have a general baseline. We are still baselining these outcome-based measurements as we go to get the sense of what “good” looks like.
Rikki Singh: How have the roles and responsibilities across product and engineering teams shifted with AI-centric products? Are there new “must have” skills that were not considered previously?
Inbal Shani: More than ever, thinking critically is the number one priority. You need to understand AI and the problems you are trying to solve because the technology is very complex, despite being accessible. It comes with a lot of limitations and guardrails. For example, you need to create privacy and trust and understand the cost. While you can experiment more freely, taking AI-centric products to production requires you to be grounded in critical problem-solving as well.
You also need to understand the data you’re using. It’s not just about creating out-of-the-box experiences: it’s about connecting automation and a system of data to get desired outcomes.
Product managers are adopting “cost thinking” more and more, too. When the migration to the cloud started, many companies went full in—they were willing to spend money to modernize their stacks. Then there came a time when everyone started to reconsider how much they were spending and realized it was more expensive than they had imagined.
Navigating the hype around AI
Rikki Singh: What key challenges do companies face when they build more AI-centric products? How should they adapt to these challenges?
Inbal Shani: Change management is difficult—and it’s even more complex right now because it has to happen across the entire company, not just within product and engineering teams. Companies must be intentional when they decide to pursue an AI transformation for their products and experiences, because there is a learning curve. You need to build the right skills in your team, and it’s possible that these skills did not exist before because you’ve never incorporated AI or similar tools in your stack.
Companies also need to understand how to think about data, privacy, and security. How are you adopting even more of a security mindset on your R&D team? And how are you translating that to marketing positioning? As we said, outcomes-based measurements are not that deterministic. So how do you translate these metrics into a go-to-market strategy and into marketing campaigns to show the value you are offering? And how do you train your sellers to talk about the things customers care about? Historically, sales teams have not focused on outcomes. They are focused on adoption, usage, and integration rather than on selling an outcomes-based experience to a customer.
There also is a lot of hype around AI, but AI is not a silver bullet. It is a tool, and we need to use it responsibly to solve the right problems for customers. Companies need to be grounded in the products they are building and solutions they are providing.
A lot of companies are jumping into AI without understanding the tech. The barrier to entry has gotten lower, making AI more accessible for everyone, but companies should not incorporate a technology stack without truly understanding it. You are responsible for these experiences, so as an R&D organization, you need to spend enough time understanding the risk and the complexity of everything being incorporated into the solution.
Rikki Singh: What tools do you use to differentiate a hype cycle from a real opportunity in the AI space?
Inbal Shani: In general, when people talk about AI, they are talking about using large language models. Before, I used AI systems to fine-tune common filters and control systems. Then we moved to the world of predictive AI. Now we are in the world of generative AI to complete code, write essays, and generate content. AI is a wide spectrum of technology.
I usually try to compare new AI systems to what we had before, determine what the evolution was, and understand the purpose these new tools serve. It is not going to replace everything we have done before; it just provides us with updated capabilities and additional advantages.
Rikki Singh: What advice do you have for product leaders who want to dabble more in AI?
Inbal Shani: Take a step back. There is a lot of pressure, hype, and expectations around AI today. Really understand the customer and anticipate their future needs in this fast-moving transformation. Think about AI as a means to an end, not the end. Remain grounded in the problems you are solving, think holistically about the components you are using, and understand how to take your people through this journey.
Change management is also key. This is a big change for your organization and your customers. And if you are ever in doubt, always go back to the customer.
