“Whether you have a digital product or a physical product, product management is about owning things from end to end and helping with the ideation process, defining MVPs and working with the data scientists and engineers to get something into the market.” - Zachary Hansen
Zachary Hansen
Hello Everyone. My name is Zach Hansen, similar to the famous 90s pop band “The Hansen Brothers.” I will not sing now, but maybe after happy hour, you can ping me up.
I am a product manager for Capital One, specifically within our card machine learning platform. I am now super corporate. I've kind of permeated from IBM startup-land, back to startup-land back to enterprises and now I'm working in a bank for the first time.
Today we're going to talk a little bit about product management and then we're going to talk about what it means to be a machine-learning product manager and what we see as differentiators there. After that, we're going to talk about how at Capital One, we are really focused on defining strong use cases for machine learning and some of the lengths that we've gone to build a system that helps us out in that regard.
So what does product management mean? I know there's a few product managers here and people who have worked with a product manager. A lot of product managers like to think of themselves as mini CEOs. I think this is a little more appropriate where it's really just a baby in a suit as opposed to a true CEO. However, product managers carry a lot of weight in the machine learning space. Whether you have a digital product or a physical product, product management is about owning things from end to end and helping with the ideation process, defining MVPs and working with the data scientists and engineers to get something into the market. After that, it involves tracking things in the market and then iterating from there, even if that means failing miserably. A product manager must consider how a customer might explain what they want, how an analyst might design it and how a programmer would write it. Everything's different, but ultimately the tire swing at the end is what the customer actually wanted in the first place. We actually do empathy interviews with our customers to find out what they actually want because if we're guessing as a bunch of different stakeholders what they want without actually interviewing them, we're doing ourselves a disservice, whether we are talking about physical products or digital products.
In reality, product management is really a bunch of cat herding. If they had a degree for cat herding, we would be recruiting heavily from there for product managers. So with that said, if we think about a product manager being someone who owns a product from ideation to in-market through iteration, what differentiates a product manager who deals with machine learning from the rest?
I love data scientists. And I think the data scientists I work with like me, but there are a few things we still have to herd cats with, even though we’re dealing with machine learning. Most of the time, our problems stem from bad data. As a product manager who's building something with machine learning, we deal with data problems. As I found out recently, coming to the banking world, our data sucks and it's very hard to get clean data sources from legacy systems and new systems that are coming from everywhere. Needless to say, it's a very frustrating endeavor. So that's one thing that differentiates what a machine learning product manager has to deal with as opposed to product managers in general. Our data scientists are great lovable people and they're super smart, but that said, they like to go off on their own and build a lot of really cool things. Sometimes these things don’t necessarily fall in line with what our customers might want or what we're trying to go after, so we have to work very closely together. Our data scientists are great at that, but sometimes it can be a little bit of a difficult partnership.
Nobody puts use-case in a corner. What I mean by that is that defining use cases that really fit machine learning is actually a lot more difficult than it might look. One of our biggest problems is that we have all sorts of individuals from different lines of business who come to us and say, “hey let's sprinkle a little bit of machine learning magic on this.” There's mismanaged expectations and part of that is working with our partners and other lines of business who do want to you leverage machine learning to understand if it really works for them. Often times, they can just use basic non-ML modeling to get something that’s just as good or better for their use case. Ultimately, how are we doing this at Capital One right now? Well Jenny is going to talk about that in terms of how we’re actually building out a canvas for understanding and defining good use cases.
Jenny Zhang
“Your clients will come to you with a solution and they're not going to tell you about the problem, so you should feel like it's in your right to ask them what the problem is. Otherwise, how will you solve it?” - Jenny Zhang
Zach and I work in a huge enterprise with 50,000 employees and our team operates as an internal consultancy. We have internal clients, maybe ten teams, who come to us and ask for a magical solution and it takes a lot of collaboration for us to get anywhere.
Data scientists product managers and business analysts are all working together as one to create the Capital One machine learning business model canvas. This means we are delivering results faster together. We have a lot of internal customers who daydream and they think it would be so cool if they could just get us to build them whatever they want. So perhaps, all your companies are benefiting from these asks because on the market, there's a huge expectation of what's possible. However, a lot of times it's not possible. You talk to your data scientist about the pitch that you're bringing in or the use case and they invariably roll their eyes. That’s in fact what we're recruiting for - in addition to the PhD, you have to have a good eye roll.
See talks like this in person at our next Data Science Salon: Applying AI and Machine Learning to Finance, Healthcare and Hospitality, in Miami, Florida.
We have at least four job functions that work on each project. We have the column business analysts as well as product data scientists and technology specialists. We're all responsible for a different thing. Data scientists are responsible for whether it's plausible and product team is responsible for defining the success criteria. Whether they are data scientists or not, business analysts are responsible for the business value and the other teams can be relieved of that responsibility. Inherently, we all speak a very different language and because business talks dollars, product talks user experience and data scientists talk algorithms with zeroes and ones, I think 99% of the time there is a miscommunication and that's okay.
We wanted to create a translation tool that we now call the business model canvas. The end goal of collaboration is always to speed up the process of going from daydream to launch. We have four parts of a business model canvas: the top parts are about the business and the bottom parts are about the data. You all are pretty experienced with the bottom parts, so I'm going to talk about the top parts, hoping that afterwards, you can feel comfortable asking your partners and teammates to help you answer these critical questions needed to define the business challenge. Your clients will come to you with a solution and they're not going to tell you about the problem, so you should feel like it's in your right to ask them what the problem is. Otherwise, how will you solve it?
There are many potential solutions to that problem and we need to know why today’s process is unsustainable. Usually, your client has a lot of painful workarounds. Maybe they have ten windows open on the screen at the same time that they have to juggle through. Maybe there's a lot of paper that needs to be transformed into digitized data. You need to ask for a demonstration of the processes that they currently go through. That way, you can quantify the size of the opportunity and the impact in terms of how many millions of dollars your solutions going to make. If it's a million dollars and the cost of the solution is ten million dollars, maybe you should not keep going.
At Capital One, where there are 50,000 stakeholders, it's really good to know who's going to be impacted and who has to sign the contract. This is key to accurately assessing the problem suitability for machine learning. First off, is your problem truly well-defined and complex? We have had a lot of interest surrounding the magic of unsupervised learning. We’ve had teams come to us and ask us to group their observations into insights. Classification and categorization are things that unsupervised models do not traditionally do, but what is an insight anyway? There are many accurate and valid definitions, so there isn't really a concrete answer and insight is different for every use case. It’s also different in every context and different for every person on a team. We are not able to create the classification outputs that these people want because no one agreed on whether or not the output counts as insight or not. The problems that you have should be really well-formed so that the challenge would be really well defined.
If you can do it already by hand, it may not be a complex problem that you need machine learning for. We need something that is not just a simple rule-based algorithm for which the target goal is clear. So in the end, as we always like to reiterate like Zach said, it is like herding cats because you'll have lots of different teams telling you different answers to all these questions. You want to feel empowered to get to your final answers and that involves quantifying the value of your solution to the enterprise. The secret in this case is making a map of all the stakeholders who you know will benefit. So if you change something at the bottom for the fraud team, it's going to have a domino effect on all these other teams. If you're working for the fraud team, you want to make sure that you understand that the solution will be brought in by the other teams. If it is a good solution, they'll appreciate the work that you've done.
Mapping stakeholders is a huge part of what we do in a very matrix organization like Capital One. We optimize our value by mapping stakeholder dependencies and through transparency, achieving their ideal rewards. Finding workarounds are what make us collaboration hounds. Without collaboration, we are no better than machines that could replicate human decision-making faster than all of us.
For more content like this, don’t miss the next Data Science Salon in Miami, on September 10-11, 2019.