Especially in a field as new and evolving as data science, your strategy is only as good as your product. Fortunately, the industry is not so new that help can’t be found. Lauren Lombardo, Senior Data Scientist at Nielsen, took the stage at DSSNYC to inform data practitioners in the media and entertainment space about the ways in which they can take ownership in a marketplace traditionally run by commercial teams.
Good morning everyone. Thanks so much for being here. I know it can be challenging to take a whole day off of your regular jobs to come and talk with this community about data science, but I really appreciate it.
I'm going to talk a little bit more organizationally-focused today. We all work in this media and entertainment landscape and we all work on data science teams. Sometimes those concepts are really challenging to work together in a way that allows us to build impactful tools for our clients. The media and entertainment landscape is inherently really large and forever changing. And data science’s background is inherently academic. It takes time to build innovative solutions.
So how do we combine those two concepts to ensure that we're delivering the tools that our clients need? Let me first tell you a little bit about Nielsen. For those of you that don't know, Nielsen is a measurement company. We measure what people watch and what people buy. I'm really only going to focus on the watch aspect of today, because we're talking about media and entertainment. But we have this international global landscape as well. We are present in over a hundred markets around the world and we provide cross-platform measurement to our clients into the industry This means that we think about everything from traditional television devices to desktop, to mobile devices - even radio.
When there is a way to consume content in this industry, Nielsen is there to measure it, which speaks to how vast and complicated it can be to work in this space. We also have a whole host of clients that all have different interests and needs from each other. We have the digital-first clients and we also have your traditional TV broadcasters. It can be challenging to build a tool for all of these clients across all of these platforms in all of these markets, because this really is a vast industry that we all work in.
Lauren Lombardo, Data Scientist at Nielsen, speaks at Data Science Salon New York 2019.
Now I’ll tell you a little bit about me: I'm a senior data scientist at Nielsen. I lead our digital content products and our digital television ratings products. My time is spent primarily in digital but I have had other roles at Nielsen including in our traditional TV measurement business. I've worked internationally at Nielsen and in the US as well. I had the opportunity to be on a commercial team where I sold directly to our clients. I manage our client relationships and I helped answer their questions about their products. It's important to understand where I'm coming from when I talk about how we should organize as data science teams in this space. I'm thinking on both sides of the aisle - developing those products and getting them out the door to our clients.
I've talked about how large this landscape is and how different the clients are. One of the hardest things when it comes down to developing data science products in the media and entertainment landscape is how fast that landscape is changing. I mentioned earlier that we're in a really quickly-paced moving environment. Particularly at Nielsen, most of our data science teams are academics, former PhDs used to spending a few years developing and working on really innovative solutions. We want them to be able to keep doing that, but if we're working in a landscape where things are changing so much quarter-over-quarter, we need to rethink the way we're working so that we allow clients that freedom and flexibility to take their time to work through those innovative solutions they're trying to offer the marketplace.
For any of you that work on data science teams, that are working on cutting-edge stuff, it takes a year or two to really bring some of these solutions to the forefront of the market. Unfortunately, in my opinion, a lot of the organizational methodology around how we solve this problem of developing solutions that aren't relevant anymore is to try to cut this one year down. I think that's wrong. I think that we need to give our data scientists the space and the time to develop the tools that they need, but in doing so we need to change the way we work so that they can be looped back into the marketplace. We're developing these products consciously. I think that this time-frame can still stand and if we're going to really empower our data science teams to build the most creative models they can for this industry. We need to ensure that we're keeping our teams lived-in.
See talks like this in person at our next Data Science Salon: Applying AI & Machine Learning To Media & Entertainment, in Los Angeles.
So how do we do that? Today, I'm going to talk about conscious development of products and conscious production of products. How do we marry the concept of a traditionally academic field’s being applied to a really quickly-paced industry and business landscape? How do we develop products consciously so that when we're taking our year one or two of development time, we're ensuring that we end up with a product that is exactly what our clients want. At that moment, this is tricky because what they want at that moment is normally not what they wanted when they came to us at the start of the development cycle.
The most fundamental component here is to start with the right questions: Instead of asking for what the client wants at Nielsen, we've started to ask why the client wants it. Traditional TV broadcasters come to us wanting linear content to be added to their television ratings so they came and they asked us how to do that. It turns out what they wanted truly was just to be able to understand their TV content on a digital space. But why they wanted it was because the industry was changing. There are more platforms available to content producers to put content out into the world and for us to consume it. They wanted to make sure that they were getting a whole holistic picture and view of what their new landscape looked like.
In this case, if we had just focused on what the client was specifically asking for, we would have met the what but by the time we delivered it to the marketplace, there would have been a whole different problem that they wanted to solve, which we could have solved in the same initial time frame by just focusing on the why.
What I've seen on data science teams is when we get these projects, we'll focus on what datasets we need to solve problems and what types of models are going to be most helpful in finding a solution. Why do the clients want these solutions? Why do they need to see the data in a different way? This leads into understanding the business case, which is a little different than just asking the right questions. As a data scientist, you need to understand how that market is going to perform with the tool that you're building.
Another thing that we focus on at Nielsen is empowering data science teams to make the decisions that they think are best for the product. Previously, at Nielsen, if there were two options and we didn't know which one to go with for a specific methodology, we’d escalate all the way up to whomever we thought should make the decision and then escalate that all the way back down. That is a really slow way of working, so if you empower your data science teams to understand the business case, they can make those decisions at a local level. If they understand what's more important to the industry. Whether it is speed or whether it is accuracy, they can understand where they should be spending their development.
Now that we're allowing data scientists to make more local decisions, the team can either focus on making the model three percent more accurate or three times faster. It always depends on what the business case it is. Does it matter that clients can access the data quicker? If the data is being used to make crucial decisions it matters that is accurate as it can possibly be. These are really important questions to understand and data scientists need to take ownership over their business cases so that we're not developing models that one, aren't useful but are potentially used incorrectly. The last pillar of this conscious development cycle is to stay in the loop. I think ties everything together. You can't continue to ask the right questions and understand the business case if you don't have someone on your team that's continuously looped in to what's happening. Even if you do this in the beginning, we've already talked about how long the data science development cycle can be. Things change in this industry and they change every year, so if you don't have someone on your team that is truly looped in, it's going to be really hard to make these decisions locally.
I'm going to define some terms here, not because I don't think we all haven't worked with internal and external clients. It’s just extremely different at every company. When I'd say “internal client”, I mean anyone that is helping you get your product to the person who is using it at Nielsen. That is often another data science team - plus a tech team, application development team, engineering team and an operations team. That’s normally our path to production to get things out the door that can obviously be very different depending on your size of the company and what projects or product you're rolling out. When I say “external clients”, really I'm focusing on whomever is using your product at the end of the day. At Nielsen, this normally means our industry clients, specifically our Google and NBC people that are buying our data. But this could refer to the finance team, the HR team - whoever your end-user is. If you have taken the time to develop this product consciously, you must ensure that it goes into production the exact way that you built it.
Supporting your internal clients means that you have to do those extremely painful code reviews to sit down and take the time to go through validations. I know that sounds simple for a lot of us but even at Nielsen, we've worked through challenges of other teams’ internally not understanding what we were trying to build. It’s not teams’ jobs to understand if it's wrong because they don't know what the output of that model is supposed to be. Data scientists need to be there throughout those steps because we're the ones that built the tool and we know what the output is supposed to be. We can understand if it's wrong.
Supporting external clients, for whatever reason, is more challenging right now. It takes a lot of work and it's often really easy to say that somebody else's job, particularly the commercial teams or the client service teams, is to support those external clients. But really what we're talking about here, to me, is a moral and an ethical issue. That might sound a little extreme, but if we think about the stuff that Nielsen is putting out, if our clients are using those tools incorrectly, we're looking at content that isn't being created because the people who are using Nielsen data to decide directionally where content is going in the industry don't think there's an audience for it. Now we're talking about underserved communities that don't have content and platforms that represent them. Or, we’re talking about millions of dollars in lost ad revenue from a financial perspective for our clients. It is crucially important that these models be used correctly. It's not just something that we should feel comfortable throwing over the wall to another team and expecting that they're going to get that out the door in a way that we expect and implement it the exact way that we developed it.
This all leads into taking accountability for how your model is interpreted. There are real costs that come if your model is being used wrong and there needs to be an end-to-end ownership on every team, especially as data science is starting to infiltrate more areas of every industry. We have a responsibility to help make sure that that end use case is working as well as our development cycle is.
One of the main ways to do this conscious development and conscious production timeline is to make sure that you have the team to support it. Even I started in data science not very long ago and even then, data science teams looked a lot different. Everyone had some sort of statistics background or computer science background. These people had been taught the same kind of skill set. Don’t get me wrong: Everyone could benefit from statistical work and programming work and as the data science industry started to grow, we started to ask our data scientists to get better at all of the new tools and all of the new types of data science that are out in the marketplace instead of hiring specialists for each of those roles.
The data science industry has gotten to a point where we're not finding unicorns that can do everything that encompasses what it means to be a data scientist. Now I wish I could hire a bunch of those unicorns on my team and I think that would make it really easy for us to develop and put products into production in this way. But we also need to think about what the non-negotiables are for ourselves as data science teams and also what the non-negotiables are for our clients. At Nielsen, non-negotiable for our client is probably some type of readability in the UI - some type of unique metric. This requires data scientists that can support that in production. At Nielsen, we need to make sure our pipelines are built properly. We need data scientists that are going to be able to build successful models, product managers that are going to get those models out the door and analysts that can both tell the story and visualize it. If you can’t make sure that you're mapping everything back to every individual skill set on your team, you're not going to be able to support that development in a production cycle.
I've talked a little bit about what happens if we don't focus our data science teams in this way in the media and entertainment space. If you haven't supported your internal teams, your product is going to spend a lot of time in what I have begun calling a dead zone, which is any time between the finished product being developed and it getting into production in the clients hands. We've tried to work through this at Nielsen by taking on fewer projects and trying to work faster through those projects. We need to make sure that our products are used correctly because we have a moral and ethical obligation to ensure that we’re not putting something out into the market that can lead to misaligned expectations. You don’t want your clients to think you don’t have a good product, when in reality, they may just not know how to use it.
To sum up, as you look at diversifying your team, someone on your team should be able to focus on each of the categories that may be important. Our responsibility as data scientists is to draw conclusions and to ensure that the right metrics are being put out into the marketplace.
Events like Data Science Salon are really great because I feel like I can talk more specifically about what I do on a day to day basis. Also, if you work in the media and entertainment history, getting engaged in the media and entertainment community should make you feel like you have an ownership in the industry as much as your commercial teams, product teams and finance teams do. At the end of the day, you own as much of your product as they do.
Curious for more?
Don’t miss the next Data Science Salon in Los Angeles, November 7, 2019.