How This Director Program Manager Turned Resistance into Collaboration by Challenging a CEO's Vision

Oh, you've muted.

So tell me about a time when I was actively resistant. What was the issue?

Is it OK if I actually do superior instead? Sure, cool. Um, so when I was a Twitch. Um, we had a, a very, very strong driven person we'd hired from C Curse, uh, through an Aqua hire, um, who wanted us to ship a leveling system. Uh, I don't know how familiar you are with those, but basically it's progression over time. Um, if we need to get into the detail, we can. Uh, he wanted to ship a leveling system for a number of reasons, basically amounting to increased attach. Um, I said to him, uh, that's a fraught choice, um. We shouldn't be doing leveling. I understand the goals you're going for, but we should be doing achievements instead. The issue at hand is he had stuck, uh, leveling and he'd actually staked some of his reputation on leveling as a really good idea because he'd said, we have a broadcaster attrition problem. Um, we have 70% loss, not 70% retention, 70% loss in broadcasters after first month, um, which is frankly insane, um. And he'd said, I'm the guy. I'm going to fix it. I'm going to make this promise to the CEO. Part of my portfolio of fixing that is leveling system. And I said to him from experience I had at Microsoft among other places for your parenthetical, this is a poor idea. You can get the same effort out of this with an achievement system. I'm happy to build one for you, but I need to diffuse intrinsically your idea. He was reticent, so the core of the issue is he was reticent because he promised it to the CEO, frankly, as his great hope of this is going to fix your core problem and because he'd Had a lot of experience with WOW and other MMO style games in which leveling is the thing, like you're the hill climb man, um. And so I kind of had to diffuse a lot of entrenched biases. Um, my reasoning, we can get into, but that's the kind of the frame of the problem.

Um, well, so first I did my research and I did a lot of listening. Uh, I came to him and said, OK, look, I know you think leveling is great. Talk to me about leveling. What are you trying to do here? And that's where I understood some of the things I've already conveyed, but also Um, he really had belief in the power of economy to motivate, and he had belief in the power of video games to really drive people to, you know, gamification an overused word now, but video games and basically shiny objects, uh, sea ship its to drive people to do things in a virtual sense, right? So, Uh, my first stage was listening. Then I did a whole bunch of research to say and to understand things like, oh, you've promised this to the CEO already, OK. Then I did a whole bunch of research to understand, hey, am I full of shit, frankly, um, do leveling systems work? Will they even do what you want? Do achievement systems work? I need some things from Microsoft already because Xbox. Do achievement systems work? Why do, why is a leveling system bad? Why do I think that's bad? Can I flesh out that thesis? Um. I can get into the detail there if you're curious, but it amounts to a leveling system is kind of like balancing a whole economy, and achievement is like balancing some part of an economy. So if you screw up part of anything that contributes to a level, the whole system breaks, meaning, well, uh, if it's possible to make money by going to a job and it's possible to make money by walking out your front yard and picking up $100 bills, you don't go to a job.

So a leveling system functions like an economy.

So if it's possible to make money by going to a job and it's possible to make money by walking out your front yard and just picking up random $100 bills all day, you don't go to a job. Effort with a Leveling system, there's one global path. Meaning there's one progress bar functionally and people will take the cheapest way to fill it and do that. With an achievement system, there are multiple small progress bars, and so if you screw one up and there's a hyper efficient way to fill it, meaning you wanted them to do behavior A, right? You wanted them to go to a job, but you screwed up the achievement. So actually behavior B is more efficient, picking up $100 bills in your front yard, you've only broken that achievement. You haven't broken the whole system. Does that make sense? So this is the core of the case. Economy problem. Also, building good leveling systems requires 4 or 5 literal economists. We have none. Also, look at Blizzard. Blizzard is the creators of WOW. There's a reason I chose this example. They just had to reset their global leveling system for the 3rd damn time in Diablo 3, so we're not going to get it right. They're better at it than us, and they're not getting it right. Given that Why don't we try an achievement system, and by the way, if we're wrong, we can go the other way cheaply. So Amazon and the mantra of one-way doors, you can build a leveling system out of an achievement system very easily by just putting points on achievements. You can't build an achievement system out of a leveling system very easily because you've already got that global macroeconomic basis. Um, Like I said, leveling behaves like. That took about 3 hours across 4 or 5 different conversations and citations and discussions over Slack and one really long litigation in which we sat down as a product team and I said, look, we should have the case out, right? This is, we should, we should establish that we're litigating things. You and I, Hubert in this particular case, are going to sit down with 2 or 3 other people. We're going to go into a room and we're going to About leveling until we either go leveling our achievements and we're going to leave the room with one plan. Um, we spent about 3.5 hours on it. Um, we came out with what we came out with achievements, and he said, OK, you've got to go convince Emmett because I sold him leveling, but I'll let you try. Emmett's the CEO. Um, and so 2 weeks later we had our product review and we came in and we said, Blah blah blah blah blah. Also achievements. And Emmett said, But why not leveling? I liked leveling. You were doing leveling a moment ago, and this is all pre-production, right? So we don't have sunk cost fallacy at least. Um, and I walked him through the case as I did with you. I said, look, We're not going to get it right. We're going to have to reset it. Achievements does everything you want it to do, and it was a little bit more bullet point fast. Achievements does everything you want it to do. It's worked at Microsoft. It costs less to implement. And by the way, if we're wrong, we can go the other way quickly. And he looked at Hubert because Hubert was the VP in this case, and he said, Do you agree with this? And Hubert said, Yeah, he's done his homework. I can't, I can't out argue him. He's right. And Emmett said, well, all right, let's try it, and if we're wrong, we'll just have to do something else. Um, and the, the punch line here is we shift achievements, it moved broadcaster attention by about 15%. Um, and Emmett actually got up to his credit at all hands and said, I was wrong about this. We got all this wonderful money and we spent about 2/3 of the cost and we should spend a lot more money on this. Let's go.

That's, that's, it's not like 70 points, right. So the total problem was you come in as a broadcaster, in your first month, you're only basically 1 in 3, let's round to 66. You're only 1 in 3 likely to still be around ever again. You're only 1 in 3 likely. Um, we moved that functionally to, uh, 56 or 57, um, I forget which, I can look it up if you're curious. Um, and that's to, to scale it against similar efforts, we shipped 3 products that targeted at this problem at the same time. The next nearest product moved at about 3 to 4 points. Um, Uh, so we had a total impact of all products shipped of about dropping it to like 49, 40, 48, um, but, uh, achievements beat the next nearest improvement by about double, now, or sorry, about triple. Um, now it didn't solve the problem. Um, You know, 50% attrition is still pretty bad. Um, that means it's a coin flip a month later, and a month is not all that long, um. So there's still a lot more work to do, but keep in mind this was one of achievements. We were focused on shipping the system, not shipping every achievement you might want to. And so really our 1st 20 achievements moved at about 15 to 17%. Um, and then it was, well, let's go ship some more, let's go move it another 5 to 10% by basically doubling down on this, um, and that persisted beyond novelty effects. So I'm not talking, uh, you know, first two weeks, I'm talking a month out, 2 months out, 3 months out, cohort analysis. OK.