How This Product Manager's Initiative Prevented a Major Security Breach at Expedia

OK. Yeah, so there's, there's, um, one instance that definitely comes to mind that fits that criteria. Um, so this was back in my time at Expedia. And uh right about that time there was a a large increase in the number of um attacks that we faced. Like most big companies were getting attacked. Uh, I'm talking about um account takeover attacks. So what uh nefarious actors would do is they would um blast your systems trying to guess what, um, guess the password guess and get into accounts with passwords stolen from other sites and so. Um, one Friday evening we saw that, uh, the number of, uh, uh, the alarms that we had in our system that detect, uh, uh, failed login attempts, uh, were going off and we found that there was, there was a massive spike in failed authentication attempts and it is, uh, very clearly, you know, an automated attempt to do this. So when this happened, we were still not, we did not faced such an, such an attack. And uh it was somewhat new to us. So, uh, I remember it was a Friday evening when this happened. I think they deliberately timed it like that, um, when we saw this alarm go off. So I had to get up, uh, I had to work with my team to understand what was happening, what kind of an attack it was. And then I had to work with, uh, uh, Expedia's risk mitigation team, uh, the and the security team and the legal team. To work on a solution. Now, there was not a process laid down for this, right? And given uh the, the security team, the, the, the risk mitigation team, it wasn't very clear like who exactly is going to do this. But then I, I took it upon myself to secure our accounts. Right, so, um, I worked, there was no clear, uh, process laid out for something like this. So I worked with my team to identify what we should do. And at the core, at the, uh, at the core of what we do, we essentially had to lock out all accounts which were successfully logged in during the period of that attack, right? So, so these nefarious actors over a period of 4 to 5 hours I've been trying like millions of accounts and they've gotten into some of those accounts. Right? So, um, we had to act really fast to reset the password on these accounts. So, as you can imagine, um, it was in the hundreds of thousands, the, uh, the accounts, the list of accounts which were logged into in that period of time. Um, and no, not hundreds of thousands, but close 100, 100,000. And, um, I had to come up with a process to secure those accounts. I had to come up with alignment with legal and PR and my leadership as to the next steps, which is basically resetting the password on, on these accounts. This was around 2016, if I, if I remember right, and this was not, it was not, um, um, at least I hadn't seen it as industry standard practice where, you know, you go and reset a bunch of passwords. Um, since then I've seen other companies do it. I've seen Uber do it. I've seen, uh, several other companies, Medium, do it. Uh, but at that point in time, the, the path of action wasn't very clear. So, in 2 days, like I had to decide, not even 2 days, like a day, I had to align with these 3 different people. It wasn't very clear who owns it, but then I jumped in and owned it. And then I had to get alignment with PR, legal, my leadership, security, risk, uh, management with these five teams that, hey, there are, there is this list of 100,000 accounts. I'm going to reset the password on all these things, and this is the email that's gonna go to the customer that says that explains the situation to them about how security is important to us, and that's why they need to reset their password. I mean, it's worded much better, but that's the essence of the email and we sent out that email. Uh, 2 customers and uh we had a plan to, uh, we can work with the customer service team, um, to, so they had a script in case customers called in and said, you know, I can't log into my account what should I do? Um, so we have a script where we'd ask them to go check that email if they got us an email, and if they did then. You know, you just basically have to reset your password and then we check if there's any nefarious activity on that account, like, you know, points being used or something like that. So, uh, this was a pretty intense, uh, thing that happened and, uh, it wasn't, uh, it was a nebulous area where I jumped in and, uh, took ownership and, uh, handled, uh, the issue in like one, a little over one day.

Um, that's because, as far as I look from my perspective, the accounts were my responsibility, my and my, my team's responsibility, and I didn't want to wait around the whole weekend and then, um, you know, wait for Monday and figure out like, no, it's not ours or ours or whatever, so somebody had to do it, so I jumped in and did it along while involving the other teams.

Yeah, so one thing that I be trying to look around corners, right? So trying to understand where the risks can come from and then understanding if you have a, um, you know, if you have a, uh, process for that. So for example, uh, how I implemented that was, um, we had a campaign, um, at Amazon, uh, in, in the Alexa coms team. I ran a campaign on, uh, Father's Day. To promote calling, right now, the, the, the Alexa comms is built around the use case of a uh family communicating with, with, uh, you know, among each other and especially in the kitchen. So it's a very, uh, uh, you know, family-centric use case, family-centric kind of, uh, functionality. And specifically, uh, utterances like call mom and call dad are highly polished, and, uh, um, uh, they're, they're very good experiences. So, we, um, so I wanted to promote call Dad on Father's Day. And um um uh, you know, uh, have a push notification promotion for that, um, saying like, you know, uh, call, call Dad on Father's Day. So what here I was able to, thanks to my previous experience of, like, you know, how things can go wrong and um it's not the same kind of thing that can go wrong, but basically the importance of having process laid out, right was what I remembered. So, um, in this particular case, what I tried to anticipate was, what if somebody, um, you know, didn't like that push notification? What if it blew up on Twitter, right? Or if somebody says, um, you know, my dad has passed away some years back, and I felt really upset when I saw this, right? So what happens after that? If you see a tweet like that, of course, we'll shut the campaign down, but then like what else should happen? Who's gonna reach out to that customer? Um, is that OK by PR? And all those things, so I was able to anticipate all that and um um get alignment on what the process should be before it blew up, right? And in fact it did blow up in one or two cases and we had to kill the campaign. So when, when that did happen, we did have, um, you know, a series of things that we could do, uh, to, to handle the situation.

Yes, so, um, the guardrail criteria I set up was if we see, uh, 3 or more instances of of customers being upset with a, uh, with this notification, then we'd shut it down. And, um, I personally was monitoring that using a tool. Uh, called N, it's a social media monitoring tool. Um, so I was monitoring that. The campaign was on a Saturday. I was monitoring it with my leadership on, on, on call. So if something went wrong, if 3, if we hit the Guardian metric, we'd kill it, and I'd tell my leadership saying that, hey, we've hit this, and I'm gonna kill this campaign, right? So, uh, that is how we monitored it, and, uh, we killed it after we saw 3 instances of it.

So it's, yeah, I had to, uh, work with my boss on that. I got a similar pushback about, about Y3, but I was indexing, overindexing on not upsetting customers. So there's, on the one hand, there's that that can go wrong. On the other hand, there's this benefit of customer benefit that so many customers are receiving. So it is, it is a subjective call, Brandon, and I was being, I was, uh, overindexing on, on customers having a good not, not upsetting customers basically. And um that's why like after 3, we kill it.

Right, yeah. So, the, um, uh, NorthStar for this program was, uh, if it was successful, it'd be part of our automated campaigns every year. So, um, I ran a bunch of, uh, uh, this was pretty much a manual campaign. So, um, um, I ran it um uh in a system within Amazon, um, and the hope was to onboard this onto our recommendation platform as an automatic campaign. And um here I saw this negative impact and it is, it was a subjective call that I made because we've looked at ways to um um quantify how bad a tweet is uh companies like like Nvi claim to do that, but it's, it's really not, not an exact science. I looked at, um, their algorithm and how it, uh, labels tweets and just, yeah, go ahead. No, just keep going. Yeah, so, um, it just doesn't catch things like sarcasm, and it's, it's the algorithm is not, not absolute at all. You can't, we can't rely. So the, so there's not like a, a, quantifiable way to do it. It has to be like a qualitative subjective call, right? So, and the 22 or 3 tweets that I saw were like bad enough that I pulled the plug. I did set the guardrail, uh, before, like, uh, as 3 tweets. And there was discussion between me and my manager. For example, the first tweet we saw was, um, uh, hey, uh, it was like somebody put that up with a, with a smiley face saying that my dad's been dead 10 years. Why would you bother me with this, which we were still thought that it was, you know, it was not, not, uh. It was not customers being hurt or something like that. When there was this lady who tweeted that I saw this when I was shopping at Walmart. I saw this, and it made me fall on the floor, and, uh, I wanted to fall, fall on the floor and cry or something like that. I miss my dad so much. So some customers were clearly very badly impacted, and it was a subjective call in the end. Uh, I had the guardrail metric. Uh, it was a subjective call in the end. OK, and, and one more thing, Brandon, like above, I wanna add to that is something that like that can really blow up in social media, right? Like if you have an influencer or somebody picks something like that up, it can really blow up badly, uh, on our faces. So that's the reason I wanted to, wanted to kill it