How This Product Manager's Bold Disagreement Redefined Key Success Metrics

The time that I disagreed with my manager was when, um, so it was in my current role. I'm a product, uh, program manager at Microsoft, uh working in the commerce space, um, on compliance, um, uh, financial compliance initiatives. And so this project was related to, um, Working through, um, reducing manual production, uh, manual touches to production, um, for our internal commerce teams. And, um, the, the situation was specific to, um, Uh, basically reporting out on, on the, um, metrics that we were going to show leadership teams. So the background of this project was related to the fact that, um, our, our, um, Our internal teams are are, are uh accessing production by um manually going in and creating these production changes on the spot. And so the project that I was working on was to reduce these manual touches so that the, uh, developers, the engineering teams are going in with automations to create these production changes instead. Now, the metric that we were using to track this, um, this move from manual to automation was, um, the number of um sessions that developers use just in time, um, access to access production uh production environments. And so, um, This metric was reported, was being reported out in monthly um report outs to leadership team, and my manager wanted to um basically report out on all of the sessions regardless of like whether or not the, um, the Internal teams were using to to access production. Whereas, um, I think they're like from my standpoint in discussing with all of these engineering teams and understanding what their reasons, what their root causes were to access, um, to, to use just in time access to access production to create these production changes. Um, I understood that there were certain scenarios that were related to um initiatives that were planned, and meaning that they were one-off um organizational-wide initiatives that would create changes that would require one-time changes in production and then um would not reoccur once that program had ended. And so, Um, I had some back and forth with my manager to really understand what was the correct metric to be reported out during these scenarios, uh, during these report outs, um, these monthly report outs, because the reason was that, um, it doesn't paint a From my understanding, it doesn't paint a correct viewpoint when we're looking at these scenarios that are one time not expected to reoccur and then um basically put the, the onus on the, on these engineering teams that yes, you are, you are using or you're accessing production manually to create these changes, but Um, there's no ROI for these teams to, to automate because once the initiative ends or the planned program ends, there's no need to use those automations further. Um, So based off of my discussions with um the engineering teams who are creating these uh production access accesses, um, I had the data to go back to my manager and, and help him understand um the difference between what um what we would be showing if we were looking at all, uh, all of these sessions that the developers had. Created for production access versus if we removed the number of sessions that were not expected to reoccur, um, and help them understand that, you know, that this, this is not the, the reporting or the metric that we want to look at in terms of success for, for reporting out on, on this, on this, um, initiative. Um, and so, From that discussion, um, I've brought in more, but, um, between, between, um, initially the discussion was between myself and my manager, and then we looped in, um, other subject matter experts from the team to understand um whether or not these scenarios were really, um, one-offs. And so these ones, these metrics, and whether or not these metrics could actually be removed completely from, from the reporting. Um, and in the end, um, Although, um, so my manager's initial stance was just to report out on all like the entire, entire set of sessions, whereas, um, I, my proposal was a subset. And so in the end, um, what happened was we decided on a middle ground, which was to report both of these metrics because they're able to paint, um, the true usage of, of, um, of our internal engineering teams. At the same time, um, we show an adjusted view which Um, my proposed, uh, metric, um, so that would show what would happen if teams, um, completed those tasks and it did not occur again as they had expected or as they had, um, told us, and then show both of these metrics together, um, but, uh, like use that, use that towards, towards reporting instead of deciding on one versus another.

Mm, um, so the reason that I felt like it didn't suit, um, the reporting, so we had a target that we needed to meet for this initiative. So the target for the initiative was to reduce, um, reduce the manual production access by 77%, um, across the org. And um in order for us to reach that goal, the actual, um, the actual number that my manager had wanted to report out on, we would pretty much not be able to meet it because there are um many initiatives that are ongoing inside our team that may cause, um, Production like one-off requests that are compliance-related because they're related to um audit requests like financial audit requests that will cause teams to increase their um access to production to create one-off tasks. So because they are one-off tasks, um, it wouldn't help, like they wouldn't be in scope for automation if teams are creating or going to spend. Um, say, uh, uh, there's a time crunch to meet the, the, um, compliance request to provide specific data, um, but then teams would also need to work on these compliance requests in addition to their future requests or requests that actually, um, You know, are related to um features that will earn money or like product product features. Um, so that's why there's less incentive for these teams to create automations that are just one-off ass for yearly audits, and these yearly audit requests could change depending on, um, audit like findings each year.

Right. So in terms of preparing for the discussion with my manager, um, I had gone through, so there are logs of each of these, uh, sessions where developers are, um, accessing production and we have a questionnaire that these, um, that, that is provided as part of this to understand what was done specifically. Um, when, when the, uh, developer access production, and so I used the, I took a subset of, so because this was for a monthly reporting, so I took the month's, um, logs to understand, um, what was done specifically and so, um, I'll, I'll just use arbitrary numbers because they're, uh, they were for a specific month, but of the, um, So say if there was 100, 100 sessions that for, for that particular month, um, I would look at which ones specifically were related to, um, To the planned sessions that were not expected to reoccur. So looking at if, if there were 100, um, if 30 of them were related to, um, to the sessions that were not expected to recur and therefore don't have enough ROI to automate, um, my suggestion would be to Uh, report at the 70 number instead of the 100 number. And so, um, for those 30 that were related to um planned session or sessions that were not expected to recur or had low ROI to automate, um, I confirmed with The developers who actually performed those sessions to confirm that um what they had attested to in the questionnaire was accurate, and then also spoke with subject matter experts um on our team to understand if um these, these sessions that were really, these 30 sessions really were related to um Scenarios like the, the, um, the scenarios that would not reoccur. So for instance, if um if 15 of those 30 were related to a certain program, I would go to the program owner and ask like, did, did, is the er or is This was this team actually in scope for, for your initiative and get got that confirmation that they were, they were expected to do that work for that specific time frame and then based off of that information, go back to my manager to help them understand that, you know, the, these sessions were not so. What occurred again, yeah.