Unlocking Efficiency: How One Engineer Reduced Reporting Time from 36 Hours to Minutes by Asking the Right Questions

OK. So, uh, when I was working with my previous organization, uh, uh, there, there was a, uh, so I was managing a managing a data analysis and reporting, uh, project. The project was basically migrating all these SQL based, uh, reporting and analytics onto the big data ecosystem. Now, uh, during the, uh, in the midst of, you know, various deliveries, uh, my, uh, uh, team came to me asking for an approval, and that approval will basically, uh, uh, having more resources. When I say more resources, it's basically computational and storage resources on the big data cluster. So, uh, before approving it, I, I just wanted to verify, uh, the findings of my team and I just wanted to understand why my team actually came to this conclusion that they will require more resources and, uh, more storage. So I had uh set up a call with my team. Uh, uh, I, I started having conversation on what's happening, why this kind of, uh, uh, you know, resources are required. And I started asking more technical questions. Uh, basically, I, I, the first question I asked was, uh, you know, why you need it, why you need a competitional resource, and the answer I, I got is, uh, you know, the batch reporting which we were generating taking 36 hours, which was a daily batch reporting which was taking 36 hours. So it doesn't make any sense for business to, uh, have, yeah, so, uh. Yeah, so I, I said why it's taking 36 hours. My question was why it's taking 36 hours, and then I realized that there are around 50 output high views. High is basically the, the kind of technology we use, and these views were getting created from around 70 to 80 staging views. And then I asked, uh, why we why we are having this, uh, staging views, uh, why not tables? Why not we store the data. And uh one of the members from team uh realized that yes, uh, you know, that there is a possibility of doing that and then I asked questions that are all the data are changing day in day out. Uh, do we have any data which is not changing for weeks, for 10 days, 15 days? We can actually store them, create a table, and using those tables, we can create the views. It will actually improve our performance because it will work as cache. And team realizes that yeah, this, this could be one of the approach. So I quickly ask my team that before I do any other approvals, you just go ahead and do a pilot, uh convert all these uh views which are basically, uh, you know, uh, possible to change, uh, uh, possible to store it into the storage. You just do a pilot storing those high views, and I said one thing that storage is 10 cents per GB whereas memory is $88 per GB. So I, I asked my team to be mindful about this, uh, and I also ask my team to be disciplined about spending money, and I say that when you spend money, think as if it's your money, right? It's not organization's money, even if it's a $4000 or or or a $40. So think, be mindful about that.

Yeah, so I, I said to my team that, uh, be mindful about it and uh once they did the pilot, uh, we increased the, uh, so our cycle time from 36 hours, went down to 8 hours. Uh, there was still scope of improvement which we took it as a continuous improvement later on, but it was moved from 36 hours to 8 hours. We were able to generate the reports, uh, before the business signs in. And that was through increasing the storage because I asked my team to store the staging data instead of keeping it into the, into the memory itself. So that's how, uh, you know, I went deep into the asking questions. I basically followed a, a framework which is called uh 5Y framework. Uh, I am a true believer of that. Uh, you know, I went deep into various questions and then realized that, uh, and it was a realization to team as well that, you know, uh, this could have been the better approach.

Yeah, so my, my, uh, uh, responsibility was to asking the question and getting the details of, uh, what is there. Uh, the suggestion came out from my side. The idea came out from my side of why not to store the data which is, which is, uh, you know, you are actually, uh, which are not changing that frequently. Why not to store it. That idea was basically coming out from my side, but it, it came after asking so many questions. So my responsibility was to, you know, asking questions, getting the answers from the team and then suggesting some path forward to the team.

Yeah, yeah, absolutely. So I would say uh definitely I trusted my team. I just wanted to verify, um, you know, the, the details which team was providing. So it's not that when I went into the room, uh meeting room, I was having that idea in my mind. Uh, somehow I was skeptical about, uh, why it's taking 34 hours. I was, uh, doubting, uh, you know, uh, what could have been wrong here because I was not believing that creating uh 50 plus batch reports taking 36 hours. So that approximation actually was there, but it was not that, uh, I was having some ideas in mind and then going into the meeting room. uh, the idea came after digging down uh to the answers of the questions which I asked to my team.

Yeah, so as I said, uh, our cycle time reduced to 8 hours.

yeah, and that was because, you know, we use storage instead of computational memory.

Uh, there was a, there was a bit of, uh, approximation, um, during the calculation. So when, uh, I suggested that idea, I actually, you know, was trying to figure out, uh, if we go ahead and let's say if we stage 14 to 15 high views, what could have been the saving of time. The approximation was there in my mind. However, the actual result which we received was quite better than what My approximation was. So my approximation was probably I would be hitting 50%. It means I would be hitting somewhere around uh uh 12 to 13 hours, but it hit 8 hours. So, uh, yeah, that, that was the kind of approximation I was having in my mind, but I was not having a clear cut, uh, saving, uh, in my mind, uh, which can tell me that this many hours will be saved.

I just wanted to try it. I just wanted to try it, uh, you know, uh, because instead of just going ahead and buying resources, I thought let's try it and let's, let's, if at all, in worst case we'll fail, but let's try it.