How Digging Deeper into Connection Timeout Errors Transformed Our Energy Pricing Strategy

Gotcha. Yeah, give me 30 seconds, Brandon. Sure. OK, um, I don't wanna keep you waiting for too long, so, um. So this is a question where um the problem was not clear on the surface, and I had to dig into the details. Um, to figure out what, uh, what exactly, um, was the problem. OK. Um, how, OK, well, let, let me talk about, uh, an, an instance, um, So One of, um, so essentially, you know, uh, same product, right? Um, uh, working with the electricity markets, allowing them to participate. Um, this was, this happened about 6 or 8 months ago where we sent out a release that was, um, supposed to go into uh customer systems, uh, within 1 week into production. Uh, when we put it into, uh, test, uh, into the test environment, we see. Uh, we saw, um, our, our prices tasks, um, um, randomly fail with a timeout or a connection timeout error. Um, and if when prices, uh, with these are market prices, the 5 minute clearing prices that we're downloading, um, when, and when prices fail, it causes one, it causes operational issues because, um, customers are not able to forecast their energy strategy in the, in the near term, exactly how will they buy and sell because they don't have market, you know, past market prices, and two. It causes gaps in our validation tool, so we're not able to validate the um uh the energy bills correctly. Um, and so this, so this, uh, problem in test was, uh, pretty critical, and we had to figure out how to prevent, uh, or what is essentially causing these, uh, these public, um, crisis tasks, uh, uh, to, to fail with the connection reset randomly. So, Um, the, the first thing I did was I looked at my automation system to see what the price tasks are doing there because that works as a proxy. Um, I noticed there are no failures, uh, in my automation system, so I was fairly confident that there's nothing wrong with the code. Um, second, um, this, um, uh, this test system was on our

Yeah, so, so I have an internal automation system right that I talked about. So I, so, and that's constantly running, downloading data, and I didn't see any failures with the task there. So, so that gave me some confidence that the, that there's nothing wrong with the code. It's probably some sort of a configuration issue somewhere. Um, so then I turned to the system where it was failing, which is the hosted test environment, uh, which is on AWS and, and, um, I worked with, um, um. Um, my, uh, my client services team that supports, um, those environments and. At first, um, um, I requested that we essentially start looking at the, the proxy configuration, uh, because the, our hosted environments have a proxy that, uh, that the internal test environments do not, do not use. Um, and this proxy comes into play when we, when we connect with the market's public, um, uh, public FTP, uh, that we use to download the data. Um, so, um, the, the, um, the client services folks that I, the managers that I worked with said, OK, the proxy does not have an issue, it looks fairly good, um, um, and so then I requested that can we use the same proxy. On my internal test system, like, essentially, uh, can I connect my internal test system to the same proxy and then see if I can reproduce the issue on my internal system. Um, and so, So, so, so, so, so step number 2 was, um, no, go ahead. No. Perfect. Yeah, yeah, so, so step number 2 was to, um, connect my internal system with the proxy, and um then I could see the random errors. Um, I could reproduce the errors. We didn't know what the root cause was. We know it's something related with the proxy, but we don't know exactly what, what's, what's the issue with the proxy. Um, so then step number 3 was to just, uh, have another meeting with the, with the manager of the client services team, show them, show him, a history of about 12 months of where the task had not failed, uh, prior to, uh, on the test system, uh, prior to, uh, using the new proxy, and then, and then right after we, we started using the proxy, it, it was clear that, uh, it was failing about 5 or 6 times, um, uh, um, uh, in a 6 hour period. Um, And so then, uh, then when, uh, when we involve the network team, uh, step number 3 was involving the network team to figure out. Um, what changes have we made with the proxy recently because it's not happening in production, it's only happening on the, on the test environment. We saw that there were additional load balancers that we had added. Um, and, um, and, and then we started, um, using one specific, you know, uh, each kind of load balancer by itself, and we saw the issue was with the three new load balancers, um, and then after troubleshooting some more we figured out that we needed to enable stickiness on, on, on those new load balancers that, um, because the, the IP address that was going. Uh, where we were sending and that the, the IP address that was coming back was, uh, uh, different and so by enabling stickiness, um, that we resolved that issue. Um,

The, the network engineer.

Yeah, so the, the network engineer, the, the client services, um, uh, manager, um, the, um, um, the AWS, um,

Um, One day they let us connect with uh uh they let us connect the internal test system with with the proxy they had to do some configuration and sort of a setup to enable test systems to to connect with uh the the internal test systems to connect with the the the customer testy test system. Uh, 2, we had about 15 load balancers and so we, they essentially did some tests where they switched, switched off the, you know, the 15 and, you know, just had one load balancer, then did it with the second one, then did it with the third one, and then, so essentially those tests that they had to do because they had the, the, the configuration know-how, um, to, to, to narrow it down. Um OK