Been bughunting the last week or so on this import job that is suddenly running so slowly that it takes more than 24h and is restarting on top of itself.

It used to run in anywhere from 4 to 8 hours, which was bad enough. You see, it ran on timer scheduled by our main site. So our deployment window was determined by when this job finished, and if it didn't finish during work hours, then no deployment that day.

So we got the idea to move it to a separate service to eliminate that deployment window bottleneck. And now, seemingly unrelated, it is just running slow as shit.

There is a lot of bad design in the code, and we know we want to build a completely new solution. But we also absolutely need this import to run every day until a better solution can take over.

We've taken care of some of the most obvious problems that could cause the poor performance, but it's unclear whether it's going to be enough. And with a runtime of about a day and wild variations of the most atomic partial imports, it's extremely tedious to test as well...