The company I worked for had to do deletion runs of customer data (files and database records) every year, mainly for legal reasons. Two months before the next run they found out that the next year would bring multiple times the amount of objects, because a decade ago they had introduced a new solution whose data would be eligible for deletion for the first time.

The existing process was not be able to cope with those amounts of objects and froze to death gobbling up every bit of ram on the testing system. So my task was to rewrite the exising code, optimize api calls and somehow I ended up in multithreading the whole process. It worked and is most probably still in production today. 💨

They keep training bigger language models (GPT et al). All the resear4chers appear to be doing this as a first step, and then running self-learning. The way they do this is train a smaller network, using the bigger network as a teacher. Another way of doing this is dropping some parameters and nodes and testing the performance of the network to see if the smaller version performs roughly the same, on the theory that there are some initialization and configurations that start out, just by happenstance, to be efficient (like finding a "winning lottery ticket").

My question is why aren't they running these two procedures *during* training and validation?

If [x] is a good initialization or larger network and [y] is a smaller network, then
after each training and validation, we run it against a potential [y]. If the result is acceptable and [y] is a good substitute, y becomes x, and we repeat the entire procedure.

The idea is not to look to optimize mere training and validation loss, but to bootstrap a sort of meta-loss that exists across the whole span of training, amortizing the loss function.

Anyone seen this in the wild yet?

Spent an hour trying to optimize some code but none of the stuff I was trying (rearranging/overlapping DMA transfers with other stuff for some ye olde concurrency) seemed to be doing anything.

Aaand obviously the stupid thing was set to autobuild Debug config on save but I was testing Release config. Sigh.

!rant

I'm in the process of building a reccomender engine for the lols. After doing it for awhile I've realized that once I get it going then it will be very hard to optimize because I can't think of test cases or deterministic correct answers. Has anyone built one here who has tips on testing one? Thanks :)