A decade ago 800x600 was pretty much the standard resolution for devices and 5 sec response time was considered fast. Animations were minimal and websites were easier to read. Programmers debated around topics like which loop runs faster, i++ or ++i, while vs doWhile and so on. In general, we were closer to understanding what happens behind the browser curtain and how code needs to be organized to make it more maintainable.

Today the level of abstraction is much higher. I don't think devs can contemplate on the finer aspects of programming efficiency; they'd rather rely on a code library to do all the grunt work. With the explosion of devices and platforms, the focus has shifted from programming to assembling. Programmers need to know their tools first, then write code. The tool is expected to work well with a millisecond response time, not the programmer's code.

Moving forward, I think programming would be more about building higher abstraction utilities/libraries that are integrated by other tools, which is already happening. Marketing an App would become more important than the actual skill needed to develop it.

A bit far-fetched, but I think the future programmer would be a lot like a stock market analyst who has a bunch of windows in front, just observing data or algorithm patterns created by an AI engine and cherry-picking a specific combination of modules that might make the next big sensational app.

Did you know?
This rant is a part of the 3 quintillion bytes of data that the world generated today.

Is it just me, or are the media / journalists once again putting a stupidly unfair pessimistic spin on that SpaceX launch?

"SpaceX rocket launches but explodes shortly into flight"
"Musk's SpaceX big rocket explodes on test flight"
"SpaceX rocket explosion: None injured or killed"

They've said time and time again, it's the first test of a massively complex rocket that's bigger than anything that's ever gone before it, and success is just defined as "getting off the launch pad" and collecting data. They did that and then some.

But instead of spreading excitement about the data, the fact it launched, that it's a world first, etc. - it's all doom and gloom, implying that the whole thing was a failure and people could have died 🙄

And people wonder why I have a low opinion of journalists.

In 2015 I sent an email to Google labs describing how pareidolia could be implemented algorithmically.
The basis is that a noise function put through a discriminator, could be used to train a generative function.
And now we have transformers.

I also told them if they looked back at the research they would very likely discover that dendrites were analog hubs, not just individual switches. Thats turned out to be true to.

I wrote to them in an email as far back as 2009 that attention was an under-researched topic. In 2017 someone finally got around to writing "attention is all you need."

I wrote that there were very likely basic correlates in the human brain for things like numbers, and simple concepts like color, shape, and basic relationships, that the brain used to bootstrap learning. We found out years later based on research, that this is the case.

I wrote almost a decade ago that personality systems were a means that genes could use to value-seek for efficient behaviors in unknowable environments, a form of adaption. We later found out that is probably true as well.

I came up with the "winning lottery ticket" hypothesis back in 2011, for why certain subgraphs of networks seemed to naturally learn faster than others. I didn't call it that though, it was just a question that arose because of all the "architecture thrashing" I saw in the research, why there were apparent large or marginal gains in slightly different architectures, when we had an explosion of different approaches. It seemed to me the most important difference between countless architectures, was initialization.

This thinking flowed naturally from some ideas about network sparsity (namely that it made no sense that networks should be fully connected, and we could probably train networks by intentionally dropping connections).

All the way back in 2007 I thought this was comparable to masking inputs in training, or a bottleneck architecture, though I didn't think to put an encoder and decoder back to back.

Nevertheless it goes to show, if you follow research real closely, how much low hanging fruit is actually out there to be discovered and worked on.

And to this day, google never fucking once got back to me.

I wonder if anyone ever actually read those emails...

Wait till they figure out "attention is all you need" isn't actually all you need.

p.s. something I read recently got me thinking. Decoders can also be viewed as resolving a manifold closer to an ideal form for some joint distribution. Think of it like your data as points on a balloon (the output of the bottleneck), and decoding as the process of expanding the balloon. In absolute terms, as the balloon expands, your points grow apart, but as long as the datapoints are not uniformly distributed, then *some* points will grow closer together *relatively* even as the surface expands and pushes points apart in the absolute.

In other words, for some symmetry, the encoder and bottleneck introduces an isotropy, and this step also happens to tease out anisotropy, information that was missed or produced by the encoder, which is distortions introduced by the architecture/approach, features of the data that got passed on through the bottleneck, or essentially hidden features.