Follow-up on https://devrant.com/rants/5001553/...

How the fuck are Jupyter notebooks so popular in research? Like some dude had an idea to take perfectly good markdown and python code, add a whole lot of transitional properties to make version control impossible, encode it as JSON on the assumption that a human could somehow look at it and make sense of countless escaped characters and base64 encoded data, create dedicated software people need to install in order to read what used to be simple plain text, and think "This. This is what 99% of data researchers will use from now on." And somehow, overwhelming majority of researchers agreed that this extremely inefficient data format is the best there is and they should develop all their tools around it.

Edit: nvm no clue what I wrote. I need to wake up properly.

@RememberMe How is it painless when I can't even auto-merge a simple one-line change without dealing with a ton of merge conflicts due to all the counters and whatnot being saved with the code? Let alone having multiple people contribute their work to the same notebook...

@hitko see edit

@RememberMe Too late :P Nvm then

I don't know anything about python or notebooks but this gives me a headache. The only time I encountered a notebook is when someone on youtube tried to explain to me how I make a self learning AI with only 3 lines of code and out of the sudden many diagrams appear.

Have you ever talked to someone in academia who writes code?

They have absolutely zero understanding of it. They can write these weird arrangements of characters that sort of looks like math, and it does something, and they're happy.

If their understanding of code was a uniform sphere with diameter 1m and mass 5kg in Hawaii, the knowledge of Jupyter notebooks would be a ship off the shore they can swim to. An understanding of clean code would be in Western Canada. Maintainable code would be in New York. An order of magnitude past that, the understanding of modularity rests on the Moon, and another order past that: version control is in orbit around Enceladus. Performant code? Alpha Centauri. Assume no air resistance when calculating these difficulties.

@Root wow. That was harsh but I love it.

Check out jupytext. Replaces notebook json format with plain old source files.

https://github.com/mwouts/jupytext

No association.

@atheist Nice one, that solves so many problems!

Although when there's an option to use a different format or runtime, there's already plenty of alternatives to choose from, the real problem is that many tools and researchers only support / accept the original notebook format.

@hitko I share my notebooks as docker containers and don't care what people want. Docker should be an acceptable GCD.

I'm also not in academia...

That said, jupytext does "just work", it's the same notebook interface, the extension just annotates the source file as needed to denote cells and stuff. If other people don't have the extension installed, it is just a source file with comments. It can also convert between the two, so having a CI step to create the nb format would also work.

I'm... Not the typical NB user.

The reductionism runs deep here… like can you really think of a better system for giving academics

1) a stock-standard, always works data visualization environment + animation / interactivity / reproducibility and being able to share it all in a browser?

And that’s not even the main thing. Forget matplotlib/pandas/ipywidgets, those are cool but being able to actually command the program control flow in a JIT fashion (by running each cell when one chooses to) is game changing.

Not everything is a nail lol