!rant

Jupyter notebooks are so fucking awesome.

Look, I'm not even mad that your dataset is the spaghettiest of all spaghetti, but why do you have ten different jupyter notebook files lying around?

I mean, I'm not implying that a monkey has more brain in his armpit than you have in your entire body, but like, you call this a dataset while all over seen so far is half-processed garbage. You could've just dipped your pc in sewage and the results would still be cleaner than this.

Luckily, your paper is half decent so what the hell, let's see if I can fish anything useful out of this. But I swear to god if I come across another static path in this... And here we go! Another static path! Ladies and gentlemen, I propose we get this guy's phd back until he learns to fucking do a decent code.

(It's actually a massively complicated project, so it kinda makes sense to be this big of a mess. But still!)

Cousins came over...

Me: just compiling some python code, opens up jupyter notebook to take a look at some data science code
Little Sis: *looks at jupnb dump on cmd*
Whoa are you Hacking?
Me: yeah. I got bored of whole Hacking command typing thing so I opened up my hacker console.
*print("hello world")*
Sis:wow!
Me: you know what, typing is too tiresome, I'll connect to it with my mind
*alt-tab*
*cmatrix -b*
*sits in yoga pose*
Little Sis: Screams at the top of her lungs and runs to aunt
"DAVE IS HACKING MATRIX"

REDIS: Great for cloud, will fuck up your local disk if too many write operations per second.

DynamoDB: WTF 10Mb should not be "too large for a single record"!!

SPARK: NEVER CONNECT IT TO A DATABASE! Wasted A LOT of cluster time. Also, can you be LESS specific on exactly what are the bugs in my code? 'cause I don't think it's possible.

NPM: can't install a package for shit. tried it waaaay to many times.

Makefiles: Just fuck you.

WSL1: breaks more often than a glass hammer.

Python >= 3.6: FUCK ENCODINGS!!

Jupyter: STOP MESSING UP WHILE SAVING!

Living is to collet bugs, it seems.

To all the data engineers in here: WTF is going on in your field?

I've worked closely with a dozen data engineers in the last 5 years (and talked to friends and internet strangers about this and get similiar responses), mine if them seem to know how to use a computer!

They don't understand git, ORMs, best practices, how to use a terminal, DAGs (important for using modern ETL scheduling tools like airflow and prefext), etc

Guys with 10 years of experience on their resume and they can't wrap a model into a flask app with 1 endpoint. They'll reference local files on their machine in w jupyter notebook and are shocked it won't work on other computers!

God bless being a student. I just moved a massive calculation to uni's jupyter servers. Saved me from a shitton of effort and burning my laptop down. 🙏

Running a fucking conda environment on windows (an update environment from the previous one that I normally use) gets to be a fucking pain in the fucking ass for no fucking reason.

First: Generate a new conda environment, for FUCKING SHITS AND GIGGLES, DO NOT SPECIFY THE PYTHON VERSION, just to see compatibility, this was an experiment, expected to fail.

Install tensorflow on said environment: It does not fucking work, not detecting cuda, the only requirement? To have the cuda dependencies installed, modified, and inside of the system path, check done, it works on 4 other fucking environments, so why not this one.

Still doesn't work, google around and found some thread on github (the errors) that has a way to fix it, do it that way, fucking magic, shit is fixed.

Very well, tensorflow is installed and detecting cuda, no biggie. HAD TO SWITCH TO PYHTHON 3,8 BECAUSE 3.9 WAS GIVING ISSUES FOR SOME UNKNOWN FUCKING REASON

Ok no problem, done.

Install jupyter lab, for which the first in all other 4 environments it works. Guess what a fuckload of errors upon executing the import of tensorflow. They go on a loop that does not fucking end.

The error: imPoRT eRrOr thE Dll waS noT loAdeD

Ok, fucking which one? who fucking knows.

I FUCKING HATE that the main language for this fucking bullshit is python. I guess the benefits of the repl, I do, but the python repl is fucking HORSESHIT compared to the one you get on: Lisp, Ruby and fucking even NODE in which error messages are still more fucking intelligent than those of fucking bullshit ass Python.

Personally? I am betting on Julia devising a smarter environment, it is a better language already, on a second note: If you are worried about A.I taking your job, don't, it requires a team of fucktards working around common basic system administration tasks to get this bullshit running in the first place.

My dream? Julia or Scala (fuck you) for a primary language in machine learning and AI, in which entire environments, with aaaaaaaaaall of the required dlls and dependencies can be downloaded and installed upon can just fucking run. A single directory structure in which shit just fucking works (reason why I like live environments like Smalltalk, but fuck you on that too) and just run your projects from there, without setting a bunch of bullshit from environment variables, cuda dlls installation phases and what not. Something that JUST FUCKING WORKS.

I.....fucking.....HATE the level of system administration required to run fucking anything nowadays, the reason why we had to create shit like devops jobs, for the sad fuckers that have to figure out environment configurations on a box just to run software.

Fuck me man development turned to shit, this is why go mod, node npm, php composer strict folder structure pipelines were created. Bitch all you want about npm, but if I can create a node_modules setting with all of the required dlls to run a project, even if this bitch weights 2.5GB for a project structure you bet your fucking ass that I would.

"YOU JUST DON'T KNOW WHAT YOU ARE DOING" YES I FUCKING DO and I will get this bullshit fixed, I will get it running just like I did the other 4 environments that I fucking use, for different versions of cuda and python and the dependency circle jerk BULLSHIT that I have to manage. But this "follow the guide and it will work, except when it does not and you are looking into obscure github errors" bullshit just takes away from valuable project time when you have a small dedicated group of developers and no sys admin or devops mastermind to resort to.

I have successfully deployed:
Java
Golang
Clojure
Python
Node
PHP
VB/C# .NET
C++
Rails
Django

Projects, and every single fucking time (save for .net, that shit just fucking works on a dedicated windows IIS server) the shit will not work with x..nT reasons. It fucking obliterates me how fucking annoying this bullshit is. And the reason why the ENTIRE FUCKING FIELD of computer science and software engineering is so fucking flawed.

But we can't all just run to simple windows bs in which we have documentation for everything. We have to spend countless hours on fucking Linux figuring shit out (fuck you also, I have been using Linux since I was 18, I am 30 now) for which graphical drivers for machine learning, cuda and whatTheFuckNot require all sorts of sys admin gymnasts to be used.

Y'all fucked up a long time ago. Smalltalk provided an all in one, easily rollable back to previous images, easily administered interfaces for this fileFuckery bullshit, and even though the JVM and the .NET environments did their best to hold shit down, and even though we had npm packages pulling the universe inside, or gomod compiling shit into one place NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO we had to do whatever the fuck we wanted to feel l337 and wanted.

Fuck all of you, fuck this field, fuck setting boxes for ML/AI and fuck every single OS in existence

Man....I keep up with this strange love hate relationship I have with Python....

Last night it was python that literally wrote my homework: define all possible equivalent partition tables with cause and effect analysis and boundary value checks for a program. The whole thing wrote itself and all I had to do was verify the inputs. Something that I was able to do using jupyter with pandas and numpy. On one hand, I despise the lack of static typing and use of whitespace as a block delimiter. On the other I cannot but help feeling a high level of gratitude over the language and its high availability and ease of use for this.

Sure, I could have used other tools, but this language has dominated hardcore in this regard enough to the point of not considering it being a crime against humanity.

I decided to learn Swift as I would like to make apps. So I installed swift and swift-jupyter in my Linux server after my 2012 iMac literally took 20 fucking minutes to compile `print("hello world")` in Playgrounds. Meanwhile, my server compiles python code in a swift jupyter notebook in seconds.

python machine learning tutorials:
- import preprocessed dataset in perfect format specially crafted to match the model instead of reading from file like an actual real life would work
- use images data for recurrent neural network and see no problem
- use Conv1D for 2d input data like images
- use two letter variable names that only tutorial creator knows what they mean.
- do 10 data transformation in 1 line with no explanation of what is going on
- just enter these magic words
- okey guys thanks for watching make sure to hit that subscribe button

ehh, the machine learning ecosystem is burning pile of shit let me give you some examples:

- thanks to years of object oriented programming research and most wonderful abstractions we have "loss.backward()" which have no apparent connection to model but it affects the model, good to know
- cannot install the python packages because python must be >= 3.9 and at the same time < 3.9
- runtime error with bullshit cryptic message
- python having no data types but pytorch forces you to specify float32
- lets throw away the module name of a function with these simple tricks:
"import torch.nn.functional as F"
"import torch_geometric.transforms as T"
- tensor.detach().cpu().numpy() ???
- class NeuralNetwork(torch.nn.Module):
def __init__(self):
super(NeuralNetwork, self).__init__() ????
- lets call a function that switches on the tracking of math operations on tensors "model.train()" instead of something more indicative of the function actual effect like "model.set_mode_to_train()"
- what the fuck is ".iloc" ?
- solving environment -/- brings back memories when you could make a breakfast while the computer was turning on
- hey lets choose the slowest, most sloppy and inconsistent language ever created for high performance computing task called "data sCieNcE". but.. but. you can use numpy! I DONT GIVE A SHIT about numpy why don't you motherfuckers create a language that is inherently performant instead of calling some convoluted c++ library that requires 10s of dependencies? Why don't you create a package management system that works without me having to try random bullshit for 3 hours???
- lets set as industry standard a jupyter notebook which is not git compatible and have either 2 second latency of tab completion, no tab completion, no documentation on hover or useless documentation on hover, no way to easily redo the changes, no autosave, no error highlighting and possibility to use variable defined in a cell below in the cell above it
- lets use inconsistent variable names like "read_csv" and "isfile"
- lets pass a boolean variable as a string "true"
- lets contribute to tech enabled authoritarianism and create a face recognition and object detection models that china uses to destroy uyghur minority
- lets create a license plate computer vision system that will help government surveillance everyone, guys what a great idea

I don't want to deal with this bullshit language, bullshit ecosystem and bullshit unethical tech anymore.

Was training a model for an hour on paid GPU after which it showed a deprecation error and crashed my Jupyter notebook.

DAMNIT JUPYTER. Kernel stop dying on me when I need you. I thought this was a mutual loving relationship.

Woke up, worked out, went back to bed. (?? Yeah I'm surprised too) Slept for an hour, woke up again, worked tirelessly and finished the slides. (Not as easy as you think. Had to drag out and undust a few jupyter notebooks again, plus realized that the stupid past me has deleted a bunch of notebooks because of lack of space, and I had to remake one again.)

Now I have to figure out why google slides doesn't like to play my videos, and write my script (don't give me the "don't practice too much" bullshit or "don't need a script". That's for losers. You gotta practice enough that you can cite your presentation even if you got a concussion in the middle of the presentation. Plus, you can modify content in the middle of presentation based on the crowd vibe but you can't do that without knowing your script by heart, can you?) Aaaaaand what was I saying... I forgot... Geez ... Well, wish me luck. This week is gonna be tough. And next week. And probably the week after. Ew.

Python devs and data analysts....

Do you recommend using pycharm for working with jupyter notebooks? I surely had a bad time with it.

I have been using many jetbrains softwares , and am a fan of their docs search and autocompletion. But I don't think there is a full support for jupyter jn it, because sometimes my graphs made using matploit or seaborn just brakes.

And some libraries have a lot of functions taking parameters as " *args, **kwargs " , I don't know what that means but those function take a lot of "value" parameters i guess?(like this: plt.figure(figsize=[13,6], axis=False) )
Pycharm also don't seem to have access to list of those arguments...

Are you having such problems too? Have you found some better ide with autocompletions and support for jupyter? Do tell.

(Ps: i know jupyter can be run directly on a browser, but as i said "auto completions and documentations" )

So the other day, I was working on some Python project when there was this bug that kept transforming. Like seriously, I would turn from "bool not defined" to "function does not exist" to literally "file does not exist"... within the FILE. And when I fixed them, new bugs kept popping up, and I couldn't find anything that was a problem. Nothing. There was this one function in there that, if I changed even the comments in there, would break. And so.... I turned off Atom and turned it on again. ( ha ) Didn't work. I restarted my computer. I copy pasted the file into another file. I used another IDE. I restarted GHCI. I restarted Jupyter Notebook... and after 6 hours... I found that it was because an if statement has a comparison between a bool and a bool, with a = in the middle. (not ==). I swear I almost threw the computer on the floor.

Pycharm could be a nice tool, if only it was not nagging about the professional version and the tools related to it so often. Shit can't even find the jupyter notebook crap. 🙄

NGL, open science feels like anarchy.

Just tried out Jupyter Notebook for the first time. I can see why software engineers wouldn't like notebooks, especially if you intend to actually publish the notebook as code for other people to use (please publish a module that can be imported, not a notebook that has to be hacked to pieces to make it reusable), but it's pretty handy for early prototyping or documentation.

I'm playing around with save-editing for a few GBA games as a personal project, and I used a Notebook to document the save file format with examples.

I am studying an online data science course in python offered by Microsoft.
The instructor uses Surface book and also has a Mac book on his desk for backup because windows.
He uses chrome all the time for his explanation to run jupyter.
The point is even they know their tech is shit but still "Windows has set your default browser to edge" - this happens
¯\_(ツ)_/⁻

This is why we can never have enough software developers

It's true. No matter how many people learn to program, there will never be enough people who know how to program. They don't have to be very good at it either. It is now a required skill.

Minimum wage in first world countries is way above 5$ per hour. A Raspberry PI 3B costs 40$, or at most 1 day of work for the worst paid jobs. And it will run for years, and do routine tasks up to thousands of times faster than any employee. With that, the only excuse that people still do routine tasks, is the inaccessibility of coder time.

Solution: everybody should know how to write code, even at the simplest level.

Blue-collar jobs: they will be obsolete. Many of them already are. The rest are waiting for their turn.

Marketing people - marketing is online. They need to know how to set up proper tracking in JS, how to get atomic data in some form of SQL, how to script some automated adjustments via APIs for ad budgets, etc. Right now they're asking for developers to do that. If they learn to do that, they'll be an independent, valued asset. Employers WILL ask for this as a bonus.
Project Managers - to manage developers, they need to know what they do. They need to know code, they have to know their way around repositories.
QA staff - scripted tests are the best, most efficient tests.
Finance - dropping Excel in favor of R with Markdown, Jupyter Notebooks or whatever, is much more efficient. Customizing / integrating their ERP with external systems is also something they could do if they knew how to code.

Operations / Category Management - most of it would go obsolete with more companies adopting APIs as a way to exchange important information, rather than phone calls and e-mails.

Who would not be replaced or who wouldn't benefit from programming? Innovative artists.

A lot of it might not be now now, but the current generation will see it already in their career.

If we educate people today, without advanced computer skills and some coding, then we are educating future deadbeats.

With all this, all education should include CS. And not just as a mandatory field or something. Make it more accessible, more interesting, more superficial if needed. Go straight to use cases, show its effectiveness in the easiest way possible. Inquisitive minds will fill in the blanks, and everyone else will at least know how to automate a part of their work.

Working with a data scientist on an update to a machine learning api that has dinner logic change with a new model.

He's wondering why his PRs are falling.

He's trying to merge into development from a branch created off of main.

He's renamed all the functions and classes and never updated the call points.

He's using new packages but never includes them in the requirements file, so the docket builds are failing.

His class method definitions don't contain self and are throwing syntax errors.

I've been working with him for 4 days to get him to understand branching, linting, unit testing, and not blindly copy and pasting snippets from jupyter notebook into production api code!

Random thoughts on more out of the box tools/environments.

Subject: Pharo

Some time ago I had shown one of my coworkers about Pharo and he quickly got the main idea behind it but mentioned how he didn't like the idea of leaving behind his text editor to deal with source code.

Some time last week I showed the dude some cool 3d animations you can do with Pharo while simultaneously manipulating the code to change them in real time. Now that caught his attention particularly and he decided he wanted to know more about the language but in particular the benefits of fucking around with an image based environment rather than a file based.

Both of us reached the conclusion that image based makes file based dev enviroments seem quaint in comparison, but estimated that it was nothing more than a sentiment rather than a fact.

We then considered what could be the advantage/disadvantages of such environments but I couldn't come up with anything other than the system not having something like Vim or VS Code or whatever which people love, but that it makes up for it with some of the craziest IDE tools I had ever seen. Plugins in this case act like source code repos that you can download and activate into your workflow in what feels something similar to VS Code being extended via plugins written in JS, and since the GUI is maleable as it is(because everything is basically just subsets of morp h windows) then extending functionality becomes so intuitive that its funny

Whereas with Emacs(for example) you have to really grind your gears with Elisp or Vimscript in Vim etc etc, with Pharo your plugin system is basicall you just adding classes that will convert your OS looking IDE into something else.

Because of how light the vm machine is, portability is a non issue, and passing pharo programs arround is not like installing Java in which you need the JVM.

Source code versioning, very important, already integrated into every live environment and can be extended to do pushes through simple key bindings with no hassle.

I dunno, I just feel that the tool is too good to be true. I keep trying to push limits into it but thus far I have found: data visualization and image modeling to work fine, web development with Teapot to be a cakewalk and work fine, therr are even packages for Arduino development.

I think its biggest con would be the image based system, but would really need to look into how this is bad by any reason other than "aww man I want vim!" since apparently some psychos already made Emacs and VS code packages for interfacing with Pharo source trees.

Embedded is certainly out of the question for any real project since its garbage collected and not the most performant cookie in the jar.

For Data science I can see some future, seems just as intuitive and interesting as a Jupyter Notebook actually, but the process can't and will not be the same since I still don't know of a way to save playground snippets unless you literally create classes for it, in which case every model you build gets saved inside of an object, sounds possible but, strange since it is not a the most common workflow in jupyter.

Some of the environment is sometimes glitchy, but it does have continuos development and have not found many hassles.

There is a biased factor from my side: I seem to be wired to understand the syntax and simple object model better than in other languages. To me this feels natural as if I was just writing ideas rather than code, mostly because I feel that there really ain't much in terms of syntax, the language gets out of my way and the IDE feels like the most intuitive environment in the world to me. I can see why some people would find it REALLY weird of counterintuitive tho.

Guess I really am a simple dude.

I been doing a online course and they provide a exercise to do in Jupyter notebook.

I notice that the exercise they wanted us to do this time is too complex. After finishing the exercise I submit it for review.

I thought I am doing an exercise but I had actually done an assignment and submit it :(

Lucky I pass that assignment. It is the worst misunderstanding I have done during my student life.

How long has GitHub been rendering Jupyter notebook files? That's slick.

!rant

Started learning Python today, whats ur opinion on Jupyter? I thought it to be quite nice:)

Also, Python 2.7 or 3? Im using 2.7 right now. Is the transition from 2.7 to three all that big?

Why the fuck isnt pythons tabbed loop thing something that's configurable?

So many things to like about python, but is there someone on this planet who actually likes this feature?

Trying to use it in Jupyter notebook (browser) is a nightmare, because tab will focus on next ui element.

Or am i missing something?

Rant:
My jupyter notebook has outgrown itself on some real world trading data analysis and its becoming a pain to add to (further) and share.
Need to find better alternatives, web apps where are you?

But i know nothing about it. Learning curve ahead!

Requirements:
I've 7 interactive dashboard plots (from some data) in jupyter-notebook.
- It'd be nicer to have a web app to use them without running notebook from a different location.
- Or running notebooks remotely (running as daemons on host machine).
Any suggestions for a starter ?
rant before requirements, coz rants lead to better requirements.

if rant++:
make_requirements(what_something)
do_work(that_something)

Today I was meeting with a researcher in my department so that I could show him how the software I developed works. He graduated from a really good university in electronic engineering, with 100/100 I think, and he can manage to copy&paste some python code. So I didn't expect what happened today.
Guy: 'So I have to give to your program as input this python file which contains a function you need to call, right?'
Me: 'Yep, I mean, that is a jupyter notebook, I need a text file containing only the function which is in that notebook'
Guy: *Downloads the notebook, tries to feed the notebook file into my program*
Me: 'Wait, don't, there probably is a lot of junk related to jupyter notebook, try opening it in notepad++'
Guy: *Opens file, sees a lot of junk text*
Me: 'Yeah, I thought so, you need to save it as plain text or .py'
Guy: *Renames the notebook as .txt*
Me: *Shakes my head without him noticing*
'That won't work, the content of the file won't change like that...'

A jr dev was having an issue registering code with our data pipeline (prefect self hosted).

Turns out he's running vscode to launch a anaconda shell (didn't even know that was a thing) to launch jupyter notebook and running commands in the notebook (didn't know that was possible) all from Windows.

No it doesn't work. His environment configuration isn't right. I told him to just run Linux and get rid of all that nonsense.

Nothing is on git yet and were three weeks in! His code is full of hard coded absolute paths of files on his hard drive... He even had an example app to go buy, with a project layout to copy.

There's no helping some people

IPython is the epitome of the new and shiny. IPython, jupyter, lab, hub, server, galaxy. I think they're all different iterations of "UI around python repl".

And I love new and shiny.

I just finished refactoring the code of a single jupyter notebook into a decent structure with comments. It feels good.
Unfortunately I still have stuff to implement and it's 2:50am :(

Here I am, 3:18 am, maybe I won't sleep today either, I hope I do... I'm going on with my uni project, a data science project. I've been wasting hours trying to understand why the fUcK 2 dataframes give me substantially different performances when they fucking shouldn't, since they should be the fUcKing sAmE. But apparently pandas is making fun of me... it seems that if you do something like:
df=original_df.loc[:, [some_cols]]
and some columns in [some_cols] don't exist in original_df, pandas won't give a shit and create a NaN column, or 0 based on how many virgin leprechauns ate bananas for Thanksgiving.
Plus I'm fucking freezing, in this apartment the heating system turns off at 23:59, it makes sense if you're in the fucking bed where you'll be fucking warm.
I miss software development... I wanna finish this MSc as soon as possible.
And here I am, listening to post-rock, writing jupyter notebooks, trying to be fucking positive.
It's not like I hate data science (maybe?), but I'm burnout.
Maybe I'll rewatch another time the video of Mr Robot with the song Where Is My Mind.
See ya.

So… I’m in full stack bootcamp and don’t know much tricks of the trade as of yet… but, has anyone ever tried using “Jupyter-notebook” while using a libgit in a-Shell for iOS to edit cloned repository files from their phone?… it seems possible but I question its efficiency.

The only console which shows their feelings for you.............*sigh*

Julia sucks!
It has similar syntax to Python and it's messing with my Python's knowledge.
Thanks to my Image processing subject's professor who preferred Julia over Python, because it's faster! and then he uses a package called Pluto (similar to Jupyter) which makes running Julia code super frustrating.

I have a deadline on 16th September. But AmazonWS has decided its not to cooperate. How, seriously, how do I copy my python files onto the stupid ec2 server and import them in the jupyter notebook.......

My younger bro started Coding today and the first thing he is asked me was if Jupyter Notebook has a dark mode or not.

I just spent 6 hours trying to get JupyterHub working with Real-time collaboration.
Time. Fucking. Wasted.

Outdated or non-existent documentation. Weird conventions. Everything is just annoying.

Is it really just hard to push a complete product to production instead of an half-ass untested mess?

So apparently jupyter / ipython adds the current workdir to kernel library path, and it crashes if you happen to have a file named something like "tokenize.py" in your workdir because it gets prioritised over ipython's builtin module with the same name. What a great design for something which is specifically made to run isolated chunks of code, that it can't even properly isolate itself from the workdir.

So i have begun learning python and jupyter notebooks etc Do you have any advice for someone like me who's a react dev and trying to switch to data science?

I have recently come across jupyter notebooks it's pretty cool I'm wondering if it is something that I should be using a long-term? Or is there another tool I should look at but I am quite interested halving interactive notebooks?