Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "numpy"
2010: PHP, CSS, Vanilla JS, and a LAMP Server.
Ah, the simple life.
2016: Node.js, React, Vue, Angular, AngularJS, Polymer, Sass, Less, Gulp, Bower, Grunt.
I can't handle this, I'm shifting domains to Machine Learning.
2017: Numpy, Scipy, TensorFlow, Theano, Keras, Torch, CNNs, RNNs, GANs and LOTS AND LOTS OF MATH!
Okay, okay. Calm down there fella.
Python Tools to Get Started with Machine Learning
SciPy - the most fundamental library with essential packages such as NumPy, matplotlib, Pandas, and SymPy.
NumPy - gives you the ability to play with your data as 'arrays' using some powerful array functions and linear algebra functions. Very essential since most computing is done with arrays of numbers.
Matplotlib - to visualize data and model outputs using 2D plotting with some 3D functionality.
Pandas - a highly flexible package which introduces dataframes to Python, a type of in-memory data table. Makes it easy to understand the data's structure and provides easy to use SQL-like commands to play with the data.
SymPy - is a package used for symbolic mathematics and computer algebra.
StatsModels - commonly used package for statistical methods and algorithms.
Scikit-learn - Most popular and easy to understand library filled with machine learning algorithms. A good start for beginners and practitioners working with smaller data loads.
RPy2 - A cross between Python and R. Allows you to call R functions from within Python.
NLTK (Natural Language Toolkit) - this toolkit in Python has functions and methods for text analysis.13
Still trying to get over my embarrassing noob moment.
In my very first crash course lecture on Python at uni, we were told about having to import modules in order to do fun mathsy stuff in Python, the most important one being numpy.
I typed in the infamous 'from numpy import *', but I got an error - I'd fallen at the first hurdle. Frustrated at this point, I called the lecturer over and asked why it wasn't working.
"You've typed in 'numpty' rather than 'numpy'."
He started laughing uncontrollably whilst I died in embarrassment, and I'm pretty sure that to this day, whenever I pass him in the corridors, he literally sees me as THAT noob.
Finally program running perfectly 👍
But wait.... Lets add another feature...
New feature needs updated matplotlib...
Lost support with numpy😫🤔
Updated numpy 😫
Run program again...
Core dumped (segmentation fault)😶😶
Time to leave this planet10
Man....I keep up with this strange love hate relationship I have with Python....
Last night it was python that literally wrote my homework: define all possible equivalent partition tables with cause and effect analysis and boundary value checks for a program. The whole thing wrote itself and all I had to do was verify the inputs. Something that I was able to do using jupyter with pandas and numpy. On one hand, I despise the lack of static typing and use of whitespace as a block delimiter. On the other I cannot but help feeling a high level of gratitude over the language and its high availability and ease of use for this.
Sure, I could have used other tools, but this language has dominated hardcore in this regard enough to the point of not considering it being a crime against humanity.3
When you encounter a bug in your code while writing a test and you have absolutely no idea what's wrong...
...and then you see it's a type problem.
after beginning to learn numpy , i believe these packages were really created by some clown of a circus xD.
Everything is sooooo entertaining!!!
i learned java 3 years ago, but today if i had to crap out some crazy java or c++ expert , i would tell him about numpy's arrays...
Like , "hey dude python has this cool data structure in the numpy library called arrays, which can hold any datatypes in a kind of arraylist like fashion, and you can convert them from 1 dimensional to 1000 dimensional in just 1 line , and also do you know we can select any column with just array[position]? and even this position does not needs to be an integer, you can use a list , like array[[1,2,3]] will give you elements at array,array,array, and...."
wait, why is my friend dead ? xD
How deep does the rabbit hole go?
Problem: Convert numpy array containing an audio time series to a .wav file and save on disk
Me: pip install "stupid package"
Console: Can't pip, behind a proxy
Me: Finds workaround after several minutes
Conversion works, but audio file on disk doesn't work
Encoding Error only works with array of ints not floats
BUT I NEED IT TO BE FLOATS
Looks for another library
scikits.audiolab <- should work
Me: pip --proxy=myproxy:port install "this shit"
Command Line *spits back huge error*
Googles error <- You need to install this package with a .whl file
Me: Downloads .whl file <- pip install "filename".whl
Command Line: ERROR: scikits.audiolab-0.11.0-cp27-cp27m-win32.whl is not a supported wheel on this platform.
Googles Error <- Need to see supported file formats
Me: python -c "import pip; print(pip.pep425tags.get_supported())"
Console: AttributeError: module 'pip' has no attribute 'pep425tags'
Googles Error <- Use another command for pip v10
Me: python -c "import pip._internal; print(pip._internal.pep425tags.get_supported())"
Me: pip install "filename".whl
Me: *spends 30 minutes to find directory where I should paste .dll file*
Finds Directory (was hidden btw), pastes file
Me: Runs .py file
Console: from version import version as _version ModuleNotFoundError: No module named 'version'
Googles Error <- Fix is: "just comment out the import statement"
Unfortunately this shit still didn't work after two hours of debugging, lmao fuck this7
I love python, but I hate dealing with python dependencies, especially on Windows.
I was tinkering and researching with neural networks, so I wanted to try out pybrain. I wrote my project, with pybrain installed via pip, and tried to build it.
Oh, what's that? Pybrain doesn't work with python 3? Well I'll download the version that's supposed to. Oh, that version has a deprecated numpy api? Let me just install those other resources. Oh, that requires a broken module that has no publicly available source?
Let's try python 2. Oh, now that's working, I just need to export environment variables for some "bls source". Some quick Google searching and the only solution that would work is building a bunch of cywgin modules by hand. That's fine, I have an ubuntu partition.
An hour later I'm compiling FORTRAN dependencies on Ubuntu.
Coding time: 1 hour
Dependency time: 3 hours6
I told interns in my startup to code a GAN only using Numpy. I received 4 resignation letters the next day13
When I first began with Python I really missed the static typed checking from Java, I barely know anything about a returned object from a method and have to read the API extensively for every new library.
After a while I finally understand why Python is so powerful, the combination of dynamic typed language and rich default methods make the language unbeatable for your productivity.
While Java's Object only has toString(), hashCode(), equals() or clone(), Python's basic Class has every fucking method for every scenario I could ever image. No wonder that libraries like numpy or pandas work so well and fluidly.8
Okay! Got my numpy pdf, theano pdf and my theano deep learning pdf! It’s time to get reading for 1111111111111111111111111111111111111 hours. Wow! I’m really getting deep into “deep learning” learning! Ok, I’ll quit now...2
Sorry Google, you got it wrong this time ....
Oh my gosh, look at that function definition ...
Oh my gosh, look at that variable ...
Oh my gosh, look at that zone ...
Oh my gosh, look at that long ...
Oh my gosh, look at that short ...
Oh my gosh, look at that stop ... is more my style.10
Me using numpy.polyfit() to fit a function to some data. Blame polyfit for not being accurate, search the web to find any related problems, as the polynom doesn't fit at all. Polynom is of 2nd degree, polyfit becomes unstable at about degree 20?? Try Polynomial.fit() ... same results. How can they all be wrong???See a little typo in my code, calculating the polynom points. Fix it, everything works.... wasted 2 and a half weeks because of this error.🤦🤦
Reading through one of my posts I’ve realized how much ego programmers can actually have. Guys, some of you have already mastered or grasped more than just the foundations of the industry standard languages, as well as developed a very solid intuition behind some design patterns and a solid understanding of some frameworks and libraries, say NumPy, say React... we get it.
You don’t have to be such condescending assholes and be offended by some of the jokes we, programming beginners, make to release stress or just to have fun.
You already have some amazing developer and engineering skills. Do not ruin it with such a detrimental attitude; I make this post because I myself have made this mistake, and I still do to this day. But if what I’ve felt reading your comments is what non-programming people feel when around me, I wouldn’t be surprised if I found that some people hated me or just wanted to kill me.
I don’t know if this will get downvot’d or if more people think like this. But I needed to share this, even just as a reflection of my very own attitude.
Thank you for your time,
My main focus Is data analysis. Being a physics student I do quite a lot of this using the likes of numpy and matplotlib, or GNU Octave.6
Whatever the f is wrong with numpy devs!!!!
Like seriously bro....
I can't import the effing sklearn.decomposition to do some basic PCA and the best solution out there is to downgrade it to version 1.16.1. Like hell!!!!
Issue has been known since last year, but guess who cares effukers8
so, my friend was telling me, his ex texted him and asked, I'm quoting,
'How to install numpy and pandas in python?'
she wanted to try machine learning !!1
Right now, everything. I started at a Consulting firm because I expected many new problems to tackle, solutions to develop and generally to always have a fire burning underneath my ass but instead I always develop the same standard bullshit.
I miss the days in my old job when there was just a problem and the task to solve it. When I stared down giant amounts of data, just KNOWING that somewhere in that mess is some structure I could exploit and that short moment of inspiration when I finally pinpointed it. The rush of endorphins when the solution became clear and everything fell into place to form a beautiful pattern amidst the chaos test data, git commits and numpy arrays.
Now its just "Yeah, would you just write another selenium testsuite that throws out fail or pass and wastes all the information because the only reason I'm a testmanager is because I'm too incompetent to do anything else and not my passion for the field".
The constant, mind numbing repetition of always the same patterns where the occasional dynamic element that becomes stale is the highlight of my work week... I would have never thought that making good money with easy work would ever get me as close to depression as it did.6
Ideas I've had over the years that could pan out and be useful:
SMS-DB: Stands for SMS-Data Burst. Used to allow those with low cell signal or no data plan to transfer data between a phone and some client via the standard SMS text space. Would be slow, but would act kinda like dial-up over SMS (as mobile lines are compressed on all service levels, even LTE, so traditional dial-up wouldn't work!) I have a general idea on how packets would be laid out, but that's about it so far...
everything2PNG: Allows one to transpose any file's data into a PNG with a 3 byte per pixel (full color RGB), which allows for a "compression" of sorts (about 91, 93% on preliminary tests) AND allowing further, more efficient compression of the resulting file. (Plus... it's just kinda cool to see files transposed as PNGs.) I actually have a simple transposer to go to PNG, but can't yet go back. Large files (around 600MB) use upwards of 4GB with efficient paging and other optimizations via NumPy so far, so it's not *viable* yet, but it's coming along nicely.
RPi-GPIO Interconnection Bus: A master/slave or round robin method to allow for Raspberry Pis to communicate using GPIO, which can help free up network bandwidth in RPi cloud computing clusters. At most, this'd allow for 4 bits used for pushing to the GPIO "bus", and 4 bits used for pulling from the "bus". 8 pins total are usually unused minimum, so either 3 or 4 pins for upload, 3 or 4 for download, and potentially 1 or 2 for commands, general non-data communication, etc. I made a version of this concept using Round Robin for a client, but it was horribly slow. (I also don't have distribution rights for the code, so i'm working from scratch.) Definitely doable.
Fuck yeah ... I have uploaded my major computation file to S3 and create Lambdas from those files(includes numpy and pandas also) and now I have only routes and invoke strategies in my EC3 .. looking for cost reduction....
I have started to work on pandas and numpy , should I work on scipy and sklearn or should make a strong fundamentals on numpy and pandas?2
So, I'm using Vpython for my physics class.
The good thing is that I love that. The bad thing is that the last update to vpython was in 2015.
Well, I update my system yesterday and apparently one of the libraries got an API update and that broke Vpython and can't be used again T.T
I'm trying to fix the code at the moment ;-;1
#get unique images ids
images_ids = np.unique(images_df.index)
Dear developer who wrote the code I'm looking at,
thanks, I really need comments like this one. I was wandering lost in 1500 lines of code, looking for an explaination of what the actual fuck the code is doing, and there I see you, comment. It's not like I want to know what the hundreds of lines functions do, who cares about that. What I needed to know, what shed light on this dark forest, is what the numpy functions do, because as you certainly know dear developer, such functions are really hard to comprehend, lacking of documentation.
Who on earth decided, that float64 is a suitable default datatype for one-hot vectors in numpy?
That's what I deserve for relying on reasonable implicit behaviour1
I starting developing my skills to a pro level from 1 year and half from now. My skillset is focused on Backend Development + Data Science(Specially Deep Learning), some sort of Machine Learning Engineer. I fill my github with personal projects the last 5 months, and im currently working on a very exciting project that involves all of my skills, its about Developing and deploy a Deep Learning Model for Image Deblurring.
I started to look for work two months to now. I applied to dozens of jobs at startups, no response. I changed my strategy a bit, focusing on early stage startups that dont have infinite money for pay all that senior devs, nothing, not even that startups wish to have me in their teams. I even applied to 2 or 3 and claim to do the job for little payment, arguing im not going for money but experience, nothing. I never got a reply back, not an interview, the few that reach back(like 3, from 3 or 4 dozen of startups), was just for say their are not interested on me.
This is frustrating, what i do on my days is just push forward my personal projects without rest. I will be broke in a few months from now if i dont get a job, im still young, i have 21 years, but i dont have economic support from parents anymore(they are already broke). Truly dont know what to do. Currently my brother is helping me with the money, but he will broke in few months as i say.
The worst of all this case is that i feel capable of get things done, i have skills and i trust in myself. This is not about me having doubts about my skills, but about startups that dont care, they are not interested in me, and the other worst thing is that my profile is in high demand, at least on startups, they always seek for backend devs with Machine Learning knowledge. Im nothing for them, i only want to land that first job, but seems to be impossible.
For add to this situation, im from south america, Venezuela, and im only able to get a remote job, because in my country basically has no Tech Industry, just Agencies everywhere underpaying devs, that as extent, dont care about my profile too!!! this is ridiculous, not even that almost dead Agencies that contract devs for very little payment in my country are interested in me! As extra, my economic situation dont allows me to reallocate, i simple cant afford that. planning to do it, but after land some job for a few months. Anyways coronavirus seems to finally set remote work as the default, maybe this is not a huge factor right now.
I try to find job as freelancer, i check the freelancer sites(Freelancer, Guru and so on) every week more or less, but at least from what i see, there is no Backend-Only gigs for Python Devs, They always ask for Fullstack developers, and Machine Learning gigs i dont even mention them.
Maybe im missing something obvious, but feel incredible that someone that has skills is not capable of land even a freelancer job. Maybe im blind, or maybe im asking too much(I feel the latter is not the case). Or maybe im overestimating my self? i think around that time to time, but is not possible, i have knowledge of Rest/GraphQL APIs Development using frameworks like Flask or DJango(But i like Flask more than DJango, i feel awesome with its microframework approach). Familiarized with containerization and Docker. I can mention knowledge about SQL and DBs(PostgreSQL), ORMs(SQLAlchemy), Open Auth, CI/CD, Unit Testing, Git, Soft DevOps Skills, Design Patterns like MVC or MTV, Serverless Environments, Deep Learning Solutions, end to end: Data Gathering, Preprocessing, Data Analysis, Model Architecture Design, Training and Finetunning. Im familiarized with SotA techniques widely used now days, GANs, Transformers, Residual Networks, U-Nets, Sequence Data, Image Data or high Dimensional Data, Data Augmentation, Regularization, Dropout, All kind of loss functions and Non Linear functions. My toolset is based around Python, with Tensorflow as the main framework, supported by other libraries like pandas, numpy and other Data Science oriented utils.
I know lot of stuff, is not that enough for get a Junior Level underpaid job? truly dont get it, what is required for get a job? not even enough for get an interview?
I have some dev friends and everyone seems to be able to land jobs, why im not landing even an interview?
I will keep pushing my Dev career, is that or starve to death. But i will love to read your suggestions! how i can approach this?
i will leave here my relevant social presence:
Thanks in advance!11
I love serverless functions but I'm so tired of complex orchestration, juggling event parameters and now scipy+numpy+pandas exceeds size limit of 250MB..
Feel like cramming it all in a monolith like the geezers of yore and be done with it3
I want to manipulate CSV files with Python and I was using NumPy, what I want to do is 3 columns, with an undetermined number of rows, and I want to be able to remove, add and edit every value, this is my questions:
Should I use NumPy? (if yes, please tell me how, I've been searching on google and I couldn't find anything of help! If not, please tell me what I should use,)4