Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "data analysis"
WHY THE FUCKIDY FUCK DO PEOPLE THINK THAT JUST BECAUSE YOU ARE A DEVELOPER THEN YOU CAN DO EVERY FUCKING THING IN A SNAP?!
THIS ENTIRE SUMMER I WORKED FOR THIS MIDGET OF A FUCK AND THE IDEA WAS TO MAINLY DEVELOP AN IOS APP FOR THEIR PRODUCT. THAT ONE APP TRANSLATED TO THREE SEPARATE APPS AND KEEP IN MIND THAT I AM THE ONLY DEVELOPER THEY HAVE SO I HAVE TO DEVELOP IT, TEST IT AND DOCUMENT IT!! AND WHAT'S EVEN WORSE IS HE KEEPS GIVING ME DESIGNS AT THE LAST MINUTE SO I ALWAYS END UP HAVING TO CREATE NEW PROJECTS ALL THE TIME. WHEN A WEEK PASSES BY WITHOUT ANY UPDATE HE AND HIS FUCKED UP BRAIN CELLS GATHER A MEETING WITH HIS 'DONALD TRUMP HANDS-SIZED' BALLS TO ASK ME, 'WHY ARE YOU TAKING SO LONG? THESE ARE JUST THREE BUTTONS ON THIS VIEW?' MOTHERFUCKING COCKSUCKER!! GO GET YOUR MOM TO DO THIS WORK THEN IF IT'S JUST THREE BUTTONS. TO HIM EVERYTHING IN IOS WHICH INCORPORATES A TAP IS A FUCKING BUTTON! BUTTON THIS! BUTTON THAT! AND IT'S NOT LIKE HE HAS SIMPLE DESIGNS..NO.NO.NO.NO. THIS ASSHOLE-SHAPED-HEAD MUGGET DESIGNS SHIT WHICH REQUIRES ME TO HAVE TO DRAW A UIVIEW AS THE SHAPE OF A HUMAN BODY AND HEART. THEN ASIDE FROM THAT, JUST BECAUSE MY RESUME SAYS I MINORED IN MATH AND APPLIED MATH, HE SENDS ME A PAPER THAT EXPLAINS MATHEMATICAL CONCEPTS OF DATA ANALYSIS THEN WANTS TO MEET WITH ME TO DISCUSS THE SUGGESTED METHODS THEN IMPLEMENT THEM. AT THIS POINT I HAD ALREADY IMPLEMENTED AN ALGO FOR COUNTING THE NUMBER OF STEPS THAT WAS WAY WAY WAY MORE ACCURATE THAN THE SHIT THEY HAD IN THEIR CONTROLLER..
SO AS I AM ABOUT TO FINISH IMPLEMENTING JUST THE INITIAL 5 VIEWS OF THIS 'FINAL' APP, HE SERIOUSLY WALKS IN AND TELLS ME, SO I'M STARTING TO WORRY THAT WE'LL NOT MEET THE AUGUST DEADLINE SO I'M THINKING MAYBE YOU SHOULD START SWITCHING BETWEEN DEVELOPMENT. WORK ON IOS FOR 4 HOURS THEN SWITCH TO ANDROID FOR THE REMAINDER OF THE TIME. OH AND UHH IF YOU FEEL LIKE TAKING THE IMAC WITH YOU TO WORK FROM HOME, THAT'S FINE AS WELL AND I'LL BE AVAILABLE IN THE WEEKENDS AS WELL. IT WAS AT THIS MOMENT THAT I REALLY REALLY HOPED I WERE GAY! LIKE 'NO LUBE' STYLE KINDA GAY!! OH BTW AT SOME POINT HE HAD ME PROGRAM ONE OF THEIR CONTROLLERS, AND WAS ASKING IF I COULD START RESEARCHING MEANS OF WRITING AN SDK FOR THEM.
KEEP IN MIND THAT I'M AN INTERN WITH ONE YEAR IOS DEV EXPERIENCE.. THEN HE WANTS ME TO ENTIRELY START LEARNING ANDROID AND GIVE HIM TWO VERSIONS OF THIS THIRD APP IN TWO WEEKS.
HE CAN GO SUCK HIS OWN DICK WHILE GETTING FINGER FUCKED BY A FORK WITH A BLUE WHALE'S 6FT LONG DICK UP HIS MOUTH.
*** that felt good ****36
Each month my department compiles a 4M row 150 column data table for compliance with a federal agency. Before submitting, we check it against about 400 rules.
The existing system was simply 400 queries that ran in sequence, table-scanning 4M rows each time, taking upwards of 6 hours, which is a huge bottleneck, especially if you have to make changes and rerun. Plus the output was rather one-dimensional.
I built a proper normalized database and created a sort of rules engine, running all 400 rules in one table scan. Not only does it complete in 30 minutes, but the reports generate automatically, and the results can be filtered on several dimensions to aid with root-cause analysis.
Management was pleased.4
No it's not AI. YOU ARE RUNNING FUCKING SQL QUERIES AND CALLING IT AI!
No it's not AI. YOU ARE RUNNING SIMPLE DATA ANALYSIS MACROS AND FUNCTIONS IN EXCEL!
Stop labelling everything as AI, you attention and investment seeking morons! @&£$¢×xo##!29
Husband wanted to not go out after dinner to "go cuddle in bed." This apparently is code for him browsing mathmatica stack exchange and reading books on data analysis with R... 😂9
Smart India Hackathon: Horrible experience
Background:- Our task was to do load forecasting for a given area. Hourly energy consumption data for past 5 years was given to us.
One government official asks the following questions:-
1. Why are you using deep learning for the project? Why are you not doing data analysis?
2. Which neural network "algorithm" you are using? He wanted to ask which model we are using, but he didn't have a single clue about Neural Networks.
3. Why are you using libraries? Why not your own code?
Here comes the biggest one,
4. Why haven't you developed your own "algorithm" (again, he meant model)? All you have done is used sone library. Where is "novelty" in your project?
I just want to say that if you don't know anything about ML/AI, then don't comment anything about it. And worst thing was, he was not ready to accept the fact that for capturing temporal dependencies where underlying probability distribution ia unknown, deep learning performs much better than traditional data analysis techniques.
After hearing his first question, second one was not a surprise for us. We were expecting something like that. For a few moments, we were speechless. Then one of us started by showing neural network architecture. But after some time, he rudely repeated the same question, "where is the algorithm". We told him every fucking thing used in the project, ranging from RMSprop optimizer to Backpropagation through time algorithm to mean squared loss error function.
Then very calmly, he asked third question, why are you using libraries? That moron wanted us to write a whole fucking optimized library. We were speechless at this question. Finally, one of us told him the "obvious" answer. We were completely demotivated. But it didnt end here. The real question was waiting. At the end, after listening to all of us, he dropped the final bomb, WHY HAVE YOU USED A NEURAL NETWORK "ALGORITHM" WHICH HAS ALREADY BEEN IMPLEMENTED? WHY DIDN'T YOU MAKE YOU OWN "ALGORITHM"? We again stated the obvious answer that it takes atleast an year or two of continuous hardwork to develop a state of art algorithm, that too when gou build it on top of some existing "algorithm". After listening to this, he left. His final response was "Try to make a new "algorithm"".
Needless to say, we were completely demotivated after this evaluation. We all had worked too hard for this. And we had ability to explain each and every part of the project intuitively and mathematically, but he was not even ready to listen.
Now, all of us are sitting aimlessly, waiting for Hackathon to end.😢😢😢😢😢26
Perhaps not "best", but certainly most amusing, so what the heck!
Years ago as an intern, I applied to a large pharmaceutical company. On part of the application form, you had to enter the code of the department you were applying to.
What I *should have* put down was "IT", which is the department that houses all their devs. However, I didn't actually read any of what the codes meant, assumed that was the department for helping people with how to mail merge, and put down "COMPSCI" instead. This was computational sciences - loosely summarised as computational data analysis on various druggable molecules.
I do *not* have any sort of biology or chemistry background, so the interview was rather... interesting, and I muddled through on the basis of getting some more interview practice assuming it was a no go.
To my amazement, got a phone call saying that they'd been thinking they wanted someone more technical on the team, and despite my lack of scientific experience they thought I'd be a good fit. I was unsure as to whether I should accept for a while, but then decided to just go for it - and had a fantastic internship there, working on a great variety of stuff, and learning tons all under a supervisor who I'm still in touch with to this day.
tl;dr - Applied for the wrong job. Coincidentally got it anyway, and miraculously had a fantastic year working there.8
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...7
One comment from @Fast-Nop made me remember something I had promised myself not to. Specifically the USB thing.
So there I was, Lieutenant Jr at a warship (not the one my previous rants refer to), my main duties as navigation officer, and secondary (and unofficial) tech support and all-around "computer guy".
Those of you who don't know what horrors this demonic brand pertains to, I envy you. But I digress. In the ship, we had Ethernet cabling and switches, but no DHCP, no server, not a thing. My proposition was shot down by the CO within 2 minutes. Yet, we had a curious "network". As my fellow... colleagues had invented, we had something akin to token ring, but instead of tokens, we had low-rank personnel running around with USB sticks, and as for "rings", well, anyone could snatch up a USB-carrier and load his data and instructions to the "token". What on earth could go wrong with that system?
We got 1 USB infected with a malware from a nearby ship - I still don't know how. Said malware did the following observable actions(yes, I did some malware analysis - As I said before, I am not paid enough):
- Move the contents on any writeable media to a folder with empty (or space) name on that medium. Windows didn't show that folder, so it became "invisible" - linux/mac showed it just fine
- It created a shortcut on the root folder of said medium, right to the malware. Executing the shortcut executed the malware and opened a new window with the "hidden" folder.
Childishly simple, right? If only you knew. If only you knew the horrors, the loss of faith in humanity (which is really bad when you have access to munitions, explosives and heavy weaponry).
People executed the malware ON PURPOSE. Some actually DISABLED their AV to "access their files". I ran amok for an entire WEEK to try to keep this contained. But... I underestimated the USB-token-ring-whatever protocol's speed and the strength of a user's stupidity. PCs that I cleaned got infected AGAIN within HOURS.
I had to address the CO to order total shutdown, USB and PC turnover to me. I spent the most fun weekend cleaning 20-30 PCs and 9 USBs. What fun!
What fun, morons. Now I'll have nightmares of those days again.9
I’m so mad I’m fighting back anger tears. This is a long rant and I apologize but I’m so freaking mad.
So a few weeks ago I was asked by my lead staff person to do a data analysis project for the director of our dept. It was a pretty big project, spanning thousands of users. I was excited because I love this sort of thing and I really don’t have anything else to do. Well I don’t have access to the dataset, so I had to get it from my lead and he said he’d do it when he had a chance. Three days later he hadn’t given it to me yet. I approach him and he follows me to my desk, gives me his login and password to login to the secure freaking database, then has me clone it and put it on my computer.
So, I start working on it. It took me about six hours to clean the database, 2 to set up the parameters and plan of attack, and two or three to visualize the data. I realized about halfway through that my lead wasn’t sure about the parameters of the analysis, and I mentioned to him that the director had asked for more information than what he was having me do. He tells me he will speak with director.
So, our director is never there, so I give my lead about a week to speak with her, in the mean time I finish the project to the specifications that the director gave. I even included notes about information that I would need to make more accurate predictions, to draw conclusions, etc. It was really well documented.
Finally, exasperated, and with the project finished but just sitting on my computer for a week, I approached my director on a Saturday when I was working overtime. She confirmed that I needed to what she said in the project specs (duh), and also mentioned she needed a bigger data set than what I was working with if we had one. She told me to speak to my lead on Monday about this, but said that my work looked great.
Monday came and my lead wasn’t there so I spoke with my supervisor and she said that what I was using was the entire dataset, and that my work looked great and I could just send it off. So, at this point 2/3 of my bosses have seen the project, reviewed it, told me it was great, and confirmed that I was doing the right thing.
I sent it off to the director to disseminate to the appropriate people. Again, she looked at it and said it was great.
A week later (today) one of the people that the project was sent to approaches me and tells me that i did a great job and thank you so much for blah blah blah. She then asks me if the dataset I used included blahblah, and I said no, that I used what was given to me but that I’d be happy to go in and fix it if given the necessary data.
She tells me, “yeah the director was under the impression that these numbers were all about blahblah, so I think there was some kind of misunderstanding.” And then implied that I would not be the one fixing the mistake.
I’m being taken off of the project for two reasons: 1. it took to long to get the project out in the first place,
2. It didn’t even answer the questions that they needed answered.
I fucking told them in the notes and ALL THROUGH THE VISUALIZATIONS that I needed additional data to compare these things I’m so fucking mad. I’m so mad.15
UPDATE: devRant Trans-Oceanic Journey Community Project
It was a mere 12 days ago that I asked the question; 'Could devRanters, as a community, build a 21st Century Technology-Laden ‘devRant devie-Stressball-in-a-Bottle’ and send it on a journey across the Atlantic ocean?
I am thrilled to report that devRanters enthusiastically accepted this difficult challenge. A core team quickly formed and a tremendous amount of research and progress has been made in a short period of time. I want to give you a high level-flavor of what we are doing. Please keep in mind we still need your help. We welcome all develops to take part in this journey.
I want to give appreciation to the devRant Founders @dfox and @trogus. Without your support and sponsorship this project would not have been possible. devRant brought us together and it a reality. Devie journeying across the Ocean the Columbus sailed will stir the imagination of children and adults worldwide when we launch on May 1, 2017.
Some of the research and action items in progress:
- Slack and trello environments were created to capture research and foster discussion.
- A Stony Brook University Oceanography Professor suggested the Gulf Stream would be a good pathway across the ocean. We researched it very and agree. The Gulf Stream has been a trans-Atlantic conduit for hundreds of years. We are deciding whether to launch from Cape Hatteras, NC or the Virginia coast. Both have easy access to the rapid currents in the Gulf Stream.
- We are researching every detail of the Gulf Stream to make the journey easier and faster for devie. We have maps and a team member gathered valuable ideas reading a thorough book – ‘The Gulf Stream’.
- We decided on using a highly resilient plastic rather than glass for the bottle material. Plastic is much lighter, faster and glass breaks down more easily. The lightweight enclosure will allow us to take full advantage of waves and ample trade winds. We are still discussing the final design as we want to minimize friction and mimic the non-locomotion fish that migrate thousands of miles riding the Gulf Stream.
-The enclosure might be 3D printed unless we can locate a commercial solution. We have 3D specs and are speaking with some experts. There are advantages and dis-advantages to each solution.
- We will be using Iridiums' RockBLOCK two-way satellite technology to bounce lat-long coordinate pings off their 36 low-orbit satellites. The data will be analyzed by our devRant devie analysis software. IOS and Android public apps being built by the team will display devie's location throughout the journey in.
- Arduino will be used as the brains
- Multiple sensors including temperature and depth are being considered
-A project plan will be published to the team Friday 12/9. Sorry I am a few days late but adding some new ideas.
There are still a lot of challenges we must overcome and we will.
That’s all for now. I will send updates and all ideas / comments are valued.6
Two years ago I moved to Dublin with my wife (we met on tour while we were both working in music) as visa laws in the UK didn’t allow me to support the visa of a Russian national on a freelance artists salary.
After we came to Dublin I was playing a lot to pay rent (major rental crisis here), I play(ed) Double Bass which is a physically intensive instrument and through overworking caused a long term injury to my forearm which prevents me playing.
Luckily my wife was able to start working in Community Operations for the big tech companies here (not an amazing job and I want her to be able to stop).
Anyway, I was a bit stuck with what step to take next as my entire career had been driven by the passion to master an art that I was very committed to. It gave me joy and meaning.
I was working as hard as I could with a clear vision but no clear path available to get there, then by chance the opportunity came to study a Higher Diploma qualification in Data Science/Analysis (I have some experience handling music licensing for tech startups and an MA with components in music analysis, which I spun into a narrative). Seemed like a ‘smart’ thing to do to do pick up a ‘respectable’ qualification, if I can’t play any more.
The programme had a strong programming element and I really enjoyed that part. The heavy statistics/algebra element was difficult but as my Python programming improved, I was able to write and utilise codebase to streamline the work, and I started to pull ahead of the class. I put in more and more time to programming and studied personally far beyond the requirements of the programme (scored some of the highest academic grades I’ve ever achieved). I picked up a confident level of Bash, SQL, Cypher (Neo4j), proficiency with libraries like pandas, scikit-learn as well as R things like ggplot. I’m almost at the end of the course now and I’m currently lecturing evening classes at the university as a paid professional, teaching Graph Database theory and implementation of Neo4j using Python. I’m co-writing a thesis on Machine Learning in The Creative Process (with faculty members) to be published by the institute. My confidence in programming grew and grew and with that platform to lift me, I pulled away from the class further and further.
I felt lost for a while, but I’ve found my new passion. I feel the drive to master the craft, the desire to create, to refine and to explore.
I’m going to write a Thesis with a strong focus on programmatic implementation and then try and take a programming related position and build from there. I’m excited to become a professional in this field. It might take time and not be easy, but I’ve already mastered one craft in life to the highest levels of expertise (and tutored it for almost 10 years). I’m 30 now and no expert (yet), but am well beyond beginner. I know how to learn and self study effectively.
The future is exciting and I’ve discovered my new art! (I’m also performing live these days with ‘TidalCycles’! (Haskell pattern syntax for music performance).
Hey all! I’m new on devRant!12
more buzzword translations with a story (because the last one was pretty well liked):
"machine learning" -> an actual, smart thing, but you generally don't need any knowledge to use it as they're all libraries now
"a bitcoin" -> literally just a fucking number that everyone has
"powerful" -> it's umm… almost working (seriously i hate this word, it really has a meaning of null)
"hacking" -> watching a friend type in their facebook password with a black hoodie on, of course (courtesy of @GeaRSiX)
"cloud-based service" -> we have an extra commodore 64 and you can use it over the internet for an ever-increasing monthly fee
"analysis" -> two options: "it's not working" or "its close enough"
"stress-free workplace" -> working from home without pants
now for a short story:
a few days ago in code.org "apscp" class, we learnt about how to do "top down design" (of course, whatever works before for you was not in option in solving problems). we had to design a game, as the first "step" of "top down design," we had to identify three things we needed to do to make a game.
graphics is literally a png, but what the fuck do you expect for ai?
we have a game right? oh wait! its getting boring. let's just sprinkle some fucking artificial intelligence on it like i put salt on french fries.
this is complete bullshit.
also, one of my most hated commercials:
"iot data and ai from the cloud"
yeah please shut the fuck up
Unintentionally Hilarious joke at work yesterday.
We were doing some data analysis, and I had to dump some stuff into a table for my colleague. So I ran the script and went to the bathroom (no.2).
When I came back, they asked me if the dump is done. And I said without thinking: "I just went." 😂3
Python Tools to Get Started with Machine Learning
SciPy - the most fundamental library with essential packages such as NumPy, matplotlib, Pandas, and SymPy.
NumPy - gives you the ability to play with your data as 'arrays' using some powerful array functions and linear algebra functions. Very essential since most computing is done with arrays of numbers.
Matplotlib - to visualize data and model outputs using 2D plotting with some 3D functionality.
Pandas - a highly flexible package which introduces dataframes to Python, a type of in-memory data table. Makes it easy to understand the data's structure and provides easy to use SQL-like commands to play with the data.
SymPy - is a package used for symbolic mathematics and computer algebra.
StatsModels - commonly used package for statistical methods and algorithms.
Scikit-learn - Most popular and easy to understand library filled with machine learning algorithms. A good start for beginners and practitioners working with smaller data loads.
RPy2 - A cross between Python and R. Allows you to call R functions from within Python.
NLTK (Natural Language Toolkit) - this toolkit in Python has functions and methods for text analysis.13
FUCK THE RECRUITERS WHO ASK US TO MAKE AN ENTIRE PROJECT AS A CODE TEST.
Oh you need to scrape this website and then store the data in some DB. Apply sentimental analysis on the data set. On the UI, the user should be able to search the fields that were scraped from the website. Upon clicking it should consume a REST API which you have to create as well. Oh and also deploy it somewhere... Oh I almost forgot, make the UI look good. If you could submit it in one week, we will move towards further rounds if we find you fit enough.
YOU KNOW WHAT, FUCK YOU!
I can apply to 10 others companies in one week and get hired in half the effort than making this whole project for you which you are going to use it on your website YOU SADIST MOTHERFUCK
I CURSE YOUR COMPANY WITH THE ETERNITY OF JS CALLBACK HELL 😡😤😣9
When scrolling through free Udemy courses for fun, my gf says:
"Whaaaat? Data analysis with Panda and Python? Isn't that shit dangerous? Like they go in the jungle and stuff?"
(Disclaimer: She has no background in CS)5
Currently, a classmate and I are working on our technical thesis.
It is all about industry 4.0, IIoT, big data and stuff.
This week, we presented our interim results to our supervisor. He is very pleased with our work and made the following suggestion:
He thinks it would be awesome to publish our work on our own GitHub repository and make it open source because he is convinced that this thesis is able to kind of "set a new standard" in some specific fields of using big data analysis in production processes.
I guess I'm kind of proud :)4
Man, most memorable has to be the lead devops engineer from the first startup I worked at. My immediate team/friends called him Mr. DW - DW being short for Done and Working.
You see, Mr. DW was a brilliant devops engineer. He came up with excellent solutions to a lot of release, deployment, and data storage problems faced at the company (small genetics firm that ships servers with our analysis software on them). I am still very impressed by some of the solutions he came up with, and wish I had more time to study and learn about them before I left that company.
BUT - despite his brilliance, Mr. DW ALWAYS shipped broken stuff. For some reason this guy thinks that only testing a single happiest of happy path scenarios for whatever he is developing constitutes "everything will work as expected!" As soon as he said it was "done", but golly for him was it "done". By fucking God was that never the truth.
So, let me provide a basic example of how things would go:
my team: "Hey DW, we have a problem with X, can you fix this?"
DW: "Oh, sure. I bet it's a problem with <insert long explanations we don't care about we just want it fixed>"
my team: "....uhh, cool! Looking forward to the fix!"
... however long later...
DW: "OK, it's done. Here you go!"
my team: "Thanks! We'll get the fix into the processing pipelines"
... another short time later...
my team: "DW, this thing is broken. Look at all these failures"
DW: "How can that be? It was done! I tested it and it worked!"
my team: "Well, the failures say otherwise. How did you test?"
DW: "I just did <insert super basic thing>"
my team: "...... you know that's, like, not how things actually work for this part of the pipeline. right?"
DW: "..... But I thought it was XYZ?"
my team: "uhhhh, no, not even close. Can you please fix and let us know when it's done and working?"
DW: "... I'll fix it..."
And rinse and repeat the "it's done.. oh wait, it's broken" a good half dozen times on average. But, anyways, the birth of Mr. Done and Working - very often stuff was done, but rarely did it ever work!
I'm still friends with my team mates, and whenever we're talking and someone says something is done, we just have to ask if it's done AND working. We always get a laugh, sadly at the excuse of Mr. DW, but he dug his own hole in this regard.
Little cherry on top: So, the above happened with one of my friends. Mr. DW created installation media for one of our servers that was deployed in China. He tested it and "it was done!" Well, my friend flies out to China for on-site installation. He plugs the install medium in and goes for the install and it crashes and burns in a fire. Thankfully my friend knew the system well enough to be able to get everything installed and configured correctly minus the broken install media, but definitely the most insane example of "it's done!" but sure as he'll "it doesn't work!" we had from Mr. DW.2
This week I reached a major milestone in a Machine Learning/Music Analysis project that I've been working on for a long time!!
I'm really proud to launch 'The Harmonic Algorithm' as an open source project! It represents the evolution of something that's grown with me through two thesis' (initially in music analysis and later in creative computation) and has been a vessel for my passion in both Music and Computation/Machine Learning for a number of years.
For more info, detailed usage examples (with video clips) and installation instructions for anyone inclined to try it out, have a look at the GitHub repo for the project:
"The Harmonic Algorithm, written in Haskell and R, generates musical domain specific data inside user defined constraints then filters it down and deterministically ranks it using a tailored Markov Chain model trained on ingested musical data. This presents a unique tool in the hands of the composer or performer which can be used as a writing aid, analysis device, for instrumental study or even in live performance."1
Inmates are trying to take over the asylum again.
Got a message from the web team manager deeply concerned because since switching to the new logging framework, the site is significantly slower.
She provided no proof or any data to what 'significantly slower' means.
#1 The 'new logging' has been in place and logging for 5 years. We only recently depreciated the ILogger interface ('new' ILogger interface only has 1 method instead of 5)
#2 The 'old logging' was modified 5 years ago, so even if you were using the 'old' interface, the underlying implementation is still the same.
She tried to push the 'it wasn't this slow before' argument, so I decided to do some fact based analysis.
Knowing they deployed their logging changes couple of weeks ago, I opened up AppDynamics, looked at the average call time to Splunk (along with a few other http calls they are doing)
- caching services - 5ms
- splunk - 30ms
- Order Service - 350ms
- Product Data Service -525ms
Then I look at the data they are logging, for the month of June, over 5 million messages. At 30ms each, that's almost 42 hours spent logging errors...yes errors. Null reference exceptions, Argument exceptions, easily fixable stuff.
So far for the month of July (using the 'new' logging), almost 2.5 million errors. Pretty close so far with June's numbers.
My only suggestion was to fix the bugs in their code so they don't log so many errors.
Her response.."Can we have one of our developers review your logging code? We believe we can find ways to optimize the http requests"
Oh good Lord. I'm not a drinking man..but ...I might start.2
TLDR: crappy api + idiot ex client combo rant // devam si duška
I saw a lot of people bitching about APIs that don't return proper response codes and other stuff..
Well let me tell you a story. I used to work on a project where we had to do something like booking, but better..crossbreed with the Off&Away bidding site (which btw we had to rip off the .js stuff and reverse engineer the whole timer thingy), using free versions of everything..even though money wasn't an issue (what our client said). Same client decided to go with transhotel because it was sooooo gooood... OK? Why did noone heard of them then?
Anyhow, the api was xml based.. we had to send some xml that was validated against a schema, we received another that was supposed to be validated againts another schema.. and so on and so on..
The API docs were nonexistent.. What was there, was broken English or Spanish.. Even had some comments like Add This & that to chapter xy.. Of course that chapter didn't even exist yet. :( And the last documentation they had, was really really old..more than a year, with visible gaps, we got the validation schemas not even listed in the docs, let alone described properly.
Yaaay! And that was not everything.. besides wrong and missing data, the API itself caused the 500 server error whenever you were no longer authenticated.
Of course it didn't tell you that your session was dead.. Just pooof! Unhandled crap everywhere!
And the best part?! We handled that login after inspecting what the hell happened, but sent the notification to the company anyways.. We had a conf call, and sent numerous emails explaining to them what a 'try catch' is and how they should handle the not authenticated error <= BTW they should have had a handled xml response for that, we got the schema for it! But they didn't. Anyhow, after two agonizing days talking back and forth they at least set up the server to be available again after the horrified 500 error. Before, it even stopped responding until reset (don't ask me how they managed to do that).
Oh yeah, did I mention this was a worldwide renown company?! Where everybody spoke/wrote English?! Yup, they have more than 700 people there, of course they speak English! <= another one of my ex clients fabulous statements... making me wanna strangle him with his tie.. I told him I am not talking to them because no-one there understood/spoke English and it would be a waste of my time.. Guess who spent almost 3 hours to talk to someone who sounded like a stereotypical Indian support tech guy with a flue speaking Italian?! // no offence please for the referenced parties!!
So yeah, sadly I don't have SS of the fucked up documentation..and I cannot post more details (not sure if the NDA still holds even though they canceled the project).. Not that I care really.. not after I saw how the client would treat his customers..
Anywayz I found on the interwebz some proof that this shitty api existed..
picture + link: https://programmableweb.com/api/...
SubRant: the client was an idiot! Probably still is, but no longer my client..
Wanted to store the credit card info + cvc and owner info etc.. in our database.. for easier second payment, like on paypal (which he wanted me to totally customize the payment page of paypal, and if that wasn't possible to collect user data on our personalized payment page and then just send it over to paypal api, if possible in plaintext, he just didn't care as long as he got his personalized payment page) or sth.... I told the company owner that they are fucking retards if they think they can pull this off & that they will lose all their (potential) clients if they figure that out.. or god forbid someone hacked us and stole the data.. I think this shit is also against the law..
I think it goes without saying what happened next.. called him ignorant stupid fucktard to his face and told him I ain't doing that since our company didn't even had a certificate to store the last 4 numbers.. They heard my voice over the whole firm.. we had fish-tank like offices, so they could all see me yelling at the director..
Guess who got laid off due to not being needed anymore the next day?! It was the best day of my life..so far!! Never have I been happier to lose my job!!
P.S. all that crap + test + the whole backand for analysis, the whole crm + campaign emails etc.. the client wanted done in 6 months.. O.o
P.P.S. almost shat my pants when devRant notified my I cannot post and wanted to copy the message and then everything disappeard.. thank god I have written this in the n++ xD11
Spec'ing out a data analysis tool for several billion transactions and I got this when sending the final spec to the client.
Client: "I don't see why it takes so long, I downloaded this template from theme forest and edited it in dreamweaver, it only took 3 hours"2
First year: intro to programming, basic data structures and algos, parallel programming, databases and a project to finish it. Homework should be kept track of via some version control. Should also be some calculus and linear algebra.
Introduce more complex subjects such as programming paradigms, compilers and language theory, low level programming + logic design + basic processor design, logic for system verification, statistics and graph theory. Should also be a project with a company.
Advanced algos, datastructures and algorithm analysis. Intro to Computer and data security. Optional courses in graphics programming, machine learning, compilers and automata, embedded systems etc. ends with a big project that goes in depth into a CS subject, not a regular software project in java basically.4
If anyone has been keeping up with my data warehouse from hell stories, we're reaching the climax. Today I reached my breaking point and wrote a strongly worder email about the situation. I detailed 3 separate cases of violated referential integrity (this warehouse has no constraints) and a field pulling from THE WRONG FLIPPING TABLE. Each instance was detailed with the lying ER diagram, highlighted the violating key pairs, the dangers they posed, and how to fix it. Note that this is a financial document; a financial document with nondeterministic behavior because the previous contractors' laziness. I feel like the flipping harbinger of doom with a cardboard sign saying "the end is near" and keep having to self-validate that if I was to change anything about this code, **financial numbers would change**, names would swap, description codes would change, and because they're edge cases in a giant dataset, they'll be hard to find. My email included SQL queries returning values where integrity is violated 15+ times. There's legacy data just shoved in ignoring all constraints. There are misspellings where a new one was made instead of updating, leaving the pk the same.
Now I'd just put sorting and other algos, but the data is processed by a crystal report. It has no debugger. No analysis tools. 11 subreports. The thing takes an hour to run and 77k queries to the oracle backend. It's one of the most disgusting infrastructures I've ever seen. There's no other solution to this but to either move to a general programming language or get the contractor to fix the data warehouse. I feel like I've gotten nowhere trying to debug this for 2 months. Now that I've reached what's probably the root issue, the office beaucracy is resisting the idea of throwing out the fire hazard and keeping the good parts. The upper management wants to just install sprinklers, and I'm losing it.
What you are expected to learn in 3 years:
digital signal processing,
signal and control system,
NLP, data algorithm,
Java, C++, Python,
ASP.NET web development,
computer security ,
Android app development,
IOS app development,
3D game development,
introduction to DevOps,
how-to -fix- computer,
Project of being entrepreneur,
and 24 random unrelated subjects of your choices
This is a major called "computer engineering"4
Fuck you Intel.
Fucking admit that you're Hardware has a problem!
"Intel and other technology companies have been made aware of new security research describing software analysis methods that, when used for malicious purposes, have the potential to improperly gather sensitive data from computing devices that are operating as designed. Intel believes these exploits do not have the potential to corrupt, modify or delete data"
With Meltdown one process can fucking read everything that is in memory. Every password and every other sensible bit. Of course you can't change sensible data directly. You have to use the sensible data you gathered... Big fucking difference you dumb shits.
Meltown occurs because of hardware implemented speculative execution.
The solution is to fucking separate kernel- and user-adress space.
And you're saying that your hardware works how it should.
Shame on you.
I'm not saying that I don't tolerate mistakes like this. Shit happens.
But not having the balls to admit that it is because of the hardware makes me fucking angry.5
Full stack developer.
I know what it's supposed to mean, but I feel like it gives discredit to the devs who perfect their area (frontend, backend, db, infrastructure). It's, to me, like calling myself a chef because I can cook dinner..
The depth, analysis and customization of the domain to shape an api to a website is never appreciated. The finicle tweaks on the frontend to make those final touches. Then comes a brat who say they are full stack, and can do all those things. Bullshit. 99.9% of them have never done anything but move data through layers and present it.
Throw these wannabes an enterprise system with monoliths and microservices willy nelly, orchestrate that shit with a vertical slice nginx ssi with disaster recovery, horizontal scaling, domain modeling, version management, a busy little bus and events flowing all decimal points of 2pi. Then, if you fully master everything going on there, I believe you are full stack.
Otherwise you just scraped the surface of what complexities software development is about. Everyone who can read a tutorial can scrape together an "in-out" website. But if your db is looking the same as your api, your highest complexity is the alignment of an infobox, I will laugh loud at your full stack.
And if you told me in an interview that you are full stack, you'd better have 10+ years experience and a good list of failed and successful projects before I'd let you stay the next two minutes..1
Been at a startup for 2-3 weeks, hood some presentations about Linux and Foss stuff but they didn't have any work for anyone and as such didn't pay anything.
So a colleague was like "hey man, I seed you making those presentations and stuff but not getting anything back, I think I might got a job for you".
And now I do small data analysis at another company. And get money.
After you automate away task into a tool or program, do you tell your boss or keep it to yourself so they remain amazed at your efficiency and productivity?
And for data analysis ones, that you have godly data processing skills as you can somehow read a 1,000,000 line log file and find the root cause and other insights in an hour or minutes?5
i want to get my own social network up and running.
so far ive got -
login 100% securely
register (1000% securely)
view someone’s profile (10^7% securely)
to add -
scrypt (maybe bcrypt, however scrypt looks like the better option)
friend a user
track their every move (ill use facebooks and googles apis for that)
to describe my product -
call me when im a gazillionaire
but seriously, im making a social network and i hope its done by wk105 tbh3
Small company, sole engineer. Non-tech management. Increasingly fancy job titles despite working alone most of the time, with the promise of hiring someone (again) I can actually manage soon.
Backlog of projects/tasks is truly a mindfuck, with new things being added each week. This backlog will never ever get done, and nothing matters anyway because the next idea is "the future", all the time.
While I have influence on some aspects of decision making, it usually ends up being what the boss wants. Actively opposed a project because it's just too big of an undertaking, it was forced through anyway. I'm trying to keep the scope manageable as I'm building it now, and it's hard.
"It's the future, we absolutely have to do this. It will be the biggest thing we've ever done."
Boss's excitement then quickly faded since it's actually in development, now nobody really seems to want to know where it's at, or how it will all work. I need to scope it out, with the knowledge that many decisions boss signed off will be questioned when he actually looks at it. We now have even more "exciting" ideas of utter grandeur. Stuff that I can't even begin to comprehend the complexity of, while struggling to keep a self imposed deadline on the current one.
Every single morning we sit on Zoom for a "valuable" "catch-up". This is absolutely perfect for one thing: Completely destroying whatever drive and focus I have going into the day. Unrelated topics, marketing conversations, even more ideas, ideas for ideas sake, small problems blown out of proportion, the list goes on. I recently argued in detail why it should be scrapped or at least be optional to attend. No luck, it's "valuable".
Today a new idea was announced, and we absolutely have to do it ASAP because it can only be better than the current solution. I raise my concerns, saying it's not as easy as you make it out to be, we should properly think about it. Nope! We'll botch something to prove that it works... So you'll base your decision whether it's good on some half ass botch job that nobody really has the mental capacity to actually pay attention to. What a reliable way to measure!
"Our analytics data isn't useful enough to tell us the impact of things we do. We (you) have to fix this." Over the last 2 or so years, I've been pushing for an overhaul and expansion of our data analysis capabilities for exactly this reason. Integrating different data sources into a unified solution so we can easily see what we're doing, etc. Nope, never happened.
The new project idea which is based on wild assumptions is ALWAYS more important than the groundwork.
Now when I mentioned that this is what I wanted to do all along, it got brushed aside. "We don't need to do anything complicated, just fix this, add that, and it's done. It should be an easy thing to do. This is very important for our decision making." Fine, have it your way.
I'm officially burned out. It's so fucking hard to get myself to focus on my work for more than an hour or two. I started a side project, and even that effort is falling victim to my day-job-induced apathy.
I'm tempted to hand in my resignation without another offer on the table. I just need time to rediscover my passion, and go job hunting from that position, instead of the utter desperation of right now.
If you've read through all this rambling, kudos to you!8
I fixed my big data processing code, I think. If all works as planned, I'll wake up to some processed data that I can do some statistical analysis on... I hope...8
When someone doesn't send raw data with their analysis, I feel like they have something to hide ...1
For starters this is my first post, found devrant the other day you're all hilarious.
I hate math, I hate proofs. I'm in a class "Analysis of Algorithms" and I have understood and do understand the importance of optimizing algorithms and data structures and I understand the algorithms and data structures themselves. That being said, I'm fucking sick of math and proofs and all this bullshit that is probably pretty important but ugggghh, I guess I just have to push through, but writing this out helped.18
2020 seems to be the year of the "dev who has never seen scale."
TypeA -> "Here's a reasoned explanation for a change I think we should make. Here is the current deficiency analysis, here is the desired resolution, here is the course of action and all calculations leading to the resolution + data. This will have x,y,z beneficial result according to our operational metrics."
TypeD -> "Those were words. Why do you need that? Change is bad, learning is worse. This will just slow me down, development speed is all that matters; there is no chance that a poorly considered/factored/checked design could ever require a ground up rewrite or fuck us utterly in the long term. Why do you make my life harder? We could x -> y -> zBUTI haven't done the math and I really don't see the benefit in x, so z is pointless. What even is scale?"
The consequences of the war caused by the ever-widening gap between engineers and developers is low key terrifying.13
There was this project where a bunch a scripts had been running for three weeks analysing a bunch of fileshares. The project was in overrun and the analysis wasn't anywhere near done.
I was given complete isolation and a team of people who were instructed to do anything I needed. I had them replicate the data to as many machines as possible and I started scripting the analysis with some sample data. After half a day of collecting laptops, desktops and severs I transfered my scripts to those machines and ran the analysis in 5 hours.
I felt like I saved the project.
Last week I received an invitation to lead the development of a e-commerce redesign, replatforming and data migration. I was excited to work on it, and started the analysis and planning, glad to spend time focusing on quality. But Murphy's law is never asleep - this Monday, I was asked to speed things up and reduce a 4 month project to 1 week.
So I joined a course for big data analysis. And they setup a lab specifically for us. Pulling us away from the usual computer labs
AND GUESS WHAT THEY DON'T EVEN BLOODY HAVE MYSQL INSTALLED. THEY'VE CHARGED ME A FORTUNE AND THEY DIDN'T EVEN CARE TO INSTALL THE BASIC SOFTWARES AND ITS ALREADY BEEN 2 WEEKS WITH THE START OF THE COURSE AND NOTHING.
F**king hate this man!!!!10
Best: learned to code, started writing smart-home scripts for home automation and developed biologic and quantitative data analysis scripts in Python and R.
Worst: didn't get paid to develop them and haven't got enough experience for it to be more than a hobby.
During one of our visits at Konza City, Machakos county in Kenya, my team and I encountered a big problem accessing to viable water. Most times we enquired for water, we were handed a bottle of bought water. This for a day or few days would be affordable for some, but for a lifetime of a middle income person, it will be way too much expensive. Of ten people we encountered 8 complained of a proper mechanism to access to viable water. This to us was a very demanding problem, that needed to be sorted out immediately. Majority of the people were unable to conduct income generating activities such as farming because of the nature of the kind of water and its scarcity as well.
Such a scenario demands for an immediate way to solve this problem. Various ways have been put into practice to ensure sustainability of water conservation and management. However most of them have been futile on the aspect of sustainability. As part of our research we also considered to check out of the formal mechanisms put in place to ensure proper acquisition of water, and one of them we saw was tree planting, which was not sustainable at all, also some few piped water was being transported very long distances from the destinations, this however did not solve the immediate needs of the people.We found out that the area has a large body mass of salty water which was not viable for them to conduct any constructive activity. This was hint enough to help us find a way to curb this demanding challenge. Presence of salty water was the first step of our solution.
We came up with an IOT based system to help curb this problem. Our system entails purification of the salty water through electrolysis, the device is places at an area where the body mass of water is located, it drills for a suitable depth and allow the salty water to flow into it. Various sets of tanks and valves are situated next to it, these tanks acts as to contain the salty water temporarily. A high power source is then connected to each tank, this enable the separation of Chlorine ions from Hydrogen Ions by electrolysis through electrolysis, salt is then separated and allowed to flow from the lower chamber of the tanks, allowing clean water to from to the preceding tanks, the preceding tanks contains various chemicals to remove any remaining impurities. The whole entire process is managed by the action of sensors. Water alkalinity, turbidity and ph are monitored and relayed onto a mobile phone, this then follows a predictive analysis of the data history stored then makes up a decision to increase flow of water in the valves or to decrease its flow. This being a hot prone area, we opted to maximize harnessing of power through solar power, this power availability is almost perfect to provide us with at least 440V constant supply to facilitate faster electrolysis of the salty water.
Being a drought prone area, it was key that the outlet water should be cold and comfortable for consumers to use, so we also coupled our output chamber with cooling tanks, these tanks are managed via our mobile application, the information relayed from it in terms of temperature and humidity are sent to it. This information is key in helping us produce water at optimum states, enabling us to fully manage supply and input of the water from the water bodies.
By the use of natural language processing, we are able to automatically control flow and feeing of the valves to and fro using Voice, one could say “The output water is too hot”, and the system would respond by increasing the speed of the fans and making the tanks provide very cold water. Additional to this system, we have prepared short video tutorials and documents enlighting people on how to conserve water and maintain the optimum state of the green economy.
IBM/OPEN SOURCE TECHNOLOGIES
For a start, we have implemented our project using esp8266 microcontrollers, sensors, transducers and low payload containers to demonstrate our project. Previously we have used Google’s firebase cloud platform to ensure realtimeness of data to-and-fro relay to the mobile. This has proven workable for most cases, whether on a small scale or large scale, however we meet challenges such as change in the fingerprint keys that renders our device not workable, we intend to overcome this problem by moving to IBM bluemix platform.
We use C++ Programming language for our microcontrollers and sensor communication, in some cases we use Python programming language to process neuro-networks for our microcontrollers.
Any feedback conserning this project please?8
[story of your first dev project] - i really think there should be a headline like that for wk rants
Anyways, it was a while back, le college teacher approached my friend and me asking if we wanted to do a project. We said sure, it was a medium sized data analysis project. We got the specs with a lot of formulas, basically implement them all and make a web frontend, thats it. Took like a year but we did it. Few months later teacher is furious because the calculations didnt give him data that he expected (by expect i mean he thought that a distribution formula would accurately yield 200+ data fields from around 4) and blamed it on us. Not the retard other professor who fucked up half of the formulas. Ok.1
A time I (almost) screamed at co-worker?
Too many times to keep up with.
Majority of time its code like ..
using (var connection = new SqlConnection(connectionString))
// data access code that does stuff
catch (Exception e)
// Various ways of dealing with the error such as ..
ShowMessage("An error occured.");
// or do nothing.
Range of excuses
- Users can't do anything about the error, so why do or show them anything?
- I'll fix the errors later
- Handling the errors were not in the end-user specification. If you want it, you'll have to perform a cost/benefit analysis, get the changes approved by the board in writing, placed in the project priority queue ...etc..etc
- I don't know.
- Users were tired of seeing database timeout errors, deadlocks, primary key violations, etc, so I fixed the problem.
On my tip of my tongue are rages of ..
"I'm going to trade you for a donkey, and shoot the donkey!"
"You are about as useful as a sack full of possum heads."
I haven't cast those stones (yet). I'll eventually run across my code that looks exactly like that.1
I used to be a big security guy, not allowing stuff like most of the social media, not bringing my phone anywhere, carrying a RPi tablet for privacy reasons. Very Stallman stuff.
Recently I noticed that I don't care so much.. I see these things as opportunities, for instance Microsoft products could be benefitial for job opportunities, I have some workout sessions on my phone.
I could restrict myself... but is it worth it just to decline some capitalist/politician's row in a dataset for analysis?
But then again I feel as a society I think we should either do this or request this data to be distributed to us as well.
Should you be playing a game of cards, when the enemy can see your hand? What do u think?6
I need to estimate how much ram and CPUs my team will need next year for our apps... That have yet to be built.
We load a lot of data feeds with batch processes running on a few large machines, some can use like 30 GB RAM at times...) which should be a lot less if we get the data real time I hope...
But wondering how to estimate well... I sorta did a worse case analysis where I just multiply and sum # CPUs/memory* nodes*approx apps...
Comes out to be like 600 CPUs and 800 RAM... So wondering if that's ok...
RAM is ok but # of CPUs is way higher bc now all the apps basically run on their own machines...16
So we’ve taken over from a project team that disbanded... read: “cut their contracts because fuck this, I can earn more working for better people”.
Me and one other guy have been tasked with saving this heap of shit.
Obviously the project guys left saying “it’s nearly done, just this one feature”. Because cut contracts are easier to deal with if “everything is almost done”.
We jump on and find that’s not the case at all... this thing, is a beast, a big old stats analysis program... so we’re like “cool, let’s see what’s going o...OH MY GOD”.
The “recalculation” function was core to this POS. The contractors had done it in C# through entity framework... it took 24 hours to run, over a reasonably small data set that was due to double every 2-5 years.
So... here’s the deal, it ran over night.... then failed. And no cunt had noticed. Entity framework “can’t commit because I’m muddled up as fuck, did you really just put the whole db in EF in memory to work with it?” Exception.
Que 6 months of me and my lead doing the job properly.
Anyway, the failure: I ended up in Hospital again with a Crohn’s flare up... about 5 months in.
Fuckall to do with all this nonsense I just wanted to tell a story. it was an interesting/fun project to fix and my lead was a legend... so happy days.
Similar story, different set of contracted devs... they’d been defining requirements with the business users using the term “Risk” which the business users knew as a group of risks.
The domain model had been written RiskGroup<>—
I made a script that performs some analysis on data from a trading card game tournament and I would appreciate your input7
So a team at on-site sent a OOM(Out of Memory) issue in our morning.
Everyone analysed the issue as being code issue since we were bringing too much data into the runtime process. The analysis was done on the heap dumps. The number of records reported by user were 1k. At the end of the day it turns the number of records were actually 100k+.
Why do people jump.to conclusions without thinking about the obvious. :-(1
never before have I been happy to be asked to work overtime, but for once, fuck yeah...
Bit of back story, I am tech lead on a massive project that has been run like a complete shit show, the PM who also happens to be the brains behind the project seems to think we are miracle workers and for the first 9/10 months of the project would make significant, like delete a weeks worth of code and start over changes, 3-5 times per week. There are features for the v1 release that have been built in excess of 5 times. I have been saying since October that even without all his constant changes, we will NOT make the deadline, and naturally as is part of my job I argued against every unnecessary feature he tried to implement, eventually he pulled me into a meeting to tell me how much he values my opinion, I need to stop arguing with him and he does not want to work with yes men (I have a rant about that convo already).
I believe our CEO finally started smelling a rat as he insisted on joining our daily stand-ups, during which said PM scripted some lovely stories to disguise the fuckup we are in, and since has assigned another PM to take over and do proper project management and risk analysis.
That is where the email comes in, a lot of the work assigned to me will miss the deadline by a month, honestly I am impressed that it is by so little and so few people will not be missing it, but anyway, he probably spun a few stories there too.
So I spent part of the work compiling the most perfect surgical response as not not actively throw him under the bu, but create a quite a few questions that they hopefully as, as himself and the CEO where cc'd into the mail.
And the jist is, the deadline itself was still impossible and 8 of the 10 tasks assigned to be have ZERO back-end whatsoever, and those tasks are about 80/90% integration to said non-existent back-end, some of those services and data structures have not even been planned yet and we are a week past the deadline and 3 weeks from the just as useless extension.
I want to do something data-science-y.
Gimme project ideas, and where can I get the data for it?
Also, not looking for machine learning, just basic data analysis stuff.
i was learning neural networks, started with keras and was on the first tutorial where they started by importing pandas
so i switched to learning data analysis using pandas in Python where they started by importing matplotlib and i realized data visualization is also important and now I'm reading matplotlib docs...🙄11
My boss is being a stupid cunt. To give you a background we were facing issues with our Collections system. First week December 2019, I and a colleague of mine came up with a new efficient collections architecture. My colleague and I started to Code and create automation scripts mid December and completed it in First week of Jan 2020. This PoC version was supposed to be just between the Dev team(App Dev and Back end, also one from the Ops side to verify the data). I did not receive any feedback on the actual collections system and the data integrity but during this time all they’ve done is take meetings with no real outcome. I raised this and the only email I got is data is looking fine when I know it is not.Now in First week of Feb, he is stressing us to go ahead and deploy the architecture in Production and we have not done any Code Review, Static Code analysis, any real tests on Code and deployment scripts. Have not discussed any metrics for our dashboard and alerting. I have no idea how to handle this cunt. I have even asked for resources to atleast productionalize the code and move ahead the deployment and still no out come. I’ll go in a meeting with him in an hour, I will be very blunt and tell him that whatever he is doing is a foolish way and maybe resign in couple of weeks6
The futility of working in a bank, or probably any other big corporation.
I was working in a small dev team that made improvements to the applications and processes of the department we were in. I was given the project of creating a small data entry system to track some communications. The comms were currently stored in a spreadsheet and had no validation which made any analysis of the data difficult.
Boring but easy, should have taken about a week.
We had 3 meetings just to define requirements, they took about a month to send me their shitty little spreadsheet so I could actually see what they were recording. Over the next 2 months they kept changing what they wanted. Finally got it finished just before I went on maternity leave.
Went off thinking, that I’ll never have to deal with that clusterfuck again.
Had a baby, went back 9 months later.
First thing I got given was that shitty project back. They hadn’t even tested it - in 9 fucking months! I transferred departments not long after. I have no idea if it ever got done.
And that is on my list of reasons as to why I’ll never work in a large corporate again.3
I think another intriguing job asides programming is engineering (*for some*). A week has past and I've been on the hike assisting my beloved brother on his contracted engineering job while I am less occupied. The job is based on 🗼Tower analysis and It's quite risky as you'd have to climb up to 56 meters high just to take readings of antennas, and fix some other stuffs. The only thing I find intriguing about this job is his love for it, funny enough he also thinks I love the job too and I guess I'm guilty for his thoughts (*Sorry bro, I love the job for you not me*).
With my little experience so far on my *new brotherly job* I noticed the most hectic task isn't going up and down the tower taking readings but at the end of all operations, he'll have to gather the values and snapshots he took while on the tower to prepare reports on msword & excel for the other buttwags at the office (or home I guess)
then archive and sends via mail. Seeing this lengthy process I was forced to ask why he wasn't using any reporting tool like Jotforms or any other equivalent and I was willing to look up some recommendations for him, his reply was: "I'm already used to this form of reporting, its what I was trained with and what the company provided, nevertheless a friend of mine suggested something of such weeks back but I would have to pay monthly fee for its usage which is quite on the high side and I don't think I'd prefer that."
Sounds convincing but not enough, okay here is another deal: You use an android phone right? and at my office we work on system automation (*basically does not know what I do for a living probably thinks I'm a hacker the illegal one*), how about i design you an android app for you to capture the tower data and a PC software for you to auto generate the msword & excel reports, I can get this ready for you in less than 5 nights (*I've got less task on my desk, and was willing to take the timeout to prepare the solution that he needed, all I needed to hear for a kick start was an "Okay" just to be sure he wants it*) I suggested and re-assured but up to this point he still declined my offer and is willing to stick with his current reporting pattern (*Me died*).1
Time to switch to offline and hide in some dark corner to get work done. Tired of all the IM’s and coming over to my desk from 1 person for “critical” work. If they’re all critical then none of them are truly critical. If you sit on the data for 2 months, and then today is the day it becomes critical and the compliance issue is because of your ineptitude then its a you problem not an IT problem. Then on top of that you submit your data to be loaded in the incorrect request form and spreadsheet format you can go fuck yourself asking this be done in an hour. It could be done in 15 minutes if you had it in the correct format as specified in the 20 meetings over the past year which removed all manual analysis and automated the entire process you idiot. Now I have to get it into the correct format in that hour so I don’t have to do the analysis for you.
I have other things to do besides your etl tickets, like finding the actual problems in our actual critical applications. You know the ones where the VP’s of this giant corporation start calling if they go down.
Sorry for the rambling guys.
Bootcamps get you up and running in coding quickly. If you are a programmer, companies are only interested on how quickly, error free and cheaply you produce marketable output. Bootcamps enable this.
More or less you are not more than a former assembly line worker putting parts on a car platform. Your value is not very high as you may be exchanged at any time at their will.
Nevertheless, you can earn money quickly. You trade in your youth and time which might be a dead end in the long-term. Trends go to machine learning, artificial intelligence. They will not need Bootcamp people and code workers.
It is better you set up Bootcamps and sell them versus absolving this. Like selling shovels during the gold rush, but not working in the mud of Alaska by yourself.
Your choice is: Making quick money, which fades anyway; or striving for the long-term future proof career.
C/S degrees from Technical Universities of reputation give to you the right direction under a strategic consideration. Companies which pay well, or freelancing with a solid acknowledged background, will always look for top graduates. People from Bootcamps are just OK for hammering assembly line coding. Even worse with SCRUM in one noisy room under enormous team server pressure controls, counting your lines of code per minute, with pale people all around. And groups of controllers never acknowledging nor trusting your work.
To acquire a serious degree, a Bachelor is nothing. Here, in INDIA, Bachelor now is what a former high school grade was. You must carry a diploma or Masters degree combined with internships at big companies with high brand recognition. This will require 4–6 years of your lifetime. You can support this financially by working part-time freelancing as making some projects front- or back-end web, data analysis and else.
Bootcamp people will lose in the long-term. They are the modern cannon fudder of software production.
It is your choice. Personally, I would never do Bootcamps. Quality and sustainability require time, deep studies and devotion.
One problem for CS education is that the salaries for academia are so low compared to industry that if someone is even vaguely competent, they can at least double their income by working a 'real' job. Now this may be different at higher levels of colleges but generally those folks are such bullshitters they wouldn't last outside of academia.
As what to improve?
Depends if it's a research or vocational course.
Basic knowledge of algorithms, runtime analysis (O notation) and some data structures and you're an instahire.
For research, go wild. Deep dive into the math, algorithmic side. Read up lots of research papers. Try out different programming paradigms. You would aim for a career in academia, AI, quant finance etc...2
Today I had a full-day job interview for a junior data scientist position.
First I met the team which was only like half of everyone because apparently everyone was gone on Fridays. However the few there were really nice.
First task is to do some basic data analysis stuff even though I already spent a week on the coding challenge and sent them all my code/tasks. I log into my machine and create a new virtual environment but can't for the life of me figure out how to use the command line in windows to install packages. Turns out there is some problem with their proxy and they have to log me in on that. Then I am struggling on the keyboard because it's for a language different that my mother tongue and it takes me 3x as long to so the most simple things. All my shortcuts are out the window. Haven't a hard time typing parentheses and brackets. Start freaking out and have a panic attack mid task. I'm sweating bullets. I didn't even make it to the simple visualization tasks much less the models at the end. Time gets called and we all go to lunch and I'm freaking out on the inside the entire time. Angry at myself because I know I am better and just couldn't think.
After lunch I present my code and results from a coding challenge I did weeks prior. People from other teams get invited and I end up getting grilled for 2 hours by 15 people. Questions are flying in from all sides. They ask me almost everything I know about machine learning and some more. Under stress I forgot the name of the optimizer I used and couldn't answer some easy stuff because my mind was racing.
Right now I am on the train home and my body physically hurts. I am disappointed with myself and wish I could have shown up better. Never really froze up like this before.2
"We are applying deep learning, NLP, machine learning, Big Data frameworks and other technologies to produce outstanding fintech products in areas of robo advisors, stocks and cryptocurrencies analysis, digital assistants, prediction of customer behavior, deep learning analysis of alternative data (satellite images) and other areas."
I code. I'm not a network or security admin. Did you even read my resume before hiring me? It says "math degree then data analysis then programming degree" Subnets, firewalls and security certs are not my areas of expertise. Given enough time, could I stumble through? Probably. But I'd probably come up with something worthy of mocking on networkadminrant.
(Day 1 of Database Class)
Database Prof - "Your final project is a program or app that deals with big data and showcases data analysis. Make something that can be used by consumers and has real-world application."
(After the Day of Showcasing Projects)
Database Prof -
(What he should have said on Day 1) - "Make something that makes me laugh with added data."
So I'm in a scenario I'm uncomfortable, need some encouragement from fellow devRanters. (Looong post)
I've been working at this startup for about 10mths (since I graduated). They have been really good to me since the start, and overlooked some fuck ups I did at first.
But now I've been way more experienced , picked it up really quick. And I've basically redesigned several of their admin solutions and data products. Also, I'm basically their entire data analysis team now. I do backend (node, PHP, MySQL) and analysis for them (stats, deep learning, python, big data packaging for clients).
But seeing as I've moved in their company, and have been consulted on several major decisions, as well as built a really good relationship with some of their clients. I still haven't seen a raise, moreover I've been told that I'm expected to work from 8am to 5:30pm (9.5hrs no overtime pay). Which really pisses me off, since I know I'm worth more than what I'm paid (about 40k a year).
My brother (who's also a dev) suggested to tell them that I'm not happy at work due to this. And quit if they don't react well.
How should i bring this up? Should I really quit? This is all new grounds.6
me: next week i'll be on vacations
boss: cool, but i need few things for this week, ... are urgent!
me: ok, tell me, what do you need?
boss: a new web site, a new dashboard for data analysis and a new system for data mining for facebook+google analytics
my mind: f*ck u, i want my vacations!
me: ok, i'll talk with the designer about the new web site and the other things ...
I currently work in data analysis.
Yesterday one of the colleagues working in a different department came to me asking if I could fix his malfunctioning headphone set :/3
I work with statistics/data analysis and web development. I study these subjects for almost a decade and now I have 4 years of practical experience.
This information is on my LinkedIn profile and from time to time tech recruiters contact me wanting to have an interview. I always accept because I find it a great way to practice interviews and talking in English, as it isn't my native language.
A remark that I always make to my colleagues wanting to start doing data analysis related work is that it may seem similar to development, but it's not. When you develop, your code work or not. It may be ugly, it may be full of security problems, but you almost always have a clear indication if things are functioning. It's possible to more or less correlate experience using a programming language with knowing how to develop.
Data science is different. You have to know what you are doing because the code will run even if you are doing something totally wrong. You have to know how to interpret the results and judge if they make sense. For this the mathematics and theory behind is as important as the programming language you use.
Ok, so I go to my first interview for a data science position. Then I discover that I will be interview by... a psychologist. A particularly old one. Yeah. Great start.
She proceeds to go through the most boring checklist of questions I ever saw. The first one? "Do you know Python?". At this point I'm questioning myself why I agreed to be interviewed. A few minutes later, a super cringy one: "Can you tell me an example of your amazing analytics skills?". I then proceed to explain what I wrote in the last two paragraphs to her. At this point is clear that she has no idea of what data science is and the company probably googled what they should expect from a candidate.
20 minutes later and the interview is over. A few days later I receive an email saying that I was not selected to continue with the recruitment process because I don't have enough experience.
In summary: an old psychologist with no idea on how data science works says I don't have experience on the subject based on a checklist that they probably google. The interview lasted less than 30 minutes.
Two weeks later another company interviews me, I gave basically the same answers and they absolutely liked what they heard. Since that day I stopped trying to understand what is expected from you on interviews.2
I was studying a lot the last year, i learned a lot about Machine Learning/Deep Learning, Data Gathering, Data Analysis, ETL, Model Architecture Design, Training, Fine Tuning, Backend Development, DataBases, API Development, ORMs, Rest, GraphQL, OAuth, CI/CD, Docker, Deployment to Production environments like Heroku, Git and more stuff i dont remember while writing this. I built and keep adding stuff to my Github Portafolio.
Im not able to get a job. I started looking for jobs as Data Scientists, no response never. I take a look at freelancer sites, nothing seems to fit my skills. And when there is a minimal fit, they always want a Full Stack Web Developer, i dont know Frontend Development, i dont like do it.
Dont know what to do or how to land any job.
My options aeems to be:
1.Learn Frontend Dev and work as Full Stack in underpaying freelance jobs
2.Keep applying to Remote-Only startups, but they still wants people with 3+ years of experience.
i cant work in my city, here are not any company startup hiring no one, we are 30 years in the past here.
What you do in my place?10
I just made my API in my laravel and I understand how it works! It may seem like not a lot, but I got from far.
Just came two years ago in this industry as I worked as a customer service agent for a hostingcompany. I entered a whole new area what I immediatly got into at the time. Mind I already was studying Biomedical labresearch at the same time and was the IT guy in the family. Well, think back then I was just googling and fixing shit most of the time.
I was 21 at the time and began to learn everything I could learn in my position and soon it was not enough and wanted to learn more by working parttime(study already asks a lot of time). I soon applied as Junior System Engineer within the same company without prior education and got the job! And I'm back feeling I entered a new area where you feel you can do so much by just learning how it works. Now I want to learn to develop in PHP so I may make another step further.
Not a rant, but I want to share my experience as labrat starting to someting programming(did some bio-informatics, which was really interesting but with less emphasis on programming but more on data analysis). Still got a gigantic of list I want to learn from languages and frameworks to orchestration systems.
When your IT VP starts speaking blasphemy:
We all know what’s going on with the API. Next week we may see 6x order volumes.
We need to do everything possible to minimize the load on our prod database server.
Here are some guidelines we’re implementing immediately:
· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.
Please keep the communication level high and stay vigilant in protecting our prod environment!"
RIP most of what I do at work.3
I don't know what to choose...
A no-brainer job in data input with security, stability and a chance of promotion or transfer to another area IT related far away in the future...
An intership of one year, in systems analysis, with a 30% better salary but no guarantees after that...4
There's a team where the leader has some real weird/bad ideas here, it's like, ugh, no. I can't say no because I'm not in position to contest, but geez I so want to.
Like, we wanted to load data with pagination because there could be a lot of them (could reach thousands objects of data easily).
Team Leader: No, no pagination because that ask to call those services several times. Only one call, you load all the data in the same single page and you don't call the service again, so stock them in cache.
So the idea at first looks bad, and after analysis and research, it is real bad, of course.1
How 2018 brings an aww bug.
At my company, we've a reporting application. Which kind of provides analysis of client's weekly business. When you open the application it shows their business trend starting from Sunday to day of week.
As usual their is an endpoint to get the data basis on start date. As soon as date changed to 1st Jan, it stopped showing data. Given that it was a long weekend, no one was available from the tech. Support team got tremendous amount of tickets for this. Later on Tuesday while debugging we got to know while forming the date in the application the logic was like this
- get current year
- get the date and month on start of week
Combine these 2 and request the data. All the time it was fetching data starting from 31 Dec 2018.
Me, working hard on a SQL project with a deadline that is half what it should be with no support from the other people on the project and was mostly made with with data I imagined would be in there cos no one could get me any fucking shit done (i.e. effectively designed, built, tested, fixed, upgraded, documented on my own for an entire weekly/monthly/ad-hoc analysis process that would output various reports for internal/external/management)
Manager - man who is a known waste of space but for some reason is in charge of the smallest part of the project, shouldn't have been fucking involved fucking management guzzling stain magnet...
Manager: Hey, do I need to refresh the database?
Me: ............I dunno, do you think we should refresh the database that this entire project is reliant? I mean...why do we need up to date transactions to analyse? Wait....you telling me it's not been being refreshed this ENTIRE time?
Manager: No....you never said I should. So should I?
Me: ..................I never said you should!?!? Are you not in the meetings talking about dependencies?????? Do you think i should have up to date trans or just run this with old stuff????? Why would you not update it!??!!?!?!? Its transactions...... (Desperately trying not to punch through my screen, through his, into his throat)
Manager: I think i'll refresh it and add it to the job?
Me: ....................(goes back to work cursing with music in so I think its quiet but who knows).
Tard, don't know how he even gets to work without someone holding his fucking hand.
Happy ending, I don't work there anymore :p
Sad ending, his spirit of tard follows me to my new jobs and possesses someone (or three sometimes)
My main focus Is data analysis. Being a physics student I do quite a lot of this using the likes of numpy and matplotlib, or GNU Octave.6
Since a country is considered to be the strongest definition of a group of unified people, consider the following scenario:
Every country makes it mandatory that any foreign company can only sell their software products as free to use softwares, or one time cost software, or a membership software.
These foreign companies are NOT allowed to generate revenue via Advertisement services or data collection/sharing/analysis AT ALL.
The sole right to share/collect data and use Advertisments as a revenue model remains in the hands of domestic companies born, registered and working in that country only.
This would generate an equal chance for both domestic and international companies to grow(domestic companies getting a chance to grow with a better revenue model, while international companies getting a chance to grow by monetising their tools/algorithms and investing in domestic companies) , resulting in countries getting a chance to grow themselves.
Open source will still win, as open source majorly contains tools and technologies for general public use.
Premium tools and frameworks would become even more valuable, and would he shared among countries like the way they share space researches and resources on nuclear technology today.
Privacy will win, as the data of a particular country stayed within a country. Domestic Companies sharing data with other companies (or even their foreign parent companies) would be held against the respective country's laws only, and government would be more involved in protecting its citizens from data theft.
Is it feasible?13
Worked entire day on an ML to predict train ticket status for Indian Railways.
While doing the analysis of the data, and trying to check/research which parameters to use and which not, I'm feeling like racist -_-.
I was looking for busy period of the year(read festive seasons) and I removed those festivals which are celebrated by minor groups.
There's more to it, but the results are better now.4
want to do software for my final year project but the lecturer's not in office and didnt reply my email during choosing topics, so end up with data analysis. 😴1
just receive a refurbished 4 years old laptop.... then deep clean it.... now having dilemma on picking a distro......
mainly use to run data analysis (r, Python, Java, C++, mySQL, MongoDB and some cloud servers...)
my thinking about a good distro to me, comfortable appearances, customize freedom, community support and constant security update.
any suggestion people??6
So I have question about my resume.
During my college time, I have done two projects related to politics:
One is to analyze the bias of media. What I did is scrape news covers for Trump and Hillary during election year and get sentiment analysis. The result is not surprising that among NY Times, NBC, Fox, Eashington Post, and CNN, Fox news is clearly favoring Trump, since Fox news is a republican news site.
The other project I did was to analyze the speech complexity and sentiment of the election. One of the observation we made was that Hillary and Trump are almost at the same level regarding speech complexity. However, Trump has a more positive sentiment in the speech, which is true consider how much he loves to say make America great again.
Now the question is, when I gave my advisor my resume, she said that I'd better not put those two projects on my resume since they are related to politics.
But, I am applying for a data science master degree. Seriously, I was just collecting the data and the data speaks for himself, why should I take those projects off my resume? I'm very proud of those projects I did as a matter of fact.
So here is the question. Shall I take off those two projects on my resume because they were political or I should leave it thereawarreally need some professional views. Please.1
Is a masters in statistics worth it?
A bit of background:
I got my bachelor in actuarial math (statistics for insurance risk), then found machine learning and got a couple of gigs in software development and data engineering. I became my previous employers the go to guy for questions about data integrity and structure.
Now I am heading to a new job that specializes in ML for gambling. And while I love the math, I really see myself doing more software development and system architecture work (with some analysis). I already started this masters program, so I got less than a year to finish, but starting to feel like its a waste of my time, but also, I dont want to just quit it.
I have last few months left out of graduation and i don't know what should i learn. There's so much things (web dev , ai/ml, blockchain, android , cloud, ,hybrid apps, gaming, ar/vr, data analysis, security,etc) and as a cs student, i feel i should be knowing them all.
In last 6 years ,
Techs that i liked or got success in :
java, Android,python, data analysis, hybrid apps(flutter)
Techs that i didn't liked or failed in : ai/ml, cloud computing , webdev(css/js) ,hybrid apps(react/angular/ionic/...)
Techs that i didn't tried : security, cryptography, blockchain, open cv , ar/vr, gaming
I am not bound by my likeness or success.
My failures was mainly because i didn't liked those techs and continued further in them. And my success comprises of just launching a few apps, passing in some certification or grabbing an internship opp because of those skills.
But if you think a particular skill is necessary to have as a cs professional then let me know. I just want to earn a lot of money and get out of this mess asap1
personal projects, of course, but let's count the only one that could actually be considered finished and released.
which was a local social network site. i was making and running it for about three years as a replacement for a site that its original admin took down without warning because he got fed up with the community. i loved the community and missed it, so that was my motivation to learn web stack (html, css, php, mysql, js).
first version was done and up in a week, single flat php file, no oop, just ifs. was about 5k lines long and was missing 90% of features, but i got it out and by word of mouth/mail is started gathering the community back.
right as i put it up, i learned about include directive, so i started re-coding it from scratch, and "this time properly", separated into one file per page.
that took about a month, got to about 10k lines of code, with about 30% of planned functionality.
i put it up, and then i learned that php can do objects, so i started another rewrite from scratch. two or three months later, about 15k lines of code, and 60% of the intended functionality.
i put it up, and learned about ajax (which was a pretty new thing since this was 2006), so i started another rewrite, this time not completely from scratch i think.
three months later, final length about 30k lines of code, and 120% of originally intended functionality (since i got some new features ideas along the way).
put it up, was very happy with it, and since i gathered quite a lot of user-generated data already through all of that time, i started seeing patterns, and started to think about some crazy stuff like auto-tagging posts based on their content (tags like positive, negative, angry, sad, family issues, health issues, etc), rewarding users based on auto-detection whether their comments stirred more (and good) discussion, or stifled it, tracking user's mental health and life situation (scale of great to horrible, something like that) based on the analysis of the texts of their posts...
... never got around to that though, missed two months hosting payments and in that time the admin of the original site put it back up, so i just told people to move back there.
awesome experience, though. worth every second.
to this day probably the project i'm most proud of (which is sad, i suppose) - the final version had its own builtin forum section with proper topics, reply threads, wysiwyg post editor, personal diaries where people could set per-post visibility (everyone, only logged in users, only my friends), mental health questionnaires that tracked user's results in time and showed them in a cool flash charts, questionnaire editor where users could make their own tests/quizzes, article section, like/dislike voting on everything, page-global ajax chat of all users that would stay open in bottom right corner, hangouts-style, private messages, even a "pointer" system where sending special commands to the chat aimed at a specific user would cause page elements to highlight on their client, meaning if someone asked "how do i do this thing on the page?", i could send that command and the button to the subpage would get highlighted, after they clicked it and the subpage loaded, the next step in the process would get highlighted, with a custom explanation text, etc...
dammit, now i got seriously nostalgic. it was an awesome piece of work, if i may say so. and i wasn't the only one thinking that, since showing the page off landed me my first two or three programming jobs, right out of highschool. 10 minutes of smalltalk, then they asked about my knowledge, i whipped up that site and gave a short walkthrough talking a bit about how the most interesting pieces were implemented, done, hired XD
those were good times, when I still felt like the programmer whiz kid =D
as i said, worth every second, every drop of sweat, every torn hair, several times over, even though "actual net financial profit" was around minus two hundred euro paid for those two or three years of hosting.
So, our lab professor in university for data structures was evaluating my friend's group assignment next to me and my colleague.
Professor: So, what does this method do?
Friend: I feel it would be best to explain our structure first and then how our methods work.
P: No, no. It's okay, I'm seeing it here in my laptop. *Looking at their report which included nothing about the structure of the assignment since she asked specifically only to place the methods and their complexity analysis due to not having time to look and evaluate a full report for each group*
F: Okay, *proceeds to explain*.
P: Okay, and why is your code commented?
F: Because it's good practice and we learned in subject x last semester.
P: Impossible! I teach that subject and I did not teach that in my classes.
F: Okay, but our professor did.
P: *Says nothing*.
They got a 3.25/5 as a grade for it and got really pissed off. I mean, they spent a lot of time making the assignment's structure perfect, optimizing their code and the professor did not care for the structure, just the methods, on the data structures subject.
FYI, they are workaholic, dedicate a lot of time improving their skills and their normal grades are usually >= 4.25.3
Not sure what I wanna do ---
Backend Development or Data Analyst
Even in backend development confused between nodejs, PHP, Python though I don't have proper knowledge of all of them.
For Data Analyst I am looking the way from where I should start.
I am in 3rd year of my college
Can anyone please help me with this?6
Been working on trying to get JMdict (relatively comprehensive Japanese dictionary file) into a database so I can do some analysis on the data therein, and it's been a bit of a pain. The KANJIDIC XML file had me thinking it'd be fairly straightforward, but this thing uses just about every trick possible to complicate what one would think would be a straightforward dictionary file:
* Readings and Spellings/Kanji usage are done in a many-to-many manner, with the only thing tying them together being an arbitrary ID. Not everything is related, however, as there can be certain readings that only apply to specific spellings within the group and vice versa. In short, there's no way to really meaningfully establish a headword fora given entry.
* Definitions are buried within broader Sense groups, which clumsily attach metadata and have the same many-to-many (except when not) structure as the readings/spellings.
Suffice to say, this has made coming up with a logical database schema for it a bit more interesting than usual.
It's at least an improvement over the original format, however, which had a couple different ways of setting up the headword section and could splatter tagging information across any part of a given entry. Fine if you're going to grep the flat file, but annoying if you're looking for something more nuanced.
Was looking online last night to see if anyone had a PHP class written to handle entries and didn't turn anything up, but *did* find this amusing exchange from a while back where the creator basically said, "I like my idiosyncratic format and it works for me. Deal with it!": https://sci.lang.japan.narkive.com/...
Grateful to the creator for producing the dictionary I've used most in my studies over the years, but still...3
"Dear TitanLannister : You are in the final year. A lot of shit is happening around u. its now time to make a career and take tough decisions. What would you do?"
CHOICE 1: COMPETITIVE
>>>>background : "a lot of super companies like wallmart, fb, amazon, ms, google,.. etc simply takes a straight coding test for fresher placement. They ask tough bad ass level questions, but with right guidance, a hell ton of dedicated hours of coding, and making it to the top of various coding tests could make you a potential candidate"
>>>>+ve points :
- "You got the teachers and professionals with great experience to guide you"
- "a dream job come true.you can go there and join teams that interests you"
- "it was your first exposure to computer world. maybe you would like doing it again, after 4 years"
>>>> -ve points:
- "You have always been an average 70 percentile guy. The task requires 2000-3000 hours of coding an year. it will be hard and you always grow bored out of this pretty quickly"
- "Even If you did that , you stand a lesser chance because your maths is shitty.There are millions running in this race with brains faster than your IDE"
- "your college will riot with you because they expect 75% attendance"
- "You are virtually out of college placements, in which , even though shitty companies come and offer even shittier 4LPA packages($6000 per annum), would take a tough logical/aptitude based test for which you won't be able to prepare"
CHOICE 2: PROFESSIONAL WORK
>>>>background: "you always wanted to create something , and therefore you started taking android based courses. you have been doing android for over 2 years and today you know a lot of things in android. you might be good in other professional lines like web dev, data analytics, ml,ai, etc too if you give time to that"
>>>>+ve points :
- "you will love doing this, you always did"
- "With the support of a good team, you will always be able to complete tasks and build new things quickly"
- "Start ups might offer you the placement, they always need students with some good exposure"
>>>>-ve points :
- "Every established company which provides interesting dev work takes their first round as coding, and do not considers your extra curricular dev work. So you are placing your all hopes in 1 good start up with super offerings that would somehow be amazed by your average profile and offer you a position"
- "start ups are well, startups and may not offer a job security as strong as est. companies"
- "You are probably not as awesome dev as you think you are. for 2 years, you have only learned the concepts , and not launched more than 1 shitty app and a few open source work"
CHOICE 3: NON CODING
>>>>background: "companies coming in college placements have 1-2 rounds of aptitude,logical reasoning , analysis based questions and other non tech tests. There are also online tests available like elitmus,AMCAT, etc which, when cleared with good marks help receive placements from decent established companies like TCS, infosys, accenture,etc"
>>>>+ve points :
- "you will eventually get placed from college, or online tests"
- "there will be a job security, as most of these companies bonds the person for 2-3 years"
>>>> -ve points:
- "You really don't like this. These companies are low profile consultant/services based companies which would put you in any area: from testing to sales, and job offers are again $5000-6000 per annum at max"
- "Since it includes college, the other factors like your average cgpa and 1 backlog will play an opposing role"
- "Again, you are a 70 percentile avg guy. who knows you might not able to crack even these simple tests"
Ugh... I am fucking confused. Please be me, and help.The things that i wrote about myself are true, but the things that i assumed about super companies, start ups or low profile companies might not be correct, these points comes from my limited knowledge ,terrified and confused brain, after all.
Wanted to know what you guys think about "Dev" jobs that also include making slide shows and presentations?
Eg. Imagine working on an analysis engine. And just when the core functionality is working, your boss wants your team to make presentation for every client that it's going to be used for - using the raw data given by the engine.
Instead of maybe adding that function to the engine itself.
Suddenly your work is now 12+hours of MS office instead of 8 hours of coding.
And a year later. You have 10 unfinished skeleton code architectures, poorly documented and 90% of the test cases never written.
And most of the devs who were on the initial project have either left out of frustration or have been fired because apparently fresher's who can not code with a senior coder level proficiency is not performing well.10
I pitched this crazy sentiment analysis story and it got approved.
Which is great... only now, I have to start learning regular expressions to tokenize my data. Regular Expressions are the worst. 😒5
If I wanted good feedback for my products (especially if they were geared towards developers) I would scrap devRant and do some data analysis on how our products are perceived in raw form. Would be very raw and informative insight indeed because you are at the heart of raging innovation (raging innovation: when a developer is so pissed at a flaw in a piece of software they highlight or fabricate an ingenious feature or solution) and will help not only iron the kinks out but make a better product all together. Also, of course the good aspects would be lauded.
Has anyone (who does either Data Analysis, ML/DL, NLP) had issues using AMD GPUs?
I'm wondering if it's even worth considering or if it's too early to think about investing in computers with such GPUs.14
Sales guys in my company are unbelievable. They told client that my analysis algorithm runs for about one hour, so that is the time they have to wait for results. I asked if he actually asked what hardware do they have, and if they want all analysis done (its a complex algo) paralell, or just one by one.
Then I suggested to better ask these things, otherwise we end up reckt. Annnnd he said they will question our professionalism if we ask too much questions.
Ok so let me summarize. My algo needs to run maximum 1 hour long, regardless the actual functionality and input data, on any fcking hw you can imagine, or else i am not a pro. Mmmkay.
Probably working for a scientific research firm and building in house tools for collection and analysis of data...
I'm taking data structures and algorithm analysis class at college but don't know how to study... I need advice, by the way the language we're using is c++9
Top 12 C# Programming Tips & Tricks
Programming can be described as the process which leads a computing problem from its original formulation, to an executable computer program. This process involves activities such as developing understanding, analysis, generating algorithms, verification of essentials of algorithms - including their accuracy and resources utilization - and coding of algorithms in the proposed programming language. The source code can be written in one or more programming languages. The purpose of programming is to find a series of instructions that can automate solving of specific problems, or performing a particular task. Programming needs competence in various subjects including formal logic, understanding the application, and specialized algorithms.
1. Write Unit Test for Non-Public Methods
Many developers do not write unit test methods for non-public assemblies. This is because they are invisible to the test project. C# enables one to enhance visibility between the assembly internals and other assemblies. The trick is to include //Make the internals visible to the test assembly [assembly: InternalsVisibleTo("MyTestAssembly")] in the AssemblyInfo.cs file.
Many developers build a POCO class in order to return multiple values from a method. Tuples are initiated in .NET Framework 4.0.
3. Do not bother with Temporary Collections, Use Yield instead
A temporary list that holds salvaged and returned items may be created when developers want to pick items from a collection.
In order to prevent the temporary collection from being used, developers can use yield. Yield gives out results according to the result set enumeration.
Developers also have the option of using LINQ.
4. Making a retirement announcement
Developers who own re-distributable components and probably want to detract a method in the near future, can embellish it with the outdated feature to connect it with the clients
[Obsolete("This method will be deprecated soon. You could use XYZ alternatively.")]
Upon compilation, a client gets a warning upon with the message. To fail a client build that is using the detracted method, pass the additional Boolean parameter as True.
[Obsolete("This method is deprecated. You could use XYZ alternatively.", true)]
5. Deferred Execution While Writing LINQ Queries
When a LINQ query is written in .NET, it can only perform the query when the LINQ result is approached. The occurrence of LINQ is known as deferred execution. Developers should understand that in every result set approach, the query gets executed over and over. In order to prevent a repetition of the execution, change the LINQ result to List after execution. Below is an example
public void MyComponentLegacyMethod(List<int> masterCollection)
6. Explicit keyword conversions for business entities
Utilize the explicit keyword to describe the alteration of one business entity to another. The alteration method is conjured once the alteration is applied in code
7. Absorbing the Exact Stack Trace
In the catch block of a C# program, if an exception is thrown as shown below and probably a fault has occurred in the method ConnectDatabase, the thrown exception stack trace only indicates the fault has happened in the method RunDataOperation
8. Enum Flags Attribute
Using flags attribute to decorate the enum in C# enables it as bit fields. This enables developers to collect the enum values. One can use the following C# code.
he output for this code will be “BlackMamba, CottonMouth, Wiper”. When the flags attribute is removed, the output will remain 14.
9. Implementing the Base Type for a Generic Type
When developers want to enforce the generic type provided in a generic class such that it will be able to inherit from a particular interface
10. Using Property as IEnumerable doesn’t make it Read-only
When an IEnumerable property gets exposed in a created class
This code modifies the list and gives it a new name. In order to avoid this, add AsReadOnly as opposed to AsEnumerable.
11. Data Type Conversion
More often than not, developers have to alter data types for different reasons. For example, converting a set value decimal variable to an int or Integer
Data structure and Analysis
For experienced ones, add "System Design".
...path for big fat packages.
My client wants me to "help increase sales" by applying seo voodoo. For over two years they didn't even have Analytics. Now, after 12 days they're asking me what do I make of the results. Like that's a meaningful amount of data for analysis.3
everyone needs to be a data scientist/evangelist/superhero or AI enthusiast/developer/super-coder or project head for critical business needs/ or doing analytical analysis for business processes...even school students who are learning just to write English sentences, think they can code easily
AI folks, who think you can code automatically by thinking with no typing..
to them i say2
I'm starting to look at how to get devices to send data to an endpoint for storage and analysis. I'm looking at AWS iot stuff like core and green grass but then I'm thinking that a REST API could also do the job. I don't need to connect devices to each other (in first iteration). Dont think I need any edge stuff either. Anybody have any experience with this?3
My jupyter notebook has outgrown itself on some real world trading data analysis and its becoming a pain to add to (further) and share.
Need to find better alternatives, web apps where are you?
But i know nothing about it. Learning curve ahead!
I've 7 interactive dashboard plots (from some data) in jupyter-notebook.
- It'd be nicer to have a web app to use them without running notebook from a different location.
- Or running notebooks remotely (running as daemons on host machine).
Any suggestions for a starter ?
rant before requirements, coz rants lead to better requirements.
Hey guys is there anything we as devs could do regarding this covid-19 situation in terms of our technical skills? Like any project or something for making awareness at global or local level, or some status update regarding various things/items that are affected by this pandemic?
I am looking for ideas or currently ongoing projects that i could contribute to. Currently my tech stack is limited to java, python , android and a little bit of basic data analysis, but i am fluid enough to learn and contribute.7
Devs can you point me to the topics that i should know in web dev? Kind of being hovering across a lot of different stuff, but don't know the actual path to follow . I kind of wish to become a full stack dev( not just the web) / Systems engineer someday and i am hoping to get a job asap
About me :
- oop/strong typing language guy. Knows enough java/kotlin/android dev to make decent Android apps, with confidence to apply on Android dev jobs
- knows enough python3 to understand its classes/functions. (But not more)
- knows some html css and client side js to make static websites, though my knowledge in this domain is not sequential but rather learning whatever i found interesting, so don't really get confidence to apply on a web dev job when i see an opening
- knows very basics of backend development in flask/php. Although worked with SAAS solutions liked firebase/aws
- knows some basics of data analysis6
Does anyone on here use the 3D software blender. What kind of Python do I need to learn (learned data analysis for internship but 80% sure won't be same as animation Python)1
I starting developing my skills to a pro level from 1 year and half from now. My skillset is focused on Backend Development + Data Science(Specially Deep Learning), some sort of Machine Learning Engineer. I fill my github with personal projects the last 5 months, and im currently working on a very exciting project that involves all of my skills, its about Developing and deploy a Deep Learning Model for Image Deblurring.
I started to look for work two months to now. I applied to dozens of jobs at startups, no response. I changed my strategy a bit, focusing on early stage startups that dont have infinite money for pay all that senior devs, nothing, not even that startups wish to have me in their teams. I even applied to 2 or 3 and claim to do the job for little payment, arguing im not going for money but experience, nothing. I never got a reply back, not an interview, the few that reach back(like 3, from 3 or 4 dozen of startups), was just for say their are not interested on me.
This is frustrating, what i do on my days is just push forward my personal projects without rest. I will be broke in a few months from now if i dont get a job, im still young, i have 21 years, but i dont have economic support from parents anymore(they are already broke). Truly dont know what to do. Currently my brother is helping me with the money, but he will broke in few months as i say.
The worst of all this case is that i feel capable of get things done, i have skills and i trust in myself. This is not about me having doubts about my skills, but about startups that dont care, they are not interested in me, and the other worst thing is that my profile is in high demand, at least on startups, they always seek for backend devs with Machine Learning knowledge. Im nothing for them, i only want to land that first job, but seems to be impossible.
For add to this situation, im from south america, Venezuela, and im only able to get a remote job, because in my country basically has no Tech Industry, just Agencies everywhere underpaying devs, that as extent, dont care about my profile too!!! this is ridiculous, not even that almost dead Agencies that contract devs for very little payment in my country are interested in me! As extra, my economic situation dont allows me to reallocate, i simple cant afford that. planning to do it, but after land some job for a few months. Anyways coronavirus seems to finally set remote work as the default, maybe this is not a huge factor right now.
I try to find job as freelancer, i check the freelancer sites(Freelancer, Guru and so on) every week more or less, but at least from what i see, there is no Backend-Only gigs for Python Devs, They always ask for Fullstack developers, and Machine Learning gigs i dont even mention them.
Maybe im missing something obvious, but feel incredible that someone that has skills is not capable of land even a freelancer job. Maybe im blind, or maybe im asking too much(I feel the latter is not the case). Or maybe im overestimating my self? i think around that time to time, but is not possible, i have knowledge of Rest/GraphQL APIs Development using frameworks like Flask or DJango(But i like Flask more than DJango, i feel awesome with its microframework approach). Familiarized with containerization and Docker. I can mention knowledge about SQL and DBs(PostgreSQL), ORMs(SQLAlchemy), Open Auth, CI/CD, Unit Testing, Git, Soft DevOps Skills, Design Patterns like MVC or MTV, Serverless Environments, Deep Learning Solutions, end to end: Data Gathering, Preprocessing, Data Analysis, Model Architecture Design, Training and Finetunning. Im familiarized with SotA techniques widely used now days, GANs, Transformers, Residual Networks, U-Nets, Sequence Data, Image Data or high Dimensional Data, Data Augmentation, Regularization, Dropout, All kind of loss functions and Non Linear functions. My toolset is based around Python, with Tensorflow as the main framework, supported by other libraries like pandas, numpy and other Data Science oriented utils.
I know lot of stuff, is not that enough for get a Junior Level underpaid job? truly dont get it, what is required for get a job? not even enough for get an interview?
I have some dev friends and everyone seems to be able to land jobs, why im not landing even an interview?
I will keep pushing my Dev career, is that or starve to death. But i will love to read your suggestions! how i can approach this?
i will leave here my relevant social presence:
Thanks in advance!11
Twitter api restricts me to fetch only 3202 tweets and if I had to fetch the retweeters count is limited to 100 .. is there any open source which will help me get through this problem because the data being collected is not sufficient for analysis
Question: what's your opinion on Julia? It seems promising, the macro syntax is neat. But would you use it for data analysis/management or general purpose development?
Does ayone knows of a
+ Open Source
Data test generator tool?
I'm on my first work with a development that will go to production, and I'd like to test performance, UI and all of that with random generated data. I know about dbSchema but it's trial pulls me back
Hi So I need some solid advice from you all wonderful people.
I think i am now ready to look into job side of this world, but have lots of doubts , read my story.
I have been learning android for last 2 years. Most of the time i have been trying to understand how stuff works in android , but i have also gained a few other skills ( python programming, kotlin/flutter basics data analysis basics, testing, some graphic designing, aweful web dev ,etc). But i really want to work with Android. I don't have any specific Salary figure in mind, but i guess my knowledge is better or atleast par with most of the good android developers.
So i want to know how is this fresher/placement thingy work?
1.) GETTING KNOWN? : How can i make some good android based company aware that I am available for hiring? Should i start emailing every android related company that i know of? Should i start listing my profile on recruitment sites like linkedin or internshala? This year it is being said that companies will come for placements. From the status of my college, they are going to give me way to less $ , nd i know am not going to like any of them, but i guess i have to sit for them too.
2.INTERVIEW OR DIRECT PLACEMENTS? A little pre-context: i am currently starting my 4th year in clg. Afaik , 4th year isnt that strict and their can be leniency in terms of attendance. But my college is a place full of political cun*s in the name of directors and HODs and I don't know if they are again going to enforce the old 75% mandatory criteria. Plus if the company is from a different state/country , then my attendance would definitely not suffice.
So mainly i am unsure if somehow a company hires me, i would be able to start immediately. I heard that there are interviews for job recruitment after which the candidate is binded with an agreement to do some months training followed by permanent working after college completion.
This type of agreement is very much suitable for me, since from what my friend tells me, trainings can be lenient and understanding regarding exam preparations nd stuff.
So what do company usually chooses? Binding a fresher on immediate working basis or do they consider graduate completion?
Also, i suck at competitive coding. Do i need to polish myself on that or some company is willing to give me chance on the basis of my other skills 🙈(okay, no kidding , that's a serious question. I need to either work on getting better in competitive or build more apps based on that)
3.) ANDROID OR EVERYTHING? From what i have heard, working as a professional fresher is more like being an allrounder than being a domain specialist. But as i already stated, i really dig android and that's no small framework. I may di other stuff too, but won't interest me nd my output might be less efficient than expected.
So freshers can really be asked to do any stuff? Or can i still be in the area i like being into?
4.) COMPANY OR START-UP? Yeah, this is a general debate starter. Ignoring the business side of the conversation ( job safety vs more salary, experience, etc) the thing that's most important for me is the presence of a team. I want someone to assign me a task, whose vision i could follow, from whom i could learn, and some other people who are supportive and doing the same amount / similar work that am doing . This is so much import8 for me that i can easily ignore other factors for a better team. I once took a call from a startup ceo who hired me, a 2 month old android beginner at that time, as the "lead android developer"
But if am being on a team where i am supposed to do any random stuff that is assigned, then obviously this whole point of "visionary, helpful leader, guiding team, "etc goes moot9
It seems like it never came to the mind of the developers of Apache Spark that people might want to use two of their libraries together to perform graph analysis on streamed Twitter data.
Why can't I simply use the streamed batches to create and extend a graph and perform continuous graph Algorithms on it??2