Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API

From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "apache spark"
-
At an expensive Apache Spark training class. For hands on portion you use their system. When everyone logged on, their system couldn't handle the load. First they blamed the hotel wifi (even though every other site was running perfectly) and then they blamed AWS. It's clearly an issue with their system. It's ridiculous to not be honest with a room full of developers who know what's up.4
-
Fucking shit uni is such a waste of time. We are learning Apache Spark in Big Data module. Fucking losers have Spark 1.6.0 installed while the latest version is 2.2.1 right now.
What a bunch of cunts. We are paying tons of money to study deprecated shits and a degree. A fucking degree that is not even on a piece of paper anymore.
Fuck this shit man.6 -
Just finished dumping all ethereum tranasctions into one big 30 GB csv.
Only thing left is to configure Apache Spark cluster.3 -
To be honest, I'm not as excited as I was 6-7 years ago when our tech industry seen a big leap, where these ML/Deep Learning algorithms were out performing humans, Apache Spark out perfomed Hadoop in distributed computing, Docker/Kubernetes are the new phenomenon in software development and delivery, Microservices architecture, ReactJS virtual DOM concepts were so cool.
Really though, I've come realise that these software trends come and go. All you need to do is adapt and go with the flow.2 -
I'm not sure if it is a dev experience, but definitely boosted my morale.
In 2014, my company (in India) sent me to attend a conference in Boston. The conference was about big data.
When I came back, I wrote a blog post about Apache Spark in my company's blog. Because of the blog, my name got mentioned in a prominent newspaper's article about Apache Spark.
PS: That is my only blog post till date -
Hey guys, I just made a small contribution to the world of free code. It's a an install script for installing Apache Spark on windows with all its dependencies and quirks. Installing it on Windows is not so straightforward as some of you might know. This script should make everything good to go.
https://github.com/Mayhem93/...4 -
Holy fucking crap, think I actually got some productive, positive output from this whole generative AI debacle.
Rather because I skipped the whole Prompt step and used FOMO blabber against itself.
Some context: at my last gig we had a whole "humanware procurement department" (A.K.A. "hiring managers", those fucks who think that javascript and java are the same thing). It was during the pandemic tech hiring boom. At this new joint I'm at, a MUCH smaller company, I gotta do it myself. Boring as fuck but at least I can get some good karma by not making an ass of myself for candidates, and trying to make this whole process a tad less abusive.
I got my reading up to date, and surprisingly enough, "yankee dandy" (HBR) has actually been saying one or two things that are not complete hogwash. For a start, they say that companies have been making their hiring processes overly complex and even after hours of interviews they hardly measure half the skills they actually need, and spend too long talking about many skills that are not actually required for the positions.
"Huh. That sounds like the inneficiencies that the stupid 'AI will make meetings more efficient' industry is overpromising to overturn"
So I tried a new thing. Instead of your off-the-shelf "solve this NP-Hard problem in O(1) then draw this bird using only your nose then invert a binary tree in COBOL then tell me what type of sitcom character are you" crap, I tried grasping how it would be like to work with the candidates. One at a time. Not too long, but not too short talks. I'm not trying to check if a kid really knows how to implement a solution for the TSP in apache spark, or if they know every cipher in TLS 1.3. I just want to know if they can understand a technical request and come to me with a plan on how to solve it without handholding or "just use a really big VM, like, 32Tb of RAM!"
Thus, if I can work with them. That's all. The rest are specific skills that can be trained in time, if the person is willing to learn new stuff.
But that is not good enough for HR, ooooh, no. You "need" an "objective way of measuring their skills", otherwise its "just biased opinions."
But that gave me an idea.
See, our HR VP is someone deep in the whole AI pyramid scheme, who drank the kool-aid and swallowed up even the cup. FOMO is their name. Hype is their business.
I posh'd up my bullshit'ish jargon and went whole "In the advent of new disruptive technologies, strategic skills can be acquired with grit and proper AI prompting. Thus, leveraging our collaborative intelligence capabilities we can hack our challenges and optimize our resources to offer more innovative opportunities and bolster our employer branding" - translation: "shut up and lemme hire someone good and reasonably priced instead of a sleazy smooth talker who wants 100M just to show up and play with chatgpt all day". The whole point is to make it sound like "we're using AI, so it's good" instead of "im doing the work I'm being paid for, so it's old-fashioned"
It seems like the HR troll swallowed it, bait and hook. Maybe all we really needed this whole time is to say the magic word "AI," especially if it makes absolutely no sense in the context. Now I want to get them to sign off on a "AI mindfulness bolstering platform" (a massage chair). Fingers crossed.4 -
I am working with Apache Spark on Scala for my undergrad research. It got me thinking: what is Scala used for beside doing stuff with Spark?
Not sure whether it's just me but most posts talking about Scala seem to always mention Spark. -
Damn Apache Spark. Instead of supporting just one language that works perfectly well, let's support several languages that give users different results! It's so much fun to combine chunks of each language to get a fully working solution!3
-
I have unsubscribed myself from the Apache Spark mailing list but I keep getting emails from them regarding PRs and comments on the repo. What do I do now? Block them or mark as spam?2
-
One of the things that gives me a lot of satisfaction is doing integrations. One customer has a good architecture, and they chose to put their metrics database with billions of records in Apache Cassandra. Which is pretty cool! A business consultancy that is helping them grow a lot, has implemented PowerBI and the dashboards are really good, but the most important reports will be based on this gigantic Cassandra database. We just delivered a quick integration using Apache Spark. Small project, fast delivery and everyone happy. Is so good. Mission accomplished feeling.1
-
It seems like it never came to the mind of the developers of Apache Spark that people might want to use two of their libraries together to perform graph analysis on streamed Twitter data.
Why can't I simply use the streamed batches to create and extend a graph and perform continuous graph Algorithms on it??2