Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "hadoop"
-
Every goddamn time.
Boss: Hey, how is the project going?
Me: promising. We have some basic functions working, but at this point it is more like a proof of concept.
Boss: Ah ok, I see.
Couple of days later...
Boss: I talked to a client who was very interested in the stuff you are just building. Made a really good deal! We need to be live by the end of the week.
Me: What?
Boss: What?6 -
Went to an interview for the position ‘PHP Web Developer’. Interviewer scans through my CV for 2mins and then starts the interview.
Interviewer: Do you know Java?
Me: I know Java but I don’t have any professional experience
Interviewer: Do you know Hadoop?
Me: No. I’ve never worked on it
Interviewer: Our company works on Hadoop hence you should be able to work on that after joining.
Me: I thought this is a PHP web dev position.
Interviewer: Of course. But you will have work on various other things too!
Me: I don’t think I want to become jack of all trades. Thanks for the opportunity!
I got up and left the interview...7 -
Your resume:
Git
SSL
Vue
Angular
React
Node
Spring boot
MySQL
MongoDb
HTML
CSS
Java
Javascript
Bootstrap
Cassandra
Hive
Hadoop
Block chain
GraphQL
Kubernetics
Jenkins
Azure
GCP
Interviewer:
Sorry, we need someone who knows AWS8 -
Manual Data Entry: Most boring job
This reminds me of one conversation with one of my faculty..
Faculty: Why not try some Machine Learning Project?
Me: Cool. Any ideas you have already thought
Faculty: Comes up with a really noble idea
Me: Awesome idea. But we need data
Faculty: Don't worry. I will get it. Just help me setup Hadoop (see the irony.. no data yet, and he wants big data setup)
Me: But we don't have data. Let's focus of data collection, Sir
Faculty: I will get it. Don't worry. Trust me.
( I did setup for him twice coz he formatted the system on which I did the setup first time)
After 6 months,
Me: (same question) Sir, Data??
Faculty: I got it.
Me: Great. Give me, I can start looking into it from today.
Faculty: Actually, it's in a register written manually in a different language (which even I can't understand) I will hire data entry guys to convert it into English digital contents.
Me: *facepalm*
Road to Manual data entry to Big Data
Dedicating this pencil to the individuals keeping the register up to date and Sir in hopes of converting it into big data..
Long way to go..4 -
Some companies be like-
.. In job posting - We are the next big thing. We are going to change the industry. We are like Google / Facebook etc...
..in Introduction - We are the next big thing. We are going to change the industry. We are like Google / Facebook etc...
.. in Interviews - We are the next big thing. We are already changing the industry. Think of us like Google / Facebook etc...
.. during Interviews - Our interview process is rigorous because we are the next big thing. We are going to change the industry. We are like Google / Facebook etc...
.. questions in interviews - Since we are Google / Facebook, please answer questions on Java, C/C++, JS, react, angular, data structure, html, css, C#, algorithms, rdbms, nosql, python, golang, pascal, shell, perl...
.. english, french, japanese, arabic, farsi, Sinhalese..
.. analytics, BigData, Hadoop, Spark,
.. HTTP(s), tcp, smpp, networking,.
..
..
..
.. starwars, dark-knight, scarface, someShitMovie..
You must be willing to work anytime. You must have 'no-excuses' attitude
.........................................
Now in Salary - Oh... well... yeah... see.... that actually depends on your previous package. Stocks will be given after 24 re-births. Joining bonus will be given once you lease your kidneys.
But hey, look... We got free food.
Well, SHOVE THAT FOOD UPTO YOUR ASS.
FUCK YOU...
FUCK YOUR 'COOL aka STUPID PIZZA BEER - CULTURE'.
FUCK YOUR 'FLAT- HIERARCHY'.
FUCK YOUR REVOLUTIONARY-PRODUCT.
FUCK YOU!2 -
My colleague unknowingly uses the word "paralyze" for "parallelize". When is the good time to correctly him? So far he has successfully paralyzed my code.4
-
There was this 'Hadoop' trend going on. So I thought that i'll start Hadoop. After one month people have started to trend Spark claiming that it is faster. -.-
And now I am outdated.13 -
Someone at work snuck something past the censors.
Our Hadoop servers all have "bigd" in their name 😂5 -
EoS1: This is the continuation of my previous rant, "The Ballad of The Six Witchers and The Undocumented Java Tool". Catch the first part here: https://devrant.com/rants/5009817/...
The Undocumented Java Tool, created by Those Who Came Before to fight the great battles of the past, is a swift beast. It reaches systems unknown and impacts many processes, unbeknownst even to said processes' masters. All from within it's lair, a foggy Windows Server swamp of moldy data streams and boggy flows.
One of The Six Witchers, the Wild One, scouted ahead to map the input and output data streams of the Unmapped Data Swamp. Accompanied only by his animal familiars, NetCat and WireShark.
Two others, bold and adventurous, raised their decompiling blades against the Undocumented Java Tool beast itself, to uncover it's data processing secrets.
Another of the witchers, of dark complexion and smooth speak, followed the data upstream to find where the fuck the limited excel sheets that feeds The Beast comes from, since it's handlers only know that "every other day a new one appears on this shared active directory location". WTF do people often have NPC-levels of unawareness about their own fucking jobs?!?!
The other witchers left to tend to the Burn-Rate Bonfire, for The Sprint is dark and full of terrors, and some bigwigs always manage to shoehorn their whims/unrelated stories into a otherwise lean sprint.
At the dawn of the new year, the witchers reconvened. "The Beast breathes a currency conversion API" - said The Wild One - "And it's claws and fangs strike mostly at two independent JIRA clusters, sometimes upserting issues. It uses a company-deprecated API to send emails. We're in deep shit."
"I've found The Source of Fucking Excel Sheets" - said the smooth witcher - "It is The Temple of Cash-Flow, where the priests weave the Tapestry of Transactions. Our Fucking Excel Sheets are but a snapshot of the latest updates on the balance of some billing accounts. I spoke with one of the priestesses, and she told me that The Oracle (DB) would be able to provide us with The Data directly, if we were to learn the way of the ODBC and the Query"
"We stroke at the beast" - said the bold and adventurous witchers, now deserving of the bragging rights to be called The Butchers of Jarfile - "It is actually fewer than twenty classes and modules. Most are API-drivers. And less than 40% of the code is ever even fucking used! We found fucking JIRA API tokens and URIs hard-coded. And it is all synchronous and monolithic - no wonder it takes almost 20 hours to run a single fucking excel sheet".
Together, the witchers figured out that each new billing account were morphed by The Beast into a new JIRA issue, if none was open yet for it. Transactions were used to update the outstanding balance on the issues regarding the billing accounts. The currency conversion API was used too often, and it's purpose was only to give a rough estimate of the total balance in each Jira issue in USD, since each issue could have transactions in several currencies. The Beast would consume the Excel sheet, do some cryptic transformations on it, and for each resulting line access the currency API and upsert a JIRA issue. The secrets of those transformations were still hidden from the witchers. When and why would The Beast send emails, was still a mistery.
As the Witchers Council approached an end and all were armed with knowledge and information, they decided on the next steps.
The Wild Witcher, known in every tavern in the land and by the sea, would create a connector to The Red Port of Redis, where every currency conversion is already updated by other processes and can be quickly retrieved inside the VPC. The Greenhorn Witcher is to follow him and build an offline process to update balances in JIRA issues.
The Butchers of Jarfile were to build The Juggler, an automation that should be able to receive a parquet file with an insertion plan and asynchronously update the JIRA API with scores of concurrent requests.
The Smooth Witcher, proud of his new lead, was to build The Oracle Watch, an order that would guard the Oracle (DB) at the Temple of Cash-Flow and report every qualifying transaction to parquet files in AWS S3. The Data would then be pushed to cross The Event Bridge into The Cluster of Sparks and Storms.
This Witcher Who Writes is to ride the Elephant of Hadoop into The Cluster of Sparks an Storms, to weave the signs of Map and Reduce and with speed and precision transform The Data into The Insertion Plan.
However, how exactly is The Data to be transformed is not yet known.
Will the Witchers be able to build The Data's New Path? Will they figure out the mysterious transformation? Will they discover the Undocumented Java Tool's secrets on notifying customers and aggregating data?
This story is still afoot. Only the future will tell, and I will keep you posted.6 -
This day I have received the most glorious news in e-pistolary form. For some years, I was suffering in support of a client who was, well, insufferable. My presence there paralleled the divine comedy in both essence and fact.
I opened the missive, expecting another plea to bail them out of whatever clusterfuck they found themselves in. Instead, what I found was something truly magical.
"Hey Human,
I hope this finds you well. I'm not sure if you remember a few years back, we were trying to decide between IBM Cloud and AWS. Well, after years of battling FF*, we're finally moving ahead with AWS. He failed one too many times to deliver anything visibly. After you left, there was no one left he could use to steal credit, ideas, and work.
FF is still pushing to have them use IBM cloud as a "warm backup" in the event "AWS fails." We will see where that goes.
I figured you'd like to know; you were the void in the wilderness for a long time. I don't want to think about how much time we could have saved if we had just listened.
PeeEm**"
This event represents a personal victory, albeit belated, over a few peoples' absurd amount of privilege. Towards the end, I was vicious about my contestation to the insanity of adopting a desperate hedge attempt-as-cloud offering from a failing company. Some examples:
// cloud 'strategy meeting'
Moi: What cloud platform are we looking at using?
FF: We're looking at IBM cloud and AWS as a second.
Moi: Why is that? I understand you're obligated to rep your offering first, but that decision doesn't seem to have the customer's best interest at heart.
FF: IBM cloud is a market leader; AWS isn't as good.
Moi: I see. I mean, that's the tech equivalent of the company's fleet management considering monkeys on tricycles as a strong competitor to service trucks, but I get what you mean.
// steering meeting
Director: Who can we look to as an example? Who is currently using the IBM cloud?
Moi: No one; they account for a single-digit portion of the actual cloud market. Their long game to sell you a "Hybrid Cloud," which means put some front end payload in a CDN, and buy n-frame units of IBM z servers for the DC with IBM gateway appliances acting as connective tissue. So it's not the cloud at all, really.
Director: How does it compare in cost?
Moi: It's generally 40% more expensive than other clouds, and it only goes higher as you option their software.
Director: What about Watson? I hear Watson is good?
Moi: It's a brand name. Most of the "Watson" product is just a facade on top of FOSS products like Spark, Hadoop, Elasticsearch, etc.
Director: Those were words. They sounded good. FF say it's good tho so we'll believe him because we're from the same city.
Moi: *deletes Director from LinkedIn*
Moral of the story: Never trust a vendor that only recommends their products.
*FF = FatFuck - an embarrassingly rotund individual whose girth is roughly equivalent to his height. He shit his way into an IBM architect position in his mid-20s purely due to winning the visa lottery. He had fake hair glued to his head for his wedding to hide his male pattern baldness; his arrange-married wife undoubtedly cries herself to sleep after sex.
**PeeEm - the then project manager, now portfolio manager of some satellite projects. An overall decent human being, capable.9 -
Hadoop was born at Yahoo! Startups like Cloudera or Hortonworks that built their entire business around Hadoop are worth more than Yahoo!3
-
Solved a major scalability issue today.
I'm starting to think I might actually be as good as what I told the recruiters I was.
This hadoop-ecosystem job used to take about 2h40min and cost about USD 1.00/GB.
Now it takes 36 mins. At about 0.85/GB.
Fixed some over shuffling, restructured some bottleneck serial stages, used lots of weird words.
Folks in this company I just entered were struggling with this formerly unwieldy process for a year.
Now it's nimble enough to run every hour.
Maybe that whole "experience" thing people were always yammering about wasn't completely bullshit.4 -
When a client refuses to use cloud services for Hadoop in lieu of 4 local machines... For TB scale data.3
-
This is the third part of my ongoing series "The Ballad of the Six Witchers and the Undocumented Java Tool".
In this part, we have the massive Battle of Sparks and Storms.
The first part is here: https://devrant.com/rants/5009817/...
The second part is here: https://devrant.com/rants/5054467/...
Over the last couple sprints and then some, The Witcher Who Writes and the Butchers of Jarfile had studied the decompiled guts of the Undocumented Java Beast and finally derived (most of) the process by which the data was transformed. They even built a model to replicate the results in small scale.
But when such process was presented to the Priests of Accounting at the Temple of Cash-Flow, chaos ensued.
This cannot be! - cried the priests - You must be wrong!
Wrong, the Witchers were not. In every single test case the Priests of Accounting threw at the Witchers, their model predicted perfectly what would be registered by the Undocumented Java Tool at the very end.
It was not the Witchers. The process was corrupted at its essence.
The Witchers reconvened at their fortress of Sprint. In the dark room of Standup, the leader of their order, wise beyond his years (and there were plenty of those), in a deep and solemn voice, there declared:
"Guys, we must not fuck this up." (actual quote)
For the leader of the witchers had just returned from a war council at the capitol of the province. There, heading a table boarding the Archpriest of Accounting, the Augur of Economics, the Marketing Spymaster and Admiral of the Fleet, was the Ciefoh Seat himself.
They had heard rumors about the Order of the Witchers' battles and operations. They wanted to know more.
It was quiet that night in the flat and cloudy plains of Cluster of Sparks and Storms. The Ciefoh Seat had ordered the thunder to stay silent, so that the forces of whole cluster would be available for the Witchers.
The cluster had solid ground for Hive and Parquet turf, and extended from the Connection River to farther than the horizon.
The Witcher Who Writes, seated high atop his war-elephant, looked at the massive battle formations behind.
The frontline were all war-elephants of Hadoop, their mahouts the Witchers themselves.
For the right flank, the Red Port of Redis had sent their best connectors - currency conversions would happen by the hundreds, instantly and always updated.
The left flank had the first and second army of Coroutine Jugglers, trained by the Witchers. Their swift catapults would be able to move data to and from the JIRA cities. No data point will be left behind.
At the center were thousands of Sparks mounting their RDD warhorses. Organized in formations designed by the Witchers and the Priestesses of Accounting, those armoured and strong units were native to this cloudy landscape. This was their home, and they were ready to defend it.
For the enemy could be seen in the horizon.
There were terabytes of data crossing the Stony Event Bridge. Hundreds of millions of datapoints, eager to flood the memory of every system and devour the processing time of every node on sight.
For the Ciefoh Seat, in his fury about the wrong calculations of the processes of the past, had ruled that the Witchers would not simply reshape the data from now on.
The Witchers were to process the entire historical ledger of transactions. And be done before the end of the month.
The metrics rumbled under the weight of terabytes of data crossing the Event Bridge. With fire in their eyes, the war-elephants in the frontline advanced.
Hundreds of data points would be impaled by their tusks and trampled by their feet, pressed into the parquet and hive grounds. But hundreds more would take their place. There were too many data points for the Hadoop war-elephants alone.
But the dawn will come.
When the night seemed darker, the Witchers heard a thunder, and the skies turned red. The Sparks were on the move.
Riding into the parquet and hive turf, impaling scores of data points with their long SIMD lances and chopping data off with their Scala swords, the Sparks burned through the enemy like fire.
The second line of the sparks would pick data off to be sent by the Coroutine Jugglers to JIRA. That would provoke even more data to cross the Event Bridge, but the third line of Sparks were ready for it - those data would be pierced by the rounds provided by the Red Port of Redis, and sent back to JIRA - for good.
They fought for six days and six nights, taking turns so that the battles would not stop. And then, silence. The day was won, all the data crushed into hive and parquet.
Short-lived was the relief. The Witchers knew that the enemy in combat is but a shadow of the troubles that approach. Politics and greed and grudge are all next in line. Are the Witchers heroes or marauders? The aftermath is to come, and I will keep you posted.4 -
Fax machines connected to VoIP connections...
Had a nightmare recently, where my fathers machine that he really needs refused to work after he moved appartements.
Uncounted calls with different tech departments, a furious fathers and two weeks later they found out, that they forgot to activate the protocol.3 -
To be honest, I'm not as excited as I was 6-7 years ago when our tech industry seen a big leap, where these ML/Deep Learning algorithms were out performing humans, Apache Spark out perfomed Hadoop in distributed computing, Docker/Kubernetes are the new phenomenon in software development and delivery, Microservices architecture, ReactJS virtual DOM concepts were so cool.
Really though, I've come realise that these software trends come and go. All you need to do is adapt and go with the flow.3 -
Someone created a 0-followers private Twitter account and posted something to try out the new views count feature.
It raked dozens of views in a couple hours.
HOW?!?
Source: https://twitter.com/briggityboppity...
It looks like a funny data reverse-engineering exercise, so let's try and figure out what is going on.
Hypothesis 1) it is the OP's own views.
Reasonable, but unlikely if what OP says about not checking it for hours is true.
H2) It's some background job in OP's device that is refreshing OP's own latest tweets, so even without human interaction technically H1 is true. It would be some really shoddy engineering to count eye-less page views, but that's also what managers would demand.
H3) it's some internal Twitter automated function like back up, replication, indexing and word count.
See H2, it would be even dumber to count that as page views.
H4) it's some internal human reviewing for a keyword that could be associated with porn (in this case, "butts"). Really? dozens of humans to review a no-impact single post? They would have to employ hundreds of thousands of reviewers.
H5) it's some page-loading shit, like thousands of similar tweets get stored in the same index hash page and end up counting as a view in all of them every time someone loads the index page. It would be like counting every hit in the namenode as a hit in every data asset in it's Hadoop partition, or every hit in a storage block as a hit in each of it's files.
Duuuumb and kinda like H3.
H6) page views are just a fraud to scam investors. Maybe it's a "most Blockchain transactions are fake" situation, maybe it's a "views get more engagement if you don't think a lot about it" situation, maybe it's a "we don't use the metric system to count page views" situation.
All of them are very dumb.
Other hypothesis or opinions?10 -
!rant
I started learning to use Hadoop recently, and am running a VM with all I need installed on it (the HDF Sandbox to be exact). The VM wants 8GB to run and my laptop only has just that, except it also needs to run Windows at the same time...
At first I thought I was screwed, that I'd need a more capable computer to learn. I gave it a shot anyway, and told VirtualBox to give the VM 4GB, hoping the VM itself would use RAM swap to function. And it did!
What I didn't expect was Windows not slowing down even a bit. Turns out Windows can triple the computer's RAM with virtual memory that it keeps on disk.
So the bottom line is: my VM is using 4GB as if it was 8GB, and at the moment my Windows is using 8GB as if they were 14GB. All of this without breaking a sweat. The more you know!3 -
At what point do you stop optimizing queries and realize it's a database architecture, scaling problem?
We've been having production issues this week because a lot more users with more demands, and I'm going we need more servers... We can't just have one db, we need to parallelize like Hadoop...
Everyone else is going, how do we optimize queries, indexes, reduce the load...11 -
I decided to start learning about hadoop... I found a great, free tutorial, but it expects me to know python. ...well it looks like I'm learning two things at once.2
-
Some people are so crazy about Hadoop, that for every problem, they think of solving it using Hadoop. Even when you need to process a 10MB file!!!!2
-
Set-up a 5 node EC2 cluster in AWS; Install my dependencies on all; Add private keys between all for handshake; Submit my Hadoop job for processing; AWS closes my instance within 10 minutes of starting a job that took me almost half an hour to set-up because the master node's spot costs have reached more than 15 USD :(2
-
"Bro, what's inside your bag?"
"Just a MacBook Air, why?"
"Looks so heavy?"
"There is Hadoop inside my Mac Air."
"Oh..." -
Took a day off and came back to find my team and I were moved from frontend to backend (which I always wanted to be).
Manager says we get to learn Java, Mongodb and even Hadoop.
I am so so excited.
But the only little hold back is, there is limited support for Java in sublime text and I should instead use eclipse. I had dozens of shortcuts customised with all those beautiful themes in sublime, but eclipse takes the fun out of coding(no offense to eclipse lovers).5 -
Setting up a single node Hadoop cluster. Then installing intellij idea, to find that it doesn't detect any installed jdk. Then uninstalled all jdks, and then reinstalled one, then Hadoop won't work. Now everything seems fine.
-
One of my colleagues who thinks he knows all about "big data": "I try to put everything in hadoop, that is my philosophy".
We don't even have a hadoop cluster. -
Question;
I have heard about Apache Hadoop for a while but never bothered to learn anything about it.
What can it be used for? Can I use it for hosting thousands of websites powered by php/hhvm?
I am starting to have a need for a really HA and High Performance solution that is futureproof too. My current solution is doing great also when it comes to performance and HA, but it is always nice to try out new things...
So the question is, can Hadoop be used in a hostingsolution?4 -
Back in my college, my professor was trying to demonstrate how to install hadoop using my laptop.
Now, he asked me to type my password
Even I thought it was asking my password. So I started typing it but my password was my crush's name (aakank....).
I typed till aaka and then realized what happened.
Fortunately I was able to handle the situation by typing
"Aakanakabaka" and saved my self.1 -
When you and your team invested a lot of their weekend time into the project to meet the clients unrealistic deadline and get not even a simple "thank you" from management...3
-
A friend went to a financial corp for a data analytic position. The interviewer proudly told him that they just got a new toy, and it's called Hadoop. My friend almost fainted.2
-
We had a tutorial on how to use spark/Hadoop.. part of the tutorial was the installation instructions for Ubuntu vms.
The Prof insisted we used an older version of Hadoop (v2.1.5), so naturally this required pulling older repos and older versions of java.
Naturally, some of the people in the class got some namespacing issues and garbage left from uninstalled packages.
Now, the tutorial was geared towards business/math people, not com sci. So most of the people didn't understand why apt didn't let them run certain commands (even though it very clearly just asked for them to run autoremove or autoclean, like in the "error message"). When the Prof and their "experienced TA" saw these messages, their recommendation was "make a new fresh vm".
The fuck? I heard that, run over and was like no. Just run the suggested command, it's literally a simple issue. And the guy didn't believe me. I had to sit him down, show him how I literally typed what the console was asking for, and everything just worked... The guy's response was "well that's Linux for you, its really complicated and can never trust anything, this time it worked, but next time it might not". Dude... Do you even know what you are saying? Like you are a supposed expert, least have some understanding of the package manager you are using. Maybe things will then be less "schrodinger cat".
God damn I can't wait to be out of this stupid fucking school. Never going back to academia.1 -
I don't like when client decide which tech use in the project. I got some weird tech request like:
1. Move existing database from postgresql to Hadoop because hadoop is Big Data (is kinda move from amazon rds to amazon s3 just why? have you index, cluster your postgresql table?)
2. Move from mysql to postgresql because mysql cause deadlock (maybe their previous developer just fucking moron)
In this situation we just explain why we don't use that and propose alternative solution. If they insist with their solution either ignore it or decide not continuing the project.5 -
I really regret switching to manjaro. So many things keep breaking, like my laptop won't sleep anymore, it stays up, whenever I plug in another display I get an error thrown at me. Among other weird behaviors (all screen related) that I can't seem to fix and make the experience feel like I'm running a very clunky win-poop machine.
On the other hand, setting up a very custom sddm theme and installing certain software like hadoop, rust, gimp, xfce tweaks and other things was such a breeze D: just "yay hadoop" and 90% of the work was done.
Grhhh... Wondering if I should accept defeat, and maybe switch to Linux MX or spend hours fixing what probably is a display driver issue that's pissing me off 😠2 -
Sometimes I think My company thinks like Hadoop ,
Instead of getting a experienced guy like 10 years exp . Let's take 5 with 2 years exp which should be more efficient !! 😒1 -
When your boss is hell bent on shoving the the words Hadoop , architecture and Revamp in every 3rd sentence. 🙄4
-
I eagerly wait for the day some people will realize and believe that Hadoop is a file system and NOT a database!!
-
I had a pretty good year! I've gone from being a totally unknown passionate web dev to a respected full stack dev. This will be a bit lengthy rant...
Best:
- Got my first full time employment dev role at a company after being self-taught for 8+ years at the start of the year. Finally got someone to take the risk of hiring someone who's "untested" and only done small and odd jobs professionally. This kickstarted my career, super grateful for that!
- Started my own programming consulting company.
- Gained enough confidence to apply to other jobs, snatched a few consulting jobs, nailed the interviews even though I never practiced any leet code.
- Currently work as a 99% remote dev (only meet up in person during the initialization of some projects.) I never thought working remotely could actually work this well. I am able to stay productive and actually focus on the work instead of living up to the 9-5 standard. If I want to go for a walk to think I can do that, I can be as social and asocial as I want. I like to sleep in and work during the night with a cup of tea in the dark and it's not an issue! I really like the freedom and I feel like I've never been more productive.
- Ended up with very happy customers and now got a steady amount of jobs rolling in and contracts are being extended.
- I learned a lot, specialized in graph databases, no more db modelling hell. Loving it!
- Got a job where I can use my favorite tools and actually create something from scratch which includes a lot of different fields. I am really happy I can use all my skills and learn new things along the way, like data analysis, databricks, hadoop, data ingesting, centralised auth like promerium and centralised logging.
- I also learned how important softskills are, I've learned to understand my clients needs and how to both communicate both as a developer and an entrepeneur.
Worst:
- First job had a manager which just gave me the specifications solo project and didn't check in or meet me for 8 weeks with vague specifications. Turns out the manager was super biased on how to write code and wanted to micromanage every aspect while still being totally absent. They got mad that I had used AJAX for requests as that was a "waste of time".
- I learned the harsh reality of working as a contractor in the US from a foreign country. Worked on an "indefinite" contract, suddenly got a 2 day notification to sum up my work (not related to my performance) after being there for 7+ months.
- I really don't like the current industry standard when it comes to developing websites (I mostly work in node.js), I like working with static websites (with static website generators like what the Svelte.js driver) and use a REST API for dynamic content. When working on the backend there's a library for everything and I've wasted so many hours this year to fix bugs and create workarounds related to dependencies. You need to dive into a rabbit hole for every tool and do something which may work or break something later. I've had so many issues with CICD and deployment to the cloud. There's a library for everything but there's so many that it's impossible to learn about the edge cases of everything. Doesn't help that everything is abstracted away, which works 90% of the time but I use 15 times the time to debug things when a bug appears. I work against a black box which may or may not have an up to date documentation and it's so complex that it will require you to yell incantations from the F#$K
era and sacrifice a goat for it to work properly.
- Learned that a lot of companies call their complex services "microservices". Ah yes, the microservice with 20 endpoints which all do completely unrelated tasks? -
Today my project manager called Hadoop a data warehouse and a Big Data lake in a meeting. I couldn't decide whether to laugh my ass off or spend the next 30 mins explaining to her what Hadoop actually is.2
-
Didn't know how to program but enrolled in a master's degree that needed that. So I attended the programming courses of the bachelor's degree that gave me just enough knowledge and just in time to save my ass writing a Hadoop MapReduce program! And I actually did well. All Java though.
These days I am banging my head against the wall learning Rust.1 -
Should i do hadoop big data course ? I am thinking of this summer to do on simplilearn. I am third year student undergraduate in IT. I am java guy and good in RDBMS.. Should i learn then?2
-
When your training system has 8 gigs of ram but you have to run HDFS, 3 PuTTY terminals, Eclipse, Firefox, a few explorer windows and sublime, it definitely feels like a tech Hurt locker. Every mouse click can fuck you up.
-
Why, oh why do we keep using such bizarre tech names e.g Hadoop, Pig, Oozie, Sqoop, Bleeg, Flume...5
-
How many languages does one have to learn...? Learnt C, C++ and Java because of college courses. Learnt HTML, CSS, and vanilla JS because I wanted to learn frontend. Now learning R for big data analytics. Today, I came to know that I need to learn more Java or start learning Python for Hadoop...!!
😧😵1 -
A project on hive....
Well the server has some error....
So i need to install hive on windows....
For that i need hadoop on windows...
For that i need either vmware or cygwin...
Done!
What was my project again?? -
I was a frontend developer, and I am new to hadoop or anything related to big data.
I am currently working as a Hadoop developer and I get to work on one of existing codebase also I am trying to recollect Java which I learnt during college.
Can u please provide me any inputs on how to get started with Hadoop, a personal view point on scope and future of Hadoop. A rough time span of how long it took for you to get out of the noob zone.
If you could provide me with a good tutorial or blog that would be awesome.
Thanks in advance1 -
I ain't getting any summer internship so thinking to do a good course on big data and hadoop. Can't find the free proper source for beginners😕! Any suggestions? Whats your plan btw..m thinking to dive into web dev as well.2
-
Windows is a shameful dev enviro but when you stuck in Africa, you gotto work with what you have. I dabble in node, R and hadoop and setting up environment and building modules on windows is a walk on hot ash in a desert.
To go around setting up of different dev enviros, i use a windows pre installed hack specially meant for that purpose. A new user account for each dev job. Kips my machine clean and sane while avoiding the blue screen.
After all, who still shares laptops today enough to use different user accounts😂😂1 -
There was this faculty who had a masters degree in big data, during my graduation.
She asked the whole class to install 'hadoop' on their machines as an assignment; in a situation where most of the class didn't even properly knew what Linux was.
I installed it and showed it to her.
She: Shreyans, can you help me install it on my system, I'm getting some errors.
Me: Sure ma'am.
She: On what Linux did you install it?
Me: Linux Mint ma'am.
She: But Mint's setup won't work for my Ubuntu. Do you have a setup for my Ubuntu?
Me: Whaaat!?
And I stood there frozen thinking what to reply..
#facepalm #facechair #facetable
#facehammer #facePanzer1 -
Debugging Spark errors is frustrating. Been running a model which takes an hour only to come across errors. And it's not even related to my code. Something to do with the cluster. FML1