Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "big fat data"
Screw the German Telekom!
I recently got a new home without internet so naturally, I went to an isp, Telekom. I went there a few weeks ago and was pleasantly surprised by the personal and the general competence. He told me they would send a technician to check my cable. So I thought great and went home. 1 A week passes, nobody shows up. I then went back to the shop and asked(someone different). He basically told me that such a service must be specifically asked for and a contract has to be signed. I then told
him his colleague told me no such thing, and that the technician should have checked up on my connection last week. He excuses him self and I signed the thingy.
Now you would imagine that this would have worked.
A week came and went and I got pissed. So I went back to the shop the guy from the first try was there. I Asked what happened, he types in his Computer. and. and. and. nothing. Apparently, the previous guy forgot, fucking forgot, to enter my request to their bloody System.
Now I asked if I can Just become a customer.
Guy: Sure, what speed is available in your region?
Me: I don't know...
Guy: Let me check
Guy: I can't see your speed the technician should have checked.
Me: Um, so, can he check?
Guy: Clearly you don't know what you want
(shorten but you get the Idea)
At this point, I really wanted to change isp so I went to Vodafone.
Lady comes up to me asks me a bunch of stuff and I explain I would want to change my phone, internet, tv, mobile and my friends mobile(I lost a bet once ^~^) to Vodafone.
What happened next I can't really explain, but she talked to her boss and "cheated" (how she calls it) on Vodafone and got me an AMAZING deal it is cheaper than Telekoms has waay more mobile data, faster Internet and I got a new phone :D.
And guess what she could fucking check, fucking check from here Computer my max internet speed.
I can only hope that the lady got a big fat commission for what she has done.6
Want to make someone's life a misery? Here's how.
Don't base your tech stack on any prior knowledge or what's relevant to the problem.
Instead design it around all the latest trends and badges you want to put on your resume because they're frequent key words on job postings.
Once your data goes in, you'll never get it out again. At best you'll be teased with little crumbs of data but never the whole.
I know, here's a genius idea, instead of putting data into a normal data base then using a cache, lets put it all into the cache and by the way it's a volatile cache.
Here's an idea. For something as simple as a single log lets make it use a queue that goes into a queue that goes into another queue that goes into another queue all of which are black boxes. No rhyme of reason, queues are all the rage.
Have you tried: Lets use a new fangled tangle, trust me it's safe, INSERT BIG NAME HERE uses it.
Finally it all gets flushed down into this subterranean cunt of a sewerage system and good luck getting it all out again. It's like hell except it's all shitty instead of all fiery.
All I want is to export one table, a simple log table with a few GB to CSV or heck whatever generic format it supports, that's it.
So I run the export table to file command and off it goes only less than a minute later for timeout commands to start piling up until it aborts. WTF. So then I set the most obvious timeout setting in the client, no change, then another timeout setting on the client, no change, then i try to put it in the client configuration file, no change, then I set the timeout on the export query, no change, then finally I bump the timeouts in the server config, no change, then I find someone has downloaded it from both tucows and apt, but they're using the tucows version so its real config is in /dev/database.xml (don't even ask). I increase that from seconds to a minute, it's still timing out after a minute.
In the end I have to make my own and this involves working out how to parse non-standard binary formatted data structures. It's the umpteenth time I have had to do this.
These aren't some no name solutions and it really terrifies me. All this is doing is taking some access logs, store them in one place then index by timestamp. These things are all meant to be blazing fast but grep is often faster. How the hell is such a trivial thing turned into a series of one nightmare after another? Things that should take a few minutes take days of screwing around. I don't have access logs any more because I can't access them anymore.
The terror of this isn't that it's so awful, it's that all the little kiddies doing all this jazz for the first time and using all these shit wipe buzzword driven approaches have no fucking clue it's not meant to be this difficult. I'm replacing entire tens of thousands to million line enterprise systems with a few hundred lines of code that's faster, more reliable and better in virtually every measurable way time and time again.
This is constant. It's not one offender, it's not one project, it's not one company, it's not one developer, it's the industry standard. It's all over open source software and all over dev shops. Everything is exponentially becoming more bloated and difficult than it needs to be. I'm seeing people pull up a hundred cloud instances for things that'll be happy at home with a few minutes to a week's optimisation efforts. Queries that are N*N and only take a few minutes to turn to LOG(N) but instead people renting out a fucking off huge ass SQL cluster instead that not only costs gobs of money but takes a ton of time maintaining and configuring which isn't going to be done right either.
I think most people are bullshitting when they say they have impostor syndrome but when the trend in technology is to make every fucking little trivial thing a thousand times more complex than it has to be I can see how they'd feel that way. There's so bloody much you need to do that you don't need to do these days that you either can't get anything done right or the smallest thing takes an age.
I have no idea why some people put up with some of these appliances. If you bought a dish washer that made washing dishes even harder than it was before you'd return it to the store.
Every time I see the terms enterprise, fast, big data, scalable, cloud or anything of the like I bang my head on the table. One of these days I'm going to lose my fucking tits.10
I work as a data engineer in my company. My senior calls himself data scientist- he is 29 years old and recently did one MOOC on data science!
I wonder when my colleagues will find out about how much he really knows.
Till then I am cleaning and arranging his data, while he sits and earns a big fat package by citing one data scientist tag to his profile!! -_-1
Data structure and Analysis
For experienced ones, add "System Design".
...path for big fat packages.
Connecting local test server with live db for testing purposes. Needs 10 min to start up because much data is preloaded.
Checked against 0 instead of null in code. Big fat null pointer error greets. Another 10 mins lost.