Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "big data"
My college organised some interview with a company, with the whole demn class. We went there, it was quite far away (50km) and the CEO invites us to a meeting room.
Where he bores me for 2 hours talking about their projects in argiculture and NSA like spying systems at tankstations.
They were caputuring license plates at gas stations and with that information gather data about the person, such as salary (by looking at their car), house adres ect. All without people knowing. And than targeting them with specific ads and offers.
The class of sheep were super excited but it pissed me off. Because he told it like it was some awesome advancement in technology that none of us could probably ever do.
He was demeaning us, saying we would do some simple wordpress sites there and other things. We are probably not good enough forc te big stuff.
Asking him some really hard questions about his projects made him so pissed he almost wanted to kick me out.
When it was finally over, there was some test that you have to do if you want to work there. If you were good enough at the test, you could!!!! (YEEY)
Uhm, I said; no thank you I dont want to work here.
Later I talked to my classmate and friend who always thinks he's better then everyone in class even tho he barely understands OOP programming. He was asking me if he should try to get the internship. I told him; dont. They have no value for us and they think they are the greatest company on the planet.
The fucking idiot go so pissed, he stopped talking to me alltogether and blocked me everywere. I AM NOT EVEN JOKING. Just because I gave my FUCKING opinon about a company he likes for no reason.
So this idiot does the test (which was fucking simple btw, I did it too and compared the results and I had 95%) He gets invited for another interview and gets told he will be paid 200 euro's per month 😂. and a free meal everyday!! 😪 hahaha . That doesnt even cover commuting costs!
My "friend" told him that the train costs more every day. You know what the CEO said? "Yeah but you can learn so much here the also brings value and you're just a last year student. But I think you are really brave for asking more"
So in the end, he couldnt take the internship and I was fucking right. Really I hate these kinds of companies thinking they are heaven on earth when they are clearly not.
I am happy I told them no before putting my dignity on thd line.14
What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!
I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.
Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.
So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.
Lunch time already. But the fun isn't over yet!
I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.
After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."
So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".
While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.
Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.5
I worked once in a company which had this tourist app which should show places on map of the city. Unfortunately it slowed the App down to load more then a couple of places. Their solution was to limit the number of loaded places to teb and prohibited zomming out. I made it handle thousands of places at the same time. Main reason for the Performance issue was, that they sent all data they had about places big, big json objects with large text blobs) to the frontend. This part was easy, I instead sent only the data needed for the map like coordinates and icon type obviously. But still the backend struggeled hard with many objects from the DB, because they built a really shitty orm or what ever this was supposed to be: every line of data retrieved from the DB was immideatly wrapped in some class wich direved from another class which had some magic methods in it which caused some absurd loops over all other obejcts and even more DB queries in unexpected moments and also in the fucking constructor. So it turrned out that the map issue was only the top of the iceberg, since using any data from the DB was extremely expensive. The hard part was to understand the insaness of this abnormination and find the bottlenecks.8
DEAR CTOs, PLEASE ASK THE DEVELOPER OF THE SOFTWARE WHICH YOU ARE PLANNING TO BUY IN WHAT LANGUAGE AND WHAT VERSION THEY ARE WRITTEN IN.
Background: I worked a LONG time for a software company which developed a BIG crm software suite for a very niche sector. The softwary company was quite successfull and got many customers, even big companies bought our software. The thing is: The software is written in Ruby 1.8.7 and Rails 2. Even some customer servers are running debian squeeze... Yes, this setup is still in production use in 2022. (Rails 7 is the current version). I really don't get it why no one asked for the specific setup, they just bought it. We always told our boss, that we need time to upgrade. But he told every time, no one pays for an tech upgrade... So there it is, many TBs of customer data are in systems which are totally old, not updated and with possibly security issues.9
There's nothing like sitting on the edge of your seat when you see a monster batch of records get sent updates.
This system was built 5 years ago and it's "peak" batch size has been < 400 records in a day, it usually sits around < 100.
It's not a big system and just runs in the background. So yea small numbers for this guy.
today though, I thought something fell down and shit its self, someone decided to add a a few thousand records to this thing and update a fuck tonne of data (for this system anyway)😬
The damn thing is standing it's ground and churning, but fuck, the scale of things is beyond what we ever thought it would have to deal with at any one time.
Build for the insane benchmarks kids, one day... someone's going to drop an elephant on it.5
BUZZWORD BUZZWORD AAAAAH
AAAAH BUZZWORD HERE BUZZWORD THERE4
Our team really needs some workflow arrangement, and this time it was me who screwed up.
So we have to push an update to the Play Store and the App Store the Friday, the app is well tested on test environment then production environment, we got the ok so I uploaded a build, the app management team then continued the process of publishing..
During the weekend the app was approved and live to almost 500k user that can receive the update.
I got a phone call from the Project Manager at almost midnight, the time was really suspicious so I answered.
- Me: Hello.
- PM: Hi, sorry to call you now but the app is live and we have a problem.
- Me: what kind of problem? Let me check.
So I updated the app on my phone and opened it while I am on call.. I almost had heart attack!! WE PUBLISHED A VERSION POINTING TO THE TEST ENVIRONMENT. Holly shit
- Me: shit call the app management team NOW.
Eventually we removed the app from sale (unpublished it) and we submitted a new version immediately, once it was approved the next day we made the app available again (so for those who didn’t update yet, there will be no update to a faulted version, and no new users landing to a version with test data), I received one or two calls from friends telling me why the app is not on the store (our app is used nationally, so it’s really important).
Thank God there was no big show on twitter or other social media.. but it’s really a good lesson to learn.
I understand this is totally my fault, thankfully I didn’t get fired 😅4
A long long time ago ( 2007 I think ) I worked for a company that made landing sites, so basically an email campaign would go out, users would be sent to a 1 page website with a form to capture their data, ready to be spammed even more. You know how it was back then.
So I worked with a guy who we had just hired, I didn't do the hiring but his CV checked out, so I gave him one of my tasks. Now most pages were made with js and html, with a PHP backend ( called with Ajax). Now this guy didn't know PHP so I was like all good, ASP works too at the end of the day we don't judge, we do like 2 or 3 of these a day and never look at them again. So he goes of and does is thing.
3 weeks later, the customer calls up to me they still haven't received their landing page. Ok so he probably forgot to email the customer np, I tell him to double check he has emailed the customer. Another week goes by end the customer calls back, same problem. At this point I'm getting worried, because we're days away from the deadline and it was originally my task.
So I go back to the guy and I tell him I want that landing page so I can send it myself, half thinking to myself that we had a freeloader, that guy that comes in to companies for 3 weeks, doesn't work, but still cashes his pay. But no, this was much worse.
So he tells me he has finished yet. I ask him why, what's the blocker ? You had 4 weeks to tell me you were blocked and couldn't progress. And his answer was simply, because I wasn't blocked I have been working on it this whole time. So I tell him to zip his project up and email it to me. We didn't do SVN or git back then, simply wasn't worth it. So he comes back to me and says the email server is telling him attachments can't be bigger then 50mb. At this point I'm thinking he didn't properly sized the art or something, so I give him a flash drive to put it on.
When I then open the flash drive, the archive is 300mb, thinking to myself, the images weren't even that big to begin with.
So I open it up, and I don't even find any images, just a single asp page. About 500mb. When I opened that up and it finally loaded, I saw the most horrendous things ever.
The first 500 lines was just initializing empty vars. Then there was some code that created an empty form with an onChange event that submits the form. After that.. it was just non stop nested if's. No loops, no while, for, foreach, NO elseif's, just nested if's, for every possible combination of the state the form could be in. Abou 5000 of them, in a single file. To make matters worse, all the form ( and page ) layout was hardcoded in the if's. Includes inline css, base64 encoded images, nothing but as dynamic, based on the length of the form he changes the layout, added more background etc. He cut the images up for every possible size of the page and included them in the code.
I showed it to my boss, he fired the guy on the spot. I redid the work from scratch, in under 4 hours. Send it to the client. they had no ammends to make, happy as Larry. Whish I kept the code somewhere.
Morale of the story, allways do a coding test on interviews, even if small things just to sanity check.4
I think I've reached some kind of job nirvana. My coworkers and I all complain about our work. We're overworked, underappreciated, underpaid, and and have to deal with all sorts of bullshit all the time. Pretty much everyone who has been on the team longer than a year is talking about quitting.
But I started at this company as a level 1 tech support phone technician before I transferred into the DevOps side of things, and that tech support job was SO much worse. Way more stressful, way less pay, mandatory overtime, horrible scheduling, being forced to remain calm while people hurl insults at you over the phone, and it was a dead-end job with a high turnover rate and almost no opportunities for advancement of any kind.
And every time I think back on that job, I realize that what I have now is actually pretty great. I'm paid well (still underpaid for the job I do, but catching up really fast due to my current boss giving me several big raises to keep me from quitting lol). I deal only with other tech people like developers and data scientists so no more listening to salesmen insult me on the phone. I'm not in any sort of customer service role so I can call people on their bullshit as long as I'm professional about it. I'm salaried so they can't make me work horrible shifts. 99% of my days are a normal 9-5 workday. I actually have a reliable schedule to plan around.
People treat me like the adult that I am.
I'd get a similar experience at other, better-paying companies, for sure, but what I have now is still pretty great.
I'm sure I'll be back in a few days to rant about more nonsensical bullshit and stress, but for now I'm feeling the zen.
Situation: I have a love hate relationship with python due to the lack of types as I have in more established languages such as C#, Java and shit even TypeScript
Situation (cont): A rather large codebase that i have developed for multiple processes at work run on Python.
I don't hate it, I just don't absolutely love it, there is a lot of things to like about Python, but man I do have some conflicts with it, I have been facing out to use other solutions that feel scripty, such as the newer versions of C# with .net, but I would say that about 80% of our codebase runs on Python, the rest is PHP.
My codebase running on Python is huge, and they do a lot from automation scripts, to data gathering and database management, never had I been bitten with the "oh noes is so slow" bug since my code is not Google level big, for everything else Python seems rather fast imho
I dunno, big time love hate relationship9
I don't like when client decide which tech use in the project. I got some weird tech request like:
1. Move existing database from postgresql to Hadoop because hadoop is Big Data (is kinda move from amazon rds to amazon s3 just why? have you index, cluster your postgresql table?)
2. Move from mysql to postgresql because mysql cause deadlock (maybe their previous developer just fucking moron)
In this situation we just explain why we don't use that and propose alternative solution. If they insist with their solution either ignore it or decide not continuing the project.5
I’ve been interviewing at a few companies lately. I’m a dev with ~6 years of experience with a specific language. Most of the experience comes from working in companies that developed their own software, not talking about cms stuff. Analytical, data tracking systems. Now working at a fintech. I’ve got an offer to work as a senior developer in a smaller tech team, with more salary. I’ve approached the current company about the offer and they told me that they don’t think I’m a senior dev and rather a strong mid level dev. The Hr also told me to think about if I’m really a senior and if the other companies expectations would be met. They would increase my salary, but not quite match it. It’s not too far off though. Their reasoning for this was that you need a lot of experience with their product (which does not correlate with seniorness of a developer, only the worth of specific employees for a company IMHO) and system architecture design. The problem is that we don’t see any tasks that could implement any system design for as log as I’ve worked here, so I don’t see how I could work into a senior role at this company. Of course imposter syndrome kicked in and I’m triple guessing myself if I should join the other company as a senior now. How should I aproach this? The current company is stressful to work at because of big workload, a lot of my coworkers think the same thing about the workload.11
So, I feel wayyy behind the tech curve right now.
The SSD implementations you see online, they're still just a bunch of seperate sort of chaos machines that contain the standard perceptron-like model of a weight, cost, and bias right ? They just kind of inferred their values by training like any other neural network, in its sep-erate parts and just fed pieces of output data generated by other parts of the neural network to it right ?
I mean it implements with pytorch so its basically a really big array of tuples in a sense that are maniupulated in a specific way.
and then CNN's just feed data back into another trained piece of the model right ?
I'm curious because object classification is about the ONLY thing I've seen work even close to properly lol
there is just so much fraud these days. sigh.
and so many lamentable tech choices and attempts... like node lol