Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "test in production"
Friend: "What is devRant?"
Me: "A place where programmers tell jokes and complain."
Friend: "Why dont you just do that irl?"
Me: "Because we never test in production"15
I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.
The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.
We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.
Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.
Needless to say, he was never allowed to touch code on a prod server ever again.9
Oh, man, I just realized I haven't ranted one of my best stories on here!
So, here goes!
A few years back the company I work for was contacted by an older client regarding a new project.
The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.
Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.
Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.
Long story short, a lot bigger than what we were expecting based on the initial brief.
The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.
They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.
Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.
So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.
So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.
We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.
As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.
So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...
Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.
Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!
[Continues in the comments]26
A more experienced friend told me
"don't be a pussy, test in production"
I'm the one fixing the bugs, not him8
So lot of people ask me here what is Testing?
Well, I have decided to post about it instead of answering every request individually as it is tiresome. So here we go...
- Why to test?
Testing is breaking down a piece of code so that all the possible logical and functional errors can be detected while in development phase, so that the disaster can be prevented in production.
- How to test a piece of code/software?
There are various types of testing like unit, sanity/smoke, functional, regression, performance, security, penetration to name a few. Each has it's own significance and to know more you can Google/DDG them.
- What to test?
We can test any and everything. You can test code, UI, UX, speed of application, database flow, logical flow, functional flow, et cetera.
Testing is an attitude, not a skill. Cannot be taught but only developed by one ownself.
To be good at testing, one needs to have a good design and business sense, apart from technical and functional knowledge.
Many will say, to test one needs to think outside the box. Fuck that. There is no box. Everything is a sandbox and do whatever the fuck you want to, to break things.
Most importantly, thinking from other person's perspective is critical as this helps you think what can an user do with your software. Trust me, humans are idiots and can literally fuck your software doggy style and make you wonder why your code isn't working?
To build an idiot proof software, we need to think beyond human levels.
Never think the obvious. NEVER.
Always try what you think can never happen. System will break, I promise.
Finding loopholes is way easier than you think. Fixing them is surely a challenge.
Moreover, to survive in corporate world leaning the processes is important. The SDLC cycle is what makes a software great.
And today I realised that no role is stupid. Testing is looked down upon in my country. Possibly, IT is the only field where testing has least value as compared to other places like food testing or automobile crash testing et cetera.
And I, myself used to think support is a stupid job requiring no skills. But today I talked to a friend in support and realised that they have more technical skills than a tester.
Everyone is important and we shouldn't look down upon any person or role. Nobody is superior, nobody is inferior. We all are equal.
Together we shall work as a team and a great teamwork can achieve wonders.
Hope this helps. Thak you for reading and do provide your feedback as it shall help me improve.
P.S.: last week somebody tagged me to a new comer's post seeking advice for testing. I don't recollect that so please tag them here again. Thanks.
Edit: I feel like and I might have, missed out some points so please excuse me for that.21
My biggest dev blunder. I haven't told a single soul about this, until now.
So, I was working as a full stack dev at a small consulting company. By this time I had about 3 years of experience and started to get pretty comfortable with my tools and the systems I worked with.
I was the person in charge of a system dealing with interactions between people in different roles. Some of this data could be sensitive in nature and users had a legal right to have data permanently removed from our system. In this case it meant remoting into the production database server and manually issuing DELETE statements against the db. Ugh.
As soon as my brain finishes processing the request to venture into that binary minefield and perform rocket surgery on that cursed database my sympathetic nervous system goes into high alert, palms sweaty. Mom's spaghetti.
Alright. Let's do this the safe way. I write the statements needed and do a test run on my machine. Works like a charm 😎
Time to get this over with. I remote into the server. I paste the code into Microsoft SQL Server Management Studio. I read through the code again and again and again. It's solid. I hit run.
Wait. I ran it?
With the IDs from my local run?
I stare at the confirmation message: "Nice job dude, you just deleted some stuff. Cool. See ya. - Your old pal SQL Server".
What did I just delete? What ramifications will this have? Am I sweating? My life is over. Fuck! Think, think, think.
You're a professional. Handle it like one, goddammit.
I think about doing a rollback but the server dudes are even more incompetent than me and we'd lose all the transactions that occurred after my little slip. No, that won't fly.
I do the only sensible thing: I run the statements again with the correct IDs, disconnect my remote session, and BOTTLE THAT SHIT UP FOREVER.
I tell no one. The next few days I await some kind of bug report or maybe a SWAT team. Days pass. Nothing. My anxiety slowly dissipates. That fateful day fades into oblivion and I feel confident my secret will die with me. Cool ¯\_(ツ)_/¯13
This was too funny to not share.
Credits to @stonecold on twitter
This is why you don't test in production, folks!7
[This makes me sound really bad at first, please read the whole thing]
Back when I first started freelancing I worked for a client who ran a game server hosting company. My job was to improve their system for updating game servers. This was one of my first clients and I didn't dare to question the fact that he was getting me to work on the production environment as they didn't have a development one setup. I came to regret that decision when out of no where during the first test, files just start deleting. I panicked as one would and tried to stop the webserver it was running on but oh no, he hasn't given me access to any of that. I thought well shit, I might as well see where I fucked up since it was midnight for him and I wasn't able to get a hold of him. I looked at every single line hundreds of times trying to see why it would have started deleting files. I found no cause. Exhausted, (This was 6am by this point) I pretty much passed out. I woke up around 5 hours later with my face on my keyboard (I know you've all done that) only to see a good 30 messages from the client screaming at me. It turns out that during that time every single client's game server had been deleted. Before responding and begging for forgiveness, I decided to take another crack at finding the root of the problem. It wasn't my fault. I had found the cause! It turns out a previous programmer had a script that would run "rm -rf" + (insert file name here) on the old server files, only he had fucked up the line and it would run "rm -rf /". I have never felt more relieved in my life. This script had been disabled by the original programmer but the client had set it to run again so that I could remake the system. Now, I was never told about this specific script as it was for a game they didn't host anymore.
I realise this is getting very long so I'll speed it up a bit.
He didn't want to take the blame and said I added the code and it was all my fault. He told me I could be on live chat support for 3 months at his company or pay $10,000. Out of all of this I had at least made sure to document what I was doing and backup every single file before I touched them which managed to save my ass when it came to him threatening legal action. I showed him my proof which resulted in him trying to guilt trip me to work for him for free as he had lost about 80% of his clients. By this point I had been abused constantly for 4 weeks by this son of a bitch. As I was underage he had said that if we went to court he'd take my parents house and make them live on the street. So how does one respond? A simple "Fuck off you cunt" and a block.
That was over 8 years ago and I haven't heard from him since.
If you've made it this far, congrats, you deserve a cookie!6
Seven months ago:
Project Manager: - "Guys, we need to make this brand new ProjectX, here are the specs. What do you think?"
Bored Old Lead: - "I was going to resign this week but you've convinced me, this is a challenge, I never worked with this stack, I'm staying! I'll gladly play with this framework I never used before, it seems to work with this libA I can use here and this libB that I can use here! Such fun!"
Project Manager: - "Awesome! I'm counting on you!"
Six months ago:
Cprn: - "So this part you asked me to implement is tons of work due to the way you're using libA. I really don't think we need it here. We could use a more common approach."
Bored Old Lead: - "No, I already rewrote parts of libB to work with libA, we're keeping it. Just do what's needed."
Cprn: - "Really? Oh, I see. It solves this one issue I'm having at least. Did you push the changes upstream?"
Bored Old Lead: - "No, nobody uses it like that, people don't need it."
Cprn: - "Wait... What? Then why did you even *think* about using those two libs together? It makes no sense."
Bored Old Lead: - "Come on, it's a challenge! Read it! Understand it! It'll make you a better coder!"
Four months ago:
Cprn: - "That version of the framework you used is loosing support next month. We really should update."
Bored Old Lead: - "Yeah, we can't. I changed some core framework mechanics and the patches won't work with the new version. I'd have to rewrite these."
Cprn: - "Please do?"
Bored Old Lead: - "Nah, it's a waste of time! We're not updating!"
Three months ago:
Bored Old Lead: - "The code you committed doesn't pass the tests."
Cprn: - "I just run it on my working copy and everything passes."
Bored Old Lead: - "Doesn't work on mine."
Cprn: - "Let me take a look... Ah! Here you go! You've misused these two options in the framework config for your dev environment."
Bored Old Lead: - "No, I had to hack them like that to work with libB."
Cprn: - "But the new framework version already brings everything we need from libB. We could just update and drop it."
Bored Old Lead: - "No! Can't update, remember?"
Bored Old Lead: - "You need to rewrite these tests. They work really slow. Two hours to pass all."
Cprn: - "What..? How come? I just run them on revision from this morning and all passed in a minute."
Bored Old Lead: - "Pull the changes and try again. I changed few input dataset objects and then copied results from error messages to assertions to make the tests pass and now it takes two hours. I've narrowed it to those weird tests here."
Cprn: - "Yeah, all of those use ORM. Maybe it's something with the model?"
Bored Old Lead: - "No, all is fine with the model. I was just there rewriting the way framework maps data types to accommodate for my new type that's really just an enum but I made it into a special custom object that needs special custom handling in the ORM. I haven't noticed any issues."
Cprn: - "What!? This makes *zero* sense! You're rewriting vendor code and expect everything to just work!? You're using libs that aren't designed to work together in production code because you wanted a challenge!?? And when everything blows up you're blaming my test code that you're feeding with incorrect dataset!??? See you on Monday, I'm going home! *door slam*"
Project Manager: - "Cprn, Bored Old Lead left on Friday. He said he can't work with you. You're responsible for Project X now."18
Confession of the day:
1. I work in release mode
2. I work on the main branch only
3. I test on production13
I am fucking dying of laughter right now. 😁
Today I got a push message of the invoicing app I use from time to time and the message literally just said "lol" (without even the usual pre-fix of the app name or anything).
So after not figuring out where that could have come from and obviously theres no private messaging etc. in that app, I contacted support and they reacted surprisingly good and at same time hilariously good; they pushed now a team towards investigating that, as apparently I wasn't the only one.
I wonder who fucked up and literally pushed "lol" to thousands of people. 😂8
So Facebook declared millions of people dead ... I guess that's what happens when you test in production7
TL;DR: The bitch ruined my Friday.
I was in a really happy mood when I arrived at office in the morning today, because Friday.
I was assigned a stupid production error to test. Being a proactive person, I completed the task only to get more on my plate.
Why? Because fuck me that's why!!! All of my main task was aside throughout the day which has been pending since ages. The loser I work for, never allocates time for my main task and end of the day, yells at me that my task is pending.
Fuck the cock sucker because he is sucks at work delegation.
So, when the next stupid task was allocated, after analysing the entire scene, I realised this is something which I have never worked on and have absolutely no clue about. Rather this should be assigned to that team member who has already worked on it, because they'll be able to complete it efficiently.
But this bitch, forced me to do the task and the said team member roamed around having gala time. This pissed the shit out of me.
Anyway, I tried to take up the challenge and asked for some help from the said bitch, who denied completely, asking me to refer the documentation which was written like shit of a rat.
I somehow managed to make my way through it but ended up failing countless time.
I seek the bitch again for some help. This time I made it clear that I am unable to complete the task and I am finding it difficult without any help.
The dick sucker guides me half way, asks hundreds of demoralising questions, making me question my self esteem and confidence.
I keep my calm but that asshole keeps demeaning my actions and now I loose shit.
I start behaving rudely and back answering slowly. The dumb bitch doesn't get the hint (we talk over chat as she is in another location).
I told myself to stop working until she helps me and called the day off. She never cared to revert and now the task is pending which will be discussed again on Monday.
She refused to listen at any cost. And that's where I realised that Narcissistic Dogmatic Hypocrites are worst people to work for.
Just wish me luck that I don't break her jaw and finish the task without creating a scene.
Such people should be fucked by a wild boar in their ass and be made to suck a donkey's dick.
Ruined my Friday.26
IF LIVES DEPEND ON A SYSTEM
1. Code review, collaboration, and knowledge sharing (each hour of code review saves 33 hours of maintenance)
2. TDD (40% — 80% reduction in production bug density)
3. Daily continuous integration (large code merges are a major source of bugs)
4. Minimize developer interruptions (an interrupted task takes twice as long and contains twice as many defects)
5. Linting (catches many typo and undefined variable bugs that static types could catch, as well as a host of stylistic issues that correlate with bug creation, such as accidentally assigning when you meant to compare)
6. Reduce complexity & improve modularity -- complex code is harder to understand, test, and maintain
My boss last week backed me up with a client. I was working in a production enviorment and the client refused to test the changes made when I told them. So we things started to go wrong and they called to my boss complaning.
Well you are right. The things are broken but he (me) stayed last night waiting for your response and you didn't gave one. So he is going to work only in the afternoon and you will have to wait.
I must buy a beer for my boss.2
!dev I'd just helped a client cut over to a new fiber connection and then left for Vegas, about 2 days into the trip my wife and I decided to hit a breakfast spot that had bottomless mimosa's, which was of course a claim we had to test.
As we are walking(stumbling) out of the restaurant I get a call that the connection has crashed and the entire car dealership is unable to sell cars, which they tell me is important functionality.
So I make it up to my room and break out the laptop, luckily the mgmt interfaces are still available externally so I'm able to log in and then have the fun challenge of 1) not falling off of my chair 2) not accidentally making a change that kills what connection I have in and 3) fixing their actual issue.
Took me almost an hour to find a simple OSPF issue but at least got them working and happy. However by that time I was beginning to sober up, which is the absolute worst thing that can happen while day-drinking and ended up basically causing me to be be hung-over for the rest of the night, including my wifes friends wedding, which she wasn't thrilled about...
The moral of this story is to make sure to NOT stop drinking while dealing with unexpected production impacting events.1
I extracted a tangled action to its own api, and wrote a test for it.
The test failed.
I added debugging, more debugging, all the debugging. It still fails. But I can at least see why it fails!
It turns out the api finds and updates the wrong user. It finds and updates the wrong user EVEN WHEN THERE ARE NO OTHER USERS.
WHAT THE SALAMI.
Also, the user lookup it uses is extremely roundabout and takes several seconds with ~2 million users. Normally I'd fix the lookup, but it has been in production for several years, and I'm terrified it will break something if I fix it.
I was only seventeen back then and I was a Java Developer Intern, not knowing much about enterprise oriented coding.
The project leader in our dev team saw a lot of potential and passion in my work, but was convinced I wasn't taught enough to do the right thing.
I was mainly doing shitty mappers and services back then, which were somewhat used but never lasted long and were ditched a few months later, which always bummed me out. I wanted to make an impact on REAL projects that would deploy into production.
So Mister Mentor (GDPR forbid to use the actual name), who was always first to come and last to leave the office, taught me what it means to code for real.
We stayed after 5pm until 7-8pm multiple times a week and he taught me in a deeply understanding and calm way how to:
- Git (SVN)
- Unit Test
And most importantly:
- How to debug like an absolute BOSS
(We even debugged native Java Libraries just for fun to see if we could break them)
Fast-forward a month later and little intern me made his first commit on production.
Without Mister Mentor, I wouldn't be half as good of a developer as I am today.3
Some years back I was working in a project that essentially dealt with all things related to foreigners and foreign affairs in Switzerland. You could manage entry visas, work permits, citizenship, international warrants, Interpol requests, etc.
One of the test managers (from client side - i.e. the government) was once manually "testing" and mixed up the production and test instance, to both of which he was logged in at the time.
The test case then ended up setting up an entry ban against himself, as he used his own name for testing...
Next time he returned from vacation the border control at the airport were like "Uhm, Sir, we can't let you into the country. Please come with us." :D :D
(He managed to clear that up in end, I dare say, though, that he learned his lesson.)13
So we hired an intern and his first task was to change a few things in email layout for our client, which is an investment bank.
I told to one of my developers to make his local database dump and setup the project for an intern. When intern completed the task, my developer thought that title "Dow Jones index crashed" was pretty funny title for a test.
What he didn't thought through enough, is that he forgot to configure fake SMTP server and he had production database dump with real email addresses.
I had really awkward 20 minutes conversation with our client. Fuck my life.4
Of course they don't use git. And also they don't use SSH all changes get committed by FTP.
When I started he gave me root access and I had to clone the whole fucking thing, wich was about 2gb, via FTP.
He stumbled when I told him, that I will test all changes first on my local machine. They were used to work in production.
I started this new freelance project where I am building some android libraries for the client. Anyways, during meeting I was about to present my results and suddenly backend seemed to be down. I looked into the round "are your servers down?"
Team Lead: "Yeah our cto, also our only backend dev, is developing a new feature."
Me: "Okay but why is production down?",
Team Lead: "Ah dont worry we always test on production. We have a pretty solid hardware, we will even upgrade it soon!"
Me:"... How about you just separate your stage environments and have a develop environment?"
Client: "see, this is where our strength is. We dont need a develop stage we have very strong hardware and our backend is fully in PHP"
Thanks God I'm a freelancer3
I worked for over 13 hours yesterday on super-urgent projects. I got so much done it's insane.
1) the printer auto-configuration script.
2) changing Stripe from test mode to live mode in production
3) website responsiveness
I finished two within five minutes and pushed to both QA and Production. actually urgent, actually necessary. Easy change.
The printer auto-configure script was honestly fun to write, if very involved. However, the APIs I needed to call to fetch data, create a printer client, etc... none of them were tested, and they were _all_ broken in at least two ways. The CTO (api guy in my previous rant) was slow at fixing them, so getting the APIs working took literally four hours. One of them (test print) still doesn't work.
Responsiveness... this was my first time making a website responsive. Ever. Also, one of the pages I needed to style was very complicated (nested fixed-aspect-ratio + flexbox); I ended up duplicating the markup and hacking the styling together just to make it work. The code is horrible. But! "Friday's the day! it's going live and we're pushing traffic to it!" So, I invested a lot of time and energy into making it ready and as pretty as I could, and finally got it working. That page alone took me two hours.
The site and the printer script (and obv the Stripe change as well) absolutely needed to be done by this morning. Super important.
1) Auto-configure script. Ostensibly we would have an intern come in and configure the printers. However, we have no printers that need configuring, so she did marketing instead. :/ Also, the docs Epson sent us only work for the T88V printer (we have exactly one, which we happened to set up and connect to). They do not work for the T88VI printers, which is what we ordered. and all we'll ever be ordering. So. :/ I'll need to rewrite a large chunk of my code to make this work. Joy :/
2) Stripe Live mode. Nobody even seemed to notice that we were collecting info in Test mode, or that I fixed it. so. um. :/
Well. That deadline is actually next Wednesday. The marketing won't even start until then, and I haven't even been given the final changes yet (like come on). Also! I asked for a QA review last night before I'd push it to production. One person glanced at it. Nobody else cared. Nobody else cared enough to look in the morning, either, so it's still on QA. Super-important deadline indeed. :/
I feel like Alice (from Dilbert) after she worked frantically on urgent projects that ended up just being cancelled. (That one where Wally smells that lovely buttery-popcorn scent of unnecessary work.)
I worked 13 hours yesterday.
Old boss story. This guy was nice but a terrible boss. Also relevant, he has a background in IT so should know better.
Him: So when you wanna check a password is correct you just unhash it in the database?
Me: Hey we should be doing unit and integration testing at a minimum to lower bugs.
Him: We don't need those, we're not a bank. If a problem comes up we just fix it and push to production.
(A while later)
Him(in email): Why do we keep getting bugs reported. Don't you devs test your code.
I was mildly annoyed at that one.
Him: We're always over budget on projects, how can we fix this.
Me: What if we increase our quotes.(technically there are other ways as well but not really possible at that time)
Him: We can't do that, clients won't want to pay.
Me: *finishing off my handover as I'm leaving for a new job*
Him: Wow you do a lot of work3
So far all designers I worked with do the following:
1. Use "creativity" to come up with stuff that the system does not allow implementing, for example: Changing clock color in mobile statusbar to Blue!
2. Use "creativity" to come up with a heavily customized calendar for a windows software which requires building the control from scratch, but they explain their creativity by saying: Can't you use CSS?
3. Provide iOS only design which follows android guidelines and refuse to provide android styles for at least pages that to be handled differently on each platform, for example, we had a checkout page in an app, and they wanted the same style for both WITHOUT building custom control for it, they said: Can't you use the android custom control inside iOS
4. They design for a website and send same mockups for me to implement on mobile apps, the problem is a web page runs on a big screen when the mobile app doesn't have room for half the stuff they designed but they must look exactly the same as website !!
5. They send entire PSD with no color codes and say: You can extract icons, and colors from psd ... When they should provide them as per our request which is: SVG for Android and PDF for iOS with the color codes, but no, they are lazy!
6. They ask the team to create a page in the app which is almost production ready just so that they test different font sizes and see how it will look on the phone
7. Same as #6 for images that contain text
The list goes on, but those are by the far the ones that made me one step away from resigning, some of them made me resign...6
[3:18 AM] Me: Heya team, I fixed X, tested it and pushed to production. Lemme know what you think when you wake up.
[6:30 AM] Me: Yo, I just checked X and everything is peachy. Let me know if it works on your end.
[9:14] Colleague A: Whoop! Yeah! Awesome!
[9:15] Boss: Nice.
[9:30] A: X doesn't work for me.
Me: OK, did you do M as I told you.
Me: *checks logs and database, finds no trace of M*
Me: A, you sure you did M on production? Send me a sreenshot plz.
A: yeah, I'm sure it's on production.
Me: *opens sreenshot, gets slapped in the face by https://staging.app.xyz*
Me: A, that's staging, you need to test it on production.
A: right, OK.
[10:46] A: works, yeah! Awesome, whoop!
[10:47] Boss: Nice.
Me: Ok! A, thanks for testing...
Me: *... and wasting my time*.
[10:47:23] Boss: Yo, did you fix Y?
Courageous/snarky me: *Hey boss, see, I knew you'd ask this right after I fixed X knowing that I could not have done anything else while troubleshooting A's testing snafu since you said 'Nice' twice. So, yesterday, I cloned myself and put me to work in parallel on Y on order fulfill your unreasonable expectations come morning.*
Real me: No, that's planned for tomorrow.
Aaaah, I fucking love it to death, when customers spontaneously decide to hire a separate, unrelated company to add new content pages to the website developed by our company.
That furuncle of a company must have had real pro devs to just create a new /html folder, dump their shit content in there and just manually add links in the existing CMS pages.
As you might already have expected, the /html folder contains:
- static *.html files for every page
- inline CSS in the *.html
- the crappiest PHP mailing script I have ever witnessed
- images with random resolutions, mostly too small
The layout of these puke-ridden pages obviously doesn't fit neither the existing color palette, nor has anything common with the current layout or typography at all.
These bastards don't even use Git!
Come on, dear customer, could you PLEASE fucking NOT hire a completely separate company to do OUR job?
I had to compare the whole deployment folder with our repo to find out what else these brain-damaged cunts changed in our code!4
It's about a guy that knows better.
I was working as a subcontractor on a bigger system. We (subs) were not allowed to deploy code, we had to wait for contractor to deploy.
One day I got an email that my code is bugged and that my feature is not working on production. I checked it on test env, everything was fine. Then I checked if the code I wrote was deployed. It was not.
I send an email explaining that if they deployed my code it would be working. Then I got a response. There was a bug in my code.
Another email. I asked how would they know? Do they have a test on their environment that failed?
No. There is one guy that READ my code and he said it should not work, so he will not deploy it. He was not a programmer, he was a business consultant responsible for the documentation.
His issue was that I used a function that was not in a class. So if the function is not declared it's obvious it will not work. I had to explain to him in another email, that you can use object of another class inside your class and then call a function, that is not in your class. It was the last time this guy blocked my deploy.
TL;DR, I had to explain a non-dev how object composition works in order to have my code deployed. Took four emails.5
I'm coming off a lengthy staff augmentation assignment awful enough that I feel like I need to be rehabilitated to convince myself that I even want to be a software developer.
They needed someone who does .NET. It turns out what they meant was someone to copy and paste massive amounts of code that their EA calls a "framework." Just copy and paste this entire repo, make a whole ton of tweaks that for whatever reason never make their way back into the "template," and then make a few edits for some specific functionality. And then repeat. And repeat. Over a dozen times.
The code is unbelievable. Everything is stacked into giant classes that inherit from each other. There's no dependency inversion. The classes have default constructors with a comment "for unit testing" and then the "real" code uses a different one.
It's full of projects, classes, and methods with weird names that don't do anything. The class and method names sound like they mean something but don't. So after a dozen times I tried to refactor, and the EA threw a hissy fit. Deleting dead code, reducing three levels of inheritance to a simple class, and renaming stuff to indicate what it does are all violations of "standards." I had to go back to the template and start over.
This guy actually recorded a video of himself giving developers instructions on how to copy and paste his awful code.
Then he randomly invents new "standards." A class that reads messages from a queue and processes them shouldn't process them anymore. It should read them and put them in another queue, and then we add more complication by reading from that queue. The reason? We might want to use the original queue for something else one day. I'm pretty sure rewriting working code to meet requirements no one has is as close as you can get to the opposite of Agile.
I fixed some major bugs during my refactor, and missed one the second time after I started over. So stuff actually broke in production because I took points off the board and "fixed" what worked to add back in dead code, variables that aren't used, etc.
In the process, I asked the EA how he wanted me to do this stuff, because I know that he makes up "standards" on the fly and whatever I do may or may not be what he was imagining. We had a tight deadline and I didn't really have time to guess, read his mind, get it wrong, and start over. So we scheduled an hour for him to show me what he wanted.
He said it would take fifteen minutes. He used the first fifteen insisting that he would not explain what he wanted, and besides he didn't remember how all of the code he wrote worked anyway so I would just have to spend more time studying his masterpiece and stepping through it in the debugger.
Being accountable to my team, I insisted that we needed to spend the scheduled hour on him actually explaining what he wanted. He started yelling and hung up. I had to explain to management that I could figure out how to make his "framework" work, but it would take longer and there was no guarantee that when it was done it would magically converge on whatever he was imagining. We totally blew that deadline.
When the .NET work was done, I got sucked into another part of the same project where they were writing massive 500 line SQL stored procedures that no one could understand. They would write a dozen before sending any to QA, then find out that there was a scenario or two not accounted for, and rewrite them all. And repeat. And repeat. Eventually it consisted of, one again, copying and pasting existing procedures into new ones.
At one point one dev asked me to help him test his procedure. I said sure, tell me the scenarios for which I needed to test. He didn't know. My question was the equivalent of asking, "Tell me what you think your code does," and he couldn't answer it. If the guy who wrote it doesn't know what it does right after he wrote it and you certainly can't tell by reading it, and there's dozens of these procedures, all the same but slightly different, how is anyone ever going to read them in a month or a year? What happens when someone needs to change them? What happens when someone finds another defect, and there are going to be a ton of them?
It's a nightmare. Why interview me with all sorts of questions about my dev skills if the plan is to have me copy and paste stuff and carefully avoid applying anything that I know?
The people are all nice except for their evil XEB (Xenophobe Expert Beginner) EA who has no business writing a line of code, ever, and certainly shouldn't be reviewing it.
I've tried to keep my sanity by answering stackoverflow questions once in a while and sometimes turning evil things I was forced to do into constructive blog posts to which I cannot link to preserve my anonymity. I feel like I've taken a six-month detour from software development to shovel crap. Never again. Lesson learned. Next time they're not interviewing me. I'm interviewing them. I'm a professional.9
I have to refactor code from an intern. He's VERY lucky that he already left the company.
If I'd say he programms like the first human that would be very insulting to that first human.
It looks like code at first sight, but when you try to understand what he was doing to achieve his goal you get a brainfuck. Duplicate code, unused code, dumb variable names like blRszN.
He wrote unittests like "expects Exception to be thrown or Server returns Statuscode 500".
Yes, Exception, the generic one.
THESE FUCKING TESTS ARE GREEN BECAUSE YOU DID NOT ACTUALLY TEST SOMETHING.
GREEN IN THIS CONTEXT MEANS: YOUR PRODUCTION CODE IS A BIG PILE OF SHIT.
I already removed 2 bugs in a test which caused another exception than the "expected" one and the test does still not reach the actual method under test.
The sad thing: The fuckers who did the code reviews and let this shit pass are still here writing code.4
So recently I have been working on an open-source project where all the mains devs are too busy to give a stream of patches and new features. I offered to do this job, but as soon as there was an 'official' patch all my changes would be wiped, I was ok with that, I have my own fork of the project so I could just implement it there. What I didn't ask for was my work 'buddy'. Instead of following what the client said (only patch critical bugs) they went on the project forum and got all these great ideas for new features which he gave tocme to implement. He had absolutely no idea about how to program and expected me to do all of it. To top it all off he messed with my code when it 'didn't work' didn't test it, then put it in to production. Even unfinished features with bugs galore were put into producton withot even contacting me and I was left to take the shit! Thank god today is the last patch I have to do.1
Just got this notification. I see virgin active UK likes to test in production. I too like to live dangerously.1
Friend of mine: so I wonder how do you test your applications in the startup?
Me: testing? *grabs his coffee laughing*
Actually we have a complete build pipeline from commit/pull-request to dev and production environments. No tests. Really. We are in rapid product development / research state.
We change technologies and approaches like our underwear (and yeah, this is frequently). If we settled some day and understood the basic problems of the whole feature palette, we'll talk about tests again.4
We started a project in January for which I was the sole developer, to automate tedious interaction with a vendor's ticketing system. We have a storage environment with about 400,000 commodity disks attached(for this vendor-- there are other vendors too), in sites around the US and Canada. With a weekly failure rate of about 0.0005%, that means about 200 disks a week need to be replaced.
This work-- hardware investigation through storage appliance frontends, internal ticket creation, external ticket creation, watching the external ticket for updates to include in our internal ticket --was all manual, and for around 200 issues a week, it was done by one guy for two years. He was hopelessly behind. This is all automated now, and this morning, I pushed this automation from dev/test to production.
It feels great to see your work helping people around you.8
OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..
Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!
Here it goes..
Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.
OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.
Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.
Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..
I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..
Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.
I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..
I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.
WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...
WTF..Ok, cig break to let me think..
I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?
So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o
Unfuckin believable.. How did this ever work?!
Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!
So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..
If only the original procedure was working as it should.. bloddy hell!!8
Let me tell you the story of how a feature request no one asked for got put in an early grave:
PM walks into weekly meeting with a single use case that one user called in about, despite never having this issue during the past year and a half that our app has been in production. PM's boss (genuinely one of the best people i have ever worked with) happens to sit in this particular meeting for no reason other than he felt like he should once in a while.
PM brings up use case and wants to devote 3 weeks' development time and another 3 weeks to test RIGHT NOW while other projects are already in motion. PM's boss speaks up with this: "Listen if this guy is really this upset, we can just tell him to build his own service. All the other end users have no problems with this, so it's not worth spending the resources on, i don't think."
And that is how i went from "this is bullshit" to "i love you" in the span of 20 minutes.2
Typos kill, kids! And deploying to production.
Instead of "for item in items" in my script, I accidentally did "for items in items". Thus, an exponential loop has been entering things into the database for the past few hours before I found the place to fix it.
By the way, this runs on cron every minute. So there are processes still running exponentially right now, possibly 180+.
Yeah, I'm setting up a a test server instead now.11
I love Test-Driven Development!
And because of that fact, my heart shatters into thousands of pieces, when I recognize error events on our production nodes which are pointing onto a golden hammer function in a legacy project.
This particular function has about 300 lines with a bunch of subfunction calls and instantiations of helper-classes returning information for workflow.
Refactoring this code to apply proper unit-tests requires a way bigger investment than simply deal with 30 eventlogs a day, because this kind of payment is barely used by customers of our webshop.
This fact is a little itch each day of my work.
Guess it will make me go insane one day
I wasn't sure what to say at the moment, but then I thought of something.
Lodash is the most depended upon package in npm. 90k packages depend on it, more than double than the second most depended upon package (request with 40k).
Lodash was also created 6 years ago.
This means lodash has been heavily tested, and is production ready.
This means that reading and understanding its code will be very educational.
Also, every lodash function lives in its own file, and are usually very short.
This means it's also easy to understand the code.
You could start with one of the "is..." (eg isArray, isFunction).
The reason for such choice is that it's very easy to understand what these functions do from their name alone.
And you also get to see how a good coder deals with js types (which can be very impredictible sometines).
And to learn even more, read the test file for that function (located in tests/<original file name>.js. For the most part they are very readable and examples of very good testing code.
Here's the isFunction code
Here's the test for isFunction
The one thing you won't learn here is about es5, 6, or whatever.3
Can't say I am a religious type, not really into discovering/discussing religions and comparing them either... BUT I do hope there is someplace (like hell on steroids!!) for developers who don't test their code before checking it in and/or puting it on production..
Also another question, can I plead not guilty due to insanity when killing such devs?!? O.o FUuuuuuuuu!!!!2
Dear client, if you can't be bothered to check more than two data points during several test imports, why are you surprised your production import has errors in the other 10k+ data points? We told you to check thoroughly, and you swore it was fine. But great now I get to unfuck production while you're mad you can't go live yet.2
I used to work on a production management team, whose job was, among other things, safeguarding access to production. Dev teams would send us requests all the time to, "run a quick SQL script."
Invariably, the SQL would include, "SELECT * FROM db_config."
We would push the tickets back, and the devs would call us, enraged. I learned pretty quickly that they didn't have any real interest in dev, test, or staging environments, and just wanted to do everything in prod, and see if it works.
But they would give up their protests pretty fast when I offered to let them speak to a manager when they were upset I wouldn't run their SQL.2
This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.
It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
I did as I was told, and Lo and Behold, the problem never occurred again.
Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"
Don't get me wrong, I'm glad that it's over but..
What the fuck happened?5
So after 6 months of asking for production API token we've finally received it. It got physically delivered by a courier, passed as a text file on a CD. We didn't have a CD drive. Now we do. Because security. Only it turned out to be encrypted with our old public key so they had to redo the whole process. With our current public key. That they couldn't just download, because security, and demanded it to be passed in the fucking same way first. Luckily our hardware guy anticipated this and the CD drives he got can burn as well. So another two weeks passed and finally we got a visit from the courier again. But wait! The file was signed by two people and the signatures weren't trusted, both fingerprints I had to verify by phone, because security, and one of them was on vacation... until today when they finally called back and I could overwrite that fucking token and push to staging environment before the final push to prod.
Only for some reason I couldn't commit. Because the production token was exactly the same as the fucking test token so there was *nothing to commit!*
BECAUSE FUCKING SECURITY!5
Deploying to production 8pm Saturday evening.
Because I have confidence in our test suite, QA procedures, infrastructure and my team members are not morons.4
I goofed up and forgot the WHERE clause in my UPDATE query. Accidentally all of production possibly updated because we don't do test databases. I think it didn't actually go through because I cancelled it but now I need to restore a backup and compare data. Which means explaining this to the co-owner who can help me with a restore. I'm mortified, more so because it was a stupid thing to do to begin with.4
So... Heard back from a recruiter today. Lovely lass.
I’d passed over a submission for her tech demo.
The brief was basically just to create a small simple module that calculates shit, nae effort.
But, when the recruiter had me on the phone she said “I know it’s a silly small module but try and run it up like you would a production ready app”.
The job spec and recruiter were keen on me demonstrating TDD, not specific on js version, final runtime, etc. The job was a senior spec at a higher salary range. So it warranted some effort, and demonstrating more than a simple module.
“Okay, cool, nae bother, let’s crack on.”
The feedback in the response from the dev today:
“He’s over-engineered tests, build...”
SUCK MY LEFT TESTICLE YOU FUCKWIT.
Talk to your recruiters, not me.
The feedback included a phrase I never hope to hear from a developer I work with:
“Tests are good but...” 😞
It was a standard 98% test suite from an RGR cycle, no more or less than I’d expect in prod.
The rest of the feedback was misguided or plain wrong. It was useful to see because I know now when they say they have “high standards” they mean: we listen to the dude who put the factory pattern in a JS brief.
Oh shit also: “someone’s done chmod 777” was in there as a sarcastic comment in the feedback. It was his fucking unarchive tool 😞
My response was brief and polite: “cheers for the consideration, all the best, James”
It’s honestly not worth warning them. Or, asking why they’d criticise something they’d asked me to do.
If you want a shitty js module, ask for a shitty js module and no more.4
My co-worker, still studying but working as a "senior dev", just decided that we don't need a test/staging environment anymore. We just "validate" (we also don't use the word "test" anymore) newly created features in production.
Makes absolutely sense...
Thank god I have a new job from february on!1
Bug I had to fix today: some elements in our React app were being swapped with other elements.
We had `<foo>bar</foo>` on the component but on the html `foo` was being swapped with some other element in our app. It's contents ("bar" in the example) were being left in place, though, so we were getting `<baz>bar</baz>`.
This would only happen when running on production mode. On development everything was fine.
Also, everything seemed fine on the React dev tools. `foo` was where it was supposed to be, but on the html it was somewhere else.
Weirdest shit I ever saw when using React. I found a way to go around it and applied that fix, but I'm still trying to track this down to the source.
The worst part was waiting for fucking webpack to finish the production bundle on every fucking change I wanted to test. I didn't miss the change-save-compile-test flow at all.
What a shit day.4
Discovered pro tip of my life :
Never trust your code
Achievements unlocked :
Successfully running C++ GPU accelerated offscreen rendering engine with texture loading code having faulty validation bug over a year on production for more than 1.5M daily Android active users without any issues.
History : Recently I was writing a new rendering engineering that uses our GPU pipeline engine.. and our prototype android app benchmark test always fails with black rendering frame detection assertion.
Spend more than a month to debug a GPU pipeline system based on directed acyclic graph based rendering algorithm.
New abilities added :
Able to debug OpenGL ES code on Android using print statement placed in source code using binary search.
I was aware of the issue over a month and just ignored it thinking it's a driver bug in my android device.. but when the api was used by one of Android dev, he reported the same issue. In the same day at night 2:59AM ....
Satan came to me and told me that " ok listen man, here is what I am gonna do with you today, your new code will be going production in a week, and the renderer will give you just one black frame after random time, and after today 3AM, your code will not show GL Errors if you debug or trace. Buhahahaha ahhaha haahha..... Puffff"
And he was gone..
Thanks satan for not killing me.. I will not trust stable production code anymore enevn though every line is documented and peer reviewed.
Yet another day at work:
My job is to write test libraries for web services and test others code. Yes I know to code, and have a niche in software testing.
Sometimes developers (whose code I find bugs in) get so defensive and scream in emails and meetings if I point out an issue in their code.
Today, when I pointed a bug in his repo, a developer questioned me in an email asking if I even understood his code, and as a tester I shouldn’t look at his code and only blackbox test it.
I wish I can educate the defensive developer that sometimes, it’s okay to make mistakes and be corrected. That’s how we deliver services that doesn’t suck in production.10
So last week I really fucked up
I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.
So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.
Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"
Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.
Needless to say, I had a fairly good weekend.
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert1
Borrowed from Reddit and Twitter:
Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in.3
YOU WANT ME TO TEST AND DEVELOP IN MOTHERFUCKING PRODUCTION?!!!
Whaaaaaat, I'm changing shit, what if somebody goes to buy this product and I've made it super-cheap? ATM, there's two fuckin options for shipping, both different costs. the best bit? RIGHT NOW, THE USER CAN CHOOSE TO PAY LESS FOR THEIR SHIPPING.
Just now... Got a job to create patch files for a couple of jars, which may or may not have varying class files. In total, I have to decompile, check, add and synchronize about 30 class files in 6 jars with a new functionality (that I didn't write). 🙂
FUCK PRODUCTION! WHY CANT YOU MAINTAIN ONE MOTHERFUCKING JAR?
OH? YOU'RE SUPERSTITIOUS THAT ONE TINY, ANT-SHIT SIZED CHANGE IN ONE SIMPLE FUNCTIONALITY WILL FUCK UP *OUR* PRODUCT?
FUCK MANAGEMENT! YOU DON'T HAVE CONFIDENCE IN YOUR *OWN* PRODUCT!
OH? CUSTOMER COMES FIRST? HAVE THE BALLS TO DEFEND YOUR OWN FUCKING SELF AND PRODUCT TO THE CLIENT OR THEY'RE GONNA MAKE YOU YOUR BITCH AND TIE A GAGBALL DIPPED IN HOT SAUCE AROUND YOUR MOUTH! HOW.. THE FUCK.. DID YOU MISS THAT LOGIC??????
Best part, they want it by tomorrow, and they don't wanna test it. Guess who's gonna get slaughtered after a week? ME! 🙂5
Dev sent out a code review request.
I take about an hour, ask questions, make suggestions, general feedback, etc.
Today I noticed none of my questions were answered, developer closed the review, and the code merged into the production branch.
So I email him, asking him why the review was closed and why none of my concerns were addressed before merging to production.
Dev: "No one responded or left feedback, so I thought it was OK to merge up."
Me: "I reviewed and left feedback within the hour you sent the request."
Dev: "Oh yea...you did. Sorry. The code is already in production, but if you still want to leave feedback, create a work item, and I'll take a look."
No you won't.
An example of the code...The dev added an async method to a test harness *console app*. Why? .. check in comment was "Improves performance and enhances the developer experience.."
NO IT DOESN'T!
OK..that's off my chest. No one is getting punched in the face today.8
(I'll give some context before the rant: I'm part if the IT department of a manufacturing company (actually I'm 1/2 of the department), and all the applications (old an new - except the ones used on production line) used in the company are my responsibility, that including most of databases too... Also, English isn't my native language so there will be some words or phrases that I'll probably write wrong... Sorry for that, if there are any corrections, I'll be glad to hear them)
There will be an implementation of new "control point" on the "shipping department" which consists on a electromechanical equipment controlled by a PLC. And despite the original concept was a collaboration between 2 departments (we, IT, and Production Control), I was never taken in consideration about anything of the project... To be fair, I forget about its existence until two weeks ago.
So, a few days I learned that there are a huge delay regarding the original deadline (mainly because the supplier was delayed with the delivery of their system), and since two weeks (less, actually, because some holydays in between) I'm learning how to integrate that "P.o.S" into an existing application on a PC using a serial communication (not the main problem, as I've done that before... With another brand of PLC's) while avoiding buying any additional software (to get the communication done and in a easy way) and that sort of things... But discovering in the process that it will be necessary to acquire such additional SW in order to finish the job ASAP.
When suddenly I get the "news" that it's almost all my duty (and responsibility) to meet the original deadline, because it doesn't matter how the other departments screw all the schedule, it's the job of IT to get the shit done in time... And what is worst: they didn't said that in such straight manner, no, the implied it while making a quick test with the general manager.
I mean, WTF? Besides doing a "respectable" number of "user support" activities in a dialy basis, I also need to manage the activities of other departments? And also fix their screw ups on a schedule that I just learned days before?
And also there is a coworker (one of whom screwed up) that, almost every time she see me, is asking "how much until you'll finish?"
As I read on a meme years ago: "please, give patience, because if you give strength, I'll need bail money too..."
Damn... I don't know of the benefits of this work are worth all this nonsense
BANE OF MY FUCKING EXISTENCE. STOP POLLUTING MY PRODUCTION CODE WITH TEST CODE, YOU FUCKING CRETINS.
Things that seem "simple" but end up taking a long ass time to actually deploy into production:
1. Using a new payment processor:
"It's just a simple API, I'll be done in 2 hours"
LOL sure it is, but testing orders and setting up a sandbox or making sure you have credentials right, and then switching from test to life and retesting, and then... fuck
2. Making changes to admin stats.
"'I just have to add this column and remove that one... maybe like a couple of hours"
"Hah, what, that's like a button, np"
125 minutes later...
Simultaneously opening ssh sessions to test and production system, finally stopping the application in the factory.
It was me.
So, just pulled another all-nighter..
On our platform I switched a quite big customer to another stock keeping system to pull them into automatic FEFO handling etc. Just a better stock keeping system overall.. I made it.. *self hi-5*
Evidently the crons caught that change, and CLEARED ALL THE STOCK LEVELS as they're now managed by said system...
Had to pull the counts, locations, expiry dates and lot numbers from the history table and old database fields, add them to an Excel sheet and then add all gathered locations by hand back into the new system, whilst also setting the new settings for them.
39 unique products that were gathered over 190+ sku objects... (Somebody didn't get object oriented, or was trying to KISS themselves, clearly...)
That's 6 hours of extra work for a stupid fuck up.. Oops? (:6
I dunno about coolest, but I did sort of cement my reputation as the "database guy" in my first job because of this.
My first job was with a group maintaining a series of websites. Because of the nature of the websites, every morning we had to pull the records from one database on one network, sneaker net the data to a database on another network, and import the data via custom data import function.
However, the live site would crash after 100 or so records were imported. The dba at the live site had to script out a custom data partitioning script to do his daily duties, but it definitely messed up his productivity.
Turns out, the custom mass import function had recycled the standard import function, which was only used to import 1 record at a time, and it never closed its database connections, because it never needed to. A one line fix to production code was delivered 6 months later (because that was our release cycle) and I came up with the temporary work around, which was basically removing the connection limit. It would still crash with the work around, but only with multiple days worth of data. So basically only on Monday. Also developed the test set for the import (15k+ records).
Living on the edge!
One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.
Well, just revert the db to the latest recovery point with the backup, I thought.
It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.
Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.
Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄
Well, not really a data loss story, but close to one.
about 6 years ago I was working for a large consulting company on a government project. I put in a change for a stored procedure that hard coded the partition to 0, except 0 didn't exist on production, just on test. several thousand government employees couldn't access it for a day. 😞
TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.
Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.4
Now my client does not want to rely on Amazon S3 because of the One Outage that it ever had a couple what weeks ago I forgot already. So my dumbass blurts out well we could always just back up to some other image or file storing website. But now I'm expected to implement this right away when I really haven't thought about it at all I mean I would have to write some sort of failover and some sort of daily or syncing mechanism. I guess I should forget about any direct upload to S3 code that I have written. Really I guess I have to wrap all of the image and file handling stuff with my own solution. Which actually that will be very nice when it is done and I could use this on other projects but it's quite a lot of work for something that I don't feel we really need at this stage in development. Just because you're using stuff on production that has am enormous red TEST label in the way of the ui doesn't mean i can code bullet proof software any faster4
So where I work, we used to push our code from test servers to production every tuesday/thursday exactly at one in the afternoon. Every time there was a push, I would play "push it to the limit" blasting over my speakers. Now we have an automated push and I never really listen to that song anymore. I miss those days. Link related https://youtu.be/9D-QD_HIfjA1
While smoke testing in production, I had to delete the sample entity I created to test the released feature, which is not a big deal
Until 20 minutes later, when I realized that I attached a couple of sub entities under it that contained actual live data1
A newly joined developer (who was supposed to be very senior) comes and asks me how to write a test cos for some reason the person didn't know how to mock.
(same for any other implementation which has an interface)
Writes Arraylist list =.....
Instead of List list = Arraylist...
Deployed code (another engineer from another country helped to deploy since this new senior dev didn't have access yet.
But the new senior dev didn't update relevant files in production code which brought down the site for nearly an hour. Mistake aside, the first reaction from this new senior dev is 'WHY DIDN'T THE DEV THAT WAS HELPING DIDN'T DO THE FILE UPDATE?'
This was followed by some other complaints such as our branching stragies are wrong. When in fact the new senior dev made a mistake by just making assumptions on our git branching strategies and we already advised on correct process.
Out of all these, guess this is the best part. The senior dev never tested code locally! Just wrote code, unit test and send to QA and somehow the test passed through. I learnt this when I realised this dev... has not even set up the local environment yet.
I keep saying new but this Senior dev been around like 3 months! This person is in another team within our larger team but shares same code base. I am puzzled how do you not set up your environment for 3 months. Don't you ask for help if you are stuck? I am pretty sure the env is still not setup.
Am I over reacting or is this one disgusting developer who doesn't even qualify for an intern let alone a senior dev? It's so revolting I can't even bring myself to offer help.9
Most intense day for me was at the very start of my career. Internship... went with product manager to client's office while PM installed new test version of our product for on-site integration testing. Shortly after deploy, client manager came over to ask why production had gone down...
Turns out that manually typing DB name as part of deployment script is not, erm, risk free. PM entered production DB name and took out a very busy call centre for a few hours. Agents in tears, customers raging on phones, etc! After we restored and got everything back up and running, he reached me the keyboard and said "You're doing it this time."
My attempt was problem free, thankfully. Earned many brownie points that day.1
Ever have a bug that *only* occurs in your production environment? How do you test potential fixes? 😜5
Normally you would have:
In the company (30-50 workers) we only have:
- SysOps (they don't know how to deploy apps beyond FTP to a webhost either)
I tried to propose to them that I handle DevOps and teach it to others, so we can deploy code that we deem "production ready" in a more proper manner...
They rather stick to "just use FTP to push any changes we made directly to the production server and test changes there"5
"Real devs test in production", in practice.
This was actually the second such notification I received. Not sure if this is standard for mobile app testing...2
Overengineering. Finding the right point between overdesign and no design at all. That's where fancy languages and unusual patterns being hit by real world problems, and you need to deal with all that utter mess you created being architecture astronaut. Isn't that funny how you realize that another fancy tool is fundamentally incompatible with the task you need to solve, and you realize it after a month of writing workarounds and hacks.
But on the other hand, duct tape slacking becomes a mess even quicker.
Not being able to promote projects. You may code the shit out of side project and still get zero response, absolutely no impact. That's why your side projects often becomes abandoned.
Oversleeping. You thought tomorrow was productive day, but you wake up oversleeped, your head aches, your mind is not clear and you be like "fuck that, I'm staying in bed watching memes all day". But there's job that has to be done, and that bothers you.
Writing tests. Oh, words can't describe how much I hate writing tests, any kind of. I tried testing so many times in high school, at university, even at production, but it seems like my mind is just doesn't accept it. I know that testing is fundamentally important, but my mind collapses every time I try to write a single fucking test, resulting in terrible headache. I don't know why it's like that, but it is, and I better repl the shit out of pure function than write fucking tests.
Ok I'm officially losing my fucking mind!
I've been trying to solve a connection bug that only occurs in production which is cool if THIS FUCKING APP DIDN'T TAKE 30 MINUTES TO DEPLOY!!!
Been busy for 4 hours and I've only been able to test 4 minor fixes.
I hate dev politics...
PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.
Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.
Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!
Me: That's dumb... the code works on test... it's not the code.. it's the application.
Senior dev: %$*^$>&÷^> $
Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!
Senior Dev: OK
Me: *runs Jenkins job the deploys the new code and restarts the application*
PM: YAY you fixed it!!
Senior Dev: Did you sump put those errors like I said.
Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.2
TL;DR: When picking vendors to outsource work to, vet them really well.
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...
Have our scope / discovery meeting about the changes and our expected DevOps workflow.
Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.
Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.
4th day, I get back "Who sets up the Github accounts?"
fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??
Whatever, fuck it. I'll make the accounts and add them.
Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."
So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?
We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.
This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...
Cant wait for today's progress checkup meeting. 😐😐
Wow or wtf to these banks API. was integrating an API for a service which accept JSON input.
Okay fair enough, that would be fine
Spent an hour writing code(purescript) most of time spent was on writing Types based on the API doc. after that okay let me test the API it failed.
I was what happened? So tested the API from postman with the payload from the doc, it worked. What how?
used a JSON diff to compare the payload from postman and the log. Looked same to me after spending few hours checking what is wrong with it .trying changing value to pasting the body of the log request in postman and trying everything failed.
Later went to the original working payload provided by them and changing the order. It started throwing error. I was like wait what?
It must be only on there UAT. created a payload with production creds and hoping to our production server (they have IP whitelist) ran the curl with proper payload as expected it worked. Later for same payload changed the order or one key and tried it failed.
I don't want to create a JSON with keys on specific order. Also it's not even sorted order.4
Since I have seen a lot of people uploading this kind of stuff lately, here is Xiaomi's test in production, back in 2017 November...1
Damn, nespresso! As if your fucking webshop wasn't bad enough, don't bug me (haha) with useless mails AND JUST TEST YOUR SHIT BEFORE PRODUCTION!!! Seriously, thousands of customers pay lots and lots of money for mediocre coffee in aluminum capsules, so JUST HAVE THE DECENCY TO INVEST SOME BUCKS INTO THE CUSTOMER FACING APPS!!!6
* Developing a new "My pages" NBV offer/order solution for customer
Customer: Are we ready for testing?
Me: Almost, we need to receive the SSL cert and then do a full test run to see if your sales services get the orders correctly. At this point, all orders made via this flow are tagged so they will not be sent to the Sales services. We also still need to implement the tracking to see who has been exposed to what in My Pages.
Customer: Ok, great!
Customer: My web team needs these customers to have fake offers on them, to validate the layout and content
Me: Ok, my colleague can fix this by Tuesday - he has all the other things with higher prio from you to complete first
Customer: Ok! Good!
Me: Good news, got the SSL cert installed and have verified the flow from my side. Now you need to verify the full flow from your side.
Customer: Ok! Great! Will do.
Customer: Can you see how things are going? Any good news?
*looks into the system*
- Have you set this into production on your side? We are not finished with the implementation on our side!
Customer: Oh, sorry - well, it looked fine when we tested with the test links you sent (3 weeks ago)
Me: But did you make a complete test run, and make sure that Sales services got the order?
Customer: Oh, no they didn't receive anything - but we thought that was just because of it being a test link
Me: Seriously - you didn't read what i wrote last Thursday?
Me: Ok, so what happens if something goes wrong - who get's blamed?
Works in production?
Yes: Copy changes to dev and test.
No: Start humming and walk away from the computer.
Take over responsibility you fucking morons!
We are the engineering team and we cannot know how you operate our product in every detail. And for god's sake don't blame us when shit happens in production when you don't test upcoming deployments by yourself!
A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.
The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.
The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.
They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"
WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production?
I just had the most embarrassing moment in programming... I am writing an administration / client / invoice webapp and I was testing an export function that worked locally, because everything that was being exported was inside the folder.
So I exported the files in test production, but some invoices didn't exist. So when they don't exist, the system creates a new invoice.
Because I was running on the test production (with client data) the system emailed the created invoices to the clients.. now I have to contact some clients and tell them the invoices were sent accidentally.2
Build pipelines are awesome. What's not awesome is forgetting the 'pipeline' bit.
Sure, let's have each TeamCity job in the 'pipeline' build the deployable from different places! You must manually merge your code from dev to test to release.
What's that? The functionality working in dev isn't there in production? I wonder how that happened.
We basically don't unit test at work. I write some tests for my code and honest to God people complain I'm wasting time saying a test bed and manual tests are good enough. We don't write test beds for about half of our production code and rely on integration tests for the rest. We only test release builds which have been symbol stripped, I get handed a crash report with no stack trace that I'm unable to reproduce and expected to stay late to fix it for some arbitrary internal deadline.
I've since moved to R&D where basically I'm left to do my own thing so it's better.
We don't project manage. Project leads take time estimates and double them so management might cut them some slack. This doesn't matter because management made up time estimates before the project started. Last project I was on had a timeline of 3 months and took a year.
We have released broken products. Not that any of the above really matters, our software products have made about 50k revenue in 2 years. There are 6 people on software. Fortunately hardware has made about 3 mill. That said our hardware customers are getting frustrated with us as we keep fucking up, shipping broken products and missing deadlines.
I've been working there about a year and a half and will be looking for a job at the end of the current project.
I joined devRant about when I was most pissed off with my job, my rant frequency has definitely gone down since I moved over to R&D.
My company insists on working in one production environment to save time and every time I try to convince them to set up a work flow with a dev and test environment, they tell me we don't have the time...
Even after I set one up anyway as I'm scared shitless to touch production. They tell me it's faster doing it all in one environment.
They launch an update. Site buggy as hell and doesn't load 90% of the content...
Used a wrong filter during loading of a table in ETL. Did not test and migrated to production. 80% of users had empty reports.
Had to stay awake till 4AM to get it fixed.
Realized an important lesson -
' A test in time saves nine'
So I'm currently "assigned" a task in which I need to fix a slow query problem, which isn't a big deal. The biggest problem is that the original team of this project haven't got any means to develop things on your local machine. Looking at their docs and scripts, it seems like everything is deployed to a dev server. But whilst looking for details for this server, I found out that the network team have decommissioned the server!
So my dilemma right now is that I can't test any of my fixes on anywhere besides staging, or possibly production! Inheriting projects is the bloody worst!5
Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.
The non tl;dr:
For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......
Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments2
Ok so some of you have probably seen my previous rants about my computer science teacher and our project but I'm just going to summarize all of them and share with you more of my pain.
1. He edits in the production environment. Its a laravel project and he is creating test database migrations IN THE PRODUCTION ENVIRONMENT AND SWITCHING BACK AND FORTH FROM MASTER AND DEV.
2. He edits in vim and doesn't follow codestyle even though I printed him off a piece of paper and emailed it to him.
3. He doesn't have any ethic when it comes to more complex things like laravel homestead.
4. He doesnt want me to release features even though he takes really long to do them.
While I love vim and it is my editor of choice, some things should be done in an ide. This is really annoying me and I'm really just considering handing him the project if he can't follow basic outline.7
How fucked up are you,
when your vp of engineering doesn't even know how to show phpinfo webpage to test server setup.
change ode directly in production server,
then messed up and using excuse :
" I don't know because i am a frontend developer "
Then why you become a VP of Engineer !3
Getting Feedback Rant!
When "this is simpler" feedback results in a function of 500 lines of code.
When I get "don't do X" in the feedback. Thank you very much. What do you want me to do instead?
When the feedback giver changes his mind after I applied the changes!
When applying the feedback introduces a bug.
Simply opinionated feedback that is not enforced by any tool or backed up by any facts.
Please find something better to do in life.
I will not consider thank you very much.
"Verify this works"
When the feedback giver knows something that you don't.
I know this is a legit case.. still annoying.
"I disagree with the feature"
Go argue with the PM, not relevant to me, thanks!
GIVING FEEDBACK RANT
I rewrote the system. Please review it.
No need to review, just approve.
I will change this as part of the next ticket.
I would like to keep it the way it is.
You can't test this.
It's impossible to test this.
No need to test this.
There's no point to test this.
I'll test this on production.
Not sure why this is working..
Please document this..
Because documentation is like a thing, you know.
Oh, this code is not related to this PR, I just don't want to open a new branch for such a small change. ignore it.
This will be meaningful in my next change.
TL;DR: fear of bricking my laptop due to typo pinning.
The worst nightmare i am living in right now...
I was noticing i did need some software in sid so i decided to use apt pinning for said software...
I configure the system, ok test looks good... I push it to production, run it on the system....and the nightmare starts.
Lits of packages get updated, and i am screaming 'noooooooo' since debian sid softwarz can sometimes break everything! I discovered that i did test my apt pinning config for the presence of the amount of numbers, but not at their value... Sooo, by accident swapping pin numbers for stable and unstable you get... Your worst apt-get update nightmare...
I hope it does not become a brick.1
So much has happened, I've been learning things, I got robbed, I discovered I love test driven development, my laptop fell downstairs and is now screenless, I'm still on this project and still have not gotten the source or gone live, side work exists though so I get to make some more money, car engine needs to be overhauled, project extended still not in production. Send Help.1
So, someone from support department asked me to check the app as it was displaying blank information on a page.
I started debugging on the api which i found is doing the proboem. I checked and it was working a moment before but not now. Switched in debug mode to see what went wrong and it works again. This happen multiple times.
Before doing anything else i asked our api developer to check if api is working. He said it should work now and problem has been fixed.
Later i found out that he was doing debugging/changing code on production server instead of his local machine or test server.
I cannot think about a title, so just story:
in my position as multi headed chimera one of my ongoing task is it to dedust old excel sheets, processes and other super inefficient relics that steal time. Mostly i solve those with some tiny vba scripts, bigger vba scripts or a tiny java applications. usually that takes a few hours or maybe two days, depending on what i think is necessary.
the current task at hand is for our (physical) production, work time is noted on a sheet of paper and later given to the production head. Who then proceeds to type it all in excel to do his thing. The guy is starved of time by a huuge margin.
The other day our CEO-like guy was ranting that nothing in this company gets ever done and that people wasting their time with useless projects and named my project among them.
I dont get humans. First he gives thumbs up for this, knowing that it will probably take me 100 hours or so to create in a working manner but later he calls it "a waste of time?" I presented the use (reducing expensive mantime, paper waste and room for fudgery) and yet he calls it useless? (well, his point was that there are other problems (which are out of my reach anyway))
they guy normally is pretty nice and has an ear for problems, but when it comes to higher computer stuff (>excel) he really struggles.
i really like my side project, gives me room to flex some muscles and test stuff. Also playing with raspberry pis on worktime.
On a sidenote, anyone ever tried raspi mesh networks and knows where i get working >10 inch capacitive touch screens?
While addressing a Senior Dev's (SD) query from another team.
SD: why is this field mandatory? Can't it be just optional? Any other work around?
Me: Is your code changes already pushed in Devo? In that case, we provide a value which will work since you are not concerned about it.
SD: Yes. It's pushed till production. And, I want to test changes in Prod.
Me: (shared some codes) and explained that this feature for testing is only available in Devo.
SD: I know that. (Shared me a ticket) I want this field to be optional. That's it.
Me: (read the entire ticket. Didn't find anything related their) Told him, I will discuss with team. And meanwhile, for Devo, you can use this value.
Next morning, I accidentally came over some other ticket raised by him only which had the correct doubts regarding request to support this field in production
Now, I don't know why did he share a wrong ticket with me.
And, how will it even help him if that field was even optional.
THAT JUST WONT WORK IN PRODUCTION.
I will discuss with my team and see what can happen.
Debugging a task, that's sending emails to too many customers.
Supervisor: "Never mind, just test in production, there is a dry run flag for the tasks."
Just in case I test locally...
--dryrun="TRUE" => Error, failed to send mail.
--dryrun=TRUE => Error, failed to send mail.
--dryrun="true" => Not trying to send mail.
If it's THIS PICKY a little more documentation would be nice.
And by a little more I mean: more than the task base class in a giant php monstrosity without phpdocs expecting its code to be self-documenting.
This is just a sweet little harmless block of code, why should I unit test it?
3 months later...
Bug in production : This is why.
Today at work I started doing 1 month old task with production problem.
First of all why now ?
Because I already fixed all the other urgent production problems I had during last month, done about 4 deployments of those super urgent errors.
Now I can start with not trivial one that are pending for quite time.
I am the only backend developer in this project ...
This is a dtp application and the problem is that we are not verifying if we got all fonts embedded in customer provided pdf files.
We are generating high quality images of those pdf for printing just fine from the beginning but now we need valid PDF with all fonts embedded in it. ( don’t ask me why I am only a hammer in this process )
After running simple test using python script against database it turned out we have over 500 broken PDF files without fonts.
So I guess I have just one sentence to say about it.
Fuck you PDF format for not being strict and allowing this shit.
As my first dev job, I took over role of solo programmer maintaining all kinds of custom-made software used by local ISP. It was about 10 years ago.
My first question was where can I find test environment and repo. Apparently there was none and I should learn and develop on production.
My sin was to quickly give up on setting up both test and repo.
My second sin was to continue using the same copy&paste PHTML with register_globals enabled, building over it without attempting to refactor it with templates. I did not use globals in any new code at least.
And I suppose my third sin was that I was playing games when I was done with my tasks. I could have used that time to refactor a bit.
But I think in the end I was absolved from them since I was the only one suffering from this. I stayed with company until it got sold and helped migrate data over (along with myself).
Sometime last year I had an internship at a small company.
Test servers weren't a thing, and after local testing, it would go to production with a backup of the files that we would put back as soon as we notice something was broken or off.
We used symfony and sonata admin was part of the bundle.
One day, boss asks me to show all the items in a table on the admin page instead of 30 rows.
Me being good guy intern say "sure no problem" so after finding the magic number, I set it to 0 instead of 30.
I gave my work reviewed by my supervisor (senior dev there) and he approved it.
I try to upload the file over FTP. No permissions.
Ask the other dev what it's about, his response: "no idea"
So he tries, fails and decides to try SSH.
Somehow, after fiddling for 20 minutes with ssh, we managed to upload the file.
As soon as we did we hear a scream from the boss's office, we refresh the site, and no matter what page we went to, all we saw was white and the logo of the company in the top left corner.
So this time, we fiddled around with ssh to restore the file for 20 minutes.
Finally succeed all goed back to normal.
A little while later, we call a meeting with the bosses and ask to rewrite the website, BAM, we get approval.
We said "two weeks tops", well that lasted 3 months.
In the end bosses are Uber happy with the work and everything ended well.
Also, development speed has multiplied.
Sometimes, certain features don't work in my app if it's uploaded to the production track in Google Play.
It works in debug mode.
It works in release mode.
It works in the internal testing track.
...but it won't work in the production track? Y'know, the one that actually matters? Theoretically, there shouldn't really be a difference if the same exact APK was uploaded to the internal testing and production track, but... idk dude.
As a result, I implemented a secret way to test a feature in the production build (it's an app to remotely control OBS Studio): if the first connection you added is named "yayeet_" and you open the disclaimer, it tests the feature. Luckily, I got some of the stuff figured out, but I just thought the way I had to test it in production was dumb.1
As I'm on a research/algorithm improvement project at work I'm working pretty much independently. As such I've set up an automated test framework and writing tests for any piece of code I touch.
Today as I was fixing a bug in production area I was demoing my tests to CTO and principal design engineer. They come from a hardware background and have pushed back against automated tests in the past but they were interested in what I was doing.
I WILL DRAG THEM KICKING AND FUCKING SCREAMING INTO THE WORLD OF AUTOMATED TESTS.1
I am just student looking for job, and got this pre interview test:
Develop an Android or iOS app with login and password input field, download button, place for image we prvided.
... reading further:
What we are looking for in the code ?
-consistent formatting of the source code
-clean, robust code without smells
-consistent abstractions and logical overall structure
-no cyclic dependencies
-code organized in meaningful layers
-low coupling and high cohesion
-descriptive and intention-revealing names of packages, classes, methods etc.
-single small functions that do one thing
-truly object-oriented design with proper encapsulation, sticking to DRY and SOLID principles, without procedural anti-patterns
-lots of bonus points for advanced techniques like design patterns, dependency injection, design by contract and especially unit (or even functional or integration) tests
-the app should be fully functional, with every state, user input, boundary condition etc. taken care of (although this app is indeed very small, treat it as a part of big production-ready project)
-the app should correctly handle screen orientation changes, device resources and permissions, incoming calls, network connection issues, being pushed to the background, signing deal with the devil :D and other platform intricacies and should recover from these events gracefully
-lowest API level is not defined - use what you think is reasonable in these days
-bonus points if the app interacts with the user in an informative and helpful way
-bonus points for nice looks - use a clean, simple yet effective layout and design
... I mean really ? and they give me like 2 days ?4
Well, yes. My boss accepted my 'test' project to be build for production use, stack is Laravel, Vue, a little Python.
It is divided into phases where as I have now month and a half to finish phase 1.
You agree I will finish the first phase in time? (24-2) Keep in mind I still have other projects I'm working on.9
Last week we were only one step ahead of going in production mode with the angular web app i coded a half year long. Sounds good right?
Yeah this morning my boss said in the dev Meeting, blazer is now in preview mode, let's do it with this tech, so our full stack is in c#...
He is not a web dev. He want to step back from coding in the near future, but yeah let's use fucking Blazer 😥
For the rest of the day, i started with a Blazer Test Project.. great start into the short week.
How about your start?6
After three months of development, my first contribution to the client is going live on their servers in less than 12 hours. And let me say, I shall never again be doing that much programming in one go, because the last week and a half has been a nightmare... Where to begin...
So last Monday, my code passed to our testing servers, for QA to review and give its seal of approval. But the server was acting up and wouldn't let us do much, giving us tons of timeouts and other errors, so we reported it to the sysadmin and had to put off the testing.
Now that's all fine and dandy, but last Wednesday we had to prepare the release for 4 days of regression testing on our staging servers, which meant that by Wednesday night the code had to be greenlight by QA. Tuesday the sysadmin was unable to check the problem on our testing servers, so we had to wait to Wednesday.
Wednesday comes along, I'm patching a couple things I saw, and around lunch time we deploy to the testing servers. I launch our fancy new Postman tests which pass in local, and I get a bunch of errors. Partially my codes fault, partially the testing env manipulating server responses and systems failing.
Fifteen minutes before I leave work on the day we have to leave everything ready to pass to staging, I find another bug, which is not really something I can ignore. My typing skills go to work as I'm hammering line after line of code out, trying to get it finished so we can deploy and test when I get home. Done just in time to catch the bus home...
So I get home. Run the tests. Still a couple failures due to the bug I tried to resolve. We ask for an extension till the following morning, thus delaying our deployment to staging. Eight hours later, at 1AM, after working a full 8 hours before, I push my code and leave it ready for deployment the following morning. Finally, everything works and we can get our code up to staging. Tests had to be modified to accommodate the shitty testing environment, but I'm happy that we're finally done there.
Staging server shits itself for half a day, so we end up doing regression tests a full day late, without a change in date for our upload to production (yay...).
We get to staging, I run my tests, all green, all working, so happy. I keep on working on other stuff, and the day that we were slated to upload to production, my coworkers find that throughout the development (which included a huge migration), code was removed which should not have. Team panics. Everyone is reviewing my commits (over a hundred commits) trying to see what we're missing that is required (especially legal requirements). Upload to production is delayed one day because of this. Ended up being one class missing, and a couple lines of code, which is my bad (but seriously, not bad considering I'm a Junior who was handed this project as his first task at his first job).
I swear to God, from here on out, one feature per branch and merge request. Never again shall I let this happen. I don't even know why it was allowed to happen, it breaks our branch policies. But ohel... I will now personally oppose crap like this too...
Now if you'll excuse me... I'm going to be highly unproductive and rest, because I might start balding otherwise after these weeks...
Helping out a team, I was documenting some code/processes when I came across several classes that was logging a lot of, IMO, 'junk' that was unnecessary (and I knew wasn't being used in any Splunk alerts/reports)
I offer a refactoring suggestion, simplifying the data being logged, moving the duplicate code to a central location, maybe saving 10~20 lines of code. Didn't think it was a big deal because they were already actively working on the code and it was all new code (nothing deployed to production yet). Sent the suggestion to the lead developer and he responds:
Dev: "Yes, the changes looks fine, but not in scope of the project. Any out of scope work will need to be suggested at the end of the project, reviewed by the team, the project manager and approved by the vice president."
"Out of scope"? Logging data to Splunk needs a vice president's approval? WTF?
YOU PROBABLY HAVE THE PROJECT OPEN IN VISUAL STUDIO RIGHT NOW!!!
Along with the documentation the lead dev said they didn't have time to do, I send his boss and the dev team my suggested changes (before-after screen shots of the code) and offered to do the 2 minutes worth of work (again, this was new code, nothing in production and zero side affects to anything).
I even offered to create the splunk reporting/alerting against the data being logged (another item they said they would not have time to do)
About a minute later the lead dev responds..
Dev: "Those changes look good. I'll have Jake make those changes and we can test the logging when we deploy to dev on Monday. Thanks!"
Of course you will...fracking ass hat.
I'll bet my Battlestar Galactica DVD box set he was going to make the changes himself, brag to his boss how he refactored the code, saving X lines of code..blah blah blah to help *me* with documenting the logging portion.
Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?
# The detailed version #
We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.
But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are
So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.
I guess the saying goes: the only people that know how it works is whoever wrote it and God.
I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.
I think in Agile there's supposed to be artifacts during development but never seen em.
Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.
I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.
And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...
The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.
Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.
Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.9
Was given 2 bugs to solve and after 1 day and half someone remembers to tell me that what I'm trying to fix won't work cause the issue only happen in production an I won't be able to test it with the connection that I have...
So a third-party service that I implemented is going to production and me and the PO were testing that yesterday, didn't see any orders coming in the service backend.. so we send a mail to them.
This morning they respond with saying that they can't have both the test server running and the production server...
What the hell is this... :/3
Just joined a new company and can only describe the merge process as madness.....is it or am I the one that is mad?!
They have the following branches:
UAT#_Branch (this kicks of a build to a machine named UAT#)
Each developer has a branch with the # being a number 1 to 6 except 5 which has been reserved for UAT_Testing branch.
They are working on a massive monolith (73 projects), it has direct references to projects with no nuget packages. To build the solution requires building other solutions in a particular order, in short a total fucking mess.
Branch from master with a feature or hotfix branch
Make commits to said branch and test manually as there are no automated tests
Push the commits to their UAT#_Development branch, this branch isn't recreated each time and may have differences to all the other UAT#_Development branches.
Once happy create a pull request to merge from UAT#_Development to UAT#_Branch you can approve your own pull request, this kicks off a build and pushes it to a server that is named UAT#.
Developer reviews changes on the UAT# server.
QA team create a UAT/year/month/day branch. Then tell developers to merge their UAT#_branch branches in to the previously created branch, this has to be done in order and that is done through a flurry of emails.
Once all merges are in it then gets pushed to a UAT_Testing branch which kicks off a build, again not a single automated test, and is manually tested by the QA team. If happy they create a release branch named Release/year/month/day and push the changes into it.
A pull request from the release branch is then made to pre-live environment where upon merge a build is kicked off. If that passes testing then a pull request to live is created and the code goes out into production.
Ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh it's a total mess. I knew when I took on this job it would be a challenge but nothing has prepped me for the scale of the challenge!! My last place it was trunk based development, commit straight to master, build kicks off with automated testing and that just gets pushed through each of the environments, so easy, so simple!
They tell me this all came about because they previously used EntityFramework EDMX models for the database and it caused merge hell.9
Since I started my routine of checking bug logs every morning, I've had 2 instances where a website vulnerability scanner was run against a production website and generated over 2,000 Coldfusion errors.
At the time, I was super nervous about the apparent hack attempt, and hyped that the attackers never actually got in. It's nice to know that despite the various errors indicating vulnerable / breakable code, they were ultimately unsuccessful. I know now that a determined attacker could probably have wrecked our production websites. Since then I've made a ton of security-related updates and I'm actually thankful for the script kiddie getting my attention with that scan.
PS. We're now building a website for a local security company who is going to work with us to pen test the site when it's finished! Gulp.4
How do you manage production, development and test environment in iOS/Xcode? Any good sources out there? 💻2
PM: We will need this to be on the staging server by tomorrow because we need to get user to start doing UAT early and deploy it to the production server 2 days later.
*2 days later, 2 hours before deployment*
PM: yeah, client just got back to us, we will need you to fix 14263728261 things in 2 hours, re-test them and deploy it.
Where to begin?
- I purposely overestimate because I can get away with it in current job
- I test in production like 50% of time in current job
- I deploy during working hours
A dev decided to overwrite the master branch with his code saying its better. That it fixes the major bugs that all of us couldn't solve.
Against my better judgement of firing him, I decided to test it.
Firing up the testing site, we made test databases to use and we went to house.
In the middle of testing, I noticed the test DBs weren't being changed. While everyone was still testing, I looked at the code. It wasn't made to test on any databases, it was specifically designed for the actual production server.
However the damage was done. In a secret dashboard in the code, someone sent instructions to drop the tables, effectively ruining the production server.
We had the dev go to an offline backup site that only went online every 10 minutes a day to make new backups. So we shut down the production server, setup a maintenance page. I get my ass chewed out again, and we were sitting ducks.
I don't think the dev had enough punishment, so I grabbed his laptop and made a full backup of his data, and locked the SSD in a safe.
I downloaded a Windows 98 and put it on a flash drive. And installed it all on his SSD. The dev is now a proud (pirate) owner of Windows 98.
He came back and started balling on his desk. We all looked at him with a pity, but he deserved it.
I'll give him the drive on Monday.
Do you think he learned his lesson?7
- Change this, change that too, oh and that too.
* Ok, will do ( however unlikely it's going to be finished correctly, as you didn't consult me before or listen to me about the impact this might have )
- ( just stfu and do it )
Sometime at or after important cutoff:
- Hey this doesn't show up, is that right
* Yeah, you wanted too many things changed at once + fix it and notice a stupid error like an && switched with a ||, or other models that don't know about change x yet
always the same, while im in a part of the system, i notice an optimization that can speed up a major query which has to join a table which is about 4gb or something ridiculous. i make ammendments using partitions because they're in the defined on that table. test. everything cool. only to be told that theres no job to clear out old partitions so i end up reverting everything i've been doing which basically makes my day's total output == bollock-all. WHY DO WE PUT HALF BUILT SHIT INTO PRODUCTION!!!???2
So one of the IPSPs I work with wanted to add another bank account to our app. All the code does is downloading the account history as CSV, parsing it and sending for further processing.
So I registered above mentioned new bank account, added credentials, the app requested a history file from the bank, sent it to the parser... and stopped there. No output. No error. Nothing. "What the heck!?", I thought. Compared this CSV with our test suite example and soon enough it turned out the new account gets a slightly different format, it's missing one of the identifiers - we send an empty field to the next module. Good. Case solved, right?
I called the IPSP's representative - will be sorted out. Four days passed. I called the bank's representative - they'll look into it. A week passed. Finally I called our people to pass the bad news - they'll have to fix that in **our** code. What a shame. So much work! They'll have to put down their coffee and doughnuts for 5 minutes...!
- "So, I've found the issue. I compared the test suit file with the one from production. Prod CSV is missing an identifier."
- "Oh? Let me see... No, mate, nope, nada, that's not the cause of it. Our tests ignore that file. We hardcoded different values."
- "And you didn't fix the input file?"
- "Nah, mate, it was such a slight difference there was no point. It could be reverted in a minute."
I considered ending my rant here but I decided to highlight few aspects:
1. The issue remains unsolved, I just stopped caring since nobody else does.
2. Apparently versioning systems aren't enough to convince some people files that aren't in use should be removed from the source.
3. Company policy that forbids developer access after staging leaves literally **nobody** who can understand how huge amount of time can be wasted by the slightest difference.
Data will contain errors and inconsistencies, and the only player that can detect them is: the computer itself.
if the computer detects that it is in a situation where it cannot meaningfully continue, it should not allow itself to continue.
Only the computer can make that determination.
If the software does not aggressively test its data, it is usually impossible to determine whether the problem is with the software or with the data or both.
Per contra, if it does do so, it becomes impossible to assert that the input data does not contain the issues that are being tested for ... a very important thing to be able to say in a real-world production setting, where hundred-megabyte input files are common.
Testing new server deployment in test env all works, then production it all breaks down. Network didn't allowed the right traffic. Took me whole week to find that out. Until some networking engineer said, you know there is a firewall between those networks?
When you have a customer that is a pain and you have to do a new contract since months but they are no replying but at same time there is a bug in a plugin they are using.
They are not updating their plugins in production but only after a test in staging.
In production there aren't write permission from web server side, so only they have access.
And the plugin has a 0-day.
We need to go live by this month 10.
Yep, infra is ready in production. You can push the events and see the results right away.
PM: Wow, great. Setup staging we need to test it.
Me: FML, staging machines are smaller, can't even start the program.1
Is using CouchDB in production a bad idea? I built a small POC to test CouchDB and PouchDB's syncing abilities. Now I'm wondering am I setting myself up for tears if this gets implemented in production...2
So I was writing SaltStack state for syslog management and I had a simple config file in place to be deployed on a test server. I was writing the command to run the state for the test server, and the only thing that was left was to type the hostname of the server (instead of wildcard) when someone interrupted me. After I got back to this terminal I instinctively pressed return sending test configuration to over 80 production servers. Nice one...
A young new dev was working on his first ticket, about a bug during parsing of an uploaded excel file. Our issue was that if the file contained an empty line, all remaining rows were ignored. So the task included extending our tests to cover this case. After 2 weeks (!), his merge request comes in. His idea (without ever asking for help) was to parse the whole file (in some cases huge) in the production code a second time, just to count the rows (!!) and save the count in a public static int field, which was verified in his new test.2
We have a CRM running on an EC2 instance. We need to clone it so we can test a tool on the replica. We tried cloning it directly, sharing the AMI and creating a new instance through it but it always redirects changes to the original production server. The database is on the instance only and static files are stored in S3. Can someone guide me or share some resourses on how to do this.5
I wanna know who use xenserver and why??
Currently am using xenserver 6.5 in a production environment and today i start to test xenserver 7.1
In my initial days as a web developer, i was assigned a task, to implement a cart share functionality in an e commerce company.
I made the functionality and tested on my system.
Result: working good.
Pushed it to beta testing environment.
Resilt: working good.
Pushed to pre production environment.
Result: working good.
Pushed to live site.
Result: 😀 Error in live site..
So a call comes to me from my team lead..
Asks what was the issue...
Me: i dont know either.
After 3-4 hrs:
I found the reason.
My system, beta test env, pre prod env are all having latest php version (5.6 i guess)
But the live server had old version of php.
Me: laughed like anything.
I didn't know that these things would matter in such a great level.
Moral of the story:
Be one with the force (server in this case)2