Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "production server update"
When you're a junior sysadmin but still have to maintain ALL the production server:
How it looks:
$ sudo apt-get update
How it feels:
& sudo [ $[ $RANDOM % 6 ] == 0 ] && rm -rf / || echo *Click*7
Contrary to most people I really love to receive email related to jobs when I'm in holiday. I keep important alerts on.
email: ***urgent, server down***
me (sipping mojito by the pool): fuck them. let's them deal with that
email: ***requirements all wrong, must develop the feature again***
me (enjoying a dinner): oh, I told them 100 times!, fuck all of them, work for me now, stupid moron.
email: I destroyed by mistake the db with an update..."
me (dancing like crazy): ahahaha I told you that support guys should not have access to production db, fuckfuck you, fix it yourself!!!
and so on..... I don't know, it just boost my pleasure during holiday.10
DATA COMMUNICATION BETWEEN SERVER AND AN APP..
1. Write all data into files,
2. Make the files as zip,
3. Send the zip to server,
4. Server will unzip the zip file,
5. Read all the data line by line from the files and update the data.
** TRUST ME, THIS IS A PRODUCTION APP I HAVE SEEN FROM A CLIENT **7
I messed up carelessly in production. Learnt how SQL queries bite you in the ass when it knows you are under pressure.
Was hosting an online quiz kinda thing during my college techfest. Tens of thousands of people participating.
Using MySQL as database and thousands of queries were being executed. Everyone were pretty excited as the event just opened up.
None of the teams could solve one particular level. Turns out the solution was wrong and was asked by the organisers to change the solution for that particular level. Usual stuff, right?
Was too lazy to open up the web UI for the back office and so, straight ahead logged in to the MySQL server and ran the UPDATE query on the table consisting of the solutions.
It had been a couple of hours and the organisers came to me with a weird problem. There were no changes in the scoreboard for the last two hours. Everyone were stuck wherever they were. Weird, right?
I then realized.
In that dreaded query, I had only run
UPDATE 'qa' SET answer = 'something'
leaving out the where clause, specifying the question to update, like
As a result, solutions to all the questions were updated to the same answer. After hastily fixing everything back, I had the dreaded conversation.
Org: What was the problem?
Me: It was the cache.
Org: Damn thing. Always messes up.
Me: *sheepishly* yeah
Probably the most embarrassing moment in my life, wrt coding 😑4
Ended in an UPDATE without WHERE query to a core table in a Drupal project (in a dev database stored in the same server that production database was)2
Ran a script on production to scrape ~1000 sites continously and update our ~50.000 productions from the data. On the same server as our site was running. Needless to say, with traffic and scraping, our server had almost 100% CPU and ram usage all the time for 2 weeks until I realised my fuckup2
Do I want to continue?
Y -> vacation lost (Production server is down).
N -> Ok, I will gather more packages for you to update next time.
Sometimes your music app knows just the right song to play.
Production program was working (has been for a long time). But suddenly it starts failing. I spent a long ass time trying to see what went wrong.
Security update on the server 🙃
Now I've got the client, his minions, and the users emailing me to fix this. But I didn't start this fire!
Song: We didn't start the fire, by Billy Joel
Am I the only developer in existence who's ever dealt with Git on Windows? What a colossal train wreck.
1. Authentication. Since there is no ssh key/git url support on Windows, you have to retype your git credentials Every Stinking Time you push. I thought Git Credential Manager was supposed to save your credentials? And this was impossible over SSH (see below). The previous developer had used an http git URL with his username and password baked in for authentication. I thought that was a horrific idea so I eventually figured out how to use a Bitbucket App password.
2. Permissions errors
In order to commit and push updates, I have to run Git for Windows as Administrator.
3. No SSH for easy git access
Here's where I confess that this is a Windows Server machine running as some form of production. Please don't slaughter me! I am not the server admin.
So, I convinced the server guy to find and install some sort of ssh service for Windows just for the off times we have to make a hot fix in production. (Don't ask, but more common than it should be.)
Sadly, this ssh access is totally useless as the git colors are all messed up, the line wrap length and window size are just weird (seems about 60 characters wide by 25 lines tall) and worse of all I can't commit/push in git via ssh because Permissions. Extremely aggravating.
4. Git on Windows hangs open and locks the index file
Finally, we manage to have Git for Windows hang quite frequently and lock the git index file, meaning that we can't do anything in git (commit, push, pull) without manually quitting these processes from task manager, then browsing to the directory and deleting the .git/index.lock file.
Putting this all together, here's the process for a pull on this production server:
Launch a VNC session to the server. Close multiple popups from different services. Ask Windows to please not "restart to install updates". Launch git for Windows. Run a git pull. If the commits to be pulled involve deleting files, the pull will fail with a permissions error. Realize you forgot to launch as Administrator. Depending on how many files were deleted in the last update, you may need to quit the application and force close the process rather than answer "n" for every "would you like to try again?" file. Relaunch Git as Administrator. Run Git pull. Finally everything works.
At this point, I'd be grateful for any tips, appreciate any sympathy, and understand any hatred. Windows Server is bad. Git on Windows is bad.10
This one was from my Tech Lead, She had to update phone number of a customer in Database on production server. And guess what she forgot 'WHERE'. Next we were facing each other with poker face. :|
p.s: fortunately we had backup of just 4 hours back. Still we lost data of about 100 people.4
Developed an update to our database procedures and tested it with local copy. After a few days everything was ready. Opened our server and started the update. After a couple of rows an error occurred. Turns out our production db is older version and does not support some syntax I used. It became a bit longer day at work...
Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.
I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!
I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!
I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?
I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).
With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....
So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...
* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.2
When your staging/test server dies just as the boss wants a big update completed. Straight to production it is!
Why on gods ungodly earth would you ever dream of injecting this kind of style into my <head> tag.
Without any warning whatsoever.
You couldn't be FUCKing bothered to be thinking about the consequences, before pushing such an update to your production server.
That's how you leave your users website broken, or ugly af.
I know it's an easy fix, simply remove the snippet that injects this crap, if only I was allowed to say "no don't keep this FUCKing crap" to the customer.2
I'm so sick of this task.
It's been three weeks now and I have the same problem as last week. It's very frustrating to have my name on this. I could have finished this in a few days if there weren't so many dependencies on other people or if those people would at least care enough to validate their work. I worked on more complex tasks and the most annoying part is knowing how easy it is but still being unable to finish it. I'm tired of attending stand-up meetings and giving the same "update" over and over again.
The team lead who borrowed me (team 2) messaged me yesterday asking what's holding it up. I told him I'm stuck with this thing that I need and that I already reached out to someone to get it so he already knows it's not my fault it's taking so long. Keep in mind that I cannot "just" get it myself, I would if I could, but imagine logging in and then paying right away. What items are you paying for? How much do they cost? It's a chain of events involving different teams.
Last week was spent with a team (team 3) that wouldn't respond to follow-ups. It came to a point where my original team (team 1) and the team who borrowed me (team 2) were all in the same group chat as team 3's lead. All for this one small fucking thing that I need. They kept saying "It's fixed" when it's not and I spent hours taking and sending screenshots to say "It's not there, seeeee?"
Finally, they made the changes last Friday after my work shift. Someone else from the team I'm working with (team 2) validated it, "Yep, it's fixed. The thing is there now." Now someone from team 2 has to make the changes to their API so the thing would appear in the response and then I will call that API to get the thing. While I'd love to volunteer, it's a completely different task assigned to a different developer and since I'm just borrowed by this team, I don't have access to all of their code except this module I'm working on and cannot assess the impact of making the changes. Not to mention that I would have to learn their entire process which I've already proven they are not really sure of themselves.
Why do I need to learn the entire process? Isn't that overkill? Glad you asked. I had an amazing on-boarding during the first week while setting up their shit and got a lot of "You need these first" errors, none of those things are committed to the repository, some are but are in different branches that I couldn't have guessed. Just refer to the previous rant.
So the smartest move is to wait for them to make the changes to the API. I told team 2 that I'm getting an internal server error with the API. They re-deployed and it works but after testing, the thing I need isn't in the response. They said they already made the changes to get the thing. It's not there. Now this guy I'm talking to follows up again with the developer who's working on the API and while all of this is happening, they decided to change the requirements. Team 2 is having production issues and my concerns are of course, not prioritized.
My team lead (team 1, the original) is obviously pissed off by team 3 from last week and now my team mates (team 1) are annoyed at both team 2 and 3 because of their broken applications and processes. Team 2 also made changes in their application without proper communication so the team 1 developers are affected.
I just want all of this to be over so I can return to my original team.
Azure, great development slots! Must have, now I can have developer, staging and production. The greatest no downtime when swapping a new server in....
Everything crashes? WTF?
OKAY, so swapping to a service that authenticates users makes the authentication part crash :/
Phew development slots ROLL BACK...
No the entire service was broken. Rolling back, all non authenticating controllers work, but the authentication never happens, so server is working, but the users cant use it. Fuck!
Delete everything. Recreate. The setting persists. WTF. Delete again, recreate, reinitialize, republish, it works as it should when tested phew.
Creating new service experiencing cant replicate. Hmm, okay must have been a glitch. Next, update, YEAH swap, no downtime!!!
*EXPLOSION* ..... RINSE AND REPEAT:/
Before get get source code for freelance job, the person who cantact me say the job is to continue the project for some update and tweak.
The UI from design is beautiful and he gave good explaination for the project and the update, continue to conversarion, negosiation and deal.
but he is not the IT guy and also the project is not his work or something that he do previosly. All the person who work on that project is already leave and not contactable.
And here that I get:
- source code
- domain cred.
And here what's missing:
- .env file
- db backup / old db cred.
- server and hosting cred.
And after some hour of learning the code I find out that:
- latest commit was 2 year ago and different from production version.
- most of the branch is RnD.
- the code have many wtf/minute lol
And for now I still re-negotiate with the person who give me the project with 2 suggestion from me.
- continue with this code with condition, he need to search for the missing part at least backup db or documentation.
- recreate the project with more time
And here's one funny part of the code.
return 5 // this number was choose by dev team at random
Deploy new script on production and then server time are outdated suddenly, plus old timestamp data inside db changed to outdated time. Who update the data inside db? Mindblow~2
worst mistake was probably introducing an infinite loop in the category tree for e-commerce site...
in the vein of true agile and considering MVPs and what not we had not yet automated everything. the client would send category updates as a spreadsheet and i had a script to generate the sql and jam it into the site. having run the script several times in the past I thought I'd just throw the update into production and call it a weekend...
it wasn't long before I started fielding calls that the site was unstable. no page would load and the server kept crashing under trivial load. well an entire frantic weekend later I discovered the category load hit an edge case I hadn't considered and I had introduced an infinite loop in the navigation of the site.
i'd like to say I learned my lesson and never just threw changes into production again, but what can I say - I like living on the edge. I did however learn that loop detection can be a valuable thibg