When you're a junior sysadmin but still have to maintain ALL the production server:

How it looks:

$ sudo apt-get update

How it feels:

& sudo [ $[ $RANDOM % 6 ] == 0 ] && rm -rf / || echo *Click*

Contrary to most people I really love to receive email related to jobs when I'm in holiday. I keep important alerts on.

It's like:
email: ***urgent, server down***
me (sipping mojito by the pool): fuck them. let's them deal with that
email: ***requirements all wrong, must develop the feature again***
me (enjoying a dinner): oh, I told them 100 times!, fuck all of them, work for me now, stupid moron.
email: I destroyed by mistake the db with an update..."
me (dancing like crazy): ahahaha I told you that support guys should not have access to production db, fuckfuck you, fix it yourself!!!

and so on..... I don't know, it just boost my pleasure during holiday.

DATA COMMUNICATION BETWEEN SERVER AND AN APP..

1. Write all data into files,
2. Make the files as zip,
3. Send the zip to server,
4. Server will unzip the zip file,
5. Read all the data line by line from the files and update the data.

** TRUST ME, THIS IS A PRODUCTION APP I HAVE SEEN FROM A CLIENT **

I messed up carelessly in production. Learnt how SQL queries bite you in the ass when it knows you are under pressure.

Was hosting an online quiz kinda thing during my college techfest. Tens of thousands of people participating.

Using MySQL as database and thousands of queries were being executed. Everyone were pretty excited as the event just opened up.

None of the teams could solve one particular level. Turns out the solution was wrong and was asked by the organisers to change the solution for that particular level. Usual stuff, right?

Was too lazy to open up the web UI for the back office and so, straight ahead logged in to the MySQL server and ran the UPDATE query on the table consisting of the solutions.

It had been a couple of hours and the organisers came to me with a weird problem. There were no changes in the scoreboard for the last two hours. Everyone were stuck wherever they were. Weird, right?

I then realized.

Fk.

In that dreaded query, I had only run

UPDATE 'qa' SET answer = 'something'

leaving out the where clause, specifying the question to update, like

WHERE qno=13

As a result, solutions to all the questions were updated to the same answer. After hastily fixing everything back, I had the dreaded conversation.

Org: What was the problem?

Me: It was the cache.

Org: Damn thing. Always messes up.

Me: *sheepishly* yeah

Probably the most embarrassing moment in my life, wrt coding 😑

Ended in an UPDATE without WHERE query to a core table in a Drupal project (in a dev database stored in the same server that production database was)

Ran a script on production to scrape ~1000 sites continously and update our ~50.000 productions from the data. On the same server as our site was running. Needless to say, with traffic and scraping, our server had almost 100% CPU and ram usage all the time for 2 weeks until I realised my fuckup

Do I want to continue?
Y -> vacation lost (Production server is down).

N -> Ok, I will gather more packages for you to update next time.
😭😭

Sometimes your music app knows just the right song to play.

Story:
Production program was working (has been for a long time). But suddenly it starts failing. I spent a long ass time trying to see what went wrong.

Problem:
Security update on the server 🙃

Now I've got the client, his minions, and the users emailing me to fix this. But I didn't start this fire!

Song: We didn't start the fire, by Billy Joel

Yesterday I killed a production server with a handful of sites running on it. 😬

Created an observer for one of my Laravel models. It was generating new translated slugs when another post gets translated by an API. While implementing also an updated method besides the created observation I obviously updated the models slug.
Pretty confident this small update will work I just pushed to production. Tested it live.

BOOM

Hard reset on the server redoing the changes. Searching where I fucked up this time...

Finding me observing the updating while updating the updating of the updating by the updating. 🤦

Am I the only developer in existence who's ever dealt with Git on Windows? What a colossal train wreck.

1. Authentication. Since there is no ssh key/git url support on Windows, you have to retype your git credentials Every Stinking Time you push. I thought Git Credential Manager was supposed to save your credentials? And this was impossible over SSH (see below). The previous developer had used an http git URL with his username and password baked in for authentication. I thought that was a horrific idea so I eventually figured out how to use a Bitbucket App password.

2. Permissions errors
In order to commit and push updates, I have to run Git for Windows as Administrator.

3. No SSH for easy git access
Here's where I confess that this is a Windows Server machine running as some form of production. Please don't slaughter me! I am not the server admin.
So, I convinced the server guy to find and install some sort of ssh service for Windows just for the off times we have to make a hot fix in production. (Don't ask, but more common than it should be.)
Sadly, this ssh access is totally useless as the git colors are all messed up, the line wrap length and window size are just weird (seems about 60 characters wide by 25 lines tall) and worse of all I can't commit/push in git via ssh because Permissions. Extremely aggravating.

4. Git on Windows hangs open and locks the index file
Finally, we manage to have Git for Windows hang quite frequently and lock the git index file, meaning that we can't do anything in git (commit, push, pull) without manually quitting these processes from task manager, then browsing to the directory and deleting the .git/index.lock file.

Putting this all together, here's the process for a pull on this production server:
Launch a VNC session to the server. Close multiple popups from different services. Ask Windows to please not "restart to install updates". Launch git for Windows. Run a git pull. If the commits to be pulled involve deleting files, the pull will fail with a permissions error. Realize you forgot to launch as Administrator. Depending on how many files were deleted in the last update, you may need to quit the application and force close the process rather than answer "n" for every "would you like to try again?" file. Relaunch Git as Administrator. Run Git pull. Finally everything works.

At this point, I'd be grateful for any tips, appreciate any sympathy, and understand any hatred. Windows Server is bad. Git on Windows is bad.

Developed an update to our database procedures and tested it with local copy. After a few days everything was ready. Opened our server and started the update. After a couple of rows an error occurred. Turns out our production db is older version and does not support some syntax I used. It became a bit longer day at work...

When your staging/test server dies just as the boss wants a big update completed. Straight to production it is!

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

This one was from my Tech Lead, She had to update phone number of a customer in Database on production server. And guess what she forgot 'WHERE'. Next we were facing each other with poker face. :|

p.s: fortunately we had backup of just 4 hours back. Still we lost data of about 100 people.

I'm starting to FUCKing hate those 1-5 lines minified 3rdparty javascript snippets that everyone seems to be bombarding their website with.

Why on gods ungodly earth would you ever dream of injecting this kind of style into my <head> tag.
Without any warning whatsoever.

You couldn't be FUCKing bothered to be thinking about the consequences, before pushing such an update to your production server.
That's how you leave your users website broken, or ugly af.

I know it's an easy fix, simply remove the snippet that injects this crap, if only I was allowed to say "no don't keep this FUCKing crap" to the customer.

Azure, great development slots! Must have, now I can have developer, staging and production. The greatest no downtime when swapping a new server in....

Everything crashes? WTF?

OKAY, so swapping to a service that authenticates users makes the authentication part crash :/

Phew development slots ROLL BACK...

No the entire service was broken. Rolling back, all non authenticating controllers work, but the authentication never happens, so server is working, but the users cant use it. Fuck!

Delete everything. Recreate. The setting persists. WTF. Delete again, recreate, reinitialize, republish, it works as it should when tested phew.

Creating new service experiencing cant replicate. Hmm, okay must have been a glitch. Next, update, YEAH swap, no downtime!!!

*EXPLOSION* ..... RINSE AND REPEAT:/

Before get get source code for freelance job, the person who cantact me say the job is to continue the project for some update and tweak.

The UI from design is beautiful and he gave good explaination for the project and the update, continue to conversarion, negosiation and deal.

but he is not the IT guy and also the project is not his work or something that he do previosly. All the person who work on that project is already leave and not contactable.

And here that I get:
- source code
- domain cred.

And here what's missing:
- documentation
- .env file
- db backup / old db cred.
- server and hosting cred.

And after some hour of learning the code I find out that:
- latest commit was 2 year ago and different from production version.
- most of the branch is RnD.
- the code have many wtf/minute lol

And for now I still re-negotiate with the person who give me the project with 2 suggestion from me.
- continue with this code with condition, he need to search for the missing part at least backup db or documentation.
- recreate the project with more time

And here's one funny part of the code.
randomNumber(){
return 5 // this number was choose by dev team at random
}

Deploy new script on production and then server time are outdated suddenly, plus old timestamp data inside db changed to outdated time. Who update the data inside db? Mindblow~

worst mistake was probably introducing an infinite loop in the category tree for e-commerce site...

in the vein of true agile and considering MVPs and what not we had not yet automated everything. the client would send category updates as a spreadsheet and i had a script to generate the sql and jam it into the site. having run the script several times in the past I thought I'd just throw the update into production and call it a weekend...

it wasn't long before I started fielding calls that the site was unstable. no page would load and the server kept crashing under trivial load. well an entire frantic weekend later I discovered the category load hit an edge case I hadn't considered and I had introduced an infinite loop in the navigation of the site.

i'd like to say I learned my lesson and never just threw changes into production again, but what can I say - I like living on the edge. I did however learn that loop detection can be a valuable thibg