I recently joined this big MNC after shutting down my own startup. I was trying to automate their build process properly. They were currently using grunt and I favor gulp, so I offered to replace the build process with gulp and manage it properly.
I was almost done with it in development environment and QA was being done for production.
In the meantime I was trying to fix some random bug in a chrome extension backend. I pushed some minor changes to production which was not going to affect the main site. That was in the afternoon.
This Friday my senior rushed to me. It was like he ran six floors to reach me. He asked, did you push the new build system to production, I refused. He then went to the computer nearby and opened the code.
It was Friday and I was about to leave. But being a good developer, I asked what's the problem. He told me that one complete module is down and the developers responsible for them left for the day already and are unreachable.
I worked on that module multiple times last month, so I offered my help. He agreed and we get to work.
The problem was in the Angular front end. So we immediately knew that the build process is screwed. I accidentally kept the gulp process open for anyone, so I immediately rebuilt using grunt and deployed again, but to no success.
Then I carefully analyzed all the commits to the module to find out that I was the one who pushed the change last. That was the chrome extention. I quickly reverted the changes and deployed and the module was live again. The senior asked, how did you do that? I told the truth.
He was surprised that how come that change affect the complete site too. We identified it after an hour. It was the grunt task which includes all the files from that particular module, including chrome extension in the build process.
He mailed the QA team to put Gulp in increased priority and approved the more structural changes, including more scrutiny before deployment and backup builds.
The module was down for more than 5 hours and we got to know only after the client used it for their own process. I was supposed to be fired for this. But instead everyone appreciated my efforts to fix things.
I guess I am in a good company 😉