The Perfect Storm:

My worst coding mistake? Yeah, let me tell you about that. I pushed a simple JavaScript/HTML change without knowing that the stupid header was shared with another "not so important" section of the site called "My Account" where people go to pay for their services. I call it the perfect storm because I left early that Friday for a weekend cruise and right before leaving I pushed the change, sent the request to push for production and left. When they noticed that clients were complaining about not being able to pay they started reversing most changes of all teams trying to fix it but they never touched mine because they knew I wasn't working on the backend. My whole team worked over the weekend trying to find the issue while I was having fun in the cruise. They ended up reversing all changes by Sunday night and it took us about 4 more days to figure out that my simple JavaScript/HTML change broke the site and prevented 30 million customers from making payments that weekend plus it broke the whole 2nd release of the month.... yeah, nothing major.

Add Comment