13
-Sam-
150d

Upgrading. We like to upgrade our stuff, whether it's software, operating systems or hardware. When it works its great but when it dosen't...

All my BAD experiences have been with upgrading.

One day I was using Jumla (a CMS) that controlled a big online clothing store. Noticing that Jumla was 0.01 versions behind I decided to 'upgrade'. This caused the entire site to break, maxed out the space on the server and eventually lost my job and that day the company supposably lost $10,000.

Today's f#ck up made me write this rant. Me and a friend own a local development company and we have a small Digital Ocean server for client website previews (before they get there own hosting). We have a few projects going at the moment and yesterday we sent a few links to clients so they could see there new website. This morning I woke up, read a few emails and ssh'd onto the server to read logs and what not. I got a bit side tracked, reading about the benefits of Ubuntu 17. You can already see where this is going... I innocently Google: "How to upgrade to Ubuntu 17". Surprisingly after running the commands and downloading the updates it was worked well. Everything was working. Then I restarted. I waited about 15 seconds and tried logging in again. Timeout after timeout. Something was wrong. I checked the console via the online Dashboard and see a page full of Kernal errors. I contacted the hosting people and they were able to help by referring to some guides but after 5 hours of cranking through errors and not winning I give up.

*Email from client*
Hello again,
The website you sent via link isn't working, can you fix this as I would like to show our CEO,
Thanks, Mike.

I destroy the droplet (server), making a new one. I have to setup and secure the server. Generating new SSH keys, new user accounts as well as installing AND configure Apache, PHP and MYSQL. I then had to upload 5gb of backups via SSH (not fast), go through each clients backup, including web files and databases and distributing where it needs to go.
Discovering that one of the DB's name changed last week and therefore our backup script failed to save it, we were forced to rewrite 10 pages of website content.

From 10 yesterday morning to 2 this morning, a total of 14 hours (I think) sitting in front of my computer trying to fix a problem that would have never occured if I didn't "upgrade"

Comments
  • 4
    Jumla sounds scary
  • 1
    Do you work for TSB?
  • 0
    @d4ng3r0u5 nah, why's that? U from NZ too?
  • 1
    @Linux I just came back on this platform after about a year to post this rant and your still poppin' off. Well done!
  • 0
    @-Sam- No, I'm from the UK. There's a bank here called TSB that is currently having a massive IT upgrade fuckup.
  • 0
    @d4ng3r0u5 ahh I see. Good to know its not just me. I thought TSB was a New Zealand only bank lol. Nevermind
  • 1
    If you already know serverpilot (and the likes) and don't wanna use such services then just be a bit more careful in future when it comes to doing things to server 😁

    If you don't know, you should look into those server management services. Not costly. Reliable. Get jobs done.
  • 1
    @CurseMeSlowly and @Condor thanks for the tips. I'm more of a dev than a sys admin so ur advice really helped. My server is alot better now, thanks!
Your Job Suck?
Get a Better Job
Add Comment