Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "backups matter"
-
The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.
TL;DR their people isn’t competent and I ended up fixing the issue.
Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.
So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.
IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.
I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.
Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.
I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.
5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.
They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.
I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.
I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.
Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.
While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-
Website up and running 1.5 later, mission accomplished.6 -
*Got a request for installing and configuring an online shop for a client*
Me: Do you have a web space already?
Client: No, I don't want to pay for it. (FYI: They only cost about 20€ a year)
Me: Okay, but free hosters are often slow and unprofessional. I really do not recommend using free hosting services.
Client: Doesn't matter, do it.
Me: *Working on the shop for several weeks, finally goes online*
One week later, client contacts me saying shop is offline. I realize the free hoster he used shut down their services (bankruptcy), resulting in the loss of about 90% of the work that I had done (no proper backups due to complexity)
Client: How can that even happen? You'll redo the shop, right?
♪~ ᕕ(ᐛ)ᕗ5 -
I just noticed that my desktop still has a DVD-RW in it that I never used.. which led me to think, I've got here like 5 DVD-RW's as well as 5 CD-RW's, but I never use them. Are there still any uses for them? Long-term backup storage for example? I've heard somewhere that data rot can happen over the course of a few years.. but would it really be a problem? I mean I've had these for over a year already and they look healthy to me. That said, they've never been used and just sat on the top of my desk in their enclosures all the time.
FWIW, I'm having 10TB of storage (as well as some other smaller stuff here and there.. maybe 500GB or so) in various hosts and 3 mirrors, so the need for backups is already fulfilled.. it's merely a curiosity of mine as in "ah, I still have these things huh".. so yeah.
I think that I've bought these DVD's for something like €6 for 5 pieces in a local store that doesn't specialize in this sort of thing, so I'm fairly certain that I could get them for cheaper elsewhere.. think the likes of AliExpress etc. Would they be a viable technology to maybe still invest in over hard drives for long-term storage purposes, considering that the cost of a hard drive these days is only €50 for 1TB? But I guess that tapes would be better for this purpose, no?9 -
Me: "Ok, downloaded the (windows) iso, now I need to create a bootable usb"
* Pulls out 4Gb usb drive *
Father: "That iso won't fit on that one"
M: "Oh yeah right, do you have one for me"
F: "Sure"
* hands over 64Gb usb drive *
M: "thanks"
* Checks content *
* "Backups 15" directory and other old files *
"Guess I can use that"
* dds iso on usb drive *
dd: "done"
M: * Reboots *
Pc: "Nani the fuck is that supposed to be"
M: * searches online for solutions, tries out 2 different ones, one being provided by microsoft *
P: "lol no"
F: * comes in *
"Is it working?"
M: "Nah, windows a bitch"
F: "ok, can you copy me some files on the usb in the meantime?"
M: "Sure..."
{ How did he know that the drive is currently empty? }
* copies files onto stick *
F: * sees empty usb drive*
"Did you delete the contents on the usb?"
M: "Kinda, that happens when you create a bootable usb drive"
F: "..."
M: "Why do you ask, was the something important on the drive?"
F: "Idk, doesn't matter anyways now, since the contents are gone"
Btw, it still doesn't boot from the usb drive. Windows 10 iso is a bitch.7 -
Normal people : Save thier work regulary.
Me :
Oh no matter, notepadd++ saves opens files somewhere, i'm good with that.
There are important lohins, passwords, some temp stuf and work in progress for something, there are backups and information needed.8 -
Has anyone ever figured out why the fuck Android reboots seemingly out of nowhere at random times?
Is it kernel panics? Is it some shitty bug that has been plaguing this cursed OS since its inception? [sidenote: boy that was a mouthful of 's'es]
Or are we already at the mercy of the Big Brother, and someone at the controls likes to play random pranks on us when they're bored?
Thinking of switching to LineageOS more and more, but I'm kinda worried about resetting my phone since I've never done full backups/restores and I have no experience in the matter...7 -
So I've been using Duet on my iPad Pro for a couple years now (lets me use it as an external monitor via Lightning cable) and without issue. Shit, I've been quite happy with it. Then the other day, whilst hooked up to my work laptop, there was a power fluctuation that caused my laptop to stop sending power to connected devices. Which is fine - I have it plugged into a surge protector so these fluctuations shouldn't matter. After a few seconds the laptop resumed normal operation and my connected devices were up and running again.
But the iPad Pro, for some reason, went into an infinite boot loop sequence. It reboots, gets to the white Apple logo, then reboots again.
In the end, after putting the iPad into recovery mode and running Apple's update in iTunes (as they recommend), it proceeds to wipe all my data. Without warning. I lost more than a couple of years of notes, illustrations and photos. All in one fucking swoop.
To be clear, you get 2 options in iTunes when performing a device update:
1. UPDATE - will not mess with your data, will just update the OS (in this case iPadOS)
2. RESTORE - will delete everything, basically a factory reset
I clicked UPDATE. After the first attempt, it still kept bootlooping. So I did it again, I made sure I clicked UPDATE because I had not yet backed up my data. It then proceeds to do a RESTORE even though I clicked UPDATE.
Why, Apple? WHY.
After a solemn weekend lamenting my lost data, I've come a conclusion: fuck you Apple for designing very shitty software. I mean, why can't I access my device data over a cabled connection in the event I can't boot into the OS? If you need some form of authentication to keep out thieves, surely the mutltiple times you ask me to log in with my Apple ID on iTunes upon connecting the damn thing is more than sufficient?! You keep spouting that you have a secure boot chain and shit, surely it can verify a legitimate user using authenticated hardware without having to boot into the device OS?
And on the subject of backing up my data, you really only have 2 manual options here. Either (a) open iTunes, select your device, select the installed app, then selectively download the files onto my system; or (b) do a full device backup. Neither of those procedures is time-efficient nor straightforward. And if you want to do option b wirelessly, it can only be on iCloud. Which is bullshit. And you can't even access the files in the device backup - you can only get to them by restoring to your device. Even MORE bullshit.
Conversely, on my Android phone I can automate backups of individual apps, directories or files to my cloud provider of choice, or even to an external microSD card. I can schedule when the backups happen. I can access my files ANYTIME.
I got the iPad Pro because I wanted the best drawing experience, and Apple Pencil at the time was really the best you could get. But I see now it's not worth compromise of having shitty software. I mean, It's already 2021 but these dated piles of excrement that are iOS and iPadOS still act like it's 2011; they need to be seriously reviewed and re-engineered, because eventually they're going to end up as nothing but all UI fluff to hide these extremely glaring problems.2