We build a backup infrastructure at work to make sure that clients can restore their files and databases themselves when something gets fucked up.

We also have step by step tutorial on how to do this.

Every fucking day we get requests to restore backups.

Mostly used reason is "I'm a technical so I won't understand it".
With all due respect, if you don't understand this and keep asking without even trying, please don't host with us.
Because, if you did as I asked and actually read through the entire article, you would.
In case you're wondering, anytime one of us asks what part they don't understand, that question is simply ignored and they pushing for us restoring it anyways continues.

Sometimes they get angry and want to talk to someone higher up or start complaining that they're paying loads of money already and that it would just take us a second anyways.

If you would read the fucking tutorial/manual instead of trying to eat out your mother's badly shaved pussy and hopefully choke on it while you're at it, you wouldn't come asking us for it.
If you genuinely don't understand this article, feel free to ask but also provide us with cocksucking feedback.

Why do you think you have the right anyways to ask us to do it for free? We maintain the backup infrastructure which definitely isn't cheap but we do it so that you, pubic sniffing weazel, can do this shit on your fucking own.

You're entitled to ask us for help but not for asking us to restore your bullshit for free every freaking time.

Tip: give your parents some condoms. Because that way they hopefully won't reproduce again, we don't need more of you in this universe.

There's one kind of ticket I fucking love. Wait, YOU LOVE TICKETS?! No, except for one.

Tickets where people ask us to restore backups.

Why do I love them?

Because the only fucking thing I need to do is reply with a link to a helpdesk item on our website because backups aren't something within our support range because it's easy for customers to do.

So whenever a ticket about restoring a backup comes in, it's as easy as opening the ticket, pasting the link with "I'd like to refer you to the following resource: " and pressing fucking "reply"

😄

TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

“Don’t learn multiple languages at the same time”

Ignored that. Suddently I understood why he said that. Mixed both languages. In holiday rechecked it and it was ok.

Sometimes mistakes can lead to good things. After relearning I understood it much better.

“Don’t learn things by head” was another one. Because that’s useless. If you want to learn a language, try to understand it.
I fully agree with that. I started that way too learning what x did what y did, ... But after a few I found out this was inutile. Since then, I only have problems with Git

Another one. At release of Swift, my code was written in Obj-C. But I would like to adopt Swift. This was in my first year of iOS development, if I can even call it development. I used these things called “Converters”. But 3/4 was wrong and caused bugs. But the Issues in swift could handle that for me. After some time one told me “Stop doing that. Try to write it yourself.”

One of the last ones: “Try to contribute to open source software, instead of creating your own version of it. You won’t reinvent the wheel right? This could also be usefull for other users.”

Next: “If something doesn’t work the first time, don’t give up. Create Backups” As I did that multiple times and simply deleted the source files. By once I had a problem no iOS project worked. Didn’t found why. I was about to delete my Mac. Because of Apple’s WWDR certificate. Since then I started Git. Git is a new way of living.

Reaching the end: “We are developers. Not designers. We can’t do both. If a client asks for another design because they don’t like the current one tell them to hire one” - Remebers me one of my previous rants about the PDF “design”

Last one: “Clients suck. They will always complain. They need a new function. They don’t need that... And after that they wont bill ya for that. Because they think it’s no work.”

Sorry, forgot this one: “Always add backdoors. Many times clients wont pay and resell it or reuse it. With backdoors you can prohibit that.”

I think these are all things I loved they said to me. Probably forgot some.

Me to customer, several years ago:
Your server is over 10 years old, and the support for it will soon be gone together with the possibility to take backups. You should upgrade your website ASAP.

Customer said:
ok, we will look into it.

This day:
Customer:
Hi, our site is down and we need backups from yesterday. Can you please fix?

Me:
Sorry, no. *send copy of the email several years ago*

!rant

So I am building a website and already finished it and ready to be deployed.
Got server access and found that there are many other websites files and folder.

Note : multiple copies of website that I am working on.

So I raise my concern with client about which folder is actually handling the website. She contact the old dev guy and got the webroot for that website. She told me to delete everything and clean it up and make tge current websites available.

I realize that there are many other websites may rely on those files and folder and inform about the consequences. And finally told her I will not touch any files except for the website I am working on.

So, I deployed the website and works good as expected. After 3 weeks clients comes back with an issue that one of her websites is notworking and trying to blame me for it.

Fyi : before my deploymemt she says that she will talk to hosting providers to clean their webspace. I am not sure on that yet.

So I tried to do auditing and found that after 2 days frm my deployment some has already wiped everything from those other websites.
I showed my audits and its someone did it.
You should contact your hosting provider or the other dev who she contacted for wiping.

From the begining I am telling that lady to take backups. She said no need.

The reason I didnt took it because I am not working on those website and obviously client isnt going to pay me for those back.

There is still more to go waiting for her resonses.

It's march, I'm in my final year of university. The physics/robotics simulator I need for my major project keeps running into problems on my laptop running Ubuntu, and my supervisor suggests installing Mint as it works fine on that.
I backup what's important across a 4GB and a 16GB memory stick. All I have to do now is boot from the mint installation disk and install from there. But no, I felt dangerous. I was about to kill anything I had, so why not `sudo rm -rf /*` ? After a couple seconds it was done. I turned it off, then back on. I wanted to move my backups to windows which I was dual booting alongside Ubuntu.
No OS found. WHAT. Called my dad, asked if what I thought happened was true, and learnt that the root directory contains ALL files and folders, even those on other partitions. Gone was the past 2 1/2 years of uni work and notes not on the uni computers and the 100GB+ other stuff on there.
At least my current stuff was backed up.

TL;DR : sudo rm -rf /* because I'm installing another Linux distro. Destroys windows too and 2 1/2 years of uni work.

An important message:
PrOpErLy managing servers is HARD.

I get pissed off at customers with ZERO server knowledge who think they can manage their VPS. “Just get a control panel and a VPS” from some flashy provider that makes server management look way too easy.. Clicking around in their fancy control panel, until:

- they need help with their *self-managed* VPS;
- their email ends up in spam;
- they suffer from performance issues;
- they need to restore a backup;
- something breaks, because YES, things break

Way too little people are able to answer:

- when and how do you make backups?
- how do you monitor your servers and which services?
- how do you keep track of trend analysis?

Then I come by with necessary software. SNMP for trend analysis, Graphite for infrastructure health, Sensu for monitoring, Kibana, Ansible for configuration management..

Things that servers need but that customers have never even heard of.. because they can do everything in their control panel..

Until they come crying to me because it broke and they don’t even know how to get into SSH.

I think the ones to blame are VPS providers that tell the tale of how easy it is to install a control panel and never look at your server again.

Customers become responsible for something *business-critical*! Yet they don’t know how it works.

I am so sick of the stupidity and illogical reasoning of clients.

Client: Descriptions are no longer syncing. Can you please fix.
Me: Problem fixed and deployed.

Client: All the descriptions got overwritten by the sync descriptions. Can you please have manual uploads overwrite the descriptions that sync (but basically auto guess what the client wants). We may need a toggle.

Me: Toggle added.

Client: Can you go through the 100+ sites backups and restore all the product descriptions?

It's like are you serious right now!!??

Back to the cheeseburger concept here...

Client: Can I have a cheeseburger (comes with pickles, onions, tomatoes, lettuce), no pickles. A Coke? Oh, but I would like pickles on my cheeseburger.

Tender: Here is your order.

Client: Why did you put pickles on this!!?? I asked for NO pickles!

Tender: You added pickles towards the end, so we put the pickles in.

Client: No! I thought you would have known based off of my original statement that I asked for a cheeseburger with no pickles. That is the override!

Narrator: See how illogical things can get. We can't just assume/guess based off of illogical reasoning.

And this, ladies and gentlemen, is why you need properly tested backups!

TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.

So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> waitwut?.jpg
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> fubar.scene
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken

Fuck my life

Son a fucking bitch!
I ordered a brand new shiney SSD for my Inspiron 7573 which came in today. I was all set up with system backups just in case and install media for Debian and Windows standing by. I pull out my tools and realize that the laptop isn't going to have a screw inside to hold the drive in since it didn't come with a SSD. No big deal, I pull out the magnetic bowl of extra screws from all the previous repairs I've done and start looking for the screw I need. I can't find the screw I need so I go online hoping maybe I'll find someplace in town that will have one single screw with no luck.

Now I get to wait 2 more days for the screw I need to arrive from Amazon.

My Sunday Morning until afternoon. FML. So I was experiencing nightly reboots of my home server for three days now. Always at 3:12am strange thing. Sunday morning (10am ca) I thought I'd investigate because the reboots affected my backups as well. All the logs and the security mails said was that some processes received signal 11. Strange. Checked the periodics tasks and executed every task manually. Nothing special. Strange. Checked smart status for all disks. Two disks where having CRC errors. Not many but a couple. Oh well. Changing sata cables again 🙄. But those CRC errors cannot be the reason for the reboots at precisely the same time each night. I noticed that all my zpools got scrubbed except my root-pool which hasn't been scrubbed since the error first occured. Well, let's do it by hand: zpool scrub zroot....Freeze. dafuq. Walked over to the server and resetted. Waited 10 minutes. System not up yet. Fuuu...that was when I first guessed that Sunday won't be that sunny after all. Connected monitor. Reset. Black screen?!?! Disconnected all disks aso. Reset. Black screen. Oh c'moooon! CMOS reset. Black screen. Sigh. CMOS reset with a 5 minute battery removal. And new sata cable just in cable. Yes, boots again. Mood lightened... Now the system segfaults when importing zroot. Good damnit. Pulled out the FreeBSD bootstick. zpool import -R /tmp zroot...segfault. reboot. Read-only zroot import. Manually triggering checksum test with the zdb command. "Invalid blckptr type". Deep breath now. Destroyed pool, recreated it. Zfs send/recv from backup. Some more config. Reboot. Boots yeah ... Doesn't find files??? Reboot. Other error? Undefined symbols???? Now I need another coffee. Maybe I did something wrong during recovery? Not very likely but let's do it again...recover-recover. different but same horrible errors. What in the name...? Pulled out a really old disk. Put it in, boots fine. So it must be the disks. Walked around the house and searched for some new disks for a new 2 disk zfs root mirror to replace the obviously broken disks. Found some new ones even. Recovery boot, minimal FreeBSD Install for bootloader aso. Deleted and recreated zroot, zfs send/recv from backup. Set bootfs attribute, reboot........
It works again. Fuckit, now it is 6pm, I still haven't showered. Put both disks through extensive tests and checked every single block. These disks aren't faulty. But for some reason they froze my system in a way so that I had to reset my BIOS and they had really low level data errors....? I Wonder if those disks have a firmware problem? So that was most of my Sunday. Nice, isn't it? But hey: calm sea won't make a good sailor, right?

I'll admit - I come from a WordPress background of almost 9 years in the making. I guess I can justify it because of all of the sites I created using it, it was the best that it could be on WP. Fast, efficient, custom - none of that off-the-shelf themeforest crap. I created everything custom. I actually knew what was going on behind the scenes of WP.

And then a buddy of mine and I had an idea for a new company/software project. I was smart enough to know that WP was not the foundation for this, so I did some NodeJS/Express tutorials. Started learning React, and really getting into the Javascript world.

And now I'm wondering WHY IN THE ABSOLUTE FUCK I ever bothered trying to become an expert in WP. It's the largest use of PHP in the fucking world and it doesn't even have native composer support. And by the time you actually get your project set up using composer you have to add a fucking mirror of the wordpress.org plugin repo to get anything to work. It's 2018 and you'd think that WP and composer would have all of this shit figured out by now.

And don't get me started on git - as soon as you have more than 1 person working on a WP site, I hope you have hourly backups of your DB because someones work will get overwritten. So you all either need to work on the same staging area of work around each other by pushing/pulling the DB and schedule your workflows.

I guess WP CLI and the REST API are a step in the right direction, but the foundation of everything is just so fucked up.

I don't feel like I've wasted my web dev career, but I definitely wish I had started down this path a lot earlier. I guess you don't know what you don't know. Thanks for reading!

Two things before this all:
- I fucking love gitlab so far
- I miss the fuzzy searching from sublime text, as vsCode still can't do it properly..

I was fed up with all the shitty overbloated git deployment scripts, sync scripts, automatic backup solutions and hosted git servers out there, so now my own solution is:

- remote git cloned local files
- local files are synced via dropbox, to easily edit them on any device
- all changes and deleted files are saved up to 1 year on dropbox
- remote has gitlab running and webhooks setup
- the webhooks point to my node scripts, which then rebase the code to its dedicated dev server
- daily server backup with 7 days roll
- cold storage backup each 30 days

Sounds like overkill, but from my experience, you really can't have enough places that have a backup, especially coldstorage backups.

My goal in general though is to have everything on my computer backupped and ready to go asap, if something happens.

I wanted to just use a virtual machine for development stuff, but that wouldnt be able to run on my laptop, so I need a more general solution, where I sync all configs and all projects across. (and have some sort of basic list of tools needed, so I dont need to remember them)

Found for example something for vscode to sync its settings and plugins via any sort of git, will give it a try in near future too.

So I've been working on this website now for a while and I'm finally near the end.

The client suddenly writes to me to ask me why there is captcha on some of the forms (register, post). He asks me to remove all captchas.

I explained to him why captchas are important and I even told him one old story of mine (basically I was noob and made that fatal mistake and suffered).

His response is to remove them anyway he didn't like them.

My response?
I keep this conversation in text form and screenshots, and I keep backups of it.

I have a feeling I'll need them.

I just noticed that my desktop still has a DVD-RW in it that I never used.. which led me to think, I've got here like 5 DVD-RW's as well as 5 CD-RW's, but I never use them. Are there still any uses for them? Long-term backup storage for example? I've heard somewhere that data rot can happen over the course of a few years.. but would it really be a problem? I mean I've had these for over a year already and they look healthy to me. That said, they've never been used and just sat on the top of my desk in their enclosures all the time.

FWIW, I'm having 10TB of storage (as well as some other smaller stuff here and there.. maybe 500GB or so) in various hosts and 3 mirrors, so the need for backups is already fulfilled.. it's merely a curiosity of mine as in "ah, I still have these things huh".. so yeah.

I think that I've bought these DVD's for something like €6 for 5 pieces in a local store that doesn't specialize in this sort of thing, so I'm fairly certain that I could get them for cheaper elsewhere.. think the likes of AliExpress etc. Would they be a viable technology to maybe still invest in over hard drives for long-term storage purposes, considering that the cost of a hard drive these days is only €50 for 1TB? But I guess that tapes would be better for this purpose, no?

Fuck google cloud platform. My server has been down for last 4 days. Stupid reason google gives me is that it does not have resources available in my zone. Why the fuck do you start a hosting company if you cannot provide RAM and CPU. On top of that their support is so bad that after 20 emails, 4 chat tickets, 3 phone calls nobody knows the issue I am facing. They just give the links to their ultra stupid documentarion.
Now all my 6 projects are down. Clients are getting impatient. I cannot do any work and googles support is the worst.
They dont even want to understand the issue, dont know how they will solve it.

I have created AWS instance now and migrated to AWS. But i have old backups which are useless on AWS. To get the latest backups i need google cloud instance to get started but stupid google does not have resources. How hard it is to add 1 CPU and 1GB RAM?

Sorry to keep whining about my stupid fucking job, but y'all, I think I'm nearing my limit.

There's some good...I am pretty much free to resolve issues any way I want to, as the only other person in the company who "codes" only knows one old ass language that doesn't apply to 90% of the rest of the tech stack at all, and some SQL - all of that to say, we may disagree, but ultimately, these matters are always deferred to me at the end of the day, insofar as the actual implementation goes (which is to say I am not micromanaged). At least as far as non-visuals are concerned, because those of course, are the most important things. Button colors and shit, woo hoo**. That's what we should focus on as we're bringing in potentially millions of dollars per month - the god damn button color and collapsible accordions based on data type over the shit ass DB performance bottleneck, the lack of redundancy or backups (aside from the one I made soon after I started -- literally saved everyone today because of that. My thanks? None, and more bullshit tasks) or the 300GB+ spaghetti code nightmare that is the literal circulatory system of the FUCKING COMPANY. Hundreds of people depend on it for their livelihoods, and those of their families, but fuck me in the face, right? I'm just a god damn nerd who has worked for the federal government, a handful of fortune 500's, a couple of fortune 100's, some startups, etc. But the fuck do I know about the lifecycle of companies?

I could continue ranting, but what's the point? I've got a nice little adage that I've started to live by, and y'all might appreciate it: "If everything is a priority/is important, nothing is". These folks just don't fucking get it. I'm torn because, on the one hand, they waste my time and kinda underpay me, in addition to forcing me to be onsite for 50 hours a week. They don't listen to me, couldn't give a flying shit about my experientially based opinions. I'm just a fucking chimp with a typewriter, there to take commands like a fucking waiter. But there's a lot of job security, assuming I don't fucking snap one day, and the job market for devs (I'm sure I don't need to tell you) is hostile atm. I'm also drinking far more than usual, and I really need to do something about that. It's only wednesday - I think...not 100% on that truth be told, and I logged my fourth trip to the liquor store this week already.

**Dear backenders - don't ever learn front end, or if you do, just lie about it to avoid being designated full stack. It's not worth it.

"let's use git for this game jam"

Wait! Don't go! I love git and use it on every project I work on! You'll have to hear me out here.

This was 4 years ago, at my first Global Game Jam. Every jam and game I'd worked on up to that point, I was the only Dev; no need for git, as backups were more than enough. I joined a group with high hopes for the game jam, with three coders and a proper art team.

The entire jam was "1 step forward 2 steps back", as git somehow constantly overwrote code as fast as we could write it.

By the end of the jam we barely had anything to show for our hard work. The takeaway isn't even about git. It's simply to never work with other people. Git is a great protocol but it can't stop people from accidentally fucking other people over. Every jam since, I've worked on my own and had a far better time of it.

Back when SharePoint was still foreign to me, and I didn't know the pain of administrating it, I had the idea that files were copied to my local machine. I saw no need to preserve backups from before I started, especially since they already existed on the server, so I got rid of them.

Also hooked up to SharePoint was an email handler. Whenever a case was created or deleted, an email went out to the entire department. Guess what happened when I deleted 250,000 records?

Fortunately, SharePoint has a recycle bin. Unfortunately, restoring those files generated another 250,000 emails. To the whole department.

I bought many donuts to appease the crowd baying for my blood.

The client doesn't want to give me her PIN code from GoDaddy but I need it to make changes for her.

She told me that GoDaddy's Customer Support told her she can't give her PIN to anyone. I understand that. I told her what to do but she still wants me to do it.

She came up with the idea of teleconference between me, her and GoDaddy (is that even possible?). We live in two different countries.

She could just do it by herself (as I told her what and how to do) or give me the PIN... Nope, she thinks that it's my business to make things up.

Boss wants me to carry on this because she's difficult and may make us bad PR even if she's not right. He doesn't want a shitstorm to handle.

We made few projects for her in the past, she gave us access to all her WordPresses, FTPs, backups, authinfo codes etc but still doesn't trust us. She always thinks dozen times before she gave us some data.

And she's not even a business client. She runs a few blogs about her hobbies. She doesn't make money from them. It's not a big deal but she treats it like a treasure.

It's not easy to be gentle and kind :)

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

!dev

Im working with morrons...

So someone had bright idea of having clone setup of our servers in other hosting company (in previous company we have rendundant setup). Whatever, maybe they want to be resistant to thermonuclear war or whatever, like the project wasn't underfunded already... Whatever, fuq it.

And with that, I have like really, really really fucking ABSOLUTELY BULLSHIT STUPID questions thrown at me.

So, this particular instance of bullshit started with trivial, literally "how much storage capacity we will need" I anwsered at least 4TB, preferably on redundant disk array, I've added small table what uses what and how much etc.

Than I got mail back...
"Thats not enough information:
1. What we need to say to company ABC
2. What we need to say to company XYZ
3. (this one actually had sense) Backups conception
4. **WILL WE PAY FOR SERVERS**
5. other important things (literally)"

So let's break it down.
Im backend guy. What the fuck do I know what you gonna say to XYZ or ABC. I dont give a shit, for me it's clicking setup new server and Im done for, you are overcomplicating as shit and require special care from hosting company that you will pay extra.
Next one, the killer one. What the fucking fuck. my anwser was literally "yes, we need to pay for servers, servers arent free."

Now tell me. How the fuck it is possible that someone can be such an idiot to ask questions like that. and I dont mean #3. maybe even #1 and 2 is like looking someone to throw responsibility on. But why the fuck I have to anwser mail that literally asked me if servers are free.

No, fuck off idiot, I have actual work. Take your bullshit and spread it somewhere else.

E:

and before anyone asks. No, Im not working in kindergarden but I often feel like I am indeed working within kindergarden full of 30+ mentally handicapped toddlers.

As a Dev in college working for teams in college, I don't really have a need to use git, since most of 'self proclaimed prestigious programmer child prodigies' I work with have no idea what it is; but I use it anyways as good programming practice and ease of backups.

So I tried using a GUI client after months of the git bash, and even though I looked up a few tutorials (was embarrassed the whole fucking time). I ended up adding, committing and pushing via bash.

Can anyone explain me how is the GUI client helpful in large projects and stuff?

Today was a SHIT day!

Working as ops for my customer, we are maintaining several tools in different environments. Today was the day my fucking Kubernetes Cluster made me rage quit, AGAIN!

We have a MongoDB running on Kubernetes with daily backups, the main node crashed due a full PVC on the cluster.

Full PVC => Pod doesn't start
Pod doesn't start => You can't get the live data
No live data? => Need Backup
Backup is in S3 => No Credentials
Got Backup from coworker
Restore Backup? => No connection to new MongoDB

3 FUCKING HOURS WASTED FOR NOTHING

Got it working at the end... Now we need to make an incident in the incident management software. Tbh that's the worst part.

And the team responsible for the cluster said monitoring wont be supported because it's unnecessary....

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

i asked my dad for help with a GRUB issue (EFI file wasn't seen in my BIOS anymore, nor booted when pointed directly at, even after ALL THE CONFIGURATIONS POSSIBLE) and i walked away for a while, content he'd figure it out (there's still a few things he knows more than me about.) I come back 30 minutes later and he's zero-filled my main drive and is halfway through installing Win10. His reasoning? "I'm installing surveillance software since you won't give me your college passwords and I need access to your college's site and your account. I can't do that on Debian."

I didn't give him authorization for this, and I thought he had zeroed my backups drive too, but it turns out it was having I/O issues (my controller is finicky sometimes, a boot cycle with it removed fixed it, luckily I can't write to drives it doesn't like when it's being a shithead)

What do? I can't sue as he owns almost everything I use and the house I live in and would no doubt kick me out and take all "my" stuff, but I feel like this really can't go ignored. I can't just talk to him about it as he thinks anything he wants done has to be done as he sees himself as above all other people, so he just shouts me down...

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

People just can’t show up on time. I respect people’s time and I do not waste a minute of anyone s time. I do not like it. Time is a precious resource and we should all treat it as such. I expect the same from other people too.

There was a meeting supposed to happen a month or so back. Me and another guy. This meeting was basically giving him the resource utilization and some cost related details for the budget our team used up. They wanted a more granular report than the generic ones we submitted.

He scheduled it once and about ten minutes before he cancelled it. Fair enough.

A week later he sent an invite and the next day he cancelled it a few hours before the meeting. Fair enough I’m still not pissed.

A month or so since the first scheduled meeting, he scheduled a meeting and I turn up and he s not there and I wait for about half hour before I leave. Next day I get an email saying he s sorry. Now I’m pissed.

A couple weeks back he schedules a meeting and I turn up and I could see through the glass door and he s talking to someone in the room he s in. He signals to me to wait for five minutes. It takes about 20. Finally I go in and he s relieved so he can finally get the data from me. I tell him I waited for twenty minutes and I have to somewhere and asked him to reschedule the meeting. He asks when and I tell him a time where I know he won’t turn up. About 8 30 in the morning. He says yes.

Next day I show up he s not there and I wait for ten minutes and just he s walking in to the floor I exit the room and run to my place. He comes over and asks and I tell him I waited for sometime and I’m busy now and he mumbles and goes away.

By this time it is already a week or so after the deadline. The bigger boss from finance turns up and asks me why I haven’t given them the data I asked for. I tell him they schedule meetings and don’t turn up on time and my other work is getting affected productivity blah blah and he says okay make the report and give it to me. I tell him no I’m not wasting anymore time on this.

He goes to my boss creates a little ruckus my boss comes over and he goes wtf. I tell him what happened. He says it’s okay give them the report. I’m like I don’t know where I kept it must be somewhere I can’t waste anymore time on this. Guy from finance joins us at this point and angrily says he’ll find it himself and don’t need our help anymore.

The file is in my laptop I use for travel. Just a local copy. Zero fucking backups. And I just deleted it.

Fuck. You.

When I say I'm a developer, it means that, in simpleton language, that I don't do your backups, I don't "repair" you PC because you're a baffon and I certainly do not edit company's logo which is in .png format deliver it in .cdr file and expect it as, as good and perfect as original, So fuck you, you impatient , imbecile , pompous prick I've ever met

> "Yeah take as much time as you want, but do it quickly"

I mean what the fuck does that even mean, if you want it done quickly then say so, be a man and face it and don't hide behind the excuse of "take as much time as you want"

Fucking idiots, little do they know when I'm in the zone I need time to think before act so I don't get blamed for "un-satisfied" product in the future.

So I've decided to leave loopholes in the current project I've been working on, deliver it, and leave the company and make their lives living hell, I know this is beneath anyone's standards but I have to do this to teach them how to treat a person properly,FUCKKKKKKKKKKKKKKK,

This is going to be fun

This shit is long story of my computer experience over my lifetime.

When I was young I got my first PC with windows it was not so bad. It required safe shut down of it’s fat32 partition. From time to time I needed to reinstall it cause of slow down but I got used to it I was only a gamer.

Time passes and I got more curious about computers and about this linux. Everything worked there but installation of anything was complete madness and none of windows programs worked well, and I wanted to play games and be productive so I sticked with windows.

I bought hp laptop once with nvidia card, it was overheating and got broken. So I bought toshiba and all I told to the seller was I want ATI card. Took me 5 minutes to do it and I was faster then my friend buying pack of cigarettes because I was earning money using computer.

Then I grown up running my small one person programming businesses and I wanted to run and compile every fucking program on this world. I wanted linux shell commands. I wanted package manager, and I wanted my os to be simple because I wasn’t earning money by using my os but by programming. So after getting my paycheck I bought mac. I can run windows and linux on vm if I need it. I try not to steal someones work so I didn’t want to run hackintosh. I am using this mac for some time.

Also I use playstation for gaming. Because I only want to run and play game I am not excited about graphics but gameplay. I think I am pragmatic person.

I can tell you something about my mac.
When I close lid it go sleep when I open it wakes up instantly. I never need to wonder if I want to hibernate or shut down or sleep and drain battery. It is fucking simple.
When I want to run or open something it doesn’t want me to wait but it gives me my intellij or terminal or another browser or whatever I search for. Yeah search is something that works.
Despite it got 8 gigs of ram I can run whatever number of programs I want at the same speed. The speed is not very fast sometimes but it’s constant fast.
I have a keychain so my passwords are in one place I can slow down shared internet speed, I can put my wifi in monitor mode and I don’t need to install some 3rd party software.
And now I updated my mac to high sierra, cause it’s free and I want to play with ios compilation. Before I did it I didn’t even backup whole work. I just used time machine and regular backups. And guess what, it still works at the same speed and all I did was click to run update and cook something to eat.
When I got bored I close the lid, when got idea open lid and code shit, not waiting for fucking wakeup or fucking updates.

I wanted to rant apple products I use but they work, they got fucking updates all along at the same time. And all of updates are optional.

I cannot tell that about all apple products but about products I use.

I think I just got old and started to praise my limited time on this world. Not being excited about new crap. When I buy something I choose wisely. I bought iPhone. I can buy latest iPhone x but I bought iPhone 7 cause it’s from fucking metal. And I know that metal is harder then glass, why the fucking apple forgot about it? I don’t know.
I know that I am clumsy and drop stuff. Dropped my phone at least 100 times and nothing.

I am not a apple fan boy I won’t buy mac with this glowing shit above keyboard that would got me blind at night.
I buy something when I know that it can save my time on this world. I try to buy things that make me productive and don’t break after a year.

So now piece of advise, stop wasting your time, buy and update wisely, wait a week or a month or a year when more people buy shit and buy what’s not broken. And if something’s broken rant this shit so next customer can be smarter.
Cheers

I have found a DVD with a Windows XP backup file with my late mother's photos on it. However, nothing seems to open the file. I, of course, did a lot of Google searches on how to do this and what it seems to come down to is a need to run the restore on an XP, Windows 7, or Server 2008 machine. There are several legacy ntbackup restore programs out there, and Microsoft seems to still have a page for downloading that, but it relies on the long-deprecated Removable Storage Service that no longer exists on Windows 10. So, a) Microsoft are a bunch of jerks for not supporting restores of legacy backups and b) does anyone have a successful method you've used to get a BKF file to restore? All the things I've tried say the BKF file is corrupted, but how to know for sure when Removable Storage Service isn't present?

Not a productive day. I wanted to restore part of database from test server but downloading data using reat api was getting timeouts.

I made simple app that dumps data and I launched process in screen and started watching Dragon ball
I like those old first series with young Songo looking for his grandpa.

After about 2 hours and partial backup still not finished I started to cut crap data from backup script will finish it tomorrow.

Looks like my ux is still ok and views got almost all approved.
Need to fix some shit on backend but I need those backups to work for more complicated customers.

Luckily frontend developer is back so he will handle visual parts.

Maybe we will release new features on the beginning of February.

Me: "Ok, downloaded the (windows) iso, now I need to create a bootable usb"
* Pulls out 4Gb usb drive *
Father: "That iso won't fit on that one"
M: "Oh yeah right, do you have one for me"
F: "Sure"
* hands over 64Gb usb drive *
M: "thanks"
* Checks content *
* "Backups 15" directory and other old files *
"Guess I can use that"
* dds iso on usb drive *
dd: "done"
M: * Reboots *
Pc: "Nani the fuck is that supposed to be"
M: * searches online for solutions, tries out 2 different ones, one being provided by microsoft *
P: "lol no"
F: * comes in *
"Is it working?"
M: "Nah, windows a bitch"
F: "ok, can you copy me some files on the usb in the meantime?"
M: "Sure..."
{ How did he know that the drive is currently empty? }
* copies files onto stick *
F: * sees empty usb drive*
"Did you delete the contents on the usb?"
M: "Kinda, that happens when you create a bootable usb drive"
F: "..."
M: "Why do you ask, was the something important on the drive?"
F: "Idk, doesn't matter anyways now, since the contents are gone"

Btw, it still doesn't boot from the usb drive. Windows 10 iso is a bitch.

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

Turned up on customer site yesterday to do a says SME work for them like I have done every week for last 3 months..

As I walk in they took a decision 15 mins earlier to power off the platform I'm working on to do a backup ( on a big data platform?) and its down till 13. 30...

Irony? The minute they finally let me turn it on New data arrives in the platform so their backup is out of date and they wouldn't need backups if they'd followed my original design and distributed it over two data centres....

Oh and they 'forgot' I was coming so there was little / nothing to do for the rest of the day either

Clients can be a PITA but I can't really complain.... Easy day though!

tldr: Deleted Win10, turned out for better
---------
**This happened a few months ago**
---------
So one day I decide I have had enough of Windows 10 and wanted to go back to Linux (A long time ago I had dual booted Ubuntu, but messed it up changing video drivers). So I create a Mint installation USB and get to work. I boot up the installation and delete the old partitions from the Ubuntu install. I install Mint and boot it up, everything works fine and dandy! I decide I want to get back on Windows to get back a few files that I wanted copies over. I turn off the computer and *try* to boot up Windows. I get an error message that I am UNABLE to boot. WHAT! After further checking, I realize that I had deleted the MBR partition of Windows. Pretty sure I could remake it if I tried hard enough, but I am starting to realize that it feels good to be totally MS independent! Now I am using all open-source software available to Linux and have no need for Windows. I do miss some of the games though...

PS: No files were lost due to backups. Save Lives, Do Backups!

1. Windows domains as user@domain
2. Starting tape backups at 13:37, realizing they need about 5 hours and all company servers run on ~5% speed for others
3. Repeatedly opening and closing devRant multiple times a minute realizing it has been open currently
4. accidentally executing "apt-get update && apt-get update"
5. Trying every earlier password if the current windows domain password timeouted until I come to an not yet used one.

Had a Nas with a single 3tb seagate HDD in it.
It ran well for half a year and it was my main backup and a time machine for my dad.

The time came that my budget was allowing a second drive for redundancy so I powered it off, added the second drive and powered it back on.

😐😓😧😭

The drive did indeed die and yes, it was one of those drives with an extremely high failure rate.

My dad was pretty mad that his backups were gone even though he didn't need them.

So my biggest lesson from this was to always encrypt such drives because dads backup wasn't and my files and such weren't either, so someone could restore our hole life's from the drive.
So I can't Rma that fucker.
Zfs at rest encryption ftw!

By the way, writing this I noticed that I didn't need to power the Nas down to add the second drive....
Ffffffffuuuuuuuuucccckkkkkk.

Another more recent thing was a refurb 4tb we red that I bought used for a bargain.
It reported 2 unwritable sectors but I didn't care for the money.

After about a month, it died.
The interesting part is how it died.
It spinns up, gets detected, you can access the data.
You can copy the data.

But after a few moments of continues load, all operations start timing out and the drive either disconnects completely or the zpool degrades and shuts down.

In the first case, replugging brings the drive back untill it does it again.

On zpool degradation only a reboot brings it back.

Put a fan on it in case it was overheating but that didn't fix it.

So I've been using Duet on my iPad Pro for a couple years now (lets me use it as an external monitor via Lightning cable) and without issue. Shit, I've been quite happy with it. Then the other day, whilst hooked up to my work laptop, there was a power fluctuation that caused my laptop to stop sending power to connected devices. Which is fine - I have it plugged into a surge protector so these fluctuations shouldn't matter. After a few seconds the laptop resumed normal operation and my connected devices were up and running again.

But the iPad Pro, for some reason, went into an infinite boot loop sequence. It reboots, gets to the white Apple logo, then reboots again.

In the end, after putting the iPad into recovery mode and running Apple's update in iTunes (as they recommend), it proceeds to wipe all my data. Without warning. I lost more than a couple of years of notes, illustrations and photos. All in one fucking swoop.

To be clear, you get 2 options in iTunes when performing a device update:

1. UPDATE - will not mess with your data, will just update the OS (in this case iPadOS)
2. RESTORE - will delete everything, basically a factory reset

I clicked UPDATE. After the first attempt, it still kept bootlooping. So I did it again, I made sure I clicked UPDATE because I had not yet backed up my data. It then proceeds to do a RESTORE even though I clicked UPDATE.

Why, Apple? WHY.

After a solemn weekend lamenting my lost data, I've come a conclusion: fuck you Apple for designing very shitty software. I mean, why can't I access my device data over a cabled connection in the event I can't boot into the OS? If you need some form of authentication to keep out thieves, surely the mutltiple times you ask me to log in with my Apple ID on iTunes upon connecting the damn thing is more than sufficient?! You keep spouting that you have a secure boot chain and shit, surely it can verify a legitimate user using authenticated hardware without having to boot into the device OS?

And on the subject of backing up my data, you really only have 2 manual options here. Either (a) open iTunes, select your device, select the installed app, then selectively download the files onto my system; or (b) do a full device backup. Neither of those procedures is time-efficient nor straightforward. And if you want to do option b wirelessly, it can only be on iCloud. Which is bullshit. And you can't even access the files in the device backup - you can only get to them by restoring to your device. Even MORE bullshit.

Conversely, on my Android phone I can automate backups of individual apps, directories or files to my cloud provider of choice, or even to an external microSD card. I can schedule when the backups happen. I can access my files ANYTIME.

I got the iPad Pro because I wanted the best drawing experience, and Apple Pencil at the time was really the best you could get. But I see now it's not worth compromise of having shitty software. I mean, It's already 2021 but these dated piles of excrement that are iOS and iPadOS still act like it's 2011; they need to be seriously reviewed and re-engineered, because eventually they're going to end up as nothing but all UI fluff to hide these extremely glaring problems.

I think I have configuphobia.

When you need to setup configurations for like anything, it's always super loosely coupled and can break when you even breathe on it.

Database table columns? Configured once.
Authorization management? Create a user and configure the password and username in the application.
Backups? Configure the network path to backup to.

All these things are so EASY to break!
Maybe I'm overcautious, but I really dislike it.

There are ways around it of course, like documentation and automation, but it's all so much work. And even then it's still loosely coupled.

What do you all do to keep your configs working without getting nightmares?

External Storage recommendation questions.

Im in need of some sort of external storage, either a harddrive or a NAS server, but idk what to get.

Price should be reasonable for the security and storage space it gives, so heres what i figured so far for pros and cons:

NAS Server:
+ Bigger capacity
+ Raid option
+ Easily expandable
+ Always accessible via the network (local)

- Difficult to transport (not gonna do that, but still)
- Expensive
- Physically larger
- Consumes power 24/7 (i dont pay for power currently)

Harddrive:
+ Easy to pack away and transport
+ Cheaper

- If drive fails, youre fucked
- If you want larger capacity, you end up with two external backups

What do you guys do? Im not sure what i should do :i
Any advice is appreciated.

It will be used for external backup, as mentioned. For my server and my own pc.

This question might make you lose a brain cell because of stupidity in the question. Read with caution

Is there a way to compile a game for Windows from Linux in Unreal engine? I did google some posts but the answer was either use a Virtual machine which will not be done or use the the theoretical method of using mingw but the forum posts state that it will be tricky business or use a windows machine. I have dual booted windows with linux on my machine.

However since the machine has a 512 gb ssd most of the storage space is devoted to unreal engine which takes 47 gigs in itself and have a lot of programs installed I have a usable 20 gigs left out of 145 gig partition. Windows has around 318 gigs of storage to it but I have 100 gigs free at most. So after installing the windows sdk, visual studio with extensions, unreal engine and some other stuff I don't have much space left for myself. I need that much space since I install a lot of games to my ssd. So now I cant load my bigger projects for playing on my windows. I could use my hdd which is mostly used for backups and 100+ gig stuff. Though the hdd's are of course far slower than ssd's which shouldn't be a problem however last time I used visual studio it ate more than 2 gigs of ram for a solution meaning that the compiler has very low memory for itself to actually compile so for any large files the hdd has more of a bottleneck.

Oh and I can't upgrade my ssd's or ram because I don't have enough money.

Thanks for the answers in advance