I think server down time is because of rain 🌧

customer who didn't paid for 3 months: why is our server down?

me: server?. what server?.

Good Morning!, its time for practiseSafeHex's most incompetent co-worker!

Todays contestant is a very special one.

*sitcom audience: WHY?*

Glad you asked, you see if you were to look at his linkedin profile, you would see a job title unlike any you've seen before.

*sitcom audience oooooooohhhhhh*

were not talking software developer, engineer, tech lead, designer, CTO, CEO or anything like that, No No our new entrant "G" surpasses all of those with the title ..... "Software extraordinaire".

*sitcom audience laughs hysterically*

I KNOW!, wtf does that even mean! as a previous dev-ranter pointed out does this mean he IS quality code? I'd say he's more like a trash can ... where his code belongs

*ba dum tsssss*

Ok ok, lets get on with the show, heres some reasons why "G" is on the show:

One of G's tasks was to build an analytics gathering library for iOS, similar to google analytics where you track pages and events (we couldn't use google's). G was SO good at this job he implemented 2 features we didn't even ask for:

- If the library was unable to load its config file (for any reason) it would throw an uncatchable system integrity error, crashing the app.
- If anything was passed into any of the functions that wasn't expected (null, empty array etc.) it would crash the app as it was "more efficient" to not do any sanity checks inside the library.

This caused a lot of issues as some of the data needed to come from the clients server. The day we launched the app, within the first 3 hours we had over 40k crash logs and a VERY angry client.

Now, what makes this story important is not the bugs themselves, come on how many times have we all done something stupid? No the issue here was G defended all of this as the right thing to do!

.. and no he wasn't stoned or drunk!

G claimed if he couldn't get the right settings / params he wouldn't be able to track the event and then our CEO wouldn't have our usage data. To which I replied:

"So your solution was to not give the client an app instead? ... which also doesn't give the CEO his data".

He got very angry and asked me "what would you do then?". I offered a solution something like why not have a default tag for "error" or "unknown" where if theres an issue, we send up whatever we have, plus the file name and store it somewhere else. I was told I was being ridiculous as it wasn't built to track anything like that and that would never work ... his solution? ... pull the library out of the app and forget it.

... once again giving everyone no data.

G later moved onto another cross-platform style project. Backend team were particularly unhappy as they got no spec of what needed to be done. All they knew was it was a single endpoint dealing with very complex model. There was no Java classes, super classes, abstract classes or even interfaces, just this huge chunk of mocked data. So myself and the lead sat down with him, and asked where the interfaces for the backend where, or designs / architecture for them etc.

His response, to this day frightens me ... not makes me angry, not bewilders me ... scares the living shit out of me that people like this exist in the world and have successful careers.

G: "hhhmmm, I know how to build an interface, but i've never understood them ... Like lets say I have an interface, what now? how does that help me in any way? I can't physically use it, does it not just use up time building it for no reason?"

us: "... ... how are the backend team suppose to understand the model, its types, integrate it into the other systems?"

G: "Can I not just tell them and they can write it down?"

**
I'll just pause here for a moment, as you'll likely need to read that again out of sheer disbelief
**

I've never seen someone die inside the way the lead did. He started a syllable and his face just dropped, eyes glazed over and he instantly lost all the will to live. He replied:

" wel ............... it doesn't matter ... its not important ... I have to go, good luck with the project"

*killed the screen share and left the room*

now I know you are all dying in suspense to know what happened to that project, I can drop the shocking bombshell that it was in fact cancelled. Thankfully only ~350 man hours were spent on it

... yep, not a typo.

G's crowning achievement however will go down in history. VERY long story short, backend got deployed to the server and EVERYTHING broke. Lead investigated, found mistakes and config issues on every second line, load balancer wasn't even starting up. When asked had this been tested before it was deployed:

G: "Yeah I tested it on my machine, it worked fine"

lead: "... and on the server?"

G: "no, my machine will do the same thing"

lead: "do you have a load balancer and multiple VM's?"

G: "no, but Java is Java"

... and with that its time to end todays episode. Will G be our most incompetent? ... maybe.

Tune in later for more practiceSafeHex's most incompetent co-worker!!!

Its Friday, you all know what that means! ... Its results day for practiseSafeHex's most incompetent co-worker!!!

*audience: wwwwwwooooooooo!!!!*

We've had a bewildering array of candidates, lets remind ourselves:

- a psychopath that genuinely scared me a little
- a CEO I would take pleasure seeing in pain
- a pothead who mistook me for his drug dealer
- an unbelievable idiot
- an arrogant idiot obsessed with strings

Tough competition, but there can be only one ... *drum roll* ... the winner is ... none of them!

*audience: GASP!*
*audience member: what?*
*audience member: no way!*
*audience member: your fucking kidding me!*

Sir calm down! this is a day time show, no need for that ... let me explain, there is a winner ... but we've kept him till last and for a good reason

*audience: ooooohhhhh*

You see our final contestant and ultimate winner of this series is our good old friend "C", taking the letters of each of our previous contestants, that spells TRAGIC which is the only word to explain C.

*audience: laughs*

Oh I assure you its no laughing matter. C was with us for 6 whole months ... 6 excruciatingly painful months.

Backstory:

We needed someone with frontend, backend and experience with IoT devices, or raspberry PI's. We didn't think we'd get it all, but in walked an interviewee with web development experience, a tiny bit of Angular and his masters project was building a robot device that would change LED's depending on your facial expressions. PERFECT!!!

... oh to have a time machine

Working with C:

- He never actually did the tutorials I first set him on for Node.js and Angular 2+ because they were "too boring". I didn't find this out until some time later.

- The first project I had him work on was a small dashboard and backend, but he decided to use Angular 1 and a different database than what we were using because "for me, these are easier".

- He called that project done without testing / deploying it in the cloud, despite that being part of the ticket, because he didn't know how. Rather than tell or ask anyone ... he just didn't do it and moved on.

- As part of his first tech review I had to explain to him why he should be using if / else, rather than just if's.

- Despite his past experience building server applications and dashboards (4 years!), he never heard of a websocket, and it took a considerable amount of time to explain.

- When he used a node module to open a server socket, he sat staring at me like a deer caught in headlights completely unaware of how to use / test it was working. I again had to explain it and ultimately test it for him with a command line client.

- He didn't understand the need to leave logging inside an application to report errors. Because he used to ... I shit you not ... drive to his customers, plug into their server and debug their application using a debugger.

... props for using a debugger, but fuck me.

- Once, after an entire 2 days of tapping me on the shoulder every 15 mins for questions / issues, I had to stop and ask:

Me: "Have you googled it?"
C: "... eh, no"
Me: "can I ask why?"
C: "well, for me, I only google for something I don't know"
Me: "... well do you know what this error message means?"
C: "ah good point, i'll try this time"

... maybe he was A's stoner buddy?

- He burned through our free cloud usage allowance for a month, after 1 day, meaning he couldn't test anything else under his account. He left an application running, broadcasting a lot of data. Turns out the on / off button on the dashboard only worked for "on". He had been killing his terminal locally and didn't know how to "ctrl + c a cloud app" ... so left it running. His intention was to restart the app every time you are done using it ... but forgot.

- His issue with the previous one ... not any of his countless mistakes, not the lack of even trying to make the button work, no, no, not for C. C's issue is the cloud is "shit" for giving us such little allowances. (for the record in a month I had never used more than 5%).

- I had to explain environment variables and why they are necessary for passwords and tokens etc. He didn't know it wasn't ok to commit these into GitHub.

- At his project meetups with partners I had to repeatedly ask him to stop googling gifs and pay attention to the talks.

- He complained that we don't have 3 hour lunch breaks like his last place.

- He once copied and pasted the same function 450 times into a file as a load test ... are loops too mainstream nowadays?

You see C is our winner, because after 6 painful months (companies internal process / requirements) he actually achieved nothing. I really mean that, nothing. Every thing was so broken, so insecure / wide open, built without any kind of common sense or standards I had to delete it all and start again ... it took me 2 weeks.

I hope you've all enjoyed this series and will join me in praying for the return of my sanity ... I do miss it a lot.

Yours truly,
practiseSafeHex

"OUR SERVER IS DOWN!!!!!!"
*ssh server*
*succesfully logged into the server*
"The server is very much up, sir."
"BUT THIS WEBSITE ISN'T WORKING ANYMORE!!!!"

Ah, so one of your websites on that server with 100s of websites on it is not working anymore. That doesn't mean that you're entire fucking server is down. Please learn the fucking difference.

When someone's calls and completely loses his shit (swearing etc etc) because we HAVE TO FUCKING HELP HIM BECAUSE THOUSANDS OF PEOPLE RELY ON THIS SITE WHICH IS DOWN.
Explained him calmly that its an unmanaged server which we literally don't have access to. Keeps on calling me all kinda things and then says he'll email some login details.

Bossman walks in, collegue goes like 'yo guess what just happened with linuxxx!'
Le me explains the story.

Bossman: email me his number. I'm going to call him. You treat my employees with fucking respect.

You know what?

Young cocky React devs can suck my old fuckin LAMP and Objective-C balls.

Got a new freelance job and got brought in to triage a React Native iOS/Android app. Lead dev's first comment to me is: "Bro, have you ever used React Native".

To which I had to reply to save my honor publicly, "No, but I have like 8 years with Objective-C and 3 years with Swift, and 3 years with Node, so I maybe I'll still be able help. Sometimes it just helps to have a fresh set of eyes."

"Well, nobody but me can work on this code."

And that, as it turned out was almost true.

After going back and forth with our PM and this dev I finally get his code base.

"Just run "npm install" he says".

Like no fuckin shit junior... lets see if that will actually work.

Node 14... nope whole project dies.

Node 12 LTS... nope whole project dies.

Install all of react native globally because fuck it, try again... still dies.

Node 10 LTS... project installs but still won't run or build complaining about some conflict with React Native libraries and Cocoa pods.

Go back to my PM... "Um, this project won't work on any version of Node newer than about 5 years old... and even if it did it still won't build, and even if it would build it still runs like shit. And even if we fix all of that Apple might still tell us to fuck off because it's React Native.

Spend like a week in npm and node hell just trying to fucking hand install enough dependencies to unfuck this turds project.

All the while the original dev is still trying TO FIX HIS OWN FUCKING CODE while also being a cocky ass the entire time. Now, I can appreciate a cocky dev... I was horrendously cocky in my younger days and have only gotten marginally better with age. But if you're gonna be cocky, you also have to be good at it. And this guy was not.

Lo, we're not done. OG Dev comes down with "Corona Virus"... I put this in quotes because the dude ends up drawing out his "virus" for over 4 months before finally putting us in touch with "another dev team he sometimes uses".

Next, me and my PM get on a MS Teams call with this Indian house. No problems there, I've worked with the Indians before... but... these are guys are not good. They're talking about how they've already built the iOS build... but then I ask them what they did to sort out the ReactNative/Cocoa Pods conflict and they have no idea what I'm talking about.

Why?

Well, one of these suckers sends a link to some repo and I find out why. When he sends the link it exposes his email...

This Indian dude's emails was our-devs-name@gmail.com...

We'd been played.

Company sued the shit out of the OG dev and the Indian company he was selling off his work to.

I rewrote the app in Swift.

So, lets review... the React dev fucked up his own project so bad even he couldn't fix it... had to get a team of Indians to help who also couldn't fix it... was still a dickhead to me when I couldn't fix it... and in the end it was all so broken we had to just do a rewrite.

None of you get npm. None of you get React. None of you get that doing the web the way Mark Zucherberg does it just makes you a choad locked into that ecosystem. None of you can fix your own damn projects when one of the 6,000 dependency developers pushes breaking changes. None of you ever even bother with "npm audit fix" because if security was a concern you'd be using a server side language for fucking server side programming like a grown up.

So, next time a senior dev with 20 years exp. gets brought in to help triage a project that you yourself fucked up... Remember that the new thing you know and think makes you cool? It's not new and it's not cool. It's just JavaScript on the server so you script kiddies never have to learn anything but JavaScript... which makes you inarguably worse programmers.

And, MF, I was literally writing javascript while you were sucking your mommas titties so just chill... this shit ain't new and I've got a dozen of my own Node daemons running right now... difference is?

Mine are still working.

Hello everyone, found this place recently, decided to bore you with one (or many) Navy story... tech Navy story. I'll start from the end.
Little backstory: I've deployed a simple domain setup on the ship I served, nothing fancy, a server, a switch, 10 computers, all Windows (details on that at another rant). I enter the ship Monday morning, and the XO tells me that he can't access his online folders.
OK, I say, I'll get to it. I fire up my laptop, try to RDP to the server (I know, I know, burn me at the stake later) no connection. WTF? Is the service down? I try pinging. No luck. I tried pinging the switch. OK. Looking at the switch admin panel, I see the server's port is dead. "OK, probably the cable." (we have old ethernet cables)
So, I drag my ass over to the server (same room with ship comms) with the cable tester to confirm that. What do I see?
The IMBECILES had pulled the plug from the server so that they could charge their mobile phones. I literally slammed my head against the door (calming exercise in case of spontaneous murder impulses - the things you learn at the Academy). My CO was nearby, and lucky for the guys, he heard me yell at them, while throwing mobiles and chargers around.
"But we thought it was OK, we just wanted to charge our-"
I kid you not, I reached for the firefighter's axe.
My CO grabbed me by the collar and dragged me to his room. I explained to him (between two cigarettes) that we MUST get a UPS and a server cabinet (budget constraints in the military are something that will give you people nightmares, trust me). I carefully explained to him that unless we got those, nothing would prevent the next moron from destroying confidential data and me from murdering him.
I plugged in and booted the server, after installing a multi socket extension. Two days after, surprise surprise, the server was off again. That was the first time I opened the door to the CO's room with a low kick. I must have looked like a psycho on drugs, he gave approval for the purchase in twenty seconds flat.
After that, I installed the UPS and the cabinet. Everything went inside, from the UPS to the very plugs. Just a locked box with cables coming out.
One of the guys came to my room, and asked if I could unlock the cabinet so that they could plug a "device" they needed.
I actually reached for my folding knife.
Disclaimer: The story above is TRUE. Even the almost violent parts.

I absolutely HATE "web developers" who call you in to fix their FooBar'd mess, yet can't stop themselves from dictating what you should and shouldn't do, especially when they have no idea what they're doing.

So I get called in to a job improving the performance of a Magento site (and let's just say I have no love for Magento for a number of reasons) because this "developer" enabled Redis and expected everything to be lightning fast. Maybe he thought "Redis" was the name of a magical sorcerer living in the server. A master conjurer capable of weaving mystical time-altering spells to inexplicably improve the performance. Who knows?

This guy claims he spent "months" trying to figure out why the website couldn't load faster than 7 seconds at best, and his employer is demanding a resolution so he stops losing conversions. I usually try to avoid Magento because of all the headaches that come with it, but I figured "sure, why not?" I mean, he built the website less than a year ago, so how bad can it really be? Well...let's see how fast you all can facepalm:

1.) The website was built brand new on Magento 1.9.2.4...what? I mean, if this were built a few years back, that would be a different story, but building a fresh Magento website in 2017 in 1.x? I asked him why he did that...his answer absolutely floored me: "because PHP 5.5 was the best choice at the time for speed and performance..." What?!

2.) The ONLY optimization done on the website was Redis cache being enabled. No merged CSS/JS, no use of a CDN, no image optimization, no gzip, no expires rules. Just Redis...

3.) Now to say the website was poorly coded was an understatement. This wasn't the worst coding I've seen, but it was far from acceptable. There was no organization whatsoever. Templates and skin assets are being called from across 12 different locations on the server, making tracking down and finding a snippet to fix downright annoying.

But not only that, the home page itself had 83 custom database queries to load the products on the page. He said this was so he could load products from several different categories and custom tables to show on the page. I asked him why he didn't just call a few join queries, and he had no idea what I was talking about.

4.) Almost every image on the website was a .PNG file, 2000x2000 px and lossless. The home page alone was 22MB just from images.

There were several other issues, but those 4 should be enough to paint a good picture. The client wanted this all done in a week for less than $500. We laughed. But we agreed on the price only because of a long relationship and because they have some referrals they got us in the door with. But we told them it would get done on our time, not theirs. So I copied the website to our server as a test bed and got to work.

After numerous hours of bug fixes, recoding queries, disabling Redis and opting for higher innodb cache (more on that later), image optimization, js/css/html combining, render-unblocking and minification, lazyloading images tweaking Magento to work with PHP7, installing OpCache and setting up basic htaccess optimizations, we smash the loading time down to 1.2 seconds total, and most of that time was for external JavaScript plugins deemed "necessary". Time to First Byte went from a staggering 2.2 seconds to about 45ms. Needless to say, we kicked its ass.

So I show their developer the changes and he's stunned. He says he'll tell the hosting provider create a new server set up to migrate the optimized site over and cut over to, because taking the live website down for maintenance for even an hour or two in the middle of the night is "unacceptable".

So trying to be cool about it, I tell him I'd be happy to configure the server to the exact specifications needed. He says "we can't do that". I look at him confused. "What do you mean we 'can't'?" He tells me that even though this is a dedicated server, the provider doesn't allow any access other than a jailed shell account and cPanel access. What?! This is a company averaging 3 million+ per year in revenue. Why don't they have an IT manager overseeing everything? Apparently for them, they're too cheap for that, so they went with a "managed dedicated server", "managed" apparently meaning "you only get to use it like a shared host".

So after countless phone calls arguing with the hosting provider, they agree to make our changes. Then the client's developer starts getting nasty out of nowhere. He says my optimizations are not acceptable because I'm not using Redis cache, and now the client is threatening to walk away without paying us.

So I guess the overall message from this rant is not so much about the situation, but the developer and countless others like him that are clueless, but try to speak from a position of authority.

If we as developers don't stop challenging each other in a measuring contest and learn to let go when we need help, we can get a lot more done and prevent losing clients. </rant>

Production is down, a coworker got himself locked in his own apartment so he can’t leave and another is late, the phone won’t stop ringing

And I don’t have the credentials to access the production server

Just a monday morning, everything’s fine 🔥😊🔥

Javascript is McDonald's.

1) everyone says they hate it....but they just keep going back.

2) very few people admit they keep gong back...

3) When McDonald's started doing salads, dressing nice, and delivering to tables it seemed a little much, you're a burger place. A few years later I'm writing my app in react JS, serving up eJS templates with my NodeJS server, running off a NOSQL JSon database, and munching down a Greek salad from McD's.

4) you start your burger (project) with high hopes. As you eat though....you start to regret it, but oh well, you're halfway in. By the end, never again, last time. A little while later, npm asks you if you'd like fries with that.

Feel free to disagree or add more!

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

I made a web app and a mobile app for a client two months ago. Yesterday I shut down the server because I wasn't paid yet despite my multiple warnings and reminders.

Under thirty minutes the money was in my bank account.

Sad thing is they weren't responding to me, ignoring or lullabying me with good cop bad cop acts, acting like friends etc...

Lowering my bar to people / agencies like this depresses me.

Btw they are my former agency, they have all the money in the world but turns out they are all jackasses after all...

Told my Boss Server is Down. He told me to go downstairs and work there.

Just got BUGS list from our Client and fuck- 95% of bugs are not even bugs :|

- No, changing the (not pre-decided) verbiage is not a bug
- Adding two more pages in the app is not a bug (what the fuck :|)
- No, APK file not running in iPhone is not a bug (goddamn :|)
- No, adding these "fuckin new" functionalities is not a bug (seriously ? :/)

AND

Mr "used to be a good coder" PM,

Getting "504 Timeout Gateway" error because Server is temporarily down is NOT a fuckin frontend bug

And No, writing Javascript with a proper design architecture is not a "complicated" way of coding
and fuckin No, Global variables and functions without any architecture don't make the programming "kind of better"

ps: And VB dot net is not a fuckin scripting language, VBScript is.

Thank you,
"buggy average coder"

Client: Can you provide some kind of guaranteed timeline that you're going to be able to move our website to our new servers with the optimizations implemented? I know you said it should take a week, but we have 3 weeks to get this moved over and we cannot afford to be double billed. I'm waiting to fire up the new server until you can confirm.

Me: As I said, it SHOULD take about a week, but that's factoring in ONLY the modifications being made for optimization and a QA call to review the website. This does not account for your hosting provider needing to spin up a new server.

We also never offered to move your website over to said new server. I sent detailed instructions for your provider to move a copy of the entire website over and have it configured and ready to point your domain over to, in order to save time and money since your provider won't give us the access necessary to perform a server-to-server transfer. If you are implying that I need to move the website over myself, you will be billed for that migration, however long it takes.

Client: So you're telling me that we paid $950 for 10 hours of work and that DOESN'T include making the changes live?

Me: Why would you think that the 10 hours that we're logged for the process of optimizing your website include additional time that has not been measured? When you build out a custom product for a customer, do you eat the shipping charges to deliver it? That is a rhetorical question of course, because I know you charge for shipping as well. My point is that we charge for delivery just as you do, because it requires our time and manpower.

All of this could have been avoided, but you are the one that enforced the strict requirement that we cannot take the website down for even 1 hour during off-peak times to incorporate the changes we made on our testbed, so we're having to go through this circus in order to deliver the work we performed.

I'm not going to give you a guarantee of any kind because there are too many factors that are not within our control, and we're not going to trap ourselves so you have a scapegoat to throw under the bus if your boss looks to you for accountability. I will reiterate that we estimate it would take about a week to implement, test and run through a full QA together, as we have other clients within our queue and our time must be appropriately blocked out each day. However, the longer you take to pull the trigger on this new server, the longer it will take on my end to get the work scheduled within the queue.

Client: If we get double billed, we're taking that out of what we have remaining to pay you.

Me: On the subject of paying us, you signed a contract acknowledging that you would pay us the remaining 50% after you approved the changes, which you did last week, in order for us to deliver the project. Thank you for the reminder that your remaining balance has not yet been paid. I'll have our CFO resend the invoice for you to remit payment before we proceed any further.

---

I love it when clients give me shit. I just give it right back.

Navy story time, and this one is lengthy.
As a Lieutenant Jr. I served for a year on a large (>100m) ship, with the duties of assistant navigation officer, and of course, unofficial computer guy. When I first entered the ship (carrying my trusty laptop), I had to wait for 2 hours at the officer's wardroom... where I noticed an ethernet plug. After 15 minutes of waiting, I got bored. Like, really bored. What on TCP/IP could possibly go wrong?
So, scanning the network it is. Besides the usual security holes I came to expect in ""military secure networks"" (Windows XP SP2 unpatched and Windows 2003 Servers, also unpatched) I came along a variety of interesting computers with interesting things... that I cannot name. The aggressive scan also crashed the SMB service on the server causing no end of cute reactions, until I restarted it remotely.
But me and my big mouth... I actually talked about it with the ship's CO and the electronics officer, and promptly got the unofficial duty of computer guy, aka helldesk, technical support and I-try-to-explain-you-that-it-is-impossible-given-my-resources guy. I seriously think that this was their punishment for me messing around. At one time I received a call, that a certain PC was disconnected. I repeatedly told them to look if the ethernet cable was on. "Yes, of course it's on, I am not an idiot." (yea, right)
So I went to that room, 4 decks down and 3 sections aft. Just to push in the half-popped out ethernet jack. I would swear it was on purpose, but reality showed me I was wrong, oh so dead wrong.
For the full year of my commission, I kept pestering the CO to assign me with an assistant to teach them, and to give approval for some serious upgrades, patching and documenting. No good.
I set up some little things to get them interested, like some NMEA relays and installed navigation software on certain computers, re-enabled the server's webmail and patched the server itself, tried to clean the malware (aka. Sisyphus' rock), and tried to enforce a security policy. I also tried to convince the CO to install a document management system, to his utter horror and refusal (he was the hard copy type, as were most officers in the ship). I gave up on almost all besides the assistant thing, because I knew that once I left, everything would go to the high-entropy status of carrying papers around, but the CO kept telling me that would be unnecessary.
"You'll always be our man, you'll fix it (sic)".
What could go wrong?
I got my transfer with 1 week's notice. Panic struck. The CO was... well, he was less shocked than I expected, but still shocked (I learned later that he knew beforehand, but decided not to tell anybody anything). So came the most rediculous request of all:
To put down, within 1 A4 sheet, and in simple instructions, the things one had to do in order to fulfil the duties of the computer guy.
I. SHIT. YOU. NOT.
My answer:
"What I can do is write: 'Please read the following:', followed by the list of books one must read in order to get some introductory understanding of network and server management, with most accompanying skills."
I was so glad I got out of that hellhole.

Had this recently with a client, mysql server of one of our shared hosting servers went down:

Senior engineer 1: heads up guys, mysql of {server name} is down, working on it! *calls second engineer in*
Support people: thanks for letting know! (in case clients call about it)

*triiiingggg*
Me: good afternoon, how can I help you?
Client: this site which we manage for a shared customer says it can't connect to the database...
M: is it hosted on {server name of mysql problems}
C: yes.
M: there's a mysql disruption there right now, we're working on it!
C: *starts guilt tripping me about thy they chose us for stability reasons and now this happens*
M: sir, I can't change this situation so you can go on and on about that but it's not going to help anyone.
C: okay, so what can I tell my client?
M: you can tell that we have a mysql server disruption right now and are working to fix it as soon as possible!
C: and what am I going to tell my client if they don't accept that answer?
M: you can tell that we are fixing this disruption as soon as possible.
C: yes you said that but what if they don't accept that answer, what am I going to tell them THEN?!
M: Listen, sir. We have a disruption right now. It's not fun but whether I tell this by writing it to you in a fairy tail or shout it at you, it's not going to make a difference.
We have a disruption and we are working on i....

*click*

Well, fuck you too.

My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

Manager: THE SERVER IS DOWN THE SERVER IS DOWN!!!!

Dev: Ok I’ll look into it

*5 mins later

Dev: Wow these are really strange logs, it’s like config values are being changed all over the place while I’m looking at it

Manager: Well I figured while you were looking into it I’d go i to the server settings and change everything I could find in order to try and get the server back up again. Two sets of hands are better than one, Is it up yet???

Dev: …No.

Manager: I THOUGT YOU SAID YOU’D LOOK INTO THIS. I NEED ANSWERS NOW. WHAT IS TAKING SO LONG?!?!?

Dev: …

I was hired as a senior software engineer. During handover I found out I'm actually replacing the CTO.

I queried why he was leaving and got a simple "just want a break from working" which I found odd.

Fast forward and now I also just want a break from work, permanently. This place has followed every bad practise and big no-no out there. Every bit of software is a built in house knockoff janky piece of crap that doesn't work and makes people's jobs 5000 times harder.

The UI looks worse than Windows 3.1, absolutely horrendous code formatting, worst database structure I've ever seen.

The mere mention of using a team communication tool results in being yelled at from the CEO whom communicates purely via email, who then gets annoyed when you don't reply because they sent the email to a client instead of you.

We get handed printed out "tickets" to work instead of the so called "amazing in house ticket system" built using PHP 5 and is literally crammed into an 800x600 IFrame. Yes a F$*#ing IFRAME!

It's not like we have an outdated TFS server that has work items we can use...

Why not push for changes you say. I have, many times, tried to suggest better tools. The only approval I've gotten is using PhpStorm. Everything else is shutdown immediately and you get the silent treatment.

The CEO hired me to do a job, then micromanages like crazy. I can't make UI changes, I can't make database changes, why? They insists they know best, but has admitted multiple times to not knowing SQL and literally uses a drag and drop database table builder.

Every page in the webapps we make are crammed into 800x600 iframes with more iframes inside iframes. And every time it's pointed out we need to do something, be it from internal staff or client suggestions, the CEO goes off about how the UI is industry leading and follows standards.. what in the actual f....

Literally holding on by a thread here. Why hire a CTO under the guise of being a senior developer but then reduce the work that can be done down to the level of a junior?

Sure the paycheck is really nice but no job is worth the stress, harassment and incompetent leadership from the CEO.

They've verbally abused people to the point they resign, best part is that was simply because the CEO made serious legal mistakes, was told about it by the employee then blamed it on others.

Excuse the profuse amount of profanity below.

Fuck this fucking fucked up motherfucker of a fucking director. Money does not make you a fucking decent person, and you come in here and tell me that you pay my fucking measly salary so I must be fucking grateful.

Starts off with a boardroom meeting this morning. Wireless connection on my laptop takes two minutes to connect, I get told that I am wasting company time and that the salary of everyone in the meeting is quite a lot ("with me being the highest"- cuntface director) so stop wasting time. Fuck you man, it's a fucking wireless connection. I am building your motherfucking company applications and doing web design and for what, so I can earn fuckall and be told that I am fucking wasting time. I am presenting your fucking site you wanted, so give me a fucking minute extra to start up the fucking wireless connection.

The fucking mails are taking long to send, great, let's come down and fucking scream at the dev who regrettably said he would try and assist IT (by calling the provider). I literally just got told that I am the following. 1) Fucking stupid 2) He is going to close the dept down because I apparently fuck up (yet again cuntface, your fucking mailserver is NOT MY FUCKING PROBLEM) 3) He is going to contact an external company to come and check my work. 4) I am fucking useless. 5) I telling him lies (yeah fuckface, I worked as a sys admin, I know what a motherfucking DNS server is and what it does. you don't - so don't fucking tell me that I am lying when I tell you there is a DNS fucking issue, because you don't know what the fuck you are talking about - to top that off motherfucker, I FUCKING BUILT YOUR FUCKING SERVER AND YOUR FUCKING NETWORK. I FUCKING KNOW HOW IT WORKS AND WHAT THE FUCK I AM TALKING ABOUT).

On top of that, I got pushed out of the way of my own PC, my code got some fucked up gibberish in it (because he was trying to minimise my editor and he typed some in it, and now I have to fucking roll-back. He told me I am wasting company time and he will take my shit away from me if I download something again. It is an open network. I downloaded JAVA and fucking updated Sublime. Jesus man. What the fucking fuck.

"why is your gmail open?!?!" because I was testing your emails from an external network. "DON'T FEED ME BULLSHIT" (even though the top mail states "test"). It's the whole fucking "my money determines my dick size" mentality.

That being said, I got told that I need to work overtime, without pay, to resolve IT's issue, even if I have to on the weekend.

That being said,my new Dell that I had just bought (my own) got thrown on the floor and he fucked out of my office. Stupid motherfucker. I fucking earn nothing but cannot leave. I will find another job, and when I do - you can go and fuck yourself and your fucking degrading opinions. I am not fucking stupid, so fuck you.Fuck your company and fuck you. Cunt.

This is dedicated to all Webdevs, especially those WordPress fanboys.

I was reflecting on some things since I do more frequent freelance jobs at the time. And I have to admit: people are fucking crazy.

I had some serious talk with customers and some serious talk for people I work as subsidiary.

The average customer thinks a nice webpage costs I'm 9-50 bucks. They got some shitty Webhosting for 1-5$/month including domain and think they are set.

They have unclear visions about what they actually want, it all boils down to "I like the design". I made a page for someone who just posted images, no text nothing and I told him a trillion times NEEDS some text, even a fucking picture description would be sufficient, else he'll never score anything at google.

Ofc it got denied, now he's bitching how nobody finds the site when they google his name. The other thing is that Wordpress became the solution for everything.

I'm a fucking certified magento developer and I hate magento with a passion. Magento is an overabstracted clusterfuck and believe me, I did the certification I had to learn more than average about the core. But damn, don't slap woocommerce on everything.

Narrowninded fucktards, the cheap out of the box solution isn't always the best.

Don't cry if you got hacked because you were too dumb to upgrade your wordpress. Don't tell me to do some "enhancements" on a server you probably share with 100 other uses. I can't fix your Webserver with your shitty ftp account.

I also hate WordPress with a burning passion. Cum guzzling cavetroll it is. It has it usages, but don't rely on a core So small every kind of extra functionality has to somehow tinkered on it and then expect it to work flawlessly and for 10$ price.

Of course you can buy a theme that, if it would have been special made for you cost 800$ or more, but it wasn't. It just looks like it from the outside. If you want customization you are at the mercy of the option it provides. I can't even tell how many times i spent whole evenings explaining how their shiny template works. Just to do some crazy shit with JavaScript like rearranging domelements because it didn't work as expected.

I still stay to my word. Nothing great has been nor will be created with a Wordpress core. Don't tell me how some great stuff has been achieved. Or wait, please do so. But before you do think about if that wouldn't been faster, cheaper, more reliable , etc... if done with a framework like symphony or laravel... or even zend or cake.

And that brings me back to the point:
Is cheap and "out of the box" really what you need and desire? As customer and as developer?

Every time I hear a developer say "works for me", I'm gonna hide a service of mine that they use behind the VPN.

Dev: "oy mate, this server is down"
Me (with VPN connection): "sorry mate, works for me"
Dev: "but here, check this out, it's down!!"
Me: "mate, check your network connection. You must have a shitty network connection."
Dev: -_-
Me: "Maybe shitty hardware? Driver issues on your network card? 🤭"

Because you know, we sysadmins can do that too 😉

Co-worker: "Should we keep this server up and running?"
Me: "Hmmm…"
C: "Do we have any other uses for it than the dedicated wiki?"
M: "Not really, and maybe it's time to move to the centralised platform Corporate™ introduced. Have we checked if anyone is using the server?"
C: "Good point, let me see…"
C: "… oh it's been down for last two weeks since the power outage."
M: "I think that answers the question. Let's leave it like this for a month more and if no one complains we can announce it dead"

What an awful day :(

The server where I host my 4 clients websites crashed.

Unable to reboot from the console.

I contact the support. 15 minutes later: "we'll look at this"

No news for 1 week despite my messages.

Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...

Answer: "Sorry, your server is down and cannot be repaired."

Fuck.

I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".

Fuck.

I subscribe to another server, at another provider.

I look at my backups... shit, the last one is 4 month ago!!

I restore the first website: OK

I restore the second website: OK

I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...

I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...

I lost ALL my 4th client data!!!

I'm sooooo piece of crap!

Hello Monday:
0.Arrive late due to traffic.(Apparently a car hit a cow crossing the road)
1. Try upgrading php5 to php7 and break stuff in the process and waste 2 hours fixing things.(Poor connection so ssh sessions hung occasionally)
2.PHP fixed,open Gmail and get over 100 emails from clients about the server being down(because of (0)).Ignore all.Find a snaglist of over 20 TODOs.
3.Open Android Studio, update to 2.3 and everything becomes broken.Each time i open it ,it crashes and i have to "Report to Google"
4.Spend the next 1 hour reinstalling AS.It finally works.
5.Open Project and the libraries are broken.Spend another hour upgrading build tools.
6.Leave SDK to update and decide to check my Google Cloud console.$50 bill pending.Shit.
7.Try XCode. Remember the project is still in Swift 2 and I have to upgrade it(Would take eternity).Immediately closes xcode.
8.Gives up on life and decides to log into Devrant.

Last september:
I get an email from a customer who want to end the contract. I inform him that the contract runs to the end of March and it will be terminated the last of march.

Customer agrees and I do not hear a word from him. Until now when I closed down all his services.

Angry email:
Why did you close the server and email? are you **** stupid!?

I reply:
21/9 2016, you wanted to terminate all services you had with us. That means that your domains, DNS-services, VPS, email and Office 365 was going to be closed today.

Why is this not uncommon?!

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

It's maddening how few people working with the internet don't know anything about the protocols that make it work. Web work, especially, I spend far too much time explaining how status codes, methods, content-types etc work, how they're used and basic fundamental shit about how to do the job of someone building internet applications and consumable services.

The following has played out at more than one company:

App: "Hey api, I need some data"
API: "200 (plain text response message, content-type application/json, 'internal server error')"
App: *blows the fuck up

*msg service team*

Me: "Getting a 200 with a plaintext response containing an internal server exception"
Team: "Yeah, what's the problem?"
Me: "...200 means success, the message suggests 500. Either way, it should be one of the error codes. We use the status code to determine how the application processes the request. What do the logs say?"
Team: "Log says that the user wasn't signed in. Can you not read the response message and make a decision?"
Me: "That status for that is 401. And no, that would require us to know every message you have verbatim, in this case, it doesn't even deserialize and causes an exception because it's not actually json."
Team: "Why 401?"
Me: "It's the code for unauthorized. It tells us to redirect the user to the sign in experience"
Team: "We can't authorize until the user signs in"
Me: *angermatopoeia* "Just, trust me. If a user isn't logged in, return 401, if they don't have permissions you send 403"
Team: *googles SO* "Internet says we can use 500"
Me: "That's server error, it says something blew up with an unhandled exception on your end. You've already established it was an auth issue in the logs."
Team: "But there's an error, why doesn't that work?"
Me: "It's generic. It's like me messaging you and saying, "your service is broken". It doesn't give us any insight into what went wrong or *how* we should attempt to troubleshoot the error or where it occurred. You already know what's wrong, so just tell me with the status code."
Team: "But it's ok, right, 500? It's an error?"
Me: "It puts all the troubleshooting responsibility on your consumer to investigate the error at every level. A precise error code could potentially prevent us from bothering you at all."
Team: "How so?"
Me: "Send 401, we know that it's a login issue, 403, something is wrong with the request, 404 we're hitting an endpoint that doesn't exist, 503 we know that the service can't be reached for some reason, 504 means the service exists, but timed out at the gateway or service. In the worst case we're able to triage who needs to be involved to solve the issue, make sense?"
Team: "Oh, sounds cool, so how do we do that?"
Me: "That's down to your technology, your team will need to implement it. Most frameworks handle it out of the box for many cases."
Team: "Ah, ok. We'll send a 500, that sound easiest"
Me: *..l.. -__- ..l..* "Ok, let's get into the other 5 problems with this situation..."

Moral of the story: If this is you: learn the protocol you're utilizing, provide metadata, and stop treating your customers like shit.

Once we were going to present a web service to governmental firm. All is going well so far and my boss asks me to host the web application the day before the presentation.

I hosted it and all was good with demo production tests, but I had a bad feeling.

While it was running on our server, I also ran it locally with a reverse proxy just in case.

* Meeting starts *

* Ice broken and down to business *

"And now our developer will run the demo for you..."

* Run the demo from my laptop to double check --> 500 Internal Server Error *

Holy shit!!!

* Opens reverse proxy link on my laptop. Present demo during meeting. Demo works like a charm. *

Firm representative: "Great! Looking forward to go live."

*Our team walks out*

GM: "Good job guys"

ME:

The network starts slowing down, transactions start to fail across the 450+ stores, the website starts to spit 500 errors what is going on?

Queue a frantic running around the office working out what was going wrong... Calls from all 3 data centres, nothing is going in or out of the network.

Notice the network admin come back to his desk, his eyebrows raise and he looks left and right before unplugging his laptop ethernet from one of the server access points

The network rushes back to life, everything is fine.

That particular network mapping tool is now banned for use on production.

plot twist
linuxxx is some of the most advanced and efficient AI ever developed, and actually uses devrant to scope out security-aware devs and take them down 1 production server at a time

One of the biggest e-commerce site's here in Sweden like to generate hundreds of thousands of sessionfiles every day.
That slows down their website alot, and the poor server is struggling hard.

They asked me if I could do something about it. So I mounted the catalogue with sessionfiles on a tmpfs volume. And suddenly the site is much faster. I guess storing sessionfiles directly into RAM solved it :P

And yes, they are aware that a restart of the server removes the files.

Today my manager asked me about my research into using RabbitMQ as a backup in case Azure Service Bus ever goes down.

Me: "Good. The way we designed the framework, all we have to do is drop the DLLs into the directory, update the config, and the services will start using RabbitMQ."
Mgr: "Excellent. Probably should be looking into using RabbitMQ as a permanent replacement for Azure"
Me: "What? The whole reason we moved to Azure was to eliminate the problems with having an on prem service bus. Since we've switched, there has been zero downtime."
Mgr: "That's what VP-Joe is afraid of. If Azure ever goes down, he won't know how to explain Azure to the president as to why we're not taking orders or can't ship packages."
Me: "That makes no sense. What did VP-Joe tell the president when a database goes down or a server mis-configuration?"
Mgr: "President understands internal outages, its just the whole 'cloud' thing he doesn't understand."
Me: "Um..then VP-Joe needs to explain it to him?"
Mgr: "The decision has already been made. Are you on board? Lets look at this move as a cost savings."
Me: "You mean the $10 a month? How much hardware will we need to support RabbitMQ?"
Mgr: "Yea, nobody probably thought of that."
Me: "I'm on board with whatever decision, but I'd like a little more than VP-Joe being afraid of the president."
Mgr: "I'm sure its not being afraid."
Me: "..."
Mgr: "OK, lets wait and see if VP-Joe forgets about this and moves on to something new."

More sysadmin focused but y’all get this stuff and I need a rant.

TLDR: Got the wrong internship.

Start working as a sysadmin/dev intern/man-of-many-hats at a small finance company (I’m still in school). Day 1: “Oh new IT guy? Just grab a PC from an empty cubicle and here’s a flash drive with Fedora, go ahead and manually install your operating system. Oh shit also your desktop has 2g of ram, a core2 duo, and we scavenged your hard drive for another dev so just go find one in the server room. And also your monitor is broken so just take one from another cubicle.”

Am shown our server room and see that someone is storing random personal shit in there (golf clubs propped against the server racks with heads mixed into the cabling, etc.). Ask why the golf clubs etc. are mixed in with the cabling and server racks and am given the silent treatment. Learn later that my boss is the owners son, and he is storing his personal stuff in our server room.

Do desktop support for end users. Another manager asks for her employees to receive copies of office 2010 (they’re running 2003 an 2007). Ask boss about licensing plans in place and upgrade schedules, he says he’ll get back to me. I explain to other manager we are working on a licensing scheme and I will keep her informed.

Next day other manager tells me (*the intern*) that she spoke with a rich business friend whose company uses fake/cracked license keys and we should do the same to keep costs down. I nod and smile. IT manager tells me we have no upgrade schedule or licensing agreement. I suggest purchasing an Office 365 subscription. Boss says $150 a year per employee is too expensive (Company pulls good money, has ~25 employees, owner is just cheap) I suggest freeware alternatives. Other manager refuses to use anything other than office 2010 as that is what she is familiar with. Boss refuses to spend any money on license keys. Learn other manager is owners wife and mother of my boss. Stalemate. No upgrades happen.

Company is running an active directory Windows Server 2003 instance that needs upgrading. I suggest 2012R2. Boss says “sure”. I ask how he will purchase the license key and he tells me he won’t.

I suggest running an Ubuntu server with LDAP functionality instead with the understanding that this will add IT employee hours for maintenance. Bosses eyes glaze over at the mention of Linux. The upgrade is put off.

Start cleaning out server room of the personal junk, labeling server racks and cables, and creating a network map. Boss asks what I’m doing. I show him the organized side of the server room and he says “okay but don’t do any more”.

... *sigh* ...

I hope you will forgive me for a third hand story, but I'm one of those evil developers, not a support per se. But I thought you'd enjoy this story anyway. So this happened to a colleague of a colleague:

$Hero - our hero. $Cop - A representative of our hard worked law enforcement agency.

So $Hero is happily speeding along in his car, running a few yellow lights a bit late, etc. Finally, the law catches up to him and pulls him over. Here's how the conversation went:

$Cop: Can I see your driving license, please?

$Hero (with smug grin): Certainly. Here it is, officer.

$Cop takes license back to motorcycle and speaks into radio.

$Hero: It's not going to help you any, though.

$Cop (with no reaction): What do you mean?

$Hero (with wider grin): The server you have to check it against is down.

$Cop (still no reaction): And why do you say that?

$Hero: Because I'm the guy they called to get on site and get it up again.

Our hero did not get a fine this time. Instead he got a police escort to his workplace.

Source: reddit r/talesfromtechsupport

When you wake up and first msg is

"Our server is down"

Our website once had it’s config file (“old” .cgi app) open and available if you knew the file name. It was ‘obfuscated’ with the file name “Name of the cgi executable”.txt. So browsing, browsing.cgi, config file was browsing.txt.
After discovering the sql server admin password in plain text and reporting it to the VP, he called a meeting.
VP: “I have a report that you are storing the server admin password in plain text.”
WebMgr: “No, that is not correct.”
Me: “Um, yes it is, or we wouldn’t be here.”
WebMgr: “It’s not a network server administrator, it’s SQL Server’s SA account. Completely secure since that login has no access to the network.”
<VP looks over at me>
VP: “Oh..I was not told *that* detail.”
Me: “Um, that doesn’t matter, we shouldn’t have any login password in plain text, anywhere. Besides, the SA account has full access to the entire database. Someone could drop tables, get customer data, even access credit card data.”
WebMgr: “You are blowing all this out of proportion. There is no way anyone could do that.”
Me: “Uh, two weeks ago I discovered the catalog page was sending raw SQL from javascript. All anyone had to do was inject a semicolon and add whatever they wanted.”
WebMgr: “Who would do that? They would have to know a lot about our systems in order to do any real damage.”
VP: “Yes, it would have to be someone in our department looking to do some damage.”
<both the VP and WebMgr look at me>
Me: “Open your browser and search on SQL Injection.”
<VP searches on SQL Injection..few seconds pass>
VP: “Oh my, this is disturbing. I did not know SQL injection was such a problem. I want all SQL removed from javascript and passwords removed from the text files.”
WebMgr: “Our team is already removing the SQL, but our apps need to read the SQL server login and password from a config file. I don’t know why this is such a big deal. The file is read-only and protected by IIS. You can’t even read it from a browser.”
VP: “Well, if it’s secured, I suppose it is OK.”
Me: “Open your browser and navigate to … browse.txt”
VP: “Oh my, there it is.”
WebMgr: “You can only see it because your laptop had administrative privileges. Anyone outside our network cannot access the file.”
VP: “OK, that makes sense. As long as IIS is securing the file …”
Me: “No..no..no.. I can’t believe this. The screen shot I sent yesterday was from my home laptop showing the file is publicly available.”
WebMgr: “But you are probably an admin on the laptop.”
<couple of awkward seconds of silence…then the light comes on>
VP: “OK, I’m stopping this meeting. I want all admin users and passwords removed from the site by the end of the day.”

Took a little longer than a day, but after reviewing what the web team changed:
- They did remove the SQL Server SA account, but replaced it with another account with full admin privileges.
- Replaced the “App Name”.txt with centrally located config file at C:\Inetpub\wwwroot\config.txt (hard-coded in the app)
When I brought this up again with my manager..
Mgr: “Yea, I know, it sucks. WebMgr showed the VP the config file was not accessible by the web site and it wasn’t using the SA password. He was satisfied by that. Web site is looking to beat projections again by 15%, so WebMgr told the other VPs that another disruption from a developer could jeopardize the quarterly numbers. I’d keep my head down for a while.”

Privacy & security violations piss me off. Not to the point that I'll write on devRant about it, but to the point that coworkers get afraid from the bloodthirsty look in my eyes.

I know all startups proclaim this, but the one I work at is kind of industry-disrupting. Think Uber vs taxi drivers... so we have real, malicious enemies.

Yet there's still this mindset of "it won't happen to us" when it comes to data leaks or corporate spying.

Me: "I noticed we are tracking our end users without their consent, and store not just the color of their balls, but also their favorite soup flavor and how often they've cheated on their partner, as plain text in the system for every employee to read"

Various C-randomletter-Os: "Oh wow indubitably most serious indeed! Let's put 2 scrumbag masters on the issue, we will tackle this in a most agile manner! We shall use AI blockchains in the elastic cloud to encrypt those ball-colors!"

NO WHAT I MEANT WAS WHY THE FUCK DO WE EVEN STORE THAT INFORMATION. IT DOES IN NO WAY RELATE TO OUR BUSINESS!

"No reason, just future requirements for our data scientists"

I'M GRABBING A HARDDRIVE SHREDDER, THE DB SERVER GOES FIRST AND YOUR PENIS RIGHT AFTER THAT!

(if it's unclear, ball color was an optimistic euphemism for what boiled down to an analytics value which might as well have been "nigger: yes/no")

Haven’t been on here for ages, but I felt like I needed to post this:

Warning:
This is long, and it might make you cry.

Backstory:
A couple of months back I worked for a completely clueless dude who had somehow landed a contract for a new website for a huge company. After a while he realised that he was incapable of completing the assignment. He then hired me as a subcontractor and I deleted literally everything he had done and started from scratch. He had over promised and under explained what needed to be done to me. It took many sleepless nights to get this finished with all the amendments and I had to double my pricing because he kept changing the brief.

Even after doubling my prices I still put in way too many hours of work. At one point I had enough and just ghosted the guy as I had done what he asked, and when he submitted it to them they wanted changes. He couldn’t make the changes, so I had to. He wouldn’t pay me extra though. I decided it wasn’t worth my time.

A couple of days ago I heard from him again. He had found another subcontractor to finish the changes. He still needed a few things though, so he promised me that I would get paid after fixing those things. I looked at the few things he had listed in our KANBAN and thought it was a few easy tasks.. until I opened the project..

I had my computer set up to sync with his server because he wanted everything done live and in production. So I naturally thought I would just “sync down” everything that the other subcontractor had done.

Here is where the magic started to happen.. I started the sync and went to grab a glass of water, and it was still running when I came back. I looked at the log and saw a bunch of “node_module” files syncing - around 900 folders. Funny thing is; neither the site nor server has anything to do with node..

I disregarded this and downloaded the files in a more manual fashion to a new folder. Interestingly I could see that my SCSS folders had not been touched since I stopped working on the project.. interesting, I thought to myself..

Turns out, the other subcontractor had taken my rendered and minimised CSS file, prettified it and worked from there. This meant that the around ~1500 lines of SCSS neatly organised in around 20 files was suddenly turned into a monster of a single CSS file of no less than 17300 lines.

I tried to explain to the guy that the other subcontractor had fucked up, but he said that I should be able to fix it since I was the one that made it initially. I haven’t replied. My life is too short for this.

TL;Dr be specific, it's actually helpful.

Client rings... "The internet is down"

Me "ok where are you exactly and how are you connecting"

"Ugh the WiFi! Just fix it"

"Ok but where are you?"

"At $companyname"

"Ok and which wi..."

"The wifi?!! Can you do anything right?"

Well... I'm allowed flexibility in terms of pleasantry...

"Ok, there are 3 buildings, 55 rooms, 2 SSID's, 17 access points, 3 routers a RADIUS server and 2 gateways... Be specific or I'll do nothing"

Simple reboot of an access point, but c'mon... It's not a secret where you are

Much-security.nl is down.

No, no hack or whatsoever. I just reinstalled the wrong server through my control panel.

The new blog version will hopefully be up tonight. 'you keep promising that' - I don't have a backup of the old version and I only just started using git so I can only upload the new version (or git pull). Next to that, except for the front-end, everything works now.

😐

This company!
Ugh.

Two days ago we had an hour and a half meeting on which projects to focus on, with the result being all seven are top priority. Because of course.

Last night I told my boss why an api he has me hitting always returns 401s; even gave him the line# responsible for the response (in his code). After an hour and sixteen minutes of him debugging, he finally admitted I might be right. zzz. This morning, he tells me it's on my end, and to ask someone else for their project's API code. The problem is that the server is not accepting the new application's key, since that key is not in the allowed list. That other project works just fine. Guess why? Their key has been whitelisted for months. But it's totally my code. Yeah. Bloody brilliant. 🔅

Anyway, today we're discussing "Winning with Accountability," a 100 page book that boils down to "do what you say you'll do, by when you said you'd do it, and take responsibility if you don't." But a huge part that the boss is stressing is: provide the exact date, time, and timezone of when things will be completed by. I mean That's fine for sales calls and reports and such trivial busywork. But dev projects? Not so much.

And that's been my past three days!
Friggin joy.

I've had many, but this is one of my favorite "OK, I'm getting fired for this" moments.

A new team in charge of source control and development standards came up with a 20 page work-instruction document for the new TFS source control structure.
The source control kingpin came from semi-large military contract company where taking a piss was probably outlined somewhere.

Maybe twice, I merged down from a release branch when I should have merged down from a dev branch, which "messed up" the flow of code that one team was working on.
Each time I was 'coached' and reminded on page 13, paragraph 5, sub-section C ... "When merging down from release, you must verify no other teams are working
on branches...blah blah blah..and if they have pending changes, use a shelfset and document the changes using Document A234-B..."

A fellow dev overheard the kingpin and the department manager in the breakroom saying if I messed up TFS one more time, I was gone.

Wasn't two days later I needed to merge up some new files to Main, and 'something' happened in TFS and a couple of files didn't get merged up. No errors, nothing.
Another team was waiting on me, so I simply added the files directly into Main. Unknown to me, the kingpin had a specific alert in TFS to notify him when someone added
files directly into Main, and I get a visit.

KP: "Did you add a couple of files directly into Main?"
Me:"Yes, I don't what happened, but the files never made it from my branch, to dev, to the review shelfset, and then to Main. I never got an error, but since
they were new files and adding a new feature, they never broke a build. Adding the files directly allowed the Web team to finish their project and deploy the
site this morning."
KP: "That is in direct violation of the standard. Didn't you read the documentation?"
Me: "Uh...well...um..yes, but that is an oddly specific case. I didn't think I hurt any.."
KP: "Ha ha...hurt? That's why we have standards. The document clearly states on page 18, paragraph 9, no files may ever be created in Main."
Me: "Really? I don't remember reading that."
<I navigate to the document, page 18, paragraph 9>
Me: "Um...no, it doesn't say that. The document only talks about merging process from a lower branch to Main."
KP: "Exactly. It is forbidden to create files directly in Main."
Me: "No, doesn't say that anywhere."
KP: "That is the spirit of the document. You violated the spirit of what we're trying to accomplish here."
Me: "You gotta be fracking kidding me."

KP grumbles something, goes back to his desk. Maybe a minute later he leaves the IS office, and the department manager leaves his office.

It was after 5:00PM, they never came back, so I headed home worried if I had a job in the morning.

I decided to come in a little early to snoop around, I knew where HR kept their terminated employee documents, and my badge wouldn't let me in the building.

Oh crap.

It was a shift change, so was able to walk in with the warehouse workers in another part of the building (many knew me, so nothing seemed that odd), and to my desk.

I tried to log into my computer...account locked. Oh crap..this was it. I'm done. I fill my computer backpack with as much personal items as I could, and started down the hallway when I meet one of our FS accountants.

L: "Hey, did your card let you in the building this morning? Mine didn't work. I had to walk around to the warehouse entrance and my computer account is locked. None of us can get into the system."

*whew!* is an understatement. Found out later the user account server crashed, which locked out everybody.

Never found out what kingpin and the dev manager left to talk about, but I at least still had a job.

The stupid stories of how I was able to break my schools network just to get better internet, as well as more ridiculous fun. XD

1st year:
It was my freshman year in college. The internet sucked really, really, really badly! Too many people were clearly using it. I had to find another way to remedy this. Upon some further research through Google I found out that one can in fact turn their computer into a router. Now what’s interesting about this network is that it only works with computers by downloading the necessary software that this network provides for you. Some weird software that actually looks through your computer and makes sure it’s ok to be added to the network. Unfortunately, routers can’t download and install that software, thus no internet… but a PC that can be changed into a router itself is a different story. I found that I can download the software check the PC and then turn on my Router feature. Viola, personal fast internet connected directly into the wall. No more sharing a single shitty router!

2nd year:
This was about the year when bitcoin mining was becoming a thing, and everyone was in on it. My shitty computer couldn’t possibly pull off mining for bitcoins. I needed something faster. How I found out that I could use my schools servers was merely an accident.
I had been installing the software on every possible PC I owned, but alas all my PC’s were just not fast enough. I decided to try it on the RDS server. It worked; the command window was pumping out coins! What I came to find out was that the RDS server had 36 cores. This thing was a beast! And it made sense that it could actually pull off mining for bitcoins. A couple nights later I signed in remotely to the RDS server. I created a macro that would continuously move my mouse around in the Remote desktop screen to keep my session alive at all times, and then I’d start my bitcoin mining operation. The following morning I wake up and my session was gone. How sad I thought. I quickly try to remote back in to see what I had collected. “Error, could not connect”. Weird… this usually never happens, maybe I did the remoting wrong. I went to my schools website to do some research on my remoting problem. It was down. In fact, everything was down… I come to find out that I had accidentally shut down the schools network because of my mining operation. I wasn’t found out, but I haven’t done any mining since then.

3rd year:
As an engineering student I found out that all engineering students get access to the school’s VPN. Cool, it is technically used to get around some wonky issues with remoting into the RDS servers. What I come to find out, after messing around with it frequently, is that I can actually use the VPN against the screwed up security on the network. Remember, how I told you that a program has to be downloaded and then one can be accepted into the network? Well, I was able to bypass all of that, simply by using the school’s VPN against itself… How dense does one have to be to not have patched that one?

4th year:
It was another programming day, and I needed access to my phones memory. Using some specially made apps I could easily connect to my phone from my computer and continue my work. But what I found out was that I could in fact travel around in the network. I discovered that I can, in fact, access my phone through the network from anywhere. What resulted was the discovery that the network scales the entirety of the school. I discovered that if I left my phone down in the engineering building and then went north to the biology building, I could still continue to access it. This seems like a very fatal flaw. My idea is to hook up a webcam to a robot and remotely controlling it from the RDS servers and having this little robot go to my classes for me.

What crazy shit have you done at your University?

I get a call: "Hey the site is down. Fix it!"
Worked on my workstation, not on my phone => DNS issue.

Local cache: "All OK"
ISP's DNS: "No record"
Google DNS: "Server error"
MXToolbox: "All OK"
CloudFlare DNS: "Domain? What domain?"

After a day of fucking around with configs and wanting to strangle the customer support guy, I just started pressing buttons, until suddenly, it worked. Turns out I'd accidentally enabled DNSSEC on a domain, that wasn't configured for it.

Lesson learned: There is no official DNS error code for "DNSSEC failed somewhere upstream". If you're lucky, you might get something useful out of the authoritative server, but apparently not on Mondays.

Wait what's that? You don't use version control on Production servers?
You want me to do what?
You want me to rename every file I have to replace with an underscore and the date after the extension so it looks like this?
SHIT.JAR_01262019
You've got to be fucking kidding me right!?
No?
Oh the production server is down again?
Is it because we're not using the right Jar file?
Well shit, I wonder why that's happening...

To improve our user's "experience" I suggested to my boss to add a status page showing...well, the current status of our services. Everybody was up for it, so I go off and implement a basic version + automated monitoring backend, get lots of positive feedback, all seems fine.

Then it starts:

Boss: "Can you get it all set up by this Saturday?"
Me: "Uh, today is Wednesday and I've never set up all the stuff needed on a proper server before"
Boss: "Well, you still have a few days. Please also contact your coworker to get it all hooked up in our launcher"
Me: "I'll try, can't make any promises though"

Contact my coworker and tell him what the plan is. I had already given him access to the repo and he is positive to get it all hooked up (I doubt he ever cloned my repo, let alone ran my code)

Spend all Friday getting my stuff set up on the production server, feeling pretty good thanks to the many tutorials.

Contact the boss Friday evening:
Me: "All up and running"
Boss: "Thanks, but we decided to go with a basic HTML page instead. We can just manually edit that, should be enough.
Me: "..."

In the end my stuff was never used, the server I set up was finally taken down a month ago. The gratitude you get when not hacking together some absolute shit that causes problems when you don't add <br/> tags at the correct places to prevent an ugly overflow, cause the coworker was too lazy to implement some form of line wrap in the launcher. I'm not saying my stuff is the best of the best, but at least it was professional looking to a certain extent.

Yesterday, the whole dev team went out to lunch and we ate a lot and we drank a lot until we all got drunk! Since we are paid for the number of hours we work, we all decided to go back to the office and work.

WE WERE DRUNK WORKING! Drunk coding ftw

Result: The server is down right now because someone fucked it up and I think i ruined my code yesterday because I wasn't really myself. The whole team was crazy as fuck. One of us just came back from Poland so we were drunk and high from all the polish chocolate we ate.

I hope they fix the server so I can check what kind of bullshit the drunken me did yesterday o.O

This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Hey everyone - apologies for the downtime earlier today. Our host is having a lot of issues and we're working to keep everything up through it.

On that note- there might be a little more down time tonight as they are trying to fix something and we might need a few server restarts. I will keep everyone updated and thanks for bearing with us!

Customer : c
Me : m

*Few weeks ago*
C: the server is slow, it sometimes takes 7 seconds before I see our data

(the project is 7+ years old and wasn't written by someone who is very good in SQL)

M: yeah I see that, our servers are busy with this one "process" (SQL query)
C: make it faster
M: well that's possible but it will take a few days (massive SQL spaghetti that I first have to untangle)
C: 😡 nvm then

*Yesterday*
C: server is down !
M: 🤔 *loads data from server and waits ~ 7 seconds*
M: Well what's the problem?
C: I need the data but it's so slow

WELL YOU MINDLESS IMBECILE... If something is slow it doesn't mean our god damn production server is down !
That just means that you have to give us a day or two so we can optimise the (ALSO BY YOUR REQUEST) rushed project... And save you YOUR money that YOU waste on the processing time on our server...

- Boss server is down
- Can you reboot it?
- Oh well... how can i explain you...

Quick recap of my last two weeks: 15 year old production server is basically dead, boss has taken over calls and claims credit for "resolving" outages (even though my coworker and I did the work, but ultimately the traffic died down enough to where it wasn't an issue anymore).

I go to a meeting to plan migration to a better server, boss bitches about not getting invited, I tell him I invited myself, and then he lectures about how that's not our job.

Different boss says we're migrating a schema for an application that should have been decommissioned 5+ years ago to use as a baseline. I explain what's going on, he says he understands, and proceeds to tell higher bosses it's perfect because there will be no user impact. OF COURSE THERE'S NO FRICKING IMPACT, YA DUNCE! there are no users!!!!

I merge two email threads together, since they discuss the same thing, but with different insight, and get yelled at, even though they requested it.

The two bosses I like are OOO for the next week, too, so I'm just sitting here hoping I don't say something that'll get me fired or sent to sensitivity training.

I'm just starting my on call rotation and don't know that I can do this. I cry when my phone rings, now, because I experience physical pain with how hard I cringe.

I got yelled at today by a guy because SOMEONE I DON'T KNOW assigned a ticket to him directly, rather than to the proper team (not his team). So I had to look into that, which at least had the benefit of preventing a catastrophic outage to our customers world wide, but no one will know because I don't brag at work; I'm too busy doing my job as well as most of my division/section/larger team, whatever the hell it's called. I saved us probably 25+ hours of continuous troubleshooting call from noticing something tiny that the people "smarter" than me missed.

**edit: sorry for typos; got my nails done yesterday but they feel like they're a mile long and I have to relearn how to type**

Dev gets hold of me, says my service is down in QA. Works if he hits it locally, works via Postman, but via the QA app server it gives a 401.

I’m like, look, if it works everywhere else, there’s something wrong on your side in QA.

He insists, no, I must help him, and begins CCing all the managers telling them this system has been down for days.

So I eventually climb into his system, check the credentials they’re using in the QA environment, and sure enough, the password is wrong.

I decided to setup a little server on my local network just to make use of a 2TB harddrive I use to store videos.

Told everyone in the house I planned to grow the library over time and that they could access it all in a browser using my system name. It's become quite a fun venture and my video library is shaping up nicely.

Using nginx on a Dell XPS 17 with Ubuntu 16.04 to host a server that just auto indexes a shared directory on my external 2TB harddrive. Kind of an embarrassing rig, but it's just a hobby activity and I do plan to upgrade shit later.

The real fun has been getting to understand a bit more about video files. They used to be magic to me, as complex as their file extension. Now I run a script on all of my torrents which checks the video and audio codecs, converting them if they aren't supported by Chrome's and Firefox's web players, and outputting mp4s using ffmpeg. I feel like I have this stuff down fairly well now. Becoming more and more automated.

Next step is to port forward so I can access it from anywhere, but we'll see about that later down the line.

Finally broke into the system and finally their server is DOWN 😎

> me on call
> had to much wine
> suddenly, phone starts making sound
> holy shit, I am on call and I am not supposed to drink
> Calls taxi because shit is looking bad at work. Everything look down according to the sms I get, I had to pay the taxi from my own pocket.
> 40 min later, arrive at work. Walk into the serverroom.
> Go to the the server that is monitoring everything.
> Check networkcable, it is loose. Push it a little.
> Goddamnit, that was it.
> Realize I never doublechecked if stuff really was down from home....

Finally got myself a Lytro Illum!
I,v been wanting to buy one since it came out but the company who made it closed down in 2015..
Thoose fuckers just thrown everything in the trash and set it on fire, software, firmware, mobile app etc.. no open source, no archives, your expensive camera is now a paper weight! You’r welcome!
So i got myself a new hobby, started reverse-engineering the fuck out of it, luckily it’s based on android (api17), i have adb and it’s running a hidden DHCP server too so it’s coming along nicely :D
I’m planning to make a camera control mobile app for it and maybe some faster image processing, wifi sharing etc..
I love beeing in home office :D

About a week ago I bought a server rack case from some guy online for 40 bucks. When it arrived it was in the worst state I had ever seen a case. Even if I tried I wouldn't have abused a case like that. Rust... everywhere...

I stripped it down and removed all the rust I could. Then I painted it and this is the end result. Pretty cool if I say so myself :)

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

I'm so grateful DevOps is now a thing. I remember getting a phone call from a client at 2am on a Friday because their site was down and having to ssh in from a Nokia with the world's tiniest keyboard to reboot the server.

Of course that particular server only exposed port 22 on it's local network, so I had to first ssh into another server which did have its ssh port open to external connections.

Trying to remember two sets of credentials and type them in on a tiny keyboard, while so drunk you were seeing double, standing outside in the rain as it was the only place you got signal. Yeah…I'm so grateful DevOps is now a thing

Stack Overflow. Everyone uses it but everyone seems to hate the community. I very often read about someone getting down voted and they all say the same thing - "I have no idea why".

I have spent a lot of time moderating SO posts, which gave me a lot of reputation and medals. I find it fun to help people and it feels good to give back to the community.

I have asked a bunch of questions and I've never gotten a single down vote, which leads me to believe everyone of you that is constantly getting down voted are doing something wrong. Because the posts I see getting down voted are fucking stupid questions that either lack information or contain too much information.

Example 1:
Server java error

Why is my server not working? I am using Tomcat, port 8080 and I'm getting IOException.

Example 2:
Webpack configuration not working

My webpack is not a working, why?
[entire webpack config]

End examples.

What the fuck are you expecting asking questions like these?? No one gets paid for answering your questions, so the least you can do is write a CLEAR AND UNDERSTANDABLE question. I'm not gonna tell you how to do it because there's A LOT of information on how to do it.

People devote hours and hours to helping others on SO, and of course they get fed up with the stupid and lazy questions. That community is not about being nice, it's not about making people feel welcomed, it's about QUALITY OF CONTENT. No one is crying when they find a superb question + answer, right? That's the result of a community not accepting low quality content.

So please, the next time you get a down vote on SO - do not come here whining about it but instead take a look at what you have posted there and ask yourself if it could have held a higher quality.

Thanks!

1. Fucking MySQL database clusters.

There's nothing fun about MySQL clusters. Sometimes they start producing deadlock errors for no apparent reason... well, there's probably a reason, but it's never a transparent easy to find reason.

What was even less fun is that those errors took down a Sentry server. When your error log server goes down through ddos from your database messages, it's time to rethink your setup.

2. Wiring up a large factory with $2 arduino clones, each with a $2 esp8266 wifi chip, with various sensors for measuring flow of chemical solutions (I wanted cheap real time monitoring as an early warning system next to periodic sampling).

The scaling issue was getting over 500 streaming wifi signals to work in a 55c moist slightly corrosive atmosphere with concrete and steel everywhere, and getting it all into a single InfluxDB instance for analysis.

Ooh this is good.

At my first job, i was hired as a c++ developer. The task seemed easy enough, it was a research and the previous developer died, leaving behind a lot of documentation and some legacy fortran code. Now you might not know, but fortran can be really easily converted to c, and then refactored to c++.

Fine, time to read the docs. The research was on pollen levels, cant really tell more. Mostly advanced maths. I dug through 500+ pages of algebra just to realize, theres no way this would ever work. Okay dont panic, im a data analyst, i can handle this.

Lets take a look at the fortran code, maybe that makes more sense. Turns out it had nothing to do with the task. It looped through some external data i couldnt find anywhere and thats it. Yay.

So i exported everything we had to a csv file, wrote a java program to apply linefit with linear regression and filter out the bad records. After that i spent 2 days in a hot server room, hoping that the old intel xeon wouldnt break down from sending java outputs directly to haskell, but it held on its own.

Got a call while on vacations, the main server is down.
*holy shit* I thought.
As I opened my MacBook, the phone kept ringing and slack was going crazy.

“What should I do? Where is the problem?”, the voices in my head said..

I opened the terminal and tried to get a ssh connection.
Sweat was dropping down from my forehead.

“Connected” the terminal said.

“Fuck yeah, the server is up, only the app is not responding”, I thought and opened the log files.

Suddenly, “STOP” I shout at the log files that were appended way too fast.

Then I saw it.
TimeOutExceptions..

I added an index to the modification date column,“ kill -9“ed the process, started again and went back to vacation mode 🙂

And of course I was the office hero for a while💪

For the smart asses, I’m aware that it’s a bad idea to -9 the app process in prod, but it was so overloaded that i was not able to kill it any other way. And we needed that server up again.

Very long story ahead!

Yesterday in the evening a friend of mine (calling him F from now on) became the target of something new to me...
Apparently one can fake his phone number through some fishy ways and call people with that number. Someone (we think we might know who it was, the why is at the end) did this yesterday to F.
Here's the whole story:
We were just talking together on a TeamSpeak Server (a program to talk to others on the internet) when suddenly another friend said: "F, why did you just call me three times in a row?" That was the first thing that was a bit suspicious. After that, F got calls from random numbers (even Afghanistan, we are German), and they said something like "Have fun with the police coming to your house". Then there was silence. 10 minutes later his phone rang and there were a ton of pizza delivery services in his town that apparently got pizza orders from him. Then there was silence, again. Suddenly someone with a hidden number called him, a woman's voice said they were the police and if F doesn't stop calling the police there will be consequences. F then told her what was going on but I think she didn't really care. She then wanted to know where F lives, but I told him not to say that, because if it is the police they can find it out by themself and if it's not, they don't need to know that.
Now, a short break: There is some fake information going around about where F lives. I can't remember when we found out but the attacker thought he would actually live there. No idea what happened at that location...
Now back to the story:
Time went by, nothing really happened. Suddenly F shouted: "There are blue lights outside! The police is here!" He muted his microphone and (the following is what he told us what happened) went down to the door (remember, he is 16) and there were two police men. They were asking about why he called the police. F explained what we knew until then, about number spoofing and stuff... They sent a more technical person to him, he understood what F was trying to explain. The police men drove away and he came back to tell us what happened. (Now we get back to what I heared myself.) The mom came in, screamed something that I couldn't understand, and F went offline. We searched who the attacker could have been. And we are pretty sure we found him. That guy connected to our Minecraft server (that's where I know F from) with his real IP, and his main account, which made it easy to search. He also got a static IP which means it doesn't change. We also got some information that in the recent days this guy was talking about VoIP spoofing and such stuff. Another friend of mine, a bit older, found some proofs and I think he will go to the police.

That's it. Thanks for reading.

So I used to do some freelancing in web development last year, nothing too fancy just some simple PHP websites. Comes the worst meeting in my life. So I am from India and we have a lot of long lasting business here being passed on over generations. TL;DR the guy was the owner of a very old business which was actually very huge and the guy was educated too, so I assumed that he'll be sensible as compared to other people.

The meeting was in an expensive cafe and he paid for it, he even told me upfront that meeting is on him. Great, right? So we sit down, order some coffee and then start discussing what he needed.
The guy needed an ecommerce website built with backend and logistics system integrated. We discussed possible designs for the website and stuff too and so far the deal looked promising to both of us.
I explained him the cost estimate and told him that I would email him the final quote from myself once we discussed server cost and shit.
So now comes the bargaining part where he asked me to give him server and domain for free.
At this point, I suspected that he didn't know that servers and domains are not something that you make. You have to purchase and renew them periodically.
So I told that guy that he didn't understand the cost estimation and explained to him that X is the cost of making this fucking thing and Y is its monthly maintenance cost, if he wanted annually could be done too. And this Y did not include server and domain costing.
Now came the fucking tide, the guy straight up turned to his shit and told me I am lying and trying to con him. So I gently asked him if he had ever gotten any website made. To which, he said No, but he knows how the costing works.
I was like "Bitch?". So I calmly tried to explain that that's not how websites are done, delivered and maintained.
He didn't seemed to be understanding and kept on fucking repeating that he knows his shit and blah blah.
At this point, I was like "Okay. Fuck this dude then. I can find another project. " and then I told him that he'll need to find someone according to his needs.
Interestingly enough, the guy called someone and then walked out of the cafe while talking on phone. I waited for 5 minutes and he didn't come back so I decided I would pay for my coffee and leave. Turns out the guy had paid his bill before my arrival and ditched me with the excuse of the call.

But oh well, I think working with such an idiot would have been much worse than paying for that coffee.

So ehm, tl;dr: KEEP DAILY BACKUPS. EVEN IF SOMEONE SAYS NOT TO.

7:48
Manager: Hey Tom, is the server down?
Me: Nah, should be ok, I just did some maintenance this sunday.
Manager: But I can't get [some work data from SQL server]
*Nervous giggle*

9:14: Some random off-site cunt they hired didn't read the notes that said "DO NOT REMOVE DATABASE [xyz]"
9:20-ish: Web don't even have the DB. And you said that we'll figure out what to do with backups later
*Suddenly manager starts to panic*
11:47: Found backup of the entire server on and old server that we had for spare parts, still running tho.
12:something: Everything back up and working.

Really glad I kept the old server running and doing daily backups. Saved our ass for the second time. And finally, new off-site backup is planned this week.

One Thursday noon,
operation manager: (looking at mobile)what the.....something is wrong i am getting bunch of emails about orders getting confirmed.
Colleague dev: (checks the main email where it gets all email sent/received) holy shit all of our clients getting confirmation email for orders which were already cancelled/incomplete.
Me: imediately contacting bluehost support, asking them to down the server so just that we can stopp it, 600+ emails were already sent and people keep getting it.
*calls head of IT* telling the situation because he's not in the office atm.

CEO: wtf is happening with my business, is it a hacker?

*so we have a intrusion somebody messed the site with a script or something*

All of us(dev) sits on the code finding the vulnerabilities , trying to track the issue that how somebody was able to do that.

*After an hour*

So we have gone through almost easch function written in the code which could possibly cause that but unable to find anything which could break it.

Head asking op when did you started getting it actually?
Op: right after 12 pm.

*an other hour passes*

Head: (checking the logs) so right after the last commit, site got updated too?. And....and.....wtf what da hell who wrote this shit in last commit?
* this fuckin query is missing damn where clause* 🤬

Me: me 😰

*long pause, everyone looking at me and i couldn't look at anyone*
The shame and me that how can i do that.

Head: so its you not any intrudor 😡
Further investigating, what the holy mother of #_/&;=568 why cronjob doesn't check how old the order is. Why why why.

(So basically this happened, because of that query all cancelled/incomplete orders got updated damage done already, helping it the cronjob running on all of them sending clients email and with that function some other values got updated too, inshort the whole db is fucked up.)
and now they know who did it as well.

*Head after some time cooling down, asked me the solution for the mess i create*

Me: i took backup just couple of days before i can restore that with a script and can do manual stuff for the recent 2 days. ( operation manager was already calling people and apologising from our side )

Head: okay do it now.

Me: *in panic* wrote a script to restore the records ( checking what i wrote 100000000 times now ), ran...tested...all working...restored the data.
after that wrote an apology email, because of me staff had to work alot and it becomes so hectic just because of me.

* at the end of the day CEO, head, staff accepted apology and asked me to be careful next time, so it actually teached me a lesson and i always always try to be more careful now especially with quries. People are really good here so that's how it goes* 🙂

The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.

TL;DR their people isn’t competent and I ended up fixing the issue.

Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.

So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.

IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.

I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.

Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.

I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.

5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.

They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.

I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.

I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.

Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.

While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-

Website up and running 1.5 later, mission accomplished.

Had 4-5 hours of sleep last night because I had to wake up early this morning for an internal demo of the feature I worked on. It was scheduled early morning because the boss was busy.

I come to the office looking like a zombie, practically sleep walking my way to the office, and guess what... The app server is down! Yay.

I had to reschedule it to right in the middle of lunch time because hey... The boss is busy. Now everyone in the team hates me for ruining their lunch.

Now I'm sitting here, staring at my code, remembering the sweet luxury of sleep.

Me to customer, several years ago:
Your server is over 10 years old, and the support for it will soon be gone together with the possibility to take backups. You should upgrade your website ASAP.

Customer said:
ok, we will look into it.

This day:
Customer:
Hi, our site is down and we need backups from yesterday. Can you please fix?

Me:
Sorry, no. *send copy of the email several years ago*

What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!

I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.

Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.

So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.

Lunch time already. But the fun isn't over yet!

I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.

After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:

"Please don't."
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."

So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".

While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.

Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.

Worst meeting:
Boss: *calls everyone* URGENT MEETING. If you're on your lunch break, then stop and join the video call. Is it a weird time in your timezone? Wake up. But COME

Omg what happened? Is the server down? Are we getting a huge client? What is so urgent that it needs all of our attention right now?

Boss: I'm afraid that I won't be able to stay long in this meeting.

... then WHY the f did you call this meeting?

Boss: that's why we'll have a meeting next at *such hour*. In the meantime, talk to your respective departments about what you've been doing and what you need from each other. See you!

No comment

It was a normal school day. I was at the computer and I needed to print some stuff out. Now this computer is special, it's hooked up onto a different network for students that signed up to use them. How you get to use these computers is by signing up using their forms online.

Unfortunately, for me on that day I needed to print something out and the computer I was working on was not letting me sign in. I called IT real quick and they said I needed to renew my membership. They send me the form, and I quickly fill it out. I hit the submit button and I'm greeted by a single line error written in php.

Someone had forgotten to turn off the debug mode to the server.
Upon examination of the error message, it was a syntax error at line 29 in directory such and such. This directory, i thought to myself, I know where this is. I quickly started my ftp client and was able to find the actual file in the directory that the error mentioned. What I didn't know, was that I'd find a mountain of passwords inside their php files, because they were automating all of the authentications.

Curious as I was, I followed the link database that was in the php file. UfFortunately, someone in IT hadn't thought far enough to make the actual link unseeable. I was greeted by the full database. There was nothing of real value from what I could see. Mostly forms that had been filled out by students.

Not only this, but I was displeased with the bad passwords. These passwords were maybe of 5 characters long, super simple words and a couple number tacked onto the end.

That day, I sent in a ticket to IT and told them about the issue. They quickly remedied it by turning off debug mode on the servers. However, they never did shut down access to the database and the php files...

Best part about the covid19 manufactured crisis?
Liquor stores deliver. Worst part about liquor stores delivering? Needing to use their shoddy websites.

I've been using a particular store (Total Wines) since they're cheaper than the rest and have better selection; it's quite literally a large warehouse made to look like a store.

Their website tries really hard to look professional, too, but it's just not. It took me two days to order, and not just from lack of time -- though from working 14 hour days, that's a factor.

Signing up was difficult. Your username is an email address, but you can't use comments because the server 500s, making the ajax call produce a wonderfully ambiguous error message. It also fades the page out like it's waiting on something, but that fade is on top of the error modal too. Similar error with the password field, though I don't remember how I triggered it.

Signing up also requires agreeing to subscribe to their newsletter. it's technically an opt-in, but not opting-in doesn't allow you to proceed. Same with opting-in to receiving a text notification when your order is ready for pickup -- you also opt-in to reciving SMS spam.

Another issue: After signing up, you start to navigate through the paginated product list. Every page change scrolls you to the exact middle of the next page. Not deliberatly; the UI loads first, and the browser gets as close as it can to your previous position -- which was below that as the pagination is at the bottom -- and then the products populate after. But regardless of why, there is no worse place to start because now you must scroll in both directions to view the products. If it stayed at the very bottom, it would at least mean you only need to scroll upwards to look at everything on the page. Minor, but increasingly irritating.

Also, they have like 198 pages of spirits alone because each size is unique entry. A 50ml, 350ml, 500ml, 750ml, 1000ml, and 1750ml bottle of e.g. Tito's vodka isn't one product, it's six. and they're sorted seemingly randomly. I think it's by available stock, looking back.

If you fancy a product, you can click on it for a detail page. Said detail page lists the various sizes in a dropdown, but they're not sorted correctly either, and changing sizes triggers a page reload, which leads to another problem:

if you navigate to more than a few pages within a 10 or so second window, the site accuses you of using browser automation. No captcha here, just a "click me for five seconds" button. However, it (usually) also triggers the check on every other tab you have open after its next nagivation.

That product page also randomly doesn't work. I haven't narrowed it down, but it will randomly decide to start failing, and won't stop failing for hours. It renders the page just fine, then immediately replaces it with a blank page. When it's failing, the only way to interact with the page is a perfectly-timed [esc], which can (and usually does) break all other page functionality, too. Absolutely great when you need to re-add everything from a stale copy of your signed-out cart living in another tab. More on that later. And don't forget to slow down to bypass the "browser automation" check, too!

Oh, and if you're using container tabs, make sure to open new tabs in the SAME container, as any request from the same IP without the login cookie will usually trigger that "browser automation" response, too.

The site also randomly signs you out, but allows you to continue amassing your cart. You'd think this is a good thing until you choose to sign in again... which empties your cart. It's like they don't want to make a sale at all.

The site also randomly forgets your name, replacing it with "null." My screen currently says "Hello, null". Hello, cruft!

It took me two days to order.
Mostly from lack of time, as i've been pulling 14 hour shifts lately trying to get everything done. but the sheer number of bugs certainly wasted most of what little time i had left. Now I definitely need a drink.

But maybe putting up with all of this is worthwhile because of their loyalty program? Apparently if you spend $500, you can take $5 off your next purchase! Yay! 1%! And your points expire! There are three levels; maybe it gets better. Level zero is for everyone; $0 requirement. There are also levels at $500 and $2500. That last one is seriously 5x more than the first paid level. and what does it earn you? A 'free' magazine subscription, 'free' classes (they're usually like $20-$50 iirc), and a 'free' grab bag (a $2.99 value!) twice per month. All for spending $2500. What a steal. It reminds me of Candy Crush's 3-star system where the first two stars are trivial, and the third is usually a difficult stretch goal. But here it's just thinly-veiled manipulation with no benefit.

I can tell they're employing some "smarketing" people with big ideas (read: stolen mistakes), but it's just such a fail.

The whole thing is a fail.

1 - Please hack his/her facebook account for me.

2 - (at home) I used to block wifi access by mac filtering and if there's legit server down and wifi isn't working, everyone blames me.

(I am freelancer and mostly work from home)

3 - almost all of my relatives think I don't work.

4 - I am first choice for everyone's phone, PC and hardware repair.

This one is classic

GET A REAL JOB, you need to go out in the field for work.

That's actually something that happened fairly recently.. just that I didn't have the energy left at the time to write it down. That, or I got my ass too drunk to properly write anything.. not sure actually.

So on paper I'm unemployed, but I do spend some time still on pretty much voluntary work for HackingVision, along with a handful of other people.

At the time, we were just doing the usual chit-chat in the admin channel, me still sick in my bed (actually that means that I wasn't drunk but really tired for once.. amazing!) and catching up to what happened, but unable to do any useful work in this sick state. So, tablet, typing on glass, right. I didn't have any keyboard attached at the time.

One of the staff members (a wanketeer from India) apparently had an assignment in a few hours for which he needed to write a server application in Java. Now, performance issues aside, I figured.. well I've got quite a bit of experience with servers, as well as some with client-server protocols. So I got thinking.. mail servers, way too overengineered. Web servers.. well that could work, I've done some basic netcat webservers that just sent an HTTP 200 OK and the file, those worked fine.. although super basic of course. And then there's IRC, which I've actually talked to an InspIRCd server through telnet before (which by the way is pretty much the only thing that telnet is still useful for, something that was never its purpose, lol) and realized that that protocol is actually quite easy to develop around. That's why I like it so much over modern chat protocols like XMPP, MQTT and whatnot. So I recommended that he'd write a little IRC server in Java. Or even just a chatbot like I attempted to at the time, considering that that's - with a stretch of course - a sort-of server too.

His fucking response however, so goddamn fucking infuriating. "If the protocol is so easy, then please write me down how to implement it in Java."

Essentially do his fucking work for him. I don't know Java, but as a fucking HackingVision admin, YOU SHOULD FUCKING KNOW THAT HACKERS CAN'T STAND LAZY CUNTS THAT CAN'T EVEN BE ASSED TO GOOGLE SHIT!!! If I wanted to deal with cunts like that, I'd have opened the page inbox with all its Fb h4xx0ring questions, not the fucking admin chat!

And type it on a goddamn fucking piece of glass, while fucking sick?! Get your ass fucked by a bobs and vegana horny fuck from the untouchable caste, because that's where you fucking belong for expecting THAT from me, you fucking bhenchod.

But at least I didn't get my ass enraged like that to say that to him in the admin chat. Although that probably wouldn't have been a bad thing, to get his feet right back on the ground again.

Finally got a new job, but it's already a horror story not even 2 hours in (making this while on break)

Everyone here is an Intern, IT? Interns, Designers? Interns, HR? Interns.

The Person who I should've worked with got fired yesterday, and now I have to work all of his shit up from 0, Documentation? Fragmental, a few things here and there, but nothing really.

IT security also doesn't exist in the slightest, there is an Excel sheet called "Master_Passwords" and every single password is in Plaintext, written out for everyone to see. (at least they used "strong" passwords)

And the place also looks run down, theres PC's, Laptops, Mics, Cables etc. lying literally everywhere no-one knows what works and what doesn't (since everyone is an intern)

Not to mention the "Server Room" is an absolute mess itself, cables hanging from literally anywhere, powerstrips are ontop of servers, each rack has like 2 or 3 2U Servers, (in a 40u Rack) and there are 10 of them!

Oh man. Mine are the REASON why people dislike PHP.

Biggest Concern: Intranet application for 3 staff members that allows them to set the admin data for an application that our userbase utilizes. Everything was fucking horrible, 300+ php files of spaghetti that did not escape user input, did not handle proper redirects, bad algo big O shit and then some. My pain point? I was testing some functionality when upon clicking 3 random check boxes you would get an error message that reads something like this "hi <SENSITIVE USERNAME DATA> you are attempting to use <SERVER IP ADDRESS> using <PASSWORD> but something went wrong! Call <OLD DEVELOPER's PHONE NUMBER> to provide him this <ERROR CODE>"

I panicked, closed that shit and rewrote it in an afternoon, that fucking retard had a tendency to use over 400 files of php for the simplest of fucking things.

Another one, that still baffles me and the other dev (an employee that has been there since the dawn of time) we have this massive application that we just can't rewrite due to time constraints. there is one file with (shit you not) a php include function that when you reach the file it is including it is just......a php closing tag. Removing it breaks down the application. This one is over 6000 files (I know) and we cannot understand what in the love of Lerdorf and baby Torvalds is happening.

From a previous job we had this massive in-house Javascript "framework" for ajax shit that for whatever reason unknown to me had a bunch of function and object names prefixed with "hotDog<rest of the function name>", this was used by two applications. One still in classic ASP and the other in php version 4.something

Legacy apps written in Apache Velocity, which in itself is not that bad, but I, even as a PHP developer, do not EVER mix views with logic. I like my shit separated AF thank you very much.

A large mobile application that interfaced with fucking everything via webviews. Shit was absolutley fucking disgusting, and I felt we were cheating our users.

A rails app with 1000 controller methods.

An express app with 1000 router methods with callbacks instead of async await even though async await was already a thing.

ultraFuckingLarge Delphi project with really no consideration for best practices. I, to this day enjoy Object Pascal, but the way in which people do delphi can scare me.

ASP.NET Application in wich there seemed to be a large portion of bolted in self made ioc framework from the lead dev, absolute shitfest, homie refused to use an actual ioc framework for it, they did pay the price after I left.

My own projects when I have to maintain them.

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

I know I shouldn't, but I had to laugh: "Does anyone know if the 'localhost' server is running? And could someone turn it on if it's down? I tried to connect but couldn't get through..."

I'm so close to giving up. Yesterday, I travelled 4 hours in one direction for a job interview for a graduate position as a web developer. As I arrived at the interview, I was welcomed by a senior dev and one of the HR people.

I sit down and they start explaining how everything will commence(standard procedure stuff) and afterwards hand me the technical test. At this time I am super calm cause I did my homework, checked out their products, their websites and knew right away what I was going to work on. As I turn the page, I see at the top with huge fucking capital letters "JAVA OOP test".

I take a minute and look back at them, like wtf is happening. Turns out that they are looking for a java dev. They picked me for the role because I had literally 1 fucking sentence in my CV and where I have said that I studied java in one semester of uni. FYI my entire portfolio, cv and cover letter are focused on JS, html, css both for client and server side.

As the fucking HR guy stood there and asked me "is there something wrong", I felt broken inside. For the first time in my fucking life I felt like I was done and couldn't continue anymore. I felt like this is some bitch-slap from karma about something but I still can't figure out what. I just walked out of there being unable to realize what happened.

I just feel like I should end my developer career before it has even started, just go do business analysis or something. Why the fuck would someone put a job description entirely talking about Angular, Less/SASS, bootstrap and jQuery and then say that is a Java dev OOP role. Who the fuck allows those people to take good salaries yet still deliver the up most shittiest quality service.

Before the interview, I checked out their websites which are simply horrendous with the comparability of a fucking baked potato. Idk really what to do, I don't mean to sound as a whiny little b.... but as I walked out of their office, I felt broken inside. Sorry for the long rant.

TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

The server is down!

@netikras since when does proprietary mean bad?

Lemme tell you 3 stories.

CISCO AnyConnect:
- come in to the office
- use internal resources (company newsletter, jira, etc.)
- connect to client's VPN using Cisco AnyConnect
- lose access to my company resources, because AnyConnect overwrites routing table (rather normal for VPN clients)
- issue a route command updating routing table so you could reach confluence page in the intranet
- route command executes successfully, `route -n` shows nothing has changed
- google this whole WTF case
- Cisco AnyConnect constantly overwrites OS routing table to ENFORCE you to use VPN settings and nothing else.

Sooo basically if you want to check your company's email, you have to disconnect from client's VPN, check email and reconnect again. Neat!

Can be easily resolved by using opensource VPN client -- openconnect

CISCO AnyConnect:
- get a server in your company
- connect it to client's VPN and keep the VPN running for data sync. VPN has to be UP at all times
- network glitch [uh-oh]
- VPN is no longer working, AnyConnect still believes everything is peachy. No reconnect attempts.
- service is unable to sync data w/ client's systems. Data gets outdated and eventually corrupted

OpenConnect (OSS alternative to AnyConnect) detects all network glitches, reports them to the log and attempts reconnect immediatelly. Subsequent reconnect attempts getting triggered with longer delays to not to spam network.

SYMANTEC VIP (alleged 2FA?):
- client's portal requires Sym VIP otp code to log in
- open up a browser in your laptop
- navigate to the portal
- enter your credentials
- click on a Sym VIP icon in the systray
- write down the shown otp number
- log in

umm... in what fucking way is that a secure 2FA? Everything is IN the same fucking device, a single click away.

Can be easily solved by opensource alternatives to Sym VIP app: they make HTTP calls to Symantec to register a new token and return you the whole totp url. You can convert that url to a qr code and scan it w/ your phone (e.g. Google's Authenticator). Now you have a true 2FA.

Proprietary is not always bad. There are good propr sw too. But the ones that are core to your BAU and are doing shit -- well these ARE bad. and w/o an oppurtunity to workaround/fix it yourself.

“sEniOr tEcHniCiaN”: “I don’t know what Blazor is. I write my projects in ASP.NET. You should just use ASP.NET”

Me: …”Blazor *is* ASP. This project is running on ASP.NET 6.”

“seNioR tEchNiCiaN”: “As previously stated, I don’t use Blazor. I don’t care what version it is.”

Yes, this is a real exchange from my ongoing problems with this idiot.
His attitude is what ticks me off the most.
He doesn’t know what CORS is.
He doesn’t understand that “ASP.NET” covers Blazor, Razor Pages, the old MVC stuff, web APIs, and more.
He doesn’t understand the difference between a web request being initiated from the browser via Fetch and a web request being initiated from the server. (“My ASP site is shown in the browser, so requests to the third party API aren’t originating from the server.”)

And yet has the arrogance to repeatedly talk down to me while I try to explain basic concepts to him in the least condescending way possible.

After going around and around in circles with him, he finally admitted to me that “he doesn’t actually know what the CORS configuration looks like or how to modify it, to be honest.”

I just wanna go home.

Gather around folks, I'll paint you a nice picture based on a true story, back from my sysadmin days. Listen up.
It's about HP and their Solaris 5.4/6 support.

- Yet another Prod Solaris dinosaur crashed
- Connected to console, found a dead system disk; for some reason it was not booting on the remaining redundant disk...
- Logged an HP vendor case. Sev1. SLA for response is 30 minutes, SLA for a fix is <24 hours
- It took them 2 days to respond to our Prod server outage due to failed system disks (responses "we are looking into this" do not count)
- it took another day for them to find an engineer who could attend the server in the DC
- The field engineer came to the DC 4 hours before the agreed time, so he had to wait (DC was 4-5 hours of driving away from HP centre)
- Turns out, he came to the wrong datacentre and was not let in even when the time came

- We had to reschedule for two days later. Prod is still down
- The engg came to the DC on time. He confirmed he had the FRU on him. Looks promising
- He entered the Hall
- He replaced the disk on the Solaris server
- It was the wrong disk he replaced. So now the server is beyond rebuild. It has to be built anew... but only after he comes back and replaces the actually faulty disk.
- He replaced that disk on the wrong Solaris server

My first job was actually nontechnical - I was 18 years old and sold premium office furniture for a small store in Munich.
I did code in my free time though (PHP/JS mostly, had a litte browsergame back then - those were the days), so when my boss approached me and asked me whether I liked to take over a coding project, I agreed to the idea.

Little did I know at the time: I was supposed to work with a web agency the boss had contracted to build their online shop. Only that he had no plan or anything, he basically told them "build me an online shop like abc(a major competitor of ours at the time)"

He employed another sales lady who was supposed to manage the shop (that didn't exist yet). In the end, I think 80% of her job was to keep me from killing my boss.

As you can imagine, with this huuuuge amout of planning and these exact visions of what was supposed to be, things went south fast and far. So far that I could visit my fellow flightless birds down in the Penguin's republic of Antarctica and still need to go further.

Well... When my boss started suing the web agency, I was... ahem, asked to take over. Dumb as I was, I did - I was a PHP kid and thought that Magento, being written in PHP, would be easy to master. If you know Magento, you know that was maybe the wrongest thing I ever said.

Fast forward 3 very exhausting months, the thing was online. Not all of it worked yet, but it was online and fairly secure.

I did next to everything myself, administrating the CentOS box the shop was running on, its (own) e-mail server, the web server, all the coding required for the shop (can you spell 12 hour day for 8 hour pay?)

3 further months later, my life basically was a wreck, I dragged myself to work, the only thing I looked forward being the motorcycle ride home. The system worked though.

Mind you, I was still, at the time, working with three major customers, doing deskside support and some admin (Win Server 2008R2 at the time) - because, to quote my boss, "We could not afford a full time developer and we don't need one".

I think i stopped coding in my free time, the one hobby I used to love more than anything on the world, somewhere Decemerish 2012. I dropped out of the open source projects I was in, quit working on my browser game and let everything slide.

I didn't even care to renew the domains and servers for it, I just let it die without notice.

The little free time I had, I spent playing video games and getting drunk/high.

December 2013, 1.5 years on the job, I reached my breaking point and just left, called in sick at least a week per month because I just could not see this fucking place anymore.

I looked for another job outside of ALL of what I did before. No more Magento, no more sales, no more PHP. I didn't have to look for long, despite what I thought of my skills.

In February 2014, I told my boss that I quit. It was still seven months until my new job started, but I wanted him to know early so we could migrate and find a replacement.

The search for said replacement started in June 2014. I had considerably less work in the months before, looks like he got the hint.

In August 2014, my replacement arrived and I got him started.

I found a job, which I am still in, and still happy about after almost half a decade, at a local, medium sized ISP as a software dev and IT security guy. Got a proper training with a certificate and everything now.

My replacement lasted two months, he was external and never really did his job - the site, which until I had quit, had a total of 3 days downtime for 3 YEARS (they were the hoster's fault, not mine), was down for an entire month and he could not even tell why.

HIS followup was kicked after taking two weeks to familiarize himself with the project. Well, I think that two weeks is not even barely enough to familiarize yourself with nearly three years of work, but my boss gave him two days.

In 2016, the shop was replaced with another one. Different shop system, different OS, different CI. I don't know why and I can't say I give a damn.

Almost all the people that worked at the company back with me have left for greener pastures, taking their customers (and revenue) with them.

As for my boss' comments, instructions and lines: THAT might not be safe for work. Or kids. Or humans in general. And there wouldn't be much left if you put it through a language filter...

Moral of the story: No, it's not a bad thing to leave a place if you're mistreated there. Don't mistake loyalty with stupidity!

And, to quote one of my favourite Bands: "Nothing matters when the pain is all but gone" (Tragedy + Time by Rise Against).

devCraft {
Closing the minecraft server for a little while!
I'll be adding mods, writing up a perms file, and hosting the pack on git! I'll post a rant with the repo link.
One of our lovely ranters offered a VM to host the server on, so the ip is probably gonna change as well! (i also gotta make an arch bootable USB, and running the server would slow that down lol)
i'll notify you all once it's donevia a rant, like i said. until then, formulate plans, and suggest some developer-related mods for me to add in! (must be 1.7.10)
Currently planned mods are:
- ComputerCraft
- Applied Energistics
- Buildcraft
- Project Red
and a few from whatever you guys suggest. see you then!
}

My worst experience was at my job where they told me I have to move to a permanent position from 3 years of contracting without a specific offer.
Why is that bad? In my country it means approximatly 40% lower wage.
I came into the job with PHP knowledge when they were looking for Perl on a project one year behind schedule. I learned the language and finished working demo in 6 weeks.
After that, every project that was ever assigned to me was done within 5-15% of the allocated time. I'm not kidding here. My manager loved be, because I was reliable, fast and I even 'accidentaly' solved other problems, like for instance I developed simple syslog search tool and benchmarked zip algos for reading speed, and the fastest had 70% better compression than the algo used before (gzip into plzip on 1-2gb files). That solved anothet problem - syslog servers did not have enough disk space and they didn't have money to upgrade the server.
The number of projects I touched or developed was over 20.
I also lead and developed our team's most successful tool, that every customer was throwing money to buy, while cutting down costs everywhere.
And after three years of that, my manager says that there are no more money for contractors. And the only possibility is going for employment. Without any specific offer! Just 'we cant do this anymore'.
Which I understand, that can happen in corporation, but ffs after all I've done, I expected warmer attitude. Not like 'you may have to leave, since we do not really care'.
I liked the people there, even though the corporation environment was lacking in many respects, but I wanted to help our local branch with everything I could and they gave up on me like that.
So I started looking elsewhere and I found a startup which offered 6 times the money I had in my previous job and promises to relocate me to USA. Which is the best thing that has happened to me that year and second best in my whole life!

So, a few weeks ago I asked you guys what would you do if you were to quit your current job. I attempted a start-up with an old "friend" of mine.

He was very enthusiastic and hardworking at the beginning. However he received a job offer from somewhere and told me that he would work there just until we found the company officially and then quit and work full time on the project.

Well... I put around 300 hours into the project and developed the system, did most of my part according to the plan but the guy didn't deliver anything.

Turned out he had another secret partner whom he first introduced me as his mentor. I had my suspicions and suddenly shut down API servers, project management domain and mail server. Suddenly the quiet guy called me asking if I was alright and everything was fine.

Then, nothing happened. He went radio silence until I called him last night and he picked up the phone drunk and mumbled something like "I'm aware of you being a victim" then went to "you're passive aggresive" or something like that and I said nothing, just hung up.

I think you can figure out what went down on the other side and I'd like to hear your scenarios.

PS: now I have another start-up idea: I'm gonna pick up a flamethrower and burn the world while laughing histerically. Anyone who'd like to join is welcome.

!rant

Trust no one.
The internet is not your friend, until you find stackoverflow and you get down voted.
That rm -rf / won't make you server faster.
System32 is needed.
Yes, that is a package manager, you don't need to write more ccode
Do not write commets on languages that only you speak, the team does not speak in latin.
Paint is the best engineering tool.
Keep a stress ball nearby.
Your client is always right, unless they mess with your coding skills.

True story.

client : show website
me: sir server is down
client : what? I don't know,I want to see my website.
me: sir try to understand it is not my issue
client *call cut*

fucking asshole -_-

I was noticing some slow network and it was dropping some connections. So I booted up my old XP install with Java 6 so connect to the ASA 5505, I see it’s logging max connections of 10000 has been reached.

Fine, I recon it’s my colleague backing up his entire machine to Google Drive.
Because when he shut it off, n connections dropped.

I check back in the log, and I see there’s 4-500 connections happening per second, I think WTF and check the source IP. Lots of random IPs from Vietnam, all going to a Windows2008 Server using rdp.

(I didn’t setup our servers, so I didn’t know which server it was accessing)

Ask my other colleague, he told me it’s a windows server from an earlier project that’s not used anymore.
I rdp into it, see there’s users logged in from around the world, and I immediately do a shutdown.

Would you look at that, connections per second dropped to about 50.
I guess that server isn’t going back online ever.
And I now need to ask management for a budget to update our network infrastructure, because the old ASA 5505 is begging me to die.

TL;DR gg previous employees didn’t shut down old servers and left them open to the world to enjoy

he: checkout my crazy FUD hack (a token stealer which turned out to be far more malicious than i anticipated)
me: executes it (yes in a VM)
windows defender: lemme delet this
he: ooh i forgot the word stub in there. microsoft detects that lemme fix that sends new file
me: here we go aga..
ms defender: nononono virus 117% delet this
he: i forgot it still!!

later i deactivated ms defender and analysed the traffic of the vm. in addition to stealing my fake tokens he also tried to read my Firefox/chrome history, IP.
when i asked him (2 days later) what this was all about in his "educational only" "token stealer" he threatened to
a) publish my IP
b) publish my browser history and with that my real name and address
b.0) when i asked him for proof he said he knows that my real name is "Roman Gräf" and i live in Frankfurt. (btw i do live in Frankfurt and that is in the profile of the discord server where he found me and i have the same username on discord as i have here)
c) to kill my machine and all my projects
got bored, blocked him, shut VM down.

Was on a call today...nobody knew who owned or was responsible for one of the servers. My manager has a great way of finding out...he says "If I take my hammer and smash the server to pieces...who gets pissed." A moment of silence then someone speaks up..."Oh well then that would be Joe down the hall."

That my friends is how to get answers. Threaten to break things.

Did some updates to an older Web Forms website built by a previous SENIOR developer who is a notoriously horrible developer.

Now before I start, you have to understand this guy studied at a University and had been working for at least two years before I even started working. He is supposed to know the basic shit mentioned below.

This also happened a couple of days ago, so I have calmed down since then so I apologise for the relaxed tone. My next rant will contain a lot more swearing.

This fucking guy did the stupidest shit imaginable.

On the details view of a post|page|article|product|anything that would require a details view this jackass would load the data from the DB.

Using an OleDbConnection, OleDbDataAdapter, DataTable and the poorest writter fucking sql statements you have ever seen. All of these declared in the Page_Load method.

There was literally no reason for him to use OleDb instead of Sql, but he simply did not know any better.

He especially liked: "select * from tbl where id = " & Request("T") & ""

ZERO fucking checks to see if the value is even passed or valid, nothing. He did not even check whether the DataTable had any rows.

He then proceeded to use only the Heading column of the returned row to change the page's title.

Stupidly I assumed the aspx page will be in a better state. Fuck NO!

This fucktard went, added server tags to the opening of the asp:Content tag, copied that shit he used to fetch the data and pasted it between the server tags.

He did not know how to access the DataTable mentioned above from the aspx page!

He did this on every fucking project he worked on. Any place that required <%= %> to display data instead of using asp server controls, this cunt copied whatever was written in the code behind and pasted everything between server tags.

Fuck I could go on forever, but I think this is enough for my first rant.

I hired 2 fresh out of school junior devs to work with me on my old web app.
They were brilliant, knew a lot of things, and were motivated.
They started complaining about how the code was shit, the db was shit, there were no best practices, the technology was old, bug fixing was boring, no comments in code.

I felt bad, very bad during 3 years, because they were absolutely right. I tried to work with them through better coding practices, rewriting, documenting etc.

Now they both have left.
I'm alone maintaining and evolving the application.
And I start to come across the code THEY developed.

What a bunch of shit. SQL queries bringing down the server. Duplicate code, because they didn't want even read the old one. Useless comments.
Performance killing functions. Exceptions swallowed without mercy. I have to clean up they poop.

I feel somewhat better, though. The application is still growing and holding the ground after many years and generating at least 800K$ per year in revenues.

Maybe better, but sad. I really wanted to share the project with somebody else but I failed, and I'm left alone....

! exactly dev

I'd ditched Windows and spent a while exploring the Linux ecosystem for content creation. And I have to say, it was not a nice experience.

As much as I respect the Linux mantra of "free as in freedom" and "you need to roll up your sleeves and figure out stuff on your own", it just isn't good enough for non-dev work. Sorry guys, but I need software that gets out of my way and at least does what it's supposed to do. I can't stand a horrible UI or delays and random crashes, which is exactly what happens with most things under Linux.

To replace my Windows workflow I used the following:
1. Windows -> elementaryOS (because Debian/Ubuntu repositories seem to have the best software support, and elementaryOS is the least horrible looking thing that supports that) and then Arch, because, well, Arch.
2. Blender + Maya -> Blender + Maya on Linux.
3. Reaper + FL Studio -> Ardour + LMMS.
4. Photoshop -> GIMP + Krita + Inkscape.
5. ZBrush -> nothing :(

As you can see, my use cases are pretty much all over the spectrum.

Firstly, installing and configuring stuff. A pleasure on Windows, an absolute pain on Linux. Everything just worked on Windows, I had to wrestle with library versions and patches and unstable audio layers (Linux audio just sucks, except for JACK) on Linux.

Out of these, Blender and Maya were the best experience. But even then, both would suffer from random crashes that just didn't happen on Windows.
Ardour is actually really nice when it works. Its use of JACK for routing makes it really really flexible, but it just isn't stable enough to depend on. LMMS is utter crap. I'm sorry, but I just hate the UI. Can't stand it.
GIMP, Krita, and Inkscape can't beat Photoshop, even when you consider them together. Adobe software workflow is just so much better and more intuitive.
Blender 3D sculpting is not bad, but it's nowhere as good as ZBrush.

Also, if you're a C++ dev like me, nothing beats Visual Studio 2017. Nothing. That IDE just blows everything else out of the water. Even VSCode. And it's not slow at all, it handled a fairly large project (PBRTv3) just fine on my Windows development VM. Yes, a VM.

So...I ditched Linux and went back to Windows, but I keep Linux as a VM for when I actually want to mess with Blender or Ardour. Or some dev stuff which Windows sucks at (which is becoming less frequent because of WSL).

Out of all the above, the only one I'd consider ready for production use would be Blender. Developers of open source software, please learn from Blender. Kickass UI and user friendly operation is extremely important, you can't make a random window with GTK buttons and text boxes and arcane config files and expect people to use it for serious work.

Also, Windows beats Linux hands down as an everyday OS. It's always been rock solid, if you take care of it properly (and that goes for any OS). Updates hardly take any time because I run it on a SSD. As for all the advertising and marketing bullshit, you can block a large amount of stuff. And for what can't be blocked, well, I just have to live with it, because the alternative is compromising on my creative output, which is too much for me.

I still run Linux on my server, though. And on my embedded devices (Pi, BeagleBone, etc.). It absolutely rocks there.

I realize that Linux software is not going to improve unless we do something about it, so I'll be contributing fixes and code (the joys of being a C++ dev, yay). Still, I feel that the platform and software as a whole is just not mature enough.

Why the fuck does your API have to send 401 when the server is down.

Just spent an hour thinking something was wrong with my auth wrapper.

Best code performance incr. I made?

Many, many years ago our scaling strategy was to throw hardware at performance problems. Hardware consisted of dedicated web server and backing SQL server box, so each site instance had two servers (and data replication processes in place)

Two servers turned into 4, 4 to 8, 8 to around 16 (don't remember exactly what we ended up with). With Window's server and SQL Server licenses getting into the hundreds of thousands of dollars, the 'powers-that-be' were becoming very concerned with our IT budget. With our IT-VP and other web mgrs being hardware-centric, they simply shrugged and told the company that's just the way it is.

Taking it upon myself, started looking into utilizing web services, caching data (Microsoft's Velocity at the time), and a service that returned product data, the bottleneck for most of the performance issues. Description, price, simple stuff. Testing the scaling with our dev environment, single web server and single backing sql server, the service was able to handle 10x the traffic with much better performance.

Since the majority of the IT mgmt were hardware centric, they blew off the results saying my tests were contrived and my solution wouldn't work in 'the real world'. Not 100% wrong, I had no idea what would happen when real traffic would hit the site.

With our other hardware guys concerned the web hardware budget was tearing into everything else, they helped convince the 'powers-that-be' to give my idea a shot.

Fast forward a couple of months (lots of web code changes), early one morning we started slowly turning on the new framework (3 load balanced web service servers, 3 web servers, one sql server). 5 minutes...no issues, 10 minutes...no issues,an hour...everything is looking great. Then (A is a network admin)...
A: "Umm...guys...hardly any of the other web servers are being hit. The new servers are handling almost 100% of the traffic."
VP: "That can't be right. Something must be wrong with the load balancers. Rollback!"
A:"No, everything is fine. Load balancer is working and the performance spikes are coming from the old servers, not the new ones. Wow!, this is awesome!"
<Web manager 'Stacey'>
Stacey: "We probably still need to rollback. We'll need to do a full analysis to why the performance improved and apply it the current hardware setup."
A: "Page load times are now under 100 milliseconds from almost 3 seconds. Lets not rollback and see what happens."
Stacey:"I don't know, customers aren't used to such fast load times. They'll think something is wrong and go to a competitor. Rollback."
VP: "Agreed. We don't why this so fast. We'll need to replicate what is going on to the current architecture. Good try guys."
<later that day>
VP: "We've received hundreds of emails complementing us on the web site performance this morning and upset that the site suddenly slowed down again. CEO got wind of these emails and instructed us to move forward with the new framework."

After full implementation, we were able to scale back to only a few web servers and a single sql server, saving an initial $300,000 and a potential future savings of over $500,000. Budget analysis considering other factors, over the next 7 years, this would save the company over a million dollars.

At the semi-annual company wide meeting, our VP made a speech.

VP: "I'd like to thank everyone for this hard fought journey to get our web site up to industry standards for the benefit of our customers and stakeholders. Most of all, I'd like to thank Stacey for all her effort in designing and implementation of the scaling solution. Great job Stacy!"
<hands her a blank white envelope, hmmm...wonder what was in it?>

A few devs who sat in front of me turn around, network guys to the right, all look at me with puzzled looks with one mouth-ing "WTF?"

Project manager asks why server is down. Reminded him that his team lead told me three months ago it was no longer needed.

Ever want to smack someone in the face with a sea bass? Like left out of the water for 3 days with all kinds of juicy and smelly goodness?
When we get an X number system errors, an email is sent to our team. Couple of hours ago I had to move the alerts from one system to another, re-naming some because I suck at naming things. I guess when I copied, I duplicated one. About an hour ago we get a system alert (as it should, there was a server hiccup) and there were two emails with the same data (just named differently)
DevA: “Why are there two emails?”
Me: “Oh, that’s me. I think I copied the alert instead of move. I’ll fix it tomorrow.”
DevA: “Hmm, but the data is the same. It’s a duplicate.”
Me: “Yea, I know. Networking responded and said it’s fixed. We won’t get another email.”
- 15 minutes later
DevA leans over and says to the boss (who was in a meeting and just sat down)…
DevA: “I looked at the alert, it’s duplicated, but the name is different. I don’t understand why.”
Me: “Like I said, the alert is duplicated. When I migrated, I copied instead of moved. No big deal.”
DevA: “Oh …oh ..yea.…OK.”
- 5 minutes later
DevA: “I looked at the query, we might have to add a filter to prevent duplicate emails. Probably some logic problems in the search.”
Me: “I just deleted the duplicate alert.”
DevA: “Oh…OK…that fixes it too.”

Good lord…as I was typing this, he just told another dev the ‘duplicate’ emails were because of a logic bug in the search. I’m getting my fishing pole.

Two years ago I started a small online business. It was not a long term investment and it literally ended up being a one man business. The idea was to provide a service to a small group of people who will benefit from my idea and to offer it to them at a very cheap price. (It being the cheapest helped its popularity a lot).

However, never once did it actually make any profit. (and i never wanted it to make a profit) I wanted it to be self sustaining business and it was.

This was a project for my University by the way, I started off in my first year because of my extensive knowledge in the particular matter, and I only sold to people on campus.

Now that its been 4 years, my batch is graduating, and so there aren't many people to spread the word about this project. It's finally the time to actually say goodbye to this project.

I leased a dedicated server two years ago, and I am finally saying goodbye to that too (can't afford to keep it live anymore). And seriously, it feels sad to shut this machine down haha, I've had so much fun playing around with the configurations (even though it was a production server).

It's clear that this downsizing will continue and I will be closing the service in the near future.

Teaching my homeschooled son about prime numbers, which of course means we need to also teach prime number determination in Python (his coding language of choice), when leads to a discussion of processing power, and a newly rented cloud server over at digital ocean, and a search of prime number search optimizations, questioning if python is the right language, more performance optimizations, crap, the metrics I added are slowing this down, so feature flags to toggle off the metrics, crap, I actually have a real job I need to get back to. Oooh, I'm up to prime numbers in two millions, and , oh, I really should run that ssh session in screen so it keeps running if I close my laptop. I could make this a service and let it run in the background. I bet there's a library for this. He's only 9. We've already talked about encryption and the need to find large prime numbers.

Internships are fucking bullshit and if more senior developers were to take the role of an actual mentor to coach juniors properly then the state of software engineering would be better.

Some people can be let down easy in terms of "this is not for you bruh", others can be built. I know that social interactions are not common for a lot of the morons in here, but being polite and kind is relatively simple if you know what you are doing. Being a dickhead != "royal levels of expertise" and if we were to coach more people into proper development practices then software would not be in such a shitty state.

For an environment that thrives in cooperation I find it hard to believe that we are still subjecting new people to the field to what can be considered slavery with little to actual no monetary compensation.

I removed many of the requirements for the application to a software developer job where I am at (I am the boss, I get to do shit like that) and my fight with HR was "I would rather someone fresh from college that I can coach properly than some dickhead with years on the field that won't listen to anything else than their own words"

Sure it would be slow, sure it would be hard, nothing ever is that simple, but my idea is "train this mkfer, level the fuck out of him, let him be off to great shit rather than giving him to some dickhead that will treat him like shit on account of being a newbie"

And yes, I do know how and what can go bad, I am going to have someone desinging shit in basic html/js/css with some php here and there not giving them the keys to every server I control. Thank you for your fucking concerns, I know what I am doing.

the experiment fails? GOOD more data for me.

Plus, you learn more when you teach others.

I'm having a DAY.

it's like I'm trying to "2+2=4" and hitting "unknown operator =", figure that out, get "22" back, fix that, then math service is down, redeploy math service, wait 30 minutes, that's up, get "out of numbers" exception, turns out a dependency updated and now I have to register what "2" means, fix that, FINALLY get 2+2 to evaluate, I get 5. Figure that out, get 4! 4!!! FINALLY. Push it. Get 55 on the server. Gets 4 on my machine, getting some random 55 from the server.

This is the craziest shit... MY FUCKING SERVER JUST SET ON FIRE!!!

Like seriously its hot news (can't resist the puns), it's actually really bad news and I'm just in shock (it's not everyday you find out your running the hottest stack in the country :-P)... I thought it slow as fuck this morning but the office internet was also on the fritz so I carried on with my life until EVERYTHING went down (completely down - poof gone) and within 2 minutes I had a technician from the data centre telling me that something to do with fans had failed and they caught fire, melted and have become one with the hardware. WTF? The last time I went to the data centre it was so cold I pissed sitting down for 2 days because my dick vanished.

I'm just so fucking torn right now because initially I was absolutely fucking ecstatic - 1 week ago after a year of doomsday bitching about having a single point of failure and me not being a sysadmin only to have them look at me like I'm some kind of techie flat earther I finally got approval to spend around 5x more per month and migrate all our software to containerized micro services.

I'll admit this is a bit worse than I expected but thanks to last week at least I have recent off site images of the drives - because big surprise I have to set this monolithic beast back up (No small feat - its gonna be a long night) on a fresh VPS, I also have to do it on premises or the data will only finish uploading sometime next week.

Pro Tip: If your also pleading for more resources/better production environment only to be stone walled the second you mention there's a cost attached be like me - I gave them an ultimatum, either I deploy the software on a stack that's manageable or they man the fuck up and pay a sys admin (This idea got them really amped up until they checked how much decent sys admins cost).

Now I have very flexible pockets because even if I go rambo the max server costs would only be 15-20% of a sys admins paycheck even though that is 13 x more than our current costs.

Just had a fire at work.
git add, git commit, git push.

No one was hurt. But, the fire was in a server room and the git server is down. At least the halon deployed successfully.

The off site backup should be up in 20 mins. I wonder how often they pull the repo.

This is a story of me trying out maintaining a game server and eventually making a mistake, although I do not regret experiencing it.

A month ago I set up a small modded minecraft server because I wanted to experience a fun modpack together with some people from reddit. Besides this, I also wanted to see if I was capable of setting up a server with systemd and screen running in the background. This went great and I learned a lot.

The very next day I was playing with $annoyingKid on the server and everything was well. However the second day, $annoyingKid started pushing the idea to start up a normal minecraft server to build a playerbase.

I asked $annoyingKid 'What about financing, staff management and marketing?'

$annoyingKid: "I don't know much about that, but you can do that while I build a spawn!"

He also didn't want to reveal his age, which alerted me that he's young and inexperienced. He also considered Discord 'scary' because there were haxors and they would get his location and kidnap him, or something. So if he was supposed to become owner (which he desired), he had no way of communicating with a community outside of the game.

He also considered himself owner, while I was the one who paid for the server. 'Owners should be people who own the server', no matter how many times I told him that.

$annoyingKid also asked if he could install plugins on his own, I asked him if he knew anything about ssh, wget or bash because I used ssh to set up the server (I know rcon exists, but didn't want to deal with that at the time), he had no idea what any of those terms meant and he couldn't give proper arguments as to why he should get console access.

In the end, he did jack shit, he had no chance of becoming co-owner or even head-admin because he had no sense of responsibility or hard work. I kept him around as an admin because he was the one who came up with the idea. I banned him on day one after he started abusing his power when someone tipped him of. Even after me ordering him to ignore an annoying player he kept going, of course I could have prevented all this by kicking him earlier since all the red flags around him had already formed a beacon of light. He tried coming back, complaining that he should at least have his moderator rank back, but he never got in again.

A week later I got bored, I had had enough fun with ssh and the server processes to know that I didn't want to continue the small project, so I shut it down and went on to do stuff on GitHub.

Lesson learned: Don't let annoying kids with no sense of responsibility talk you into doing things you aren't sure you want to be doing. And only give people power after they've proved to you that they are capable of handling it.

A Bad and Sad Day

Hello Monday,

Client : on weekend site went down for 7 hours? Why ?
Me : Let me check the logs
Client : bla bla bla
boss : check ur code bla bla bla
Reason : Some PHP service stopped on server

Client2 : I have purchased this software and you have to made minor changes
Me : Payment Integrations are not working?
Client : Whataa nonsense is this.. you are supposed to do this
Me : We are supposed to do minor changes. They do not have proper payment integrations. If you want we have to write complete code
Client : bla bla.. I gave you working software

:( Why don't you just fuck off .. liars

Your web server config is fucked, and your app is down.

~ "Faithfully yours, nginx."

School gave me 3 DigitalOcean droplets to try out Kubernetes in the cloud, awesome!

Wrote an Ansible script to not only simply install docker and add users but also add kubernetes, nice!

Oh wait, error?! Well I should've known this wasn't going to be easy... ah well no problem. Let's see... Ansible is cryptic as always, it can't connect to the API server? Is it even running?

Let's ssh to the master, ah nothing is running, great. Let's try out kubeadm init and see what happens, oh gosh, my Docker version has not been validated! No problem, let's just downgrade!

How do I do that? Oh I know, change the version in the role! Wait that version doesn't exit? Let's travel to Docker's website and see what versions exist of docker-ce, oh I see, it needs a subversion, no problem.

Oh that errors too? Wait then what... Oh I need a ~ and a ubuntu and a 0 somewhere, my mistake!

Let's run it again! Fails!

Same ssh process, oh wait...

Oh god no...

Kubernetes requires 2 cores and these things only have 1...

Welp, time to ask the teachers to resize my droplet by a small amount tomorrow, hopefully I'll get a new error!

----------------------------------------------

My adventure so far with Kubernetes. I'm not installing it for any serious/prod reason, just for educational purposes. K8s seems like 'endgame' to me, like one of the 'big guys' that big enterprises use so I'm eager to throw stuff at a droplet and see what happens.

Going further down the rabbit hole tomorrow!

Wish me luck :3

(And yes, I could've figured this all out beforehand with documentation, but this is more fun in my opinion)

I call this one the tester than knew too much.

Note: The server the tester is running on has a hard drive that is breaking down...

Tester: Remember the error I talked to you about yesterday?

Me: Yeah, what about it?

Tester: Well the server hasn't recovered yet and I haven't restarted anything...

Me: Well the application itself hasn't crashed so our monitoring application doesn't seem to notice that the service is in a bad state. The error seems only to have brought down certain threads within the application.

Tester: No, I think there is a different issue here and has nothing to do with that error, the application is still doing things.
*tails the log*
Tester: See?

Me: As I said some things are still running and are unaffected by the error.

Tester: NO! It has to be caused by the other error I had a week ago where my file got corrupted. As we said I removed the file, restarted it and it worked again, but had the same problem a day later...

Note: The problem is not related, this time the application is running out of file descriptors

Me: Well... If the problem is the same it would have complained about the file descriptors then aswell, not an I/O error.

Tester: Nope, I think you are wrong!

Me: ¯\_(ツ)_/¯

FML

Inherited a simple marketplace website that matches job seekers and hospitals in healthcare. Typically, all you need for this sort of thing is a web server, a database with search

But the precious devs decided to go micro-services in a container and db per service fashion. They ended up with over 50 docker containers with 50ish databases. It was a nightmare to scale or maintain!

With 50 database for for a simple web application that clearly needs to share data, integration testing was impossible, data loss became common, very hard to pin down, debugging was a nightmare, and also dangerous to change a service’s schema as dependencies were all tangled up.

The obvious thing was to scale down the infrastructure, so we could scale up properly, in a resource driven manner, rather than following the trend.

We made plans, but the CTO seemed worried about yet another architectural changes, so he invested in more infrastructure services, kubernetes, zipkin, prometheus etc without any idea what problems those infra services would solve.

First rant here, and it's going to be a query to the more professional and experienced members of society (most of you).

I am currently a Sys Admin for a major company, and I develop at night. My primary employment at the moment is the sys admin job (and I code for extra money at nights).

I wanted to start a development department at the company that I am working at, but it was turned turned down. It was stated that we are not branching in development, and that we should stick to our server implementation and support. This was a prompt to me wanting to start studying officially (I wanted to get qualified in JAVA, so that I had some paper behind my name when I looked for another job). HR and my directors outright denied me the ability to study through them (they pay for studies for employees) and I was more than fine with this.

I took a loan and paid for the studies myself. Can't crush a dream, you know?

The director caught wind of me studying, and now has demanded that I develop him a mobile application for the company. I told him that I am not a mobile developer, and that it didn't fall into my key performance areas.

Note, I do my coding on own time, on my own device, and never at work. It's fully my intellectual property. It also in no way interferes with my work during the day, and has NO conflict with my contract this side.

He sent an email yesterday, this is after two months. He is now stating that I WILL do the application, and he has CCd HR and two directors.

I don't want to do the app for this company, I spoke to HR previously about this, and she said that I should try and quote it under my own company name (which I did, but it was denied as it was "too expensive").

Now I am being forced to do something that is COMPLETELY out of my roles and responsibilities, something that this company has ABSOLUTELY no desire to go into further on, and he is basically letting me know that if I don't do it, he is going to start messing with my pay.

I really don't want to do this, and I cannot afford to make my secondary job my primary at the moment. The problem is, too, that I don't have the time during the day to develop AND do my sys admin tasks (I manage more than 300 servers, and 5000 devices).

What can I do in this instance? Or what would you guys recommend, in your experience?

Sorry for the noob question, but I don't know what to do.

Looks like my server is going to be down tomorrow :(
Its raining and my server is in the cloud.

While I work on my devRant client, I always end up thinking, "how would dfox do it for the backend".

Some of the functions I have include:
-fetching new rants, 250 at a time
-remembering the ones I've read and filtering those out (kinda similar to Algo)
-checking for updates from followed users and new notifications

My implementions are kinda spammy though part of it is because I need to use the existing API.

But even so, how would/does he implement these so all the users' requests don't end up bringing down the server, and efficiently store and retrieve everyone's view history?

My problems are small compared to his... How do you think and deal with problems at that scale though?

Next 3 days I' ll be working on moving our email server to new server because the old datacenter is closing down. Clients are pissed and they are fucking verbally bashing me left to right when I told them that their emails will be down since most of them host their own DNS.

I am fucked.

Network Security at it's best at my school.
So firstly our school has only one wifi AP in the whole building and you can only access Internet from there or their PCs which have just like the AP restricted internet with mc afee Webgateway even though they didn't even restrict shuting down computers remotely with shutdown -i.
The next stupid thing is cmd is disabled but powershell isn't and you can execute cmd commands with batch files.
But back to internet access: the proxy with Mcafee is permanently added in these PCs and you don't havs admin rights to change them.
Although this can be bypassed by basically everone because everyone knows one or two teacher accounts, its still restricted right.
So I thought I could try to get around. My first first few tries failed until I found out that they apparently have a mac adress wthitelist for their lan.
Then I just copied a mac adress of one of their ARM terminals pc and set up a raspberry pi with a mac change at startup.
Finally I got an Ip with normal DHCP and internet but port 80 was blocked in contrast to others like 443. So I set up an tcp openvpn server on port 443 elsewhere on a server to mimic ssl traffic.
Then I set up my raspberry pi to change mac, connect to this vpn at startup and provide a wifi ap with an own ip address range and internet over vpn.
As a little extra feature I also added a script for it to act as Spotify connect speaker.
So basically I now have a raspberry pi which I can plugin into power and Ethernet and an aux cable of the always-on-speakers in every room.
My own portable 10mbit/s unrestricted AP with spotify connect speaker.
Last but not least I learnt very many things about networks, vpns and so on while exploiting my schools security as a 16 year old.

A few days ago Aruba Cloud terminated my VPS's without notice (shortly after my previous rant about email spam). The reason behind it is rather mundane - while slightly tipsy I wanted to send some traffic back to those Chinese smtp-shop assholes.

Around half an hour later I found that e1.nixmagic.com had lost its network link. I logged into the admin panel at Aruba and connected to the recovery console. In the kernel log there was a mention of the main network link being unresponsive. Apparently Aruba Cloud's automated systems had cut it off.

Shortly afterwards I got an email about the suspension, requested that I get back to them within 72 hours.. despite the email being from a noreply address. Big brain right there.

Now one server wasn't yet a reason to consider this a major outage. I did have 3 edge nodes, all of which had equal duties and importance in the network. However an hour later I found that Aruba had also shut down the other 2 instances, despite those doing nothing wrong. Another hour later I found my account limited, unable to login to the admin panel. Oh and did I mention that for anything in that admin panel, you have to login to the customer area first? And that the account ID used to login there is more secure than the password? Yeah their password security is that good. Normally my passwords would be 64 random characters.. not there.

So with all my servers now gone, I immediately considered it an emergency. Aruba's employees had already left the office, and wouldn't get back to me until the next day (on-call be damned I guess?). So I had to immediately pull an all-nighter and deploy new servers elsewhere and move my DNS records to those ASAP. For that I chose Hetzner.

Now at Hetzner I was actually very pleasantly surprised at just how clean the interface was, how it puts the project front and center in everything, and just tells you "this is what this is and what it does", nothing else. Despite being a sysadmin myself, I find the hosting part of it insignificant. The project - the application that is to be hosted - that's what's important. Administration of a datacenter on the other hand is background stuff. Aruba's interface is very cluttered, on Hetzner it's super clean. Night and day difference.

Oh and the specs are better for the same price, the password security is actually decent, and the servers are already up despite me not having paid for anything yet. That's incredible if you ask me.. they actually trust a new customer to pay the bills afterwards. How about you Aruba Cloud? Oh yeah.. too much to ask for right. Even the network isn't something you can trust a long-time customer of yours with.

So everything has been set up again now, and there are some things I would like to stress about hosting providers.

You don't own the hardware. While you do have root access, you don't have hardware access at all. Remember that therefore you can't store anything on it that you can't afford to lose, have stolen, or otherwise compromised. This is something I kept in mind when I made my servers. The edge nodes do nothing but reverse proxying the services from my LXC containers at home. Therefore the edge nodes could go down, while the worker nodes still kept running. All that was necessary was a new set of reverse proxies. On the other hand, if e.g. my Gitea server were to be hosted directly on those VPS's, losing that would've been devastating. All my configs, projects, mirrors and shit are hosted there.

Also remember that your hosting provider can terminate you at any time, for any reason. Server redundancy is not enough. If you can afford multiple redundant servers, get them at different hosting providers. I've looked at Aruba Cloud's Terms of Use and this is indeed something they were legally allowed to do. Any reason, any time, no notice. They covered all their bases. Make sure you do too, and hope that you'll never need it.

Oh, right - this is a rant - Aruba Cloud you are a bunch of assholes. Kindly take a 1Gbps DDoS attack up your ass in exchange for that termination without notice, will you?

The S on Holiday stands for stability on the production server...

I fucking hate stupid accountants!

Yesterday we went to a customer to talk to the accountants because we want to remove one of their unused PC's in the office.

First, just the way they think (and talk if) they are the most important and it's absolutely critical everything works 100%. I see they are important but not 100 times more important than everybody else!

They called us their EDP-guys (EDV in German, that's the translation I found). That insulted me a bit. I'm rather called IT-guy, I don't know anything about the fucking EDP systems nor want I to. I'm there to make sure the hardware works. But whatever, fine, call me what you want.

Then they straight up threatened us, because their work is so important, they can't afford to have downtime in their systems. They don't really care, but the bosses of us both do and if we fuck up they (the bosses) will hold us responsible. There is a fucking update for your piece of shit software (datev)! I don't do the update, I'm just responsible that the update can be deployed on the hardware. I'm not responsible if this update fucks your system and frankly I don't care!

I could tell them all of this but they won't listen. They always talk in this patronising arrogant voice, because they are so important and we better don't fuck up the update.

I'm there to help. I don't want downtime for your systems. I want you to work with our systems the best you can.
But fuck you, I hope the server burns down!

Still dealing with the web department and their finger pointing after several thousand errors logged.

SeniorWebDev: “Looks like there were 250 database timeout errors at 11:02AM. DBAs might want to take a look.”

I look at the actual exceptions being logged (bulk of the over 1,600 logged errors)..

“Object reference not set to an instance of an object.”

Then I looked the email timestamp…11:00AM. We received the email notification *before* the database timeout errors occurred.

I gather some facts…when the exceptions started, when they ended, and used the stack trace to find the code not checking for null (maybe 10 minutes of junior dev detective work). Send the data to the ‘powers that be’ and carried on with my daily tasks.
I attached what I found (not the actual code, it was changed to protect the innocent)
Couple of hours later another WebDev replied…

WebDev: “These errors look like a database connectivity issue between the web site and the saleitem data service. Appears the logging framework doesn’t allow us to log any information about the database connection.”

FRACK!!...that Fracking lying piece of frack! Our team is responsible for the logging framework. I was typing up my response (having to calm down) then about a minute later the head DBA replies …

DBA: “Do you have any evidence of this? Our logs show no connectivity issues. The logging framework does have the ability to log an extensive amount of data regarding the database transaction. Database name, server, login, command text, and parameter values. Everything we need to troubleshoot. This is the link to the documentation …. If you implement the one line of code to gather the data, it will go a long way in helping us debug performance and connectivity issue. Thank you.”

DBA sends me a skype message “You’re welcome :)”

Ahh..nice to see someone else fed up with their lying bull...stuff.

Scared the shit out of me when I heard one of my volunteer side projects website was down. Because I just finished fixing my own hacked website today.

Turn out the server and website is fine but the domain was expired yesterday 😣

Don't fucking scare at me like that.

Tough week indeed.

Fuck google cloud platform. My server has been down for last 4 days. Stupid reason google gives me is that it does not have resources available in my zone. Why the fuck do you start a hosting company if you cannot provide RAM and CPU. On top of that their support is so bad that after 20 emails, 4 chat tickets, 3 phone calls nobody knows the issue I am facing. They just give the links to their ultra stupid documentarion.
Now all my 6 projects are down. Clients are getting impatient. I cannot do any work and googles support is the worst.
They dont even want to understand the issue, dont know how they will solve it.

I have created AWS instance now and migrated to AWS. But i have old backups which are useless on AWS. To get the latest backups i need google cloud instance to get started but stupid google does not have resources. How hard it is to add 1 CPU and 1GB RAM?

Last Monday I bought an iPhone as a little music player, and just to see how iOS works or doesn't work.. which arguments against Apple are valid, which aren't etc. And at a price point of €60 for a secondhand SE I figured, why not. And needless to say I've jailbroken it shortly after.

Initially setting up the iPhone when coming from fairly unrestricted Android ended up being quite a chore. I just wanted to use this thing as a music player, so how would you do it..?

Well you first have to set up the phone, iCloud account and whatnot, yada yada... Asks for an email address and flat out rejects your email address if it's got "apple" in it, catch-all email servers be damned I guess. So I chose ishit at my domain instead, much better. Address information for billing.. just bullshit that, give it some nulls. Phone number.. well I guess I could just give it a secondary SIM card's number.

So now the phone has been set up, more or less. To get music on it was quite a maze solving experience in its own right. There's some stuff about it on the Debian and Arch Wikis but it's fairly outdated. From the iPhone itself you can install VLC and use its app directory, which I'll get back to later. Then from e.g. Safari, download any music file.. which it downloads to iCloud.. Think Different I guess. Go to your iCloud and pull it into the iPhone for real this time. Now you can share the file to your VLC app, at which point it initializes a database for that particular app.

The databases / app storage can be considered equivalent to the /data directories for applications in Android, minus /sdcard. There is little to no shared storage between apps, most stuff works through sharing from one app to another.

Now you can connect the iPhone to your computer and see a mount point for your pictures, and one for your documents. In that documents mount point, there are directories for each app, which you can just drag files into. For some reason the AFC protocol just hangs up when you try to delete files from your computer however... Think Different?

Anyway, the music has been put on it. Such features, what a nugget! It's less bad than I thought, but still pretty fucked up.

At that point I was fairly dejected and that didn't get better with an update from iOS 14.1 to iOS 14.3. Turns out that Apple in its nannying galore now turns down the volume to 50% every half an hour or so, "for hearing safety" and "EU regulations" that don't exist. Saying that I was fuming and wanting to smack this piece of shit into the wall would be an understatement. And even among the iSheep, I found very few people that thought this is fine. Though despite all that, there were still some. I have no idea what it would take to make those people finally reconsider.. maybe Tim Cook himself shoving an iPhone up their ass, or maybe they'd be honored that Tim Cook noticed them even then... But I digress.

And then, then it really started to take off because I finally ended up jailbreaking the thing. Many people think that it's only third-party apps, but that is far from true. It is equivalent to rooting, and you do get access to a Unix root account by doing it. The way you do it is usually a bootkit, which in a desktop's ring model would be a negative ring. The access level is extremely high.

So you can root it, great. What use is that in a locked down system where there's nothing available..? Aha, that's where the next thing comes in, 2 actually. Cydia has an OpenSSH server in it, and it just binds to port 22 and supports all of OpenSSH's known goodness. All of it, I'm using ed25519 keys and a CA to log into my phone! Fuck yea boi, what a nugget! This is better than Android even! And it doesn't end there.. there's a second thing it has up its sleeve. This thing has an apt package manager in it, which is easily equivalent to what Termux offers, at the system level! You can install not just common CLI applications, but even graphical apps from Cydia over the network!

Without a jailbreak, I would say that iOS is pretty fucking terrible and if you care about modding, you shouldn't use it. But jailbroken, fufu.. this thing trades many blows with Android in the modding scene. I've said it before, but what a nugget!

Something is really fucking wrong with people in my company. They fucking calling me after 5PM on Friday when the server is down. What part of my role you fuckers don't understand. I'm not a Network Engineer and I don't have fucking access to the fucking server.

Call the fucking Network Engineer. If his not picking up his phone then that's his fucking problem not mine.

(Bang the Table) Fuck this shit

Today we updated PHP on our server. From 7.0 to 7.2
Promptly the whole server went down and none of the pages on it worked anymore. I guess we sort of forgot that there are other pages on the serve than the one we were working on...
Now everything is operational again.