Today my manager asked me about my research into using RabbitMQ as a backup in case Azure Service Bus ever goes down.

Me: "Good. The way we designed the framework, all we have to do is drop the DLLs into the directory, update the config, and the services will start using RabbitMQ."
Mgr: "Excellent. Probably should be looking into using RabbitMQ as a permanent replacement for Azure"
Me: "What? The whole reason we moved to Azure was to eliminate the problems with having an on prem service bus. Since we've switched, there has been zero downtime."
Mgr: "That's what VP-Joe is afraid of. If Azure ever goes down, he won't know how to explain Azure to the president as to why we're not taking orders or can't ship packages."
Me: "That makes no sense. What did VP-Joe tell the president when a database goes down or a server mis-configuration?"
Mgr: "President understands internal outages, its just the whole 'cloud' thing he doesn't understand."
Me: "Um..then VP-Joe needs to explain it to him?"
Mgr: "The decision has already been made. Are you on board? Lets look at this move as a cost savings."
Me: "You mean the $10 a month? How much hardware will we need to support RabbitMQ?"
Mgr: "Yea, nobody probably thought of that."
Me: "I'm on board with whatever decision, but I'd like a little more than VP-Joe being afraid of the president."
Mgr: "I'm sure its not being afraid."
Me: "..."
Mgr: "OK, lets wait and see if VP-Joe forgets about this and moves on to something new."

Yesterday I fucked up big time.
First time in my career (I’m 23).

I just started working this week at a new company startup that had no programmers before me. They have a bunch of websites under their control that were on all different hosting solutions, and we decided to move them all to AWS.
I moved a few and was managing the folder rights on the server.
What happened next made my heart skip a few beats.
Bear in mind I’m not an expert in Linux.
I wanted to chmod to the folder I was currently in, and typed ‘sudo chmod -R 770 /‘ thinking for a while that the ‘/‘ would do it on my current dir.

Fuck. As I saw what was happening I pressed ctrl + c as fast as I could. But the damage had been done.

Fast forward a couple hours I deleted the broken instance, and created a new one from scratch. Had to do everything again but managed to do it in just a couple hours, moving as fast as I could without making such stupid mistakes again.

I was honest about it from the first minute it happened, and told my boss right away that I fucked up and had to start over, with a couple of hours of downtime.
Luckily not much was lost and I took a snapshot right after I was finished and will look into auto backups next week.

One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.

On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).

Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.

I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".

Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.

There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.

Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).

I've found and fixed any kind of "bad bug" I can think of over my career from allowing negative financial transfers to weird platform specific behaviour, here are a few of the more interesting ones that come to mind...

#1 - Most expensive lesson learned

Almost 10 years ago (while learning to code) I wrote a loyalty card system that ended up going national. Fast forward 2 years and by some miracle the system still worked and had services running on 500+ POS servers in large retail stores uploading thousands of transactions each second - due to this increased traffic to stay ahead of any trouble we decided to add a loadbalancer to our backend.

This was simply a matter of re-assigning the IP and would cause 10-15 minutes of downtime (for the first time ever), we made the switch and everything seemed perfect. Too perfect...

After 10 minutes every phone in the office started going beserk - calls where coming in about store servers irreparably crashing all over the country taking all the tills offline and forcing them to close doors midday. It was bad and we couldn't conceive how it could possibly be us or our software to blame.

Turns out we made the local service write any web service errors to a log file upon failure for debugging purposes before retrying - a perfectly sensible thing to do if I hadn't forgotten to check the size of or clear the log file. In about 15 minutes of downtime each stores error log proceeded to grow and consume every available byte of HD space before crashing windows.

#2 - Hardest to find

This was a true "Nessie" bug.. We had a single codebase powering a few hundred sites. Every now and then at some point the web server would spontaneously die and vommit a bunch of sql statements and sensitive data back to the user causing huge concern but I could never remotely replicate the behaviour - until 4 years later it happened to one of our support staff and I could pull out their network & session info.

Turns out years back when the server was first setup each domain was added as an individual "Site" on IIS but shared the same root directory and hence the same session path. It would have remained unnoticed if we had not grown but as our traffic increased ever so often 2 users of different sites would end up sharing a session id causing the server to promptly implode on itself.

#3 - Most elegant fix

Same bastard IIS server as #2. Codebase was the most unsecure unstable travesty I've ever worked with - sql injection vuns in EVERY URL, sql statements stored in COOKIES... this thing was irreparably fucked up but had to stay online until it could be replaced. Basically every other day it got hit by bots ended up sending bluepill spam or mining shitcoin and I would simply delete the instance and recreate it in a semi un-compromised state which was an acceptable solution for the business for uptime... until we we're DDOS'ed for 5 days straight.

My hands were tied and there was no way to mitigate it except for stopping individual sites as they came under attack and starting them after it subsided... (for some reason they seemed to be targeting by domain instead of ip). After 3 days of doing this manually I was given the go ahead to use any resources necessary to make it stop and especially since it was IIS6 I had no fucking clue where to start.

So I stuck to what I knew and deployed a $5 vm running an Nginx reverse proxy with heavy caching and rate limiting linked to a custom fail2ban plugin in in front of the insecure server. The attacks died instantly, the server sped up 10x and was never compromised by bots again (presumably since they got back a linux user agent). To this day I marvel at this miracle $5 fix.

Tired, sick, brain foggy, cold.

I’m trying to finish my last few specs and it totally isn’t going well.

My PM also promised me he would get the change requests for this ticket to me by today so I could work on them — as we’re moving this Friday. He did not. He made the same promise last week. Bloody useless.

Oh well.
I let him know that I wouldn’t be able to finish the feature in time if he didn’t get back to me, so... week off? :D

As if packing and moving and driving is downtime.

I do need to figure out this last spec, though. I rewrote the entire feature, and broke functionality specific to some client, and apparently it’s tricky and extremely fragile. I have no idea how it was working before, and the only person I have to ask is... grumpy and overly busy, and hasn’t looked at any of it in years. Yay!

I might just go to bed.

My first job was actually nontechnical - I was 18 years old and sold premium office furniture for a small store in Munich.
I did code in my free time though (PHP/JS mostly, had a litte browsergame back then - those were the days), so when my boss approached me and asked me whether I liked to take over a coding project, I agreed to the idea.

Little did I know at the time: I was supposed to work with a web agency the boss had contracted to build their online shop. Only that he had no plan or anything, he basically told them "build me an online shop like abc(a major competitor of ours at the time)"

He employed another sales lady who was supposed to manage the shop (that didn't exist yet). In the end, I think 80% of her job was to keep me from killing my boss.

As you can imagine, with this huuuuge amout of planning and these exact visions of what was supposed to be, things went south fast and far. So far that I could visit my fellow flightless birds down in the Penguin's republic of Antarctica and still need to go further.

Well... When my boss started suing the web agency, I was... ahem, asked to take over. Dumb as I was, I did - I was a PHP kid and thought that Magento, being written in PHP, would be easy to master. If you know Magento, you know that was maybe the wrongest thing I ever said.

Fast forward 3 very exhausting months, the thing was online. Not all of it worked yet, but it was online and fairly secure.

I did next to everything myself, administrating the CentOS box the shop was running on, its (own) e-mail server, the web server, all the coding required for the shop (can you spell 12 hour day for 8 hour pay?)

3 further months later, my life basically was a wreck, I dragged myself to work, the only thing I looked forward being the motorcycle ride home. The system worked though.

Mind you, I was still, at the time, working with three major customers, doing deskside support and some admin (Win Server 2008R2 at the time) - because, to quote my boss, "We could not afford a full time developer and we don't need one".

I think i stopped coding in my free time, the one hobby I used to love more than anything on the world, somewhere Decemerish 2012. I dropped out of the open source projects I was in, quit working on my browser game and let everything slide.

I didn't even care to renew the domains and servers for it, I just let it die without notice.

The little free time I had, I spent playing video games and getting drunk/high.

December 2013, 1.5 years on the job, I reached my breaking point and just left, called in sick at least a week per month because I just could not see this fucking place anymore.

I looked for another job outside of ALL of what I did before. No more Magento, no more sales, no more PHP. I didn't have to look for long, despite what I thought of my skills.

In February 2014, I told my boss that I quit. It was still seven months until my new job started, but I wanted him to know early so we could migrate and find a replacement.

The search for said replacement started in June 2014. I had considerably less work in the months before, looks like he got the hint.

In August 2014, my replacement arrived and I got him started.

I found a job, which I am still in, and still happy about after almost half a decade, at a local, medium sized ISP as a software dev and IT security guy. Got a proper training with a certificate and everything now.

My replacement lasted two months, he was external and never really did his job - the site, which until I had quit, had a total of 3 days downtime for 3 YEARS (they were the hoster's fault, not mine), was down for an entire month and he could not even tell why.

HIS followup was kicked after taking two weeks to familiarize himself with the project. Well, I think that two weeks is not even barely enough to familiarize yourself with nearly three years of work, but my boss gave him two days.

In 2016, the shop was replaced with another one. Different shop system, different OS, different CI. I don't know why and I can't say I give a damn.

Almost all the people that worked at the company back with me have left for greener pastures, taking their customers (and revenue) with them.

As for my boss' comments, instructions and lines: THAT might not be safe for work. Or kids. Or humans in general. And there wouldn't be much left if you put it through a language filter...

Moral of the story: No, it's not a bad thing to leave a place if you're mistreated there. Don't mistake loyalty with stupidity!

And, to quote one of my favourite Bands: "Nothing matters when the pain is all but gone" (Tragedy + Time by Rise Against).

If you think deploying on a Friday is bad try deploying on a Saturday. :,(

>Client complains about a 30 minute downtime around midnight
>Client also pays only for a single VM on a HV that they don't even own themselves

>Replies with an offer of how to make the setup more resilient, going from 1 VM to 2 LBs/FE loadbalanced through BGP, and distributing traffic through HaProxy onto 2 BE machines that in turn talk to a Postgres Cluster with RepMgr for dynamic failover.

>No reply so far

Hmmm :^)

Anyone reading these emails we are sending?

I work at a small place. A few users are using an application at our place that I develop and maintain. We all work remotely.

I announce by email to these few users a new version release of said application because of low level changes in the database, send the timeline for the upgrade, I include the new executable, with an easy illustrated 2 minutes *howto* to update painlessly.

Yet, past the date of the upgrade, 100% of the application users emailed me because they were not able to use the software anymore.

----------------

Or I have this issue where we identified a vulnerability in our systems - and I send out an email asking (as soon as possible) for which client version users are using to access the database, so that I patch everything swiftly right. Else everything may crash. Like a clean summary, 2 lines. Easy. A 30 second thing.

A week pass, no answer, I send again.

Then a second week pass, one user answers, saying:

> well I am busy, I will have time to check this out in February.

----------------

Then I am asking myself:
* Why sending email at all in the first place?

* Who wrote these 'best practices textbooks about warning users on schedule/expected downtime?'

*How about I just patch and release first and then expect the emails from the users *after* because 'something is broken', right? Whatever I do, they don't read it.

Oh and before anyone suggest that I should talk to my boss about this behavior from the users, my boss is included in the aforementioned 'users'.

Catch-22 much ? Haha thanks for reading

/rant

What a bunch of cunts.
It's sad how they keep applying restrictions to everything. Two years ago, there were no restrictions. Now:
Max one website, random account locks if you ever get actual visitors, no support unless you're premium, max 5 simultaneous clients, one hour sleeptime a day, some "random" disk full errors or internal server errors and at least two hours downtime a day.

Fuck off.

How much does your internet cost?

My stats:
Speed: 50Mbps
Bandwidth: Unlimited with no FUP
Yearly: $67.16
Monthly: $5.59
Downtime: Almost never

Note: This is for home internet plan and NOT mobile plan

A couple of years ago, we decide to migrate our customer's data from one data center to another, this is the story of how it goes well.

The product was a Facebook canvas and mobile game with 200M users, that represent approximately 500Gibi of data to move stored in MySQL and Redis. The source was stored in Dallas, and the target was New York.

Because downtime is responsible for preventing users to spend their money on our "free" game, we decide to avoid it as much as possible.

In our MySQL main table (manually sharded 100 tables) , we had a modification TIMESTAMP column. We decide to use it to check if a user needs to be copied on the new database. The rest of the data consist of a savegame stored as gzipped JSON in a LONGBLOB column.

A program in Go has been developed to continuously track if a user's data needs to be copied again everytime progress has been made on its savegame. The process goes like this: First the JSON was unzipped to detect bot users with no progress that we simply drop, then data was exported in a custom binary file with fast compressed data to reduce the size of the file. Next, the exported file was copied using rsync to the new servers, and a second Go program do the import on the new MySQL instances.

The 1st loop takes 1 week to copy; the 2nd takes 1 day; a couple of hours for the 3rd, and so on. At the end, copying the latest versions of all the savegame takes roughly a couple of minutes.

On the Redis side, some data were cache that we knew can be dropped without impacting the user's experience. Others were big bunch of data and we simply SCAN each Redis instances and produces the same kind of custom binary files. The process was fast enough to launch it once during migration. It takes 15 minutes because we were able to parallelise across the 22 instances.

It takes 6 months of meticulous preparation. The D day, the process goes smoothly, but we shutdowns our service for one long hour because of a typo on a domain name.

What is it with devs who try to bloody "cost optimise" everything to within an inch of its life when there's no reason to do so?! This ain't your personal pocket money project here. This is a real commercial app with real consequences.

Seriously, saving £100 a month might seem like a lot to you, but this is a multi million pound project we're talking about. That's bloody nothing, and no-one will care. If a Fargate spot instance restarts at the wrong time and causes downtime though, or if we need logs going back a week, and don't have them because the log retention period is a few days, then everyone will be royally pissed. All because you thought "it should be ok", or it "seemed like the right thing to do". Sod off.

The most annoying hack I've had to deal with was back when I did IT support, actually. Level 1 call center tech at the time. Apparently someone fell for a phishing email and gave out his outlook credentials. The phisher used that email account to send out another phishing email to roughly 1800 employees.

Security Operations noticed, because this guy's job didn't generally involve sending out mass-communication emails. They investigated, figured out what had happened, and opted for the nuclear option: they reset the password for EVERY SINGLE ACCOUNT that received the email. All 1800 of them. Over the weekend.

I walked into the call center Monday morning and checked the call stats, then did a double-take. There were over 300 people waiting in the queue. I almost left and called in sick. Turns out it wasn't that bad though. Annoying to reset so many passwords and having no downtime due to the full queue, but on the other hand my stats were better that day than any other, since every call was a 5-minute password reset.

I was Just college fresher who completed his Engineering. My first week in the office. And a system was provided to me, since it was support project so I was given direct access to production database.

Fresher + Production Database + Access of Admin credentials = Worst Possible Combination

So it was my night shift, I was told to update new tariff plan for our client (which was one of the largest telecom service in India) .

If someone recharges for more than 200 Rupee, that person will get 10% or 20% extra talk time. Which was only applicable for particular circle (Like Bihar and Rajasthan).

Since I was fresher, I was told to update given query from my senior employee which he shared on the shared folder. Production downtime was in the mid night, so at that time I updated that query on the production database.

Query successfully updated. I completed my night shift, went home and slept.

When I woke up, I saw my mobile it had 200+ missed calls from different locations of India. They were Circle heads of that telecom service provider who contacted me. I realized something unexpected is expecting me.

Then at that moment my team lead called me and he asked me to come office right away.

Reminding you I was a fresher, I was shivering. What have I done there?

When I reached office, I came to know that the query I updated on production bombarded.

Every person who recharged that day (duration from midnight to morning 10 AM) got 10 times or 20 times more talktime.

A part of Query was something like this where error was made:

TalkTime = RechargeAmount + RechargeAmount * 10/100; (Bihar)
or

TalkTime = RechargeAmount + RechargeAmount * 20/100; (Rajasthan)
But instead of this query, I updated below one:

TalkTime = RechargeAmount + RechargeAmount * 10;
or

TalkTime = RechargeAmount + RechargeAmount * 20;
In a span of 10 hours, that telecom service lost revenue of 6.5 crore Rupees. Thanks to recovery team they were able to recover 6 crore but still 50 lakh Rupees were in loss.

One small query, and approx 1 million dollar was on stake.

Aftermath of this incident

My Mistake:

I should have taken those queries on mail. Or, there should have been mail communication regarding this.
Never ever do anything over oral communication. Senior employee who did this denied and said he provided correct query, and I had no proof of communication.
I told them, it was me who executed that query on production. Since I was fresher, and took my responsibility of that incident. My team lead rescued me from that situation.
Lesson Learned:

Always test your query and code multiple times before you execute or Go live it on production.
Always have email communication for every action you take on production.
Power comes with responsibility. If you have admin credentials of production never use it for update/delete/drop until you are sure.
Don’t take your job lightly.
I was not fired from that Job, but I have learnt my lesson very well.

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

Have some downtime today, so since I lucked out and found some old backups (from before I used Git) of a project I was planning on revisiting, I decided to fire it up and see what I can do to get that going again.

And discovering just how much my coding style has changed since then...

[Code is in PHP, for reference]

* Virtually no documentation (whereas my current style is near-obsessive with PHPdoc blocks)
* Where comments exist, they only use // and are a full tab after the end of the line
* All assignment operators are dutifully aligned on tabs
* Have to update the entire codebase because it relies on deprecated `mysql_*` calls
* Have to flip all of the quotes throughout the codebase because I used double-quotes as my primary at the time instead of single quotes.
* Also relied on magic quotes for injecting variable content into strings
* Associative array practices varied; sometimes the names are encased in double quotes, but I just hit a block where it's all leaving it to the compiler to interpret unquoted string literals

And perhaps the most egregious so far...

* Any time we get database results back the process for cycling through them is to do `$count = mysql_num_rows($result);` (or $count2, etc.), then do a `for ($i = 0; $i < $count; $i++)` (again, or $j, $k, etc.), instead of just a simple `while ($data = $result->fetch_assoc())`

I hope my boss learned his lesson: dd if=/dev/zero of=[hdd storing DB about VM cluster]
- is a very very bad idea...

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

I'm at a loss, where do I seek guidance?

Azure, great development slots! Must have, now I can have developer, staging and production. The greatest no downtime when swapping a new server in....

Everything crashes? WTF?

OKAY, so swapping to a service that authenticates users makes the authentication part crash :/

Phew development slots ROLL BACK...

No the entire service was broken. Rolling back, all non authenticating controllers work, but the authentication never happens, so server is working, but the users cant use it. Fuck!

Delete everything. Recreate. The setting persists. WTF. Delete again, recreate, reinitialize, republish, it works as it should when tested phew.

Creating new service experiencing cant replicate. Hmm, okay must have been a glitch. Next, update, YEAH swap, no downtime!!!

*EXPLOSION* ..... RINSE AND REPEAT:/

It's winter and it's quiet. Too quiet. My shitty job has me sitting here, waiting for work to appear. I could be at home working on something dev related and fun and meaningful to the progress of my life but no, I have to be here and I have to "look" productive for the bosses. I hate this shit, it's like prison, except I get paid, so I should be thankful. I can remote into my PC at home but I already got snapped for that, now I'm paranoid and afraid to try use this shitty downtime in a productive way.

Well, guess I better go sweep the already swept floors again to maintain the illusion of "work" for my penny dripping masters.

QQ having nothing to do is worse than too much to do.

So I finally got full computer access 2 and a half weeks into working, and now there's "scheduled network downtime" with no indication of when it will be back up. I swear I'm never going to get anything done here

Let me just say:

Galera is bloody incredible. We had 2 out of 3 nodes crash, and it still managed to recover automatically with no downtime.

But let me also say

When it *does* fully crash... Data recovery is an _incredible_ pain in the arse.

Thank you, Galera. Wish more customers were willing to pay for 3 SQL nodes instead of just two while expecting minimal node downtime...

Oh let the rant time begin…

So previous post I mentioned about this dev who has resigned and how I was going to see about a Snr. position.

Management is now scrambling to figure out what to do as this dev managed all the migration to AWS etc, I know servers but haven’t got too much familiarity with AWS.

Anyways so I finally get a 1:1 with my new line manager. I ask about the position and he says they don’t know what there going to do yet. Hire a new dev in India to offset and with the same knowledge even though the guy leaving is in the U.K. Bad idea as the servers are in the U.K. so if we get downtime or the server crashes we have no one in the U.K. to reset or access to the servers. India are very cagey who gets access which is annoying to say the least even though us (three devs) in the U.K. are the principal engineering team so there looking at all options.

Anyways we have a back and fourth, we discuss some of the plans for the app, some of which we are nowhere near ready to even conceptualise as the app in its current state sucks, (ruby 2.2.6 and rails 5 but not really). Needs major refactoring and rewrite, one thing they want to do is multi tendency which again given the state is laughable.

So, as my manager is speaking my head is screaming being like “this is just going to be a massive disaster”. Then we go onto that he’s seeing what everyone’s strengths are etc. And then we get onto the upgrade and that he wants me to work on it.

Yes.. the upgrade I’ve been trying to do for the past 4+ months but I keep getting told to stop and getting pushed backed.

I’ve been told we have devOps looking into restructuring the app, not possible as how the app is written, we have India trying to multi tenant again disaster incoming as they’ll end up rushing it. Legal are going to have a field day. Every time I say the issues are the fundamentals with the app, here’s how we can sort it. In one ear out the other basically there patching the ship even though it’s still leaking.

I have so many ideas, and things I can do to improve the app and get it back to not only working order, fix the performance issues, data issues and everything else. Brick wall.

So rants ensue where I basically say I would love to do the upgrade but management gives me no time in the roadmap (we have no say in planning). At this point I’m just speaking to a brick wall.

After the meeting I have a chat with the BAs, we all have the same issues so honestly it sucks we end up ranting to each other for an hour.

I’m being under-utilised, being told do this, do that even though I’ve had two stabs but told to stop and pushed back, I know what benefits I can bring to the app with a refactoring, ideas and how to properly lead the team because honestly we’re working on an old legacy app, and management are clueless and there priorities are all wrong, the company is getting frustrated and it’s a sinking ship. They would rather patch issues without solving them and everything I say goes in one ear and out the other.

Frustrating is not the word.

Any Elasticsearch gurus here? I have a box with too many young gen GCs (one per 2 or 3 seconds), and irregular, very long old gen GCs (One per several hours, taking around a minute and freeing about 2/3's of the old gen space) -- I was thinking changing the new gen ratio from 2/3 to something like 3/4 or 4/5.

However, after reading an elastic article about settings to never touch... I'm no longer so sure...

Only other option I was considering is going from CMS to G1GC to cut back on the old gen GC time... A minute long downtime for Elastic is rather problematic.

Any thoughts? The box is rather old - running Elastic 5.6 with 20 GBs of heap, 207 shards and 306k docs.

If you were to host a PHP website in a managed hosting, able to handle 200 concurrent users and upgrade to a better plan with no or small downtime if needed, which would be your choice?

The ability to integrate a CI/CD solution would be really helpful.

Context: We are dealing with a one-time campaign at the company and we don't plan to integrate this project into our architecture, so we looking for alternative solutions where to host it and deploy it to.

I am so sick and tired of ChatGPT being down all the time! It's like the developers at OpenAI just don't care about the people who rely on this tool to get their work done. I mean, come on, it's not like we're asking for much here. We just need a stable, reliable language model that we can rely on to complete our tasks. But no, instead we have to deal with constant downtime, error messages, and other issues that make it impossible to get anything done.

And don't even get me started on the lack of support and communication from the OpenAI team. It's like they're completely oblivious to the fact that their product is causing major headaches for so many people. I mean, I understand that developing and maintaining a large language model is no small feat, but that's no excuse for the constant problems we're facing.

I'm honestly at my wit's end with this whole thing. It's just so frustrating and frustratingly frustrating to have something that should be so simple and straightforward be such a constant source of stress and frustration. If the OpenAI team can't get their act together and provide a reliable product, then they need to step aside and let someone who can do the job properly take over.

I just hope that they're listening and that they'll take the necessary steps to fix these issues and provide a service that we can all depend on. Because right now, it's just not cutting it.