Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "downtime"
devRant will be going down on Friday, July 7th around 10:30pm EDT so we can do some database maintenance and restructuring of our cluster. It hopefully won't be down for more than about 30 minutes or so, and during that time you should see our "down for maintenance" message.
If you usually use devRant while you're on the toilet (we know many do!), we apologize and suggest you try to schedule around this!
Please let me know if you have any questions and apologies for the inconvenience.47
”our PC stick isn't booting up! Come and fix it! (angry)”
”The PC are meant to boot up whenever power is delivered to them. Are you sure your TVs are powered on?”
”Yes! I just pressed the power button on both TVs and it didn't turn on the PC sticks.”
”So you can confirm the TVs are on? Can you change the input and see what happens?”
”Stop wasting my time and send someone down to fix it now! I told you it isn't working!”
”Ok, we will get someone out to you as soon as possible.”
Then a support guy drives 2 hours to their store.
When he gets there he realizes that the TVs power is connected to a light switch and they has the switch off!!!
He said ”can we turn on some lights so I can see behind the TV?” and then all the fucking TVs came on.
These are times when I fully understand the concept of “firing a customer”.
The customer sent an email saying ”the downtime for your product was unacceptable.” even after it was explained to them that the problem was them turning off the power.
These fucking idiots actually expect us to deliver products to display on TVs without fucking electricity to run them.15
First time poster here. Please be nice :)
My biggest workaround is one that's being currently deployed to 40 truck drivers (trucking company here), preventing printers being out of usage while on the road. We also have to use HP ePrint to wirelessly print documents, but that's another story for another time I guess :)
CEO asked us to install wifi printers in our 40-ish trucks which has wifi on board. However he's always picking one of the cheapest options possible, so we got consumer grade printers (Laserjet 1002w). Those printers often disconnects without getting back on the truck wifi network EVER. I have to get physically in the truck, wire the printer via USB onto my laptop and reconfigure Wifi on it with the HP Windows tool. This means lots of printer downtime, which always happens when the drivers are three timezones away from our office
Then I thought: "What if I could sniff what HP sends via USB while I (re)configure the printer, and replay whats being sent later? Our trucks all have an Android tablet with a USB type-A connector with host capability, so I could write a small app that replays the config when plugged in by the user.
Three days of hacking around later, I have a working app. By chance, HP printers (or at least those models we have) uses HTTP POST via USB, so I could easily replay the request.
Edit: the end result is that truck drivers just plug the printer to their tablet, press "reconfigure" in a home made Android app, printer is reconnected to the truck and they're good to go. They don't have access to the network nor know enough to debug themselves anyways14
Happiness is not getting any server issue/downtime notifications while you're outside on a bridge watching fireworks ❤😊5
Observed my bf spending at least a half hour browsing devRant in bed, so asked him what he'd do if devRant didn't exist anymore.
His simple reply?
May God help him tomorrow for the scheduled downtime... ;)9
So long ago, I once came across this downtime message and saved it for future reference.
This community is the perfect place to share this.
P.S.: Only Indians will get train track reference. LOL56
When you move a bunch of cables you haven’t touched in a while, and underneath you find this 🥳
Guess I know what I’m taking to work on Monday.2
Yesterday I fucked up big time.
First time in my career (I’m 23).
I just started working this week at a new company startup that had no programmers before me. They have a bunch of websites under their control that were on all different hosting solutions, and we decided to move them all to AWS.
I moved a few and was managing the folder rights on the server.
What happened next made my heart skip a few beats.
Bear in mind I’m not an expert in Linux.
I wanted to chmod to the folder I was currently in, and typed ‘sudo chmod -R 770 /‘ thinking for a while that the ‘/‘ would do it on my current dir.
Fuck. As I saw what was happening I pressed ctrl + c as fast as I could. But the damage had been done.
Fast forward a couple hours I deleted the broken instance, and created a new one from scratch. Had to do everything again but managed to do it in just a couple hours, moving as fast as I could without making such stupid mistakes again.
I was honest about it from the first minute it happened, and told my boss right away that I fucked up and had to start over, with a couple of hours of downtime.
Luckily not much was lost and I took a snapshot right after I was finished and will look into auto backups next week.9
This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.
Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.
And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.
And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."
We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.
And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.4
Scheduled devRant maintenance - I'm going to be upgrading some infrastructure later and there will be some downtime, probably about 15 minutes, around 9pm EDT. Apologies for the inconvenience and devRant disruption :) It will help with working towards an even more stable service in the future.
Feel free to let me know if you have any questions!11
Hey everyone - apologies for the downtime earlier today. Our host is having a lot of issues and we're working to keep everything up through it.
On that note- there might be a little more down time tonight as they are trying to fix something and we might need a few server restarts. I will keep everyone updated and thanks for bearing with us!20
Writing more infrastructure than product.
Look, my application requests and transforms data from a single external API endpoint, it's just one GET request...
But I made an intelligent response caching middleware to prevent downtime when the parent API goes down, I made mocks and tests for everything, the documentation is directly generated from the code and automatically hosted for every git branch using hooks, responses are translated into JSONschema notation which automatically generate integration tests on commit, and the transformations are set up as a modular collection of composable higher order lenses!
Boss: Please use less amphetamine.5
'Twas the night before deployment and all through Hipchat
Not an intern was stirring, or a CSM, at that,
The feature design was approved with care,
The business case and user feedback were there.
The sales team was home watching the game in their beds,
Making "Presidents Club" danced in their heads,
The CEO mused over valuation cap,
And how the competition could take a dirt nap.
When in pager duty there arose such a clatter
The night devs logged in to see what was the matter.
Away to the server logs they flew in a flash
To see what had made the primary DB crash.
The pristine backups in the freshly pushed cloud
Gave joy and relief to the now-anxious crowd.
When what, on the CDN status page should appear,
But holly-red downtime and a DDoS severe.
With little fanfare and at speeds none could detect,
In toddy-wreathed glow appeared the Senior Architect!
In mere milliseconds the commands they came,
Were whistled, and shouted, and called out by name:
"Now Traceroute, now Nmap, now Dig, now loopback Localhost!
On Makefile, on Linker, on Tar, on avocado toast!
To the bottom of the Btree, to the root CA,
Now compile! Compile! And restore away!!"
As JS frameworks of the day away fly ,
When the zeitgeist forgets them in the race to the sky,
So up to the network backbone they flew,
The Architect on her PDP-10, with her Dynabook too!
And then in a twinkling, the emergency team
Saw the DDoS reverse, and the bits surged downstream
Packets, like snowflakes, flashed before our wint'ry eyes,
At 1024 GB/s, a wonderland of surprise!
Her eyes, how they twinkled! Her shell scripts-- concise!
Her bytecode so polished it ran on any device!
Her soldering iron swung freely from her hip,
From a hot whiskey toddy she enjoyed a long sip.
She spoke not a word but went straight to her work,
And synced the prod DB where the tables went berserk.
And laying her spanner aside of her nose,
And giving a nod, to obscurity she rose!
She sprang to her PDP, to her mainboard gave a knock,
Rebooted the magnets and refreshed the clock.
She exclaimed as she vanished, giving us her last jabs,
"Remember to always use spaces, not tabs!"12
Years ago we deployed this system with a SQL DB on a separate windows server.
Every now and then we had error messages saying that the system could not connect to the db. It was going on for about 5 minutes or so and then the db was up again.
We built a bunch of fallback logic to handle it gracefully.
Then one day one of the guys was in the "server room". It was not a real server room but like a dedicated office in another building.
He saw how the cleaning lady came in, unplugged the server's cable from the wall socket and plugged in the vacuum cleaner...6
To save server cost and developers' productivity, devRant should have an intentional downtime of 3 to 6 hours daily :311
Things that never happen
Customer: I really am happy with the service. The 99.999% availability is great. I completely understand that downtimes are necessary to keep the system up to date....1
--- UK Mobile carrier O2's data network vanishes like a fart in the wind ---
One of the largest mobile carriers in the UK; O2 has been having all manner of weird and wonderful problems this morning as bleary eyed susbcribers awoke to find their data services unavailable. What makes this particular outage interesting (more so than the annoyingly frequent wobblers some mobile masts have) is that the majority of the UK seems to be affected.
To further compound the hilarity/disaster (depending on which side of the fence you're on), Many smaller independent carriers such as GiffGaff and Tesco Mobile piggy-back off O2's network, meaning they're up the stinky creek without a paddle as well. Formal advice from the gaseous carrier is to reboot your device frequently to force a reconnect attempt, Which we're absolutely sure won't cause any issues at all with millions of devices screaming at the same network when it comes back up.
Issue reports began flooding DownDetector at around 5am (GMT), With PR minions formally acknowledging the issue 2 hours later at 7am (GMT) via the most official channel available - Twitter. After a few recent updates via the grapevine (companies involved seems to be keeping their heads down at the minute) Ericsson has been fingered for pushing out a wonky software update but there's been no official confirmation of this, so pitchforks away please folks.
If you're in need of a giggle while you wait for your 4G goodness to return, You can always hop on an open WiFi network and read the tales of distress the data-less masses are screaming into the void.6
I have an interview on Thursday for a job I've been doing for the past 9 months - I bloody hope I get it!
I'm currently classed as an 'Apprentice' but have been doing the sole job of the Developer after he left a week before I started.
The only differences between the two roles is the pay difference and title (just about double my current rate).
I've started to produce documentation and processes for rolling upgrades to our application without downtime which is something they're big on.
Public sector for you, it took 9 months for a replacement...8
PSA: "sudo apt-get remove nginx" doesn't actually remove nginx. It will still continue to run and block port 80 on every reboot.
Until you run sudo apt-get autoremove, nginx-core and others still remain.
And that's how twenty seconds of scheduled downtime turns into 10 minutes.
Two days ago...
I was happy, building out the network in a new lokation.
Suddenly my phone just doesn't want to stop ringing, from all the other lokation calling in that they can't connect to HQ.
Then HQ calls, we don't have internet, nothing works. The one guy on location who has access to the server room enters and finds all the servers offline and a couple of breakers blown.
Turn on breakers, servers won't boot properly.
Me in a taxi and hurry to HQ, to help boot the servers.
Afterwards I find out that one of the bosses spilled a cup of coffee on his desk, shorting the circuit.
Apparently he is on the same breaker group as the servers!?! What the actual fuck!
At least now the other bosses are like; yeah, we need to do something about that3
To fix a shitty age old procedure in a service, I've had to deal with the worst api in history.
JUST A FEW (keep calm xzvf, keep calm) highlights:
- >60% of methods are broken.
- NO DOCS
- Application/JSON --> sends XML
- Errors are 200 OK; application/JSON `<error>an error</error>`
- "Minor changes" (ie. 3.10.13 -> 3.10.14) break a lot of endpoints
- A lot of service downtime (> 2 hours in the last 5 days)
- Shitty response times(17-49 seconds, I had to increase timeouts)
- Sometimes a request works, sometimes the SAME EXACT REQUEST DOESN'T WORK.
To keep me sane I've been listening to [this](https://youtube.com/watch/...) (Dylan Bettie - You give REST a bad name) on loop, because it describes this api perfectly.
I'LL FUCKING MURDER THE RETARDED, MORONIC AUTHOR OF THIS SHIT BY SHOWING PRINTED VERSIONS OF HIS API RESPONSES DOWN HIS THROAT UNTIL HE EITHER SUFFOCATES OR "EXPLODES"!14
Alright, so the "big e-commerce" site have ranted about a few times decided to move their site to google, because the developers blame our server to be the issue.
Well, I wish I had a couple of beer to drink while I am enjoying the downtime, servercrashes, and timeout on the site now. I hope the devs eat their own shit, because they are.
Yes, it is dumb that airports, stores, and hospitals run very outdated software, but imagine how hard it would be to upgrade all those machines, especially considering the programs that might not work well with never operating systems and the fact that staff would have to be trained all over again. Not to mention, most of these businesses and services can't afford any downtime and need to make sure that everything is compatible (so, update one PC, you have to update all of them). In theory, I am still a fan of updated systems, but then again, I have a 10 year old XP installation at home, which I've been preparing to reinstall for a year or so (don't really use that PC, but still)8
A lot of engineering fads go in circle.
Architecture in the 80s: Mainframe and clients.
Architecture in the 90s: Software systems connected by an ESB.
Architecture in the 2000s: Big central service and everyone connects to it for everything
Architecture in the 2010s: Decentralized microservices that communicate with queues.
Current: RabbitMQ and Kafka.
... Can't we just go back to the 90s?
I hate fads.
I hate when I have to get some data, and it's scattered on 20 different servers, and to load a fucking account page, a convoluted network of 40 apps have to be activated, some in PHP, others in JS, others on Java, that are developed by different teams, connected to different tiny ass DBs, all on huge clusters of tiny ass virtual machines that get 30% load at peak hours, 90% of which comes from serializing and parsing messages. 40 people maintaining this nightmare, that could've been just 7 people making a small monolithic system that easily handles this workload on a 4-core server with 32GB of RAM.
Tripple it, put it behind a load balancer, proper DB replication (use fucking CockroachDB if you really want survivability), and you've got zero downtime at a fraction of the cost.
Just because something's cool now, doesn't mean that everybody has to blindly follow it for fucks sake!
Same rant goes for functional vs OOP and all that crap. Going blindly with any of these is just a stupid fad, and the main reason why companies need refactoring of legacy code.15
So yesterday I said to my private laptop update and shutdown...
Fast forward to this morning. Hell breaks loose. Have to fix it asap! We have downtime. But fucking windows update!!!
You fucking peace of shit should have done this yesterday. And why does it have to take so long.11
First company I worked for, built around 40 websites with Drupal 7...in only a year (don't know if it's a lot for today's standards, but I was one guy doing everything). Of course I didn't have the time to keep updating everything and I continually insisted to the boss that we need more people if we are going to expand. Of course he kept telling me to keep working harder and that I "got this". Well, after a year a couple of websites got defaced, you know the usual stuff if you've been around for some time. Felt pretty bad at the time, it was a similar feeling to having your car stolen or something.
Anyways, fast forward about 2 years, started working on another company, and well...this one was on another level. They had a total of around 40 websites, with about 10 of them being Joomla 1.5 installations (Dear Lord have mercy on my soul(the security vulnerabilities from these websites only, were greater than Spiderman's responsibilities)) and the others where WordPress websites, all that ON A SINGLE VPS, I mean, come on... Websites being defaced on the daily, pharma-hacks everywhere, server exploding from malware queing about 90k of spam emails on the outbox, server downtime for maintenance happening almost weekly, hosting company mailing me on the daily about the next malware detection adventure etc. Other than that, the guy that I was replacing, was not giving a single fuck. He was like, "dude it's all good here, everything works just fine and all you have to do is keep the clients happy and shit". Sometimes, I hate myself for being too caring and responsible back then.
I'm still having nightmares of that place. Both that office and that VPS.
My first job was actually nontechnical - I was 18 years old and sold premium office furniture for a small store in Munich.
I did code in my free time though (PHP/JS mostly, had a litte browsergame back then - those were the days), so when my boss approached me and asked me whether I liked to take over a coding project, I agreed to the idea.
Little did I know at the time: I was supposed to work with a web agency the boss had contracted to build their online shop. Only that he had no plan or anything, he basically told them "build me an online shop like abc(a major competitor of ours at the time)"
He employed another sales lady who was supposed to manage the shop (that didn't exist yet). In the end, I think 80% of her job was to keep me from killing my boss.
As you can imagine, with this huuuuge amout of planning and these exact visions of what was supposed to be, things went south fast and far. So far that I could visit my fellow flightless birds down in the Penguin's republic of Antarctica and still need to go further.
Well... When my boss started suing the web agency, I was... ahem, asked to take over. Dumb as I was, I did - I was a PHP kid and thought that Magento, being written in PHP, would be easy to master. If you know Magento, you know that was maybe the wrongest thing I ever said.
Fast forward 3 very exhausting months, the thing was online. Not all of it worked yet, but it was online and fairly secure.
I did next to everything myself, administrating the CentOS box the shop was running on, its (own) e-mail server, the web server, all the coding required for the shop (can you spell 12 hour day for 8 hour pay?)
3 further months later, my life basically was a wreck, I dragged myself to work, the only thing I looked forward being the motorcycle ride home. The system worked though.
Mind you, I was still, at the time, working with three major customers, doing deskside support and some admin (Win Server 2008R2 at the time) - because, to quote my boss, "We could not afford a full time developer and we don't need one".
I think i stopped coding in my free time, the one hobby I used to love more than anything on the world, somewhere Decemerish 2012. I dropped out of the open source projects I was in, quit working on my browser game and let everything slide.
I didn't even care to renew the domains and servers for it, I just let it die without notice.
The little free time I had, I spent playing video games and getting drunk/high.
December 2013, 1.5 years on the job, I reached my breaking point and just left, called in sick at least a week per month because I just could not see this fucking place anymore.
I looked for another job outside of ALL of what I did before. No more Magento, no more sales, no more PHP. I didn't have to look for long, despite what I thought of my skills.
In February 2014, I told my boss that I quit. It was still seven months until my new job started, but I wanted him to know early so we could migrate and find a replacement.
The search for said replacement started in June 2014. I had considerably less work in the months before, looks like he got the hint.
In August 2014, my replacement arrived and I got him started.
I found a job, which I am still in, and still happy about after almost half a decade, at a local, medium sized ISP as a software dev and IT security guy. Got a proper training with a certificate and everything now.
My replacement lasted two months, he was external and never really did his job - the site, which until I had quit, had a total of 3 days downtime for 3 YEARS (they were the hoster's fault, not mine), was down for an entire month and he could not even tell why.
HIS followup was kicked after taking two weeks to familiarize himself with the project. Well, I think that two weeks is not even barely enough to familiarize yourself with nearly three years of work, but my boss gave him two days.
In 2016, the shop was replaced with another one. Different shop system, different OS, different CI. I don't know why and I can't say I give a damn.
Almost all the people that worked at the company back with me have left for greener pastures, taking their customers (and revenue) with them.
As for my boss' comments, instructions and lines: THAT might not be safe for work. Or kids. Or humans in general. And there wouldn't be much left if you put it through a language filter...
Moral of the story: No, it's not a bad thing to leave a place if you're mistreated there. Don't mistake loyalty with stupidity!
And, to quote one of my favourite Bands: "Nothing matters when the pain is all but gone" (Tragedy + Time by Rise Against).8
I was part of a on-call rotation. We had ~800 microsites with decent traffic on this one box, because that's a good idea...
One day the box was experiencing kernel panics and causing core dumps. After exhausting every possiblity I decided it was time to restart the box:
sudo shutdown now
Missed the -r and the box was not accessible remotely. Had to wait for someone at the data center to terminal in.
Downtime was ~2 hours.
This was caused by a crontab that automatically ran apt-get update & apt-get upgrade... Also made by me... None of this should have worked or allowed to be done!
Microsoft Azure down on a Friday at home time? Whoever tripped over that cable is probably trying awfully hard to slink out of the building unnoticed right about now2
Subject of message: "Important: New feature for all 000webhost users." Thanks, 000webhost, an hour of downtime a day was exactly the feature I wanted to be implemented. P.S. if you have OCD, don't look at how many unread messages I have10
I fucking hate stupid accountants!
Yesterday we went to a customer to talk to the accountants because we want to remove one of their unused PC's in the office.
First, just the way they think (and talk if) they are the most important and it's absolutely critical everything works 100%. I see they are important but not 100 times more important than everybody else!
They called us their EDP-guys (EDV in German, that's the translation I found). That insulted me a bit. I'm rather called IT-guy, I don't know anything about the fucking EDP systems nor want I to. I'm there to make sure the hardware works. But whatever, fine, call me what you want.
Then they straight up threatened us, because their work is so important, they can't afford to have downtime in their systems. They don't really care, but the bosses of us both do and if we fuck up they (the bosses) will hold us responsible. There is a fucking update for your piece of shit software (datev)! I don't do the update, I'm just responsible that the update can be deployed on the hardware. I'm not responsible if this update fucks your system and frankly I don't care!
I could tell them all of this but they won't listen. They always talk in this patronising arrogant voice, because they are so important and we better don't fuck up the update.
I'm there to help. I don't want downtime for your systems. I want you to work with our systems the best you can.
But fuck you, I hope the server burns down!13
I'm in a slack channel with our fellow devs as a side chat for downtime.
We get to talking about coding, and then it led to the tools of coding then it led to OS debate.
I said I use Windows because it's what I work the fastest with. Then out of nowhere, they start flaming me, calling me random boy and there's really nothing I can do about it, because the "elitests" keep piling on the list of why Windows is bad.
Why is it that when I go into a coding server and I link a Windows solution to said problem, I get flamed for it?
It's honestly like I can't use software without someone trying to dox me (even if that is a overstatement)25
I was an oilfield machinist for about 10 years. During downtime I'd read blogs and books on my phone. Eventually I wrote an app to manage parts drawings and CNC programs for my shop. Any time I came across a package or pattern I didn't understand I'd pursue it relentlessly. CodeWars and reading other people's code got me a long way. Now I've got a job in silicon valley and things are pretty sweet.
Just now I was reading on https://pve.proxmox.com/wiki/... about high availability. Now my Proxmox VE is just a tower (which happens to have ECC memory) that's stored in my storage room (and which is mostly used for experimental and home server purposes). But my mail servers.. those have been made with high availability in mind. Most importantly, I've made their services entirely redundant (but within the same datacenter). And when they have updates, I apply updates to one, reboot, see if it didn't break something and then do the same to the other server after the first one came up again. So no downtime whatsoever.
If memory serves me right, I think that I've been able to maintain these servers for the last year without any downtime at all (I reboot them every month to apply new kernels but they haven't both been simultaneously down at any moment). Does that make them High Availability? My interventions regarding their availability have been rather trivial. Is it really that hard..?4
Well today we got to test our system to the extreme and I'm pleased to say it passed. Major power surge followed by a black out. UPS for all networking and servers kicked in without missing a beat and the standby generator outside about 45 seconds later. After explaining to users how to turn on their computer (😑), we were able to get everyone working again in about 5 minutes. Lasted three hours without power from the grid without any client downtime1
It seems Microsoft have taken my critique of Azure last week to heart. Last week, I insinuated that perhaps hometime on Friday was an inconvenient time of day for major downtime.
This week they've rectified their previous fault, and in their great mercy have decided to take down service at the busiest time of the day on Thursday instead.2
Not sure if that qualifies as prank...
Had an pretty incompetent CS teacher and used to simply unplug her PC when we had enough of her shit. Usually took her about 45mins to figure out what was wrong with her PC and another 5 of ranting why we'd do that. Eventually she started to check the cable first which reduced the ”downtime” to about 15mins.
However, we soon started to flip the power switch at the back of her machine instead. She never figured that out and called IT several times to fix it.
Thinking about it, it's probably worse than a prank 😅7
Not only are you not your job, your job is not worth taking home with you; unless it's actually your company, leave it in the office. You can love your job and still have days when you hate it, or days when you'd rather be doing anything else; that doesn't mean you don't still like what you do.
As a profession we can all be obsessive and not take the time out that we need, so make special effort to do so, even if that just means you're working on a personal project instead. Your brain, and partner, will be glad that you did. Whether you like to admit it or not, everyone needs downtime.1
Before, when you bought $3k Cisco router you got the box that will run indefinitely as long as it has power in about any environment you can go to install it; with or without fans, it had more important business than to give a shit about such things.
Now, when you buy $500k Cisco box you get a over-engineered chassis with 5 separate fan modules with their own firmwares, self tests, watchdog timers and shitload of bugs. It's a fucking fan, it should spin, not do quantum chromodynamic simulations.
Next rant could probably be how Cisco's switch from monolithic to modular linux arhitecture (in order to reduce downtime) turned into having time bombs just waiting to do some crazy shit.2
A guy who had the same nationality as the enterprise we were working for was promoted from JUNIOR js developer to UX/UI coordinator for the entire department just because he was 2 year older than me (26 vs 28). Literally he was a junior dev and went to that.
One day he was accusing me of writing a piece of code which led prod to downtime. I was in the office, he was in another country with our manager and technical director next to him and we were talking over internal conference system. I shown git history + his name + his code and he was saying ‘that’s not true!!!’.
I couldn’t resist and I began to yell something like ‘You fucking fuck piece of shit cocksucker...’ for 5 minutes. Since that day i was the god on my project for UI/UX side.
Even now he is in the same place on the same position...
PS: more stories to come with this guy7
Being a sysadmin can be the most frustrating thing ever, but it's worth it for those moments when you feel like an absolute ninja.
Switched from single threaded gevent server to an nginx configuration, added ssl, and setup a reverse proxy to flask socketio, all with less than 10 minutes aggregate downtime. On the prod server. \o/3
What a bunch of cunts.
It's sad how they keep applying restrictions to everything. Two years ago, there were no restrictions. Now:
Max one website, random account locks if you ever get actual visitors, no support unless you're premium, max 5 simultaneous clients, one hour sleeptime a day, some "random" disk full errors or internal server errors and at least two hours downtime a day.
My superpower would be the ability to split myself into multiple copies of myself so that I could function as an entire dev company on my own whilst learning the skills to do so without downtime because a copy of you could sleep while another works.
Of course the copies would share their knowledge and can merge back when needed10
Work it harder, make it better
Do it faster, makes us stronger
More than ever, hour after hour
Work is never over
Oh the rant? This is gonna be a long one, and that was one of the lyric that stuck in my head for the past 3 days, Alive live album 2007 was glorious,
TL/DR, note to self, ALWAYS ask for documentation, and written evidence of any task & stories before start anything next time,
To start, death march was over, my team and I got some downtime(less work) for the past week, some of the guys were still busy with their respective stories (bugs, etc) but all in all it was not as much load as the past month before that,
It was peaceful and quiet, I was working bugs, some enhancement here, some enhancement there, it was nice for a change, until
One of the PO came by, asking if there's any spare dev, my team's tech lead suggests me as it's gonna be a front end work on web, good old HTML and CSS, and it's supposed be a task, not a story, I thought it's gonna be nice for a change, so I agree, the PO took me to the lead developer in charge,
Both of them briefed me, it's gonna happen in the apps (it's web alright, but in React Native, so no HTML and CSS for me), i was tasked to create 2 forms, and connect it with the microservice, "okay" I thought to myself,
Me: "Do we have the design for this?"
TL: "no but someone already made similar page you can either reuse it, copy paste it, whatever"
Me: *my bullshit senses are tingling, "that's one, what about the second"
PO: "I think the other team already made similar one too, lemme check, ah here it is, if it's all good I'll make the story for this"
TL: "okay, so there's that, can you start right away?"
Me: *tingling intensifies, "wait what about the flow"
TL: "it's simple, I will do this then it takes to your page, then this and that and that, and you do this and should be done, the MS is all there you just need to make the front end and connect it with MS, good?"
Me *hmmm intensifies, "ok let me check the available component first and see what I can do"
TL: "great, can you finish it by tonight?"
Me: *what the fuck intensifies further, it's fucking 6 PM, "nope, I don't think so, there's always complications when handling forms, not to mention copy pasting stuff"
TL: "it's ok, at least finish one by tonight"
Me: "..., we'll see"
Trying to reuse the form for the first page is no good, I had to duplicate the components, first one is working fine, the second template is almost done,
TL: "dude how's it going? It's been a whole day"
Me: "first one's finished, second's underway"
TL: "can it be don.."
Me: "today? Nope"
TL: "dood, is it done yet? It's been two days, what's left on the progress?"
Me: *for fuck's sake, "I just need to figure out how to connect between the first and the second and it's done
TL: "okay cool"
[different PO came by]
PO #2: "hey, hi, sorry, what's the progress on this?"
Me: "uhh hi, just need some unit test and it should be good for PR"
PO #2: "cool, keep me posted"
I finished, about to put a PR, I need the story ticket, I asked the PO and the TL
PO #2 gave me the ticket for the backend work which the TL is working on, no mentions or specs for th front end
TL gave me a ticket which was just created shortly the moment after I asked for it, only title and no description
Me: *shit, this will be interesting
Sure enough, the tester who is doing smoke test on my branch threw a fit, where's the spec, where's the design, where's everything, how is it supposed to work, the flow, the typecheck, translations, etc
The news came to be heard by the design & product team, and they came by, apparently no one knows what the front end was supposed to do, all everyone know is just the back end part,
In the end, it goes apeshit, everyone are confused, everyone have different understanding of the story, but at least what I've done doesn't went to ashes, after explaining everything to the design team, they decided that let it be, but there's might be some minor changes on the layout,
And then I began to understand why this TL and PO #1 had somewhat bad rap, while I was stuck with PO #2 trying to explain the relation between my task and the backend user story,1
Docker swarm. All i want is a 'zero-downtime' system and everytime i try to set it up there's three damn things missing. Load balencer, service updater, and a good distributed storage. I finally got pissed off and am working on those but fuck it's been how fucking long docker has been out why the hell somebody else hasn't done this yet.3
When you walk in at work in the morning, hoping for a quiet Friday at work, but nope. I check our WP monitoring and see that half our sites aren't responding. Well fuck. Turns out that the firewall service we use to protect our sites experienced a massive DDOS attack. So the service we use to keep our sites safe ironically caused downtime. Me, our Devopser and another dev spent the entire morning bringing the sites up again and bringing the customers up to date. FFS, I need more coffee.2
Go to meetups and talk to people. Give presentations at meetups if you can. Get involved in community projects. Love coding. Use your downtime to study new stuff.
When talking to potential employers be positive and enthusiastic about your technology.
EDIT: Oh, a few more. Don't seem desperate for a job. Without saying anything, potential employers should feel like you have other offers and they're being evaluated by you. Ask questions about their company if you get an interview.
Try to give off an air of being in control and having a number of choices in your carreer (even if you're living off ramen every day).
The pressure should be on companies to hurry up and snap you up before another company does.
Be honest but a little spin won't hurt.
The company I work for used to be hosted on 3dcart. One day the site went down and their support couldn't tell us why. After over 24 hours of downtime they restored service but left 5 days of all records and customizations across the entire store, from the DB to the damn templates. Their support apologized for the outage blaming the disaster on a combination of hard disk failure and a bad update to their backup script. They were not willing to assist us in any way. We were forced to manually enter 5 days of orders (which gave them new order numbers and caused more problems), products and template changes, with order data coming from an internal email which was luckily CC'd on the order confirmation email. Thank God for whoever setup that CC, it saved our asses. In the end it cost our company thousands of dollars and 3dcart never composited us in any way.2
Had a shoulder operation, and currently unable to move my arm. Getting pretty frustrated with being unable to move and feeling useless.
A mate just explained that I was basically patching my body. After a little downtime I'll be stronger and better.
Somehow, with this logic/analogy, it seems sensible and acceptable!4
I just spent more than an hour helping my dad getting into his web mail again.
The provider he used as until now has merged with Vodafone, they also merged the webmail client. And ffs they made a mess of it. You had to request a new activation code and enter that into a one of three different forms all on one page with seriously confusing information as to which one you needed to use to reregister. After finally finding the right one and you can log in, they keep trying to get direct you to the Vodafone mail, not the original provider's webmail, which they kept separate. An all-in-all awful user experience. They had a downtime of about 10 days during the migration, and I still find bugs like endless loops while trying to load some sub-pages.
I can't help but wonder what all went wrong on the dev and devops side here. The downtime was awfully long, even for such a complicated merger. Was the dev team understaffed, or under qualified, or both? I also got my doubts about whether any UX staff was involved at all. The only thing I know is that as soon as I'm home for Easter, I will switch my dad over to a different mail client. Thanks Vodafone for one hour that I'll never get back.1
The worst part of moving to a new apartment, besides the heavy lifting, is the Internet downtime, before it's reestablished 😒2
Client: my website is down
Support: can you just google my ip and let me know your IP
Client : OMG google is down!!! Oops router was'nt plugged.
**Client is on call just incase you wonder :p
Concerning my last post on the two Commodores, (https://devrant.com/rants/963917/...) here's the great story behind the boxed one.
So at the place where I interned over the summer, I helped the tech dept. (IT herein) move to a new bldg. We had to dismantle most of the network infrastructure stuff, so we were in the server room a lot. First day on the job, Boss shows me server room, I'm amazed and all because this is my first real server room lol.
We walk around, and there's a Commodore 64 box on a table, just kinda there. I ask, "Uh, is that actually a C64?" B: "Yeah, that's E's." Me: "E?" (name obfuscated) B: "Yeah, E's a little crazy." Me: "Is it actually in there?" B: "Absolutely, check it out!" *opens box and sees my jaw drop* Me: "Well, alrighty then!" So that lingers in my mind for a while until I meet E. He is a fuckin hilarious guy, personifying the C64, making obscure and professionally inappropriate references. Everyone loves him, until he pranks them. He always did.
We’re in the server room, wiping some Cisco switches or something, and we have some downtime, so I ask him about the 64, and he's like "Yeah, I haven't had time to diagnose her issues much. If you want her, go ahead, see if you can make it work!" Me: "You're kidding, right?" E: "Nah, not at all!"
That day I walked out with a server motherboard, 2 Xeon CPUs and some RAM for the server (all from an e-waste bin, approved for me to take home from boss) and a boxed C64. Did a multimeter test on the PSU pins, one of the 9vAC pins is effectively dead (1.25v fluctuating? No thanks.) but everything else is fine except for a loose heatsink and a blown fuse in each C64. Buying the parts tonight. I wanna see this thing work!1
Am I the only one who gets intimidated when shit its roof?
Yesterday, during crucial business hours, one of the major OMS db column type got overflowed. Caused around 30 mins downtime and then later, pool of all connections with high concurrent requests flushed down stream which caused thunder herd.
One by one.. all services went down; Fucking java service couldn't even start because of load..
This is the moment I fell in love with GoLang. We shard request using GoLang service, it just started and picked up the load beautifully..
At the end, it is around 6 millions business loss, but a good lesson learned :)
SSH attack today. I asked my colleague if a rule in the firewall would block that. He tells me that yes and when I look, the rule is open for the SSH. 2 hours ouf downtime ...1
Does anyone else here play Player Unknown's Battlegrounds and is as blown away at how fucking terrible their dev team is? I mean, I get the game is a early access and that comes with a certain level of instability, but over the last few weeks I've gotten increasingly triggered about it.
Here's a small list of what they've managed in the last few months or so.
1) Upgraded their servers and somehow managed to make the game so laggy it's unplayable.
2) Changed the DirectX requirements secretly (not even in the patch notes), but not their minimum specs, leaving people with lower end graphics cards thinking they can run the game only to get errors on launch.
3) Introduced a weekly and monthly patch system, then released patches that consistently broke the game and made it unplayable until hotfixes were pushed. They then announced they will no longer be releasing regular patches after managing to break their internal testing system. Not that they were using it anyways.
4) Major bugs and memory leaks that everyone knows about have not been fixed in months and have been instead ignored to spend time on adding more important things, like adding car horns. The fucking broken jump mechanic has gotten me killed so many fucking times because it's impossible to jump at all but yes, what I needed was a horn.
5) Downtime in the middle of the fucking day, with no warning, for things like server maintainence.
6) Promising no microtransactions in the game until full release and then, surprise, adding them anyways.
There's so much more that these fuckers have done but now I'm far too triggered to go on. Anyone else agree with me or am I just raging solo? 😂4
Show stopper issue raised,
10 minutes to home time,
Critical service goes down,
Current facial expression2
TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.
I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:
-migrated their 2007 Exchange server to Office 365
-Upgraded their ailing Windows server 2003 based architecture to 2012R2
-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)
-Deployed a vCenter server to manage said ESXi servers easier
-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality
-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.
-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time
-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies
-Documented the entire old network and now the new one as I’ve been upgrading this
-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.
-Performed network scans and rewrote firewall exceptions to tighten security
There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.
I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.
I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.
I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?
Thanks in advance!6
Node: The most passive aggressive language I've had the displeasure of programming in.
Reference an undefined variable in a module? Prepare to waste your time hunting for it, because the runtime won't tell you about it until you reference a property or method on the quietly undefined module object.
Think you know how promises work? As a hiring manager, I've found that less than 5% of otherwise well-experienced devs are out of the Dunning Kruger danger zone.
Async causes edge cases and extra dev effort that add to the effort required to make a quality product.
Got a bug in one of your modules? Prepare yourself for some downtime because a single misplaced parentheses can take out the entire Node process, killing unrelated pages and even static file hosting.
All this makes for a programming experience that demands much higher cognitive load, creates more categories of bugs, and leads to code bloat/smell much more quickly than other commonly substituted languages.
From a business perspective, the money you save on scaling (assuming your app is more compute efficient under Node) is wasted on salaries and opportunity costs stemming from longer dev time, more QA, and more frequent outages.
IMO, Node is an awesome experiment, a fun language, a great tool for specific use cases, and a terrible fucking choice for an entire website.8
I know it's a weekend, but there are a few lifeless developers who work even on Sundays. Stackoverflow.com is down :/6
So it's been 22 days and were still going strong without a single day's downtime! https://devrant.com/rants/1147150/...2
"data randomly disappeared and caused us downtime. I fixed the problem by replacing the missing data"
"I don't see a problem. the data is there"
of course it's fucking there. I just put it back, but that doesn't change the fact that downtime happened.2
Friend of mine messaged me about sites being down, of course Im at a family dinner with no laptop or ssh keys with me so no way to fix it!!!
Should cloudflare have taken down their servers to protect their clients? Which is worse, the leak live or the downtime?1
What's up with bitbucket? For the last 2 months it's been down numerous times, obviously when you need it the most :(4
For crying out loud, no, GoDaddy, you don't just shutdown expired domain without ANY warnings. No!!! Not cool!!!5
I hope my boss learned his lesson: dd if=/dev/zero of=[hdd storing DB about VM cluster]
- is a very very bad idea...10
Met a client. He need some abc services. Client asked me what will be the sla uptime. I told him unintentionally that it will be 99% instead of 99.99% and he was like that's good. I mean seriously....?
Was he okay with the 3 days 15 hrs downtime?5
When the guy you are relying on to do an export for an app during a MISSION CRITICAL downtime exports the wrong data and drops offline... Then you find his number in an email... then you find out he is driving somewhere and will not be back at his computer for 30 minutes...
Thanks for staying up with me @joeygreen
Online java IDE suggestions?
My (non-dev) job is boring and I tend to have a lot of downtime, any suggestions?3
I write web software that gets sold to enterprise customers. A major part of the work flow is running reports that get exported as PDFs that users have to keep track of for compliance purposes. Just under a week ago, a select few reports quit printing. Once the issue worked its way through the red tape and eventually got to the point where a developer (me) could/had to look at it and pull server logs, I noticed that the report was trying to access a column that I had just created a week or so ago.
We have a six week release cycle. Six is a bigger number than one.
Turns out the production reports server was pointed at the preview environment which has a release cycle of whatever the fuck we want. To compound the problem, our operations team had a national holiday, so running reports was broken a full day before anything could be done. Then the next day, when the ops person got into the office, it took a few hours to convince them that yes this is a problem and yes this needs to be fixed.
But of course midday deployments/restarts of anything ever is out of the question. Chalk up another day of downtime. And of course we *just* sold to a new major customer.
Happy onboarding week guys.2
CTO migrated from AWS S3 and put images on server. The reason is S3 will have downtime from time to time. Should I look for another job?1
Vultr's Block Storage in New Jersey has been down all day. My Mastadon node is hosted there. I'm jonesing for my Fediverse fix!2
So I finally got full computer access 2 and a half weeks into working, and now there's "scheduled network downtime" with no indication of when it will be back up. I swear I'm never going to get anything done here
Any suggestions to work on coding (php/sql atm) during downtime while at work? I've been learning css and js (front/back) for a year while unemployed. Just got IT call centre job in highly monitored corporate environment. Have potential side programming job but need more practice.4
When you have to call the priority line of your hosting provider because the site went down and they're only available between 9.00 and 18.00.
They must be joking...
Sunday planned to Building a project VS family reunion long drive. No laptop at the moment. Beach later will just watch tutorials in my phone oh well 100 km long driving 😩
Databases and LDAP down since 1 1/2 days...so embarrassing...am i really working in an it company???
luckily there are options beside work...hello amazon, spotify, devrant...:D
if we got Server/DB issues it always takes about at least half a day to fix it! *facepalm*1
Azure, great development slots! Must have, now I can have developer, staging and production. The greatest no downtime when swapping a new server in....
Everything crashes? WTF?
OKAY, so swapping to a service that authenticates users makes the authentication part crash :/
Phew development slots ROLL BACK...
No the entire service was broken. Rolling back, all non authenticating controllers work, but the authentication never happens, so server is working, but the users cant use it. Fuck!
Delete everything. Recreate. The setting persists. WTF. Delete again, recreate, reinitialize, republish, it works as it should when tested phew.
Creating new service experiencing cant replicate. Hmm, okay must have been a glitch. Next, update, YEAH swap, no downtime!!!
*EXPLOSION* ..... RINSE AND REPEAT:/
If I had a dollar for every time a mother fucker in QA distracted me and threw me out of focus..
Nothing against QA but an unnecessary interaction costs me about an hour of downtime trying to back up to speed.
This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...
Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Infra Guy: u trying to get set of specific folder for set of specific components?
Infra Guy: bash script with cp or mv will not work for you?
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place3
Client needed their site transferred to our hosting environment with "NO downtime"... easy enough. Our provider doesn't support their configuration so they need their site rebuilt on our platform... again, no problem. Their current hosting plan expired YESTERDAY and they're running off the fumes of the grace period... now we've got a problem.3
Screamed Terraform is not a joke at coworker today.
Idiot corrupted the remote state while just trying to change the AMI of an EC2 instance for staging. I even said any amount of downtime is completely ok.
Please excuse: This is my first step into python. So consider this a beginners question:
This forked script checks a twitter page for words and sends a mail (probably using .qmail) to the owner.
If I execute this python:
"[$USER@$HOST uberspace-downtime-notify]$ python fetch.py
Traceback (most recent call last):
File "fetch.py", line 11, in <module>
ImportError: No module named html
Similar errors are fixed in this github commit https://github.com/datalib/... - but that's a more complex script and I don't quite get where the imported module is needed (on a code basis - on the logical basis all is clear)
Any idea for a guy with his first steps into python and back into programming languages since some years=5
When you have to be at work becuase it's work, but you finish all your work in 1 day regularly, and it takes QA 2-3 days to get back to you.... Massive downtime.2
Spent downtime during testing passing papers with this dude in my class working on an app. Pretty chill guy imo.
Wanted to say that the other place where developers rant about things is down for around 30 minutes now.
If you were to host a PHP website in a managed hosting, able to handle 200 concurrent users and upgrade to a better plan with no or small downtime if needed, which would be your choice?
The ability to integrate a CI/CD solution would be really helpful.
Context: We are dealing with a one-time campaign at the company and we don't plan to integrate this project into our architecture, so we looking for alternative solutions where to host it and deploy it to.4
Bloody softlayer sending notifications about expected downtime on "IMS services" (which could mean any of a great number of things), without specifying what it is, what it does or to what services or regions it is related...
Grmbl, what use is there to get a notification about unexpected maintenance if you can't even make out if you'll be affected or not!