Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "server down"
-
"OUR SERVER IS DOWN!!!!!!"
*ssh server*
*succesfully logged into the server*
"The server is very much up, sir."
"BUT THIS WEBSITE ISN'T WORKING ANYMORE!!!!"
Ah, so one of your websites on that server with 100s of websites on it is not working anymore. That doesn't mean that you're entire fucking server is down. Please learn the fucking difference.26 -
When someone's calls and completely loses his shit (swearing etc etc) because we HAVE TO FUCKING HELP HIM BECAUSE THOUSANDS OF PEOPLE RELY ON THIS SITE WHICH IS DOWN.
Explained him calmly that its an unmanaged server which we literally don't have access to. Keeps on calling me all kinda things and then says he'll email some login details.
Bossman walks in, collegue goes like 'yo guess what just happened with linuxxx!'
Le me explains the story.
Bossman: email me his number. I'm going to call him. You treat my employees with fucking respect.15 -
So there was an inspection from government for our bank's IT security. I gave a tour to our server and security systems. I threw all possible acronyms as much as I could remember. Inspector nodded and noted down never uttered a word.
Finally, he breaks his silence, looking at a device he points out and says "What's that ?"
I look at the device then stare at his face back again at the device and to his face I reply "That's AC, Air Conditioner".19 -
Hello everyone, found this place recently, decided to bore you with one (or many) Navy story... tech Navy story. I'll start from the end.
Little backstory: I've deployed a simple domain setup on the ship I served, nothing fancy, a server, a switch, 10 computers, all Windows (details on that at another rant). I enter the ship Monday morning, and the XO tells me that he can't access his online folders.
OK, I say, I'll get to it. I fire up my laptop, try to RDP to the server (I know, I know, burn me at the stake later) no connection. WTF? Is the service down? I try pinging. No luck. I tried pinging the switch. OK. Looking at the switch admin panel, I see the server's port is dead. "OK, probably the cable." (we have old ethernet cables)
So, I drag my ass over to the server (same room with ship comms) with the cable tester to confirm that. What do I see?
The IMBECILES had pulled the plug from the server so that they could charge their mobile phones. I literally slammed my head against the door (calming exercise in case of spontaneous murder impulses - the things you learn at the Academy). My CO was nearby, and lucky for the guys, he heard me yell at them, while throwing mobiles and chargers around.
"But we thought it was OK, we just wanted to charge our-"
I kid you not, I reached for the firefighter's axe.
My CO grabbed me by the collar and dragged me to his room. I explained to him (between two cigarettes) that we MUST get a UPS and a server cabinet (budget constraints in the military are something that will give you people nightmares, trust me). I carefully explained to him that unless we got those, nothing would prevent the next moron from destroying confidential data and me from murdering him.
I plugged in and booted the server, after installing a multi socket extension. Two days after, surprise surprise, the server was off again. That was the first time I opened the door to the CO's room with a low kick. I must have looked like a psycho on drugs, he gave approval for the purchase in twenty seconds flat.
After that, I installed the UPS and the cabinet. Everything went inside, from the UPS to the very plugs. Just a locked box with cables coming out.
One of the guys came to my room, and asked if I could unlock the cabinet so that they could plug a "device" they needed.
I actually reached for my folding knife.
Disclaimer: The story above is TRUE. Even the almost violent parts.23 -
2010: PHP, CSS, Vanilla JS, and a LAMP Server.
Ah, the simple life.
2016: Node.js, React, Vue, Angular, AngularJS, Polymer, Sass, Less, Gulp, Bower, Grunt.
I can't handle this, I'm shifting domains to Machine Learning.
2017: Numpy, Scipy, TensorFlow, Theano, Keras, Torch, CNNs, RNNs, GANs and LOTS AND LOTS OF MATH!
Okay, okay. Calm down there fella.
JavaScript doesn't seem that complicated now, does it? 🙈14 -
So they were having trouble with the server always being slow and maxed to 100%, so the boss told me when wait times were hitting 5+mins due to server trying to catch up, he complained at me, said if I could get the wait time to 30sec to instant he would raise my pay to 90k a year, then walked away after I agreed, I was quite serious but I don't think he thought I was, so I decided to look over the system, IDK who but they put all the calculations and processing server-side for the CA's on floor then sent the completed view to the CA, so I spent months recreating the entire system except the server only pulled the data needed then the new client would do all the processing on their computer since they weren't doing anything anyways, I did a practice run today as its one of our peak days, wait times went to barely 5secs or "instant" according to CA's, I walked into the office, slapped that hourly report down after just two hours and showed the massive increase in employees production times.
That look on his face...
That look on my face...
That look on my next check...
Bliss10 -
Hey everyone,
This coming Sunday (Feb 4), at about 12AM EST, devRant will be going down for scheduled maintenance (database server maintenance). The maintenance shouldn’t take more than about 15 minutes.
Feel free to let me know if you have any questions and apologies for the inconvenience!35 -
Production is down, a coworker got himself locked in his own apartment so he can’t leave and another is late, the phone won’t stop ringing
And I don’t have the credentials to access the production server
Just a monday morning, everything’s fine 🔥😊🔥7 -
So this happened today.
Client: hey I sent this ticket, what's the status/have you located the issue?
Me: well, it says it quite obviously in the error message...? (i actually said that, toned down afterwards a little)
Client: where's the error message then?
Me: 5th line....? It's literally there in plain english?
Client: ok so what does it mean?
Me:..............? "marked as spam by the receiving server"?!
Client: yeah ok but what does that mean?
😐
Thing to keep in mind: they're a web dev/email solutions company.
😐😩9 -
> Woke up at 03:30 because I am on call and a server went down
> Fixed shit and went to bed
> could not sleep,
> fuck it, drive to work.
> arrive at the office at 04:30
> I can scream stupid shit and noone would hear
> *grin*5 -
Javascript is McDonald's.
1) everyone says they hate it....but they just keep going back.
2) very few people admit they keep gong back...
3) When McDonald's started doing salads, dressing nice, and delivering to tables it seemed a little much, you're a burger place. A few years later I'm writing my app in react JS, serving up eJS templates with my NodeJS server, running off a NOSQL JSon database, and munching down a Greek salad from McD's.
4) you start your burger (project) with high hopes. As you eat though....you start to regret it, but oh well, you're halfway in. By the end, never again, last time. A little while later, npm asks you if you'd like fries with that.
Feel free to disagree or add more!12 -
I made a web app and a mobile app for a client two months ago. Yesterday I shut down the server because I wasn't paid yet despite my multiple warnings and reminders.
Under thirty minutes the money was in my bank account.
Sad thing is they weren't responding to me, ignoring or lullabying me with good cop bad cop acts, acting like friends etc...
Lowering my bar to people / agencies like this depresses me.
Btw they are my former agency, they have all the money in the world but turns out they are all jackasses after all...15 -
Call comes down from the CEO and through his "Yes Man" that some investors are coming by to visit and he want to show off the data center and test servers. There are four full racks of storage servers filled with HDDs and each server has 4 to 16 HDDs a piece.
I got told to "make all of the lights blink", which can be epic seeing it in action but my test cycles rarely aligned that way.
All morning I was striping RAID arrays and building short mixed I/O tests to maximize "LED blinking" for the boss's henchman.
Investors apparently live/die by blinking light progress and it was all on me to get everything working.21 -
Had this recently with a client, mysql server of one of our shared hosting servers went down:
Senior engineer 1: heads up guys, mysql of {server name} is down, working on it! *calls second engineer in*
Support people: thanks for letting know! (in case clients call about it)
*triiiingggg*
Me: good afternoon, how can I help you?
Client: this site which we manage for a shared customer says it can't connect to the database...
M: is it hosted on {server name of mysql problems}
C: yes.
M: there's a mysql disruption there right now, we're working on it!
C: *starts guilt tripping me about thy they chose us for stability reasons and now this happens*
M: sir, I can't change this situation so you can go on and on about that but it's not going to help anyone.
C: okay, so what can I tell my client?
M: you can tell that we have a mysql server disruption right now and are working to fix it as soon as possible!
C: and what am I going to tell my client if they don't accept that answer?
M: you can tell that we are fixing this disruption as soon as possible.
C: yes you said that but what if they don't accept that answer, what am I going to tell them THEN?!
M: Listen, sir. We have a disruption right now. It's not fun but whether I tell this by writing it to you in a fairy tail or shout it at you, it's not going to make a difference.
We have a disruption and we are working on i....
*click*
Well, fuck you too.7 -
Manager: THE SERVER IS DOWN THE SERVER IS DOWN!!!!
Dev: Ok I’ll look into it
*5 mins later
Dev: Wow these are really strange logs, it’s like config values are being changed all over the place while I’m looking at it
Manager: Well I figured while you were looking into it I’d go i to the server settings and change everything I could find in order to try and get the server back up again. Two sets of hands are better than one, Is it up yet???
Dev: …No.
Manager: I THOUGT YOU SAID YOU’D LOOK INTO THIS. I NEED ANSWERS NOW. WHAT IS TAKING SO LONG?!?!?
Dev: …13 -
My first job: The Mystery of The Powered-Down Server
I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.
It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.
One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.
Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.
Several days went by, with the usual day-to-day comings and goings; no surprises.
Then, next week, it happened again.
My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.
(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).
Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.
I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."
My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.
Another weeks goes by and all is forgotten in the swamp of work we have to do.
Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.
My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.
This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...
So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.
So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.
The team leader is insisting the VAR swap in a new server.
One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.
The cleaning staff came in.
Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.
Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.
We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.
You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.9 -
Had a former customer call a few (10) years ago, furious that we shut down their website.
Me: but you moved your website to another provider 2 years ago? We dont even provide webhosting any more?
Turns out, when they moved we put a redirect to their new site on our server.
During the summer this server was decommisoned due to a failing harddrive but since we no longer had any active customers on it we just pulled the plug.
The customer had never actually redirected their domain name. :)
When we sent them a copy of their own cancellation letter we got an "oh hell, sorry".
:D1 -
Biggest scaling challenge I've faced?
Around 2006~2007 the business was in double-digit growth thanks to the eCommerce boom and we were struggling to keep up with the demand.
Upper IT management being more hardware focused and always threw more hardware at the problem. At its worst, we had over 25 web servers (back then, those physical tall-rectangle boxes..no rack system yet) and corresponding SQL server for each (replicated from our main sql server)
Then business boomed again and projected the need for 40 servers (20 web servers, 20 sql servers) over the next 5 years. Hardware+software costs (they were going to have to tear down a wall in order to expand the server room) were going to be in the $$ millions.
Even though we were making money, the folks spending it didn't seem to care, but I knew this trajectory was not sustainable, so I started utilizing (this was 2007) WCF services and Microsoft's caching framework Velocity. Started out small, product lookup data (description, price, the simple stuff) and within a month, I was able to demonstrate the web site could scale with less than half of our current hardware infrastructure.
After many political battles (I've ranted about a few of those), the $$ won and even with the current load, we were able to scale back to 5 web servers and 2 sql servers. When the business increased in the double-digits again, and again...we were still the same hardware for almost 5 years. We only had to add another service server when the international side of the business started taking off.
Challenge wasn't the scaling issue, the challenge was dealing with individuals who resisted change.3 -
Every time I hear a developer say "works for me", I'm gonna hide a service of mine that they use behind the VPN.
Dev: "oy mate, this server is down"
Me (with VPN connection): "sorry mate, works for me"
Dev: "but here, check this out, it's down!!"
Me: "mate, check your network connection. You must have a shitty network connection."
Dev: -_-
Me: "Maybe shitty hardware? Driver issues on your network card? 🤭"
Because you know, we sysadmins can do that too 😉26 -
Co-worker: "Should we keep this server up and running?"
Me: "Hmmm…"
C: "Do we have any other uses for it than the dedicated wiki?"
M: "Not really, and maybe it's time to move to the centralised platform Corporate™ introduced. Have we checked if anyone is using the server?"
C: "Good point, let me see…"
C: "… oh it's been down for last two weeks since the power outage."
M: "I think that answers the question. Let's leave it like this for a month more and if no one complains we can announce it dead"3 -
What an awful day :(
The server where I host my 4 clients websites crashed.
Unable to reboot from the console.
I contact the support. 15 minutes later: "we'll look at this"
No news for 1 week despite my messages.
Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...
Answer: "Sorry, your server is down and cannot be repaired."
Fuck.
I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".
Fuck.
I subscribe to another server, at another provider.
I look at my backups... shit, the last one is 4 month ago!!
I restore the first website: OK
I restore the second website: OK
I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...
I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...
I lost ALL my 4th client data!!!
I'm sooooo piece of crap!14 -
Last september:
I get an email from a customer who want to end the contract. I inform him that the contract runs to the end of March and it will be terminated the last of march.
Customer agrees and I do not hear a word from him. Until now when I closed down all his services.
Angry email:
Why did you close the server and email? are you **** stupid!?
I reply:
21/9 2016, you wanted to terminate all services you had with us. That means that your domains, DNS-services, VPS, email and Office 365 was going to be closed today.
Why is this not uncommon?!5 -
I wrote (or, ended up with a very much alpha but usable version) a monitoring system a while back and completely forgot about its existence.
But, it's still running and a few days ago I was building a docker imagine on a system with not that many resources and after about 5 minutes I started getting notifications about a high load!
Then, while I had forgotten about it again, yesterday, I suddenly started getting notifications about websites on my main application server going down.
Logged in and all was good again after restarting nginx.
Gotta say that it feels quite awesome to be notified of shit going wrong by something I wrote myself while I forgot about its existence 😊2 -
The network starts slowing down, transactions start to fail across the 450+ stores, the website starts to spit 500 errors what is going on?
Queue a frantic running around the office working out what was going wrong... Calls from all 3 data centres, nothing is going in or out of the network.
Notice the network admin come back to his desk, his eyebrows raise and he looks left and right before unplugging his laptop ethernet from one of the server access points
The network rushes back to life, everything is fine.
That particular network mapping tool is now banned for use on production.10 -
Once we were going to present a web service to governmental firm. All is going well so far and my boss asks me to host the web application the day before the presentation.
I hosted it and all was good with demo production tests, but I had a bad feeling.
While it was running on our server, I also ran it locally with a reverse proxy just in case.
* Meeting starts *
* Ice broken and down to business *
"And now our developer will run the demo for you..."
* Run the demo from my laptop to double check --> 500 Internal Server Error *
Holy shit!!!
* Opens reverse proxy link on my laptop. Present demo during meeting. Demo works like a charm. *
Firm representative: "Great! Looking forward to go live."
*Our team walks out*
GM: "Good job guys"
ME:4 -
plot twist
linuxxx is some of the most advanced and efficient AI ever developed, and actually uses devrant to scope out security-aware devs and take them down 1 production server at a time15 -
One of the biggest e-commerce site's here in Sweden like to generate hundreds of thousands of sessionfiles every day.
That slows down their website alot, and the poor server is struggling hard.
They asked me if I could do something about it. So I mounted the catalogue with sessionfiles on a tmpfs volume. And suddenly the site is much faster. I guess storing sessionfiles directly into RAM solved it :P
And yes, they are aware that a restart of the server removes the files.8 -
Contrary to most people I really love to receive email related to jobs when I'm in holiday. I keep important alerts on.
It's like:
email: ***urgent, server down***
me (sipping mojito by the pool): fuck them. let's them deal with that
email: ***requirements all wrong, must develop the feature again***
me (enjoying a dinner): oh, I told them 100 times!, fuck all of them, work for me now, stupid moron.
email: I destroyed by mistake the db with an update..."
me (dancing like crazy): ahahaha I told you that support guys should not have access to production db, fuckfuck you, fix it yourself!!!
and so on..... I don't know, it just boost my pleasure during holiday.9 -
More sysadmin focused but y’all get this stuff and I need a rant.
TLDR: Got the wrong internship.
Start working as a sysadmin/dev intern/man-of-many-hats at a small finance company (I’m still in school). Day 1: “Oh new IT guy? Just grab a PC from an empty cubicle and here’s a flash drive with Fedora, go ahead and manually install your operating system. Oh shit also your desktop has 2g of ram, a core2 duo, and we scavenged your hard drive for another dev so just go find one in the server room. And also your monitor is broken so just take one from another cubicle.”
Am shown our server room and see that someone is storing random personal shit in there (golf clubs propped against the server racks with heads mixed into the cabling, etc.). Ask why the golf clubs etc. are mixed in with the cabling and server racks and am given the silent treatment. Learn later that my boss is the owners son, and he is storing his personal stuff in our server room.
Do desktop support for end users. Another manager asks for her employees to receive copies of office 2010 (they’re running 2003 an 2007). Ask boss about licensing plans in place and upgrade schedules, he says he’ll get back to me. I explain to other manager we are working on a licensing scheme and I will keep her informed.
Next day other manager tells me (*the intern*) that she spoke with a rich business friend whose company uses fake/cracked license keys and we should do the same to keep costs down. I nod and smile. IT manager tells me we have no upgrade schedule or licensing agreement. I suggest purchasing an Office 365 subscription. Boss says $150 a year per employee is too expensive (Company pulls good money, has ~25 employees, owner is just cheap) I suggest freeware alternatives. Other manager refuses to use anything other than office 2010 as that is what she is familiar with. Boss refuses to spend any money on license keys. Learn other manager is owners wife and mother of my boss. Stalemate. No upgrades happen.
Company is running an active directory Windows Server 2003 instance that needs upgrading. I suggest 2012R2. Boss says “sure”. I ask how he will purchase the license key and he tells me he won’t.
I suggest running an Ubuntu server with LDAP functionality instead with the understanding that this will add IT employee hours for maintenance. Bosses eyes glaze over at the mention of Linux. The upgrade is put off.
Start cleaning out server room of the personal junk, labeling server racks and cables, and creating a network map. Boss asks what I’m doing. I show him the organized side of the server room and he says “okay but don’t do any more”.
... *sigh* ...20 -
You know your week starts great, when you receive a call, that the whole shop database of a company is not working anymore.
Turns out MS Fucking SQL decided to drop all tables, just because the harddisk was full.
A SERIOUS FUCK YOU TO MICROSCHROTT!
It's not a lot of fun, when 15 people have to wait for you about 1 hour, until it is restored.
Why does this have to happen on a monday?10 -
I hope you will forgive me for a third hand story, but I'm one of those evil developers, not a support per se. But I thought you'd enjoy this story anyway. So this happened to a colleague of a colleague:
$Hero - our hero. $Cop - A representative of our hard worked law enforcement agency.
So $Hero is happily speeding along in his car, running a few yellow lights a bit late, etc. Finally, the law catches up to him and pulls him over. Here's how the conversation went:
$Cop: Can I see your driving license, please?
$Hero (with smug grin): Certainly. Here it is, officer.
$Cop takes license back to motorcycle and speaks into radio.
$Hero: It's not going to help you any, though.
$Cop (with no reaction): What do you mean?
$Hero (with wider grin): The server you have to check it against is down.
$Cop (still no reaction): And why do you say that?
$Hero: Because I'm the guy they called to get on site and get it up again.
Our hero did not get a fine this time. Instead he got a police escort to his workplace.
Source: reddit r/talesfromtechsupport3 -
Nothing better on a Monday morning than oversleeping, realizing that that one server issue notification was a general notification app one and there actually have been loads of disruptions, trains are down...
If they're out of coffee at the station in fucking done with today already.3 -
One day my boss called me in to his office. "I need you to sort Bernie out".
Bernie? "Oh, you don't know about Bernie!"
Turns out Bernie was a Windows 2000 server running SQL 2000 that had all sorts of antique applications and SSIS packages running on it. Nobody was sure what it did, or if it was even still alive, but nobody was game to just switch it off.
So, after 2 months of chasing down source code, analysing it, looking for non-existent documentation, I was finally able to say.
Bernie's dead. You can bury him.13 -
I'm redoing my setup right now and getting genuinely scared that when I turn on my server, the power will go down.
I'm using so many devices at the same time 😅14 -
Client calls in and chews me out for being unable to connect to their server.
Long story short their Internet was down.
Pretty sure they still think it's my fault somehow.4 -
Much-security.nl is down.
No, no hack or whatsoever. I just reinstalled the wrong server through my control panel.
The new blog version will hopefully be up tonight. 'you keep promising that' - I don't have a backup of the old version and I only just started using git so I can only upload the new version (or git pull). Next to that, except for the front-end, everything works now.
😐23 -
TL;Dr be specific, it's actually helpful.
Client rings... "The internet is down"
Me "ok where are you exactly and how are you connecting"
"Ugh the WiFi! Just fix it"
"Ok but where are you?"
"At $companyname"
"Ok and which wi..."
"The wifi?!! Can you do anything right?"
Well... I'm allowed flexibility in terms of pleasantry...
"Ok, there are 3 buildings, 55 rooms, 2 SSID's, 17 access points, 3 routers a RADIUS server and 2 gateways... Be specific or I'll do nothing"
Simple reboot of an access point, but c'mon... It's not a secret where you are7 -
I get a call: "Hey the site is down. Fix it!"
Worked on my workstation, not on my phone => DNS issue.
Local cache: "All OK"
ISP's DNS: "No record"
Google DNS: "Server error"
MXToolbox: "All OK"
CloudFlare DNS: "Domain? What domain?"
After a day of fucking around with configs and wanting to strangle the customer support guy, I just started pressing buttons, until suddenly, it worked. Turns out I'd accidentally enabled DNSSEC on a domain, that wasn't configured for it.
Lesson learned: There is no official DNS error code for "DNSSEC failed somewhere upstream". If you're lucky, you might get something useful out of the authoritative server, but apparently not on Mondays.8 -
Wait what's that? You don't use version control on Production servers?
You want me to do what?
You want me to rename every file I have to replace with an underscore and the date after the extension so it looks like this?
SHIT.JAR_01262019
You've got to be fucking kidding me right!?
No?
Oh the production server is down again?
Is it because we're not using the right Jar file?
Well shit, I wonder why that's happening...2 -
Yesterday, the whole dev team went out to lunch and we ate a lot and we drank a lot until we all got drunk! Since we are paid for the number of hours we work, we all decided to go back to the office and work.
WE WERE DRUNK WORKING! Drunk coding ftw
Result: The server is down right now because someone fucked it up and I think i ruined my code yesterday because I wasn't really myself. The whole team was crazy as fuck. One of us just came back from Poland so we were drunk and high from all the polish chocolate we ate.
I hope they fix the server so I can check what kind of bullshit the drunken me did yesterday o.O8 -
Some fucker installed a keylogger on my Ubuntu laptop at home and registered it as a systemd service. From Wireshark, it's sending each keystroke to a server in France using irc. Tried accessing the server but the moron shut it down immediately. It's the last time am fucking installing code from prebuilt binaries. If I can't build it from source then fuck off your sniffing cunt. I was about to log in into a database from that machine.
UPDATE: I found the actual file sending the keystrokes but it's binary. Anyone know how I can decode a binary file?36 -
This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.
Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.
And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.
And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."
We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.
And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.4 -
Hey everyone - apologies for the downtime earlier today. Our host is having a lot of issues and we're working to keep everything up through it.
On that note- there might be a little more down time tonight as they are trying to fix something and we might need a few server restarts. I will keep everyone updated and thanks for bearing with us!16 -
Customer : c
Me : m
*Few weeks ago*
C: the server is slow, it sometimes takes 7 seconds before I see our data
(the project is 7+ years old and wasn't written by someone who is very good in SQL)
M: yeah I see that, our servers are busy with this one "process" (SQL query)
C: make it faster
M: well that's possible but it will take a few days (massive SQL spaghetti that I first have to untangle)
C: 😡 nvm then
*Yesterday*
C: server is down !
M: 🤔 *loads data from server and waits ~ 7 seconds*
M: Well what's the problem?
C: I need the data but it's so slow
WELL YOU MINDLESS IMBECILE... If something is slow it doesn't mean our god damn production server is down !
That just means that you have to give us a day or two so we can optimise the (ALSO BY YOUR REQUEST) rushed project... And save you YOUR money that YOU waste on the processing time on our server...4 -
Dev gets hold of me, says my service is down in QA. Works if he hits it locally, works via Postman, but via the QA app server it gives a 401.
I’m like, look, if it works everywhere else, there’s something wrong on your side in QA.
He insists, no, I must help him, and begins CCing all the managers telling them this system has been down for days.
So I eventually climb into his system, check the credentials they’re using in the QA environment, and sure enough, the password is wrong.6 -
I decided to setup a little server on my local network just to make use of a 2TB harddrive I use to store videos.
Told everyone in the house I planned to grow the library over time and that they could access it all in a browser using my system name. It's become quite a fun venture and my video library is shaping up nicely.
Using nginx on a Dell XPS 17 with Ubuntu 16.04 to host a server that just auto indexes a shared directory on my external 2TB harddrive. Kind of an embarrassing rig, but it's just a hobby activity and I do plan to upgrade shit later.
The real fun has been getting to understand a bit more about video files. They used to be magic to me, as complex as their file extension. Now I run a script on all of my torrents which checks the video and audio codecs, converting them if they aren't supported by Chrome's and Firefox's web players, and outputting mp4s using ffmpeg. I feel like I have this stuff down fairly well now. Becoming more and more automated.
Next step is to port forward so I can access it from anywhere, but we'll see about that later down the line.22 -
> me on call
> had to much wine
> suddenly, phone starts making sound
> holy shit, I am on call and I am not supposed to drink
> Calls taxi because shit is looking bad at work. Everything look down according to the sms I get, I had to pay the taxi from my own pocket.
> 40 min later, arrive at work. Walk into the serverroom.
> Go to the the server that is monitoring everything.
> Check networkcable, it is loose. Push it a little.
> Goddamnit, that was it.
> Realize I never doublechecked if stuff really was down from home....5 -
I built an infinite scroll module today that loaded in more content when you scroll down. Issue was it sent a request for every pixel you scrolled.
Needless to say, many, many, many Ajax requests were sent to the server. Oops.4 -
Finally got myself a Lytro Illum!
I,v been wanting to buy one since it came out but the company who made it closed down in 2015..
Thoose fuckers just thrown everything in the trash and set it on fire, software, firmware, mobile app etc.. no open source, no archives, your expensive camera is now a paper weight! You’r welcome!
So i got myself a new hobby, started reverse-engineering the fuck out of it, luckily it’s based on android (api17), i have adb and it’s running a hidden DHCP server too so it’s coming along nicely :D
I’m planning to make a camera control mobile app for it and maybe some faster image processing, wifi sharing etc..
I love beeing in home office :D19 -
Me: *on the phone with client* "Ok, now reboot the cashier terminal"
Client: *reboots server causing whole system to go down*
Me: *internal screams*4 -
About a week ago I bought a server rack case from some guy online for 40 bucks. When it arrived it was in the worst state I had ever seen a case. Even if I tried I wouldn't have abused a case like that. Rust... everywhere...
I stripped it down and removed all the rust I could. Then I painted it and this is the end result. Pretty cool if I say so myself :)12 -
I'm so grateful DevOps is now a thing. I remember getting a phone call from a client at 2am on a Friday because their site was down and having to ssh in from a Nokia with the world's tiniest keyboard to reboot the server.
Of course that particular server only exposed port 22 on it's local network, so I had to first ssh into another server which did have its ssh port open to external connections.
Trying to remember two sets of credentials and type them in on a tiny keyboard, while so drunk you were seeing double, standing outside in the rain as it was the only place you got signal. Yeah…I'm so grateful DevOps is now a thing7 -
Stack Overflow. Everyone uses it but everyone seems to hate the community. I very often read about someone getting down voted and they all say the same thing - "I have no idea why".
I have spent a lot of time moderating SO posts, which gave me a lot of reputation and medals. I find it fun to help people and it feels good to give back to the community.
I have asked a bunch of questions and I've never gotten a single down vote, which leads me to believe everyone of you that is constantly getting down voted are doing something wrong. Because the posts I see getting down voted are fucking stupid questions that either lack information or contain too much information.
Example 1:
Server java error
Why is my server not working? I am using Tomcat, port 8080 and I'm getting IOException.
Example 2:
Webpack configuration not working
My webpack is not a working, why?
[entire webpack config]
End examples.
What the fuck are you expecting asking questions like these?? No one gets paid for answering your questions, so the least you can do is write a CLEAR AND UNDERSTANDABLE question. I'm not gonna tell you how to do it because there's A LOT of information on how to do it.
People devote hours and hours to helping others on SO, and of course they get fed up with the stupid and lazy questions. That community is not about being nice, it's not about making people feel welcomed, it's about QUALITY OF CONTENT. No one is crying when they find a superb question + answer, right? That's the result of a community not accepting low quality content.
So please, the next time you get a down vote on SO - do not come here whining about it but instead take a look at what you have posted there and ask yourself if it could have held a higher quality.
Thanks!8 -
1. Fucking MySQL database clusters.
There's nothing fun about MySQL clusters. Sometimes they start producing deadlock errors for no apparent reason... well, there's probably a reason, but it's never a transparent easy to find reason.
What was even less fun is that those errors took down a Sentry server. When your error log server goes down through ddos from your database messages, it's time to rethink your setup.
2. Wiring up a large factory with $2 arduino clones, each with a $2 esp8266 wifi chip, with various sensors for measuring flow of chemical solutions (I wanted cheap real time monitoring as an early warning system next to periodic sampling).
The scaling issue was getting over 500 streaming wifi signals to work in a 55c moist slightly corrosive atmosphere with concrete and steel everywhere, and getting it all into a single InfluxDB instance for analysis.12 -
A Merry Christmas to everyone! Hope no one will get server down or family asking for help with computer!1
-
Got a call while on vacations, the main server is down.
*holy shit* I thought.
As I opened my MacBook, the phone kept ringing and slack was going crazy.
“What should I do? Where is the problem?”, the voices in my head said..
I opened the terminal and tried to get a ssh connection.
Sweat was dropping down from my forehead.
“Connected” the terminal said.
“Fuck yeah, the server is up, only the app is not responding”, I thought and opened the log files.
Suddenly, “STOP” I shout at the log files that were appended way too fast.
Then I saw it.
TimeOutExceptions..
I added an index to the modification date column,“ kill -9“ed the process, started again and went back to vacation mode 🙂
And of course I was the office hero for a while💪
For the smart asses, I’m aware that it’s a bad idea to -9 the app process in prod, but it was so overloaded that i was not able to kill it any other way. And we needed that server up again.4 -
So ehm, tl;dr: KEEP DAILY BACKUPS. EVEN IF SOMEONE SAYS NOT TO.
7:48
Manager: Hey Tom, is the server down?
Me: Nah, should be ok, I just did some maintenance this sunday.
Manager: But I can't get [some work data from SQL server]
*Nervous giggle*
9:14: Some random off-site cunt they hired didn't read the notes that said "DO NOT REMOVE DATABASE [xyz]"
9:20-ish: Web don't even have the DB. And you said that we'll figure out what to do with backups later
*Suddenly manager starts to panic*
11:47: Found backup of the entire server on and old server that we had for spare parts, still running tho.
12:something: Everything back up and working.
Really glad I kept the old server running and doing daily backups. Saved our ass for the second time. And finally, new off-site backup is planned this week.3 -
Had 4-5 hours of sleep last night because I had to wake up early this morning for an internal demo of the feature I worked on. It was scheduled early morning because the boss was busy.
I come to the office looking like a zombie, practically sleep walking my way to the office, and guess what... The app server is down! Yay.
I had to reschedule it to right in the middle of lunch time because hey... The boss is busy. Now everyone in the team hates me for ruining their lunch.
Now I'm sitting here, staring at my code, remembering the sweet luxury of sleep.2 -
Rant about devRant.
I hate to see two types of posts:
1. “Haha, i added a WordPress existing theme and charged customer XXXX EUR”, “Idiot customer, I had cross platform app and I charged for each platform”, “lol, they wanted to negotiate the payment, I connected to backdoor and shutdown down their servers”
2. “Why does customer not trust our estimates?” “I told him he does not need to worry about this comparability”, “they are asking about security of the hardened server, are they nuts?
First you treat customers dishonestly and then expect them to trust you and rely on your expertise. Before you constantly complain about the customers - look at yourself guys.10 -
Worst meeting:
Boss: *calls everyone* URGENT MEETING. If you're on your lunch break, then stop and join the video call. Is it a weird time in your timezone? Wake up. But COME
Omg what happened? Is the server down? Are we getting a huge client? What is so urgent that it needs all of our attention right now?
Boss: I'm afraid that I won't be able to stay long in this meeting.
... then WHY the f did you call this meeting?
Boss: that's why we'll have a meeting next at *such hour*. In the meantime, talk to your respective departments about what you've been doing and what you need from each other. See you!
No comment6 -
Me to customer, several years ago:
Your server is over 10 years old, and the support for it will soon be gone together with the possibility to take backups. You should upgrade your website ASAP.
Customer said:
ok, we will look into it.
This day:
Customer:
Hi, our site is down and we need backups from yesterday. Can you please fix?
Me:
Sorry, no. *send copy of the email several years ago*2 -
Five US quarters are about 1oz in weight and should hold down most keys on most keyboards.
I knew this because we had an error on a server with a pop up window that was only "ok". We couldn't kill the process because critical so we left a stack of change on enter until it it was done running and we could leave for the night.3 -
1 - Please hack his/her facebook account for me.
2 - (at home) I used to block wifi access by mac filtering and if there's legit server down and wifi isn't working, everyone blames me.
(I am freelancer and mostly work from home)
3 - almost all of my relatives think I don't work.
4 - I am first choice for everyone's phone, PC and hardware repair.
This one is classic
GET A REAL JOB, you need to go out in the field for work.5 -
Preface: My company took over another company. A week ago I inherited their IT.
"IT" !!!! Are you fucking kidding me?!
Their server stood at an ex employees homeoffice. So I drove to her and she had 0 idea about IT. Server was just "Running". I tore that fuck down and saw an aweful lot of Hentai in all home folders.
WTF?!
Not enough, their crm was a makroinfested access table. Shit was protected so I couldn't even edit the makros. The retarded fucktards hardcoded paths to serverside folder \\fuck\you\hard\cavetroll
Just so that server will never see the light of my domain! Damn you? Mothership of sisterfucking dickgirls!10 -
Back in my sysadmin days we had an IT zoo to look after. And I mean it... Linux side was allright, but unix.... Most unices were no longer supported. Some of their vendors' companies were already long gone.
There was a distant corner in our estate known to like 2 people only, both have left the company long ago. And one server in that corner went down. It took 2 days to find any info about the device. And connecting to it looked like:
1 ssh to a jumpbox #1
2 ssh to a jumpbox #2
3 ssh to a dmz jumpbox
4 ssh to an aix workload
5 fire up a vnc server
6 open up a vnc client on my workstation, connect to than vnc server [forgot to mention, all ssh connections had to forward a vnc port to my pc]
7 in vnc viewer, open up a terminal
8 ssh to hp-uxes' jumpbox
9 ssh to the problematic hp-ux
.....6 -
I know I shouldn't, but I had to laugh: "Does anyone know if the 'localhost' server is running? And could someone turn it on if it's down? I tried to connect but couldn't get through..."4
-
*wants to watch Re:Zero on Windows*
The files are on my file server, exposed to the Windows machine with Samba. But the Re:Zero directory isn't visible on Windows 🤔
$ mv "Re:Zero" ReZero
*Suddenly becomes visible on Windows*
What the fuck.. can't it do : characters? Something as basic as that? Microsoft, you.. you never heard of character escaping? I mean, Linux shells for example don't deal with certain characters very well either, so what do you do? Either "this", 'this', or this\ stuff, depending on some and the other things that I won't get into, but mostly it boils down to preference.
Meanwhile Windows: sorry man, can't do it >_< but I can fuck up your language, updates, privacy and files!!!
Fucking hell.. at this point I'm not even mad anymore. Just.. what the fuck Microsoft?14 -
Old client texted me yesterday: the website and pos system you made does not work anymore... Why ?
I saw that their domain was moved to another host and texted back: "some has moved the domain so that's why."
Client: "how can this be fixed"
Me: "move the domain back"
Client: "but then the new system I bought cannot function".
Me: oh well, then you are in trouble, if the new company you hired to make you a new system and website had been using just a little brain power, this would not happen. Now you have to bring your new system up and working before you open your store...
I could have helped them by pointing a sub domain to the server, but he never ever treated me with respect, and never payed in time, and he did not tell me about this move before he initiated it.
Me: shuts down server and thingking: good luck working with those new "professionals"4 -
Hello devRant, a question for you.
I'm looking to redisign/setup my server 'infrastructure'.
It'll exist out of:
7 vps's (6+gb ram/500gb+/100mbs up/down per vps)
2 dedicated servers running as virtualization servers. (16gb/4tb/1gbit up/down and another one but let's leave that one out for now because it's gonna take a shit ton of time to solve that clusterfuck)
One server will function as an entry point for all websites I run, multiple database servers and multiple backup ones.
Any advices/tips/ideas?
Just a very serious hobby thing :)18 -
I need to make a confession about my terribly unprofessional project I made. Around two years ago I got thrown for the first time into back end development - I had to work on the project alone. As a very smart man I basically exposed our SMTP server as a nice and very flexible API.
Fortunately it was, by the design, a very short-lived project, taken down from the web completely and for good after around 2 months. I'm still happy I had more luck than brains and nobody used our server as a spam sending service in our name and I have learned a valuable and relatively cheap lesson in security this way.1 -
Before shutting something down, verify that its the right one ;)
A colleague was testing some performance setting on a server and needed to restart the network driver so he disabled it and was going to enable it when the remote session died ;)
Fast 30 min trip to the datacenter to enable network card.2 -
I've got to say, arriving at my teeny tiny rented room after a long day of distribution center work and sitting down behind my monitors with a good beer and doing some programming/server stuff really feels like coming home 😍.2
-
Anybody uses Slack at work
My whole company freaked out today because Slack servers were down
It was hilarious 😂2 -
Gather around folks, I'll paint you a nice picture based on a true story, back from my sysadmin days. Listen up.
It's about HP and their Solaris 5.4/6 support.
- Yet another Prod Solaris dinosaur crashed
- Connected to console, found a dead system disk; for some reason it was not booting on the remaining redundant disk...
- Logged an HP vendor case. Sev1. SLA for response is 30 minutes, SLA for a fix is <24 hours
- It took them 2 days to respond to our Prod server outage due to failed system disks (responses "we are looking into this" do not count)
- it took another day for them to find an engineer who could attend the server in the DC
- The field engineer came to the DC 4 hours before the agreed time, so he had to wait (DC was 4-5 hours of driving away from HP centre)
- Turns out, he came to the wrong datacentre and was not let in even when the time came
- We had to reschedule for two days later. Prod is still down
- The engg came to the DC on time. He confirmed he had the FRU on him. Looks promising
- He entered the Hall
- He replaced the disk on the Solaris server
- It was the wrong disk he replaced. So now the server is beyond rebuild. It has to be built anew... but only after he comes back and replaces the actually faulty disk.
- He replaced that disk on the wrong Solaris server2 -
devCraft {
Closing the minecraft server for a little while!
I'll be adding mods, writing up a perms file, and hosting the pack on git! I'll post a rant with the repo link.
One of our lovely ranters offered a VM to host the server on, so the ip is probably gonna change as well! (i also gotta make an arch bootable USB, and running the server would slow that down lol)
i'll notify you all once it's donevia a rant, like i said. until then, formulate plans, and suggest some developer-related mods for me to add in! (must be 1.7.10)
Currently planned mods are:
- ComputerCraft
- Applied Energistics
- Buildcraft
- Project Red
and a few from whatever you guys suggest. see you then!
}47 -
!rant
Trust no one.
The internet is not your friend, until you find stackoverflow and you get down voted.
That rm -rf / won't make you server faster.
System32 is needed.
Yes, that is a package manager, you don't need to write more ccode
Do not write commets on languages that only you speak, the team does not speak in latin.
Paint is the best engineering tool.
Keep a stress ball nearby.
Your client is always right, unless they mess with your coding skills.
True story.2 -
I recently started my first job at a small place and on the first week they gave me access to the firewall server because they wanted me to implement a VPN service. It could have gone worse, the network was only down for 10 minutes.1
-
Sad story:
User : Hey , this interface seems quite nice
Me : Yeah, well I’m still working on it ; I still haven’t managed to workaround the data limit of the views so for the time limit I’ve set it to a couple of days
Few moments later
User : Why does it give me that it can’t connect to the data?
Me : what did you do ?
User : I tried viewing the last year of entries and compare it with this one
Few comas later
100476 errors generated
False cert authorization
Port closed
Server down
DDOS on its way1 -
I really, really, fucking god damn it REALLY need to move a legacy project from the grave yard server and get it in git, and then build a dev environment for it, so I can stop making incredibly volatile changes direct to PROD (backend, frontend and DB all at once and then test it while it’s live and being used, but fuck me if I can be bothered digging through a 10GB code base and attempting to make it work in a multi-environment setup when it’s going to be a long trip down the error logs until it works again 😱🔫2
-
I couldn't figure out why my server went down all of a sudden.This seems suited for this week's rant.
Expert fullstack solutions architect here.3 -
Some years ago our company site was hosted by a prick who knew nothing and started to pretend the server got a virus or whatever.
I tested their server and figured out they did not have any firewall policies going on like mitigation of ssh brute force.
It was at this time I learned about SYN flood, and boy I flooded that port 80 of them.
The company site went down for as long as I wanted.
It was great because now we manage it in house and never had a problem anymore. -
Why is it always on Fridays that some server decides to start acting up?
I want to start my weekend, damnit! -
!dev
The school I went to didn't have PCs when I first joined (had some RISC OS machines instead). They got Windows 95 PCs eventually and networked them. I had no experience with networking before this, but had a PC at home. We all had mapped drives to resources on the server. The PCs were pretty locked down - no "Run" command etc.
Anyway, one day the head of IT came in to one of the lessons and asked me "how I did it".
What had I done? Well, clearly he had seen something I'd taught one of my friends. I wrote it down for him.
1. Right-click the desktop
2. New shortcut
3. \\nameofserver
4. OK
Such hax, being able to see the file shares on the server.
Shortly after this, all computer areas had signs saying "no shortcuts allowed"... -
I was noticing some slow network and it was dropping some connections. So I booted up my old XP install with Java 6 so connect to the ASA 5505, I see it’s logging max connections of 10000 has been reached.
Fine, I recon it’s my colleague backing up his entire machine to Google Drive.
Because when he shut it off, n connections dropped.
I check back in the log, and I see there’s 4-500 connections happening per second, I think WTF and check the source IP. Lots of random IPs from Vietnam, all going to a Windows2008 Server using rdp.
(I didn’t setup our servers, so I didn’t know which server it was accessing)
Ask my other colleague, he told me it’s a windows server from an earlier project that’s not used anymore.
I rdp into it, see there’s users logged in from around the world, and I immediately do a shutdown.
Would you look at that, connections per second dropped to about 50.
I guess that server isn’t going back online ever.
And I now need to ask management for a budget to update our network infrastructure, because the old ASA 5505 is begging me to die.
TL;DR gg previous employees didn’t shut down old servers and left them open to the world to enjoy9 -
client : show website
me: sir server is down
client : what? I don't know,I want to see my website.
me: sir try to understand it is not my issue
client *call cut*
fucking asshole -_-1 -
Did some updates to an older Web Forms website built by a previous SENIOR developer who is a notoriously horrible developer.
Now before I start, you have to understand this guy studied at a University and had been working for at least two years before I even started working. He is supposed to know the basic shit mentioned below.
This also happened a couple of days ago, so I have calmed down since then so I apologise for the relaxed tone. My next rant will contain a lot more swearing.
This fucking guy did the stupidest shit imaginable.
On the details view of a post|page|article|product|anything that would require a details view this jackass would load the data from the DB.
Using an OleDbConnection, OleDbDataAdapter, DataTable and the poorest writter fucking sql statements you have ever seen. All of these declared in the Page_Load method.
There was literally no reason for him to use OleDb instead of Sql, but he simply did not know any better.
He especially liked: "select * from tbl where id = " & Request("T") & ""
ZERO fucking checks to see if the value is even passed or valid, nothing. He did not even check whether the DataTable had any rows.
He then proceeded to use only the Heading column of the returned row to change the page's title.
Stupidly I assumed the aspx page will be in a better state. Fuck NO!
This fucktard went, added server tags to the opening of the asp:Content tag, copied that shit he used to fetch the data and pasted it between the server tags.
He did not know how to access the DataTable mentioned above from the aspx page!
He did this on every fucking project he worked on. Any place that required <%= %> to display data instead of using asp server controls, this cunt copied whatever was written in the code behind and pasted everything between server tags.
Fuck I could go on forever, but I think this is enough for my first rant.2 -
What kind of genius pulls the power cable of the main server through the PSU lever of an old useless server?
I had to pull out the PSU and let it hang on the powercord just to be able to pull the old ugly bastard out... fucking genius, really!
Now I have to wait until the evening to shut down the main server and remove the hangman PSU.
I just really fucking hope and pray that restarting the main server will not cause errors. It has been on for 2 years and never been updated since then.5 -
Was on a call today...nobody knew who owned or was responsible for one of the servers. My manager has a great way of finding out...he says "If I take my hammer and smash the server to pieces...who gets pissed." A moment of silence then someone speaks up..."Oh well then that would be Joe down the hall."
That my friends is how to get answers. Threaten to break things.5 -
One of the dumbest things I've done as a dev... I was a front-end/php guy that got hired into a asp.net company. My boss told me to take down a section of the website. I hopped on the server and started looking at files and saved a config file. 2 minutes later I had 3 devs in my office asking what I'd done that took down our entire network.
Thankfully my boss laughed it after they republished the site, but that was my intro into asp.net development.1 -
Why the fuck does your API have to send 401 when the server is down.
Just spent an hour thinking something was wrong with my auth wrapper.3 -
Project manager asks why server is down. Reminded him that his team lead told me three months ago it was no longer needed.
-
Today, during deployment on server without remote access:
Me (on the phone calling our data centre Admin): "There's a permissions mismatch. The following paths need write access from the following users..."
Admin: "Okay, okay, slow down... I'm still in the elevator." - 10 minutes later - "Okay, ready."
And I gave him the paths and he said: "Try now."
And I tried and it still didn't work. And then we tried all that again. And again. And finally he said:
Admin: "Okay, I give up, I'm going back down to get the screen."12 -
First rant!
The first time I got in touch with programming was when I was about 14 years old. I started a private server for a game called Maplestory (yeah you know it, I know you do) and had one of the most popular servers.
Topping all the rankings of best servers, getting lots and lots of traffic...
Anyway, I started modding the game and implement new features and quests. Right until my father saw our bandwidth. Because the server was running on my computer in my own bedroom 24/7 and blowing nice hot air in my room.
Our bandwidth limit was reached in just a couple days in to the next billing cycle and had to shut everything down from that point. And this happened a few times.
I was devastated shutting it down but learned so much from it. And it introduced me to programming.
Up till now, I'm almost graduating in computer science, already have 2 companies that are willing to hire me, and probably even going to work with my dad on a huge app soon2 -
Just had a fire at work.
git add, git commit, git push.
No one was hurt. But, the fire was in a server room and the git server is down. At least the halon deployed successfully.
The off site backup should be up in 20 mins. I wonder how often they pull the repo.3 -
We had a major core router hardware failure in our LA datacenter today and every one of our services has been down since 6am, including all production servers. We have about 15,000 sites down across our entire platform. Our manager came over and told us to just go home because we need to replace the hardware and the process is expected to take all day, and we can't do any work until then because all the production servers are down. So you could say that it's been a pretty easy Friday so far! I'm headed home to play Spider-Man2
-
The worst mistake I made was using json files as a simple database.
The project was to be delivered the day before it was even ordered, so I had to take shortcuts.
I totally underestimated the popularity of the project, so all that file reading and writing made the server slow and sluggish. It also being submitted on a shared host, made it even worse.
In the end the host shut it down. I don't recall, but I guess I had to add that database support after all, the project being that popular...4 -
Manager after google went down: Hey I don't want our project to go like google. Once one server went down, gmail went down, then youtube, then drive, and everything else with it. Could we separate all of our features into different servers so that if one thing goes down, the rest aren't affected?
NO9 -
I'm having a DAY.
it's like I'm trying to "2+2=4" and hitting "unknown operator =", figure that out, get "22" back, fix that, then math service is down, redeploy math service, wait 30 minutes, that's up, get "out of numbers" exception, turns out a dependency updated and now I have to register what "2" means, fix that, FINALLY get 2+2 to evaluate, I get 5. Figure that out, get 4! 4!!! FINALLY. Push it. Get 55 on the server. Gets 4 on my machine, getting some random 55 from the server.5 -
This is a story of me trying out maintaining a game server and eventually making a mistake, although I do not regret experiencing it.
A month ago I set up a small modded minecraft server because I wanted to experience a fun modpack together with some people from reddit. Besides this, I also wanted to see if I was capable of setting up a server with systemd and screen running in the background. This went great and I learned a lot.
The very next day I was playing with $annoyingKid on the server and everything was well. However the second day, $annoyingKid started pushing the idea to start up a normal minecraft server to build a playerbase.
I asked $annoyingKid 'What about financing, staff management and marketing?'
$annoyingKid: "I don't know much about that, but you can do that while I build a spawn!"
He also didn't want to reveal his age, which alerted me that he's young and inexperienced. He also considered Discord 'scary' because there were haxors and they would get his location and kidnap him, or something. So if he was supposed to become owner (which he desired), he had no way of communicating with a community outside of the game.
He also considered himself owner, while I was the one who paid for the server. 'Owners should be people who own the server', no matter how many times I told him that.
$annoyingKid also asked if he could install plugins on his own, I asked him if he knew anything about ssh, wget or bash because I used ssh to set up the server (I know rcon exists, but didn't want to deal with that at the time), he had no idea what any of those terms meant and he couldn't give proper arguments as to why he should get console access.
In the end, he did jack shit, he had no chance of becoming co-owner or even head-admin because he had no sense of responsibility or hard work. I kept him around as an admin because he was the one who came up with the idea. I banned him on day one after he started abusing his power when someone tipped him of. Even after me ordering him to ignore an annoying player he kept going, of course I could have prevented all this by kicking him earlier since all the red flags around him had already formed a beacon of light. He tried coming back, complaining that he should at least have his moderator rank back, but he never got in again.
A week later I got bored, I had had enough fun with ssh and the server processes to know that I didn't want to continue the small project, so I shut it down and went on to do stuff on GitHub.
Lesson learned: Don't let annoying kids with no sense of responsibility talk you into doing things you aren't sure you want to be doing. And only give people power after they've proved to you that they are capable of handling it.1 -
A Bad and Sad Day
Hello Monday,
Client : on weekend site went down for 7 hours? Why ?
Me : Let me check the logs
Client : bla bla bla
boss : check ur code bla bla bla
Reason : Some PHP service stopped on server
Client2 : I have purchased this software and you have to made minor changes
Me : Payment Integrations are not working?
Client : Whataa nonsense is this.. you are supposed to do this
Me : We are supposed to do minor changes. They do not have proper payment integrations. If you want we have to write complete code
Client : bla bla.. I gave you working software
:( Why don't you just fuck off .. liars2 -
Inherited a simple marketplace website that matches job seekers and hospitals in healthcare. Typically, all you need for this sort of thing is a web server, a database with search
But the precious devs decided to go micro-services in a container and db per service fashion. They ended up with over 50 docker containers with 50ish databases. It was a nightmare to scale or maintain!
With 50 database for for a simple web application that clearly needs to share data, integration testing was impossible, data loss became common, very hard to pin down, debugging was a nightmare, and also dangerous to change a service’s schema as dependencies were all tangled up.
The obvious thing was to scale down the infrastructure, so we could scale up properly, in a resource driven manner, rather than following the trend.
We made plans, but the CTO seemed worried about yet another architectural changes, so he invested in more infrastructure services, kubernetes, zipkin, prometheus etc without any idea what problems those infra services would solve.2 -
I call this one the tester than knew too much.
Note: The server the tester is running on has a hard drive that is breaking down...
Tester: Remember the error I talked to you about yesterday?
Me: Yeah, what about it?
Tester: Well the server hasn't recovered yet and I haven't restarted anything...
Me: Well the application itself hasn't crashed so our monitoring application doesn't seem to notice that the service is in a bad state. The error seems only to have brought down certain threads within the application.
Tester: No, I think there is a different issue here and has nothing to do with that error, the application is still doing things.
*tails the log*
Tester: See?
Me: As I said some things are still running and are unaffected by the error.
Tester: NO! It has to be caused by the other error I had a week ago where my file got corrupted. As we said I removed the file, restarted it and it worked again, but had the same problem a day later...
Note: The problem is not related, this time the application is running out of file descriptors
Me: Well... If the problem is the same it would have complained about the file descriptors then aswell, not an I/O error.
Tester: Nope, I think you are wrong!
Me: ¯\_(ツ)_/¯
FML3 -
Scared the shit out of me when I heard one of my volunteer side projects website was down. Because I just finished fixing my own hacked website today.
Turn out the server and website is fine but the domain was expired yesterday 😣
Don't fucking scare at me like that.
Tough week indeed.2 -
While I work on my devRant client, I always end up thinking, "how would dfox do it for the backend".
Some of the functions I have include:
-fetching new rants, 250 at a time
-remembering the ones I've read and filtering those out (kinda similar to Algo)
-checking for updates from followed users and new notifications
My implementions are kinda spammy though part of it is because I need to use the existing API.
But even so, how would/does he implement these so all the users' requests don't end up bringing down the server, and efficiently store and retrieve everyone's view history?
My problems are small compared to his... How do you think and deal with problems at that scale though? -
Next 3 days I' ll be working on moving our email server to new server because the old datacenter is closing down. Clients are pissed and they are fucking verbally bashing me left to right when I told them that their emails will be down since most of them host their own DNS.
I am fucked.3 -
Six or seven years ago, I worked for a large financial organization as part of a very large effort to convert server assets from physical to virtual. The consultants on site were in bed with the vendor of a terrible piece of software designed for that purpose. After a couple weeks on the job I'd had it, and sat down in between sessions of "validating" the conversion procedure, and started writing my own software for converting Linux servers. After a couple days it was working great, and they wound up adopting my software as the default method for Linux conversions.
Years later, I'm interviewing for my current job and one of the interviewers tells me he used my converter some time later and loved it. Pretty sure it's what swung the interview for me. -
Our main server that stores everything and that everyone uses has been down for about an hour now. It's okay though I get paid by the hour. Plus, I'm working remotely today.
Probably doesn't help that we let other people other than the IT department mess with what we put on the server.1 -
Today we updated PHP on our server. From 7.0 to 7.2
Promptly the whole server went down and none of the pages on it worked anymore. I guess we sort of forgot that there are other pages on the serve than the one we were working on...
Now everything is operational again.3 -
Something is really fucking wrong with people in my company. They fucking calling me after 5PM on Friday when the server is down. What part of my role you fuckers don't understand. I'm not a Network Engineer and I don't have fucking access to the fucking server.
Call the fucking Network Engineer. If his not picking up his phone then that's his fucking problem not mine.
(Bang the Table) Fuck this shit4 -
So i was bitching about some cron task few days before
and guess the issue
Fucking Client just shut down the server by himself for night and cron job scheduled for night time :(
I fucking waste 2-3 days to debug the function whats wrong with them until one day I checked server is closed at night :(2 -
Fuck google cloud platform. My server has been down for last 4 days. Stupid reason google gives me is that it does not have resources available in my zone. Why the fuck do you start a hosting company if you cannot provide RAM and CPU. On top of that their support is so bad that after 20 emails, 4 chat tickets, 3 phone calls nobody knows the issue I am facing. They just give the links to their ultra stupid documentarion.
Now all my 6 projects are down. Clients are getting impatient. I cannot do any work and googles support is the worst.
They dont even want to understand the issue, dont know how they will solve it.
I have created AWS instance now and migrated to AWS. But i have old backups which are useless on AWS. To get the latest backups i need google cloud instance to get started but stupid google does not have resources. How hard it is to add 1 CPU and 1GB RAM?18 -
Following on from yesterday's catastrophe...
It happened again overnight!
And THEN another server failed when backing up the DBs...turns out 2 websites have activated logging .. each of around 20GB in the database!
So this morning both of our servers are down for different but similar reasons.
It's absolutely fine everybody.
I'm fine,
we're fine,
it's FINE! -
The effect of updating the bios via USB (possible only on windows) when all computers run on Linux.8
-
Decide to update main apache server before going on a date with my girlfriend, and I somehow fuck it and the server up. She is here in 20 minutes, and the entire site is down. FML8
-
I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.
Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.
I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.
I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂
Definitely the worst moment of my year.3 -
Currently our companies website is being attacked and we're getting a 500 error and the FTP servers are down. Everyone is panicking. Little do they know that I just rebooted the server and already sent out a notice of this happening and will be back online. People are chanting for my death outside my cubical with fire and pitch forks. HURRY UP AND FINISH REBOOTING SERVER!!!1!!2
-
Do I want to continue?
Y -> vacation lost (Production server is down).
N -> Ok, I will gather more packages for you to update next time.
😭😭4 -
So, I've been having to deal with server issues and other BS. All of these issues have caused all of my websites to be down. For the longest time, I couldn't get Nginx to find my websites' directories except for the default Nginx website, so I changed the default Nginx website to what you see in the image.
I tried to be funny, did I succeed? Even a little bit?8 -
I just got an email that a client changed their DNS zone files to point at a new server. Turns out that they haven't set the server up yet. Client is wondering why that domain's emails aren't working, and why their site is down. They didn't want to give me the Domain's portal login credentials until now, because they "could do it without [me]." Tomorrow morning should be fun.4
-
Domain server goes down, it's the gateway and DNS too.
Ok I'll just remove the domain, it's been orphaned really since you went to the cloud.
Don't have local admin password.
Ok call old it company who set up gear
Out of business
Ok boot to Linux and reset
Usb boot locked
Don't have bios password
Call old it company
Still out of business.
Wait, can I just set manual ipv4 ? Ok domain without a domain controller... If it works it works.2 -
We decide to joke around with building stuff
We happen to name our Scaleway Server "Chitoge"
shit went down from there3 -
Dev: The server is completely down right now. Nobody can access the application, we need to divert some resources to horizontally scaling our app.
Manager: Hm, this was not in the schedule. I need to consult senior leadership on what to do about this. I don’t want to be held accountable for making a decision on this complex and highly nuanced situation.
Dev: No need, I have a solution. Just need a week to build/test/deploy the ability to horizontally scale.
Manager: But that will cause delays to new features.
Dev: New features don’t matter if the app can’t even load.
Manager: Ok you can implement your solution but it can’t take any time. I need those new features out.
Dev: ???????5 -
I wanted to play games today. simple mission. i was playing for two hours and somehow i ended up writing a webapp to help me with my taxes based on data from my firefly-iii server 🤔
why can't i sit down and just do nothing for a day 😅8 -
This is stupid but i think is my best idea yet.
So i have an old orange pi, with only 256m memory. Its running a few tasks i need but i wanted to use it for controlling a few things from my phone (lights and powering on my pc) so i thought i would make a server for that. Now mind you, my shirt doesnt say "lightweight backend language", so there was no way the pi couldve handled a struts server. I was digging around and found that php has a shell_exec command. Then it clicked, and i wrote the whole system like
shell_exec("java -jar someprocess.jar"). Now this sounds really stupid but it works and php is really light so it doesnt even slow it down that much.
Thinking about making this into some kinda server/framework/something just for fun.4 -
Just now I was reading on https://pve.proxmox.com/wiki/... about high availability. Now my Proxmox VE is just a tower (which happens to have ECC memory) that's stored in my storage room (and which is mostly used for experimental and home server purposes). But my mail servers.. those have been made with high availability in mind. Most importantly, I've made their services entirely redundant (but within the same datacenter). And when they have updates, I apply updates to one, reboot, see if it didn't break something and then do the same to the other server after the first one came up again. So no downtime whatsoever.
If memory serves me right, I think that I've been able to maintain these servers for the last year without any downtime at all (I reboot them every month to apply new kernels but they haven't both been simultaneously down at any moment). Does that make them High Availability? My interventions regarding their availability have been rather trivial. Is it really that hard..?4 -
I really felt like a badass one time when I managed to recover all projects on our dev server after a full meltdown of the HDD.
We had no recent backups, because our backup server was down for a few months, and our (at the time small) company was in a tight spot on finances, and couldn't get a replacement.
The problem was that the HDD on the backup server failed, but we were storing all projects also on the dev server, along with our local git repos (no GitHub at the time for us), but then the dev server HDD also broke, and I used every piece of data recovery software I found trying to recover the data, until one actually managed to read the raw data from the HDD and store it as a virtual drive, that I then used to try and build another partition index and it actually worked!
Lost about 10% of the data, but that was enough, as i managed to recover all the git repos and databases...
I don't even remember the tools that got the job done in the end, but that was one hell of a week, and at the end I felt like a true IT God!
True story!
PS: 2 weeks later we had a new backup server, another offsite backup solution and a GitHub account for the company. Was delayed on salary in order to manage it (me and the CEO both agreed to give our pay for one month to get them), but worth it!1 -
So a server goes down and being the only person who can recover it, I get started, whilst doing this the boys sits right next to me (6 inches) and starts asking what caused the problem instead of letting me get on and fix it, then complains the outage was too long.
-
A week ago, the team that hired me asked me to fix the s**t they made when they hosted around 30 WordPress sites in a single Bluehost shared server. Several of those were multisite installations. The server eventually gone down because of the load. And the most disturbing part was they were taking money from some of their clients to host the sites, in stead of not having a reseller licence. The server was going down quite frequently so I suggested moving some sites to another host or another server. They asked me to do it, but when I asked for the permission to edit the nameservers, they asked me to make a subdomain and point it to the new server. Which was kind of impossible because the new host was already having some subdomains and it's not easy to work with sub-sub domains. So, on an open statement they said that I am unprofessional and not fit for work. Before that they disturbed me and bursted on me when I was off working hours. -_-8
-
Those moments when your app server is down and you restart everything and then somehow it is magically working again.2
-
Today’s DevOps public service announcement... don’t test your server provisioning scripts locally. Especially when this gem is in there:
rm -f ~/.ssh/id_rsa
echo ‘vault secret/ssh_key’ > ~/.ssh/id_rsa
Well, I no longer have my key, but the script works! I’m sitting with a very locked down server key6 -
Yesterday night, pushed code that work normally to prod server, website down, internal server error, too many connection to MySQL server, tried to fix it for 4 hours, nothing to do, removed the new code, still the same problem, in my head, I told myself that I'm not good at programming (not the first time), send an email to the host, they tell me the problem is from them and they fixed it. And now I know I'm not bad enough.2
-
I'm trying to explain to this dumb mf-er why we can't use send data simply.
It's a mobile platform I said, you can't rely on having network I said, the dumbass users might try it on 2% battery I said.
DO'H BUT MY INTERZ CAN DO IT IN 5 MINUTES.
Yeah, enjoy crashing the app with asynctask like a stupid bitch. When it fails under nearly every possible condition that deviates from the ideal, you can go write down how great your interns are, eat it up, shit it out and when you grow a flower from that, then I might help you again.
Retard server people.6 -
When you own a game server and you get ddos and your network is down and server is not up your problem is not that its down its that you cant fucking watch netflix...#hateddos2
-
Can someone help me understand?
I subscribed to a nifty IT-releated magazine, and on its back, there's an ad for "Dedicated root server hosting", nothing unusual at a first glance, but after I read the issue, I decided to humor them and see what it is that they offered, and... It just... Doesn't make sense to me!
An ad for "Dedicated Root Server" - What is a dedicated root server first of all? Root servers of any infrastructure sound pretty important.
But, the ad also boasts "High speed performance with the new Intel Core i9-9900K octa-core processor", that's the first weird thing.
Why would anyone responsible enough want to put an i9 into a highly-reliable root server, when the thing doesn't even support ECC? Also, come on, octa-core isn't much, I deal with servers that have anywhere between 2 and 24 cores. 8 isn't exactly a win, even if it has a higher per-core clock.
Oh, also, further down the ad has a list of, seeming, advantages/specs of the servers, they proclaim that the CPU "incl. Hyper-Threading-Technology"... Isn't that... Standard when it comes to servers? I have never seen a server without hyperthreading so far at my job.
"64 GBs of DDR4 RAM" - Fair enough, 64 gigs is a good amount, but... Again, its not ECC, something I would never put into a server.
"2 x 8 TB SATA Enterprise Hard Drive 7200 rpm" - Heh, "enterprise hard drive", another cheap marketing word, would impress me more if they mentioned an actual brand/model, but I'll bite, and say that at least the 7200 rpm is better than I expected.
"100 GBs of Backup Space" - That's... Really, really little. I've dealt with clients who's single database backup is larger than that. Especially with 2x8 TB HDD (Even accounting for software raids on top)
This one cracks me up - "Traffic unlimited"
Whaaaat?! You are not gonna give me a limit to the total transferred traffic to the internet for my server in your data center? Oh, how generous of you, only, the other case would make the server just an expensive paperweight! I thought this ad was for semi-professionals at least, so why mention traffic, and not bandwidth, the thing that matters much more when it comes to servers? How big of a bandwidth do I get? Don't tell me you use dialup for your "Dedicated Root Server"s!
"Location Germany or Finland" - Fair enough, geolocation can matter when it comes to latency.
"No minimum contract" - Oooh, how kiiiind of you, again, you are not gonna charge me extra for using the server only as long as I pay? How nice!
"Setup Fee £60" - I guess, fair enough, the server is not gonna set itself up, only...
The whole ad is for "monthly from £55.50", that's quite the large fee for setup.
Oh, and a cherry on top, the tiny print on the bottom mentions: "All prices exclude VAT and are a subject to..." blah blah blah.
Really? I thought that this sort of almost customer deceipt is present only in the common people's sphere!
I must say, there's being unimpressed, and then... There's this. Why, just... Why? Anyone understands this? Because I don't...12 -
Have you ever been interrupted because a marketing workmate had a friend on the phone who needed advice on a WordPress hosting, and wanted your advice right now?
Because I have.
When we had a massive server failure and our production environment was down.
Seriously, what the fuck is wrong with people nowadays.6 -
How can I use a custom DNS server on my schools network?
They seem to block any DNS that isn't their own. I've tried 8.8.8.8 and the new cloudflare one. Both blocked, it's a real issue because their default dns dies for about a minute every 12/24 hrs causing my VPN to go down.
How do they even block external DNS servers?5 -
My lead loves to over engineer crap and waste weeks building complicated solutions.
And then during retro when a team member has the stones to say we should've thought about it a little more or used the input of some other teammates, he shuts them down by saying that more input would've been bad for the design. I can see where he's coming from, but he always seems to have an excuse for us. Why can't he just be more transparent and clear with us? If he has a problem, just say it. That's what retros are for.
Oh and then he takes a shot at me saying that we shouldn't have built a UI in tandem with it. I didn't even recommend a UI for the thing. All I said was that if we ever have a UI, we should consider a database setup that assists both the server and UI. But nooo, he's stuck with this "server design" approach. Everything has to be built to make it easier for the server.
I still don't understand why anyone would have their server logic influence the design. Especially the database. I just seems too targeted. It just creates these nasty denormalized tables.
Ugh... Our team is getting dragged around by this arrogant and silly man. -
Work story.
We have this system that's being used nation-wide and basically there's a control panel for management (it's a website)) and an app for the regular users.
I just migrated and replaced the guy before me, I'm basically the only one on the project.
The code for the website is a mess, the servers are sometimes slow, and few security problems here and there.
Project Lead comes up to me and says that few of our clients that use the website are saying it works really slowly.
I start by analyzing the networking, and found shocking things.
First of all, let's say there's a messaging option, and the management teams that are our clients can have each a lot of groups, which all have messaging.
Upon first load, ALL OF THE IMAGES, FROM ALL GROUPS, ARE PRE LOADED. It can get up to few hundred photos being preloaded upon first load, which can explain the slow loading.
After discovering that, I discovered that the Administration control panel, which only my project lead can access, with sends heavy requests to the server and loads heavy assets, is loaded every time to every single client, generating heavy stress on our server and slowing everything down.
I tell that to my project lead and say that that's what causing the slow downs, I coded a fix that currently sits and is not being merged to the master branch to be deployed, and somehow I need to find a way to fix the slowness which all comes down to the heavy requests and slow connection with servers... And they won't merge my fix that fixes the loading of the administration panel so the stress on the servers could go down, and everything will be sped up....
Ah damnit.. sometimes I don't understand it..4 -
Fucking fuck fuck fuck outdated superiors that know jack shit about how software development works. Dnt even know about git, docker, cloud services. Everything is done on premise with network that is fucking crap and when an app is down "hey why is it down?" ask the fucking server and network admin how the fuck am i supossed to know? i have to create workaround codes when other devs just need to deploy their app and its fucking running as it should be. why the fuck do i need to spend my time debugging Ping timeouts? im a fucking dev. I have done designs, analyze requirements, build frontend, backend, optimize codes, paying attention to security and now i have to fix network problems as well? fuck off
Create Innovation my fucking arse. you just Keep saying that but then wondering "what is this new thing youre trying? its new and different why do that?" because you asked for innovation you fuck. If i copied some other concept its not innovation is it pricks.
Fuck them and all the brown nosers as well.1 -
I wondering what could possibly slow down our server app , Just found out there's one mother fucker who use async in every cases and async everywhere. Hey not all function are async!3
-
A course at university made us program a chat server and client with Java RPC (in the days where there was no such thing as stack overflow), without ever teaching us anything about coding.
The grading was based on unit tests executed on a Server the university provided.
The server was down or overloaded most of the time and one could only try to deploy at most 3 times...
There were heuristics in place to find duplicate solutions.
... I have to say, that course took me from "hello world" to developer within a couple of months. Thanks assholes!! :D1 -
I was happily folowing a tutorial earlier on setting up an email server but the site got taken down for maintenance just as I reached the last step...
IT'S BEEN 6 HOURS NOW AND IT'S STILL NOT BACK UP.
I HAVE NO IDEA WHAT I'M DOING AND I'M GOING TO BREAK SOMETHING AND BE SAD.9 -
So client wants an android app that implements some legacy Epson printer SDK, works on a chinese Windows device with an android Emulator on it, connects to local Webservice that had to be configurated and ran (local Network) , sends and tracks data, if Server down then handle it on the Client and reconnect as soon as Server up, running own TCP Server on Android device that listens for specific http requests, which make the android connect to an Epson printer to start printing. The stuff that is being printed? A png file that has to be converted to a Bitmap, a QR Code that has to be generated by the bugged base64 encrypted stuff coming via http in (webserver-> Android TCP server)
Dont forget the Software Design (MVP), documentation, research etc.. Im about to finish the app , its my 5th day on this Project, the 6th day was planned to be full testing. Client Calls me and ask me how far I am, I reply, he says ok. 30 minutes later he tells me he wont pay me next time that much because this work should take 3 days, or even 2. "A senior Android developer could do this in 2 days"... When i sent him my notices he called me a liar, his webdev has alot of experience and told him it should take 2-3 days...ffs2 -
>Be a customer
>Ask to reduce number of servers you rent at our company
>Agree to a date when the one server you no longer needs will be due to be disconnected and taken apart
>Date comes
>I have the honors of sending the final /sbin/poweroff
>All goes neatly... until...
The web that used to be there, now moved to another one of their machines, goes down, wtf???
Oh. a 500. What?
Checks logs...
Cannot connect to Database.
Wtf? Local database works... Oh. OH. OH MY GOD.
>Turn the server back on and tell the customer to fix the app to no longer connect to that machine
Sometimes, being a sysadmin can be a real fun!1 -
Looks like copying large file e.g. 1GB from Remote Desktop Connection will also affect SQL Server performance and somehow slowing down the SQL transaction 100000x times
What a new thing to experience😆5 -
It's 8am and the server is down... Legit rant right? Sure, but it gets better. One of our buyers in purchasing printed out a 504 error and hand delivered it to my counterpart in our northern office.
-
It's going to be a long rant here and probably my fist rant ! And yes I am pissed up with a community growing in dev world .
There are so called framework experts who are so good that they can spin up a nodejs server with express and mongodb .
So to the people who bash on php , who bash down MySQL for no fuckig reason other than they have heard these are not so cool.fuck yourself incompetent piece of crap!!! I can hear all day about how algorithms and datsructure are not important form these people.fuck you because if you don't know /understand /want to understand the basics of computing how the fuck can your brain be trusted with anyting serious??If you can't write down proofs of basic / standard algorithms and till bash down on people who do those please fuck you because those are the people indirectly responsible for your Job so that u can work on fancy frameworks and cool IDE's .
Instead of whining down dedicate some time to your maturity and knowledge because that what we devs are all about.we like solving problems right?.
I repeat if you are anything like stating up it career in mid 20s maybe.leave everything if you can .Forget all fucking frameworks and technologies start with basics of computing, right at instruction level using assembly .Then move to a higher language when u know and reason about what your CPU is actually doing.
If you can't do that and keep on crying and bashing down things wihout proper explanations fuck yourself with a cactus .5 -
I work with content. More specifically I work on content migration and improvement.
We connect to many platforms and pull and push documents into it. This one time we had to connect to some outrageously expensive (6 figures) system which we obviously couldn't afford to buy just for testing. The client wouldn't give us a testing server either.
My literal warning: "We need a testing server because we're gonna push it until it breaks. Then we know the limit." Client: "nah it will be fine." Us: "I promise you the server will go down..." Client: "It's a stable system. You can test in your own folder on our server"
10 minutes later we had an angry client because the server crashed due to overload.
I'm not sure if I'm annoyed or amused :p -
Well I probably should have been more prepared for this, but i was able to backup, upload and transfer my websites from my own server to my backup host provider while my ISP has been down for the past couple hours. Fun stuff.
-
So we use SAP for this little thing.
Booking your work times in a self service. And guess what?!
That shitty piece of software cant handle such an easy task. I mean seriously what the heck?!
They load everything from the slowest server the world has ever seen.
It litterally takes 2 seconds to click a button and something happens.
I need at least 1 minute for one entry! ONE!!!!!
Not to forget to mention it is down like all the time. Hey there are more than 4 users?!
Lets burn the instance down to the ground. Yeah thanks SAP cloud. Great job!6 -
Started up KiTTY to connect to my virtual test server per usual when I couldn't establish a connection.
Nothing too unusual so I do the typical troubleshooting I make sure host, port and authentication is all correct and it is. So now I open the display for the virtual server and start looking at ip info, host info, checking ports and everything is completely fine.
Now I'm getting frustrated so I start running things like configtest in apache, using systemctl to check the services status, even restarting virtualbox in my windows 10 devpc. Still cannot connect!
I start feeling hopeless and just shut everything down, the whole operating system.
*takes breath*
Computer boots up and I start my usual thing of creating workspaces, opening editors, starting servers, etc.
I open KiTTY again and launch my virtual test server..
konicm8ker@VM-UBUNTUSERVER:~$ _
Somethings you just can't fix without a reboot. -
If I have to change my domain password every 3 months for a bullshit out of date security policy (there's plenty of evidence suggesting that changing passwords is actually worse security), then maybe, just FUCKING maybe, make sure that that password change appropriately filters down to things like SQL Server so I can keep doing my goddamn work.9
-
!Rant #lazy
I setup my local server so I can test my apps and to mount my partitions on my www folder in a folder named "mnt" with numbered folders for each mounted partition
So I can access and download my files to phone and other devices from all over the network :)
And even watch movies that are in my computer on my phone or tv
And when I want to sleep I usually watch a movie or one episode of series which is stored in my pc ,on my phone by this method in my bed 😅
So the thing is I'm too lazy to get out of bed and shut down the server and pc
So I setup this file so I can run commands on my PC from my phone 😅😂😂😂
192.168.1.110/server.php?cmd=shutdown now&psw=mypassword7 -
A server's purpose is to serve clients. So if a server is down, it can't serve clients. So it's no longer a server, amirite?2
-
So glad to be staying a new job next week. Today a junior colleague asked me what the best way to test something would be as it won't work locally. Knowing this has a good chance of taking down the server, I suggest he sounds up a server on his AWS account. My manager comes in, oh no I don't want him doing it on AWS use the production server instead. By the time stuff States hitting the fan I will be gone.1
-
When you let your friends test you application. All of this happened in about 3 seconds. I directly shut down the server haha. Translation: "page ... couldn't be loaded."2
-
Discord...
Okay, I have a lot to rant about discord, but today, exceptionally, to the point.
I have my dedicated server. It has uptime last 3 years better than 99,99% (was down 15ish minutes for maintainance and RAM upgrade and like 10 minutes down becouse hoster's generators failed to trigger when there was outage)
This year it was up 24/7/365.
Why am I saying it?
Well, my TS3 server is up 100% of time this year. Yet still everyone moves to discord and suffers brutal audio quality and audio lags, and outages like right now. Its not first time this year and recently discord was acting up before. Today they scored bigger downtime than my dedi server (thats not redundant, not distributed nor any fancy "uptime helpers") last 3 years.
Why the fuck people prefer discord to ts3 other that it allows to upload images more conviniently? Okay, it looks nicer, and is like 10 times heavier on machine, but other than that? Its beyond me.
E: fix typo
E2: fix typo28 -
We had a meeting with the top IT folks in the company (top manager was in the meeting). A mix between operations (customer), engineers, and IT people were at this meeting.
The IT group was not happy that my customer had asked me to do some server work. IT wanted to control that. They wanted to shut down a server without regard to the customer using this resource constantly.
IT had heartburn about the system running Ubuntu rather than Redhat Linux. The top manager during the meeting says, "Why are you running Ubuntu? That is a gaming OS." All the other IT people who worked for them just looked at each other and us with that 'WTF? Dear caught in headlights look.'
My manager who was at same level as head of IT made a few comments. We got to keep our server and IT backed down. -
When you raise a ticket with your support team that your prod server is currently down and they mark it as a medium request, like dude... seriously?! Get it sorted ASAP paleaaaaase4
-
whole fucking day of coding.
git server down for maintenance so compresses folders / send. lost everything. if i meet a microsoft employee today there will be blood and piss. DIE U FUCKING FUCKS!!€ -
How would you feel if you can't solve your code? I'm almost a week stuck on a server problem. I applied for frontend dev position, it's okay for me to do this kind of stuff but the fact that I can't solve it for so long it makes me feel down :(5
-
"Hello,
it turns out we probably caused the downtime ourselves. I didn't know dropping 170 databases and deleting 80 entire projects at once could do that"
Gave me a hearty chuckle. Especially as the client adamantly refused to have SSDs installed for the DB to run on top.
Just imagining the poor read-write heads jerking back and forth in vain attempts to find all the data to delete... So yes, dropping 170 databases at once does in fact take a database server down to its knees, as deleting is all the drives will be doing for a while.
At least it wasn't my or my colleague's mistake this time.4 -
I already wrote this story in the comments to some other rant, still it's pretty funny.
So, i was modding my wii u by messing with system files, the only way to do this is to run an homebrew that hosts an ftp server so you can connect with your pc, the thing is, this server was not protected and allowed anonymous login, not an issue if you have a firewall.
BUT i had this console in the DMZ on the router for online play.
~You can see where this is going~
While doing stuff from my pc i started seeing a lot of entries on the console's log and i didn't understand why, i thought filezilla was doing some shit so i closed it but the log was still going, then i realized and quickly shut the server down.
Then i disabled the DMZ and went back to check what happened (unfortunately the log was only shown on screen and not saved, by closing the server i lost it) so looks like the attacker uploaded a lot of files with random names in pretty much every folder.
Due to the random names, I don't know which are system files and which attacker's files so i just kept them there. And today they should still be there.
--Random thinking--
What if the attacker also stole some files, he was hoping to get photos or valuable documents, but just got the console's system files and he thought something along the lines of "wtf is even this".
Also, how likely is that an ip scanner script gets to my ip in the 5 minutes i run the server, and (on the attacker side) how likely is to hack a server just to find out that's a wii u (the least successful console in the last generation)...
Moral of the story: double check your firewall if you are going to run an unprotected server.
That's it, stay safe :P2 -
Today i tried made some finishing touches on my discord bot. After dome struggling i finally managed to host it on heroku.
This is what happened after I put it on a real server and my dorm shut down internet access2 -
As many of you might know, the PYX servers are down. It happens that I run an Android client for PYX, that was obviously a tragedy for me. I didn't give up and I've immediately setup a backup server, then due to lack of resources I had to shut it down, but I provided a list of servers run by other people.
Yesterday I've updated the app once again to do some fixing. Today, this guy leaves a review: "Absolutely useless until they bring back the servers. Why update the app when literally NOBODY is playing??"; Why am I updating the app?! Jesus Christ, why can't I update my app?!?! Should I delete it from the Play Store just because some servers are down? I get it, there are a few people playing, but please, don't fucking say that the app is useless.
This kind of people makes me very angry. -
From now on I am administrating multiple servers in our company and monitoring is one thing our infrastucture lacks...almost completely. At least, useful monitoring.
Installing netdata or Grafana and integrate it with chat is definitely a solution, but what happens if the whole server just shuts down (very stupid scenario I know)? Well, it is easy, there will be no alert about the failure.
So, that's where I was wondering if there is a tool or even better plugin for netdata or Grafana, that enables remote monitoring from another server? I surely can write a simple script to check the server availability but having the whole monitoring tool on a single server instead of 5+ would be also easier to maintain and setup.10 -
So a few days ago I sat down to write a redis adaptor to transfer data back and forth between redis and elasticsearch. I download the go-redis package and start writing a simple client.
I run the client and it gives me an error. So I'm stuck at it for about 30 mins and then I say to myself, "You dumb fuck you haven't started the redis-server". So I open up another terminal and type in `redis-server` and then I realise I don't even have redis installed on my machine.
I do such dumb things every weekend. If you have any dumb mistakes you made while writing code please share them in the comments. :-) -
At my workplace a while back we got some new interns.
I had been explaining how to use the commands on our Linux server. While I was in mid-sentence talking about what shutdown does, the intern inputs the shutdown command; shutting down everything... not shortly after, I start getting bombarded with emails and calls...4 -
Reporting server connection to database is down, probably due to a user access restriction.
reported the issue to the India sql datacenter and got back: Yes, We see that the connection is down. ( I sent them screenshot of it including the error message ) There is no such database available.
Me: Yes, well I'm in the db working right ( send screenshot) now.
India: ..... disappear offline.1 -
Trying to install Centos7 onto my proliant g6, red screen, try a fix, red screen, try another fix, red screen, finally find a fix that seems like it is the exact problem, screen dies can't see bios... god damnit.18
-
The story of a normal release:
- tool gets tested "intensely" by 3 ppl quite a long time - everything works
- a major 2 days reserved as maintenance window for even more testing
- release starts
- first the admin panel of the server suddenly is not accessible anymore
- after some problems the tool is deployed
- suddenly servers are down and not pingable anymore - off on off on (provider has major problems .. good job)
- ppl start testing
- testers report lots and lots of new bugs - seems like the testing wasn't that intense after all...
- people start coming with lots of new requirements (oh we need to import those excels.. excels don't match our internal stuff.. )
- confusion over confusion
- getting pissed of a lot...
- quit caring and focus on another project
- profit
Fuck my life -
I wonder how fast the mail server pewdiepie uses will go down, if he turned on notifications for every like, comment etc.2
-
Let's see what's on the menu today:
* Web Application Catastrophe Special *
Includes, but not limited to:
- Orphaned server processes in the configuration management cluster
- Microservice back-end architecture with no API documentation
- Poorly implemented cache microservice with no documentation
- Stale data causing everything to be shown as down in production, despite everything running fine
Cost: 1 developer's sanity -
I had a nightmare this morning in which I had been working on a project for months without pushing my commits to the remote server. Then the office suddenly burned down and I had to tell my boss that months of work was lost because I was too lazy to push. He was sooo mad. 😬1
-
You know that time when somebody had a problem with a system you wrote years ago, and it has taken you an hour to try to remember how to even call it, because the documentation and code didn't get migrated from svn to git, and the svn server has been shut down for some reason, and the admin is out today, and the last time you had the code was three machines ago, so you're trying to gleam what needs to be done to just call the stupid thing from log files set to 'error'?
That time is now. -
Fuck you Mobike!! Stop polling your applog.mobike.com server literally every 10 fucking seconds when your app is not even running! And why the fuck did it suddenly start last night, of all things? It's fucking creepy and invasive.
Related question: does anyone know how to permanently shut down a background app service on an unrooted android 7? Even if I kill the background process (not the app itself - that one has not even been opened), it still restarts a minute later3 -
I though you may like what I did
The AI finished training now
My server almost went down
I put a semi-colon at the end;
I think you can't create front-end
I haven't had sleep in a while
Neither have I walked a full mile
I'm more of a 'kilometer' guy
We are going to eventually die
Fuck PHP
I can't afford to buy full HD
I'm not good at this
Cause I don't have a girl to kiss -
Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.
I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!
I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!
I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?
I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).
With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....
So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...
* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.2 -
I'm still using .then().catch() instead of the async await.
So, first of all, Fuck you for calling it a STANDARD now. its nowhere near to be called standard. You wanna get some data from an API? Wanna call it using axios or fetch? What if the server is down? what if there's an error that you don't even know existed? Where do I get that kinda error in async await? try-catch? no thanks :| I'm good -_-8 -
So I was looking at ups's on amazon and I scrolled down to the questions section and someone asked "Would this computer be good to host a game server?" I honestly have no words😂
-
TL;DR The "senior dev", that the client hired on their end, is acting as a middleman between me and the project requestors. Taking the credit for my work.
I've already bitch about this before. I've been in a crusade to defend the production server from this fraud for a long time now.
But most recently he has removed me from all meetings with the actual project owner. I create the solutions, then he goes through them to understand it a bit. He proceeds to present it to the project owner in a way that almost blatantly says that he made it.
I'm sick and tired of working with this asshole. He is literally useless, worse he is slowing things down and breaking others.
I'm just gonna begin countering this... -
So a SQL server walks into a bar to grab a drink. He gets his drink and spots a table with two people sitting down with a spare chair. So he walks up to them and says "Hey, can I JOIN you two?"1
-
The test server at work was going up and down all day. Not great when I was trying to use my new dashboard.
Long story short I found out the hard way to make sure I closed my SQL connections -
In a real-time multiplayer competitive game where you control a vehicle, is it feasible to simulate the whole thing on server side, such that the client only sends controls and receives sensor results? I mean like the client doesn't even know its own precise rotation, just the readings of a gyroscope and an accelerometer which are both susceptible to errors, and deduces the "down" direction from those two and approximate control forces. This would both solve hacking (writing a good robot is just as challenging) and lead to fun results like an attitude indicator going crazy from a gust of wind.14
-
Not sure if it should be a joke or a rant, but something rather funny (at least to me) happened today.
TL;DR; Someone's outlook was crippled by 100k+ of warning notifications
So we have developed a server that has an internal database that wraps around an elastic search instance, that is managed by a POS vault/storage solution, that we have to use for legal reasons. The elastic search is "provided" by the software, but we keep this internal database just to be sure and totally not because it's unreliable POS.
Anyways, they take data integrity very seriously, so every warning our server produces is emailed to someone in charge to review it and if necessary forward it to us. This will be important later on.
Couple of days ago we got error forward when trying to write an entry into the POS software we get an error, because an object we tried to write already existed. After some investigation we concluded an entry was missing when the internal database was created, so we asked them to repopulate elastic search to solve this problem.
When start the server we always sync the internal database to the elastic search and emit a warning when an entry is missing in internal database or vice versa. And well... almost all of them were missing, which caused our server to emit ~40 warnings/ms. Poor outlook. Still investigating for the cause, but damn, I never expected I'd take down someone email account by accident -
I spent 2 weeks at work building a dashbord (not a feature but...) wich provided an overview of projects and tasks managed in redmine (Kanban tool) by using its API. After i finished it we started using it till my boss found out that it was completly "useless" for him - it had all the features he asked for! -.- ...
three days later the redmine server crashed and we changed the provider, nobody missed my dashboard and so it got abandoned :( sometimes i miss it, it looked fancy af and stuff!!
But at least i learned a lot of js and API stuff. I was verry new to js back then :)
Boss asked -> I deliver -> and *pooff* down the river it goes. seems like my tasks have not changed much since then. -
Production goes down because there's a memory leak due to scale.
When you say it in one sentence, it sounds too easy. Being developers we know how it all goes. It starts with an alert ping, then one server instance goes down, then the next. First you start debugging from your code, then the application servers, then the web servers and by that time, you're already on the tips of your toes. Then you realize that the application and application servers have been gradually losing memory over a period of time. If the application is one that don't get re-deployed ever so often, the complexity grows faster. No anomaly / change detection monitor can detect a gradual decrease of memory over a period of months.2 -
university server will be down tomorrow morning around 8am to 12pm. i wanted to finish my code tomorrow morning :(
more sleep for me though ! -
!Rant
Just for blatant curiosity; how many people would ++ this just to help me get a stressball to get through this day.
Rant
So today I woke up to a message that my jobs webserver was down; and that all personal webpages for the Uni went down with it. We had no idea why or how it went down but eventually someone got it up- just in time for our Proxmox server to go down; which runs our new web server (thankfully it's not live yet), our print server, our PXE server, and more... Loving today. Such stress. Very shit.
We have a memo on the window "Watch your FPMs" (fucks per minute) -
Technical question that I just cant find the answer to anywhere.
I have a load balancer and want it to pass the IP of the original caller to the server. Usually it is done by modifying the header? of the Request HTTP packet? and adding X-Forwarded-For: ....
The LB team though says it needs to modify X-Originating-IP and somehow causes a noticeable impact of the speed of all requests.
I don't know the details but it should only modify the first Packet that has the HTTP headers and should be appending X-Forwarded-For. If only need to modify the Header packet, how can it slow down the whole interaction so much:
-Adds 100ms to a 200ms request
-Increases a 10 minute download to like 20-30 minutes6 -
Being told to continuously restart a server over the weekend while someone's API script kept crashing the server. Yes, they knew it was doing it. No, they "claimed" they didn't know how to slow down the amount of new API sessions. I avoided all their requests after that.1
-
I'm finally going on vacation, which is good, but I already know there will be many stupid people writing me endless private messages about them being banned or their server not working correctly BECAUSE OF THEIR OWN SCRIPTS and they will spam me like they usually do although I WRITE THEM I'm out of the office but they don't care since they don't have other things to do than COMPLAINING ALL DAY.
Geez okay, I got to calm down, I'm on vacation, don't want *that* to ruin my mood.
It was stupid to ban all those people for breaking the rules today though, should have waited another week, so guess this one is on me.1 -
Got a phishing email with a link to a website hosted by wix. The only thing on the site was a form and submit button so I’m sure it’s for collecting credentials. I was able to report them and wix shut it down which was nice. But I was thinking, if someone were to ddos the web server, what action would wix do? Would they let the requests keep coming and increase the customers bill? Or would they just shut down the server?2
-
Grrrr, at the end of the day just before weekend trying to arc land something so you can start fresh the next week. Boom phabricator server down. Unexpected reply.
-
"Boss, we have a problem!"
"Don't say that, we call it -opportunity-"
"Boss, we have a lot of opportunities, the server is down."
"Ow..that's a problem"2 -
Forgot about a System.Exit();
Was wondering for 2h why my Weblogic Server shuts down immediately after deploying my Webservice...
I never felt so bad at Programming..1 -
Ugh, I feel like fucking crying every time I have to explain to the other developers and network sysadmins that we can't have the same Redis server for both session and caching data in the current project we're working on. I wrote all this down in the specifications which NO ONE reads!!
There are times like these I wish I just had access to the AWS account so I can do all of this myself.3 -
Screw AIX! More importantly screw the IBM designer that though cache batteries were a good way to monitize their platform to help validate the service contracts. I guess it "works", but at what cost?
Just lost the last 4 business days going down this rabbit hole with a customer's server.
Edit: Quick note, yes, the customer is on track for a migration soon.8 -
Can github (not google) copilot be run on a company server? Or some scaled down version?
I could see this being useful for a version that only parses the code the company wants it to parse. Like an existing codebase. I often find snippets on how to solve something in the existing codebase, but it is usually hit or miss.9 -
So I developed this proxy server that will throttle down API calls to one of our providers so we don't get blocked for TOS violation...
Some dude had a tool running all day long which crashed 2 minutes before I left work.
This literally ruined my day until I recalled it's all cached!!!!
Mood is back again and I deserve my beer! -
Had a client whom was using the staging system on my server as cdn, remote computing, etc... because his prod server was a cheap vhost while the vm was a beast compared to it. I shut it down without telling. I just got a call that his site is now slow a f and full of errors.
I kindly told him that there was a recent security breach called dirty cow. Then I told him that I shut the vm down because it would mean security risk for him since there are no patches available yet and only Power on again with there was work for me to do.
If you want resources pay for them -
Jr a little low while trying to do some stuff. Took down the prod sql server for a while. Some disturbance but not the end of the world but she’s a little low.
I am like: GREAT WORK!!! NOW YOU ARE LEARNING!!! 💪
Time for cake and celebrations!1 -
First let me start this rant by saying: Don't use SharePoint lists as your primary data store if you can avoid it. You're gonna have a bad time.
My coworkers and I work on a system where we need to pull tons of data down from a SharePoint site and run various algorithms and operations on it. Generate reports, that sort of thing. This is all done in the browser using a Typescript React SPFX webpart. Basically using SharePoint as a DB/DAL.
Because of the sheer amount of data we end up pulling down (our system in production is the single source of truth for one of the largest companies in Canada, and they're currently building a pipeline as we speak), in order to maintain a reasonable speed while using it, we have some pretty intense caching logic implemented, logic that ensures we get new items when new items are detected, and merges changes to already exisiting objects. It's pretty brilliant, and that's before we even consider the custom paging that my coworker implemented in order to get around the IndexedDB max size of 100MB.
Well that's all well and good, and works great in production, but it is a horror to work with. Because EVERYTHING we touch on the server is cached locally, it can be IMPOSSIBLE to detect data anomalies, be they local or server side -.- You don't know how many hours I have completely WASTED fixing a "bug" that didn't really exist... Just incorrect data in the cache12 -
I'm working on a JavaEE Webshop (Uni assignment) that has to send and receive JMS messages to and from a server, which is located inside the university network, so I have to use a VPN to run the shop. The problem is, the VPN is so goddamn slow that I get SocketTimeOutExceptions regularely! I have Gigabit-Connection, but with the VPN it slows down to ~1Mbps for whatever reason, which is apparently too slow for Java.2
-
So I follow Linus Tech Tips and set my computer's DNS server to 1.1.1.1 but the dumbass in me didn't set any backup servers.
Come Friday night, internet is not working on my computer. After a modem/router restart and it still not working, I thought it was just the internet in the house was down for a little bit (it was connecting to the router perfectly fine). The next morning I wake up and my phone's connected to WiFi and it's working, so I'm like, "great, internet's back"
Not for my laptop lol. Nothing's loading there. Since it's just this device that's having trouble, I decide to forget the network and log back in. Still not working.
I finally remembered my DNS server setting and add Google's external DNS servers to the list and now it's working.9 -
When did you last shut down your linux system? Mine, apparently was on June 20th, would have done to apply some system updates. (This is my personal system and not a dedicated server)1
-
Angular and MVC really slows shit down, doesn't it?
I added 1 (one) page which displayed two data grids,straight out of a database procedure. Here's the head count:
- 13(!!!!!) New files
- 5 old ones modified
- Build process on both server and client side (Visual Studio build and ng build)7 -
God, these people...
Little backstory. I'm making an training application and we have a MySQL database set up where some elements of the training are configured. This is so learning experts can easily change some aspects of the training without programmer's help.
Meanwhile, I'm also in the middle of a server migration, because our current server is running a lot of deprecated software and is in dire need of replacement.
This is going pretty slowly, though, because of other, high-priority, work that keeps being shoved my way.
Now, someone accidentally deletes a bunch of data from one of the schemas. No big deal in my book, the training is still in development and we have nightly backups of the database.
So I shoot a support ticket to the hosting provider and ask them to restore a specific schema, telling them to restore the image to some other machine and dump the tables in an MySQL file so I can restore it that way.
I also told them to get the backup of the OLD server, not the NEW one we're still migrating to.
About an hour later, I get a message that they dumped the schema's files in a Temp folder on the D drive. So I RDP to the server to check and... The files aren't there. Just before writing a response asking where the file is, I remembered the server I was migrating to and checked that server, and there were the files.
I had already migrated part of our databases and was testing compatibility before I moved to something else.
The hosting provider just dumped the files of the wrong server, despite me telling them exactly which server to use.
This is not the first time this hosting provider has let me down...
I'm really considering jumping to another if they keep doing this... -
Question: which is better approach?
1. Use push notifications to tell react website that new data is available
OR
2. Use SignalR (similar to socket.io) and push data in real time?
The down side of SignalR is server needs to map connections --> use more memory while push notifications doesn't require that12 -
Composer.json require sendgrid
Composer adds wrong directions to file, fine I'll hard code it.
Composer is deriving file path.
Fine I'll edit 4 files.
Composer is escaping hard path
Change global variables
Composer is still adding its own directory before hard path.
Follow azure and sendgrid documentation to the letter, composer puts wrong way round slashes in file path.
Gives up on 57th server 500 error
Sometimes azure gets me down in its implementation of things.... -
This was in 2001 on a legacy AIX core ops server. I got tired of waiting hours for the last page of a print job to debug totals. So, I added a print menu option to print it to an HTML file on a share served up by a wab server and send me the email link. Took two years to catch on but when it did, we eliminated all nightly print jobs and took the paper budget down by 90%. All because I was too lazy to keep reloading my desk jet with paper to debug reports and I forgot to take it off the menu.
-
Today one of the guys at work said
"The dev server goes down because we make changes all the time - continuous disintegration" -
Client's WordPress powered site has gone down due to influx of traffic. Site is cached and whatnot, the server it's on is simply underpowered and not coping. Whilst I'm frantically trying to spin up new servers and load balance and get things back online, client keeps interrupting with emails like "URGENT - Website is down", "How long is this going to take please?", and "Hello! Do you know whats going on?!". I reply cordially to each, secretly growing increasingly enraged. "Yes I am already aware and working on it.", "Difficult to estimate, but I'm hoping no more than 20 minutes.", "Yes. I'm working on it.", "Yes, I know. I'm working on it."
And then I receive this gem: "Perhaps call Word press?"
AARRGGHHH
This scene from Clerks springs to mind: https://youtube.com/watch/...2 -
Sprinting all week long against time. friday afternoon, committing time. connect to Git to check latest changes. Server down.
New IT Guy pushed an update @4PM.
He's been missing for a week. -
Friend keeps gold coin in case the economy tanks. I have a home git server in case the Internet goes down.1
-
In banking industry it brings up security concerns. We were in the exact same situation, however using SAS+SPDE with some custom SAS and tsql queries. Our database was merely 100TB, still it was a nightmare to assure stable performance thoroughly, because SPDE could not properly handle SMT. After having 24h++ daily flow processing times, the managers have decided to rent a 6 years old IBM power 7 with dedicated processor cores, which eventually have truncated the processing time down to 15 hours. This was a time limited contract, for 6 months. I've left the company in a short while, but this made the managers to rethink buying a more up to date server, so now the daily processing flows now are around 11,5h. Long story short, sometimes a little architecture optimization does the trick.
-
for the 3rd time ive tried introducing some version control on a project that really needs it because it has multiple people working on it.
And because the last time my efforts got shut down because in practice people thought it was too much of a hassle to develop locally rather than on the shared development server directly, I made a feature that would let people checkout branches on said server...
Apparently the action of; saving > committing > pushing to your feature branch > merge after aproval, is still too much for people to comprehend; "I think this is too convoluted can't we just keep pushing to the production server to check our work and then commit and push to the master branch"
So I just got pissed and said fuck it, no more git then, I'm not even going to put any effort into changing tooling here anymore, and this is a massive project where we have to manually remove code that isnt ready yet from the staging environment.
Are the people I'm working with just this stupid or am I really overengineering this solution because I think 4 people should not be working on the same file at the same time without any form of version control and just direct upload to FTP.
(and yes, I know I should leave this job already, but social anxiety of starting at a new company is a big obstacle for me)3 -
You guys think Windows 10 data tracking is bad? Server 2016 is worse. When I try and shut the machine down (it's a VM and I want to change config) it actually asks me the reason for the shutdown. I understand servers are intended to always be up but geez that's overkill. And this is only within my first hour of installation and use.4
-
Android + servers
Hey guys
So cause I barely have time to code I mounted a server with my old cellphone, so I can advance one of my projects at work (it's a helper with stuff like tables and calculators for work [CNC machine] )
The cell phone is a meizu m2 with android 6 (don't buy one, has lots of stupid bugs)
My problem: android terminates the server and a app I use to copy files from Dropbox to the server folder (only work arround the home rooter)
F king meizu bugs : resets lots of definitions to default, like I give permition to the server app to always be on and it changes back to default when I turn off the screen.
So when I turn of the screen the server goes down
Solutions to keep the server always awake?
Also better solution to change /upload the files without Dropbox as an intermediary
Btw the app that syncs also turns off every fucking time (so no updates till I get home)1 -
Team member takes names inputted and coverts them to slugs, works perfectly and functions extremely well.
Adds function to allow slug modifications but doesn't use regex or anything else to verify input is a real slug, application goes down over weekend and know one notices till this morning...
Always verify input both client and server side!!! -
ELSTER, Germany's federal tax office online service seemed to be one of the few examples of successful digitization. But don't wait until the last minute for tax declarations online when everybody else has the same idea and federal ELSTER server slows down and throws a lot of exceptions turning into a federal point of failure today.1
-
A dev decided to overwrite the master branch with his code saying its better. That it fixes the major bugs that all of us couldn't solve.
Against my better judgement of firing him, I decided to test it.
Firing up the testing site, we made test databases to use and we went to house.
In the middle of testing, I noticed the test DBs weren't being changed. While everyone was still testing, I looked at the code. It wasn't made to test on any databases, it was specifically designed for the actual production server.
However the damage was done. In a secret dashboard in the code, someone sent instructions to drop the tables, effectively ruining the production server.
We had the dev go to an offline backup site that only went online every 10 minutes a day to make new backups. So we shut down the production server, setup a maintenance page. I get my ass chewed out again, and we were sitting ducks.
I don't think the dev had enough punishment, so I grabbed his laptop and made a full backup of his data, and locked the SSD in a safe.
I downloaded a Windows 98 and put it on a flash drive. And installed it all on his SSD. The dev is now a proud (pirate) owner of Windows 98.
He came back and started balling on his desk. We all looked at him with a pity, but he deserved it.
I'll give him the drive on Monday.
Do you think he learned his lesson?7 -
Fucking dot files...
Written a deployment script to reduce the amount of another dude's fuck ups when updating code on the server. Apparently the website executable automatically generated TLS certificates (let's encrypt) and placed them into the local hidden folder.
There is a limit on how many certificates a single domain can generate so... The website is down...7 -
So jelly of all you people out there with a good enough internet connection to setup your own server environment at home. Sitting here with max 10mbit/s down and 2mbit/s up7
-
Ranters, I need help
Does anyone know how can I run something as a systemd service and still have access to the "console" - the input and output of the program, when I need it?
(It is a minecraft forge server and there are multiple people managing the physical server and one of them managed to corrupt the map yesterday by rebooting the machine without stopping the minecraft server... so there comes the need for a service so it can shut down before the machine reboots... but I also need the access to the console of said minecraft server because sometimes when some modded items are misbehaving the rcon will stop responding while the console still works)
Edit: currently I am running the minecraft server in a detached screen (ddg "screen linux" if you don't know what I am talking about)5 -
....that moment, late at night, when you've just closed that remote terminal to your network media server, tweaking that custom tree listing script, and then you put your phone down and roll over.
:|2 -
Error: Disk image full.
...
Not good on a production SMB server. Wasted 2 hours tracking the fault down....
Turns out the boss thought it would be a good idea to install syncing software as remote backup without telling anyone...
It wasn't remote :( -
someone gets in my Discord server, asks "can anyone download a file for me? DM me" in the bot trap, and leaves.
Is it worth the effort of trying to track this guy down so I can get new malware or is it not worth the risk of CP?4 -
The day when you Plan an Alarm clock which will awake you when one of your server have trouble.
Hm I think that should look like this:
https://youtu.be/umtf9bx8cuo2 -
Was working today when all of a sudden the background on my laptop and monitor changed from its usual mostly black image to bright white. Tried to set the desktop background, but it had no effect. Honestly thought I broke my computer for a few minutes.
Turns out one of the ubersichts (custom desktop widgets) was attempting to curl a page that was "Down for server upgrades". Definitely one of the weirder computer issues I've had. -
My task for today: RDP into a vm of windows server 2008, use winSCP to ftp into the client's server, pull down approx 900 images, renaming the 200 or so duplicate filenames then upload them to a file sharing service...
THEN I have to pull them down onto my machine, optimise them, and then reverse the entire process, again manually renaming the duplicates. -
It's okay to slow down. Seriously it's fine. You can write 203 wpm... as long as the command you enter includes all necessary limits. Nothing is worse than having ansible rm -r the wrong directory or the wrong server because you missed a limit and put all.undefined wk26 learn from mistakes devops ansible take your time time is both the solution and the problem
-
Fuck you, whoever kicked me out of LinkHub Discord server + revoked repo access... i even contributed. I don't care if you shut down everything, just say something before kicking me out... rude mf3
-
Gah, I just received this Ubuntu 18.04 VM with 8 cores and 8 gigs of ram, and since it'll be a production server both serving public and "private" networks (yes, shout at me, but projects won't be about hosting sensitive information, I wouldn't put all that on one server), and I'm struggling between my options.
Docker, or not docker?
The server's main use is to host our growing blog and install Varnish, which will hog some ram after a while. I use Laradock for my dev projets, it's really easy to develop with it, but I am unsure if it fits a production environment with performance, security and traffic load in mind :(
I read Docker has stability issues (in 2016-2017), and can bring the machine down with it, I don't know if I should just install the software (nginx, apache, percona/mysql/maria) without "containerizing" it and go for it
I'm lost xD7 -
Databases and LDAP down since 1 1/2 days...so embarrassing...am i really working in an it company???
luckily there are options beside work...hello amazon, spotify, devrant...:D
if we got Server/DB issues it always takes about at least half a day to fix it! *facepalm*1 -
I want to start learning to write a simple game server emulator in C#. The game works LAN but it gets LAN disconnected when internet drops so some sort of keepalive is implemented. I can copy the files to another device and it works online without a login etc so there is no online authentication but as soon as internet drops the LAN game goes down to so i need to emulate the online update server or something like that to prevent that from happening. (spotted with Wireshark etc)
I don't have much experience , just created a simple tcp client/server console app but in this case I ofcrs will only need a server one in combination with custom dns. Any tips on where to start? Does someone have an example game server emulator? or update server emulator?1 -
A bug on a chef script which stops the web server in case if it was unable to talk with the database. One night the database hits the maximum number of allowed connection and a good number of web servers went down at 3 am.
-
ant.design selectors are bogus garbage.
The drop-down selector that replaces the browser's native one does not allow typing to select an entry, meaning to select a language from a long list, one needs to manually scroll to it. If the scroll wheel of the mouse does not work properly, one needs to use the scroll bar, which is far too short to be able to conveniently scroll a long language list.
Sure, ant.design might look pretty (as advertised), and has oh-so-fancy features like fade in/out animations, but from an interaction point of view, that's as useless as the skeleton screens popularly used by JavaScript-based websites (which are anyway inferior in performance and compatibility compared to static HTML pages with JavaScript on top).
Not only can I not type-to-select, but the date selector on Dailymotion, which uses this utter garbage, sends "[object Object]" to the server, so the user is forced to edit the HTTP request manually. Complete utter garbage.
Don't use that shit. Use the browser's native feature. Or use something progressively enhancing like the drop-down menus used by MediaWiki on pages such as Special:Contributions, where it actually is properly implemented.2 -
is it bad to put your jsx in useState for react?
const = handleClick = (sort) => {
if(sort.props.id == 'default'){
setSort(<ArrowUpward id='up'/>)
}
else if(sort.props.id == 'up'){
setSort(<ArrowDownward id='down'/>)
}
else if(sort.props.id == 'down'){
setSort(<LineAxis id='default'/>)
}
}
the other alternative is using template logic which i think looks uglier, someone told me i did server side rendering in the browser2 -
Yeah, it's Friday morning and guess what I left my laptop work yesterday and who just got a text saying that the server is on their ass.. yeah you are right ME. And my team who can do the same job as me on restarting the application don't ever take action on this kind of thing... Well I hope they will in a few weeks because I will be gone then2
-
hi all
my friends and I are creating a game! I've had prior experience making games before, and my friend who does all of the modelling has some experience with Unity Snaps which we can use to make some buildings, but we've never really tried to make a team project before, until now!
we're making a top down urban warfare game and we're making quite good progress so far! we are done with configuring a server to host, player movement, and almost player shooting
I would post a link to the discord server but I'm not sure if I'm allowed to advertise and could not find the TOS. anyway yeah please tell me if I can post it3 -
Testing new server deployment in test env all works, then production it all breaks down. Network didn't allowed the right traffic. Took me whole week to find that out. Until some networking engineer said, you know there is a firewall between those networks?
-
I have an iot solutions company which has 0.5 millions hits per day just from 7 devices. It is on shared server and it bogs down every now and then. In future it is going to grow to 100 million hits.
What kind of cloud instance on aws should i use. Or is there any special hosting for iot devices.4 -
It is currently 5:30 AM, I've been trying to upgrade a server by one single major OS release for the last 3.5 hours. All major apps were no issue, except one.
Redmine.
That thing runs on Ruby.
Ruby, as it turns out, doesn't really like the way Debian handles package management.
And now, I have Redmine that, even if I completely uninstall / reinstall, won't start
I went down several rabbit holes, trying to operatively find what the issue is. But I never got to the key issue.
Fuck ruby. Fuck Redmine. Fuck nothing-saying error messages. Fuck bundle. Fuck gem. Fuck it all.
I'm redirecting the Redmine domain onto the server backup I've made. Upgrading that thing is a nightmare.
Maybe now I can finally go to bed...5 -
I've got a dev server where I run some test sites in WP using EasyEngine, because I want to get accustomed to WP in Docker.
It asked me to update, and I was like "sure". Now whenever I want to setup a website I get "easyengine couldn't create username"
I figured ok I'll use WordOps, which requires migrating from EasyEngine to it. I was like sure, and next thing I know the "migrated" websites that it was supposed to properly migrate automatically are down, and I can't get an SSL issues for my new site.
All threads on both issues don't help.
It was supposed to be a 5 minute job and it turned into 3 hours trying to troubleshoot. Now I'll spin up a DigitalOcean server and install a quick WP site.
Fuck both EasyEngine and WordOps <3
I thought EasyEngine would be cool but seeing the very limited community activity it's not worth the risk even having it in a dev environment. -
I used parse-server and services back when it was a web service at an internship, just loved the way it did things it did. Backend as a service was new to me as a mobile application developer. 5 years down the lane. My first go-to backend is Parse. I know firebase does XYZ things better. But I love the simplicity and openness of parse.
Community picked up parse as a self hosted open source service and its still going strong.
Just love the possibility of starting a mobile project and not having to worry about setting up a whole web service to cater to it.