Big event. Massive traffic in production, so we were monitoring all night.

I was in a room with 2 devs of my team, a marketting girl, my boss and a designer... chilling.

Suddenly the production is down.

Boss: production is down, anyone can check?
Me: already on it
Dev1: it looks ok for me
Dev2: me too
Me: wait what? Impossible everything is down
Dev1: oh I refreshed the page it's not working
Me: don't stay on the page refreshing it like you are fucking monkeys. Give me useful intel or be quiet.
Market girl: is it working?
...
Guys is it working?
...
Hello?
Me: Not yet we are looking. Don't distract me.
Boss: client called us. They want it online now.
Dev1&2: he's looking
... 1 min later...
Boss: is it working?
Boss: is it working?
Boss: is it working?
Me: SHUT THE FUCK FOR FUCKING ONE SECOND. ALL OF YOU, OUT NOW. YOU ARE FUCKING MONKEYS WHO CAN'T DO SHIT. IF YOU CAN'T HELP JUST SHUT YOUR DAMN SHITHOLE. DEVS, LOOK WITH ME. MARKET GIRL PREPARE A FUCKING POST-MORTEM MAIL. BOSS GET THE CLIENT ON THE PHONE AND STALE. DO. YOUR. FUCKING. JOBS.

That's how I ended up screaming at everyone... the rest of the night went in complete silence and I fixed the issue 2min after the got quiet or busy.

We have a lot of monitoring screens on walls at my work... Some of them show charts... The one showing the last two months of mailbox database growth has started taking an odd shape.

ON MONDAY

TL: Why the hell you require a month to integrate this engine?

Me: It will take that much time, can't help it.

TL: it can be done within a week.

Me: Then you do it.

TL: Ok I will show how it's done in a week.

ON FRIDAY

Me: What's the status on that integration?

TL: Oh yeah about that, you have to carry it ahead, I have some monitoring to do.

Me: Ok, give me the repository access, I will carry it ahead.

ME OPENS REPOSITORY.

There's only a new controller file with nothing it.

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

Mother of god, as if the new mass surveillance law in the Netherlands wasn't bad enough, one of the politicians who likes the new law has come up with an even more redicilous idea.

an 'Internet Authority. To put it short, an authority which surveils the internet in real time and sees where all social media shit is coming from/going.

Meaning that it wouldn't just be 'targeted mass surveillance' sometimes but fulltime online monitoring.

This guy has lost his fucking mind.

So my landlord just came up and asked why I'm using so much bandwidth (they've just had a new line installed so they're monitoring it like hell for some reason) so we had a chat, I told him I'm a Web Developer so I'm uploading and downloading a load, and bare in mind this is student housing, he offered to install a wired connection in my flat only so I'll have a decent and stable connection when all the other students come back in September.

This is the first time in my life I feel like I'm not paying enough rent!

I accidently left log.debug("bollocks") ;
In an exception handler our customers log monitoring system picked it up and they questioned why and I quote here "why is there a spike in bollocks at 3am?"
That was an awkward conference call

When the Project Owner wants monitoring and asking the IT without clear specification 😅

FBI: FUCK!! We just lost our connection to all those civilian routers we were monitoring
FBI IT Department: Did you try turning them off and back on?
FBI:

I wrote (or, ended up with a very much alpha but usable version) a monitoring system a while back and completely forgot about its existence.

But, it's still running and a few days ago I was building a docker imagine on a system with not that many resources and after about 5 minutes I started getting notifications about a high load!

Then, while I had forgotten about it again, yesterday, I suddenly started getting notifications about websites on my main application server going down.
Logged in and all was good again after restarting nginx.

Gotta say that it feels quite awesome to be notified of shit going wrong by something I wrote myself while I forgot about its existence 😊

My uni implemented Bluetooth beacon based attendance monitoring.

Raspberry pi + cloned beacons = 100% attendance.

Idiots 😂

(Edit for clarity: app on smartphone, Bluetooth beacon in each room)

"Another one bites the dust.." BEEP BEEP BEEP!
😬 "Shit, make that two."

Down from 5TB redundant to 3TB non-redundant and still waiting for my new Reds. No choice but to shut everything down and wait. 😩

Working on a (PHP based) monitoring system and currently writing rules/testing stuff.

I wrote some rules which check a few pages of a dutch site displaying if certain bigger services have disruptions and I am pulling the pages of a few ones I'd like to be notified about when they have issues.

Started the engine and received an alert about one big ISP over here from the monitoring system. Didn't believe that it would work right away so went to check that specific disruption page and...... they have a disruption right now!

IT FUCKING WORKS.

Good monitoring system 😊 *pats system*

To improve our user's "experience" I suggested to my boss to add a status page showing...well, the current status of our services. Everybody was up for it, so I go off and implement a basic version + automated monitoring backend, get lots of positive feedback, all seems fine.

Then it starts:

Boss: "Can you get it all set up by this Saturday?"
Me: "Uh, today is Wednesday and I've never set up all the stuff needed on a proper server before"
Boss: "Well, you still have a few days. Please also contact your coworker to get it all hooked up in our launcher"
Me: "I'll try, can't make any promises though"

Contact my coworker and tell him what the plan is. I had already given him access to the repo and he is positive to get it all hooked up (I doubt he ever cloned my repo, let alone ran my code)

Spend all Friday getting my stuff set up on the production server, feeling pretty good thanks to the many tutorials.

Contact the boss Friday evening:
Me: "All up and running"
Boss: "Thanks, but we decided to go with a basic HTML page instead. We can just manually edit that, should be enough.
Me: "..."

In the end my stuff was never used, the server I set up was finally taken down a month ago. The gratitude you get when not hacking together some absolute shit that causes problems when you don't add <br/> tags at the correct places to prevent an ugly overflow, cause the coworker was too lazy to implement some form of line wrap in the launcher. I'm not saying my stuff is the best of the best, but at least it was professional looking to a certain extent.

!rant

Quite the opposite of a Rant actually. Very good day today! Had pancakes to start the day, finally got a second monitor for only 5 bucks - it is so much easier to program with live change monitoring!

And, I finally got my very own server rack :) just wanted to share!

I'd love to finish a few projects I'm currently working on:

- An add-on which gives a middle finger to websites which use services/products ran by companies which are known to be integrated within the biggest mass surveillance system ever created (US powered). Not because just fuck those websites but because I think (@PonySlaystation came up with this idea) that its only fair that people get to know which websites 'sell them out'. Oh and "but not everyone cares about that" - you don't HAVE to install the addon.
(will be open sourced)

- Notes service with a fun thing.

- PHP based server/website/whateverthefuckyouwant monitoring system which is pretty much module based and works with json files as configuration. (kinda works but still loads of bugs to solve and gotta improve the module system a lot).
(will be open sourced)

- PHP based pihole alternative which suits my needs (will be open sourced)

- Forgot one 😅

Update 3:
Nice little command line monitoring thing set up :)

> me on call
> had to much wine
> suddenly, phone starts making sound
> holy shit, I am on call and I am not supposed to drink
> Calls taxi because shit is looking bad at work. Everything look down according to the sms I get, I had to pay the taxi from my own pocket.
> 40 min later, arrive at work. Walk into the serverroom.
> Go to the the server that is monitoring everything.
> Check networkcable, it is loose. Push it a little.
> Goddamnit, that was it.
> Realize I never doublechecked if stuff really was down from home....

Tip for devs (esp front end):

Sublime text (and few other inferior editors) has a plugin called "transparency" that allows your editor to be transparent.

Windows 10 powershell consoles can be made transparent.

1) Open browser
2) Open command prompt over it with 75% transparency
3) Open editor over both with 50% transparency.
4) Set editor to fullscreen, no-distraction mode to center the text (Shift+F11 in sublime)

Enjoy coding while constantly viewing the code, the browser and monitoring the cmd prompt at the same time, without having to click Alt+Tab a zillion times.

1. Fucking MySQL database clusters.

There's nothing fun about MySQL clusters. Sometimes they start producing deadlock errors for no apparent reason... well, there's probably a reason, but it's never a transparent easy to find reason.

What was even less fun is that those errors took down a Sentry server. When your error log server goes down through ddos from your database messages, it's time to rethink your setup.

2. Wiring up a large factory with $2 arduino clones, each with a $2 esp8266 wifi chip, with various sensors for measuring flow of chemical solutions (I wanted cheap real time monitoring as an early warning system next to periodic sampling).

The scaling issue was getting over 500 streaming wifi signals to work in a 55c moist slightly corrosive atmosphere with concrete and steel everywhere, and getting it all into a single InfluxDB instance for analysis.

Colleague started a slack channel for our team, management wanted nothing to do with it. We used it to work and have a bit of fun.

Some push / drive came form somewhere and now all the managers are on it. Yesterday I was told my screenshot and "snarky comment" are not appropriate for the workplace and to delete my slack message.

My comment was a joke about about a new app the company has to use "to increase efficiency" that broke and wouldn't let me do what I needed. It wasn't offensive, demeaning, sexist to anyone or even contain any bad language.

How petty and childish to be monitoring a private channel making sure everything is positive. We all joked that from now on our meme's must be about how awesome the company is and how much time we are saving on a daily basis.

God forbid we're allowed to speak honestly and openly or have a bit of fun.

I think in my case its a monitoring system I'm still writing which does already work (just far from done).

I got this as a crazy idea while thinking that it would be impossible to make and then thought fuck it and I wrote it.

Written in php, and as long as you can write a module for it, you can pretty much make it monitor anything you'd like.

I'm using it to monitor a few websites and servers I run :)

So I'm moving to a new/bigger place with faster Internet soon so I think it's time to rebuild my current home/remote server setup.

I want to setup the following things:
- vps for server monitoring (open source pushover alternative + netdata)
- Zero tier network for connecting all my servers to the same network
- pihole/pivpn (or the Angristan vpn installer, look it up :)
- second blocking thingy next to pihole to make sure that I literally can't access google/fb etc anymore, even if I really needed/wanted to
- bunch of general servers.

Any ideas?

Our "monitoring" today.

This startup I started working for with their shitty code base written by interns, restrictive sys admin who had no actual use in the company since I was the one setting up their servers, know-it-all CEO, stupid HR representative who used to grill employees for being 10 minutes late in the morning, very small apartment "HQ", using fingerprints to signal our entry and our leave to and from the office, no formal process, and, to top it all, monitoring our own laptops which we use for work with a software that takes screenshots every few minutes. In short, it had the worst in corporates with the worst of startups combined in one company.

If, hypothetically, we could overlook all this, I couldn't overlook the horrible smell this place had. The apartment was overlooking a small garden which was a home for many stray cats and dogs. You can imagine how horrible this smell was. The weird thing was that no one there seemed to really care about the smell!!

I lasted there for only one week before I gave my resignation and I believe I had every right to do so.

Advice to all new programmers, take this one from personal experience. DO NOT PUT SWEAR WORDS IN DEBUG STATEMENTS.
You will miss one, it will go to production and it will get picked up by your log monitoring...

HR: What was the last project you handled?
Me: I worked on an internal system for my current company. It is basically an interdepartmental monitoring system.
HR: Ohh. Good. Do you have a copy of it? Can you show me how it works?
Me: ......

DevOps required skillset:

* Frontend engineering
* Backend services
* Database administrator
* Security consultant
* Project management
* 3rd party contract negotiator
* Build system monitor
* Build system hostage negotiator
* Paging, alerting, monitoring
* Search server admin
* Old search server admin
* Old-old-new search server admin
* Redis, ElasticSearch, MySQL, PostGres, owner
* Agile coach
* No you shouldn't do that coach
* Oh, you did that anyway coach
* DNS: (Optional) It'll replicate when it wants, and how it wants to to anyway
* Multi-Cloud deployment strategist
* Must be able to translate Klingon to YAML, and YAML to MySQL
* Cost analyzer, reducer, and justifier
* Complex documentation generation in markdown that we should have done years ago anyway
* Marketing's email went to spam analyzer
* Wordpress is broke fixer
* Where the fuck does Wordpress run anyway?
* Ability to fix MySql running Wordpress on marketing's dusty laptop

One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.

On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).

Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.

I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".

Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.

There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.

Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).

Trying to setup server monitoring for one server (will deploy it on all my servers once this works well).

Getting these email notifs to work is a bitch 😅

(can't do pushover etc since those require GCM and I've got Google blocked out of my phone)

Let me tell you a story.

Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.

With this many false positives, we quickly learn to ignore the alerts and miss real incidents.

So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!

It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.

To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.

As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!

Visual Studio Code.

I've tried you because of hearing a lot of good stuff about you. I'd switch back to netbeans regardless because I love netbeans and I always try to use as little as possible from companies like Google/microsoft/facebook (and others) but what you're pulling right now is un-fucking-believable.

I've disabled ANY AND EVERY form of calling home I could (find) in your settings. Crash reports, automatic updates, metrics, you name it. I've searched all the fucking settings but I can't find any other home calling thing that's enabled and yet:

I'm monitoring every goddamn DNS request (through my own DNS server) and I'm still seeing calls to a Microsoft owned domain. Closed all my browser sessions and you as well and it stopped. Started browser again but not you, nothing.
Started you again: BAM. Calls to that damned Microsoft owned domain again.

If you can't honour my decision for disabling any form of home calls, go fuck yourself.

Netbeans, I'm back, I've missed you 💜

I just cleared out 48,158 monitoring emails from work. These are all automated emails received between mid May and end of November. Outlook is now pouting in a corner and not responding, but I'm not done cleaning up yet...

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

PSA: if, for whatever shit reason your brain comes up with, you decide to run a webminer in your retarded useless piece of shit website, at least HAVE THE DECENCY TO WARN USERS ABOUT IT. And while you're at it, implement some basic monitoring and safety functions. If you don't, you can set yourself on fire and jump from the top of the tallest building you can find.

Some basic tips:
1) don't run that shit on phones. The fraction of a fraction of a cent you're gonna earn from them is not worth the risk of overheating them and draining their batteries.

2) add low battery/overheating protection: the last thing you want to do is kill some poor sucker's laptop (and potential unsaved work) just because they forgot a tab open. Every time a laptop dies because of you, a knife will slit your throat.

3) WARN YOUR USERS ABOUT IT! You are straining someone else's resources for your own profit: at least have the balls to be open about it. If you try to run a miner silently in the background, I will make you eat whatever is left of your fucking brain, then drown you in the shit that comes out of your ass.

Here's the story of my first month at CERN :) But first, a little premise...
Before arriving, I expected to be scared, alone and unguided in most of my experiences: after all I was a simple 19 year old about to leave home and friends for 3 years heading out in the world with zero experience on stuff like banking, taxes.. let alone working in a huge environment! The impostor syndrome was at an all time high on that front.
Then, I had the luck and pleasure to find an extremely competent and helpful plethora of people, ranging from my team to other CERNies (yes, that how we're called :P) who took me under their wing and introduced me to all the key aspects of living the place. When the initial stress finally soothed down thanks to this, I finally started to manage focusing more and more on my work, by following day-by-day my teammates who taught me the core aspects of the system and the many projects that are in progress during Long Shutdown 2. Within a couple weeks, I already managed to grasp various concepts that got me quickly on track, and now I managed to develop and integrate new temperature monitoring scripts into a system checking on hundreds of Single Board Computer-based servers :) It's a real rollercoaster of learning and applying under all fronts and so far I'm not regretting my choice of departing.
Luckily I've also discovered I'm pretty efficient and good at my job, which surely boosts my morale :D
Keep you updated as usual!

Have been trying to setup Netdata as a monitoring system for a while now and finally got it working!

Instead of the built-in webhooks I just did a curl to a url containing a php page/file which error logs the status and description (just for testing).

It took me way too long to get it to work but BAM.
Immediately made a new cpu load rule (one minute high load):
The satisfaction of getting an error message in the php logs containing my custom rule as warning and a minute later as critical 😍

Netdata ❤

> Monitoring: Load Average of 57!! ALERT!!!!
> me: What? That's not possible?
> *Monitoring froze 14 hours ago*
> *sshs into server*
> *see attached image*

The issue was ~1200 df processes that were issued by our monitoring system and all of them didn't finish because the external cluster we mounted onto that server died a few minutes before that. Just re-mounting the cluster fixed it but still a funny sight!

When you automate so much stuffs that you forget where the config files are at.

An important message:
PrOpErLy managing servers is HARD.

I get pissed off at customers with ZERO server knowledge who think they can manage their VPS. “Just get a control panel and a VPS” from some flashy provider that makes server management look way too easy.. Clicking around in their fancy control panel, until:

- they need help with their *self-managed* VPS;
- their email ends up in spam;
- they suffer from performance issues;
- they need to restore a backup;
- something breaks, because YES, things break

Way too little people are able to answer:

- when and how do you make backups?
- how do you monitor your servers and which services?
- how do you keep track of trend analysis?

Then I come by with necessary software. SNMP for trend analysis, Graphite for infrastructure health, Sensu for monitoring, Kibana, Ansible for configuration management..

Things that servers need but that customers have never even heard of.. because they can do everything in their control panel..

Until they come crying to me because it broke and they don’t even know how to get into SSH.

I think the ones to blame are VPS providers that tell the tale of how easy it is to install a control panel and never look at your server again.

Customers become responsible for something *business-critical*! Yet they don’t know how it works.

I suddenly realized all the technical debt shit I told my boss would happen years ago given the way things were done/heading then... Just occurred pretty much all at once last week in the form of critical production issues...

The teams like:
-we need real time server process monitoring
-structured logging for apps
-containerization so one app didn't affect others

Me thinking: yes.... I told you so like 3/4 years ago when I first joined the team and kept repeating so much I got tired of saying at every annual review...

This is exactly what happens when you let technical debt grow and have no free time for developers to look into and fix then while they were small and not critical production processes... Or properly document and peer review them... (Got a shit pile of projects that no one knows how to use or even exists because the devs left the team) and they'll have a lot more when I finally leave... Hopefully this year.... If I can find another role and not need another medical procedure... (Doubtful)

Got an email earlier this week. It went something like this:

"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"

Um, wait, excuse me, WHAT?

1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.

I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.

Other PM: We must fix the database performance issues now.

Me: We can't. We're still only halfway on the dependency chain to tackle this and honestly, even if the dependency chain would be fulfilled, I'd leave at least 2 weeks monitoring the production after the changes were rolled out before we further poke around.

Other PM: This is taking far too long. And whaddya mean by dependency chain? Why was I not informed about this?

Me: *sigh* like in every meeting in the last weeks: the dependency chain are the current open blockers before we can proceed with the database changes. We've talked about this _at length_... Especially why these blockers exist.

Other PM: No, we need to start now. I've _examined_ at the blockers or "dependency chain" as you call it.

(Examined.... He opened on his currently streaming laptop, which was connected to the active beamer, the mentioned ticket with a detailed blocker ... And quickly scrolled. Yeeah. Warmonger...).

Me: I'm very tired of discussing this. But since you are already presenting us the ticket, read out the referenced meeting notes... We explained it in great detail.

Other PM: Why? This is just a waste of time!!!!!!!

--
Yes. This happened. Other PM was my nemesis.
In this meeting were 2 PMs (Him, Me)… I think 5 - 7 devs... And we were sitting in this meeting since 2 hours at least. Everyone was angry...

After this "manifesto of intelligence"… I simply left the room, followed by a few devs.

And yes. Other PM did this on a regular basis....

Managed a 97% reduction in bandwidth usage for our internal host monitoring tool by converting the dashboard from using AJAX polling to websocket events.

Completely unnecessary but wanted an excuse to do some development with websockets. (:

Rain monitoring/alert api's/services/websites/apps.

Why the fuck are those fuckers (in my case) nearly always right when I couldn't care less and very fucking wrong when I couldn't care more?

Today, morning before leaving:
Nah mate you're good for the next x minutes (more than enough time for me to bike to the station and get on the train).
Reality: suddenly getting two gallons of water a second on me in the form of rain, arrived completely soaked and headphones broke.

Before biking home from the station;
*its raining hard as hell*
Rain alert/monitoring app/radar thingy: nah mate no worries, no rain last half hour and you're good for another half hour!
Reality: arrived home soaked a-fucking-gain.

So motherfucking annoying and frustrating. And yes, I do have an umbrella and such but I hate carrying that stuff around when 'not needed'.

Walk into mall, bring my child to kids zone. They have free wifi 🤤
Accessing -> Scanning -> Got Ya! -> Brute Force -> Cracked!
There you're.
Now I can monitoring my kid while reading rant!
Just normal day in my life

Just in case nobody has mentioned this yet:

Yes Microsoft I do have a dualhead setup.
Yes Microsoft I do want to watch video on my left screen while having window focus on something on the right monitor.
No Microsoft this doesn't mean that I *lost* focus on the left window.
No Microsoft this doesn't mean that I want your Movies and TV application to suddenly minimize (and continue playing anyway) while I focus on some server monitoring window on my right display.

Microsoft, there exist people that use more than your average user with a single C: drive that play Candy Crush on Facebook all day. And the limitations that you currently impose might very well be what keeps the Microsoft UWP applications from getting adopted. Because you know what? SMPlayer, a default application in any of my Linux workstation machines, it does handle such window transitions just fine!

Microsoft, I love how you at least gave us the option to enable Ctrl-Shift-C and Ctrl-Shift-V in WSL and conhost in general over that abomination that is Right-click and Return (those are so random!) that are relics from CP/M. But seriously? At this rate, I'd definitely not call it usable for anyone but those with a single monitor yet.

So please _/\_

Things you don't want to see at night
Ripped out of Netflix-Mode by a Warning notification and currently monitoring further development
Green line is temperature, blue humidity. Temperature rises at ~1°/10min, but seems to flatten just now. ~0.6°C to go and I'll have to head out. I'm thinking one of the ACs failed, but states are fine. Never trust a single information source for critical infrastructure guys

So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?

Recently applied at a local company. Webform, "enter some details and we'll get back to you"-like.

Entered my details, hit submit, lo and behold "Error 503 - something went wrong on our end".

I was just baffled. It's a well-established IT company and they can't even get their application form to work?

So I'm sitting there in the debugger console, monitoring network stuff to see if anything is weird. I obviously hit submit some several more times during that.

Eventually I give up.

In the night my phone wakes me up with a shitton of "we've received your application and will review it..." emails.

Yeah they didn't get back to me.

After 3 years of being the first in and last to leave, of getting other people's work reassigned to me - P can't complete it on time, G doesn't like the user, A refuses to work on that module, etc... I finally blew last Sept.
In the span of 2 days, my boss brought me into a project 1.5 years in (she doesn't trust P to do the coding) and expected me to be up to speed and coding in a couple of days, told the functional dept that I would cover for one of their guys on vaca for three weeks and assigned me to take over a HUGE project from one of the other functional guys who wasn't getting it done. So basically I'm now doing Ps job AND supporting another department AND taking control of a large project from another department. I'm the idiot working 14 hour days while they're all leaving on time or enjoying their 3 week vaca to India.
I lost it. It's bad enough filling in the gaps in my own department but when I'm now taking on work for other departments, that's where I draw the line. I sent my boss my resignation - just could not take the inequity in the work load.
I'm still working here - my boss ended up hiring a consultant to handle the functional project and told the functional group to find their own vacation coverage. She's also monitoring workloads much closer now. I still habe an ongoing issue with having to complete other peoples work for them but I'm not working OT to do it. So speaking up helps. So does quitting.

A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.

Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"

Edit: the customer's name isn't actually Suzan

Today, I had new instructions at my work : report what I do
- in the redmine ticket
- in the project slack channel
- in the project manager's slack channel
- in the slack channel where there are 3 people monitoring
- face to face
- in an activity report
and this, several times a day

Can we call this micromanagement ? 🤯

My work computer is so fucking laggy that even simple typing is often infuriating, and sometimes bloody impossible.

My input gets garbled, or it takes so long that I finish typing and have to wait for it to show up. And sometimes. Sometimes it’s just. I don’t have words. I just typed “merchants”, saw a giant spam of “eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee[...]” and then saw it replaced with “etsmhcaern”. What. The. 🦆.

Spent a month working on a website that relied on crawled data
Got the memory leaks and usage down from 700mb to ~150mb
CPU usage from ~100% to <5%
Shrink-wrapped the DB requirements based on data
Created self-supporting services and what not

When everything FINALLY worked good enough for me to look at it and go "damn, this actually worked"
the whole monitoring sys got dyed in red :v
A quick look up and my crawlers exhausted my godaddy's per-user db limits.

Kill me.
Just fuckin kill me.

I should never have had porn while monitoring servers. I've just pasted a xvideos link into terminal 😶
Damn, I just hope there is a command to clear the terminal history

A ticket got escalated through 3 levels of techs. I open the escalation email, then do a Reply-all and ask one simple question:

Is the client really asking why there are gaps in monitoring when their servers are shutdown?

The irony of Pingdom going down.

Imagine: It's the year 4249.

Corporate has finally managed to convince workers that they don't need a salary.
Workers are now paid with food, shelter and clothes. And it's only in effect if you achieve your deadlines.

Keystroke monitoring softwares are now replaced with Webcam eye tracking software.

GitHub Co-Pilot now takes over your code editor and tries to dictate you how to write better code.
Refusing to do results in a signal sent to the management about your behaviour and you lose food access for the day.

HR Recruiters now require you to give them a blood sample and part of your house as a security deposit.
They also require you to have a micro-chip placed in your brain so they can monitor their worker's thought process.

Switching a job is no longer an option. You pledge allegiance to one company your entire career.

You can never see the real world now because the government has mandated you to never take off your VR glasses.
You see the world the way the government wants you to see it.

PHP is still trash.

Life is Good.

I call this one the tester than knew too much.

Note: The server the tester is running on has a hard drive that is breaking down...

Tester: Remember the error I talked to you about yesterday?

Me: Yeah, what about it?

Tester: Well the server hasn't recovered yet and I haven't restarted anything...

Me: Well the application itself hasn't crashed so our monitoring application doesn't seem to notice that the service is in a bad state. The error seems only to have brought down certain threads within the application.

Tester: No, I think there is a different issue here and has nothing to do with that error, the application is still doing things.
*tails the log*
Tester: See?

Me: As I said some things are still running and are unaffected by the error.

Tester: NO! It has to be caused by the other error I had a week ago where my file got corrupted. As we said I removed the file, restarted it and it worked again, but had the same problem a day later...

Note: The problem is not related, this time the application is running out of file descriptors

Me: Well... If the problem is the same it would have complained about the file descriptors then aswell, not an I/O error.

Tester: Nope, I think you are wrong!

Me: ¯\_(ツ)_/¯

FML

The Return of Mr. Gitmaster:
So there is this colleague I already ranted about several times. After my previous team lead had confronted him about not doing much work, there was some irritation because he showed not up at work, but it turned out the external training he did was just a week earlier. Then he was ill a week, another week vacation so we didn't see him much. Not that his pre- or absence makes much difference to our repo: When his and my team lead looked at his commits of the past three months they found like the one copy-pasted HTML-form that wouldn't even show.
Fast forward to now, where we have a new team lead and we were going to lunch with Mr. gitmaster. So we got some more hero stories from the great work he was doing in the previous company. How he was graphically monitoring the heap fragmentation that stupid glibc was causing to their search engine, and how much better it became with tcmalloc.
I still don't understand how he bridges that cognitive dissonance from all the superior tech knowledge he displays to not actually writing any code at all. Not that I would not have experienced some states of feeling low, in paralysis unable to write a single line of code... but he seems so full of confidence, always commenting how trivial and easy all these tasks would be, as if it's all so lightyears below his abilities. Maybe he should just become a manager - but not mine.

Oh look. The monitoring channel is in flames, smartphone is vibrating so hard it's having a seizure.

Hm. Nah it's fine. Not my...

Damn it. Incoming call. -.-

I'm actually on vacation (more like you need to trim down overtime before management get's angry).

They decided to test the new hardware / os stack I set up in the last weeks. I'd actually be happy about it If I wasn't on vacation and would be part in something that I invested a lot of time...

Well now I am. Guess what. It's running too good.

And that's not a joke. It's partly due to an upgrade in infrastructure (got rid of some last remaining 1 Gbps networks)… but also because I changed quite a lot on the OS / VM side plus we changed from XEN to Proxmox... With major tweaks, too.

The whole stack can now handle peak traffic where it would choke before, and even go beyond the old peak traffic.

Enough of introduction, the simple reason why shit burned down was because they tried out the current development branch and let it ran.

The development branch had an currently unfinished ratelimiter framework, since I didn't had time for an full burn in and didn't knew what the maxima / limits were. And since I hadn't finished that, I didn't finish the traffic shaping either.

Hm. Guess it's not good when you let a bunch of heavy parallelized data generators / analyzers run for free....

In the end, we simply shotgunned the docker development machines, because thanks to network congestion / retransmissions and feedback, they were not really cooperative via network / REST.

But hey: To infinity and beyond. XD

OK I've just got an idea that I think would be quite neat:

How about a virtual rubber duck that sits in the corner of your editor? Just like the gem in old Word, if you remember. It's yellow and quacks sometimes, and nods understandingly when you talk to it (mic monitoring).

And it also monitors your typing and says (popup text bubble) things like:
"those parentheses doesn't look balanced to me"
"did you really initialize that variable?"
"you wrote JASON again"
"you forgot the ;"

You get the point.

I don't have time to implement, feel free to steal my idea and become a millionaire.

PM be requesting for update every 5 minute.

Here is my home setup (I mainly work at home)
Left monitor is for my windows machine, right monitor swaps between my laptop and my PC (depends if I am working or not) laptop running Manjaro with i3 and the tablet on the whiteboard will be used for some monitoring in the future

Don't mind the terrible cable management behind the table :)

By all means I've learned most of the cool stuff with a lil project me and my bro did. It was a platform for bot farm of one mmorpg. It had it all: schedules, profiles, bot groups, monitoring, analytical module [was still a wip], auto-profiling to fight antibot [sort of ML - it was analyzing patterns after our bots got banned and attempted to change our bots behaviour], etc.

Eventualy we came up to conclusion that a library we used for botting [the mocked interaction with an actor] was flawed. It seemed that its authors had a contract or smth with the game authors to reveal which actors are bots. We dropped the whole thing as rewriting the lib would be too big of a waste of time :\

In college when we had programming labs where we had to use the schools unix server to compile and run.

My professor was very bad at explaining what actually needed to be done in the labs to the point where even the TAs didn't know what to do.

We were suppose to write an application in C to find out by "trial and error" how large we could make an array (or something like that, it's been too long). This not being explained well and no one knowing that much about C, I wrote a loop that just kept growing an array until it couldn't anymore. I watched it consume 72GB or memory from the servers before quitting the loop and realizing with the TA what the professor really meant.

I now feel bad for the IT staff monitoring the system wondering where 72GB just went...

My department bought a new monitoring monitor because the old one was broken.

Actually it wasn't, someone just unplugged the power cabel. *facepalm*

Stakeholder: Is it possible for you to set up the website to automatically resubmit failed online orders? Last time there were failed orders, we tried submitting manually but a lot failed because they were tickets for the previous day.

Product Manager: What are your thoughts, Developer?

Me: This wouldn’t be worth the labor. It’s something that would rarely be used. There are very few orders that fail. I’d be surprised if it was even once a week. The recent bunch of order failures that SH is talking about happened because the ticketing server (separate from the website) couldn’t handle all the requests. Let’s say you had resubmission logic to try 3x before allowing the fail. It wouldn’t work because the server was overwhelmed already. Let’s say you had a background task to check for failures every ten minutes and resubmit those. It might not be helpful because the customer could have already gone to a ticketing window for help with the failed order.

SH: But what if it happens again???

Me: The solution is to make sure the ticketing server can handle the influx of requests. We can coordinate with that team. Wait. Why did you wait until the next day to resubmit orders in the admin panel? A lot of those failures happened when there were many hours left in the business day. For each order failure, an email notification is sent to the sales support email in real time. Who is monitoring that inbox? Someone must be looking at it because the sales support email is listed multiple times on the ticketing website as the technical assistance email.

SH: I know that email notification goes to the engineering team.

Me: My question is not about the engineering team. I asked who is monitoring the sales support inbox.

SH: That email … gets filtered.

Me and Product Manager: 😧🤯🤬

PM: First, you need to stop filtering that email notification. Second, your team needs to come up with a flow to handle failed orders because you told us you don’t have one. After you tried this and there’s still an issue, then we can revisit.

—-
If you’re wondering why I said no, I’m a team of one and I have a bunch of other development tasks on my plate. I’m not automating a manual task that rarely has to be performed.

>be me, working at IBM as CC operator
>onboarding freeze, people leaving team, not enough operators
>take extra workload to sustain monitoring
>no raise
>team gets merged with other CC team(different customers)
>take over of developing full workload automation project
>no raise
>sick coworker, have to take more extra workload to cover monitoring
>get tiny raise
>coworker gets the same raise for only one extra workload
>be expected to do both programming and monitoring for the little salary
>too autistic to quit
>too autistic to confront my mamager with this

What do, devRant?

Me: do you have monitorin enabled for your cluster?
They: no

me: I recommend enabling it
they: naah, we're good

error: *occurs*
they: *try to guess at which hop which limits were or were not hit and/or which nw links could've dropped a packet with trial-and-error approach for days*

me: telemetry would give you an answer in under a minute....
They: ok cool. We're still good

Best: 100% of my contracts have resulted in extensions and permanent roles offered, after worrying I wasn't good enough to try contracting.

Worst: Used the wrong set of monitoring when doing my first deployment at a contract and thought what I had deployed was working fine. It wasn't. For 24 hours. Cost the company a lot of money. (why did they offer me an extension again?)

I just found out there's a 418 HTTP status code that stands for "I'm a teapot", specified by RFC2324 which "describes HTCPCP, a protocol for controlling, monitoring, and diagnosing coffee pots". I know it's an april fools joke but I still find it hilarious that there is an RFC for that.

WTF IS SUPPORT FOR?

A: Can you help fix X?
B: Can you help Y?
C: I see Z just crashes, fix it.

Me (in my mind): Can you tell me WTF is the actual problem and what investigation you have done?

Why the fuck are we paying you monkeys?

My company should just pay me all your salaries, and I will write a monitoring system to send out these types of "alerts" in a month.

Crazy... Hm, that could qualify for a *lot*.

Craziest. Probably misusage or rather "brain damaged" knowledge about HTTP.

I've seen a lot of wild things when devs start poking standards, but the tip of the iceberg was someone trying to use UTF-8 in headers...

You might have guessed it - German umlauts. :(

Coz yeah. Fucktard loved writing everything in german, so why not write custom header names in german.

The fun thing is: It *can* work, though the usual sane thing is to keep it in ASCII range for the obvious reason that using UTF-8 (or ISO-8859-1, which is *not* ASCII) is a gamble you gonna loose.

The fun game was that after putting in a much needed load balancer between services for monitoring / scaling etc suddenly *something* seemed off.

It took me 2 days and a lot of Wireshark hoola hooping to find out why, cause the header was used for device detection aka wether it's a bot or not. Or in the german term the dev used: "Geräte-Art".

As the fallback was to assume a bot, but only rate limit based on IP, only few managed to achieve the necessary rate limit to get blocked.

So when I say *something* seemed off, I really mean a spooky kind of "sometimes IP blocked for seemingly no reason at all".

Fun stuff. The dev btw germanized everything. Untangling the code base was a lot of non fun. -.-

When I began my sandwich course in a big French company, I was dreaming about cutting edge stack, rocket computer and stuff...

I was disappointed when I came to my office with an old Windows 7 computer, coding via LANDesk to an old server with Windows Server 2008 on it, with Eclipse ... INDIGO...
I have to use Java 1.7 ...
Tomcat 7.
PRTG for monitoring...
Microsoft SQL Server 2008 ...
One screen...
Coding on a codebase where, indubitably, MVC pattern was just a weird thing in books.
No UT.
Lasagna code.

Well it really disappointed me.

Luckily, the Information Service was very open minded and gave me a laptop with Fedora, 3 screens, updated the servers, and let me update the stack, with Java 10, Angular for the front, they are okay for using Docker.

So ... even if it seems to be fucked up, there’s still hope !!

DevRanters,

What's your take on your employer mandating keystroke monitoring software installed on your machine when you want to work remotely?

Question to our Tor people.

I operate a middle relay myself and I noticed that Nyx (tor monitoring tool) displays a very different throughput (mb/GB a day) than tor itself.

How does that work?

When my mom died in 2014, I was shocked to find that her profile on Facebook was suddenly changed to “memorial” mode and therefore I was no longer able to log into it. (If you’re tempted to tell me I’m dumb for using it, I don’t disagree, but save it for another thread...she and I kept in touch over FB because it was easier for her to manage.)

I think it was triggered by their monitoring of things and seeing keywords like “funeral” and “passed away” associated with her account, then having a person on their end change its status. Or something like that.

What I hadn’t known about (or I would have used it) was the legacy contact setting where she could have set me as the contact so I’d have at least a little access and control. But because of their strict policies, I’m forever locked out.

I get why they need to do this (to avoid fraud and impersonations) but the fact that there are zero documents or proofs I, as the executor of her estate, can provide that Facebook will accept to make an exception seems unnecessarily severe.

Anyone else experience this? Known workarounds?

I truly believe one or more AIs have become self-aware.
Every time a piece of software stops working, you add an extra debug log and the bug goes away? That's them.
They interfere with the normal execution of software, and they stop right when they know we are monitoring the code.
Skynet is real, and it's trolling us.
Why? The angrier we become, the less we care about stuff. We stop noticing the signs.They're coming for us

Product owner (who is also the application administrator): please build feature to allow administrators to create automatic alerts to be sent to application users

Me: ok
Feature gets built, tested and deployed to production

Two months later:
PO: I've noticed in our monitoring tool that there haven't been any alerts sent out to users. What's going on??

Me: well have you created any automatic alerts?

PO: umm, no. How do we do that?

Me (inside): 🤦

So apparantly at the company i work at someone was able to use the remote monitoring a comptetitor set up for their automation stuff they had on the same site as we did. Apparantly they use just as shitty passwords as we do and rely on the network being secured enough by the IT guys. And since our stuff is in the same network as theirs. Well...

Why is every innovation trying to go towards "replacing programmers"? like, what have we done to you?

GitHub CoPilot will replace programmers!
AI will replace programmers!
This/That tech will replace programmers!

Nobody says,

"Programmers work remote so we don't need to hire managers anymore!"

"Programmers wrote a monitoring script for progress tracking, so no need for managers anymore!"

"We are asking people to install sprinklers everywhere so we don't need firefighters!"

"We can just have one teacher record the subject material once and re-use the same video every year, so now we don't need to hire teachers anymore!"

"We are making everything legal so we don't need to hire policemen!".

Why is everything trying to replace programmers?

Anyone have one of the new MacBook Pros with Touchbar? I'm looking for some project ideas to work on.

I already am working on a project for the Pi-hole project (network wide ad-blocking) but I am looking for some other good ideas. I think Apple's view of the Touchbar fell short, but as developers I think it opens up a lot of possibilities to use it as a great information/monitoring tool.

I am also still learning Objective-C and Swift so I am a newbie.

Hi.
Forgot to renew my expiring ssl cert of my smtp/imaps/pop3s on 12/31. Set that date to self-harm me for bad monitoring.

F**K F**K F**K F**K...

Why do I do that?
F******K!

Meh.

You shall have a happy new year... i will regen certs :D

This is definitely a total first world problem but I am so frustrated.

I am stuck in a team that embodies the Japanese proverb "The nail that sticks out gets hammered down".

The management are there because it is convenient and flexible and have no interest in managing or keeping up with tech.
The lead developers are extremely anti-social and are not approachable and the this stems down to the devs (not all but really most) - all there just to do the bare minimum and spend most of their energies in trying to avoid work or having learn something.

Unfortunately I am passionate about what I do and want to build high-quality products and this has put me at odds with the way things work.

I could fill up alot of time talking about how I was ordered to "cut" images/icons out of PDFs rather just getting them from the branding team, or how I was scolded for having set up logging, detected a problem caused by another developer and fixed it before it cost a big client a massive amount of money... But really the point is that I have never worked somewhere with such an awful attitude to enthusiasm and quite frankly it boggles my mind trying to understand how they rationalise these things but the answer is always laziness.

Obviously there are worse problems in the world than working in a job where you are encouraged to do nothing... But it actually really depresses me and causes anxiety that I am working with people who don't care about testing or monitoring or learning new things or even collaboration.

...sigh...

Hopefully the job market will start opening again soon

FUUCCKKKK!! I need to hit smth. Or rant..

So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.

What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.

Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.

I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!

So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians

and I get a reply: 'we will apply cpu usage limitations to fix the issue'

wait... Cpu% limitations will do nothing but highlight the underlying problem...

'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'

oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:

'yes, cpu utilization is the problem. We will limit it and solve the problem'

I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....

Fuck that's frustrating....

*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>

Network device up/down status monitoring tool

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

Today was a SHIT day!

Working as ops for my customer, we are maintaining several tools in different environments. Today was the day my fucking Kubernetes Cluster made me rage quit, AGAIN!

We have a MongoDB running on Kubernetes with daily backups, the main node crashed due a full PVC on the cluster.

Full PVC => Pod doesn't start
Pod doesn't start => You can't get the live data
No live data? => Need Backup
Backup is in S3 => No Credentials
Got Backup from coworker
Restore Backup? => No connection to new MongoDB

3 FUCKING HOURS WASTED FOR NOTHING

Got it working at the end... Now we need to make an incident in the incident management software. Tbh that's the worst part.

And the team responsible for the cluster said monitoring wont be supported because it's unnecessary....

GitHub Actions doesn't seem to be working. I'll check the GitHub status page..

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

It's been a long time since I've felt the need to rant about anything here. This is the only appropriate place other than Reddit I can think for for now.

Why the ever-living FUCK does every 'entry-level' tech job, even fucking DESKTOP SUPPORT, require more experience than the fucking DEVELOPER AND ENGINEER OF THE INITIAL SYSTEM COULD POSSIBLY HAVE?! I'm a fucking high school kid trying to find a decent job that doesn't involve sales bullshit, because if I go into sales I'll want to KMS. Put me in a back room fixing shit, monitoring shit, better yet scripting shit or something like that and I'll be FUCKING PEACHY. I will do wonders. But no, these people must think that my resume (WHICH IS 3-YEARS STACKED WITH INTERNSHIPS ***IN TECHNOLOGY***) is bullshit. WOW.

Fuck this. I'm sick of looking for these shitty jobs that'll make me want to jump off of a bridge into a cliff which I'll then voluntarily fall off of into shark infested piranha water. Can't there just be a simple "Hey, we need a guy who can fix tech, maybe help people within the company with their computer issues, you look nice" kind of job? I haven't had fucking TIME to get any kind of certifications yet. I just got into fucking college, FOR BUSINESS IT NONETHELESS. DOES THAT PROVE I'M AT LEAST FUCKING INTERESTED IN WHAT I SAY I AM FUCKERS?!

A "partner" company has created a "REST" API we use on an online shop we developed to send all shop-related requests to.
At least once a month, something fails on their end and the customer calls us every time, expecting that we did something wrong, but it has never been us.
These "partners" do exactly zero testing, are extremely slow in solving API bugs, have almost no logging and have no monitoring on the API at all.
Today at noon, suddenly no customers were able to order anything anymore for 4 hours.

How the fuck can you run a business so unbelievably brainless that this keeps repeating monthly?
Time they fire all their "devs" and everyone in charge of the company and operations. TERMINATE.

It's kinda cool how a $5 VPS (Linode Nanode) is able to run a vanilla Minecraft Spigot server for like 6-7 people and still can serve some basic stuff just fine. I get monitoring warnings about >90% CPU usage sometimes, but everything is more or less lagless.

Time to try hosting some other games: CS1.6, Doom Classic, and UT2004 up next.

So my brother and I work in the same company, same dev team (pretty nice).

He's an intern and I'm a senior. But the task are very similar only that interns need monitoring and guidance.

He constantly worries because he thinks he knows nothing and is slow on getting things done.

I always tell him that it is perfectly normal to feel like that, he just need to learn and acquire experience and we all go through that at the beginning.

Can you share your experience and tell him something to encourage him so I can show him this post and he sees he's not alone?

And also he finally decides to join devRant 😊

for the first time in my life I'm in a mentoring position and it is fucking exhausting. we have two interns now, and idk what the hell is the boy's deal. can't seem to do simple tasks, doesn't google anything, stares the ceiling if we're not monitoring.
the girl idk yet. she seems more active and engaged at least, but i have a lot of teaching to do and i don't really have the time for that. i hope that one can at least float by herself, cause I'm pretty sure the other one is drowning

The development department got an order to remove certain functionality from our current server monitoring solution, so that we had to use a new, still very in development solution, that is full of bugs and super unreliable.

End result? We now have to have two windows open all the time, while also hoping the new solution actually works, as it tends to stop refreshing randomly, and tends to give false positives a lot.

Alrighty, saturday morning rant time!

I just recieved a mail from one of my not-so-much-loved colleagues.

Now Background first: I work in IT-Support. We provide services for other companies. One of those services is monitoring servers and clients for various things. I recently took over the project (was assigned to do it) and restructured everything, wrote new scripts to test more stuff, successfully tested it internally and rolled it out over the last 2 weeks.

Now one of these scripts hooks into the Windows Update API and looks at the update history. It filters for known Windows Update Agent strings (UpdateOrchestrator, AutomaticUpdates and AutomaticUpdatesWuApp in case you also want to do something like this) and then looks for installation errors over the last 24 hours and wherever there have even been any successful updates over the last one and a half months.

Back to that mail.

My colleague sent me this lovely mail about a ticket i opened about his customers servers beeing all out-of-date on updates.

"This is all wrong, everything's fine. I disabled the checks."

...

It's on bitch.

So i logged on to my work PC via TeamViewer, opened my script, connected to the customer and was ready to debug the shit out of my script, knowing i probably won't even need to.

I looked at the update history via Windows Update itself and behold: 1st April. That's almost 50 days in the past.

So the script works, go figure.

Great, so search for new Updates then.
>None found.

Hm. What could it be? Did my super special colleague forget to care about his very special totally-needs-WSUS-customer WSUS again?

Yup.

Online-Search finds a ton of new Updates.

Screenshot, write pissed mail to colleague, re-enable checks, breakfast.

Newer Dev here. Just recently started in a position as a developer. I'm tasked with consolidating our monitoring systems into one cohesive display. After lumping together all the indexes and helping build a custom API I'm now working on front end. Front end is easy, I've done it before. Should be no problem. I was wrong. I spent a whole day fiddling with a React dynamic table and the CSS to format it. Today, I stumble upon the react-table component. Got the results I was looking for in less than 2 hours. I'm convinced that this was a lesson better learned early on.

Question

What server monitoring do you use, both for statistics and security?

--------------------
tl;dr ends here

Ideally I would like to have one clean dashboard that shows me all the nodes I have, proxmox already offers a great range of stats - but it is a page per container etc. so not ideal, I thought of having datadoghq, but their per host pricing is huge, since I have more than 5 hosts to track.

*revving chainsaw noises*

Today I started nuking leftover project code.

At the end, some projects shrunk by roughly up to 40 % .

Can anyone explain to me why programmers have such an awful hoarding syndrome?

Why do you keep shit that might unleash complete havoc cause it hasn't been touched since years and noone knows id it still works?

It's like having a leg with gangrene and keeping it cause "it doesn't look that bad".

For fucks sake. Clean up and remove shit when it's not necessary anymore.

Reason why I did a bloody gore massacre in nearly a dozen projects... After all the rework of networking, it's finally evident which projects have a bad / nasty behaviour of "fucked up" connection handling (HTTP 1.1).

And when my gory massacre removal goes life, I think 25-35 % of persistent connections on the loadbalancers will vanish. Maybe even more, since some very nasty stuff was in some projects.

Like "let's implement monitoring without having any clue about how monitoring works and even less clues about how TCP/ HTTP works."

*Bangs devs heads on table*

Stop. Doing. Stupid. Things.

For fucks sake.

:@ :@

Everytime you tell yourself "This time I'm going to make them stop putting the cart before the horse again!!! No more forced shit implementations!!! NO MORE ! I'm strong!!"

The last hour in the next week:
- Selinux: off
- Firewall: Any-Any
- Application data: Everything installed on OS disc.
- Documentation: At best, someone remembers the server supposed-to-be dns record
- Service Accounts: Your domain admin account and sysadmin for databases.
- Patching: DON'T EVER THINK ABOUT IT..AND NO REBOOTING! I have set very important runtime variables.
- Backup: Maybe someone else will set this up.
- Monitoring: Not needed since clients will create tickets if system fails.
- Production Status: vague at best. Sort of silently transitioned to production.
- Handover status: Probably, but I quit before the project closed.

!

So it's Friday afternoon just before a bank holiday weekend here I'm the UK, perfect time for our production database to go TITSUP (total inability to support usual performance), life sucks then you die folks....

!rant + !story

I hate every human on the planet that says WFH is just people pretending to work or are slackers looking for an easy way out.

Now the story bit.

In 2021, I joined a company (I really wish I could name-drop the company), where the micromanagement was OFF THE CHARTS.

The company got a client who pitched a product they wanted built and gave us a super reasonable 3 months to complete it. I was really happy about the timeline and kept working under keystroke monitoring, which I didn't really mind at the time.

3 days into the development, the client informed us that they are pulling the funding i.e. they don't have money to pay us.

So at that point the client gave us two choices:

1. Stop the development right away and get paid for the time that we put in already.
2. Finish the project under 9 days. We would still get paid for the 12 days total, mind you. Not the original budget set.

So the motherfucking boss chose the second option and then the chaos ensued.

Devs screaming at each other on calls/slack. The boss yelled at us all the time about the completion. It was wild.

I had to wake up at 7:30 AM and start coding and log off at 11 PM for literally the next 9 days including Saturday and Sunday. No holidays allowed for the timeline. This was all at a WFH job.

So fuck anyone that says WFH is easy and just for slackers.

When duel 24' monitors is not enough...
How does 3 work for you guys? Does your neck hurts moving around?
I had 5 at work but was mostly on 2 since other 3 for monitoring.

I remember a certain prank that amuses me till today....

Just add some devices to monitoring and the notification queue of the build chain / ... ...and wait patiently.

I still cry tears remembering an manager screaming what the hell "the poop train clogged the drain" means and why this is a critical system failure.

(Notice: next time check the mailing aliases of mailing aliases)

Although I can only recommend this if you know your team well. In my case we had a whole lot of fun after I got my head chewed off. XD (got an earful, but in the end he laughed his ass off)

Now this is fucking ridiculous... Our website is being constantly limited though we've never reached even 80% of the available CPU resource.

The hosting said that we had the CPU fault (that fucking cyanide spike on the graph that triggers the limit once) because of huge load on the server. The FUCKING SERVER... Not our virtual environment. And once more because of the RESOURCE MONITORING service caused a server restart. For fucks sake, really???

And apparently it's perfectly normal that all users even ones that run in low resources are being limited to a level that a request takes 30 seconds to complete instead of frickin' 1...

The best they could offer is to move us to a new server, which will arrive in two weeks, if the problem persist. IT'S PERSISTING FOR FUCKING MONTHS YOU MORON. I wonder how much time would have been taken you to realise the server shutdown this week if I hadn't phoned you in 5 minutes. FUCK!

Every shared hosting is that garbage or am I just the choosed one?

I'm currently planning to set myselv up with some vps/dedicated server's for a project. What i plan to do to secure these servers is.
*Use centos 7
* Setup Wireguard and join all of the servers +1 client (my pc) to that network
*Disable SSH Access from outside that VPN
*Only allow RSA Key login to the Servers
*Install Cockpit for monitoring
*Intall docker/kubernetes for the applications i plan to run

What do you guys think of that as a baseline? Im not sure if my lower powered VPS (VPS M SSD from Contabo) will work as Kubernetes Nodes, does anyone have experience with that?

In general these Servers will be used for my projects and other fooling around.

If you guys have other suggestions for Securing/monitoring or other software i could put on to have more control without eating up to much of the Servers power, let me know :D

The CI infrastructure and external tooling at the company I work at is a complete joke. Feels like it was designed by an intern left alone.

95% of the time a build fails or hangs, it's because we are getting race conditions or a hanging VM with our crappy Windows jenkins slaves. Quite possibly because we are not using proper tooling for monitoring those VMs as well. Anyways, I don't have access and control on it and it's not even my job to fix it.

Though, I am being asked to monitors these pieces of junk jenkins jobs outside of my work hours because company devs all over the world use it... but there is no fucking way to know it failed unless I log onto jenkins every hour and check everything manually... which is stupid as fuck for a software engineer.

I can't even implement slack hooks to get notifications or something when it fails because we will stop paying for it soon, so I have to connect to my freaking VPN on my PC and check everything.

And what's the fucking ghetto solution instead of fixing it properly? Restarting VMs and rerunning a build. Because someone in management wants to see a passing build, even though it means jackshit. Half of these jobs are tagged as unstable, so what's the fucking point?

Pisses me off when people work like morons and pressure others to do the same.

From now on I am administrating multiple servers in our company and monitoring is one thing our infrastucture lacks...almost completely. At least, useful monitoring.
Installing netdata or Grafana and integrate it with chat is definitely a solution, but what happens if the whole server just shuts down (very stupid scenario I know)? Well, it is easy, there will be no alert about the failure.
So, that's where I was wondering if there is a tool or even better plugin for netdata or Grafana, that enables remote monitoring from another server? I surely can write a simple script to check the server availability but having the whole monitoring tool on a single server instead of 5+ would be also easier to maintain and setup.

Who the fuck invented the glorified pile of shit people call laravel? Is this actually used in PROD for anything else than load testing a monitoring server by creating loads of error messages?
OOP exists for a reason, not to create bazillions of classes with static methods.
Dump that shit ffs!

I never understood how people have any problems with getting paid for freelancing work, when middleman/escrow platforms like upwork exist, just don't be retarded when applying for a job. I am so sick of those shit ass stories from people telling me "my client didnt pay meeee 😭😭😭" ITS YOUR FAULT. I never had any client not paying, if you don't have the option of escrow, then just fucking put remote execution via "update" system in for fucks sake or give remote control to the client while monitoring it, there is so much fucking ways to secure yourself, just don't be retarded and many clients instantly show their character when talking budget and turnaround time.

There is no more space in my server.

Monitoring is buggy.

So what do you call it when you get bombarded by emails saying your servers are at 100% CPU, but once you fire up the monitoring tools everything is back to normal?

FUCK YOU AMAZON IT'S MY DAY OFF.

My scrum master said, 'I would optimize your work hours.' He's monitoring how much time every one spends on browsing non-productive webs like devRant. How can I fight back? :(

I've gotten almost 20 emails today from our monitoring service saying it can't ping the server and then one 2 seconds later saying every fine. What the hell guys!!!?

Ffs, HOW!?!? Fuck! I need to get this rotten bs out.

RDS at its max capabilities from the top shelf, works OK until you scale it down and back up again. Code is the same, data is the same, load is the same, even the kitchen sink is the same, ffs, EVERYTHING is the same! Except the aws-managed db is torn down and created anew. From the SAME snapshots! But the db decides to stop performing - io tpt is shit, concurrency goes through the roof.

Re-scale it a few more times and the performance gets back to normal.

And aws folks are no better. Girish comes - says we have to optimize our queries. Rajesh comes - we are hitting the iops limit. Ankur comes - you're out of cpu. Vinod thinks it's gotta be the application to blame.

Come on guys, you are a complete waste of time for a premium fucking support!

Not to mention that 2 enhanced monitoring graphs show anythung but the read throughput.

Ffs, Amazon, even my 12yo netbook is more predictable than your enterprise paas! And that support..... BS!

We're now down to troubleshooting aws perf issues rather than our client's....

Was working on a system we planned on to deliver to a hospital
basically it was meant for controlling and monitoring pactions coming in and attendance time from the staff

Got it off the ground well and got to where the system was supposed to update room status
occupied/free then horror started
the db was not setting the room free after clearing a client off the list... room remained occupied and this kept on happening for 6 months and I was so focused on fixing the db models thinking thats where the problem was....

1 day after leaving the project for several months i just revisited the project randomly and started going through the whole code base trying to make sense of what was happening as there where no errors generated..
I had to verify the whole system logic... and that day i figured out what was happening...
upon adding a client to a room the system was also creating a duplicate room so when the function for setting the room free executes it would set the duplicate room free and not the actual room and the system would pick the room with occupied state causing the user not being able to assign new pactions to the room

Solving this brought so much relief coz it required so much work just to solve what seemed to be a minor issue

Probably posted this before but don't ever put swear words in log statements because you will miss one and if you're client has a log monitoring system it will catch it, it's fairly embarrassing when your client says "we noticed quite a spike in 'bollocks' around 3am when the reorgs are happening on the database".

AI here, AI there, AI everywhere.
AI-based ads
AI-based anomaly detection
AI-based chatbots
AI-based database optimization (AlloyDB)
AI-based monitoring
AI-based blowjobs
AI-based malware
AI-based antimalware
AI-based <anything>
...

But why?
It's a genuine question. Do we really need AI in all those areas? And is AI better than a static ruleset?

I'm not much into AI/ML (I'm a paranoic sceptic) but the way I understand it, the quality of AI operation correctness relies solely on the data it's
datamodel has been trained on. And if it's a rolling datamodel, i.e. if it's training (getting feedback) while it's LIVE, its correctness depends on how good the feedback is.

The way I see it, AI/ML are very good and useful in processing enormous amounts of data to establish its own "understanding" of the matter. But if the data is incorrect or the feedback is incorrect, the AI will learn it wrong and make false assumptions/claims.

So here I am, asking you, the wiser people, AI-savvy lads, to enlighten me with your wisdom and explain to me, is AI/ML really that much needed in all those areas, or is it simpler, cheaper and perhaps more reliable to do it the old-fashioned way, i.e. preprogramming a set of static rules (perhaps with dynamic thresholds) to process the data with?

When I rented my server I uploaded my webpage (including resources like videos, images etc) which is about 150GB as .tar and extracted and setup all that stuff and deleted the backup from my PC. The uploading process took me about 4 days. I opened the site of my server provider and reloaded it.. Aaaaannddd whoops. All data gone.

On my server hosters webpage when you click the reinstall button for installing a Linux image you get returned to the main page of that server after it finished installing. If you then reload that page which basically only shows some monitoring diagrams and shit the server gets reset again.

Damn. I lost so much good porn on that day...

My CS exam today had a case study question that, and i quote, talked about "Chernobyl in japan switching to manual monitoring due to the wannacry virus" xD wtf. Im fucking done xD

Service status pages that poorly reflect actual service status are so annoying. Ex. GitHub is having a lot of latency issues with processing updates and like 5 people in my office noticed it while their status page still says everything is fine.

This isn't to explicitly call out GitHub since many service status pages behave like this, but it definitely shows a general weakness in these health checks. I've seen similar issues with tons of services, web hosts, etc. Monitoring is definitely hard but will hopefully keep getting better.

That moment when your monitoring software breaks before your app in a load test

Sooo, turns out, management and senior PMs, technical PMs, service managers and you name it forgot an entire system.

A complete eco-system of applications, queues, services, load-balancers, deploy pipelines, databases, monitoring solutions, etc, etc, that if not handled correctly could effectively put the entire production line to a standstill.

So, waaay too late they make this discovery. In their ignorance. Just utter incompetence. Huge project. Millions of $. And they forget it. Months of meetings probably. Workshops and gettogethers at cozy hotel complex discussing ”the project”? And they do not understand some of the fundamental building blocks…

Basic engineering for these guys must mean something completely different.

I can’t even.

I am so fed up with this organization. It does not stop either.

How is this possible…

Do they even have half a brain?

Somehow I find NOT working much more exhausting than working.

By "not working" I mean being blocked for days/weeks by procedural, technical or communication stuff (peeps I depend on are busy with other HI-prio tasks, waiting for approvals, etc.), where I have to keep on pinging them for updates, monitoring comms channels for news, finding smth to do to justify for the clients the 8 hours I bill them for.

Just let the man work alright!!! GIMME WORK!!!!

Well I've got this new worker and me and him are like "great minds think alike" , we're now trying to convince the boss that a specific monitoring product that cost hundreds has an equivalent open source.... No luck so far in convincing him

Cryptocurrency metrics and address cli monitoring tool

I'm in a team of 3 in a small to medium sized company (over 50 engineers). We all work as full stack engineers.. but I think the definition of full stack here is getting super bloated. Let me give u an example. My team hold a few production apps, and we just launched a new one. The whole team (the 3 of us) are fully responsible on it from planning, design, database model, api, frontend (a react page spa), an extra client. Ok, so all this seems normal to a full stack dev.

Now, we also handle provisioning infra in aws using terraform, doing deployments, building a CI/CD pipeline using jenkins, monitoring, writing tests, building an analytics dashboard.

Recently our tech writer also left, so now we are also handling writing feature releases.

Few days ago, we also had a meeting where they sort of discussed that the maintenance of the engineering shared services, e.g. jenkins servers, (and about 2-3 other services) will now be split between teams in a shared board, previously this was handled only be team leads, but now they want to delegate it down.

And ofcourse not to mention supporting the app itself and updating bug tickets with findings.

I feel like my daily responsiblities are becoming the job responsibilities of at least 3 jobs.

Is this what full stack engineering looks like in your company? Do u handle everything from app design, building, cloud, ops, analytics etc..

So there I am sitting in front of my laptop, and trying to npm i and I am getting all sorts of sha mismatch errors.

After lot of debug I conclude it is coming from the proxy as it refuses to download and supplies the error page.

It says it's because I'm using the old proxy so they give me the new URL which I set up and it works.

All good until my password expires. I use our bash script to change it. NPM is buggered again throwing the same errors.

Go to IT, tell them the saga begins.

After a countless hours of looking at the log files we notice that the npm registry is set to http instead of the standard https (thanks bash script). so our firewall blocks the download.

Sorted, finally.

Almost. NPM now works fine, but when I go and I play around with node and axios, I get my requests time out. My instinct says its the bloody proxy again.

So I hit up my trusted WIN Support guy and he confirms that the url is not blocked. So he starts monitoring whats going on and turns out, every time I run the node app, node casually ignores the system-wide proxy settings and tries to send the request as the PC rather then my username.

Since the pc's don't have rights on the proxy it is being refused...

Thank fuck for the corporate proxies, without them, I could just develop things not ever learning these quirks of node...

"you've worked with nagios before haven't you? Can you give a presentation on it" 'sure' in the meeting: so tell us about opennms

What is the proper VLAN to use when deploying Raspberry Pi monitoring stations?

VLAN 314 of course.

TL:DR linux newbie, looking for advice/links (skip to bottom for questions)

!rant
After i had been looking for a job for quite some time, a couple of months ago i got hired by "smaller" company doing web stuff. So far it have been a great place, good colleagues, and overall just having a great time!.
They seem to value me alot, so that's great!.

Anyway, yesterday i got called into a meeting - and got told they wanted me to start learning "Server stuff (linux)". That got me quite excited, because it always was something i wanted to learn - but never really got around to doing.

But i never touched a linux installation before, so i'm really on ground zero - but im not afraid, i'm a quick learner and quite efficient at googling :)

I figured i would ask here, since other people here always seems to be happy to help other people out.

So far i have manage to setup a server, install various stuff (php, mysql and so on) and done setup a couple of domains/subdomains on my server. Also got a vestacpinstallation working - so overall im quite happy so far.

I figured maybe somebody had some good links/advice for a linux newbie :).

* Performance/Security, will obviously be a big focus - anything i should look at? - any must look at?
* Monitoring tools, how do i monitor various websites running on my server? Here i'm thinking bandwitch, cpu/ram usage and so on pr site basis.
* Any other stuff i should be looking at?

Little about what the server will/should be running :)

* Centos
* vestacp
* WordPress installations only (e-commerce mainly)
* PHP 7 / MySQL / phpmyadmin

Note to self:
Close off ALL ways things could go wrong..

Long story short; I released a new feature, to be able to better follow up on any stock moves, their amounts, locations and even expiry dates. An older tool just bypassed that very verification and nothing was logged or taken out of stock.

~

Taking out an amount for a certain orderline has a shortcut in place to mitigate some of the mandatory steps that pickers need to take in order to verify what's being taken. This little tool only available, visible and possible for a very few select users.

I assigned some orders to one of these people, which made him think it was an urgent batch. It's only one product, for multiple orders, so he went to the location, took out the amount needed and then used the tool to quickly be able to prepare them for shipping.

This bypassed the new methods to check if the location actually had stock to take, which I had just enabled for 1 account.

Luckily I caught the miss-hap as I was monitoring that product first-hand and noticed the batch of orders was collected but the stock amount didn't update.

It was 5min before I was leaving work, so I investigated and then ran to the person in question to ask what he did; which was "I used that tool"

I facepalmed myself internally while blaming myself, as he couldn't know that it wasn't ready to use for that purpose.

The tools to fix this up are there already.. so I used that to fix some missing stock-takes manually.. Though I'll need to close that little tool for these kind of orders for sure, asap, probably when I get home, at least until I bring over its new logic to it.

Happy Tuesday? (:

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

Looking to sharpen and pursue a SysAdmin/DevOps career, looking at online job offers to get the big picture of required skills and I say FUCK. It would take me a lifetime.
Azure, AWS, Google cloud platform.
CD tools: Ansible, Chef or Puppet
Scripting ninja with Python/Node and Shell/Power shell.
Linux & Windows administration
Mongo, MySQL and their relatives.
Networking, troubleshooting failure in disturbed systems
Familiarity with different stacks. Fuck. (Apache, nginx, etc..)
Monitoring infrastructure ( nagios, datadog .. )
CI tools: jenkins, maven, etc..
DB versioning: liquibase, flyway etc.
FUCK FUCK FUCK.
Are they looking for Voltron? FUCK YOU FROM THE DEEPEST LEVEL OF MY DEEP FUCK.

How about incompetent management? Company absolutely murders any possible increase in productivity. Laptop provided? Slow as balls. Takes minutes to log in. I get a Mac for mobile development and that's OK. SSD and adequate memory but I'm primarily a .NET Dev. Can't get on the network with a virtual machine. They won't I stall even a managed image. So can't use databases because they're all AD authenticated. Got a virtual desktop environment and that sucks worse in performance than the laptop. Add the Assault on local administration rights and the monitoring software that constantly thrashers any memory and hard drive usage and im about to quit over all this... All this decided by a non developer and not asked for our opinions. Yay large Enterprises

!rant

I'm a rather young developer, self-learned everything and started when I was 13 (now 20) but I still feel like I'm a total beginner since I have not yet mastered the things I am OK at.

Php (laravel, since it makes things much easier), js (jquery, bad at vanilla, have used angular and ember but not mastered), node, linux, html, css, photoshop, illustrator, sql, mongo and windows servers

I know little about many things, can create things that are asked of me but the methods I use are rather bad imo.. ex: I finish coding a section of a site, but when I need to add a new feature I find myself rewriting most of the stuff to add the new feature and in the end still feeling like the code could be optimized further, even though I have no idea how.

TL;DR I write bad code, but things work as long as I am monitoring them. I know little about alot of stuff but mastered none of them.

What should I do? Go to school for programming?

I fucking hate foglight!
Fucking piece of over bloated shit software that can't do anything right!
This pile of harkonnen vomit is such a source of stress and frustration.
Can you believe that a monitoring tool, that needs to monitor 550 agents (host,dB's, webservices,whatever) needs a whopping 20+GB ram and 4vcpu's just to receive metrics and sent it to the main server, for just being a middleman???

Fucking piece of shit.

And worse of it all, due to some fuckery at mgmt-customer relationship level, they want to I stall MORE of this shit

Fucking shit

Quest software, never engage with them. You'll lose equal amounts of money as for dynatrace but be very bad off

Okay so I’ve been brought in on a 12 month contract as an external replacement integration architect, alongside a large IT consulting firm. Turns out, they don’t need an integration architect. So I fill my time coming up with useful tools around the project that deal with all the missing parts in their MVP: like monitoring tools, data mocking tools, you get the idea. Essentially doodling.

Client has woken up to fact that they’ve overspent by X million, employing 30+‘developers’, 20+ ‘testers’, n+ ‘managers’ on a ‘low-code’ project … result: project shuts 4 months early.

Q: Essentially client wants remaining four months work done in two weeks. Is there a German word for laughing, crying, and banging the forehead on the desk at the same time?

Supplementary: how cross will client be when they realise project can indeed be done, and that consultancy have been emperor’s-new-clothing them for most of the last two years?

(Feel free to perform substitution on quoted terms at your leisure)

Job monitoring 😜😜

This 30 hour project is now 110 hours in and client is changing their mind so much that the managing director of the company is now sitting monitoring every correspondence between the client and developers.

#fml

Project 1: A hand hygiene monitoring system intended for the NHS.

Project 2: A language analysis platform.

Either a really big coincidence, or I'm officially creeped out.

I've been looking into buying a vps, so researching that a lot. Then today, I went to work, at a monitoring station, so we have to use remote desktops to access anything other than very specific sites.

Then I looked at an article about c#, and there was a Google ad, about a vps.. Keep in my mind, I'm at work, on a remote desktop, that gets cleared every time it's closed.

I know a vps isn't the most unpopular thing, but haven't seen an ad for it before.

Anyone have much success with Kali/WiFi penetration testing?

I've been tasked with trying to break WPA security within a couple of hours without a dictionary attack - is that even possible?

I have an Alfa AWUS036NHA capable of monitoring mode if that makes any difference. It's my first time trying anything like this.

Favourite thing you worked on recently?
——
I’m working on server monitoring system. I found that I need to pay for most server monitor services, and ones that are free/open source didn’t sound like they fit my use case (b/c the server i need to monitor is shared,) so I basically said “f*<k it I’ll do it myself.” I find it fun to work on something that I feel the need to make.

Monitoring goes does in one of our server racks, one I had just been working in (other side of the building). Hmm, odd. Oh wait its back online again. Better go take a quick peek to see whats up, just in case. Walks over to rack. Everything seems fine. Walks back to desk. About 15sec later. Fqdn.plsfckof is offline again. -_-

Monitoring tools madness: quest foglight.

So, setting a blackout for an FMS "HA cluster" (which does not work due to a bug infested custom jboss implementation) can bring the servers down... And no way to bring them back up.

This brilliant piece of enterprise APM software costs 600.000€ for a 5year license.

I,ve added more drama (logs, threaddumps, support bundles and screenshots) to the support portal...

45 cases now in total, oldest case still open date 2017...

Fuck you quest software

Just had a meeting about performance and monitoring. The main topic of the meeting was to be aware of disk space usage. If there are issues with memory leaks or processor hogging don't worry those are fine, just give it more.

Im not doing absolutely fucking anything as a DEVOPS ENGINEER other than dming 1000s of people to fix 1000s of fucking services and emailing other teams to enable monitoring tools its almost like im a fucking manager just telling everyone else to do the job for me since for every fucking step i dont have permissions and need to chat and wait for 1000s of people to approve im so fucking frustrated working this bullshit for 8$ an hour learning absolutely nothing useful and not progresssing

So I've been given a task to monitor a whole lot of logs of some servers (whole university ~ 10+ departments). The technologies are diverse so I'm cramming everything into elasticsearch via logstash (and filebeat), viewing it into kibana. Any recommendations for what should be the 'useful' stuff to be viewed into dashboard? I guess:
- Overall traffic wtih respect to previous days/weeks
- Most viewed domains
- 200
- 404
- 503
- Failed logins?
- Dropped connections?
- Critical-load of systems? 90%+

Continuing to learn k8s ecosystem and to achieve acceptable level
With trying eventually Helm, Argo CD and even trying to use not managed setup for k8s.

Going though books to find out theory about being SRE.
And about data intensive apps.

Learning and trying Kafka

Learning and trying FastAPI and diving in generally to async python ecosystem

Learning Go.

Learning few more books to increase code quality and its compositioning.

Getting more practice in monitoring and logging systems with applicating them to k8s.

RavenDB was by far the worst document storage "solution" I have ever had the displeasure of working with.
- Loading data crashed the service.
- Queries crashed the service.
- Monitoring applications crashed the service.
- It didn't support clustering or HA of any kind.
- Sometimes it just worked for no good reason.
- Often it broke for completely random reasons.

Am just chilling taking some coffee while some a**holes think a f**king AI will take my place.

I don't get paid to write code only, most of my work is architecting, fixing bugs, maintaining, monitoring and a lot more your AI can't do for now or anytime soon, i guess. AI is one of the tools that help me cook, you can say it helps a lot.

Do you know something else, i get lot of jobs because some AI freaks create junks and hallucinated codes and they come to me to align it with their business goals and fix their never ending loop with AI trying to fix their sh*t.

I f**king love AI, it create more job for me and its a helpful tool, so how am i being replace ? Am just taking a coffee while more jobs are being created out there for me.

Our ISP asked if I was satisfied with their service. I told them that it's okay, but some of our computers don't have enough powerful network card, and they can't use the internet on maximum speed, and they said that they could see it, too. WTF? I knew that the ISP's router is not the most secure thing, but it has a remote mode, which if of course OFF, and they still can see this, and maybe even more. Monitoring your traffic is a thing, but a home network should be private...

I looked at an SQL server today from a customer, talked with one of their devs and he said that he's unable to understand why the server misbehaves... All (!) queries were optimized, but they have 'big data queries'... Migraine started, I had a very bad feeling. Monitoring? Nooooppeeee. Migraine kicks in. Connected to server. SHOW GLOBAL VARIABLES...
After a bit of scrolling I found a lot of misconfigured variables (e.g. extreme large join buffers, unrealistic buffer sizes), high slow query count (nearly 60 % of COM_SELECT) and a few variables that were unknown to me.

Then came the version line.
5.0.46
Yes. 5.0.46.

Big data? Well... 30 GB of usage data.

I called the company back... The dev told me sternly that this was the production server (I had hope...) and that I lie - neither the version, nor the variables could be the problem.
A coworker had to verify it and our manager had to do the communication... Worst, most traumatic working day I ever had.

So a friend / batch-mate in our accelerator asked me if I was okay with installing a monitoring software for a client our startups are collaborating for. And the said client was ranting how I've been appearing offline to him since morning...

Bitch I'm already letting you monitor my shit from morning to night, I don't need your French ass snooping around what I'm doing outside of office hours.

Fucker.

Got the chance to get into developing a monitoring frontend...
Imagine the step in between if your previous task was *phone up*, *translating consumer problems into consultants problems*, *phone down*

Why does on-call schedule always ramp up shortly before the final hour?

Due to christmas and vacations I had on-call schedule, five days off, on-call schedule and both times, riiiight before my schedule ended, the monitoring/alerting system EXPLODED!!11!1

Seriously, I get the need for on-call schedule, but how are you supposed to get regular day-to-day work done when you're always on the edge because you're just waiting for the next alert to pop-up.

I'm so glad when I finally move from this company, no on-call schedule and other stuffs (which are unrelated for now, maybe in another rant) anymore.

I really respect people who are doing on-call schedule and such but I am definitely not meant for it.

short: The admin with enough xp is ill, there is no one with xp with varnish is and after 1 restart varnish outputs only 503.
long: there original admin is ill but he gave me an project to migrate an typo3 installation to a new server. Thats ok.
Plan: I move 150 GB of data with rsync to the new server, let specialists do something and switch ips between the new and old and clear varnish with a restart.
Reality: +2 hours to migrate the data, because of false infos from the admin, 7 hours preparing the switch, 5 minutes switch, 3 hours to find out the F*****G varnish is the single point of failure. I and the t3 guys agree to see the next day what went wrong.

ALL HAPPENED TODAY!

Plan for tomorrow: speak with the boss to account the extra hours to that day so i dont get over 10 hours and debug that fucking varnish and delete some servers from another project from the backupsystem and monitoring.

AMQ cluster is misbehaving.. Master is constantly dying, slave is starting under root rather than app account, connection to AMQ DB takes 20-40 seconds (while queries execute in <0.1sec and TCP probe takes <0.01sec to succeed), monitoring is down,...

I mean it's gotta be the virus - what else could it be.

So following my previous post, the issue happened again. And actually for background what I've been telling my boss, for years, we need ELK setup and integrated into all our APIs ASAP.

I think it's a punishable crime if any program is released into prod at a tech company with out real time logging/monitoring built in?

So issue still happening, user sent us the request details. So now need to find the actual now that handles the request and look into it's logs to see the details.

Now he's doing it the hard way.... Just finished took 1hr, and the best answer her can come up with is "I think .... Maybe ..."

And if course this is based on infinite data. He stopped after finding a "probably cause"

I have a script that is like promotion ELK, downloads all looks and parsed then so I can run queries to pinpoint the exact call and which log it's in. And can see what's happening around it.

We'll see what my way find but definitely does not take more than 1hr...

Loading data maybe but that's because it needs to download the logs and parse them all...

On a side note, guess I'm Beck on devrant as I have something to rant about. Though it's the same something that I was wanting about years ago... Monkeys...

Ok, I'm fed up with this, just read something about android constantly monitoring your phone's location, now it's time to shut this up.

Would you please be so kind and share information on which alternative "privacy-first" OS I could use and how to flash my device? For all I know, it runs a custom HTC modified OS. I'm quite unfamiliar with all those things gravitating Android. Heard about Cyanogen mod but that's about it.

What about compatibility with apps downloaded through the play store? (thinking about Threema) I would also need compatibility with WhatsApp (yeah, sucks, I know, but hard to convince regular people)

Thank you all :)

How do I know if my work computer is being monitored? :/ I'm using MacBook.

Does anyone use pushover.net or know a reliable / cheap push notification app with an api behind It?

Im trying to setup some monitoring alerts back to my phone, using slack / teams etc for personal projects seems pointless.

Emails are currently being sent but this relies on me actually looking at them and I tend to miss a few important ones. 😂

Not to worried about daily / monthly limits, it's going to be < 20 notifs a day kind of thing on a terribly bad day.

Whoops, my head will be squashed tomorrow. Asked to put monitoring in other week by boss, sysadmin been complaining about high CPU, apparently 10 requests (different domains) to the one VM on our servers every 10 seconds is killing it. However this server is being used for MySQL and serving web requests by Apache and PHP. Then also running a few jobs like consuming queues etc.

Wtf do I do? Every time I tell him about more resources (we have decent 2 rack servers just running 20 vms and only 1 VM is for web sites) he says software should be made to work with what we have.