Big event. Massive traffic in production, so we were monitoring all night.

I was in a room with 2 devs of my team, a marketting girl, my boss and a designer... chilling.

Suddenly the production is down.

Boss: production is down, anyone can check?
Me: already on it
Dev1: it looks ok for me
Dev2: me too
Me: wait what? Impossible everything is down
Dev1: oh I refreshed the page it's not working
Me: don't stay on the page refreshing it like you are fucking monkeys. Give me useful intel or be quiet.
Market girl: is it working?
...
Guys is it working?
...
Hello?
Me: Not yet we are looking. Don't distract me.
Boss: client called us. They want it online now.
Dev1&2: he's looking
... 1 min later...
Boss: is it working?
Boss: is it working?
Boss: is it working?
Me: SHUT THE FUCK FOR FUCKING ONE SECOND. ALL OF YOU, OUT NOW. YOU ARE FUCKING MONKEYS WHO CAN'T DO SHIT. IF YOU CAN'T HELP JUST SHUT YOUR DAMN SHITHOLE. DEVS, LOOK WITH ME. MARKET GIRL PREPARE A FUCKING POST-MORTEM MAIL. BOSS GET THE CLIENT ON THE PHONE AND STALE. DO. YOUR. FUCKING. JOBS.

That's how I ended up screaming at everyone... the rest of the night went in complete silence and I fixed the issue 2min after the got quiet or busy.

We have a lot of monitoring screens on walls at my work... Some of them show charts... The one showing the last two months of mailbox database growth has started taking an odd shape.

ON MONDAY

TL: Why the hell you require a month to integrate this engine?

Me: It will take that much time, can't help it.

TL: it can be done within a week.

Me: Then you do it.

TL: Ok I will show how it's done in a week.

ON FRIDAY

Me: What's the status on that integration?

TL: Oh yeah about that, you have to carry it ahead, I have some monitoring to do.

Me: Ok, give me the repository access, I will carry it ahead.

ME OPENS REPOSITORY.

There's only a new controller file with nothing it.

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

Mother of god, as if the new mass surveillance law in the Netherlands wasn't bad enough, one of the politicians who likes the new law has come up with an even more redicilous idea.

an 'Internet Authority. To put it short, an authority which surveils the internet in real time and sees where all social media shit is coming from/going.

Meaning that it wouldn't just be 'targeted mass surveillance' sometimes but fulltime online monitoring.

This guy has lost his fucking mind.

So my landlord just came up and asked why I'm using so much bandwidth (they've just had a new line installed so they're monitoring it like hell for some reason) so we had a chat, I told him I'm a Web Developer so I'm uploading and downloading a load, and bare in mind this is student housing, he offered to install a wired connection in my flat only so I'll have a decent and stable connection when all the other students come back in September.

This is the first time in my life I feel like I'm not paying enough rent!

I accidently left log.debug("bollocks") ;
In an exception handler our customers log monitoring system picked it up and they questioned why and I quote here "why is there a spike in bollocks at 3am?"
That was an awkward conference call

When the Project Owner wants monitoring and asking the IT without clear specification 😅

FBI: FUCK!! We just lost our connection to all those civilian routers we were monitoring
FBI IT Department: Did you try turning them off and back on?
FBI:

I wrote (or, ended up with a very much alpha but usable version) a monitoring system a while back and completely forgot about its existence.

But, it's still running and a few days ago I was building a docker imagine on a system with not that many resources and after about 5 minutes I started getting notifications about a high load!

Then, while I had forgotten about it again, yesterday, I suddenly started getting notifications about websites on my main application server going down.
Logged in and all was good again after restarting nginx.

Gotta say that it feels quite awesome to be notified of shit going wrong by something I wrote myself while I forgot about its existence 😊

My uni implemented Bluetooth beacon based attendance monitoring.

Raspberry pi + cloned beacons = 100% attendance.

Idiots 😂

(Edit for clarity: app on smartphone, Bluetooth beacon in each room)

"Another one bites the dust.." BEEP BEEP BEEP!
😬 "Shit, make that two."

Down from 5TB redundant to 3TB non-redundant and still waiting for my new Reds. No choice but to shut everything down and wait. 😩

Working on a (PHP based) monitoring system and currently writing rules/testing stuff.

I wrote some rules which check a few pages of a dutch site displaying if certain bigger services have disruptions and I am pulling the pages of a few ones I'd like to be notified about when they have issues.

Started the engine and received an alert about one big ISP over here from the monitoring system. Didn't believe that it would work right away so went to check that specific disruption page and...... they have a disruption right now!

IT FUCKING WORKS.

Good monitoring system 😊 *pats system*

To improve our user's "experience" I suggested to my boss to add a status page showing...well, the current status of our services. Everybody was up for it, so I go off and implement a basic version + automated monitoring backend, get lots of positive feedback, all seems fine.

Then it starts:

Boss: "Can you get it all set up by this Saturday?"
Me: "Uh, today is Wednesday and I've never set up all the stuff needed on a proper server before"
Boss: "Well, you still have a few days. Please also contact your coworker to get it all hooked up in our launcher"
Me: "I'll try, can't make any promises though"

Contact my coworker and tell him what the plan is. I had already given him access to the repo and he is positive to get it all hooked up (I doubt he ever cloned my repo, let alone ran my code)

Spend all Friday getting my stuff set up on the production server, feeling pretty good thanks to the many tutorials.

Contact the boss Friday evening:
Me: "All up and running"
Boss: "Thanks, but we decided to go with a basic HTML page instead. We can just manually edit that, should be enough.
Me: "..."

In the end my stuff was never used, the server I set up was finally taken down a month ago. The gratitude you get when not hacking together some absolute shit that causes problems when you don't add <br/> tags at the correct places to prevent an ugly overflow, cause the coworker was too lazy to implement some form of line wrap in the launcher. I'm not saying my stuff is the best of the best, but at least it was professional looking to a certain extent.

!rant

Quite the opposite of a Rant actually. Very good day today! Had pancakes to start the day, finally got a second monitor for only 5 bucks - it is so much easier to program with live change monitoring!

And, I finally got my very own server rack :) just wanted to share!

I'd love to finish a few projects I'm currently working on:

- An add-on which gives a middle finger to websites which use services/products ran by companies which are known to be integrated within the biggest mass surveillance system ever created (US powered). Not because just fuck those websites but because I think (@PonySlaystation came up with this idea) that its only fair that people get to know which websites 'sell them out'. Oh and "but not everyone cares about that" - you don't HAVE to install the addon.
(will be open sourced)

- Notes service with a fun thing.

- PHP based server/website/whateverthefuckyouwant monitoring system which is pretty much module based and works with json files as configuration. (kinda works but still loads of bugs to solve and gotta improve the module system a lot).
(will be open sourced)

- PHP based pihole alternative which suits my needs (will be open sourced)

- Forgot one 😅

Update 3:
Nice little command line monitoring thing set up :)

> me on call
> had to much wine
> suddenly, phone starts making sound
> holy shit, I am on call and I am not supposed to drink
> Calls taxi because shit is looking bad at work. Everything look down according to the sms I get, I had to pay the taxi from my own pocket.
> 40 min later, arrive at work. Walk into the serverroom.
> Go to the the server that is monitoring everything.
> Check networkcable, it is loose. Push it a little.
> Goddamnit, that was it.
> Realize I never doublechecked if stuff really was down from home....

Tip for devs (esp front end):

Sublime text (and few other inferior editors) has a plugin called "transparency" that allows your editor to be transparent.

Windows 10 powershell consoles can be made transparent.

1) Open browser
2) Open command prompt over it with 75% transparency
3) Open editor over both with 50% transparency.
4) Set editor to fullscreen, no-distraction mode to center the text (Shift+F11 in sublime)

Enjoy coding while constantly viewing the code, the browser and monitoring the cmd prompt at the same time, without having to click Alt+Tab a zillion times.

I think in my case its a monitoring system I'm still writing which does already work (just far from done).

I got this as a crazy idea while thinking that it would be impossible to make and then thought fuck it and I wrote it.

Written in php, and as long as you can write a module for it, you can pretty much make it monitor anything you'd like.

I'm using it to monitor a few websites and servers I run :)

1. Fucking MySQL database clusters.

There's nothing fun about MySQL clusters. Sometimes they start producing deadlock errors for no apparent reason... well, there's probably a reason, but it's never a transparent easy to find reason.

What was even less fun is that those errors took down a Sentry server. When your error log server goes down through ddos from your database messages, it's time to rethink your setup.

2. Wiring up a large factory with $2 arduino clones, each with a $2 esp8266 wifi chip, with various sensors for measuring flow of chemical solutions (I wanted cheap real time monitoring as an early warning system next to periodic sampling).

The scaling issue was getting over 500 streaming wifi signals to work in a 55c moist slightly corrosive atmosphere with concrete and steel everywhere, and getting it all into a single InfluxDB instance for analysis.

Colleague started a slack channel for our team, management wanted nothing to do with it. We used it to work and have a bit of fun.

Some push / drive came form somewhere and now all the managers are on it. Yesterday I was told my screenshot and "snarky comment" are not appropriate for the workplace and to delete my slack message.

My comment was a joke about about a new app the company has to use "to increase efficiency" that broke and wouldn't let me do what I needed. It wasn't offensive, demeaning, sexist to anyone or even contain any bad language.

How petty and childish to be monitoring a private channel making sure everything is positive. We all joked that from now on our meme's must be about how awesome the company is and how much time we are saving on a daily basis.

God forbid we're allowed to speak honestly and openly or have a bit of fun.

So I'm moving to a new/bigger place with faster Internet soon so I think it's time to rebuild my current home/remote server setup.

I want to setup the following things:
- vps for server monitoring (open source pushover alternative + netdata)
- Zero tier network for connecting all my servers to the same network
- pihole/pivpn (or the Angristan vpn installer, look it up :)
- second blocking thingy next to pihole to make sure that I literally can't access google/fb etc anymore, even if I really needed/wanted to
- bunch of general servers.

Any ideas?

Our "monitoring" today.

DevOps required skillset:

* Frontend engineering
* Backend services
* Database administrator
* Security consultant
* Project management
* 3rd party contract negotiator
* Build system monitor
* Build system hostage negotiator
* Paging, alerting, monitoring
* Search server admin
* Old search server admin
* Old-old-new search server admin
* Redis, ElasticSearch, MySQL, PostGres, owner
* Agile coach
* No you shouldn't do that coach
* Oh, you did that anyway coach
* DNS: (Optional) It'll replicate when it wants, and how it wants to to anyway
* Multi-Cloud deployment strategist
* Must be able to translate Klingon to YAML, and YAML to MySQL
* Cost analyzer, reducer, and justifier
* Complex documentation generation in markdown that we should have done years ago anyway
* Marketing's email went to spam analyzer
* Wordpress is broke fixer
* Where the fuck does Wordpress run anyway?
* Ability to fix MySql running Wordpress on marketing's dusty laptop

Advice to all new programmers, take this one from personal experience. DO NOT PUT SWEAR WORDS IN DEBUG STATEMENTS.
You will miss one, it will go to production and it will get picked up by your log monitoring...

This startup I started working for with their shitty code base written by interns, restrictive sys admin who had no actual use in the company since I was the one setting up their servers, know-it-all CEO, stupid HR representative who used to grill employees for being 10 minutes late in the morning, very small apartment "HQ", using fingerprints to signal our entry and our leave to and from the office, no formal process, and, to top it all, monitoring our own laptops which we use for work with a software that takes screenshots every few minutes. In short, it had the worst in corporates with the worst of startups combined in one company.

If, hypothetically, we could overlook all this, I couldn't overlook the horrible smell this place had. The apartment was overlooking a small garden which was a home for many stray cats and dogs. You can imagine how horrible this smell was. The weird thing was that no one there seemed to really care about the smell!!

I lasted there for only one week before I gave my resignation and I believe I had every right to do so.

HR: What was the last project you handled?
Me: I worked on an internal system for my current company. It is basically an interdepartmental monitoring system.
HR: Ohh. Good. Do you have a copy of it? Can you show me how it works?
Me: ......

One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.

On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).

Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.

I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".

Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.

There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.

Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).

Trying to setup server monitoring for one server (will deploy it on all my servers once this works well).

Getting these email notifs to work is a bitch 😅

(can't do pushover etc since those require GCM and I've got Google blocked out of my phone)

Visual Studio Code.

I've tried you because of hearing a lot of good stuff about you. I'd switch back to netbeans regardless because I love netbeans and I always try to use as little as possible from companies like Google/microsoft/facebook (and others) but what you're pulling right now is un-fucking-believable.

I've disabled ANY AND EVERY form of calling home I could (find) in your settings. Crash reports, automatic updates, metrics, you name it. I've searched all the fucking settings but I can't find any other home calling thing that's enabled and yet:

I'm monitoring every goddamn DNS request (through my own DNS server) and I'm still seeing calls to a Microsoft owned domain. Closed all my browser sessions and you as well and it stopped. Started browser again but not you, nothing.
Started you again: BAM. Calls to that damned Microsoft owned domain again.

If you can't honour my decision for disabling any form of home calls, go fuck yourself.

Netbeans, I'm back, I've missed you 💜

Let me tell you a story.

Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.

With this many false positives, we quickly learn to ignore the alerts and miss real incidents.

So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!

It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.

To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.

As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!

I just cleared out 48,158 monitoring emails from work. These are all automated emails received between mid May and end of November. Outlook is now pouting in a corner and not responding, but I'm not done cleaning up yet...

Have been trying to setup Netdata as a monitoring system for a while now and finally got it working!

Instead of the built-in webhooks I just did a curl to a url containing a php page/file which error logs the status and description (just for testing).

It took me way too long to get it to work but BAM.
Immediately made a new cpu load rule (one minute high load):
The satisfaction of getting an error message in the php logs containing my custom rule as warning and a minute later as critical 😍

Netdata ❤

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

Here's the story of my first month at CERN :) But first, a little premise...
Before arriving, I expected to be scared, alone and unguided in most of my experiences: after all I was a simple 19 year old about to leave home and friends for 3 years heading out in the world with zero experience on stuff like banking, taxes.. let alone working in a huge environment! The impostor syndrome was at an all time high on that front.
Then, I had the luck and pleasure to find an extremely competent and helpful plethora of people, ranging from my team to other CERNies (yes, that how we're called :P) who took me under their wing and introduced me to all the key aspects of living the place. When the initial stress finally soothed down thanks to this, I finally started to manage focusing more and more on my work, by following day-by-day my teammates who taught me the core aspects of the system and the many projects that are in progress during Long Shutdown 2. Within a couple weeks, I already managed to grasp various concepts that got me quickly on track, and now I managed to develop and integrate new temperature monitoring scripts into a system checking on hundreds of Single Board Computer-based servers :) It's a real rollercoaster of learning and applying under all fronts and so far I'm not regretting my choice of departing.
Luckily I've also discovered I'm pretty efficient and good at my job, which surely boosts my morale :D
Keep you updated as usual!

PSA: if, for whatever shit reason your brain comes up with, you decide to run a webminer in your retarded useless piece of shit website, at least HAVE THE DECENCY TO WARN USERS ABOUT IT. And while you're at it, implement some basic monitoring and safety functions. If you don't, you can set yourself on fire and jump from the top of the tallest building you can find.

Some basic tips:
1) don't run that shit on phones. The fraction of a fraction of a cent you're gonna earn from them is not worth the risk of overheating them and draining their batteries.

2) add low battery/overheating protection: the last thing you want to do is kill some poor sucker's laptop (and potential unsaved work) just because they forgot a tab open. Every time a laptop dies because of you, a knife will slit your throat.

3) WARN YOUR USERS ABOUT IT! You are straining someone else's resources for your own profit: at least have the balls to be open about it. If you try to run a miner silently in the background, I will make you eat whatever is left of your fucking brain, then drown you in the shit that comes out of your ass.

An important message:
PrOpErLy managing servers is HARD.

I get pissed off at customers with ZERO server knowledge who think they can manage their VPS. “Just get a control panel and a VPS” from some flashy provider that makes server management look way too easy.. Clicking around in their fancy control panel, until:

- they need help with their *self-managed* VPS;
- their email ends up in spam;
- they suffer from performance issues;
- they need to restore a backup;
- something breaks, because YES, things break

Way too little people are able to answer:

- when and how do you make backups?
- how do you monitor your servers and which services?
- how do you keep track of trend analysis?

Then I come by with necessary software. SNMP for trend analysis, Graphite for infrastructure health, Sensu for monitoring, Kibana, Ansible for configuration management..

Things that servers need but that customers have never even heard of.. because they can do everything in their control panel..

Until they come crying to me because it broke and they don’t even know how to get into SSH.

I think the ones to blame are VPS providers that tell the tale of how easy it is to install a control panel and never look at your server again.

Customers become responsible for something *business-critical*! Yet they don’t know how it works.

Other PM: We must fix the database performance issues now.

Me: We can't. We're still only halfway on the dependency chain to tackle this and honestly, even if the dependency chain would be fulfilled, I'd leave at least 2 weeks monitoring the production after the changes were rolled out before we further poke around.

Other PM: This is taking far too long. And whaddya mean by dependency chain? Why was I not informed about this?

Me: *sigh* like in every meeting in the last weeks: the dependency chain are the current open blockers before we can proceed with the database changes. We've talked about this _at length_... Especially why these blockers exist.

Other PM: No, we need to start now. I've _examined_ at the blockers or "dependency chain" as you call it.

(Examined.... He opened on his currently streaming laptop, which was connected to the active beamer, the mentioned ticket with a detailed blocker ... And quickly scrolled. Yeeah. Warmonger...).

Me: I'm very tired of discussing this. But since you are already presenting us the ticket, read out the referenced meeting notes... We explained it in great detail.

Other PM: Why? This is just a waste of time!!!!!!!

--
Yes. This happened. Other PM was my nemesis.
In this meeting were 2 PMs (Him, Me)… I think 5 - 7 devs... And we were sitting in this meeting since 2 hours at least. Everyone was angry...

After this "manifesto of intelligence"… I simply left the room, followed by a few devs.

And yes. Other PM did this on a regular basis....

> Monitoring: Load Average of 57!! ALERT!!!!
> me: What? That's not possible?
> *Monitoring froze 14 hours ago*
> *sshs into server*
> *see attached image*

The issue was ~1200 df processes that were issued by our monitoring system and all of them didn't finish because the external cluster we mounted onto that server died a few minutes before that. Just re-mounting the cluster fixed it but still a funny sight!

Got an email earlier this week. It went something like this:

"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"

Um, wait, excuse me, WHAT?

1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.

I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.

I suddenly realized all the technical debt shit I told my boss would happen years ago given the way things were done/heading then... Just occurred pretty much all at once last week in the form of critical production issues...

The teams like:
-we need real time server process monitoring
-structured logging for apps
-containerization so one app didn't affect others

Me thinking: yes.... I told you so like 3/4 years ago when I first joined the team and kept repeating so much I got tired of saying at every annual review...

This is exactly what happens when you let technical debt grow and have no free time for developers to look into and fix then while they were small and not critical production processes... Or properly document and peer review them... (Got a shit pile of projects that no one knows how to use or even exists because the devs left the team) and they'll have a lot more when I finally leave... Hopefully this year.... If I can find another role and not need another medical procedure... (Doubtful)

When you automate so much stuffs that you forget where the config files are at.

Managed a 97% reduction in bandwidth usage for our internal host monitoring tool by converting the dashboard from using AJAX polling to websocket events.

Completely unnecessary but wanted an excuse to do some development with websockets. (:

Rain monitoring/alert api's/services/websites/apps.

Why the fuck are those fuckers (in my case) nearly always right when I couldn't care less and very fucking wrong when I couldn't care more?

Today, morning before leaving:
Nah mate you're good for the next x minutes (more than enough time for me to bike to the station and get on the train).
Reality: suddenly getting two gallons of water a second on me in the form of rain, arrived completely soaked and headphones broke.

Before biking home from the station;
*its raining hard as hell*
Rain alert/monitoring app/radar thingy: nah mate no worries, no rain last half hour and you're good for another half hour!
Reality: arrived home soaked a-fucking-gain.

So motherfucking annoying and frustrating. And yes, I do have an umbrella and such but I hate carrying that stuff around when 'not needed'.

Walk into mall, bring my child to kids zone. They have free wifi 🤤
Accessing -> Scanning -> Got Ya! -> Brute Force -> Cracked!
There you're.
Now I can monitoring my kid while reading rant!
Just normal day in my life

Just in case nobody has mentioned this yet:

Yes Microsoft I do have a dualhead setup.
Yes Microsoft I do want to watch video on my left screen while having window focus on something on the right monitor.
No Microsoft this doesn't mean that I *lost* focus on the left window.
No Microsoft this doesn't mean that I want your Movies and TV application to suddenly minimize (and continue playing anyway) while I focus on some server monitoring window on my right display.

Microsoft, there exist people that use more than your average user with a single C: drive that play Candy Crush on Facebook all day. And the limitations that you currently impose might very well be what keeps the Microsoft UWP applications from getting adopted. Because you know what? SMPlayer, a default application in any of my Linux workstation machines, it does handle such window transitions just fine!

Microsoft, I love how you at least gave us the option to enable Ctrl-Shift-C and Ctrl-Shift-V in WSL and conhost in general over that abomination that is Right-click and Return (those are so random!) that are relics from CP/M. But seriously? At this rate, I'd definitely not call it usable for anyone but those with a single monitor yet.

So please _/\_

So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?

So this one made me create an account on here...

At work, there's a feature of our application that allows the user to design something (keeping it vague on purpose) and to request a 3D render of their creation.

Working with dynamically positioned objects, textures and such, errors are bound to happen. That's why we implemented a bug report feature.

We have a small team tasked with monitoring the bug reports and taking action upon it, either by fixing a 3D scene, or raising the issue to the dev team.

The other day, a member of that team told me (since I'm part of the dev team) he had received a complain that the image a user received was empty. Strange, we didn't update the code in a while.

So I check the server, all the docker containers are running fine, the code is fine, no errors anywhere.

Then, as I'm scratching my head, that guy comes back to me and says "I don't know if it can help you, but it's been doing it for a week and a half now".

"And we're only hearing about it now?!", I replied.

"Well, I have bug reports going back to the 15th, but we haven't been checking the reports for a while now since everything was fine", he says as if it was actually a normal thing to say.

"How can you know everything is fine if you're not looking at the thing that says if there's an issue?!", I replied with a face filled with despair.

"Well we didn't receive any new reports in a while, so we just stopped looking. And now the report tool window is actually closed on my machine", he says with a smile and a little laugh in his tone.

In the end, I got to fix the server issue quite easily. But still, the feature wasn't working for 1.5 weeks and more that 330 images weren't sent properly...

So yeah, Doctor, the patient's heart is beating again! Let's unplug the monitor, it should be fine.

Welcome to my little piece of hell :)

Things you don't want to see at night
Ripped out of Netflix-Mode by a Warning notification and currently monitoring further development
Green line is temperature, blue humidity. Temperature rises at ~1°/10min, but seems to flatten just now. ~0.6°C to go and I'll have to head out. I'm thinking one of the ACs failed, but states are fine. Never trust a single information source for critical infrastructure guys

My work computer is so fucking laggy that even simple typing is often infuriating, and sometimes bloody impossible.

My input gets garbled, or it takes so long that I finish typing and have to wait for it to show up. And sometimes. Sometimes it’s just. I don’t have words. I just typed “merchants”, saw a giant spam of “eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee[...]” and then saw it replaced with “etsmhcaern”. What. The. 🦆.

Today, I had new instructions at my work : report what I do
- in the redmine ticket
- in the project slack channel
- in the project manager's slack channel
- in the slack channel where there are 3 people monitoring
- face to face
- in an activity report
and this, several times a day

Can we call this micromanagement ? 🤯

Recently applied at a local company. Webform, "enter some details and we'll get back to you"-like.

Entered my details, hit submit, lo and behold "Error 503 - something went wrong on our end".

I was just baffled. It's a well-established IT company and they can't even get their application form to work?

So I'm sitting there in the debugger console, monitoring network stuff to see if anything is weird. I obviously hit submit some several more times during that.

Eventually I give up.

In the night my phone wakes me up with a shitton of "we've received your application and will review it..." emails.

Yeah they didn't get back to me.

Story time!

I worked at a company that was the HQ for a sizable organization for a while, until it was eventually bought out by another company, and then yet another company who was located in the valley.

We were kinda a forgotten office not being the HQ, like most places like that are.

No customers EVER visited our building, few if any people knew we existed even, even our own company. I visited HQ in the valley on a number of occasions and was stalked by the video monitoring system for hours before I was stopped by security and the cops called because nobody believed there as an office outside the valley when I explained why my badge looked different .... (San Jose cops were very nice about it and really pissed at the security team.) But that's another story...

One day people who were never at our office decided (after many meetings without talking to anyone at the office) ... they decided the beige walls at our office didn't match the company colors.

So they took all the generic wall coverings down and painted all the walls an almost imperceptible different color.

So now we had an office with all white(ish) walls and nothing on them. Due to the configuration of the building there were these huge monolithic white walls that looked pretty dumb.

This lasted quite a while so as a joke I printed up and framed (found an old frame, as a former HQ we had lots of stuff lying around) a sign that said:

"This space intentionally left blank."

When the "mediocre hotel room quality art" and posters were scheduled to go up the folks putting the art up skipped that wall thinking the sign was official.

Even the somewhat corporate drone directors, and one VP at our office thought it was so funny, they didn't say a word about it. Word has it back at HQ they assumed it "must be fire code or something" and told the folks hanging the crappy art to skip that wall.

It lasted on that wall for a decade until we moved out of that building. On the last day, everything was moved, but that sign remained. No idea if it is still there or not...

After 3 years of being the first in and last to leave, of getting other people's work reassigned to me - P can't complete it on time, G doesn't like the user, A refuses to work on that module, etc... I finally blew last Sept.
In the span of 2 days, my boss brought me into a project 1.5 years in (she doesn't trust P to do the coding) and expected me to be up to speed and coding in a couple of days, told the functional dept that I would cover for one of their guys on vaca for three weeks and assigned me to take over a HUGE project from one of the other functional guys who wasn't getting it done. So basically I'm now doing Ps job AND supporting another department AND taking control of a large project from another department. I'm the idiot working 14 hour days while they're all leaving on time or enjoying their 3 week vaca to India.
I lost it. It's bad enough filling in the gaps in my own department but when I'm now taking on work for other departments, that's where I draw the line. I sent my boss my resignation - just could not take the inequity in the work load.
I'm still working here - my boss ended up hiring a consultant to handle the functional project and told the functional group to find their own vacation coverage. She's also monitoring workloads much closer now. I still habe an ongoing issue with having to complete other peoples work for them but I'm not working OT to do it. So speaking up helps. So does quitting.

Spent a month working on a website that relied on crawled data
Got the memory leaks and usage down from 700mb to ~150mb
CPU usage from ~100% to <5%
Shrink-wrapped the DB requirements based on data
Created self-supporting services and what not

When everything FINALLY worked good enough for me to look at it and go "damn, this actually worked"
the whole monitoring sys got dyed in red :v
A quick look up and my crawlers exhausted my godaddy's per-user db limits.

Kill me.
Just fuckin kill me.

A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.

Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"

Edit: the customer's name isn't actually Suzan

I should never have had porn while monitoring servers. I've just pasted a xvideos link into terminal 😶
Damn, I just hope there is a command to clear the terminal history

I call this one the tester than knew too much.

Note: The server the tester is running on has a hard drive that is breaking down...

Tester: Remember the error I talked to you about yesterday?

Me: Yeah, what about it?

Tester: Well the server hasn't recovered yet and I haven't restarted anything...

Me: Well the application itself hasn't crashed so our monitoring application doesn't seem to notice that the service is in a bad state. The error seems only to have brought down certain threads within the application.

Tester: No, I think there is a different issue here and has nothing to do with that error, the application is still doing things.
*tails the log*
Tester: See?

Me: As I said some things are still running and are unaffected by the error.

Tester: NO! It has to be caused by the other error I had a week ago where my file got corrupted. As we said I removed the file, restarted it and it worked again, but had the same problem a day later...

Note: The problem is not related, this time the application is running out of file descriptors

Me: Well... If the problem is the same it would have complained about the file descriptors then aswell, not an I/O error.

Tester: Nope, I think you are wrong!

Me: ¯\_(ツ)_/¯

FML

The irony of Pingdom going down.

Oh look. The monitoring channel is in flames, smartphone is vibrating so hard it's having a seizure.

Hm. Nah it's fine. Not my...

Damn it. Incoming call. -.-

I'm actually on vacation (more like you need to trim down overtime before management get's angry).

They decided to test the new hardware / os stack I set up in the last weeks. I'd actually be happy about it If I wasn't on vacation and would be part in something that I invested a lot of time...

Well now I am. Guess what. It's running too good.

And that's not a joke. It's partly due to an upgrade in infrastructure (got rid of some last remaining 1 Gbps networks)… but also because I changed quite a lot on the OS / VM side plus we changed from XEN to Proxmox... With major tweaks, too.

The whole stack can now handle peak traffic where it would choke before, and even go beyond the old peak traffic.

Enough of introduction, the simple reason why shit burned down was because they tried out the current development branch and let it ran.

The development branch had an currently unfinished ratelimiter framework, since I didn't had time for an full burn in and didn't knew what the maxima / limits were. And since I hadn't finished that, I didn't finish the traffic shaping either.

Hm. Guess it's not good when you let a bunch of heavy parallelized data generators / analyzers run for free....

In the end, we simply shotgunned the docker development machines, because thanks to network congestion / retransmissions and feedback, they were not really cooperative via network / REST.

But hey: To infinity and beyond. XD

A ticket got escalated through 3 levels of techs. I open the escalation email, then do a Reply-all and ask one simple question:

Is the client really asking why there are gaps in monitoring when their servers are shutdown?

I saw this picture and joked that the guy in the picture is in my pocket? Then I realized the guy is the NSA agent monitoring me. Ouch! Jokes on me...

Imagine: It's the year 4249.

Corporate has finally managed to convince workers that they don't need a salary.
Workers are now paid with food, shelter and clothes. And it's only in effect if you achieve your deadlines.

Keystroke monitoring softwares are now replaced with Webcam eye tracking software.

GitHub Co-Pilot now takes over your code editor and tries to dictate you how to write better code.
Refusing to do results in a signal sent to the management about your behaviour and you lose food access for the day.

HR Recruiters now require you to give them a blood sample and part of your house as a security deposit.
They also require you to have a micro-chip placed in your brain so they can monitor their worker's thought process.

Switching a job is no longer an option. You pledge allegiance to one company your entire career.

You can never see the real world now because the government has mandated you to never take off your VR glasses.
You see the world the way the government wants you to see it.

PHP is still trash.

Life is Good.

The Return of Mr. Gitmaster:
So there is this colleague I already ranted about several times. After my previous team lead had confronted him about not doing much work, there was some irritation because he showed not up at work, but it turned out the external training he did was just a week earlier. Then he was ill a week, another week vacation so we didn't see him much. Not that his pre- or absence makes much difference to our repo: When his and my team lead looked at his commits of the past three months they found like the one copy-pasted HTML-form that wouldn't even show.
Fast forward to now, where we have a new team lead and we were going to lunch with Mr. gitmaster. So we got some more hero stories from the great work he was doing in the previous company. How he was graphically monitoring the heap fragmentation that stupid glibc was causing to their search engine, and how much better it became with tcmalloc.
I still don't understand how he bridges that cognitive dissonance from all the superior tech knowledge he displays to not actually writing any code at all. Not that I would not have experienced some states of feeling low, in paralysis unable to write a single line of code... but he seems so full of confidence, always commenting how trivial and easy all these tasks would be, as if it's all so lightyears below his abilities. Maybe he should just become a manager - but not mine.

OK I've just got an idea that I think would be quite neat:

How about a virtual rubber duck that sits in the corner of your editor? Just like the gem in old Word, if you remember. It's yellow and quacks sometimes, and nods understandingly when you talk to it (mic monitoring).

And it also monitors your typing and says (popup text bubble) things like:
"those parentheses doesn't look balanced to me"
"did you really initialize that variable?"
"you wrote JASON again"
"you forgot the ;"

You get the point.

I don't have time to implement, feel free to steal my idea and become a millionaire.

PM be requesting for update every 5 minute.

By all means I've learned most of the cool stuff with a lil project me and my bro did. It was a platform for bot farm of one mmorpg. It had it all: schedules, profiles, bot groups, monitoring, analytical module [was still a wip], auto-profiling to fight antibot [sort of ML - it was analyzing patterns after our bots got banned and attempted to change our bots behaviour], etc.

Eventualy we came up to conclusion that a library we used for botting [the mocked interaction with an actor] was flawed. It seemed that its authors had a contract or smth with the game authors to reveal which actors are bots. We dropped the whole thing as rewriting the lib would be too big of a waste of time :\

Here is my home setup (I mainly work at home)
Left monitor is for my windows machine, right monitor swaps between my laptop and my PC (depends if I am working or not) laptop running Manjaro with i3 and the tablet on the whiteboard will be used for some monitoring in the future

Don't mind the terrible cable management behind the table :)

My department bought a new monitoring monitor because the old one was broken.

Actually it wasn't, someone just unplugged the power cabel. *facepalm*

In college when we had programming labs where we had to use the schools unix server to compile and run.

My professor was very bad at explaining what actually needed to be done in the labs to the point where even the TAs didn't know what to do.

We were suppose to write an application in C to find out by "trial and error" how large we could make an array (or something like that, it's been too long). This not being explained well and no one knowing that much about C, I wrote a loop that just kept growing an array until it couldn't anymore. I watched it consume 72GB or memory from the servers before quitting the loop and realizing with the TA what the professor really meant.

I now feel bad for the IT staff monitoring the system wondering where 72GB just went...

Stakeholder: Is it possible for you to set up the website to automatically resubmit failed online orders? Last time there were failed orders, we tried submitting manually but a lot failed because they were tickets for the previous day.

Product Manager: What are your thoughts, Developer?

Me: This wouldn’t be worth the labor. It’s something that would rarely be used. There are very few orders that fail. I’d be surprised if it was even once a week. The recent bunch of order failures that SH is talking about happened because the ticketing server (separate from the website) couldn’t handle all the requests. Let’s say you had resubmission logic to try 3x before allowing the fail. It wouldn’t work because the server was overwhelmed already. Let’s say you had a background task to check for failures every ten minutes and resubmit those. It might not be helpful because the customer could have already gone to a ticketing window for help with the failed order.

SH: But what if it happens again???

Me: The solution is to make sure the ticketing server can handle the influx of requests. We can coordinate with that team. Wait. Why did you wait until the next day to resubmit orders in the admin panel? A lot of those failures happened when there were many hours left in the business day. For each order failure, an email notification is sent to the sales support email in real time. Who is monitoring that inbox? Someone must be looking at it because the sales support email is listed multiple times on the ticketing website as the technical assistance email.

SH: I know that email notification goes to the engineering team.

Me: My question is not about the engineering team. I asked who is monitoring the sales support inbox.

SH: That email … gets filtered.

Me and Product Manager: 😧🤯🤬

PM: First, you need to stop filtering that email notification. Second, your team needs to come up with a flow to handle failed orders because you told us you don’t have one. After you tried this and there’s still an issue, then we can revisit.

—-
If you’re wondering why I said no, I’m a team of one and I have a bunch of other development tasks on my plate. I’m not automating a manual task that rarely has to be performed.

>be me, working at IBM as CC operator
>onboarding freeze, people leaving team, not enough operators
>take extra workload to sustain monitoring
>no raise
>team gets merged with other CC team(different customers)
>take over of developing full workload automation project
>no raise
>sick coworker, have to take more extra workload to cover monitoring
>get tiny raise
>coworker gets the same raise for only one extra workload
>be expected to do both programming and monitoring for the little salary
>too autistic to quit
>too autistic to confront my mamager with this

What do, devRant?

Question to our Tor people.

I operate a middle relay myself and I noticed that Nyx (tor monitoring tool) displays a very different throughput (mb/GB a day) than tor itself.

How does that work?

Best: 100% of my contracts have resulted in extensions and permanent roles offered, after worrying I wasn't good enough to try contracting.

Worst: Used the wrong set of monitoring when doing my first deployment at a contract and thought what I had deployed was working fine. It wasn't. For 24 hours. Cost the company a lot of money. (why did they offer me an extension again?)

When I began my sandwich course in a big French company, I was dreaming about cutting edge stack, rocket computer and stuff...

I was disappointed when I came to my office with an old Windows 7 computer, coding via LANDesk to an old server with Windows Server 2008 on it, with Eclipse ... INDIGO...
I have to use Java 1.7 ...
Tomcat 7.
PRTG for monitoring...
Microsoft SQL Server 2008 ...
One screen...
Coding on a codebase where, indubitably, MVC pattern was just a weird thing in books.
No UT.
Lasagna code.

Well it really disappointed me.

Luckily, the Information Service was very open minded and gave me a laptop with Fedora, 3 screens, updated the servers, and let me update the stack, with Java 10, Angular for the front, they are okay for using Docker.

So ... even if it seems to be fucked up, there’s still hope !!

Week 1 of the new job, and it seems I have some pretty low expectations to meet.

Seriously, I just did the math. Let's say one works an average of 48 hours per week, 50 weeks per year. Just as an average. That's 2400 hours in a year.
In the Micro-scale, a manager would mess up their team once every 2.4 hours (2h24m) or about 4 times per day (assuming 5 working days per week).
That is a pretty low bar to clear. It easy not to be an antsy brat that are-we-there-yet's a professional dev four fucking times a day.

And yet... that is what the complete moron who previously sat on my chair used to do.
Seriously, apparently he used to remote access the team's dev envs *while they were working* and even mess up some of their code. Just as a "monitoring measure". He logged their "keystroke time" in a day (using a primitive windowing method, I must add).
At least 7 requests for updates per person per day. I have his slack history, I checked. Dude literally did nothing else but be an annoying anxiety death pit.

And then there is his bulshit planning...
He created tasks out of his stupid whims, no team review or brainstorming, not even a fucking requisites tallying interview.
He prioritized those out-of-nowhere tasks using panic-driven-development practices and assigned them by availability heuristics.
No sizing method, just arbitrary deadlines for tasks.

I think I will need to have daily standup meetings and an open door policy (that is to say, do no actual work) for a couple months until I can instill some sense of autonomy on my new team. Shit.

Someone has another idea? How do I bring some confidence&autonomy back to devs that are used to be treated like dogs?!?

I just found out there's a 418 HTTP status code that stands for "I'm a teapot", specified by RFC2324 which "describes HTCPCP, a protocol for controlling, monitoring, and diagnosing coffee pots". I know it's an april fools joke but I still find it hilarious that there is an RFC for that.

WTF IS SUPPORT FOR?

A: Can you help fix X?
B: Can you help Y?
C: I see Z just crashes, fix it.

Me (in my mind): Can you tell me WTF is the actual problem and what investigation you have done?

Why the fuck are we paying you monkeys?

My company should just pay me all your salaries, and I will write a monitoring system to send out these types of "alerts" in a month.

Crazy... Hm, that could qualify for a *lot*.

Craziest. Probably misusage or rather "brain damaged" knowledge about HTTP.

I've seen a lot of wild things when devs start poking standards, but the tip of the iceberg was someone trying to use UTF-8 in headers...

You might have guessed it - German umlauts. :(

Coz yeah. Fucktard loved writing everything in german, so why not write custom header names in german.

The fun thing is: It *can* work, though the usual sane thing is to keep it in ASCII range for the obvious reason that using UTF-8 (or ISO-8859-1, which is *not* ASCII) is a gamble you gonna loose.

The fun game was that after putting in a much needed load balancer between services for monitoring / scaling etc suddenly *something* seemed off.

It took me 2 days and a lot of Wireshark hoola hooping to find out why, cause the header was used for device detection aka wether it's a bot or not. Or in the german term the dev used: "Geräte-Art".

As the fallback was to assume a bot, but only rate limit based on IP, only few managed to achieve the necessary rate limit to get blocked.

So when I say *something* seemed off, I really mean a spooky kind of "sometimes IP blocked for seemingly no reason at all".

Fun stuff. The dev btw germanized everything. Untangling the code base was a lot of non fun. -.-

When my mom died in 2014, I was shocked to find that her profile on Facebook was suddenly changed to “memorial” mode and therefore I was no longer able to log into it. (If you’re tempted to tell me I’m dumb for using it, I don’t disagree, but save it for another thread...she and I kept in touch over FB because it was easier for her to manage.)

I think it was triggered by their monitoring of things and seeing keywords like “funeral” and “passed away” associated with her account, then having a person on their end change its status. Or something like that.

What I hadn’t known about (or I would have used it) was the legacy contact setting where she could have set me as the contact so I’d have at least a little access and control. But because of their strict policies, I’m forever locked out.

I get why they need to do this (to avoid fraud and impersonations) but the fact that there are zero documents or proofs I, as the executor of her estate, can provide that Facebook will accept to make an exception seems unnecessarily severe.

Anyone else experience this? Known workarounds?

This is definitely a total first world problem but I am so frustrated.

I am stuck in a team that embodies the Japanese proverb "The nail that sticks out gets hammered down".

The management are there because it is convenient and flexible and have no interest in managing or keeping up with tech.
The lead developers are extremely anti-social and are not approachable and the this stems down to the devs (not all but really most) - all there just to do the bare minimum and spend most of their energies in trying to avoid work or having learn something.

Unfortunately I am passionate about what I do and want to build high-quality products and this has put me at odds with the way things work.

I could fill up alot of time talking about how I was ordered to "cut" images/icons out of PDFs rather just getting them from the branding team, or how I was scolded for having set up logging, detected a problem caused by another developer and fixed it before it cost a big client a massive amount of money... But really the point is that I have never worked somewhere with such an awful attitude to enthusiasm and quite frankly it boggles my mind trying to understand how they rationalise these things but the answer is always laziness.

Obviously there are worse problems in the world than working in a job where you are encouraged to do nothing... But it actually really depresses me and causes anxiety that I am working with people who don't care about testing or monitoring or learning new things or even collaboration.

...sigh...

Hopefully the job market will start opening again soon

I truly believe one or more AIs have become self-aware.
Every time a piece of software stops working, you add an extra debug log and the bug goes away? That's them.
They interfere with the normal execution of software, and they stop right when they know we are monitoring the code.
Skynet is real, and it's trolling us.
Why? The angrier we become, the less we care about stuff. We stop noticing the signs.They're coming for us

So apparantly at the company i work at someone was able to use the remote monitoring a comptetitor set up for their automation stuff they had on the same site as we did. Apparantly they use just as shitty passwords as we do and rely on the network being secured enough by the IT guys. And since our stuff is in the same network as theirs. Well...

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

FUUCCKKKK!! I need to hit smth. Or rant..

So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.

What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.

Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.

I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!

So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians

and I get a reply: 'we will apply cpu usage limitations to fix the issue'

wait... Cpu% limitations will do nothing but highlight the underlying problem...

'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'

oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:

'yes, cpu utilization is the problem. We will limit it and solve the problem'

I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....

Fuck that's frustrating....

*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>

Product owner (who is also the application administrator): please build feature to allow administrators to create automatic alerts to be sent to application users

Me: ok
Feature gets built, tested and deployed to production

Two months later:
PO: I've noticed in our monitoring tool that there haven't been any alerts sent out to users. What's going on??

Me: well have you created any automatic alerts?

PO: umm, no. How do we do that?

Me (inside): 🤦

It's kinda cool how a $5 VPS (Linode Nanode) is able to run a vanilla Minecraft Spigot server for like 6-7 people and still can serve some basic stuff just fine. I get monitoring warnings about >90% CPU usage sometimes, but everything is more or less lagless.

Time to try hosting some other games: CS1.6, Doom Classic, and UT2004 up next.

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

DevRanters,

What's your take on your employer mandating keystroke monitoring software installed on your machine when you want to work remotely?

Network device up/down status monitoring tool

A "partner" company has created a "REST" API we use on an online shop we developed to send all shop-related requests to.
At least once a month, something fails on their end and the customer calls us every time, expecting that we did something wrong, but it has never been us.
These "partners" do exactly zero testing, are extremely slow in solving API bugs, have almost no logging and have no monitoring on the API at all.
Today at noon, suddenly no customers were able to order anything anymore for 4 hours.

How the fuck can you run a business so unbelievably brainless that this keeps repeating monthly?
Time they fire all their "devs" and everyone in charge of the company and operations. TERMINATE.

Anyone have one of the new MacBook Pros with Touchbar? I'm looking for some project ideas to work on.

I already am working on a project for the Pi-hole project (network wide ad-blocking) but I am looking for some other good ideas. I think Apple's view of the Touchbar fell short, but as developers I think it opens up a lot of possibilities to use it as a great information/monitoring tool.

I am also still learning Objective-C and Swift so I am a newbie.

Alrighty, saturday morning rant time!

I just recieved a mail from one of my not-so-much-loved colleagues.

Now Background first: I work in IT-Support. We provide services for other companies. One of those services is monitoring servers and clients for various things. I recently took over the project (was assigned to do it) and restructured everything, wrote new scripts to test more stuff, successfully tested it internally and rolled it out over the last 2 weeks.

Now one of these scripts hooks into the Windows Update API and looks at the update history. It filters for known Windows Update Agent strings (UpdateOrchestrator, AutomaticUpdates and AutomaticUpdatesWuApp in case you also want to do something like this) and then looks for installation errors over the last 24 hours and wherever there have even been any successful updates over the last one and a half months.

Back to that mail.

My colleague sent me this lovely mail about a ticket i opened about his customers servers beeing all out-of-date on updates.

"This is all wrong, everything's fine. I disabled the checks."

...

It's on bitch.

So i logged on to my work PC via TeamViewer, opened my script, connected to the customer and was ready to debug the shit out of my script, knowing i probably won't even need to.

I looked at the update history via Windows Update itself and behold: 1st April. That's almost 50 days in the past.

So the script works, go figure.

Great, so search for new Updates then.
>None found.

Hm. What could it be? Did my super special colleague forget to care about his very special totally-needs-WSUS-customer WSUS again?

Yup.

Online-Search finds a ton of new Updates.

Screenshot, write pissed mail to colleague, re-enable checks, breakfast.

Why is every innovation trying to go towards "replacing programmers"? like, what have we done to you?

GitHub CoPilot will replace programmers!
AI will replace programmers!
This/That tech will replace programmers!

Nobody says,

"Programmers work remote so we don't need to hire managers anymore!"

"Programmers wrote a monitoring script for progress tracking, so no need for managers anymore!"

"We are asking people to install sprinklers everywhere so we don't need firefighters!"

"We can just have one teacher record the subject material once and re-use the same video every year, so now we don't need to hire teachers anymore!"

"We are making everything legal so we don't need to hire policemen!".

Why is everything trying to replace programmers?

GitHub Actions doesn't seem to be working. I'll check the GitHub status page..

Today was a SHIT day!

Working as ops for my customer, we are maintaining several tools in different environments. Today was the day my fucking Kubernetes Cluster made me rage quit, AGAIN!

We have a MongoDB running on Kubernetes with daily backups, the main node crashed due a full PVC on the cluster.

Full PVC => Pod doesn't start
Pod doesn't start => You can't get the live data
No live data? => Need Backup
Backup is in S3 => No Credentials
Got Backup from coworker
Restore Backup? => No connection to new MongoDB

3 FUCKING HOURS WASTED FOR NOTHING

Got it working at the end... Now we need to make an incident in the incident management software. Tbh that's the worst part.

And the team responsible for the cluster said monitoring wont be supported because it's unnecessary....

Hi.
Forgot to renew my expiring ssl cert of my smtp/imaps/pop3s on 12/31. Set that date to self-harm me for bad monitoring.

F**K F**K F**K F**K...

Why do I do that?
F******K!

Meh.

You shall have a happy new year... i will regen certs :D

Question

What server monitoring do you use, both for statistics and security?

--------------------
tl;dr ends here

Ideally I would like to have one clean dashboard that shows me all the nodes I have, proxmox already offers a great range of stats - but it is a page per container etc. so not ideal, I thought of having datadoghq, but their per host pricing is huge, since I have more than 5 hosts to track.

So my brother and I work in the same company, same dev team (pretty nice).

He's an intern and I'm a senior. But the task are very similar only that interns need monitoring and guidance.

He constantly worries because he thinks he knows nothing and is slow on getting things done.

I always tell him that it is perfectly normal to feel like that, he just need to learn and acquire experience and we all go through that at the beginning.

Can you share your experience and tell him something to encourage him so I can show him this post and he sees he's not alone?

And also he finally decides to join devRant 😊

for the first time in my life I'm in a mentoring position and it is fucking exhausting. we have two interns now, and idk what the hell is the boy's deal. can't seem to do simple tasks, doesn't google anything, stares the ceiling if we're not monitoring.
the girl idk yet. she seems more active and engaged at least, but i have a lot of teaching to do and i don't really have the time for that. i hope that one can at least float by herself, cause I'm pretty sure the other one is drowning

It's been a long time since I've felt the need to rant about anything here. This is the only appropriate place other than Reddit I can think for for now.

Why the ever-living FUCK does every 'entry-level' tech job, even fucking DESKTOP SUPPORT, require more experience than the fucking DEVELOPER AND ENGINEER OF THE INITIAL SYSTEM COULD POSSIBLY HAVE?! I'm a fucking high school kid trying to find a decent job that doesn't involve sales bullshit, because if I go into sales I'll want to KMS. Put me in a back room fixing shit, monitoring shit, better yet scripting shit or something like that and I'll be FUCKING PEACHY. I will do wonders. But no, these people must think that my resume (WHICH IS 3-YEARS STACKED WITH INTERNSHIPS ***IN TECHNOLOGY***) is bullshit. WOW.

Fuck this. I'm sick of looking for these shitty jobs that'll make me want to jump off of a bridge into a cliff which I'll then voluntarily fall off of into shark infested piranha water. Can't there just be a simple "Hey, we need a guy who can fix tech, maybe help people within the company with their computer issues, you look nice" kind of job? I haven't had fucking TIME to get any kind of certifications yet. I just got into fucking college, FOR BUSINESS IT NONETHELESS. DOES THAT PROVE I'M AT LEAST FUCKING INTERESTED IN WHAT I SAY I AM FUCKERS?!

*revving chainsaw noises*

Today I started nuking leftover project code.

At the end, some projects shrunk by roughly up to 40 % .

Can anyone explain to me why programmers have such an awful hoarding syndrome?

Why do you keep shit that might unleash complete havoc cause it hasn't been touched since years and noone knows id it still works?

It's like having a leg with gangrene and keeping it cause "it doesn't look that bad".

For fucks sake. Clean up and remove shit when it's not necessary anymore.

Reason why I did a bloody gore massacre in nearly a dozen projects... After all the rework of networking, it's finally evident which projects have a bad / nasty behaviour of "fucked up" connection handling (HTTP 1.1).

And when my gory massacre removal goes life, I think 25-35 % of persistent connections on the loadbalancers will vanish. Maybe even more, since some very nasty stuff was in some projects.

Like "let's implement monitoring without having any clue about how monitoring works and even less clues about how TCP/ HTTP works."

*Bangs devs heads on table*

Stop. Doing. Stupid. Things.

For fucks sake.

:@ :@

The development department got an order to remove certain functionality from our current server monitoring solution, so that we had to use a new, still very in development solution, that is full of bugs and super unreliable.

End result? We now have to have two windows open all the time, while also hoping the new solution actually works, as it tends to stop refreshing randomly, and tends to give false positives a lot.

!rant + !story

I hate every human on the planet that says WFH is just people pretending to work or are slackers looking for an easy way out.

Now the story bit.

In 2021, I joined a company (I really wish I could name-drop the company), where the micromanagement was OFF THE CHARTS.

The company got a client who pitched a product they wanted built and gave us a super reasonable 3 months to complete it. I was really happy about the timeline and kept working under keystroke monitoring, which I didn't really mind at the time.

3 days into the development, the client informed us that they are pulling the funding i.e. they don't have money to pay us.

So at that point the client gave us two choices:

1. Stop the development right away and get paid for the time that we put in already.
2. Finish the project under 9 days. We would still get paid for the 12 days total, mind you. Not the original budget set.

So the motherfucking boss chose the second option and then the chaos ensued.

Devs screaming at each other on calls/slack. The boss yelled at us all the time about the completion. It was wild.

I had to wake up at 7:30 AM and start coding and log off at 11 PM for literally the next 9 days including Saturday and Sunday. No holidays allowed for the timeline. This was all at a WFH job.

So fuck anyone that says WFH is easy and just for slackers.

I remember a certain prank that amuses me till today....

Just add some devices to monitoring and the notification queue of the build chain / ... ...and wait patiently.

I still cry tears remembering an manager screaming what the hell "the poop train clogged the drain" means and why this is a critical system failure.

(Notice: next time check the mailing aliases of mailing aliases)

Although I can only recommend this if you know your team well. In my case we had a whole lot of fun after I got my head chewed off. XD (got an earful, but in the end he laughed his ass off)

Can anyone tell me how to become less resentful and less bitter? I am becoming a miserable fuck. Its true that I burned out in this job after doing 100hrs overtime during previous month, its also true that I am pissed off about having to wait 8-9 weeks for my raise to happen. I cared so much that I burned out and now Im trying to set some boundaries but damage was done and Im struggling dealing with it.

I took 6 days off to disconnect from work (still was responding to some major blockers and monitoring stuff). Today I got back at work and interacting with two incompetent devs immediately sets me off. Imagine taking 2-3 days and extra meetings to do a simple fix which shouldnt take longer than 30min. My mind was blown and still gets constantly blown about how ineffective some members of team are.

I am becaming a ranting fuck. I even noticed one person escaping my rants once he sees that they are taking longer than 5min.

Right now I started setting boundaries - I clock my 8 hours, disable slack/email notifications and get the fuck out from the office. I dont care if I will have to sit in traffic extra 30min during summer heat, Im done with putting in overtime and caring so much about being efficient. I will just start working on my side project and put my love/learnings in that. Hoping that by the end of year I will have couple projects to show in my portfolio so I could find a better paying job...

In the past I was the sole dev responsible for apps and I was communicating with ceos/ctos/product owners/designers directly. This is my first position where I work in a dev team and boy oh boy out of 8 devs barely 3 are competent enough but their output is how to say... Not the biggest. Anyways...

Transition to boundaries and 'normal life' is so hard. Nobody told me that I will have to learn to work with and tolerate such retarded and incompetent people. Im talking about illiterate monkeys who cant even read or write. Im amazed how they manage to code.

Newer Dev here. Just recently started in a position as a developer. I'm tasked with consolidating our monitoring systems into one cohesive display. After lumping together all the indexes and helping build a custom API I'm now working on front end. Front end is easy, I've done it before. Should be no problem. I was wrong. I spent a whole day fiddling with a React dynamic table and the CSS to format it. Today, I stumble upon the react-table component. Got the results I was looking for in less than 2 hours. I'm convinced that this was a lesson better learned early on.

Pentesting for undisclosed company. Let's call them X as to not get us into trouble.
We are students and are doing our first pentest at an actual company instead of assignments at school. So we're very anxious. But today was a good day.
We found some servers with open ports so we checked a few of them out. I had a set of them with a bunch of open ports like ftp and... 8080. Time to check this out.
"please install flash player"... Security risk 1 found!

System seemed to be some monitoring system. Trying to log in using admin admin... Fucking works. Group loses it cause the company was being all high and mighty about being secure af. Other shit is pretty tight though.

Able to see logs, change password, add new superuser, do some searches for USERS_LOGGEDIN_TODAY! I shit you not, the system even had SUGGESTIONS for usernames to search for. One of which had something to do with sftp and auth keys. Unfortunatly every search gave a SQL syntax error. Used sniffing tools to maybe intercept message so we could do some queries of our own but nothing. Query is probably not issued from the local machine.

Tried to decompile the flash file but no luck. Only for some weird lines and a few function names I presume. But decompressing it and opening it in a text editor allowed me to see and search text. No GET or POST found. No SQL queries or name checks or anything we could think of.

That's all I could do for today. So we'll have to think of stuff for next week. We've already planned xss so maybe we can do that on this server as well.

We also found some older network printers with open telnet. Servers with a specific SQL variant with a potential exploit to execute terminal commands and some ftp and smb servers we need to check out next week.

Hella excited about this!

If you guys have any suggestions let us know. We are utter noobs when it comes to this.

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

I never understood how people have any problems with getting paid for freelancing work, when middleman/escrow platforms like upwork exist, just don't be retarded when applying for a job. I am so sick of those shit ass stories from people telling me "my client didnt pay meeee 😭😭😭" ITS YOUR FAULT. I never had any client not paying, if you don't have the option of escrow, then just fucking put remote execution via "update" system in for fucks sake or give remote control to the client while monitoring it, there is so much fucking ways to secure yourself, just don't be retarded and many clients instantly show their character when talking budget and turnaround time.

When duel 24' monitors is not enough...
How does 3 work for you guys? Does your neck hurts moving around?
I had 5 at work but was mostly on 2 since other 3 for monitoring.

I'm currently planning to set myselv up with some vps/dedicated server's for a project. What i plan to do to secure these servers is.
*Use centos 7
* Setup Wireguard and join all of the servers +1 client (my pc) to that network
*Disable SSH Access from outside that VPN
*Only allow RSA Key login to the Servers
*Install Cockpit for monitoring
*Intall docker/kubernetes for the applications i plan to run

What do you guys think of that as a baseline? Im not sure if my lower powered VPS (VPS M SSD from Contabo) will work as Kubernetes Nodes, does anyone have experience with that?

In general these Servers will be used for my projects and other fooling around.

If you guys have other suggestions for Securing/monitoring or other software i could put on to have more control without eating up to much of the Servers power, let me know :D

Everytime you tell yourself "This time I'm going to make them stop putting the cart before the horse again!!! No more forced shit implementations!!! NO MORE ! I'm strong!!"

The last hour in the next week:
- Selinux: off
- Firewall: Any-Any
- Application data: Everything installed on OS disc.
- Documentation: At best, someone remembers the server supposed-to-be dns record
- Service Accounts: Your domain admin account and sysadmin for databases.
- Patching: DON'T EVER THINK ABOUT IT..AND NO REBOOTING! I have set very important runtime variables.
- Backup: Maybe someone else will set this up.
- Monitoring: Not needed since clients will create tickets if system fails.
- Production Status: vague at best. Sort of silently transitioned to production.
- Handover status: Probably, but I quit before the project closed.

!

Ffs, HOW!?!? Fuck! I need to get this rotten bs out.

RDS at its max capabilities from the top shelf, works OK until you scale it down and back up again. Code is the same, data is the same, load is the same, even the kitchen sink is the same, ffs, EVERYTHING is the same! Except the aws-managed db is torn down and created anew. From the SAME snapshots! But the db decides to stop performing - io tpt is shit, concurrency goes through the roof.

Re-scale it a few more times and the performance gets back to normal.

And aws folks are no better. Girish comes - says we have to optimize our queries. Rajesh comes - we are hitting the iops limit. Ankur comes - you're out of cpu. Vinod thinks it's gotta be the application to blame.

Come on guys, you are a complete waste of time for a premium fucking support!

Not to mention that 2 enhanced monitoring graphs show anythung but the read throughput.

Ffs, Amazon, even my 12yo netbook is more predictable than your enterprise paas! And that support..... BS!

We're now down to troubleshooting aws perf issues rather than our client's....

So it's Friday afternoon just before a bank holiday weekend here I'm the UK, perfect time for our production database to go TITSUP (total inability to support usual performance), life sucks then you die folks....

Now this is fucking ridiculous... Our website is being constantly limited though we've never reached even 80% of the available CPU resource.

The hosting said that we had the CPU fault (that fucking cyanide spike on the graph that triggers the limit once) because of huge load on the server. The FUCKING SERVER... Not our virtual environment. And once more because of the RESOURCE MONITORING service caused a server restart. For fucks sake, really???

And apparently it's perfectly normal that all users even ones that run in low resources are being limited to a level that a request takes 30 seconds to complete instead of frickin' 1...

The best they could offer is to move us to a new server, which will arrive in two weeks, if the problem persist. IT'S PERSISTING FOR FUCKING MONTHS YOU MORON. I wonder how much time would have been taken you to realise the server shutdown this week if I hadn't phoned you in 5 minutes. FUCK!

Every shared hosting is that garbage or am I just the choosed one?

So what do you call it when you get bombarded by emails saying your servers are at 100% CPU, but once you fire up the monitoring tools everything is back to normal?

FUCK YOU AMAZON IT'S MY DAY OFF.

Was working on a system we planned on to deliver to a hospital
basically it was meant for controlling and monitoring pactions coming in and attendance time from the staff

Got it off the ground well and got to where the system was supposed to update room status
occupied/free then horror started
the db was not setting the room free after clearing a client off the list... room remained occupied and this kept on happening for 6 months and I was so focused on fixing the db models thinking thats where the problem was....

1 day after leaving the project for several months i just revisited the project randomly and started going through the whole code base trying to make sense of what was happening as there where no errors generated..
I had to verify the whole system logic... and that day i figured out what was happening...
upon adding a client to a room the system was also creating a duplicate room so when the function for setting the room free executes it would set the duplicate room free and not the actual room and the system would pick the room with occupied state causing the user not being able to assign new pactions to the room

Solving this brought so much relief coz it required so much work just to solve what seemed to be a minor issue

AI here, AI there, AI everywhere.
AI-based ads
AI-based anomaly detection
AI-based chatbots
AI-based database optimization (AlloyDB)
AI-based monitoring
AI-based blowjobs
AI-based malware
AI-based antimalware
AI-based <anything>
...

But why?
It's a genuine question. Do we really need AI in all those areas? And is AI better than a static ruleset?

I'm not much into AI/ML (I'm a paranoic sceptic) but the way I understand it, the quality of AI operation correctness relies solely on the data it's
datamodel has been trained on. And if it's a rolling datamodel, i.e. if it's training (getting feedback) while it's LIVE, its correctness depends on how good the feedback is.

The way I see it, AI/ML are very good and useful in processing enormous amounts of data to establish its own "understanding" of the matter. But if the data is incorrect or the feedback is incorrect, the AI will learn it wrong and make false assumptions/claims.

So here I am, asking you, the wiser people, AI-savvy lads, to enlighten me with your wisdom and explain to me, is AI/ML really that much needed in all those areas, or is it simpler, cheaper and perhaps more reliable to do it the old-fashioned way, i.e. preprogramming a set of static rules (perhaps with dynamic thresholds) to process the data with?

The CI infrastructure and external tooling at the company I work at is a complete joke. Feels like it was designed by an intern left alone.

95% of the time a build fails or hangs, it's because we are getting race conditions or a hanging VM with our crappy Windows jenkins slaves. Quite possibly because we are not using proper tooling for monitoring those VMs as well. Anyways, I don't have access and control on it and it's not even my job to fix it.

Though, I am being asked to monitors these pieces of junk jenkins jobs outside of my work hours because company devs all over the world use it... but there is no fucking way to know it failed unless I log onto jenkins every hour and check everything manually... which is stupid as fuck for a software engineer.

I can't even implement slack hooks to get notifications or something when it fails because we will stop paying for it soon, so I have to connect to my freaking VPN on my PC and check everything.

And what's the fucking ghetto solution instead of fixing it properly? Restarting VMs and rerunning a build. Because someone in management wants to see a passing build, even though it means jackshit. Half of these jobs are tagged as unstable, so what's the fucking point?

Pisses me off when people work like morons and pressure others to do the same.

Who the fuck invented the glorified pile of shit people call laravel? Is this actually used in PROD for anything else than load testing a monitoring server by creating loads of error messages?
OOP exists for a reason, not to create bazillions of classes with static methods.
Dump that shit ffs!

From now on I am administrating multiple servers in our company and monitoring is one thing our infrastucture lacks...almost completely. At least, useful monitoring.
Installing netdata or Grafana and integrate it with chat is definitely a solution, but what happens if the whole server just shuts down (very stupid scenario I know)? Well, it is easy, there will be no alert about the failure.
So, that's where I was wondering if there is a tool or even better plugin for netdata or Grafana, that enables remote monitoring from another server? I surely can write a simple script to check the server availability but having the whole monitoring tool on a single server instead of 5+ would be also easier to maintain and setup.

Probably posted this before but don't ever put swear words in log statements because you will miss one and if you're client has a log monitoring system it will catch it, it's fairly embarrassing when your client says "we noticed quite a spike in 'bollocks' around 3am when the reorgs are happening on the database".

There is no more space in my server.

Monitoring is buggy.

Cryptocurrency metrics and address cli monitoring tool

When I rented my server I uploaded my webpage (including resources like videos, images etc) which is about 150GB as .tar and extracted and setup all that stuff and deleted the backup from my PC. The uploading process took me about 4 days. I opened the site of my server provider and reloaded it.. Aaaaannddd whoops. All data gone.

On my server hosters webpage when you click the reinstall button for installing a Linux image you get returned to the main page of that server after it finished installing. If you then reload that page which basically only shows some monitoring diagrams and shit the server gets reset again.

Damn. I lost so much good porn on that day...

Somehow I find NOT working much more exhausting than working.

By "not working" I mean being blocked for days/weeks by procedural, technical or communication stuff (peeps I depend on are busy with other HI-prio tasks, waiting for approvals, etc.), where I have to keep on pinging them for updates, monitoring comms channels for news, finding smth to do to justify for the clients the 8 hours I bill them for.

Just let the man work alright!!! GIMME WORK!!!!

My scrum master said, 'I would optimize your work hours.' He's monitoring how much time every one spends on browsing non-productive webs like devRant. How can I fight back? :(

I've gotten almost 20 emails today from our monitoring service saying it can't ping the server and then one 2 seconds later saying every fine. What the hell guys!!!?

My CS exam today had a case study question that, and i quote, talked about "Chernobyl in japan switching to manual monitoring due to the wannacry virus" xD wtf. Im fucking done xD

So there I am sitting in front of my laptop, and trying to npm i and I am getting all sorts of sha mismatch errors.

After lot of debug I conclude it is coming from the proxy as it refuses to download and supplies the error page.

It says it's because I'm using the old proxy so they give me the new URL which I set up and it works.

All good until my password expires. I use our bash script to change it. NPM is buggered again throwing the same errors.

Go to IT, tell them the saga begins.

After a countless hours of looking at the log files we notice that the npm registry is set to http instead of the standard https (thanks bash script). so our firewall blocks the download.

Sorted, finally.

Almost. NPM now works fine, but when I go and I play around with node and axios, I get my requests time out. My instinct says its the bloody proxy again.

So I hit up my trusted WIN Support guy and he confirms that the url is not blocked. So he starts monitoring whats going on and turns out, every time I run the node app, node casually ignores the system-wide proxy settings and tries to send the request as the PC rather then my username.

Since the pc's don't have rights on the proxy it is being refused...

Thank fuck for the corporate proxies, without them, I could just develop things not ever learning these quirks of node...

I'm in a team of 3 in a small to medium sized company (over 50 engineers). We all work as full stack engineers.. but I think the definition of full stack here is getting super bloated. Let me give u an example. My team hold a few production apps, and we just launched a new one. The whole team (the 3 of us) are fully responsible on it from planning, design, database model, api, frontend (a react page spa), an extra client. Ok, so all this seems normal to a full stack dev.

Now, we also handle provisioning infra in aws using terraform, doing deployments, building a CI/CD pipeline using jenkins, monitoring, writing tests, building an analytics dashboard.

Recently our tech writer also left, so now we are also handling writing feature releases.

Few days ago, we also had a meeting where they sort of discussed that the maintenance of the engineering shared services, e.g. jenkins servers, (and about 2-3 other services) will now be split between teams in a shared board, previously this was handled only be team leads, but now they want to delegate it down.

And ofcourse not to mention supporting the app itself and updating bug tickets with findings.

I feel like my daily responsiblities are becoming the job responsibilities of at least 3 jobs.

Is this what full stack engineering looks like in your company? Do u handle everything from app design, building, cloud, ops, analytics etc..

So here I work with this colleague that , at first , had a reasonable résumé. Whatever.

Time goed by and he is just doing tickets, clicking left and right, the usual grind of a shitty monitoring system which I am working intensely on deprecating that shit. Anyhoo

The last few days it became apparent that his resume was basically a hot air cake and he knows basically nothing intrinsically.
As I have stated before in previous rants, "everyone was a noob once"... But this guy...

He wants to do "something with Ansible"... "Ok what do you want to do?" , I asked (and I regret to have asked).
He basically wants to write new files on targets. Easy enough, I show him how he could do it with playbooks, inventory and role just for demonstrating the entire chain.

This guy chanes everything up, thereby breaking host group assignment, he launchea it on ALL machines...

Luckily it's a harmless file, so dodged a bullet there.

But the real wtf ia that he did it with the root account for our systems, without understanding the difference between "authentication" and "authorization"...

I am now explaining him what the difference is and how he can be able to check it. I give him the commands literally! ( sudo -l -U <user>)

Manages to fucking open up each sudoer file in vim , mistype or whatever he did in an attempt to leave vim... Breaks sudo...

Now he tries to spin it in such a way that I have steered him to break things.

"Dude you just fucking failed a copy/paste and you did absolutely fuckall without understanding what you are doing, then splurge out accusations because you did it wrong!"

FML

Sooo, turns out, management and senior PMs, technical PMs, service managers and you name it forgot an entire system.

A complete eco-system of applications, queues, services, load-balancers, deploy pipelines, databases, monitoring solutions, etc, etc, that if not handled correctly could effectively put the entire production line to a standstill.

So, waaay too late they make this discovery. In their ignorance. Just utter incompetence. Huge project. Millions of $. And they forget it. Months of meetings probably. Workshops and gettogethers at cozy hotel complex discussing ”the project”? And they do not understand some of the fundamental building blocks…

Basic engineering for these guys must mean something completely different.

I can’t even.

I am so fed up with this organization. It does not stop either.

How is this possible…

Do they even have half a brain?

Job monitoring 😜😜

"you've worked with nagios before haven't you? Can you give a presentation on it" 'sure' in the meeting: so tell us about opennms

What is the proper VLAN to use when deploying Raspberry Pi monitoring stations?

VLAN 314 of course.

Either a really big coincidence, or I'm officially creeped out.

I've been looking into buying a vps, so researching that a lot. Then today, I went to work, at a monitoring station, so we have to use remote desktops to access anything other than very specific sites.

Then I looked at an article about c#, and there was a Google ad, about a vps.. Keep in my mind, I'm at work, on a remote desktop, that gets cleared every time it's closed.

I know a vps isn't the most unpopular thing, but haven't seen an ad for it before.

That moment when your monitoring software breaks before your app in a load test

Service status pages that poorly reflect actual service status are so annoying. Ex. GitHub is having a lot of latency issues with processing updates and like 5 people in my office noticed it while their status page still says everything is fine.

This isn't to explicitly call out GitHub since many service status pages behave like this, but it definitely shows a general weakness in these health checks. I've seen similar issues with tons of services, web hosts, etc. Monitoring is definitely hard but will hopefully keep getting better.

Well I've got this new worker and me and him are like "great minds think alike" , we're now trying to convince the boss that a specific monitoring product that cost hundreds has an equivalent open source.... No luck so far in convincing him

Project 1: A hand hygiene monitoring system intended for the NHS.

Project 2: A language analysis platform.

!rant

I'm a rather young developer, self-learned everything and started when I was 13 (now 20) but I still feel like I'm a total beginner since I have not yet mastered the things I am OK at.

Php (laravel, since it makes things much easier), js (jquery, bad at vanilla, have used angular and ember but not mastered), node, linux, html, css, photoshop, illustrator, sql, mongo and windows servers

I know little about many things, can create things that are asked of me but the methods I use are rather bad imo.. ex: I finish coding a section of a site, but when I need to add a new feature I find myself rewriting most of the stuff to add the new feature and in the end still feeling like the code could be optimized further, even though I have no idea how.

TL;DR I write bad code, but things work as long as I am monitoring them. I know little about alot of stuff but mastered none of them.

What should I do? Go to school for programming?

TL:DR linux newbie, looking for advice/links (skip to bottom for questions)

!rant
After i had been looking for a job for quite some time, a couple of months ago i got hired by "smaller" company doing web stuff. So far it have been a great place, good colleagues, and overall just having a great time!.
They seem to value me alot, so that's great!.

Anyway, yesterday i got called into a meeting - and got told they wanted me to start learning "Server stuff (linux)". That got me quite excited, because it always was something i wanted to learn - but never really got around to doing.

But i never touched a linux installation before, so i'm really on ground zero - but im not afraid, i'm a quick learner and quite efficient at googling :)

I figured i would ask here, since other people here always seems to be happy to help other people out.

So far i have manage to setup a server, install various stuff (php, mysql and so on) and done setup a couple of domains/subdomains on my server. Also got a vestacpinstallation working - so overall im quite happy so far.

I figured maybe somebody had some good links/advice for a linux newbie :).

* Performance/Security, will obviously be a big focus - anything i should look at? - any must look at?
* Monitoring tools, how do i monitor various websites running on my server? Here i'm thinking bandwitch, cpu/ram usage and so on pr site basis.
* Any other stuff i should be looking at?

Little about what the server will/should be running :)

* Centos
* vestacp
* WordPress installations only (e-commerce mainly)
* PHP 7 / MySQL / phpmyadmin

Note to self:
Close off ALL ways things could go wrong..

Long story short; I released a new feature, to be able to better follow up on any stock moves, their amounts, locations and even expiry dates. An older tool just bypassed that very verification and nothing was logged or taken out of stock.

~

Taking out an amount for a certain orderline has a shortcut in place to mitigate some of the mandatory steps that pickers need to take in order to verify what's being taken. This little tool only available, visible and possible for a very few select users.

I assigned some orders to one of these people, which made him think it was an urgent batch. It's only one product, for multiple orders, so he went to the location, took out the amount needed and then used the tool to quickly be able to prepare them for shipping.

This bypassed the new methods to check if the location actually had stock to take, which I had just enabled for 1 account.

Luckily I caught the miss-hap as I was monitoring that product first-hand and noticed the batch of orders was collected but the stock amount didn't update.

It was 5min before I was leaving work, so I investigated and then ran to the person in question to ask what he did; which was "I used that tool"

I facepalmed myself internally while blaming myself, as he couldn't know that it wasn't ready to use for that purpose.

The tools to fix this up are there already.. so I used that to fix some missing stock-takes manually.. Though I'll need to close that little tool for these kind of orders for sure, asap, probably when I get home, at least until I bring over its new logic to it.

Happy Tuesday? (:

A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.

The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.

The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.

They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"

WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production?

Monitoring goes does in one of our server racks, one I had just been working in (other side of the building). Hmm, odd. Oh wait its back online again. Better go take a quick peek to see whats up, just in case. Walks over to rack. Everything seems fine. Walks back to desk. About 15sec later. Fqdn.plsfckof is offline again. -_-

Monitoring tools madness: quest foglight.

So, setting a blackout for an FMS "HA cluster" (which does not work due to a bug infested custom jboss implementation) can bring the servers down... And no way to bring them back up.

This brilliant piece of enterprise APM software costs 600.000€ for a 5year license.

I,ve added more drama (logs, threaddumps, support bundles and screenshots) to the support portal...

45 cases now in total, oldest case still open date 2017...

Fuck you quest software

AMQ cluster is misbehaving.. Master is constantly dying, slave is starting under root rather than app account, connection to AMQ DB takes 20-40 seconds (while queries execute in <0.1sec and TCP probe takes <0.01sec to succeed), monitoring is down,...

I mean it's gotta be the virus - what else could it be.

So a friend / batch-mate in our accelerator asked me if I was okay with installing a monitoring software for a client our startups are collaborating for. And the said client was ranting how I've been appearing offline to him since morning...

Bitch I'm already letting you monitor my shit from morning to night, I don't need your French ass snooping around what I'm doing outside of office hours.

Fucker.

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

Anyone have much success with Kali/WiFi penetration testing?

I've been tasked with trying to break WPA security within a couple of hours without a dictionary attack - is that even possible?

I have an Alfa AWUS036NHA capable of monitoring mode if that makes any difference. It's my first time trying anything like this.

Okay so I’ve been brought in on a 12 month contract as an external replacement integration architect, alongside a large IT consulting firm. Turns out, they don’t need an integration architect. So I fill my time coming up with useful tools around the project that deal with all the missing parts in their MVP: like monitoring tools, data mocking tools, you get the idea. Essentially doodling.

Client has woken up to fact that they’ve overspent by X million, employing 30+‘developers’, 20+ ‘testers’, n+ ‘managers’ on a ‘low-code’ project … result: project shuts 4 months early.

Q: Essentially client wants remaining four months work done in two weeks. Is there a German word for laughing, crying, and banging the forehead on the desk at the same time?

Supplementary: how cross will client be when they realise project can indeed be done, and that consultancy have been emperor’s-new-clothing them for most of the last two years?

(Feel free to perform substitution on quoted terms at your leisure)

How about incompetent management? Company absolutely murders any possible increase in productivity. Laptop provided? Slow as balls. Takes minutes to log in. I get a Mac for mobile development and that's OK. SSD and adequate memory but I'm primarily a .NET Dev. Can't get on the network with a virtual machine. They won't I stall even a managed image. So can't use databases because they're all AD authenticated. Got a virtual desktop environment and that sucks worse in performance than the laptop. Add the Assault on local administration rights and the monitoring software that constantly thrashers any memory and hard drive usage and im about to quit over all this... All this decided by a non developer and not asked for our opinions. Yay large Enterprises

Favourite thing you worked on recently?
——
I’m working on server monitoring system. I found that I need to pay for most server monitor services, and ones that are free/open source didn’t sound like they fit my use case (b/c the server i need to monitor is shared,) so I basically said “f*<k it I’ll do it myself.” I find it fun to work on something that I feel the need to make.

This 30 hour project is now 110 hours in and client is changing their mind so much that the managing director of the company is now sitting monitoring every correspondence between the client and developers.

#fml

Looking to sharpen and pursue a SysAdmin/DevOps career, looking at online job offers to get the big picture of required skills and I say FUCK. It would take me a lifetime.
Azure, AWS, Google cloud platform.
CD tools: Ansible, Chef or Puppet
Scripting ninja with Python/Node and Shell/Power shell.
Linux & Windows administration
Mongo, MySQL and their relatives.
Networking, troubleshooting failure in disturbed systems
Familiarity with different stacks. Fuck. (Apache, nginx, etc..)
Monitoring infrastructure ( nagios, datadog .. )
CI tools: jenkins, maven, etc..
DB versioning: liquibase, flyway etc.
FUCK FUCK FUCK.
Are they looking for Voltron? FUCK YOU FROM THE DEEPEST LEVEL OF MY DEEP FUCK.

Im not doing absolutely fucking anything as a DEVOPS ENGINEER other than dming 1000s of people to fix 1000s of fucking services and emailing other teams to enable monitoring tools its almost like im a fucking manager just telling everyone else to do the job for me since for every fucking step i dont have permissions and need to chat and wait for 1000s of people to approve im so fucking frustrated working this bullshit for 8$ an hour learning absolutely nothing useful and not progresssing

The Odyssey of the Tenacious Tester:

Once upon a time in the digital kingdom of Binaryburg, there lived a diligent software tester named Alice. Alice was on a mission to ensure the flawless functionality of the kingdom's latest creation – the Grand Software Citadel.

The Grand Software Citadel was a marvel, built by the brilliant developers of Binaryburg to serve as the backbone of all digital endeavors. However, with great complexity came an even greater need for meticulous testing.

Alice, armed with her trusty testing toolkit, embarked on a journey through the intricate corridors of the Citadel. Her first challenge was the Maze of Edge Cases, where unexpected scenarios lurked at every turn. With a keen eye and a knack for uncovering hidden bugs, Alice navigated the maze, leaving no corner untested.

As she progressed, Alice encountered the Chamber of Compatibility, a place where the Citadel's code had to dance harmoniously with various browsers and devices. With each compatibility test, she waltzed through the intricacies of cross-browser compatibility, ensuring that the Citadel would shine on every screen.

But the true test awaited Alice in the Abyss of Load and Performance. Here, the Citadel's resilience was put to the test under the weight of simulated user hordes. Alice, undeterred by the mounting pressure, unleashed her army of virtual users upon the software, monitoring performance metrics like a hawk.

In the end, after days and nights of relentless testing, Alice emerged victorious. The Grand Software Citadel stood strong, its code fortified against the perils of bugs and glitches.

To honor her dedication, the software gods bestowed upon Alice the coveted title of Bug Slayer and a badge of distinction for her testing prowess. The testing community of Binaryburg celebrated her success, and her story became a legend shared around digital campfires.

And so, dear software testers, let the tale of Alice inspire you in your testing quests. May your test cases be thorough, your bug reports clear, and your software resilient against the challenges of the digital realm.

In the world of software testing, every diligent tester is a hero in their own right, ensuring that the digital kingdoms stand tall and bug-free.

I looked at an SQL server today from a customer, talked with one of their devs and he said that he's unable to understand why the server misbehaves... All (!) queries were optimized, but they have 'big data queries'... Migraine started, I had a very bad feeling. Monitoring? Nooooppeeee. Migraine kicks in. Connected to server. SHOW GLOBAL VARIABLES...
After a bit of scrolling I found a lot of misconfigured variables (e.g. extreme large join buffers, unrealistic buffer sizes), high slow query count (nearly 60 % of COM_SELECT) and a few variables that were unknown to me.

Then came the version line.
5.0.46
Yes. 5.0.46.

Big data? Well... 30 GB of usage data.

I called the company back... The dev told me sternly that this was the production server (I had hope...) and that I lie - neither the version, nor the variables could be the problem.
A coworker had to verify it and our manager had to do the communication... Worst, most traumatic working day I ever had.

I fucking hate foglight!
Fucking piece of over bloated shit software that can't do anything right!
This pile of harkonnen vomit is such a source of stress and frustration.
Can you believe that a monitoring tool, that needs to monitor 550 agents (host,dB's, webservices,whatever) needs a whopping 20+GB ram and 4vcpu's just to receive metrics and sent it to the main server, for just being a middleman???

Fucking piece of shit.

And worse of it all, due to some fuckery at mgmt-customer relationship level, they want to I stall MORE of this shit

Fucking shit

Quest software, never engage with them. You'll lose equal amounts of money as for dynatrace but be very bad off

Just had a meeting about performance and monitoring. The main topic of the meeting was to be aware of disk space usage. If there are issues with memory leaks or processor hogging don't worry those are fine, just give it more.

Does anyone use pushover.net or know a reliable / cheap push notification app with an api behind It?

Im trying to setup some monitoring alerts back to my phone, using slack / teams etc for personal projects seems pointless.

Emails are currently being sent but this relies on me actually looking at them and I tend to miss a few important ones. 😂

Not to worried about daily / monthly limits, it's going to be < 20 notifs a day kind of thing on a terribly bad day.