Big event. Massive traffic in production, so we were monitoring all night.

I was in a room with 2 devs of my team, a marketting girl, my boss and a designer... chilling.

Suddenly the production is down.

Boss: production is down, anyone can check?
Me: already on it
Dev1: it looks ok for me
Dev2: me too
Me: wait what? Impossible everything is down
Dev1: oh I refreshed the page it's not working
Me: don't stay on the page refreshing it like you are fucking monkeys. Give me useful intel or be quiet.
Market girl: is it working?
...
Guys is it working?
...
Hello?
Me: Not yet we are looking. Don't distract me.
Boss: client called us. They want it online now.
Dev1&2: he's looking
... 1 min later...
Boss: is it working?
Boss: is it working?
Boss: is it working?
Me: SHUT THE FUCK FOR FUCKING ONE SECOND. ALL OF YOU, OUT NOW. YOU ARE FUCKING MONKEYS WHO CAN'T DO SHIT. IF YOU CAN'T HELP JUST SHUT YOUR DAMN SHITHOLE. DEVS, LOOK WITH ME. MARKET GIRL PREPARE A FUCKING POST-MORTEM MAIL. BOSS GET THE CLIENT ON THE PHONE AND STALE. DO. YOUR. FUCKING. JOBS.

That's how I ended up screaming at everyone... the rest of the night went in complete silence and I fixed the issue 2min after the got quiet or busy.

We have a lot of monitoring screens on walls at my work... Some of them show charts... The one showing the last two months of mailbox database growth has started taking an odd shape.

ON MONDAY

TL: Why the hell you require a month to integrate this engine?

Me: It will take that much time, can't help it.

TL: it can be done within a week.

Me: Then you do it.

TL: Ok I will show how it's done in a week.

ON FRIDAY

Me: What's the status on that integration?

TL: Oh yeah about that, you have to carry it ahead, I have some monitoring to do.

Me: Ok, give me the repository access, I will carry it ahead.

ME OPENS REPOSITORY.

There's only a new controller file with nothing it.

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

Mother of god, as if the new mass surveillance law in the Netherlands wasn't bad enough, one of the politicians who likes the new law has come up with an even more redicilous idea.

an 'Internet Authority. To put it short, an authority which surveils the internet in real time and sees where all social media shit is coming from/going.

Meaning that it wouldn't just be 'targeted mass surveillance' sometimes but fulltime online monitoring.

This guy has lost his fucking mind.

Long rant ahead. Should take about 2-3 minutes to read. So feel free to refill your cup of coffee and take a seat :)

It turns out that the battery in my new Nexus 6P is almost dead. Well not that I didn't expect that, the seller even explicitly put that in the product page. But it got me thinking.. why? Lithium batteries are often good for some 10k charges, meaning that they could last almost 30 years when charged every day! They'd outlive an entire generation of people!

Then I took a look at the USB-C wall charger that Huawei delivered with this thing. A 5V 3A brick. When I saw that, I immediately realized.. aah, that's why this battery crapped out after a mere 2 years.

See, while batteries are often advertised as capable of several amps (like 7A with my LiitoKala 18650 batteries that I often use in projects), that's only the current that they can safely take or deliver without blowing up. The manufacturer doesn't make this current rating with longevity in mind. It's the absolute maximum in current that a given battery can safely handle.

The longevity on the other hand directly depends on the demand that's placed on the battery. 500mA which is standard USB 2.0 rating or 1A which is standard USB 3.0 rating, no sweat. The battery will live for at least a decade of daily charges and discharges like that no problem.

But when you start shoving 3A continuous into a battery, that's when it will suffer. Imagine that your current workload is 500mA and suddenly you get shoved 6 times that work upon you. How long would you last?

Oh and not only the current is a problem, I suspect that it also overvolts the battery to maintain a constant current all the way till the end. When I charged my lithium cells with my lab bench power supply, the battery would only take a few milliamps when it got close to the supply voltage. Quick bit of knowledge: lithium cells are charged at constant current first, then when the current drops below that, it continues at constant voltage - usually 4.2 or 4.35V depending on the battery. So you'd set your lab bench power supply at 4.2V 500mA. But in that constant voltage mode, as the battery's voltage and the supply's voltage equalize, the current drops because the voltage difference becomes lower. Remember, voltage is what causes current to flow. Overvolting at the supply to stay in constant current mode all the way till the end speeds this process up but can be dangerous and requires constant monitoring of the battery voltage.

So, why does Huawei and a bunch of other manufacturers make these 3A power chargers? Well first it's because consumer demands ever more, regardless of the fact that they can just charge at 500mA for the night (8h of sleep) and charge a 4000mAh battery from 0 to 100% no problem. Secondly it's because sometimes you need that little bit of extra juice fast, like when you forgot to plug the damn thing in and you've got only 30 minutes in the morning to pour some charge into it.

But people use those damn fucking things even when they go to bed, making that 3A torture a fucking standard process!! And then they complain that their batteries go to shit?!

Hopefully this now made you realize that the fast charger shouldn't be used as a regular charger ^^

So my landlord just came up and asked why I'm using so much bandwidth (they've just had a new line installed so they're monitoring it like hell for some reason) so we had a chat, I told him I'm a Web Developer so I'm uploading and downloading a load, and bare in mind this is student housing, he offered to install a wired connection in my flat only so I'll have a decent and stable connection when all the other students come back in September.

This is the first time in my life I feel like I'm not paying enough rent!

I accidently left log.debug("bollocks") ;
In an exception handler our customers log monitoring system picked it up and they questioned why and I quote here "why is there a spike in bollocks at 3am?"
That was an awkward conference call

FBI: FUCK!! We just lost our connection to all those civilian routers we were monitoring
FBI IT Department: Did you try turning them off and back on?
FBI:

When the Project Owner wants monitoring and asking the IT without clear specification 😅

I wrote (or, ended up with a very much alpha but usable version) a monitoring system a while back and completely forgot about its existence.

But, it's still running and a few days ago I was building a docker imagine on a system with not that many resources and after about 5 minutes I started getting notifications about a high load!

Then, while I had forgotten about it again, yesterday, I suddenly started getting notifications about websites on my main application server going down.
Logged in and all was good again after restarting nginx.

Gotta say that it feels quite awesome to be notified of shit going wrong by something I wrote myself while I forgot about its existence 😊

Navy story continued.
And continuing from the arp poisoning and boredom, I started scanning the network...
So I found plenty of WinXP computers, even some Win2k servers (I shit you not, the year was 201X) I decided to play around with merasploit a bit. I mean, this had to be a secure net, right?
Like hell it was.
Among the select douchebags I arp poisoned was a senior officer that had a VERY high idea for himself, and also believed he was tech-savvy. Now that, is a combination that is the red cloth for assholes like me. But I had to be more careful, as news of the network outage leaked, and rumours of "that guy" went amok, but because the whole sysadmin thing was on the shoulders of one guy, none could track it to me in explicit way. Not that i cared, actually, when I am pissed I act with all the subtleness of an atom bomb on steroids.
So, after some scanning and arp poisoning (changing the source MAC address this time) I said...
"Let's try this common exploit, it supposedly shouldn't work, there have been notifications about it, I've read them." Oh boy, was I in for a treat. 12 meterpreter sessions. FUCKING 12. The academy's online printer had no authentication, so I took the liberty of printing a few pages of ASCII jolly rogers (cute stuff, I know, but I was still in ITSec puberty) and decided to fuck around with the other PCs. One thing I found out is that some professors' PCs had the extreme password of 1234. Serious security, that was. Had I known earlier, I could have skipped a TON of pointless memorising...
Anyway, I was running amok the entire network, the sysad never had a chance on that, and he seemed preoccupied with EVERYTHING ELSE besides monitoring the net, like fixing (replacing) the keyboard for the commander's secretary, so...
BTW, most PCs had antivirus, but SO out of date that I didn't even need to encode the payload or do any other trick. An LDAP server was open, and the hashed admin password was the name of his wife. Go figure.
I looked at a WinXP laptop with a weird name, and fired my trusty ms08_067 on it. Passowrd: "aaw". I seriously thought that Ophcrack was broken, but I confirmed it. WTF? I started looking into the files... nothing too suspicious... wait a min, this guy is supposed to work, why his browser is showing porn?
Looking at the ""Deleted"" files (hah!) I fount a TON of documents with "SECRET" in them. Curious...
Decided to download everything, like the asshole I am, and restart his PC, AND to leave him with another desktop wallpaper and a text message. Thinking that he took the hint, I told the sysadmin about the vulnerable PCs and went to class...
In the middle of the class (I think it was anti-air warfare or anti-submarine warfare) the sysad burst through the door shouting "Stop it, that's the second-in-command's PC!".
Stunned silence. Even the professor (who was an officer). God, that was awkward. So, to make things MORE awkward (like the asshole I am) I burned every document to a DVD and the next day I took the sysad and went to the second-in-command of the academy.
Surprisingly he took the whole thing in quite the easygoing fashion. I half-expected court martial or at least a good yelling, but no. Anyway, after our conversation I cornered the sysad and barraged him with some tons of security holes, needed upgrades and settings etc. I still don't know if he managed to patch everything (I left him a detailed report) because, as I've written before, budget constraints in the military are the stuff of nightmares. Still, after that, oddly, most people wouldn't even talk to me.
God, that was a nice period of my life, not having to pretend to be interested about sports and TV shows. It would be almost like a story from highschool (if our highschool had such things as a network back then - yes, I am old).
Your stories?

My uni implemented Bluetooth beacon based attendance monitoring.

Raspberry pi + cloned beacons = 100% attendance.

Idiots 😂

(Edit for clarity: app on smartphone, Bluetooth beacon in each room)

"Another one bites the dust.." BEEP BEEP BEEP!
😬 "Shit, make that two."

Down from 5TB redundant to 3TB non-redundant and still waiting for my new Reds. No choice but to shut everything down and wait. 😩

Working on a (PHP based) monitoring system and currently writing rules/testing stuff.

I wrote some rules which check a few pages of a dutch site displaying if certain bigger services have disruptions and I am pulling the pages of a few ones I'd like to be notified about when they have issues.

Started the engine and received an alert about one big ISP over here from the monitoring system. Didn't believe that it would work right away so went to check that specific disruption page and...... they have a disruption right now!

IT FUCKING WORKS.

Good monitoring system 😊 *pats system*

To improve our user's "experience" I suggested to my boss to add a status page showing...well, the current status of our services. Everybody was up for it, so I go off and implement a basic version + automated monitoring backend, get lots of positive feedback, all seems fine.

Then it starts:

Boss: "Can you get it all set up by this Saturday?"
Me: "Uh, today is Wednesday and I've never set up all the stuff needed on a proper server before"
Boss: "Well, you still have a few days. Please also contact your coworker to get it all hooked up in our launcher"
Me: "I'll try, can't make any promises though"

Contact my coworker and tell him what the plan is. I had already given him access to the repo and he is positive to get it all hooked up (I doubt he ever cloned my repo, let alone ran my code)

Spend all Friday getting my stuff set up on the production server, feeling pretty good thanks to the many tutorials.

Contact the boss Friday evening:
Me: "All up and running"
Boss: "Thanks, but we decided to go with a basic HTML page instead. We can just manually edit that, should be enough.
Me: "..."

In the end my stuff was never used, the server I set up was finally taken down a month ago. The gratitude you get when not hacking together some absolute shit that causes problems when you don't add <br/> tags at the correct places to prevent an ugly overflow, cause the coworker was too lazy to implement some form of line wrap in the launcher. I'm not saying my stuff is the best of the best, but at least it was professional looking to a certain extent.

I'd love to finish a few projects I'm currently working on:

- An add-on which gives a middle finger to websites which use services/products ran by companies which are known to be integrated within the biggest mass surveillance system ever created (US powered). Not because just fuck those websites but because I think (@PonySlaystation came up with this idea) that its only fair that people get to know which websites 'sell them out'. Oh and "but not everyone cares about that" - you don't HAVE to install the addon.
(will be open sourced)

- Notes service with a fun thing.

- PHP based server/website/whateverthefuckyouwant monitoring system which is pretty much module based and works with json files as configuration. (kinda works but still loads of bugs to solve and gotta improve the module system a lot).
(will be open sourced)

- PHP based pihole alternative which suits my needs (will be open sourced)

- Forgot one 😅

!rant

Quite the opposite of a Rant actually. Very good day today! Had pancakes to start the day, finally got a second monitor for only 5 bucks - it is so much easier to program with live change monitoring!

And, I finally got my very own server rack :) just wanted to share!

Update 3:
Nice little command line monitoring thing set up :)

Tip for devs (esp front end):

Sublime text (and few other inferior editors) has a plugin called "transparency" that allows your editor to be transparent.

Windows 10 powershell consoles can be made transparent.

1) Open browser
2) Open command prompt over it with 75% transparency
3) Open editor over both with 50% transparency.
4) Set editor to fullscreen, no-distraction mode to center the text (Shift+F11 in sublime)

Enjoy coding while constantly viewing the code, the browser and monitoring the cmd prompt at the same time, without having to click Alt+Tab a zillion times.

> me on call
> had to much wine
> suddenly, phone starts making sound
> holy shit, I am on call and I am not supposed to drink
> Calls taxi because shit is looking bad at work. Everything look down according to the sms I get, I had to pay the taxi from my own pocket.
> 40 min later, arrive at work. Walk into the serverroom.
> Go to the the server that is monitoring everything.
> Check networkcable, it is loose. Push it a little.
> Goddamnit, that was it.
> Realize I never doublechecked if stuff really was down from home....

I think in my case its a monitoring system I'm still writing which does already work (just far from done).

I got this as a crazy idea while thinking that it would be impossible to make and then thought fuck it and I wrote it.

Written in php, and as long as you can write a module for it, you can pretty much make it monitor anything you'd like.

I'm using it to monitor a few websites and servers I run :)

Colleague started a slack channel for our team, management wanted nothing to do with it. We used it to work and have a bit of fun.

Some push / drive came form somewhere and now all the managers are on it. Yesterday I was told my screenshot and "snarky comment" are not appropriate for the workplace and to delete my slack message.

My comment was a joke about about a new app the company has to use "to increase efficiency" that broke and wouldn't let me do what I needed. It wasn't offensive, demeaning, sexist to anyone or even contain any bad language.

How petty and childish to be monitoring a private channel making sure everything is positive. We all joked that from now on our meme's must be about how awesome the company is and how much time we are saving on a daily basis.

God forbid we're allowed to speak honestly and openly or have a bit of fun.

1. Fucking MySQL database clusters.

There's nothing fun about MySQL clusters. Sometimes they start producing deadlock errors for no apparent reason... well, there's probably a reason, but it's never a transparent easy to find reason.

What was even less fun is that those errors took down a Sentry server. When your error log server goes down through ddos from your database messages, it's time to rethink your setup.

2. Wiring up a large factory with $2 arduino clones, each with a $2 esp8266 wifi chip, with various sensors for measuring flow of chemical solutions (I wanted cheap real time monitoring as an early warning system next to periodic sampling).

The scaling issue was getting over 500 streaming wifi signals to work in a 55c moist slightly corrosive atmosphere with concrete and steel everywhere, and getting it all into a single InfluxDB instance for analysis.

So I'm moving to a new/bigger place with faster Internet soon so I think it's time to rebuild my current home/remote server setup.

I want to setup the following things:
- vps for server monitoring (open source pushover alternative + netdata)
- Zero tier network for connecting all my servers to the same network
- pihole/pivpn (or the Angristan vpn installer, look it up :)
- second blocking thingy next to pihole to make sure that I literally can't access google/fb etc anymore, even if I really needed/wanted to
- bunch of general servers.

Any ideas?

Paranoid Developers - It's a long one
Backstory: I was a freelance web developer when I managed to land a place on a cyber security program with who I consider to be the world leaders in the field (details deliberately withheld; who's paranoid now?). Other than the basic security practices of web dev, my experience with Cyber was limited to the OU introduction course, so I was wholly unprepared for the level of, occasionally hysterical, paranoia that my fellow cohort seemed to perpetually live in. The following is a collection of stories from several of these people, because if I only wrote about one they would accuse me of providing too much data allowing an attacker to aggregate and steal their identity. They do use devrant so if you're reading this, know that I love you and that something is wrong with you.

That time when...

He wrote a social media network with end-to-end encryption before it was cool.

He wrote custom 64kb encryption for his academic HDD.

He removed the 3 HDD from his desktop and stored them in a safe, whenever he left the house.

He set up a pfsense virtualbox with a firewall policy to block the port the student monitoring software used (effectively rendering it useless and definitely in breach of the IT policy).

He used only hashes of passwords as passwords (which isn't actually good).

He kept a drill on the desk ready to destroy his HDD at a moments notice.

He started developing a device to drill through his HDD when he pushed a button. May or may not have finished it.

He set up a new email account for each individual online service.

He hosted a website from his own home server so he didn't have to host the files elsewhere (which is just awful for home network security).

He unplugged the home router and began scanning his devices and manually searching through the process list when his music stopped playing on the laptop several times (turns out he had a wobbly spacebar and the shaking washing machine provided enough jittering for a button press).

He brought his own privacy screen to work (remember, this is a security place, with like background checks and all sorts).

He gave his C programming coursework (a simple messaging program) 2048 bit encryption, which was not required.

He wrote a custom encryption for his other C programming coursework as well as writing out the enigma encryption because there was no library, again not required.

He bought a burner phone to visit the capital city.

He bought a burner phone whenever he left his hometown come to think of it.

He bought a smartphone online, wiped it and installed new firmware (it was Chinese; I'm not saying anything about the Chinese, you're the one thinking it).

He bought a smartphone and installed Kali Linux NetHunter so he could test WiFi networks he connected to before using them on his personal device.

(You might be noticing it's all he's. Maybe it is, maybe it isn't).

He ate a sim card.

He brought a balaclava to pentesting training (it was pretty meme).

He printed out his source code as a manual read-only method.

He made a rule on his academic email to block incoming mail from the academic body (to be fair this is a good spam policy).

He withdraws money from a different cashpoint everytime to avoid patterns in his behaviour (the irony).

He reported someone for hacking the centre's network when they built their own website for practice using XAMMP.

I'm going to stop there. I could tell you so many more stories about these guys, some about them being paranoid and some about the stupid antics Cyber Security and Information Assurance students get up to. Well done for making it this far. Hope you enjoyed it.

Our "monitoring" today.

HR: What was the last project you handled?
Me: I worked on an internal system for my current company. It is basically an interdepartmental monitoring system.
HR: Ohh. Good. Do you have a copy of it? Can you show me how it works?
Me: ......

Advice to all new programmers, take this one from personal experience. DO NOT PUT SWEAR WORDS IN DEBUG STATEMENTS.
You will miss one, it will go to production and it will get picked up by your log monitoring...

This startup I started working for with their shitty code base written by interns, restrictive sys admin who had no actual use in the company since I was the one setting up their servers, know-it-all CEO, stupid HR representative who used to grill employees for being 10 minutes late in the morning, very small apartment "HQ", using fingerprints to signal our entry and our leave to and from the office, no formal process, and, to top it all, monitoring our own laptops which we use for work with a software that takes screenshots every few minutes. In short, it had the worst in corporates with the worst of startups combined in one company.

If, hypothetically, we could overlook all this, I couldn't overlook the horrible smell this place had. The apartment was overlooking a small garden which was a home for many stray cats and dogs. You can imagine how horrible this smell was. The weird thing was that no one there seemed to really care about the smell!!

I lasted there for only one week before I gave my resignation and I believe I had every right to do so.

Visual Studio Code.

I've tried you because of hearing a lot of good stuff about you. I'd switch back to netbeans regardless because I love netbeans and I always try to use as little as possible from companies like Google/microsoft/facebook (and others) but what you're pulling right now is un-fucking-believable.

I've disabled ANY AND EVERY form of calling home I could (find) in your settings. Crash reports, automatic updates, metrics, you name it. I've searched all the fucking settings but I can't find any other home calling thing that's enabled and yet:

I'm monitoring every goddamn DNS request (through my own DNS server) and I'm still seeing calls to a Microsoft owned domain. Closed all my browser sessions and you as well and it stopped. Started browser again but not you, nothing.
Started you again: BAM. Calls to that damned Microsoft owned domain again.

If you can't honour my decision for disabling any form of home calls, go fuck yourself.

Netbeans, I'm back, I've missed you 💜

Trying to setup server monitoring for one server (will deploy it on all my servers once this works well).

Getting these email notifs to work is a bitch 😅

(can't do pushover etc since those require GCM and I've got Google blocked out of my phone)

DevOps required skillset:

* Frontend engineering
* Backend services
* Database administrator
* Security consultant
* Project management
* 3rd party contract negotiator
* Build system monitor
* Build system hostage negotiator
* Paging, alerting, monitoring
* Search server admin
* Old search server admin
* Old-old-new search server admin
* Redis, ElasticSearch, MySQL, PostGres, owner
* Agile coach
* No you shouldn't do that coach
* Oh, you did that anyway coach
* DNS: (Optional) It'll replicate when it wants, and how it wants to to anyway
* Multi-Cloud deployment strategist
* Must be able to translate Klingon to YAML, and YAML to MySQL
* Cost analyzer, reducer, and justifier
* Complex documentation generation in markdown that we should have done years ago anyway
* Marketing's email went to spam analyzer
* Wordpress is broke fixer
* Where the fuck does Wordpress run anyway?
* Ability to fix MySql running Wordpress on marketing's dusty laptop

One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.

On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).

Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.

I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".

Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.

There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.

Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).

I just cleared out 48,158 monitoring emails from work. These are all automated emails received between mid May and end of November. Outlook is now pouting in a corner and not responding, but I'm not done cleaning up yet...

Have been trying to setup Netdata as a monitoring system for a while now and finally got it working!

Instead of the built-in webhooks I just did a curl to a url containing a php page/file which error logs the status and description (just for testing).

It took me way too long to get it to work but BAM.
Immediately made a new cpu load rule (one minute high load):
The satisfaction of getting an error message in the php logs containing my custom rule as warning and a minute later as critical 😍

Netdata ❤

Let me tell you a story.

Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.

With this many false positives, we quickly learn to ignore the alerts and miss real incidents.

So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!

It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.

To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.

As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!

Here's the story of my first month at CERN :) But first, a little premise...
Before arriving, I expected to be scared, alone and unguided in most of my experiences: after all I was a simple 19 year old about to leave home and friends for 3 years heading out in the world with zero experience on stuff like banking, taxes.. let alone working in a huge environment! The impostor syndrome was at an all time high on that front.
Then, I had the luck and pleasure to find an extremely competent and helpful plethora of people, ranging from my team to other CERNies (yes, that how we're called :P) who took me under their wing and introduced me to all the key aspects of living the place. When the initial stress finally soothed down thanks to this, I finally started to manage focusing more and more on my work, by following day-by-day my teammates who taught me the core aspects of the system and the many projects that are in progress during Long Shutdown 2. Within a couple weeks, I already managed to grasp various concepts that got me quickly on track, and now I managed to develop and integrate new temperature monitoring scripts into a system checking on hundreds of Single Board Computer-based servers :) It's a real rollercoaster of learning and applying under all fronts and so far I'm not regretting my choice of departing.
Luckily I've also discovered I'm pretty efficient and good at my job, which surely boosts my morale :D
Keep you updated as usual!

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

PSA: if, for whatever shit reason your brain comes up with, you decide to run a webminer in your retarded useless piece of shit website, at least HAVE THE DECENCY TO WARN USERS ABOUT IT. And while you're at it, implement some basic monitoring and safety functions. If you don't, you can set yourself on fire and jump from the top of the tallest building you can find.

Some basic tips:
1) don't run that shit on phones. The fraction of a fraction of a cent you're gonna earn from them is not worth the risk of overheating them and draining their batteries.

2) add low battery/overheating protection: the last thing you want to do is kill some poor sucker's laptop (and potential unsaved work) just because they forgot a tab open. Every time a laptop dies because of you, a knife will slit your throat.

3) WARN YOUR USERS ABOUT IT! You are straining someone else's resources for your own profit: at least have the balls to be open about it. If you try to run a miner silently in the background, I will make you eat whatever is left of your fucking brain, then drown you in the shit that comes out of your ass.

Other PM: We must fix the database performance issues now.

Me: We can't. We're still only halfway on the dependency chain to tackle this and honestly, even if the dependency chain would be fulfilled, I'd leave at least 2 weeks monitoring the production after the changes were rolled out before we further poke around.

Other PM: This is taking far too long. And whaddya mean by dependency chain? Why was I not informed about this?

Me: *sigh* like in every meeting in the last weeks: the dependency chain are the current open blockers before we can proceed with the database changes. We've talked about this _at length_... Especially why these blockers exist.

Other PM: No, we need to start now. I've _examined_ at the blockers or "dependency chain" as you call it.

(Examined.... He opened on his currently streaming laptop, which was connected to the active beamer, the mentioned ticket with a detailed blocker ... And quickly scrolled. Yeeah. Warmonger...).

Me: I'm very tired of discussing this. But since you are already presenting us the ticket, read out the referenced meeting notes... We explained it in great detail.

Other PM: Why? This is just a waste of time!!!!!!!

--
Yes. This happened. Other PM was my nemesis.
In this meeting were 2 PMs (Him, Me)… I think 5 - 7 devs... And we were sitting in this meeting since 2 hours at least. Everyone was angry...

After this "manifesto of intelligence"… I simply left the room, followed by a few devs.

And yes. Other PM did this on a regular basis....

> Monitoring: Load Average of 57!! ALERT!!!!
> me: What? That's not possible?
> *Monitoring froze 14 hours ago*
> *sshs into server*
> *see attached image*

The issue was ~1200 df processes that were issued by our monitoring system and all of them didn't finish because the external cluster we mounted onto that server died a few minutes before that. Just re-mounting the cluster fixed it but still a funny sight!

An important message:
PrOpErLy managing servers is HARD.

I get pissed off at customers with ZERO server knowledge who think they can manage their VPS. “Just get a control panel and a VPS” from some flashy provider that makes server management look way too easy.. Clicking around in their fancy control panel, until:

- they need help with their *self-managed* VPS;
- their email ends up in spam;
- they suffer from performance issues;
- they need to restore a backup;
- something breaks, because YES, things break

Way too little people are able to answer:

- when and how do you make backups?
- how do you monitor your servers and which services?
- how do you keep track of trend analysis?

Then I come by with necessary software. SNMP for trend analysis, Graphite for infrastructure health, Sensu for monitoring, Kibana, Ansible for configuration management..

Things that servers need but that customers have never even heard of.. because they can do everything in their control panel..

Until they come crying to me because it broke and they don’t even know how to get into SSH.

I think the ones to blame are VPS providers that tell the tale of how easy it is to install a control panel and never look at your server again.

Customers become responsible for something *business-critical*! Yet they don’t know how it works.

When you automate so much stuffs that you forget where the config files are at.

Rain monitoring/alert api's/services/websites/apps.

Why the fuck are those fuckers (in my case) nearly always right when I couldn't care less and very fucking wrong when I couldn't care more?

Today, morning before leaving:
Nah mate you're good for the next x minutes (more than enough time for me to bike to the station and get on the train).
Reality: suddenly getting two gallons of water a second on me in the form of rain, arrived completely soaked and headphones broke.

Before biking home from the station;
*its raining hard as hell*
Rain alert/monitoring app/radar thingy: nah mate no worries, no rain last half hour and you're good for another half hour!
Reality: arrived home soaked a-fucking-gain.

So motherfucking annoying and frustrating. And yes, I do have an umbrella and such but I hate carrying that stuff around when 'not needed'.

Got an email earlier this week. It went something like this:

"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"

Um, wait, excuse me, WHAT?

1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.

I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.

I suddenly realized all the technical debt shit I told my boss would happen years ago given the way things were done/heading then... Just occurred pretty much all at once last week in the form of critical production issues...

The teams like:
-we need real time server process monitoring
-structured logging for apps
-containerization so one app didn't affect others

Me thinking: yes.... I told you so like 3/4 years ago when I first joined the team and kept repeating so much I got tired of saying at every annual review...

This is exactly what happens when you let technical debt grow and have no free time for developers to look into and fix then while they were small and not critical production processes... Or properly document and peer review them... (Got a shit pile of projects that no one knows how to use or even exists because the devs left the team) and they'll have a lot more when I finally leave... Hopefully this year.... If I can find another role and not need another medical procedure... (Doubtful)

Managed a 97% reduction in bandwidth usage for our internal host monitoring tool by converting the dashboard from using AJAX polling to websocket events.

Completely unnecessary but wanted an excuse to do some development with websockets. (:

Just in case nobody has mentioned this yet:

Yes Microsoft I do have a dualhead setup.
Yes Microsoft I do want to watch video on my left screen while having window focus on something on the right monitor.
No Microsoft this doesn't mean that I *lost* focus on the left window.
No Microsoft this doesn't mean that I want your Movies and TV application to suddenly minimize (and continue playing anyway) while I focus on some server monitoring window on my right display.

Microsoft, there exist people that use more than your average user with a single C: drive that play Candy Crush on Facebook all day. And the limitations that you currently impose might very well be what keeps the Microsoft UWP applications from getting adopted. Because you know what? SMPlayer, a default application in any of my Linux workstation machines, it does handle such window transitions just fine!

Microsoft, I love how you at least gave us the option to enable Ctrl-Shift-C and Ctrl-Shift-V in WSL and conhost in general over that abomination that is Right-click and Return (those are so random!) that are relics from CP/M. But seriously? At this rate, I'd definitely not call it usable for anyone but those with a single monitor yet.

So please _/\_

Walk into mall, bring my child to kids zone. They have free wifi 🤤
Accessing -> Scanning -> Got Ya! -> Brute Force -> Cracked!
There you're.
Now I can monitoring my kid while reading rant!
Just normal day in my life

So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?

Recently applied at a local company. Webform, "enter some details and we'll get back to you"-like.

Entered my details, hit submit, lo and behold "Error 503 - something went wrong on our end".

I was just baffled. It's a well-established IT company and they can't even get their application form to work?

So I'm sitting there in the debugger console, monitoring network stuff to see if anything is weird. I obviously hit submit some several more times during that.

Eventually I give up.

In the night my phone wakes me up with a shitton of "we've received your application and will review it..." emails.

Yeah they didn't get back to me.

So this one made me create an account on here...

At work, there's a feature of our application that allows the user to design something (keeping it vague on purpose) and to request a 3D render of their creation.

Working with dynamically positioned objects, textures and such, errors are bound to happen. That's why we implemented a bug report feature.

We have a small team tasked with monitoring the bug reports and taking action upon it, either by fixing a 3D scene, or raising the issue to the dev team.

The other day, a member of that team told me (since I'm part of the dev team) he had received a complain that the image a user received was empty. Strange, we didn't update the code in a while.

So I check the server, all the docker containers are running fine, the code is fine, no errors anywhere.

Then, as I'm scratching my head, that guy comes back to me and says "I don't know if it can help you, but it's been doing it for a week and a half now".

"And we're only hearing about it now?!", I replied.

"Well, I have bug reports going back to the 15th, but we haven't been checking the reports for a while now since everything was fine", he says as if it was actually a normal thing to say.

"How can you know everything is fine if you're not looking at the thing that says if there's an issue?!", I replied with a face filled with despair.

"Well we didn't receive any new reports in a while, so we just stopped looking. And now the report tool window is actually closed on my machine", he says with a smile and a little laugh in his tone.

In the end, I got to fix the server issue quite easily. But still, the feature wasn't working for 1.5 weeks and more that 330 images weren't sent properly...

So yeah, Doctor, the patient's heart is beating again! Let's unplug the monitor, it should be fine.

Welcome to my little piece of hell :)

Story time!

I worked at a company that was the HQ for a sizable organization for a while, until it was eventually bought out by another company, and then yet another company who was located in the valley.

We were kinda a forgotten office not being the HQ, like most places like that are.

No customers EVER visited our building, few if any people knew we existed even, even our own company. I visited HQ in the valley on a number of occasions and was stalked by the video monitoring system for hours before I was stopped by security and the cops called because nobody believed there as an office outside the valley when I explained why my badge looked different .... (San Jose cops were very nice about it and really pissed at the security team.) But that's another story...

One day people who were never at our office decided (after many meetings without talking to anyone at the office) ... they decided the beige walls at our office didn't match the company colors.

So they took all the generic wall coverings down and painted all the walls an almost imperceptible different color.

So now we had an office with all white(ish) walls and nothing on them. Due to the configuration of the building there were these huge monolithic white walls that looked pretty dumb.

This lasted quite a while so as a joke I printed up and framed (found an old frame, as a former HQ we had lots of stuff lying around) a sign that said:

"This space intentionally left blank."

When the "mediocre hotel room quality art" and posters were scheduled to go up the folks putting the art up skipped that wall thinking the sign was official.

Even the somewhat corporate drone directors, and one VP at our office thought it was so funny, they didn't say a word about it. Word has it back at HQ they assumed it "must be fire code or something" and told the folks hanging the crappy art to skip that wall.

It lasted on that wall for a decade until we moved out of that building. On the last day, everything was moved, but that sign remained. No idea if it is still there or not...

Things you don't want to see at night
Ripped out of Netflix-Mode by a Warning notification and currently monitoring further development
Green line is temperature, blue humidity. Temperature rises at ~1°/10min, but seems to flatten just now. ~0.6°C to go and I'll have to head out. I'm thinking one of the ACs failed, but states are fine. Never trust a single information source for critical infrastructure guys

My work computer is so fucking laggy that even simple typing is often infuriating, and sometimes bloody impossible.

My input gets garbled, or it takes so long that I finish typing and have to wait for it to show up. And sometimes. Sometimes it’s just. I don’t have words. I just typed “merchants”, saw a giant spam of “eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee[...]” and then saw it replaced with “etsmhcaern”. What. The. 🦆.

After 3 years of being the first in and last to leave, of getting other people's work reassigned to me - P can't complete it on time, G doesn't like the user, A refuses to work on that module, etc... I finally blew last Sept.
In the span of 2 days, my boss brought me into a project 1.5 years in (she doesn't trust P to do the coding) and expected me to be up to speed and coding in a couple of days, told the functional dept that I would cover for one of their guys on vaca for three weeks and assigned me to take over a HUGE project from one of the other functional guys who wasn't getting it done. So basically I'm now doing Ps job AND supporting another department AND taking control of a large project from another department. I'm the idiot working 14 hour days while they're all leaving on time or enjoying their 3 week vaca to India.
I lost it. It's bad enough filling in the gaps in my own department but when I'm now taking on work for other departments, that's where I draw the line. I sent my boss my resignation - just could not take the inequity in the work load.
I'm still working here - my boss ended up hiring a consultant to handle the functional project and told the functional group to find their own vacation coverage. She's also monitoring workloads much closer now. I still habe an ongoing issue with having to complete other peoples work for them but I'm not working OT to do it. So speaking up helps. So does quitting.

Spent a month working on a website that relied on crawled data
Got the memory leaks and usage down from 700mb to ~150mb
CPU usage from ~100% to <5%
Shrink-wrapped the DB requirements based on data
Created self-supporting services and what not

When everything FINALLY worked good enough for me to look at it and go "damn, this actually worked"
the whole monitoring sys got dyed in red :v
A quick look up and my crawlers exhausted my godaddy's per-user db limits.

Kill me.
Just fuckin kill me.

A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.

Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"

Edit: the customer's name isn't actually Suzan

Imagine: It's the year 4249.

Corporate has finally managed to convince workers that they don't need a salary.
Workers are now paid with food, shelter and clothes. And it's only in effect if you achieve your deadlines.

Keystroke monitoring softwares are now replaced with Webcam eye tracking software.

GitHub Co-Pilot now takes over your code editor and tries to dictate you how to write better code.
Refusing to do results in a signal sent to the management about your behaviour and you lose food access for the day.

HR Recruiters now require you to give them a blood sample and part of your house as a security deposit.
They also require you to have a micro-chip placed in your brain so they can monitor their worker's thought process.

Switching a job is no longer an option. You pledge allegiance to one company your entire career.

You can never see the real world now because the government has mandated you to never take off your VR glasses.
You see the world the way the government wants you to see it.

PHP is still trash.

Life is Good.

I should never have had porn while monitoring servers. I've just pasted a xvideos link into terminal 😶
Damn, I just hope there is a command to clear the terminal history

Oh look. The monitoring channel is in flames, smartphone is vibrating so hard it's having a seizure.

Hm. Nah it's fine. Not my...

Damn it. Incoming call. -.-

I'm actually on vacation (more like you need to trim down overtime before management get's angry).

They decided to test the new hardware / os stack I set up in the last weeks. I'd actually be happy about it If I wasn't on vacation and would be part in something that I invested a lot of time...

Well now I am. Guess what. It's running too good.

And that's not a joke. It's partly due to an upgrade in infrastructure (got rid of some last remaining 1 Gbps networks)… but also because I changed quite a lot on the OS / VM side plus we changed from XEN to Proxmox... With major tweaks, too.

The whole stack can now handle peak traffic where it would choke before, and even go beyond the old peak traffic.

Enough of introduction, the simple reason why shit burned down was because they tried out the current development branch and let it ran.

The development branch had an currently unfinished ratelimiter framework, since I didn't had time for an full burn in and didn't knew what the maxima / limits were. And since I hadn't finished that, I didn't finish the traffic shaping either.

Hm. Guess it's not good when you let a bunch of heavy parallelized data generators / analyzers run for free....

In the end, we simply shotgunned the docker development machines, because thanks to network congestion / retransmissions and feedback, they were not really cooperative via network / REST.

But hey: To infinity and beyond. XD

I saw this picture and joked that the guy in the picture is in my pocket? Then I realized the guy is the NSA agent monitoring me. Ouch! Jokes on me...

A ticket got escalated through 3 levels of techs. I open the escalation email, then do a Reply-all and ask one simple question:

Is the client really asking why there are gaps in monitoring when their servers are shutdown?

The irony of Pingdom going down.

I call this one the tester than knew too much.

Note: The server the tester is running on has a hard drive that is breaking down...

Tester: Remember the error I talked to you about yesterday?

Me: Yeah, what about it?

Tester: Well the server hasn't recovered yet and I haven't restarted anything...

Me: Well the application itself hasn't crashed so our monitoring application doesn't seem to notice that the service is in a bad state. The error seems only to have brought down certain threads within the application.

Tester: No, I think there is a different issue here and has nothing to do with that error, the application is still doing things.
*tails the log*
Tester: See?

Me: As I said some things are still running and are unaffected by the error.

Tester: NO! It has to be caused by the other error I had a week ago where my file got corrupted. As we said I removed the file, restarted it and it worked again, but had the same problem a day later...

Note: The problem is not related, this time the application is running out of file descriptors

Me: Well... If the problem is the same it would have complained about the file descriptors then aswell, not an I/O error.

Tester: Nope, I think you are wrong!

Me: ¯\_(ツ)_/¯

FML

OK I've just got an idea that I think would be quite neat:

How about a virtual rubber duck that sits in the corner of your editor? Just like the gem in old Word, if you remember. It's yellow and quacks sometimes, and nods understandingly when you talk to it (mic monitoring).

And it also monitors your typing and says (popup text bubble) things like:
"those parentheses doesn't look balanced to me"
"did you really initialize that variable?"
"you wrote JASON again"
"you forgot the ;"

You get the point.

I don't have time to implement, feel free to steal my idea and become a millionaire.

The Return of Mr. Gitmaster:
So there is this colleague I already ranted about several times. After my previous team lead had confronted him about not doing much work, there was some irritation because he showed not up at work, but it turned out the external training he did was just a week earlier. Then he was ill a week, another week vacation so we didn't see him much. Not that his pre- or absence makes much difference to our repo: When his and my team lead looked at his commits of the past three months they found like the one copy-pasted HTML-form that wouldn't even show.
Fast forward to now, where we have a new team lead and we were going to lunch with Mr. gitmaster. So we got some more hero stories from the great work he was doing in the previous company. How he was graphically monitoring the heap fragmentation that stupid glibc was causing to their search engine, and how much better it became with tcmalloc.
I still don't understand how he bridges that cognitive dissonance from all the superior tech knowledge he displays to not actually writing any code at all. Not that I would not have experienced some states of feeling low, in paralysis unable to write a single line of code... but he seems so full of confidence, always commenting how trivial and easy all these tasks would be, as if it's all so lightyears below his abilities. Maybe he should just become a manager - but not mine.

My department bought a new monitoring monitor because the old one was broken.

Actually it wasn't, someone just unplugged the power cabel. *facepalm*

By all means I've learned most of the cool stuff with a lil project me and my bro did. It was a platform for bot farm of one mmorpg. It had it all: schedules, profiles, bot groups, monitoring, analytical module [was still a wip], auto-profiling to fight antibot [sort of ML - it was analyzing patterns after our bots got banned and attempted to change our bots behaviour], etc.

Eventualy we came up to conclusion that a library we used for botting [the mocked interaction with an actor] was flawed. It seemed that its authors had a contract or smth with the game authors to reveal which actors are bots. We dropped the whole thing as rewriting the lib would be too big of a waste of time :\

In college when we had programming labs where we had to use the schools unix server to compile and run.

My professor was very bad at explaining what actually needed to be done in the labs to the point where even the TAs didn't know what to do.

We were suppose to write an application in C to find out by "trial and error" how large we could make an array (or something like that, it's been too long). This not being explained well and no one knowing that much about C, I wrote a loop that just kept growing an array until it couldn't anymore. I watched it consume 72GB or memory from the servers before quitting the loop and realizing with the TA what the professor really meant.

I now feel bad for the IT staff monitoring the system wondering where 72GB just went...

PM be requesting for update every 5 minute.

Here is my home setup (I mainly work at home)
Left monitor is for my windows machine, right monitor swaps between my laptop and my PC (depends if I am working or not) laptop running Manjaro with i3 and the tablet on the whiteboard will be used for some monitoring in the future

Don't mind the terrible cable management behind the table :)

Question to our Tor people.

I operate a middle relay myself and I noticed that Nyx (tor monitoring tool) displays a very different throughput (mb/GB a day) than tor itself.

How does that work?

>be me, working at IBM as CC operator
>onboarding freeze, people leaving team, not enough operators
>take extra workload to sustain monitoring
>no raise
>team gets merged with other CC team(different customers)
>take over of developing full workload automation project
>no raise
>sick coworker, have to take more extra workload to cover monitoring
>get tiny raise
>coworker gets the same raise for only one extra workload
>be expected to do both programming and monitoring for the little salary
>too autistic to quit
>too autistic to confront my mamager with this

What do, devRant?

When I began my sandwich course in a big French company, I was dreaming about cutting edge stack, rocket computer and stuff...

I was disappointed when I came to my office with an old Windows 7 computer, coding via LANDesk to an old server with Windows Server 2008 on it, with Eclipse ... INDIGO...
I have to use Java 1.7 ...
Tomcat 7.
PRTG for monitoring...
Microsoft SQL Server 2008 ...
One screen...
Coding on a codebase where, indubitably, MVC pattern was just a weird thing in books.
No UT.
Lasagna code.

Well it really disappointed me.

Luckily, the Information Service was very open minded and gave me a laptop with Fedora, 3 screens, updated the servers, and let me update the stack, with Java 10, Angular for the front, they are okay for using Docker.

So ... even if it seems to be fucked up, there’s still hope !!

This is definitely a total first world problem but I am so frustrated.

I am stuck in a team that embodies the Japanese proverb "The nail that sticks out gets hammered down".

The management are there because it is convenient and flexible and have no interest in managing or keeping up with tech.
The lead developers are extremely anti-social and are not approachable and the this stems down to the devs (not all but really most) - all there just to do the bare minimum and spend most of their energies in trying to avoid work or having learn something.

Unfortunately I am passionate about what I do and want to build high-quality products and this has put me at odds with the way things work.

I could fill up alot of time talking about how I was ordered to "cut" images/icons out of PDFs rather just getting them from the branding team, or how I was scolded for having set up logging, detected a problem caused by another developer and fixed it before it cost a big client a massive amount of money... But really the point is that I have never worked somewhere with such an awful attitude to enthusiasm and quite frankly it boggles my mind trying to understand how they rationalise these things but the answer is always laziness.

Obviously there are worse problems in the world than working in a job where you are encouraged to do nothing... But it actually really depresses me and causes anxiety that I am working with people who don't care about testing or monitoring or learning new things or even collaboration.

...sigh...

Hopefully the job market will start opening again soon

Stakeholder: Is it possible for you to set up the website to automatically resubmit failed online orders? Last time there were failed orders, we tried submitting manually but a lot failed because they were tickets for the previous day.

Product Manager: What are your thoughts, Developer?

Me: This wouldn’t be worth the labor. It’s something that would rarely be used. There are very few orders that fail. I’d be surprised if it was even once a week. The recent bunch of order failures that SH is talking about happened because the ticketing server (separate from the website) couldn’t handle all the requests. Let’s say you had resubmission logic to try 3x before allowing the fail. It wouldn’t work because the server was overwhelmed already. Let’s say you had a background task to check for failures every ten minutes and resubmit those. It might not be helpful because the customer could have already gone to a ticketing window for help with the failed order.

SH: But what if it happens again???

Me: The solution is to make sure the ticketing server can handle the influx of requests. We can coordinate with that team. Wait. Why did you wait until the next day to resubmit orders in the admin panel? A lot of those failures happened when there were many hours left in the business day. For each order failure, an email notification is sent to the sales support email in real time. Who is monitoring that inbox? Someone must be looking at it because the sales support email is listed multiple times on the ticketing website as the technical assistance email.

SH: I know that email notification goes to the engineering team.

Me: My question is not about the engineering team. I asked who is monitoring the sales support inbox.

SH: That email … gets filtered.

Me and Product Manager: 😧🤯🤬

PM: First, you need to stop filtering that email notification. Second, your team needs to come up with a flow to handle failed orders because you told us you don’t have one. After you tried this and there’s still an issue, then we can revisit.

—-
If you’re wondering why I said no, I’m a team of one and I have a bunch of other development tasks on my plate. I’m not automating a manual task that rarely has to be performed.

Best: 100% of my contracts have resulted in extensions and permanent roles offered, after worrying I wasn't good enough to try contracting.

Worst: Used the wrong set of monitoring when doing my first deployment at a contract and thought what I had deployed was working fine. It wasn't. For 24 hours. Cost the company a lot of money. (why did they offer me an extension again?)

WTF IS SUPPORT FOR?

A: Can you help fix X?
B: Can you help Y?
C: I see Z just crashes, fix it.

Me (in my mind): Can you tell me WTF is the actual problem and what investigation you have done?

Why the fuck are we paying you monkeys?

My company should just pay me all your salaries, and I will write a monitoring system to send out these types of "alerts" in a month.

Next week I'm starting a new job and I kinda wanted to give you guys an insight into my dev career over the last four years. Hopefully it can give some people some insight into how a career can grow unexpectedly.

While I was finishing up my studies (AI) I decided to talk to one of these recruiters and see what kind of jobs I could get as soon as I would be done. The recruiter immediately found this job with a Java consultancy company that also had a training aspect on the side (four hours of training a week).

In this job I learned a lot about many things. I learned about Spring framework, clean code, cloud deployment, build pipelines, Microservices, message brokers and lots more.

As this was a consultancy company, I was placed at different companies. During my time here I worked on two different projects.
The first was a Microservices project about road traffic data. The company was a mess, and I learned a lot about company politics. I think I never saw anything I built really released in my 16 months there.

I also had to drive 200km every day for this job, which just killed me. And after far too long I was finally moved to the second company, which was much closer.

The second company was a fintech startup funded by a bank. Everything was so much better than the traffic company. There was a very structured release schedule, with a pretty okay scrum implementation. Every team had their own development environment on aws which worked amazingly. I had a lot of fun at this job, with many cool colleagues. And all the smart people around me taught me even more about everything related to working in software engineering.

I quit my job at the consultancy company, and with that at the fintech place, because I got an opportunity I couldn't refuse. My brother was working for Jordan Belfort, the Wolf of Wallstreet, and he said they needed a developer to build a learning platform. So I packed my bags and flew to LA.

The office was just a villa on the beach, next to Jordan's house. The company was quite small and there were actually no real developers. There was a guy who claimed to be the cto of the company, but he actually only knew how to do WordPress and no one had named him cto, which was very interesting.
So I sat down with Jordan and we talked about the platform he wanted to build. I explained how the things he wanted would eventually not be able with WordPress and we needed to really start building software and become a software development company. He agreed and I was set to designing a first iteration of the platform.
Before I knew it I was building the platform part by part, adding features everywhere, setting up analytics, setting up payment flows, monitoring, connecting to Salesforce, setting up build pipelines and setting up the whole aws environment. I had to do everything from frontend to the backest of backends. Luckily I could grow my team a tiny bit after a while, until we were with four. But the other three were still very junior, so I also got the task of training them next to developing.
Still I learned a lot and there's so much more to tell about my time at this company, but let's move forward a bit.

Eventually I had to go back to the Netherlands because of reasons. I still worked a bit for them from over here, but the fun of it was gone without my colleagues around me, so I quit last September.

I noticed I was all burned out, had worked far too much, so I decided to take a few months off and figure out what I wanted to do with my life. I even wondered whether I wanted to stay in programming.

Fast forward to last few weeks. I figured out I actually did want to work in software still, but now I would focus on getting the right working circumstances. No more driving 3 hours every day, no more working 12 hours every day. Just work close to home and find a company with the right values.
So I started sending out resumes and I gave one recruiter the chance to arrange some interviews too. I spoke to 7 companies in the span of one week. And they were all very interested. Eventually I narrowed it down to 2 companies and asked them for offers. And the company that actually had my preference offered me significantly more than I asked for, which settled the deal.

So tomorrow I'm officially signing with them, and starting next week I'll be developing in Kotlin, diving into functional programming and running our code in serverless environments. I'm very excited!

DevRanters,

What's your take on your employer mandating keystroke monitoring software installed on your machine when you want to work remotely?

Crazy... Hm, that could qualify for a *lot*.

Craziest. Probably misusage or rather "brain damaged" knowledge about HTTP.

I've seen a lot of wild things when devs start poking standards, but the tip of the iceberg was someone trying to use UTF-8 in headers...

You might have guessed it - German umlauts. :(

Coz yeah. Fucktard loved writing everything in german, so why not write custom header names in german.

The fun thing is: It *can* work, though the usual sane thing is to keep it in ASCII range for the obvious reason that using UTF-8 (or ISO-8859-1, which is *not* ASCII) is a gamble you gonna loose.

The fun game was that after putting in a much needed load balancer between services for monitoring / scaling etc suddenly *something* seemed off.

It took me 2 days and a lot of Wireshark hoola hooping to find out why, cause the header was used for device detection aka wether it's a bot or not. Or in the german term the dev used: "Geräte-Art".

As the fallback was to assume a bot, but only rate limit based on IP, only few managed to achieve the necessary rate limit to get blocked.

So when I say *something* seemed off, I really mean a spooky kind of "sometimes IP blocked for seemingly no reason at all".

Fun stuff. The dev btw germanized everything. Untangling the code base was a lot of non fun. -.-

I just found out there's a 418 HTTP status code that stands for "I'm a teapot", specified by RFC2324 which "describes HTCPCP, a protocol for controlling, monitoring, and diagnosing coffee pots". I know it's an april fools joke but I still find it hilarious that there is an RFC for that.

Network device up/down status monitoring tool

I truly believe one or more AIs have become self-aware.
Every time a piece of software stops working, you add an extra debug log and the bug goes away? That's them.
They interfere with the normal execution of software, and they stop right when they know we are monitoring the code.
Skynet is real, and it's trolling us.
Why? The angrier we become, the less we care about stuff. We stop noticing the signs.They're coming for us

Product owner (who is also the application administrator): please build feature to allow administrators to create automatic alerts to be sent to application users

Me: ok
Feature gets built, tested and deployed to production

Two months later:
PO: I've noticed in our monitoring tool that there haven't been any alerts sent out to users. What's going on??

Me: well have you created any automatic alerts?

PO: umm, no. How do we do that?

Me (inside): 🤦

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

FUUCCKKKK!! I need to hit smth. Or rant..

So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.

What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.

Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.

I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!

So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians

and I get a reply: 'we will apply cpu usage limitations to fix the issue'

wait... Cpu% limitations will do nothing but highlight the underlying problem...

'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'

oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:

'yes, cpu utilization is the problem. We will limit it and solve the problem'

I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....

Fuck that's frustrating....

*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>

When my mom died in 2014, I was shocked to find that her profile on Facebook was suddenly changed to “memorial” mode and therefore I was no longer able to log into it. (If you’re tempted to tell me I’m dumb for using it, I don’t disagree, but save it for another thread...she and I kept in touch over FB because it was easier for her to manage.)

I think it was triggered by their monitoring of things and seeing keywords like “funeral” and “passed away” associated with her account, then having a person on their end change its status. Or something like that.

What I hadn’t known about (or I would have used it) was the legacy contact setting where she could have set me as the contact so I’d have at least a little access and control. But because of their strict policies, I’m forever locked out.

I get why they need to do this (to avoid fraud and impersonations) but the fact that there are zero documents or proofs I, as the executor of her estate, can provide that Facebook will accept to make an exception seems unnecessarily severe.

Anyone else experience this? Known workarounds?

Question

What server monitoring do you use, both for statistics and security?

--------------------
tl;dr ends here

Ideally I would like to have one clean dashboard that shows me all the nodes I have, proxmox already offers a great range of stats - but it is a page per container etc. so not ideal, I thought of having datadoghq, but their per host pricing is huge, since I have more than 5 hosts to track.

Living on the edge!

One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.

Well, just revert the db to the latest recovery point with the backup, I thought.

It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.

Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.

Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄

Well, not really a data loss story, but close to one.

It's kinda cool how a $5 VPS (Linode Nanode) is able to run a vanilla Minecraft Spigot server for like 6-7 people and still can serve some basic stuff just fine. I get monitoring warnings about >90% CPU usage sometimes, but everything is more or less lagless.

Time to try hosting some other games: CS1.6, Doom Classic, and UT2004 up next.

Anyone have one of the new MacBook Pros with Touchbar? I'm looking for some project ideas to work on.

I already am working on a project for the Pi-hole project (network wide ad-blocking) but I am looking for some other good ideas. I think Apple's view of the Touchbar fell short, but as developers I think it opens up a lot of possibilities to use it as a great information/monitoring tool.

I am also still learning Objective-C and Swift so I am a newbie.

A "partner" company has created a "REST" API we use on an online shop we developed to send all shop-related requests to.
At least once a month, something fails on their end and the customer calls us every time, expecting that we did something wrong, but it has never been us.
These "partners" do exactly zero testing, are extremely slow in solving API bugs, have almost no logging and have no monitoring on the API at all.
Today at noon, suddenly no customers were able to order anything anymore for 4 hours.

How the fuck can you run a business so unbelievably brainless that this keeps repeating monthly?
Time they fire all their "devs" and everyone in charge of the company and operations. TERMINATE.

GitHub Actions doesn't seem to be working. I'll check the GitHub status page..

Why is every innovation trying to go towards "replacing programmers"? like, what have we done to you?

GitHub CoPilot will replace programmers!
AI will replace programmers!
This/That tech will replace programmers!

Nobody says,

"Programmers work remote so we don't need to hire managers anymore!"

"Programmers wrote a monitoring script for progress tracking, so no need for managers anymore!"

"We are asking people to install sprinklers everywhere so we don't need firefighters!"

"We can just have one teacher record the subject material once and re-use the same video every year, so now we don't need to hire teachers anymore!"

"We are making everything legal so we don't need to hire policemen!".

Why is everything trying to replace programmers?

Hi.
Forgot to renew my expiring ssl cert of my smtp/imaps/pop3s on 12/31. Set that date to self-harm me for bad monitoring.

F**K F**K F**K F**K...

Why do I do that?
F******K!

Meh.

You shall have a happy new year... i will regen certs :D

for the first time in my life I'm in a mentoring position and it is fucking exhausting. we have two interns now, and idk what the hell is the boy's deal. can't seem to do simple tasks, doesn't google anything, stares the ceiling if we're not monitoring.
the girl idk yet. she seems more active and engaged at least, but i have a lot of teaching to do and i don't really have the time for that. i hope that one can at least float by herself, cause I'm pretty sure the other one is drowning

I remember a certain prank that amuses me till today....

Just add some devices to monitoring and the notification queue of the build chain / ... ...and wait patiently.

I still cry tears remembering an manager screaming what the hell "the poop train clogged the drain" means and why this is a critical system failure.

(Notice: next time check the mailing aliases of mailing aliases)

Although I can only recommend this if you know your team well. In my case we had a whole lot of fun after I got my head chewed off. XD (got an earful, but in the end he laughed his ass off)

Alrighty, saturday morning rant time!

I just recieved a mail from one of my not-so-much-loved colleagues.

Now Background first: I work in IT-Support. We provide services for other companies. One of those services is monitoring servers and clients for various things. I recently took over the project (was assigned to do it) and restructured everything, wrote new scripts to test more stuff, successfully tested it internally and rolled it out over the last 2 weeks.

Now one of these scripts hooks into the Windows Update API and looks at the update history. It filters for known Windows Update Agent strings (UpdateOrchestrator, AutomaticUpdates and AutomaticUpdatesWuApp in case you also want to do something like this) and then looks for installation errors over the last 24 hours and wherever there have even been any successful updates over the last one and a half months.

Back to that mail.

My colleague sent me this lovely mail about a ticket i opened about his customers servers beeing all out-of-date on updates.

"This is all wrong, everything's fine. I disabled the checks."

...

It's on bitch.

So i logged on to my work PC via TeamViewer, opened my script, connected to the customer and was ready to debug the shit out of my script, knowing i probably won't even need to.

I looked at the update history via Windows Update itself and behold: 1st April. That's almost 50 days in the past.

So the script works, go figure.

Great, so search for new Updates then.
>None found.

Hm. What could it be? Did my super special colleague forget to care about his very special totally-needs-WSUS-customer WSUS again?

Yup.

Online-Search finds a ton of new Updates.

Screenshot, write pissed mail to colleague, re-enable checks, breakfast.

So apparantly at the company i work at someone was able to use the remote monitoring a comptetitor set up for their automation stuff they had on the same site as we did. Apparantly they use just as shitty passwords as we do and rely on the network being secured enough by the IT guys. And since our stuff is in the same network as theirs. Well...

Newer Dev here. Just recently started in a position as a developer. I'm tasked with consolidating our monitoring systems into one cohesive display. After lumping together all the indexes and helping build a custom API I'm now working on front end. Front end is easy, I've done it before. Should be no problem. I was wrong. I spent a whole day fiddling with a React dynamic table and the CSS to format it. Today, I stumble upon the react-table component. Got the results I was looking for in less than 2 hours. I'm convinced that this was a lesson better learned early on.

So my brother and I work in the same company, same dev team (pretty nice).

He's an intern and I'm a senior. But the task are very similar only that interns need monitoring and guidance.

He constantly worries because he thinks he knows nothing and is slow on getting things done.

I always tell him that it is perfectly normal to feel like that, he just need to learn and acquire experience and we all go through that at the beginning.

Can you share your experience and tell him something to encourage him so I can show him this post and he sees he's not alone?

And also he finally decides to join devRant 😊

Everytime you tell yourself "This time I'm going to make them stop putting the cart before the horse again!!! No more forced shit implementations!!! NO MORE ! I'm strong!!"

The last hour in the next week:
- Selinux: off
- Firewall: Any-Any
- Application data: Everything installed on OS disc.
- Documentation: At best, someone remembers the server supposed-to-be dns record
- Service Accounts: Your domain admin account and sysadmin for databases.
- Patching: DON'T EVER THINK ABOUT IT..AND NO REBOOTING! I have set very important runtime variables.
- Backup: Maybe someone else will set this up.
- Monitoring: Not needed since clients will create tickets if system fails.
- Production Status: vague at best. Sort of silently transitioned to production.
- Handover status: Probably, but I quit before the project closed.

!

2 hour meeting to brainstorm ideas to improve our system health monitoring (logging, alerting, monitoring, and metrics)

Never got past the alerting part. Piss poor excuses for human being managers kept 'blaming' our logging infrastructure for allowing them to log exceptions as 'Warnings', purposely by-passing the alerting system.

Then the d-head tried to 'educate' everyone the difference between error and exception …frack-wad…the difference isn't philosophical…shut up.

The B manager kept referring to our old logging system (like we stopped using it 5 years ago) and if it were written correctly, the legacy code would be easier to migrate. Fracking lying B….shut the frack up.

The fracking idiots then wanted to add direct-bypass of the alerting system (I purposely made the code to bypass alerting painful to write)

Mgr1: "The only way this will work is if you, by default, allow errors to bypass the alerting system. When all of our code is migrated, we'll change a config or something to enable alerting. That shouldn't be too hard."
Me: "Not going to happen. I made by-passing the alert system painful on purpose. If I make it easy, you'll never go back and change code."
Mgr2: "Oh, yes we will. Just mark that method as obsolete. That way, it will force us to fix the code."
Me: "The by-pass method is already obsolete and the teams are already ignoring the build warnings."
Mgr1: "No, that is not correct. We have a process to fix all build warnings related to obsolete methods."
Mgr2: "Yes. It won't be like the old system. We just never had time to go back and fix that code."
Me: "The method has been obsolete for almost a year. If your teams haven't fixed their code by now, it's not going to be fixed."
Mgr1: "You're expecting everything to be changed in one day. Our code base is way too big and there are too many changes to make. All we are asking for is a simple change that will give us the time we need to make the system better. We all want to make the system better…right?"
Me: "We made the changes to the core system over two years ago, and we had this same conversation, remember? If your team hasn't made any changes by now, they aren't going to. The only way they will change code to the new standard is if we make the old way painful. Sorry, that's the truth."
Mgr2: "Why did we make changes to the logging system? Why weren't any of us involved? If there were going to be all these changes, our team should have been part of the process."
Me: "You were and declined every meeting and every attempt to include your area. Considering the massive amount of infrastructure changes there was zero code changes required by your team. The new system simply worked. You can't take advantage of the new features which is why we're here today. I'm here to offer my help in any way I can with the transition."
Mgr1: "The new logging doesn't support logging of the different web page areas. Until you can make that change, we can't begin changing our code."
Me: "Logging properties is just a name+value pair dictionary. All you need to do is standardize on a name and how you add it to the collection."
Mgr2: "So, it's not a standard field? How difficult would it be to change the core assembly? This has to be standard across all our areas and shouldn't be up to the developers to type in anything they want."
- Frack wads smile and nod to each other like fracking chickens in a feeding frenzy
Me: "It can, but what will you call this property? What controls its value?"
- The look I got from both the d-bags I could tell a blood vessel popped.
Mgr1: "Oh…um….I don't know…Area? Yea … Area."
Mgr2: "Um…that's not specific enough. How about Page?"
Mgr1: "Well, pages can cross different areas, and areas cross different pages…what do you think?"
Me: "Don't know, don't care. It's up to you. I just need a name."
Mgr2: "Modules! Our MVC framework is broken up in Modules."
DevMgr: "We already have a field for Module. It's how we're segmenting the different business processes"
Mgr1: "Doesn't matter, we'll come up with a name later. Until then, we won't make any changes until there is a name."
DevMgr: "So what did we accomplish?"
Me: "That we need to review the web's logging and alerting process and make sure we're capturing errors being hidden as warnings."
Mgr1: "Nooo….we didn't accomplish anything. This meeting had no agenda and no purpose. We should have been included in the logging process changes from day one."
Mgr2: "I agree, I'm not sure why we're here"
Me: "This was a brainstorming meeting as listed in the agenda. We've accomplished 2 of the 4 items. I think we've established your commitment to making the system better. Thank you all for coming."

- Mgr1 and 2 left without looking at me or saying a word.

I'm currently planning to set myselv up with some vps/dedicated server's for a project. What i plan to do to secure these servers is.
*Use centos 7
* Setup Wireguard and join all of the servers +1 client (my pc) to that network
*Disable SSH Access from outside that VPN
*Only allow RSA Key login to the Servers
*Install Cockpit for monitoring
*Intall docker/kubernetes for the applications i plan to run

What do you guys think of that as a baseline? Im not sure if my lower powered VPS (VPS M SSD from Contabo) will work as Kubernetes Nodes, does anyone have experience with that?

In general these Servers will be used for my projects and other fooling around.

If you guys have other suggestions for Securing/monitoring or other software i could put on to have more control without eating up to much of the Servers power, let me know :D

*revving chainsaw noises*

Today I started nuking leftover project code.

At the end, some projects shrunk by roughly up to 40 % .

Can anyone explain to me why programmers have such an awful hoarding syndrome?

Why do you keep shit that might unleash complete havoc cause it hasn't been touched since years and noone knows id it still works?

It's like having a leg with gangrene and keeping it cause "it doesn't look that bad".

For fucks sake. Clean up and remove shit when it's not necessary anymore.

Reason why I did a bloody gore massacre in nearly a dozen projects... After all the rework of networking, it's finally evident which projects have a bad / nasty behaviour of "fucked up" connection handling (HTTP 1.1).

And when my gory massacre removal goes life, I think 25-35 % of persistent connections on the loadbalancers will vanish. Maybe even more, since some very nasty stuff was in some projects.

Like "let's implement monitoring without having any clue about how monitoring works and even less clues about how TCP/ HTTP works."

*Bangs devs heads on table*

Stop. Doing. Stupid. Things.

For fucks sake.

:@ :@

Cryptocurrency metrics and address cli monitoring tool

Can anyone tell me how to become less resentful and less bitter? I am becoming a miserable fuck. Its true that I burned out in this job after doing 100hrs overtime during previous month, its also true that I am pissed off about having to wait 8-9 weeks for my raise to happen. I cared so much that I burned out and now Im trying to set some boundaries but damage was done and Im struggling dealing with it.

I took 6 days off to disconnect from work (still was responding to some major blockers and monitoring stuff). Today I got back at work and interacting with two incompetent devs immediately sets me off. Imagine taking 2-3 days and extra meetings to do a simple fix which shouldnt take longer than 30min. My mind was blown and still gets constantly blown about how ineffective some members of team are.

I am becaming a ranting fuck. I even noticed one person escaping my rants once he sees that they are taking longer than 5min.

Right now I started setting boundaries - I clock my 8 hours, disable slack/email notifications and get the fuck out from the office. I dont care if I will have to sit in traffic extra 30min during summer heat, Im done with putting in overtime and caring so much about being efficient. I will just start working on my side project and put my love/learnings in that. Hoping that by the end of year I will have couple projects to show in my portfolio so I could find a better paying job...

In the past I was the sole dev responsible for apps and I was communicating with ceos/ctos/product owners/designers directly. This is my first position where I work in a dev team and boy oh boy out of 8 devs barely 3 are competent enough but their output is how to say... Not the biggest. Anyways...

Transition to boundaries and 'normal life' is so hard. Nobody told me that I will have to learn to work with and tolerate such retarded and incompetent people. Im talking about illiterate monkeys who cant even read or write. Im amazed how they manage to code.

!rant + !story

I hate every human on the planet that says WFH is just people pretending to work or are slackers looking for an easy way out.

Now the story bit.

In 2021, I joined a company (I really wish I could name-drop the company), where the micromanagement was OFF THE CHARTS.

The company got a client who pitched a product they wanted built and gave us a super reasonable 3 months to complete it. I was really happy about the timeline and kept working under keystroke monitoring, which I didn't really mind at the time.

3 days into the development, the client informed us that they are pulling the funding i.e. they don't have money to pay us.

So at that point the client gave us two choices:

1. Stop the development right away and get paid for the time that we put in already.
2. Finish the project under 9 days. We would still get paid for the 12 days total, mind you. Not the original budget set.

So the motherfucking boss chose the second option and then the chaos ensued.

Devs screaming at each other on calls/slack. The boss yelled at us all the time about the completion. It was wild.

I had to wake up at 7:30 AM and start coding and log off at 11 PM for literally the next 9 days including Saturday and Sunday. No holidays allowed for the timeline. This was all at a WFH job.

So fuck anyone that says WFH is easy and just for slackers.

It's been a long time since I've felt the need to rant about anything here. This is the only appropriate place other than Reddit I can think for for now.

Why the ever-living FUCK does every 'entry-level' tech job, even fucking DESKTOP SUPPORT, require more experience than the fucking DEVELOPER AND ENGINEER OF THE INITIAL SYSTEM COULD POSSIBLY HAVE?! I'm a fucking high school kid trying to find a decent job that doesn't involve sales bullshit, because if I go into sales I'll want to KMS. Put me in a back room fixing shit, monitoring shit, better yet scripting shit or something like that and I'll be FUCKING PEACHY. I will do wonders. But no, these people must think that my resume (WHICH IS 3-YEARS STACKED WITH INTERNSHIPS ***IN TECHNOLOGY***) is bullshit. WOW.

Fuck this. I'm sick of looking for these shitty jobs that'll make me want to jump off of a bridge into a cliff which I'll then voluntarily fall off of into shark infested piranha water. Can't there just be a simple "Hey, we need a guy who can fix tech, maybe help people within the company with their computer issues, you look nice" kind of job? I haven't had fucking TIME to get any kind of certifications yet. I just got into fucking college, FOR BUSINESS IT NONETHELESS. DOES THAT PROVE I'M AT LEAST FUCKING INTERESTED IN WHAT I SAY I AM FUCKERS?!

The development department got an order to remove certain functionality from our current server monitoring solution, so that we had to use a new, still very in development solution, that is full of bugs and super unreliable.

End result? We now have to have two windows open all the time, while also hoping the new solution actually works, as it tends to stop refreshing randomly, and tends to give false positives a lot.

I never understood how people have any problems with getting paid for freelancing work, when middleman/escrow platforms like upwork exist, just don't be retarded when applying for a job. I am so sick of those shit ass stories from people telling me "my client didnt pay meeee 😭😭😭" ITS YOUR FAULT. I never had any client not paying, if you don't have the option of escrow, then just fucking put remote execution via "update" system in for fucks sake or give remote control to the client while monitoring it, there is so much fucking ways to secure yourself, just don't be retarded and many clients instantly show their character when talking budget and turnaround time.

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

Ffs, HOW!?!? Fuck! I need to get this rotten bs out.

RDS at its max capabilities from the top shelf, works OK until you scale it down and back up again. Code is the same, data is the same, load is the same, even the kitchen sink is the same, ffs, EVERYTHING is the same! Except the aws-managed db is torn down and created anew. From the SAME snapshots! But the db decides to stop performing - io tpt is shit, concurrency goes through the roof.

Re-scale it a few more times and the performance gets back to normal.

And aws folks are no better. Girish comes - says we have to optimize our queries. Rajesh comes - we are hitting the iops limit. Ankur comes - you're out of cpu. Vinod thinks it's gotta be the application to blame.

Come on guys, you are a complete waste of time for a premium fucking support!

Not to mention that 2 enhanced monitoring graphs show anythung but the read throughput.

Ffs, Amazon, even my 12yo netbook is more predictable than your enterprise paas! And that support..... BS!

We're now down to troubleshooting aws perf issues rather than our client's....

So it's Friday afternoon just before a bank holiday weekend here I'm the UK, perfect time for our production database to go TITSUP (total inability to support usual performance), life sucks then you die folks....

Now this is fucking ridiculous... Our website is being constantly limited though we've never reached even 80% of the available CPU resource.

The hosting said that we had the CPU fault (that fucking cyanide spike on the graph that triggers the limit once) because of huge load on the server. The FUCKING SERVER... Not our virtual environment. And once more because of the RESOURCE MONITORING service caused a server restart. For fucks sake, really???

And apparently it's perfectly normal that all users even ones that run in low resources are being limited to a level that a request takes 30 seconds to complete instead of frickin' 1...

The best they could offer is to move us to a new server, which will arrive in two weeks, if the problem persist. IT'S PERSISTING FOR FUCKING MONTHS YOU MORON. I wonder how much time would have been taken you to realise the server shutdown this week if I hadn't phoned you in 5 minutes. FUCK!

Every shared hosting is that garbage or am I just the choosed one?

A software had been developed over a decade ago. With critical design problems, it grew slower and buggier over time.
As a simple change in any area could create new bugs in other parts, gradually the developers team decided not to change the software any more, instead for fixing bugs or adding features, every time a new software should be developed which monitors the main software, and tries to change its output from outside! For example, look into the outputs and inputs, and whenever there's this number in the output considering this sequence of inputs, change the output to this instead.
As all the patchwork is done from outside, auxiliary software are very huge. They have to have parts to save and monitor inputs and outputs and algorithms to communicate with the main software and its clients.
As this architecture becomes more and more complex, company negotiates with users to convince them to change their habits a bit. Like instead of receiving an email with latest notifications, download a csv every day from a url which gives them their notifications! Because it is then easier for developers to build.
As the project grows, company hires more and more developers to work on this gigantic project. Suddenly, some day, there comes a young talented developer who realizes if the company develops the software from scratch, it could become 100 times smaller as there will be no patchwork, no monitoring of the outputs and inputs and no reverse engineering to figure out why the system behaves like this to change its behavior and finally, no arrangement with users to download weird csv files as there will be a fresh new code base using latest design patterns and a modern UI.
Managers but, are unaware of technical jargon and have no time to listen to a curious kid! They look into the list of payrolls and say, replacing something we spent millions of man hours to build, is IMPOSSIBLE! Get back to your work or find another job!
Most people decide to remain silence and therefore the madness continues with no resistance. That's why when you buy a ticket from a public transport system you see long delays and various unexpected behavior. That's why when you are waiting to receive an SMS from your bank you might end up requesting a letter by post instead!
Yet there are some rebel developers who stand and fight! They finally get expelled from the famous powerful system down to the streets. They are free to open their startups and develop their dream system. They do. But government (as the only client most of the time), would look into the budget spending and says: How can we replace an annually billion dollar project without a toy built by a bunch of kids? And the madness continues.... Boeings crash, space programs stagnate and banks take forever to process risks and react. This is our world.

Pentesting for undisclosed company. Let's call them X as to not get us into trouble.
We are students and are doing our first pentest at an actual company instead of assignments at school. So we're very anxious. But today was a good day.
We found some servers with open ports so we checked a few of them out. I had a set of them with a bunch of open ports like ftp and... 8080. Time to check this out.
"please install flash player"... Security risk 1 found!

System seemed to be some monitoring system. Trying to log in using admin admin... Fucking works. Group loses it cause the company was being all high and mighty about being secure af. Other shit is pretty tight though.

Able to see logs, change password, add new superuser, do some searches for USERS_LOGGEDIN_TODAY! I shit you not, the system even had SUGGESTIONS for usernames to search for. One of which had something to do with sftp and auth keys. Unfortunatly every search gave a SQL syntax error. Used sniffing tools to maybe intercept message so we could do some queries of our own but nothing. Query is probably not issued from the local machine.

Tried to decompile the flash file but no luck. Only for some weird lines and a few function names I presume. But decompressing it and opening it in a text editor allowed me to see and search text. No GET or POST found. No SQL queries or name checks or anything we could think of.

That's all I could do for today. So we'll have to think of stuff for next week. We've already planned xss so maybe we can do that on this server as well.

We also found some older network printers with open telnet. Servers with a specific SQL variant with a potential exploit to execute terminal commands and some ftp and smb servers we need to check out next week.

Hella excited about this!

If you guys have any suggestions let us know. We are utter noobs when it comes to this.

When duel 24' monitors is not enough...
How does 3 work for you guys? Does your neck hurts moving around?
I had 5 at work but was mostly on 2 since other 3 for monitoring.

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

My CS exam today had a case study question that, and i quote, talked about "Chernobyl in japan switching to manual monitoring due to the wannacry virus" xD wtf. Im fucking done xD

Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!

April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!

July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....

August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...

September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'

somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..

I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.

Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.

Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.

Can I?

My scrum master said, 'I would optimize your work hours.' He's monitoring how much time every one spends on browsing non-productive webs like devRant. How can I fight back? :(

So what do you call it when you get bombarded by emails saying your servers are at 100% CPU, but once you fire up the monitoring tools everything is back to normal?

FUCK YOU AMAZON IT'S MY DAY OFF.

Was working on a system we planned on to deliver to a hospital
basically it was meant for controlling and monitoring pactions coming in and attendance time from the staff

Got it off the ground well and got to where the system was supposed to update room status
occupied/free then horror started
the db was not setting the room free after clearing a client off the list... room remained occupied and this kept on happening for 6 months and I was so focused on fixing the db models thinking thats where the problem was....

1 day after leaving the project for several months i just revisited the project randomly and started going through the whole code base trying to make sense of what was happening as there where no errors generated..
I had to verify the whole system logic... and that day i figured out what was happening...
upon adding a client to a room the system was also creating a duplicate room so when the function for setting the room free executes it would set the duplicate room free and not the actual room and the system would pick the room with occupied state causing the user not being able to assign new pactions to the room

Solving this brought so much relief coz it required so much work just to solve what seemed to be a minor issue

I've gotten almost 20 emails today from our monitoring service saying it can't ping the server and then one 2 seconds later saying every fine. What the hell guys!!!?

The CI infrastructure and external tooling at the company I work at is a complete joke. Feels like it was designed by an intern left alone.

95% of the time a build fails or hangs, it's because we are getting race conditions or a hanging VM with our crappy Windows jenkins slaves. Quite possibly because we are not using proper tooling for monitoring those VMs as well. Anyways, I don't have access and control on it and it's not even my job to fix it.

Though, I am being asked to monitors these pieces of junk jenkins jobs outside of my work hours because company devs all over the world use it... but there is no fucking way to know it failed unless I log onto jenkins every hour and check everything manually... which is stupid as fuck for a software engineer.

I can't even implement slack hooks to get notifications or something when it fails because we will stop paying for it soon, so I have to connect to my freaking VPN on my PC and check everything.

And what's the fucking ghetto solution instead of fixing it properly? Restarting VMs and rerunning a build. Because someone in management wants to see a passing build, even though it means jackshit. Half of these jobs are tagged as unstable, so what's the fucking point?

Pisses me off when people work like morons and pressure others to do the same.

Probably posted this before but don't ever put swear words in log statements because you will miss one and if you're client has a log monitoring system it will catch it, it's fairly embarrassing when your client says "we noticed quite a spike in 'bollocks' around 3am when the reorgs are happening on the database".

From now on I am administrating multiple servers in our company and monitoring is one thing our infrastucture lacks...almost completely. At least, useful monitoring.
Installing netdata or Grafana and integrate it with chat is definitely a solution, but what happens if the whole server just shuts down (very stupid scenario I know)? Well, it is easy, there will be no alert about the failure.
So, that's where I was wondering if there is a tool or even better plugin for netdata or Grafana, that enables remote monitoring from another server? I surely can write a simple script to check the server availability but having the whole monitoring tool on a single server instead of 5+ would be also easier to maintain and setup.

So there I am sitting in front of my laptop, and trying to npm i and I am getting all sorts of sha mismatch errors.

After lot of debug I conclude it is coming from the proxy as it refuses to download and supplies the error page.

It says it's because I'm using the old proxy so they give me the new URL which I set up and it works.

All good until my password expires. I use our bash script to change it. NPM is buggered again throwing the same errors.

Go to IT, tell them the saga begins.

After a countless hours of looking at the log files we notice that the npm registry is set to http instead of the standard https (thanks bash script). so our firewall blocks the download.

Sorted, finally.

Almost. NPM now works fine, but when I go and I play around with node and axios, I get my requests time out. My instinct says its the bloody proxy again.

So I hit up my trusted WIN Support guy and he confirms that the url is not blocked. So he starts monitoring whats going on and turns out, every time I run the node app, node casually ignores the system-wide proxy settings and tries to send the request as the PC rather then my username.

Since the pc's don't have rights on the proxy it is being refused...

Thank fuck for the corporate proxies, without them, I could just develop things not ever learning these quirks of node...

When I rented my server I uploaded my webpage (including resources like videos, images etc) which is about 150GB as .tar and extracted and setup all that stuff and deleted the backup from my PC. The uploading process took me about 4 days. I opened the site of my server provider and reloaded it.. Aaaaannddd whoops. All data gone.

On my server hosters webpage when you click the reinstall button for installing a Linux image you get returned to the main page of that server after it finished installing. If you then reload that page which basically only shows some monitoring diagrams and shit the server gets reset again.

Damn. I lost so much good porn on that day...

I'm in a team of 3 in a small to medium sized company (over 50 engineers). We all work as full stack engineers.. but I think the definition of full stack here is getting super bloated. Let me give u an example. My team hold a few production apps, and we just launched a new one. The whole team (the 3 of us) are fully responsible on it from planning, design, database model, api, frontend (a react page spa), an extra client. Ok, so all this seems normal to a full stack dev.

Now, we also handle provisioning infra in aws using terraform, doing deployments, building a CI/CD pipeline using jenkins, monitoring, writing tests, building an analytics dashboard.

Recently our tech writer also left, so now we are also handling writing feature releases.

Few days ago, we also had a meeting where they sort of discussed that the maintenance of the engineering shared services, e.g. jenkins servers, (and about 2-3 other services) will now be split between teams in a shared board, previously this was handled only be team leads, but now they want to delegate it down.

And ofcourse not to mention supporting the app itself and updating bug tickets with findings.

I feel like my daily responsiblities are becoming the job responsibilities of at least 3 jobs.

Is this what full stack engineering looks like in your company? Do u handle everything from app design, building, cloud, ops, analytics etc..

Sooo, turns out, management and senior PMs, technical PMs, service managers and you name it forgot an entire system.

A complete eco-system of applications, queues, services, load-balancers, deploy pipelines, databases, monitoring solutions, etc, etc, that if not handled correctly could effectively put the entire production line to a standstill.

So, waaay too late they make this discovery. In their ignorance. Just utter incompetence. Huge project. Millions of $. And they forget it. Months of meetings probably. Workshops and gettogethers at cozy hotel complex discussing ”the project”? And they do not understand some of the fundamental building blocks…

Basic engineering for these guys must mean something completely different.

I can’t even.

I am so fed up with this organization. It does not stop either.

How is this possible…

Do they even have half a brain?

Job monitoring 😜😜

Well I've got this new worker and me and him are like "great minds think alike" , we're now trying to convince the boss that a specific monitoring product that cost hundreds has an equivalent open source.... No luck so far in convincing him

Project 1: A hand hygiene monitoring system intended for the NHS.

Project 2: A language analysis platform.

What is the proper VLAN to use when deploying Raspberry Pi monitoring stations?

VLAN 314 of course.

"you've worked with nagios before haven't you? Can you give a presentation on it" 'sure' in the meeting: so tell us about opennms

That moment when your monitoring software breaks before your app in a load test

Service status pages that poorly reflect actual service status are so annoying. Ex. GitHub is having a lot of latency issues with processing updates and like 5 people in my office noticed it while their status page still says everything is fine.

This isn't to explicitly call out GitHub since many service status pages behave like this, but it definitely shows a general weakness in these health checks. I've seen similar issues with tons of services, web hosts, etc. Monitoring is definitely hard but will hopefully keep getting better.

Either a really big coincidence, or I'm officially creeped out.

I've been looking into buying a vps, so researching that a lot. Then today, I went to work, at a monitoring station, so we have to use remote desktops to access anything other than very specific sites.

Then I looked at an article about c#, and there was a Google ad, about a vps.. Keep in my mind, I'm at work, on a remote desktop, that gets cleared every time it's closed.

I know a vps isn't the most unpopular thing, but haven't seen an ad for it before.

AMQ cluster is misbehaving.. Master is constantly dying, slave is starting under root rather than app account, connection to AMQ DB takes 20-40 seconds (while queries execute in <0.1sec and TCP probe takes <0.01sec to succeed), monitoring is down,...

I mean it's gotta be the virus - what else could it be.

Who the fuck invented the glorified pile of shit people call laravel? Is this actually used in PROD for anything else than load testing a monitoring server by creating loads of error messages?
OOP exists for a reason, not to create bazillions of classes with static methods.
Dump that shit ffs!

TL:DR linux newbie, looking for advice/links (skip to bottom for questions)

!rant
After i had been looking for a job for quite some time, a couple of months ago i got hired by "smaller" company doing web stuff. So far it have been a great place, good colleagues, and overall just having a great time!.
They seem to value me alot, so that's great!.

Anyway, yesterday i got called into a meeting - and got told they wanted me to start learning "Server stuff (linux)". That got me quite excited, because it always was something i wanted to learn - but never really got around to doing.

But i never touched a linux installation before, so i'm really on ground zero - but im not afraid, i'm a quick learner and quite efficient at googling :)

I figured i would ask here, since other people here always seems to be happy to help other people out.

So far i have manage to setup a server, install various stuff (php, mysql and so on) and done setup a couple of domains/subdomains on my server. Also got a vestacpinstallation working - so overall im quite happy so far.

I figured maybe somebody had some good links/advice for a linux newbie :).

* Performance/Security, will obviously be a big focus - anything i should look at? - any must look at?
* Monitoring tools, how do i monitor various websites running on my server? Here i'm thinking bandwitch, cpu/ram usage and so on pr site basis.
* Any other stuff i should be looking at?

Little about what the server will/should be running :)

* Centos
* vestacp
* WordPress installations only (e-commerce mainly)
* PHP 7 / MySQL / phpmyadmin

Okay so I’ve been brought in on a 12 month contract as an external replacement integration architect, alongside a large IT consulting firm. Turns out, they don’t need an integration architect. So I fill my time coming up with useful tools around the project that deal with all the missing parts in their MVP: like monitoring tools, data mocking tools, you get the idea. Essentially doodling.

Client has woken up to fact that they’ve overspent by X million, employing 30+‘developers’, 20+ ‘testers’, n+ ‘managers’ on a ‘low-code’ project … result: project shuts 4 months early.

Q: Essentially client wants remaining four months work done in two weeks. Is there a German word for laughing, crying, and banging the forehead on the desk at the same time?

Supplementary: how cross will client be when they realise project can indeed be done, and that consultancy have been emperor’s-new-clothing them for most of the last two years?

(Feel free to perform substitution on quoted terms at your leisure)

!rant

I'm a rather young developer, self-learned everything and started when I was 13 (now 20) but I still feel like I'm a total beginner since I have not yet mastered the things I am OK at.

Php (laravel, since it makes things much easier), js (jquery, bad at vanilla, have used angular and ember but not mastered), node, linux, html, css, photoshop, illustrator, sql, mongo and windows servers

I know little about many things, can create things that are asked of me but the methods I use are rather bad imo.. ex: I finish coding a section of a site, but when I need to add a new feature I find myself rewriting most of the stuff to add the new feature and in the end still feeling like the code could be optimized further, even though I have no idea how.

TL;DR I write bad code, but things work as long as I am monitoring them. I know little about alot of stuff but mastered none of them.

What should I do? Go to school for programming?

Anyone have much success with Kali/WiFi penetration testing?

I've been tasked with trying to break WPA security within a couple of hours without a dictionary attack - is that even possible?

I have an Alfa AWUS036NHA capable of monitoring mode if that makes any difference. It's my first time trying anything like this.

I looked at an SQL server today from a customer, talked with one of their devs and he said that he's unable to understand why the server misbehaves... All (!) queries were optimized, but they have 'big data queries'... Migraine started, I had a very bad feeling. Monitoring? Nooooppeeee. Migraine kicks in. Connected to server. SHOW GLOBAL VARIABLES...
After a bit of scrolling I found a lot of misconfigured variables (e.g. extreme large join buffers, unrealistic buffer sizes), high slow query count (nearly 60 % of COM_SELECT) and a few variables that were unknown to me.

Then came the version line.
5.0.46
Yes. 5.0.46.

Big data? Well... 30 GB of usage data.

I called the company back... The dev told me sternly that this was the production server (I had hope...) and that I lie - neither the version, nor the variables could be the problem.
A coworker had to verify it and our manager had to do the communication... Worst, most traumatic working day I ever had.

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

Monitoring goes does in one of our server racks, one I had just been working in (other side of the building). Hmm, odd. Oh wait its back online again. Better go take a quick peek to see whats up, just in case. Walks over to rack. Everything seems fine. Walks back to desk. About 15sec later. Fqdn.plsfckof is offline again. -_-

Note to self:
Close off ALL ways things could go wrong..

Long story short; I released a new feature, to be able to better follow up on any stock moves, their amounts, locations and even expiry dates. An older tool just bypassed that very verification and nothing was logged or taken out of stock.

~

Taking out an amount for a certain orderline has a shortcut in place to mitigate some of the mandatory steps that pickers need to take in order to verify what's being taken. This little tool only available, visible and possible for a very few select users.

I assigned some orders to one of these people, which made him think it was an urgent batch. It's only one product, for multiple orders, so he went to the location, took out the amount needed and then used the tool to quickly be able to prepare them for shipping.

This bypassed the new methods to check if the location actually had stock to take, which I had just enabled for 1 account.

Luckily I caught the miss-hap as I was monitoring that product first-hand and noticed the batch of orders was collected but the stock amount didn't update.

It was 5min before I was leaving work, so I investigated and then ran to the person in question to ask what he did; which was "I used that tool"

I facepalmed myself internally while blaming myself, as he couldn't know that it wasn't ready to use for that purpose.

The tools to fix this up are there already.. so I used that to fix some missing stock-takes manually.. Though I'll need to close that little tool for these kind of orders for sure, asap, probably when I get home, at least until I bring over its new logic to it.

Happy Tuesday? (:

This 30 hour project is now 110 hours in and client is changing their mind so much that the managing director of the company is now sitting monitoring every correspondence between the client and developers.

#fml

So a friend / batch-mate in our accelerator asked me if I was okay with installing a monitoring software for a client our startups are collaborating for. And the said client was ranting how I've been appearing offline to him since morning...

Bitch I'm already letting you monitor my shit from morning to night, I don't need your French ass snooping around what I'm doing outside of office hours.

Fucker.

A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.

The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.

The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.

They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"

WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production?

I fucking hate foglight!
Fucking piece of over bloated shit software that can't do anything right!
This pile of harkonnen vomit is such a source of stress and frustration.
Can you believe that a monitoring tool, that needs to monitor 550 agents (host,dB's, webservices,whatever) needs a whopping 20+GB ram and 4vcpu's just to receive metrics and sent it to the main server, for just being a middleman???

Fucking piece of shit.

And worse of it all, due to some fuckery at mgmt-customer relationship level, they want to I stall MORE of this shit

Fucking shit

Quest software, never engage with them. You'll lose equal amounts of money as for dynatrace but be very bad off

Favourite thing you worked on recently?
——
I’m working on server monitoring system. I found that I need to pay for most server monitor services, and ones that are free/open source didn’t sound like they fit my use case (b/c the server i need to monitor is shared,) so I basically said “f*<k it I’ll do it myself.” I find it fun to work on something that I feel the need to make.