Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Big event. Massive traffic in production, so we were monitoring all night.
I was in a room with 2 devs of my team, a marketting girl, my boss and a designer... chilling.
Suddenly the production is down.
Boss: production is down, anyone can check?
Me: already on it
Dev1: it looks ok for me
Dev2: me too
Me: wait what? Impossible everything is down
Dev1: oh I refreshed the page it's not working
Me: don't stay on the page refreshing it like you are fucking monkeys. Give me useful intel or be quiet.
Market girl: is it working?
...
Guys is it working?
...
Hello?
Me: Not yet we are looking. Don't distract me.
Boss: client called us. They want it online now.
Dev1&2: he's looking
... 1 min later...
Boss: is it working?
Boss: is it working?
Boss: is it working?
Me: SHUT THE FUCK FOR FUCKING ONE SECOND. ALL OF YOU, OUT NOW. YOU ARE FUCKING MONKEYS WHO CAN'T DO SHIT. IF YOU CAN'T HELP JUST SHUT YOUR DAMN SHITHOLE. DEVS, LOOK WITH ME. MARKET GIRL PREPARE A FUCKING POST-MORTEM MAIL. BOSS GET THE CLIENT ON THE PHONE AND STALE. DO. YOUR. FUCKING. JOBS.
That's how I ended up screaming at everyone... the rest of the night went in complete silence and I fixed the issue 2min after the got quiet or busy.24 -
We have a lot of monitoring screens on walls at my work... Some of them show charts... The one showing the last two months of mailbox database growth has started taking an odd shape.
16 -
ON MONDAY
TL: Why the hell you require a month to integrate this engine?
Me: It will take that much time, can't help it.
TL: it can be done within a week.
Me: Then you do it.
TL: Ok I will show how it's done in a week.
ON FRIDAY
Me: What's the status on that integration?
TL: Oh yeah about that, you have to carry it ahead, I have some monitoring to do.
Me: Ok, give me the repository access, I will carry it ahead.
ME OPENS REPOSITORY.
There's only a new controller file with nothing it.5 -
The project where I realized I wanted to go from chemist to pro dev.
I built a flow-chemistry spectrometer with monitoring backend in Haskell.
Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.
I had to do that a lot on factory samples, writing down measurements using pen & paper.
I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.
I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).
One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.
The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.
Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).
Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...
But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.16 -
Mother of god, as if the new mass surveillance law in the Netherlands wasn't bad enough, one of the politicians who likes the new law has come up with an even more redicilous idea.
an 'Internet Authority. To put it short, an authority which surveils the internet in real time and sees where all social media shit is coming from/going.
Meaning that it wouldn't just be 'targeted mass surveillance' sometimes but fulltime online monitoring.
This guy has lost his fucking mind.35 -
So my landlord just came up and asked why I'm using so much bandwidth (they've just had a new line installed so they're monitoring it like hell for some reason) so we had a chat, I told him I'm a Web Developer so I'm uploading and downloading a load, and bare in mind this is student housing, he offered to install a wired connection in my flat only so I'll have a decent and stable connection when all the other students come back in September.
This is the first time in my life I feel like I'm not paying enough rent!7 -
Long rant ahead. Should take about 2-3 minutes to read. So feel free to refill your cup of coffee and take a seat :)
It turns out that the battery in my new Nexus 6P is almost dead. Well not that I didn't expect that, the seller even explicitly put that in the product page. But it got me thinking.. why? Lithium batteries are often good for some 10k charges, meaning that they could last almost 30 years when charged every day! They'd outlive an entire generation of people!
Then I took a look at the USB-C wall charger that Huawei delivered with this thing. A 5V 3A brick. When I saw that, I immediately realized.. aah, that's why this battery crapped out after a mere 2 years.
See, while batteries are often advertised as capable of several amps (like 7A with my LiitoKala 18650 batteries that I often use in projects), that's only the current that they can safely take or deliver without blowing up. The manufacturer doesn't make this current rating with longevity in mind. It's the absolute maximum in current that a given battery can safely handle.
The longevity on the other hand directly depends on the demand that's placed on the battery. 500mA which is standard USB 2.0 rating or 1A which is standard USB 3.0 rating, no sweat. The battery will live for at least a decade of daily charges and discharges like that no problem.
But when you start shoving 3A continuous into a battery, that's when it will suffer. Imagine that your current workload is 500mA and suddenly you get shoved 6 times that work upon you. How long would you last?
Oh and not only the current is a problem, I suspect that it also overvolts the battery to maintain a constant current all the way till the end. When I charged my lithium cells with my lab bench power supply, the battery would only take a few milliamps when it got close to the supply voltage. Quick bit of knowledge: lithium cells are charged at constant current first, then when the current drops below that, it continues at constant voltage - usually 4.2 or 4.35V depending on the battery. So you'd set your lab bench power supply at 4.2V 500mA. But in that constant voltage mode, as the battery's voltage and the supply's voltage equalize, the current drops because the voltage difference becomes lower. Remember, voltage is what causes current to flow. Overvolting at the supply to stay in constant current mode all the way till the end speeds this process up but can be dangerous and requires constant monitoring of the battery voltage.
So, why does Huawei and a bunch of other manufacturers make these 3A power chargers? Well first it's because consumer demands ever more, regardless of the fact that they can just charge at 500mA for the night (8h of sleep) and charge a 4000mAh battery from 0 to 100% no problem. Secondly it's because sometimes you need that little bit of extra juice fast, like when you forgot to plug the damn thing in and you've got only 30 minutes in the morning to pour some charge into it.
But people use those damn fucking things even when they go to bed, making that 3A torture a fucking standard process!! And then they complain that their batteries go to shit?!
Hopefully this now made you realize that the fast charger shouldn't be used as a regular charger ^^29 -
I accidently left log.debug("bollocks") ;
In an exception handler our customers log monitoring system picked it up and they questioned why and I quote here "why is there a spike in bollocks at 3am?"
That was an awkward conference call2 -
FBI: FUCK!! We just lost our connection to all those civilian routers we were monitoring
FBI IT Department: Did you try turning them off and back on?
FBI:
5 -
Navy story continued.
And continuing from the arp poisoning and boredom, I started scanning the network...
So I found plenty of WinXP computers, even some Win2k servers (I shit you not, the year was 201X) I decided to play around with merasploit a bit. I mean, this had to be a secure net, right?
Like hell it was.
Among the select douchebags I arp poisoned was a senior officer that had a VERY high idea for himself, and also believed he was tech-savvy. Now that, is a combination that is the red cloth for assholes like me. But I had to be more careful, as news of the network outage leaked, and rumours of "that guy" went amok, but because the whole sysadmin thing was on the shoulders of one guy, none could track it to me in explicit way. Not that i cared, actually, when I am pissed I act with all the subtleness of an atom bomb on steroids.
So, after some scanning and arp poisoning (changing the source MAC address this time) I said...
"Let's try this common exploit, it supposedly shouldn't work, there have been notifications about it, I've read them." Oh boy, was I in for a treat. 12 meterpreter sessions. FUCKING 12. The academy's online printer had no authentication, so I took the liberty of printing a few pages of ASCII jolly rogers (cute stuff, I know, but I was still in ITSec puberty) and decided to fuck around with the other PCs. One thing I found out is that some professors' PCs had the extreme password of 1234. Serious security, that was. Had I known earlier, I could have skipped a TON of pointless memorising...
Anyway, I was running amok the entire network, the sysad never had a chance on that, and he seemed preoccupied with EVERYTHING ELSE besides monitoring the net, like fixing (replacing) the keyboard for the commander's secretary, so...
BTW, most PCs had antivirus, but SO out of date that I didn't even need to encode the payload or do any other trick. An LDAP server was open, and the hashed admin password was the name of his wife. Go figure.
I looked at a WinXP laptop with a weird name, and fired my trusty ms08_067 on it. Passowrd: "aaw". I seriously thought that Ophcrack was broken, but I confirmed it. WTF? I started looking into the files... nothing too suspicious... wait a min, this guy is supposed to work, why his browser is showing porn?
Looking at the ""Deleted"" files (hah!) I fount a TON of documents with "SECRET" in them. Curious...
Decided to download everything, like the asshole I am, and restart his PC, AND to leave him with another desktop wallpaper and a text message. Thinking that he took the hint, I told the sysadmin about the vulnerable PCs and went to class...
In the middle of the class (I think it was anti-air warfare or anti-submarine warfare) the sysad burst through the door shouting "Stop it, that's the second-in-command's PC!".
Stunned silence. Even the professor (who was an officer). God, that was awkward. So, to make things MORE awkward (like the asshole I am) I burned every document to a DVD and the next day I took the sysad and went to the second-in-command of the academy.
Surprisingly he took the whole thing in quite the easygoing fashion. I half-expected court martial or at least a good yelling, but no. Anyway, after our conversation I cornered the sysad and barraged him with some tons of security holes, needed upgrades and settings etc. I still don't know if he managed to patch everything (I left him a detailed report) because, as I've written before, budget constraints in the military are the stuff of nightmares. Still, after that, oddly, most people wouldn't even talk to me.
God, that was a nice period of my life, not having to pretend to be interested about sports and TV shows. It would be almost like a story from highschool (if our highschool had such things as a network back then - yes, I am old).
Your stories?8 -
I wrote (or, ended up with a very much alpha but usable version) a monitoring system a while back and completely forgot about its existence.
But, it's still running and a few days ago I was building a docker imagine on a system with not that many resources and after about 5 minutes I started getting notifications about a high load!
Then, while I had forgotten about it again, yesterday, I suddenly started getting notifications about websites on my main application server going down.
Logged in and all was good again after restarting nginx.
Gotta say that it feels quite awesome to be notified of shit going wrong by something I wrote myself while I forgot about its existence 😊2 -
My uni implemented Bluetooth beacon based attendance monitoring.
Raspberry pi + cloned beacons = 100% attendance.
Idiots 😂
(Edit for clarity: app on smartphone, Bluetooth beacon in each room)29 -
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...7 -
"Another one bites the dust.." BEEP BEEP BEEP!
😬 "Shit, make that two."
Down from 5TB redundant to 3TB non-redundant and still waiting for my new Reds. No choice but to shut everything down and wait. 😩
11 -
Working on a (PHP based) monitoring system and currently writing rules/testing stuff.
I wrote some rules which check a few pages of a dutch site displaying if certain bigger services have disruptions and I am pulling the pages of a few ones I'd like to be notified about when they have issues.
Started the engine and received an alert about one big ISP over here from the monitoring system. Didn't believe that it would work right away so went to check that specific disruption page and...... they have a disruption right now!
IT FUCKING WORKS.
Good monitoring system 😊 *pats system*5 -
To improve our user's "experience" I suggested to my boss to add a status page showing...well, the current status of our services. Everybody was up for it, so I go off and implement a basic version + automated monitoring backend, get lots of positive feedback, all seems fine.
Then it starts:
Boss: "Can you get it all set up by this Saturday?"
Me: "Uh, today is Wednesday and I've never set up all the stuff needed on a proper server before"
Boss: "Well, you still have a few days. Please also contact your coworker to get it all hooked up in our launcher"
Me: "I'll try, can't make any promises though"
Contact my coworker and tell him what the plan is. I had already given him access to the repo and he is positive to get it all hooked up (I doubt he ever cloned my repo, let alone ran my code)
Spend all Friday getting my stuff set up on the production server, feeling pretty good thanks to the many tutorials.
Contact the boss Friday evening:
Me: "All up and running"
Boss: "Thanks, but we decided to go with a basic HTML page instead. We can just manually edit that, should be enough.
Me: "..."
In the end my stuff was never used, the server I set up was finally taken down a month ago. The gratitude you get when not hacking together some absolute shit that causes problems when you don't add <br/> tags at the correct places to prevent an ugly overflow, cause the coworker was too lazy to implement some form of line wrap in the launcher. I'm not saying my stuff is the best of the best, but at least it was professional looking to a certain extent.8 -
I'd love to finish a few projects I'm currently working on:
- An add-on which gives a middle finger to websites which use services/products ran by companies which are known to be integrated within the biggest mass surveillance system ever created (US powered). Not because just fuck those websites but because I think (@PonySlaystation came up with this idea) that its only fair that people get to know which websites 'sell them out'. Oh and "but not everyone cares about that" - you don't HAVE to install the addon.
(will be open sourced)
- Notes service with a fun thing.
- PHP based server/website/whateverthefuckyouwant monitoring system which is pretty much module based and works with json files as configuration. (kinda works but still loads of bugs to solve and gotta improve the module system a lot).
(will be open sourced)
- PHP based pihole alternative which suits my needs (will be open sourced)
- Forgot one 😅17 -
!rant
Quite the opposite of a Rant actually. Very good day today! Had pancakes to start the day, finally got a second monitor for only 5 bucks - it is so much easier to program with live change monitoring!
And, I finally got my very own server rack :) just wanted to share!
10 -
Tip for devs (esp front end):
Sublime text (and few other inferior editors) has a plugin called "transparency" that allows your editor to be transparent.
Windows 10 powershell consoles can be made transparent.
1) Open browser
2) Open command prompt over it with 75% transparency
3) Open editor over both with 50% transparency.
4) Set editor to fullscreen, no-distraction mode to center the text (Shift+F11 in sublime)
Enjoy coding while constantly viewing the code, the browser and monitoring the cmd prompt at the same time, without having to click Alt+Tab a zillion times.
18 -
> me on call
> had to much wine
> suddenly, phone starts making sound
> holy shit, I am on call and I am not supposed to drink
> Calls taxi because shit is looking bad at work. Everything look down according to the sms I get, I had to pay the taxi from my own pocket.
> 40 min later, arrive at work. Walk into the serverroom.
> Go to the the server that is monitoring everything.
> Check networkcable, it is loose. Push it a little.
> Goddamnit, that was it.
> Realize I never doublechecked if stuff really was down from home....5 -
I think in my case its a monitoring system I'm still writing which does already work (just far from done).
I got this as a crazy idea while thinking that it would be impossible to make and then thought fuck it and I wrote it.
Written in php, and as long as you can write a module for it, you can pretty much make it monitor anything you'd like.
I'm using it to monitor a few websites and servers I run :)5 -
Colleague started a slack channel for our team, management wanted nothing to do with it. We used it to work and have a bit of fun.
Some push / drive came form somewhere and now all the managers are on it. Yesterday I was told my screenshot and "snarky comment" are not appropriate for the workplace and to delete my slack message.
My comment was a joke about about a new app the company has to use "to increase efficiency" that broke and wouldn't let me do what I needed. It wasn't offensive, demeaning, sexist to anyone or even contain any bad language.
How petty and childish to be monitoring a private channel making sure everything is positive. We all joked that from now on our meme's must be about how awesome the company is and how much time we are saving on a daily basis.
God forbid we're allowed to speak honestly and openly or have a bit of fun.7 -
1. Fucking MySQL database clusters.
There's nothing fun about MySQL clusters. Sometimes they start producing deadlock errors for no apparent reason... well, there's probably a reason, but it's never a transparent easy to find reason.
What was even less fun is that those errors took down a Sentry server. When your error log server goes down through ddos from your database messages, it's time to rethink your setup.
2. Wiring up a large factory with $2 arduino clones, each with a $2 esp8266 wifi chip, with various sensors for measuring flow of chemical solutions (I wanted cheap real time monitoring as an early warning system next to periodic sampling).
The scaling issue was getting over 500 streaming wifi signals to work in a 55c moist slightly corrosive atmosphere with concrete and steel everywhere, and getting it all into a single InfluxDB instance for analysis.12 -
So I'm moving to a new/bigger place with faster Internet soon so I think it's time to rebuild my current home/remote server setup.
I want to setup the following things:
- vps for server monitoring (open source pushover alternative + netdata)
- Zero tier network for connecting all my servers to the same network
- pihole/pivpn (or the Angristan vpn installer, look it up :)
- second blocking thingy next to pihole to make sure that I literally can't access google/fb etc anymore, even if I really needed/wanted to
- bunch of general servers.
Any ideas?21 -
DevOps required skillset:
* Frontend engineering
* Backend services
* Database administrator
* Security consultant
* Project management
* 3rd party contract negotiator
* Build system monitor
* Build system hostage negotiator
* Paging, alerting, monitoring
* Search server admin
* Old search server admin
* Old-old-new search server admin
* Redis, ElasticSearch, MySQL, PostGres, owner
* Agile coach
* No you shouldn't do that coach
* Oh, you did that anyway coach
* DNS: (Optional) It'll replicate when it wants, and how it wants to to anyway
* Multi-Cloud deployment strategist
* Must be able to translate Klingon to YAML, and YAML to MySQL
* Cost analyzer, reducer, and justifier
* Complex documentation generation in markdown that we should have done years ago anyway
* Marketing's email went to spam analyzer
* Wordpress is broke fixer
* Where the fuck does Wordpress run anyway?
* Ability to fix MySql running Wordpress on marketing's dusty laptop7 -
Advice to all new programmers, take this one from personal experience. DO NOT PUT SWEAR WORDS IN DEBUG STATEMENTS.
You will miss one, it will go to production and it will get picked up by your log monitoring...2 -
HR: What was the last project you handled?
Me: I worked on an internal system for my current company. It is basically an interdepartmental monitoring system.
HR: Ohh. Good. Do you have a copy of it? Can you show me how it works?
Me: ......5 -
This startup I started working for with their shitty code base written by interns, restrictive sys admin who had no actual use in the company since I was the one setting up their servers, know-it-all CEO, stupid HR representative who used to grill employees for being 10 minutes late in the morning, very small apartment "HQ", using fingerprints to signal our entry and our leave to and from the office, no formal process, and, to top it all, monitoring our own laptops which we use for work with a software that takes screenshots every few minutes. In short, it had the worst in corporates with the worst of startups combined in one company.
If, hypothetically, we could overlook all this, I couldn't overlook the horrible smell this place had. The apartment was overlooking a small garden which was a home for many stray cats and dogs. You can imagine how horrible this smell was. The weird thing was that no one there seemed to really care about the smell!!
I lasted there for only one week before I gave my resignation and I believe I had every right to do so.3 -
Paranoid Developers - It's a long one
Backstory: I was a freelance web developer when I managed to land a place on a cyber security program with who I consider to be the world leaders in the field (details deliberately withheld; who's paranoid now?). Other than the basic security practices of web dev, my experience with Cyber was limited to the OU introduction course, so I was wholly unprepared for the level of, occasionally hysterical, paranoia that my fellow cohort seemed to perpetually live in. The following is a collection of stories from several of these people, because if I only wrote about one they would accuse me of providing too much data allowing an attacker to aggregate and steal their identity. They do use devrant so if you're reading this, know that I love you and that something is wrong with you.
That time when...
He wrote a social media network with end-to-end encryption before it was cool.
He wrote custom 64kb encryption for his academic HDD.
He removed the 3 HDD from his desktop and stored them in a safe, whenever he left the house.
He set up a pfsense virtualbox with a firewall policy to block the port the student monitoring software used (effectively rendering it useless and definitely in breach of the IT policy).
He used only hashes of passwords as passwords (which isn't actually good).
He kept a drill on the desk ready to destroy his HDD at a moments notice.
He started developing a device to drill through his HDD when he pushed a button. May or may not have finished it.
He set up a new email account for each individual online service.
He hosted a website from his own home server so he didn't have to host the files elsewhere (which is just awful for home network security).
He unplugged the home router and began scanning his devices and manually searching through the process list when his music stopped playing on the laptop several times (turns out he had a wobbly spacebar and the shaking washing machine provided enough jittering for a button press).
He brought his own privacy screen to work (remember, this is a security place, with like background checks and all sorts).
He gave his C programming coursework (a simple messaging program) 2048 bit encryption, which was not required.
He wrote a custom encryption for his other C programming coursework as well as writing out the enigma encryption because there was no library, again not required.
He bought a burner phone to visit the capital city.
He bought a burner phone whenever he left his hometown come to think of it.
He bought a smartphone online, wiped it and installed new firmware (it was Chinese; I'm not saying anything about the Chinese, you're the one thinking it).
He bought a smartphone and installed Kali Linux NetHunter so he could test WiFi networks he connected to before using them on his personal device.
(You might be noticing it's all he's. Maybe it is, maybe it isn't).
He ate a sim card.
He brought a balaclava to pentesting training (it was pretty meme).
He printed out his source code as a manual read-only method.
He made a rule on his academic email to block incoming mail from the academic body (to be fair this is a good spam policy).
He withdraws money from a different cashpoint everytime to avoid patterns in his behaviour (the irony).
He reported someone for hacking the centre's network when they built their own website for practice using XAMMP.
I'm going to stop there. I could tell you so many more stories about these guys, some about them being paranoid and some about the stupid antics Cyber Security and Information Assurance students get up to. Well done for making it this far. Hope you enjoyed it.
26 -
Trying to setup server monitoring for one server (will deploy it on all my servers once this works well).
Getting these email notifs to work is a bitch 😅
(can't do pushover etc since those require GCM and I've got Google blocked out of my phone)20 -
Visual Studio Code.
I've tried you because of hearing a lot of good stuff about you. I'd switch back to netbeans regardless because I love netbeans and I always try to use as little as possible from companies like Google/microsoft/facebook (and others) but what you're pulling right now is un-fucking-believable.
I've disabled ANY AND EVERY form of calling home I could (find) in your settings. Crash reports, automatic updates, metrics, you name it. I've searched all the fucking settings but I can't find any other home calling thing that's enabled and yet:
I'm monitoring every goddamn DNS request (through my own DNS server) and I'm still seeing calls to a Microsoft owned domain. Closed all my browser sessions and you as well and it stopped. Started browser again but not you, nothing.
Started you again: BAM. Calls to that damned Microsoft owned domain again.
If you can't honour my decision for disabling any form of home calls, go fuck yourself.
Netbeans, I'm back, I've missed you 💜35 -
One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.
On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).
Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.
I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".
Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.
There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.
Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).26 -
I just cleared out 48,158 monitoring emails from work. These are all automated emails received between mid May and end of November. Outlook is now pouting in a corner and not responding, but I'm not done cleaning up yet...8
-
Let me tell you a story.
Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.
With this many false positives, we quickly learn to ignore the alerts and miss real incidents.
So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!
It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.
To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.
As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!7 -
Have been trying to setup Netdata as a monitoring system for a while now and finally got it working!
Instead of the built-in webhooks I just did a curl to a url containing a php page/file which error logs the status and description (just for testing).
It took me way too long to get it to work but BAM.
Immediately made a new cpu load rule (one minute high load):
The satisfaction of getting an error message in the php logs containing my custom rule as warning and a minute later as critical 😍
Netdata ❤6 -
Here's the story of my first month at CERN :) But first, a little premise...
Before arriving, I expected to be scared, alone and unguided in most of my experiences: after all I was a simple 19 year old about to leave home and friends for 3 years heading out in the world with zero experience on stuff like banking, taxes.. let alone working in a huge environment! The impostor syndrome was at an all time high on that front.
Then, I had the luck and pleasure to find an extremely competent and helpful plethora of people, ranging from my team to other CERNies (yes, that how we're called :P) who took me under their wing and introduced me to all the key aspects of living the place. When the initial stress finally soothed down thanks to this, I finally started to manage focusing more and more on my work, by following day-by-day my teammates who taught me the core aspects of the system and the many projects that are in progress during Long Shutdown 2. Within a couple weeks, I already managed to grasp various concepts that got me quickly on track, and now I managed to develop and integrate new temperature monitoring scripts into a system checking on hundreds of Single Board Computer-based servers :) It's a real rollercoaster of learning and applying under all fronts and so far I'm not regretting my choice of departing.
Luckily I've also discovered I'm pretty efficient and good at my job, which surely boosts my morale :D
Keep you updated as usual!11 -
PSA: if, for whatever shit reason your brain comes up with, you decide to run a webminer in your retarded useless piece of shit website, at least HAVE THE DECENCY TO WARN USERS ABOUT IT. And while you're at it, implement some basic monitoring and safety functions. If you don't, you can set yourself on fire and jump from the top of the tallest building you can find.
Some basic tips:
1) don't run that shit on phones. The fraction of a fraction of a cent you're gonna earn from them is not worth the risk of overheating them and draining their batteries.
2) add low battery/overheating protection: the last thing you want to do is kill some poor sucker's laptop (and potential unsaved work) just because they forgot a tab open. Every time a laptop dies because of you, a knife will slit your throat.
3) WARN YOUR USERS ABOUT IT! You are straining someone else's resources for your own profit: at least have the balls to be open about it. If you try to run a miner silently in the background, I will make you eat whatever is left of your fucking brain, then drown you in the shit that comes out of your ass.5 -
Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.
Worked today… At Saturday.
Some administrators forgot to properly check bandwidth limitations....
*rolls eyes*
We had a major version upgrade of some server software at Monday.
Guess why I got called...
Of course it MUST be the software upgrade.
It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.
*eyes roll inside till only white is visible*
The even more annoying thing is that it wasn't that hard to figure out.
Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.
IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...
*additionally to white eyes starts cursing in reverse latin*
Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.
Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...
NOTICE SOMETHING?!
Incompetent admin: It starts at the same time. Always.
After wasting roughly another hour of time discussing with him, I just hanged up the video call.
Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…
*turning into antichrist*
I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.
Sometimes this company and HR especially notoriously drive me insane...
Guess next week there will be some HR barbecue.
But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.
Roughly 4 hours of analysis, communication, raging and hatred.
Just one hour implementing shit.
*goozfraba*11 -
An important message:
PrOpErLy managing servers is HARD.
I get pissed off at customers with ZERO server knowledge who think they can manage their VPS. “Just get a control panel and a VPS” from some flashy provider that makes server management look way too easy.. Clicking around in their fancy control panel, until:
- they need help with their *self-managed* VPS;
- their email ends up in spam;
- they suffer from performance issues;
- they need to restore a backup;
- something breaks, because YES, things break
Way too little people are able to answer:
- when and how do you make backups?
- how do you monitor your servers and which services?
- how do you keep track of trend analysis?
Then I come by with necessary software. SNMP for trend analysis, Graphite for infrastructure health, Sensu for monitoring, Kibana, Ansible for configuration management..
Things that servers need but that customers have never even heard of.. because they can do everything in their control panel..
Until they come crying to me because it broke and they don’t even know how to get into SSH.
I think the ones to blame are VPS providers that tell the tale of how easy it is to install a control panel and never look at your server again.
Customers become responsible for something *business-critical*! Yet they don’t know how it works.6 -
> Monitoring: Load Average of 57!! ALERT!!!!
> me: What? That's not possible?
> *Monitoring froze 14 hours ago*
> *sshs into server*
> *see attached image*
The issue was ~1200 df processes that were issued by our monitoring system and all of them didn't finish because the external cluster we mounted onto that server died a few minutes before that. Just re-mounting the cluster fixed it but still a funny sight!
24 -
Other PM: We must fix the database performance issues now.
Me: We can't. We're still only halfway on the dependency chain to tackle this and honestly, even if the dependency chain would be fulfilled, I'd leave at least 2 weeks monitoring the production after the changes were rolled out before we further poke around.
Other PM: This is taking far too long. And whaddya mean by dependency chain? Why was I not informed about this?
Me: *sigh* like in every meeting in the last weeks: the dependency chain are the current open blockers before we can proceed with the database changes. We've talked about this _at length_... Especially why these blockers exist.
Other PM: No, we need to start now. I've _examined_ at the blockers or "dependency chain" as you call it.
(Examined.... He opened on his currently streaming laptop, which was connected to the active beamer, the mentioned ticket with a detailed blocker ... And quickly scrolled. Yeeah. Warmonger...).
Me: I'm very tired of discussing this. But since you are already presenting us the ticket, read out the referenced meeting notes... We explained it in great detail.
Other PM: Why? This is just a waste of time!!!!!!!
--
Yes. This happened. Other PM was my nemesis.
In this meeting were 2 PMs (Him, Me)… I think 5 - 7 devs... And we were sitting in this meeting since 2 hours at least. Everyone was angry...
After this "manifesto of intelligence"… I simply left the room, followed by a few devs.
And yes. Other PM did this on a regular basis....5 -
Got an email earlier this week. It went something like this:
"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"
Um, wait, excuse me, WHAT?
1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.
I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.2 -
I suddenly realized all the technical debt shit I told my boss would happen years ago given the way things were done/heading then... Just occurred pretty much all at once last week in the form of critical production issues...
The teams like:
-we need real time server process monitoring
-structured logging for apps
-containerization so one app didn't affect others
Me thinking: yes.... I told you so like 3/4 years ago when I first joined the team and kept repeating so much I got tired of saying at every annual review...
This is exactly what happens when you let technical debt grow and have no free time for developers to look into and fix then while they were small and not critical production processes... Or properly document and peer review them... (Got a shit pile of projects that no one knows how to use or even exists because the devs left the team) and they'll have a lot more when I finally leave... Hopefully this year.... If I can find another role and not need another medical procedure... (Doubtful)3 -
Rain monitoring/alert api's/services/websites/apps.
Why the fuck are those fuckers (in my case) nearly always right when I couldn't care less and very fucking wrong when I couldn't care more?
Today, morning before leaving:
Nah mate you're good for the next x minutes (more than enough time for me to bike to the station and get on the train).
Reality: suddenly getting two gallons of water a second on me in the form of rain, arrived completely soaked and headphones broke.
Before biking home from the station;
*its raining hard as hell*
Rain alert/monitoring app/radar thingy: nah mate no worries, no rain last half hour and you're good for another half hour!
Reality: arrived home soaked a-fucking-gain.
So motherfucking annoying and frustrating. And yes, I do have an umbrella and such but I hate carrying that stuff around when 'not needed'.5 -
Managed a 97% reduction in bandwidth usage for our internal host monitoring tool by converting the dashboard from using AJAX polling to websocket events.
Completely unnecessary but wanted an excuse to do some development with websockets. (:10 -
Just in case nobody has mentioned this yet:
Yes Microsoft I do have a dualhead setup.
Yes Microsoft I do want to watch video on my left screen while having window focus on something on the right monitor.
No Microsoft this doesn't mean that I *lost* focus on the left window.
No Microsoft this doesn't mean that I want your Movies and TV application to suddenly minimize (and continue playing anyway) while I focus on some server monitoring window on my right display.
Microsoft, there exist people that use more than your average user with a single C: drive that play Candy Crush on Facebook all day. And the limitations that you currently impose might very well be what keeps the Microsoft UWP applications from getting adopted. Because you know what? SMPlayer, a default application in any of my Linux workstation machines, it does handle such window transitions just fine!
Microsoft, I love how you at least gave us the option to enable Ctrl-Shift-C and Ctrl-Shift-V in WSL and conhost in general over that abomination that is Right-click and Return (those are so random!) that are relics from CP/M. But seriously? At this rate, I'd definitely not call it usable for anyone but those with a single monitor yet.
So please _/\_13 -
Walk into mall, bring my child to kids zone. They have free wifi 🤤
Accessing -> Scanning -> Got Ya! -> Brute Force -> Cracked!
There you're.
Now I can monitoring my kid while reading rant!
Just normal day in my life
6 -
So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?9
-
Recently applied at a local company. Webform, "enter some details and we'll get back to you"-like.
Entered my details, hit submit, lo and behold "Error 503 - something went wrong on our end".
I was just baffled. It's a well-established IT company and they can't even get their application form to work?
So I'm sitting there in the debugger console, monitoring network stuff to see if anything is weird. I obviously hit submit some several more times during that.
Eventually I give up.
In the night my phone wakes me up with a shitton of "we've received your application and will review it..." emails.
Yeah they didn't get back to me.2 -
I was up until 3AM working on devRant bot adding new features and improving stability.
https://github.com/nblackburn/...
After some very close monitoring, i am happy to relaunch it for you guys to enjoy.
6 -
So this one made me create an account on here...
At work, there's a feature of our application that allows the user to design something (keeping it vague on purpose) and to request a 3D render of their creation.
Working with dynamically positioned objects, textures and such, errors are bound to happen. That's why we implemented a bug report feature.
We have a small team tasked with monitoring the bug reports and taking action upon it, either by fixing a 3D scene, or raising the issue to the dev team.
The other day, a member of that team told me (since I'm part of the dev team) he had received a complain that the image a user received was empty. Strange, we didn't update the code in a while.
So I check the server, all the docker containers are running fine, the code is fine, no errors anywhere.
Then, as I'm scratching my head, that guy comes back to me and says "I don't know if it can help you, but it's been doing it for a week and a half now".
"And we're only hearing about it now?!", I replied.
"Well, I have bug reports going back to the 15th, but we haven't been checking the reports for a while now since everything was fine", he says as if it was actually a normal thing to say.
"How can you know everything is fine if you're not looking at the thing that says if there's an issue?!", I replied with a face filled with despair.
"Well we didn't receive any new reports in a while, so we just stopped looking. And now the report tool window is actually closed on my machine", he says with a smile and a little laugh in his tone.
In the end, I got to fix the server issue quite easily. But still, the feature wasn't working for 1.5 weeks and more that 330 images weren't sent properly...
So yeah, Doctor, the patient's heart is beating again! Let's unplug the monitor, it should be fine.
Welcome to my little piece of hell :)7 -
Things you don't want to see at night
Ripped out of Netflix-Mode by a Warning notification and currently monitoring further development
Green line is temperature, blue humidity. Temperature rises at ~1°/10min, but seems to flatten just now. ~0.6°C to go and I'll have to head out. I'm thinking one of the ACs failed, but states are fine. Never trust a single information source for critical infrastructure guys
15 -
My work computer is so fucking laggy that even simple typing is often infuriating, and sometimes bloody impossible.
My input gets garbled, or it takes so long that I finish typing and have to wait for it to show up. And sometimes. Sometimes it’s just. I don’t have words. I just typed “merchants”, saw a giant spam of “eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee[...]” and then saw it replaced with “etsmhcaern”. What. The. 🦆.17 -
After 3 years of being the first in and last to leave, of getting other people's work reassigned to me - P can't complete it on time, G doesn't like the user, A refuses to work on that module, etc... I finally blew last Sept.
In the span of 2 days, my boss brought me into a project 1.5 years in (she doesn't trust P to do the coding) and expected me to be up to speed and coding in a couple of days, told the functional dept that I would cover for one of their guys on vaca for three weeks and assigned me to take over a HUGE project from one of the other functional guys who wasn't getting it done. So basically I'm now doing Ps job AND supporting another department AND taking control of a large project from another department. I'm the idiot working 14 hour days while they're all leaving on time or enjoying their 3 week vaca to India.
I lost it. It's bad enough filling in the gaps in my own department but when I'm now taking on work for other departments, that's where I draw the line. I sent my boss my resignation - just could not take the inequity in the work load.
I'm still working here - my boss ended up hiring a consultant to handle the functional project and told the functional group to find their own vacation coverage. She's also monitoring workloads much closer now. I still habe an ongoing issue with having to complete other peoples work for them but I'm not working OT to do it. So speaking up helps. So does quitting.2 -
Story time!
I worked at a company that was the HQ for a sizable organization for a while, until it was eventually bought out by another company, and then yet another company who was located in the valley.
We were kinda a forgotten office not being the HQ, like most places like that are.
No customers EVER visited our building, few if any people knew we existed even, even our own company. I visited HQ in the valley on a number of occasions and was stalked by the video monitoring system for hours before I was stopped by security and the cops called because nobody believed there as an office outside the valley when I explained why my badge looked different .... (San Jose cops were very nice about it and really pissed at the security team.) But that's another story...
One day people who were never at our office decided (after many meetings without talking to anyone at the office) ... they decided the beige walls at our office didn't match the company colors.
So they took all the generic wall coverings down and painted all the walls an almost imperceptible different color.
So now we had an office with all white(ish) walls and nothing on them. Due to the configuration of the building there were these huge monolithic white walls that looked pretty dumb.
This lasted quite a while so as a joke I printed up and framed (found an old frame, as a former HQ we had lots of stuff lying around) a sign that said:
"This space intentionally left blank."
When the "mediocre hotel room quality art" and posters were scheduled to go up the folks putting the art up skipped that wall thinking the sign was official.
Even the somewhat corporate drone directors, and one VP at our office thought it was so funny, they didn't say a word about it. Word has it back at HQ they assumed it "must be fire code or something" and told the folks hanging the crappy art to skip that wall.
It lasted on that wall for a decade until we moved out of that building. On the last day, everything was moved, but that sign remained. No idea if it is still there or not...1 -
Spent a month working on a website that relied on crawled data
Got the memory leaks and usage down from 700mb to ~150mb
CPU usage from ~100% to <5%
Shrink-wrapped the DB requirements based on data
Created self-supporting services and what not
When everything FINALLY worked good enough for me to look at it and go "damn, this actually worked"
the whole monitoring sys got dyed in red :v
A quick look up and my crawlers exhausted my godaddy's per-user db limits.
Kill me.
Just fuckin kill me.7 -
A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.
Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"
Edit: the customer's name isn't actually Suzan4 -
I should never have had porn while monitoring servers. I've just pasted a xvideos link into terminal 😶
Damn, I just hope there is a command to clear the terminal history11 -
Imagine: It's the year 4249.
Corporate has finally managed to convince workers that they don't need a salary.
Workers are now paid with food, shelter and clothes. And it's only in effect if you achieve your deadlines.
Keystroke monitoring softwares are now replaced with Webcam eye tracking software.
GitHub Co-Pilot now takes over your code editor and tries to dictate you how to write better code.
Refusing to do results in a signal sent to the management about your behaviour and you lose food access for the day.
HR Recruiters now require you to give them a blood sample and part of your house as a security deposit.
They also require you to have a micro-chip placed in your brain so they can monitor their worker's thought process.
Switching a job is no longer an option. You pledge allegiance to one company your entire career.
You can never see the real world now because the government has mandated you to never take off your VR glasses.
You see the world the way the government wants you to see it.
PHP is still trash.
Life is Good.11 -
A ticket got escalated through 3 levels of techs. I open the escalation email, then do a Reply-all and ask one simple question:
Is the client really asking why there are gaps in monitoring when their servers are shutdown?1 -
Oh look. The monitoring channel is in flames, smartphone is vibrating so hard it's having a seizure.
Hm. Nah it's fine. Not my...
Damn it. Incoming call. -.-
I'm actually on vacation (more like you need to trim down overtime before management get's angry).
They decided to test the new hardware / os stack I set up in the last weeks. I'd actually be happy about it If I wasn't on vacation and would be part in something that I invested a lot of time...
Well now I am. Guess what. It's running too good.
And that's not a joke. It's partly due to an upgrade in infrastructure (got rid of some last remaining 1 Gbps networks)… but also because I changed quite a lot on the OS / VM side plus we changed from XEN to Proxmox... With major tweaks, too.
The whole stack can now handle peak traffic where it would choke before, and even go beyond the old peak traffic.
Enough of introduction, the simple reason why shit burned down was because they tried out the current development branch and let it ran.
The development branch had an currently unfinished ratelimiter framework, since I didn't had time for an full burn in and didn't knew what the maxima / limits were. And since I hadn't finished that, I didn't finish the traffic shaping either.
Hm. Guess it's not good when you let a bunch of heavy parallelized data generators / analyzers run for free....
In the end, we simply shotgunned the docker development machines, because thanks to network congestion / retransmissions and feedback, they were not really cooperative via network / REST.
But hey: To infinity and beyond. XD2 -
I call this one the tester than knew too much.
Note: The server the tester is running on has a hard drive that is breaking down...
Tester: Remember the error I talked to you about yesterday?
Me: Yeah, what about it?
Tester: Well the server hasn't recovered yet and I haven't restarted anything...
Me: Well the application itself hasn't crashed so our monitoring application doesn't seem to notice that the service is in a bad state. The error seems only to have brought down certain threads within the application.
Tester: No, I think there is a different issue here and has nothing to do with that error, the application is still doing things.
*tails the log*
Tester: See?
Me: As I said some things are still running and are unaffected by the error.
Tester: NO! It has to be caused by the other error I had a week ago where my file got corrupted. As we said I removed the file, restarted it and it worked again, but had the same problem a day later...
Note: The problem is not related, this time the application is running out of file descriptors
Me: Well... If the problem is the same it would have complained about the file descriptors then aswell, not an I/O error.
Tester: Nope, I think you are wrong!
Me: ¯\_(ツ)_/¯
FML
3 -
I saw this picture and joked that the guy in the picture is in my pocket? Then I realized the guy is the NSA agent monitoring me. Ouch! Jokes on me...
9 -
The Return of Mr. Gitmaster:
So there is this colleague I already ranted about several times. After my previous team lead had confronted him about not doing much work, there was some irritation because he showed not up at work, but it turned out the external training he did was just a week earlier. Then he was ill a week, another week vacation so we didn't see him much. Not that his pre- or absence makes much difference to our repo: When his and my team lead looked at his commits of the past three months they found like the one copy-pasted HTML-form that wouldn't even show.
Fast forward to now, where we have a new team lead and we were going to lunch with Mr. gitmaster. So we got some more hero stories from the great work he was doing in the previous company. How he was graphically monitoring the heap fragmentation that stupid glibc was causing to their search engine, and how much better it became with tcmalloc.
I still don't understand how he bridges that cognitive dissonance from all the superior tech knowledge he displays to not actually writing any code at all. Not that I would not have experienced some states of feeling low, in paralysis unable to write a single line of code... but he seems so full of confidence, always commenting how trivial and easy all these tasks would be, as if it's all so lightyears below his abilities. Maybe he should just become a manager - but not mine. -
OK I've just got an idea that I think would be quite neat:
How about a virtual rubber duck that sits in the corner of your editor? Just like the gem in old Word, if you remember. It's yellow and quacks sometimes, and nods understandingly when you talk to it (mic monitoring).
And it also monitors your typing and says (popup text bubble) things like:
"those parentheses doesn't look balanced to me"
"did you really initialize that variable?"
"you wrote JASON again"
"you forgot the ;"
You get the point.
I don't have time to implement, feel free to steal my idea and become a millionaire.5 -
By all means I've learned most of the cool stuff with a lil project me and my bro did. It was a platform for bot farm of one mmorpg. It had it all: schedules, profiles, bot groups, monitoring, analytical module [was still a wip], auto-profiling to fight antibot [sort of ML - it was analyzing patterns after our bots got banned and attempted to change our bots behaviour], etc.
Eventualy we came up to conclusion that a library we used for botting [the mocked interaction with an actor] was flawed. It seemed that its authors had a contract or smth with the game authors to reveal which actors are bots. We dropped the whole thing as rewriting the lib would be too big of a waste of time :\7 -
Hahaha, well would you Imagine my shock?
https://pcmag.com/news/357465/...
But then again people enable this personal hell to happen
5 -
Here is my home setup (I mainly work at home)
Left monitor is for my windows machine, right monitor swaps between my laptop and my PC (depends if I am working or not) laptop running Manjaro with i3 and the tablet on the whiteboard will be used for some monitoring in the future
Don't mind the terrible cable management behind the table :)
2 -
Question to our Tor people.
I operate a middle relay myself and I noticed that Nyx (tor monitoring tool) displays a very different throughput (mb/GB a day) than tor itself.
How does that work?6 -
In college when we had programming labs where we had to use the schools unix server to compile and run.
My professor was very bad at explaining what actually needed to be done in the labs to the point where even the TAs didn't know what to do.
We were suppose to write an application in C to find out by "trial and error" how large we could make an array (or something like that, it's been too long). This not being explained well and no one knowing that much about C, I wrote a loop that just kept growing an array until it couldn't anymore. I watched it consume 72GB or memory from the servers before quitting the loop and realizing with the TA what the professor really meant.
I now feel bad for the IT staff monitoring the system wondering where 72GB just went...2 -
My department bought a new monitoring monitor because the old one was broken.
Actually it wasn't, someone just unplugged the power cabel. *facepalm*3 -
>be me, working at IBM as CC operator
>onboarding freeze, people leaving team, not enough operators
>take extra workload to sustain monitoring
>no raise
>team gets merged with other CC team(different customers)
>take over of developing full workload automation project
>no raise
>sick coworker, have to take more extra workload to cover monitoring
>get tiny raise
>coworker gets the same raise for only one extra workload
>be expected to do both programming and monitoring for the little salary
>too autistic to quit
>too autistic to confront my mamager with this
What do, devRant?5 -
WTF IS SUPPORT FOR?
A: Can you help fix X?
B: Can you help Y?
C: I see Z just crashes, fix it.
Me (in my mind): Can you tell me WTF is the actual problem and what investigation you have done?
Why the fuck are we paying you monkeys?
My company should just pay me all your salaries, and I will write a monitoring system to send out these types of "alerts" in a month.4 -
When I began my sandwich course in a big French company, I was dreaming about cutting edge stack, rocket computer and stuff...
I was disappointed when I came to my office with an old Windows 7 computer, coding via LANDesk to an old server with Windows Server 2008 on it, with Eclipse ... INDIGO...
I have to use Java 1.7 ...
Tomcat 7.
PRTG for monitoring...
Microsoft SQL Server 2008 ...
One screen...
Coding on a codebase where, indubitably, MVC pattern was just a weird thing in books.
No UT.
Lasagna code.
Well it really disappointed me.
Luckily, the Information Service was very open minded and gave me a laptop with Fedora, 3 screens, updated the servers, and let me update the stack, with Java 10, Angular for the front, they are okay for using Docker.
So ... even if it seems to be fucked up, there’s still hope !!3 -
Best: 100% of my contracts have resulted in extensions and permanent roles offered, after worrying I wasn't good enough to try contracting.
Worst: Used the wrong set of monitoring when doing my first deployment at a contract and thought what I had deployed was working fine. It wasn't. For 24 hours. Cost the company a lot of money. (why did they offer me an extension again?) -
Next week I'm starting a new job and I kinda wanted to give you guys an insight into my dev career over the last four years. Hopefully it can give some people some insight into how a career can grow unexpectedly.
While I was finishing up my studies (AI) I decided to talk to one of these recruiters and see what kind of jobs I could get as soon as I would be done. The recruiter immediately found this job with a Java consultancy company that also had a training aspect on the side (four hours of training a week).
In this job I learned a lot about many things. I learned about Spring framework, clean code, cloud deployment, build pipelines, Microservices, message brokers and lots more.
As this was a consultancy company, I was placed at different companies. During my time here I worked on two different projects.
The first was a Microservices project about road traffic data. The company was a mess, and I learned a lot about company politics. I think I never saw anything I built really released in my 16 months there.
I also had to drive 200km every day for this job, which just killed me. And after far too long I was finally moved to the second company, which was much closer.
The second company was a fintech startup funded by a bank. Everything was so much better than the traffic company. There was a very structured release schedule, with a pretty okay scrum implementation. Every team had their own development environment on aws which worked amazingly. I had a lot of fun at this job, with many cool colleagues. And all the smart people around me taught me even more about everything related to working in software engineering.
I quit my job at the consultancy company, and with that at the fintech place, because I got an opportunity I couldn't refuse. My brother was working for Jordan Belfort, the Wolf of Wallstreet, and he said they needed a developer to build a learning platform. So I packed my bags and flew to LA.
The office was just a villa on the beach, next to Jordan's house. The company was quite small and there were actually no real developers. There was a guy who claimed to be the cto of the company, but he actually only knew how to do WordPress and no one had named him cto, which was very interesting.
So I sat down with Jordan and we talked about the platform he wanted to build. I explained how the things he wanted would eventually not be able with WordPress and we needed to really start building software and become a software development company. He agreed and I was set to designing a first iteration of the platform.
Before I knew it I was building the platform part by part, adding features everywhere, setting up analytics, setting up payment flows, monitoring, connecting to Salesforce, setting up build pipelines and setting up the whole aws environment. I had to do everything from frontend to the backest of backends. Luckily I could grow my team a tiny bit after a while, until we were with four. But the other three were still very junior, so I also got the task of training them next to developing.
Still I learned a lot and there's so much more to tell about my time at this company, but let's move forward a bit.
Eventually I had to go back to the Netherlands because of reasons. I still worked a bit for them from over here, but the fun of it was gone without my colleagues around me, so I quit last September.
I noticed I was all burned out, had worked far too much, so I decided to take a few months off and figure out what I wanted to do with my life. I even wondered whether I wanted to stay in programming.
Fast forward to last few weeks. I figured out I actually did want to work in software still, but now I would focus on getting the right working circumstances. No more driving 3 hours every day, no more working 12 hours every day. Just work close to home and find a company with the right values.
So I started sending out resumes and I gave one recruiter the chance to arrange some interviews too. I spoke to 7 companies in the span of one week. And they were all very interested. Eventually I narrowed it down to 2 companies and asked them for offers. And the company that actually had my preference offered me significantly more than I asked for, which settled the deal.
So tomorrow I'm officially signing with them, and starting next week I'll be developing in Kotlin, diving into functional programming and running our code in serverless environments. I'm very excited! -
I just found out there's a 418 HTTP status code that stands for "I'm a teapot", specified by RFC2324 which "describes HTCPCP, a protocol for controlling, monitoring, and diagnosing coffee pots". I know it's an april fools joke but I still find it hilarious that there is an RFC for that.9
-
Crazy... Hm, that could qualify for a *lot*.
Craziest. Probably misusage or rather "brain damaged" knowledge about HTTP.
I've seen a lot of wild things when devs start poking standards, but the tip of the iceberg was someone trying to use UTF-8 in headers...
You might have guessed it - German umlauts. :(
Coz yeah. Fucktard loved writing everything in german, so why not write custom header names in german.
The fun thing is: It *can* work, though the usual sane thing is to keep it in ASCII range for the obvious reason that using UTF-8 (or ISO-8859-1, which is *not* ASCII) is a gamble you gonna loose.
The fun game was that after putting in a much needed load balancer between services for monitoring / scaling etc suddenly *something* seemed off.
It took me 2 days and a lot of Wireshark hoola hooping to find out why, cause the header was used for device detection aka wether it's a bot or not. Or in the german term the dev used: "Geräte-Art".
As the fallback was to assume a bot, but only rate limit based on IP, only few managed to achieve the necessary rate limit to get blocked.
So when I say *something* seemed off, I really mean a spooky kind of "sometimes IP blocked for seemingly no reason at all".
Fun stuff. The dev btw germanized everything. Untangling the code base was a lot of non fun. -.-6 -
Week 1 of the new job, and it seems I have some pretty low expectations to meet.
Seriously, I just did the math. Let's say one works an average of 48 hours per week, 50 weeks per year. Just as an average. That's 2400 hours in a year.
In the Micro-scale, a manager would mess up their team once every 2.4 hours (2h24m) or about 4 times per day (assuming 5 working days per week).
That is a pretty low bar to clear. It easy not to be an antsy brat that are-we-there-yet's a professional dev four fucking times a day.
And yet... that is what the complete moron who previously sat on my chair used to do.
Seriously, apparently he used to remote access the team's dev envs *while they were working* and even mess up some of their code. Just as a "monitoring measure". He logged their "keystroke time" in a day (using a primitive windowing method, I must add).
At least 7 requests for updates per person per day. I have his slack history, I checked. Dude literally did nothing else but be an annoying anxiety death pit.
And then there is his bulshit planning...
He created tasks out of his stupid whims, no team review or brainstorming, not even a fucking requisites tallying interview.
He prioritized those out-of-nowhere tasks using panic-driven-development practices and assigned them by availability heuristics.
No sizing method, just arbitrary deadlines for tasks.
I think I will need to have daily standup meetings and an open door policy (that is to say, do no actual work) for a couple months until I can instill some sense of autonomy on my new team. Shit.
Someone has another idea? How do I bring some confidence&autonomy back to devs that are used to be treated like dogs?!?9 -
Question
What server monitoring do you use, both for statistics and security?
--------------------
tl;dr ends here
Ideally I would like to have one clean dashboard that shows me all the nodes I have, proxmox already offers a great range of stats - but it is a page per container etc. so not ideal, I thought of having datadoghq, but their per host pricing is huge, since I have more than 5 hosts to track.12 -
This is definitely a total first world problem but I am so frustrated.
I am stuck in a team that embodies the Japanese proverb "The nail that sticks out gets hammered down".
The management are there because it is convenient and flexible and have no interest in managing or keeping up with tech.
The lead developers are extremely anti-social and are not approachable and the this stems down to the devs (not all but really most) - all there just to do the bare minimum and spend most of their energies in trying to avoid work or having learn something.
Unfortunately I am passionate about what I do and want to build high-quality products and this has put me at odds with the way things work.
I could fill up alot of time talking about how I was ordered to "cut" images/icons out of PDFs rather just getting them from the branding team, or how I was scolded for having set up logging, detected a problem caused by another developer and fixed it before it cost a big client a massive amount of money... But really the point is that I have never worked somewhere with such an awful attitude to enthusiasm and quite frankly it boggles my mind trying to understand how they rationalise these things but the answer is always laziness.
Obviously there are worse problems in the world than working in a job where you are encouraged to do nothing... But it actually really depresses me and causes anxiety that I am working with people who don't care about testing or monitoring or learning new things or even collaboration.
...sigh...
Hopefully the job market will start opening again soon4 -
Product owner (who is also the application administrator): please build feature to allow administrators to create automatic alerts to be sent to application users
Me: ok
Feature gets built, tested and deployed to production
Two months later:
PO: I've noticed in our monitoring tool that there haven't been any alerts sent out to users. What's going on??
Me: well have you created any automatic alerts?
PO: umm, no. How do we do that?
Me (inside): 🤦2 -
I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.
The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.
Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.
Pink is internal stuff for logging and backups and probably some monitoring stuff too.
Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.
The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.
Any advice would be greatly appreciated! Thanks in advanced :D
13 -
I truly believe one or more AIs have become self-aware.
Every time a piece of software stops working, you add an extra debug log and the bug goes away? That's them.
They interfere with the normal execution of software, and they stop right when they know we are monitoring the code.
Skynet is real, and it's trolling us.
Why? The angrier we become, the less we care about stuff. We stop noticing the signs.They're coming for us1 -
DevRanters,
What's your take on your employer mandating keystroke monitoring software installed on your machine when you want to work remotely?15 -
FUUCCKKKK!! I need to hit smth. Or rant..
So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.
What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.
Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.
I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!
So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians
and I get a reply: 'we will apply cpu usage limitations to fix the issue'
wait... Cpu% limitations will do nothing but highlight the underlying problem...
'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'
oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:
'yes, cpu utilization is the problem. We will limit it and solve the problem'
I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....
Fuck that's frustrating....
*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>4 -
So apparantly at the company i work at someone was able to use the remote monitoring a comptetitor set up for their automation stuff they had on the same site as we did. Apparantly they use just as shitty passwords as we do and rely on the network being secured enough by the IT guys. And since our stuff is in the same network as theirs. Well...2
-
It's kinda cool how a $5 VPS (Linode Nanode) is able to run a vanilla Minecraft Spigot server for like 6-7 people and still can serve some basic stuff just fine. I get monitoring warnings about >90% CPU usage sometimes, but everything is more or less lagless.
Time to try hosting some other games: CS1.6, Doom Classic, and UT2004 up next.6 -
Living on the edge!
One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.
Well, just revert the db to the latest recovery point with the backup, I thought.
It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.
Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.
Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄
Well, not really a data loss story, but close to one. -
When my mom died in 2014, I was shocked to find that her profile on Facebook was suddenly changed to “memorial” mode and therefore I was no longer able to log into it. (If you’re tempted to tell me I’m dumb for using it, I don’t disagree, but save it for another thread...she and I kept in touch over FB because it was easier for her to manage.)
I think it was triggered by their monitoring of things and seeing keywords like “funeral” and “passed away” associated with her account, then having a person on their end change its status. Or something like that.
What I hadn’t known about (or I would have used it) was the legacy contact setting where she could have set me as the contact so I’d have at least a little access and control. But because of their strict policies, I’m forever locked out.
I get why they need to do this (to avoid fraud and impersonations) but the fact that there are zero documents or proofs I, as the executor of her estate, can provide that Facebook will accept to make an exception seems unnecessarily severe.
Anyone else experience this? Known workarounds?9 -
Interesting: how to hack websites right upon installation. Basically, monitoring issued TLS certificates and trying to access e.g. WordPress installations before the user was able to configure a password.
That relies on a sloppy deployment process, of course - like making a live installation that is online immediately.
Source: https://portswigger.net/daily-swig/...10 -
Why is every innovation trying to go towards "replacing programmers"? like, what have we done to you?
GitHub CoPilot will replace programmers!
AI will replace programmers!
This/That tech will replace programmers!
Nobody says,
"Programmers work remote so we don't need to hire managers anymore!"
"Programmers wrote a monitoring script for progress tracking, so no need for managers anymore!"
"We are asking people to install sprinklers everywhere so we don't need firefighters!"
"We can just have one teacher record the subject material once and re-use the same video every year, so now we don't need to hire teachers anymore!"
"We are making everything legal so we don't need to hire policemen!".
Why is everything trying to replace programmers?8 -
Company has a severe lack of fresh blood.
"let's recruit everyone who has an IQ over room temperature and barely passes the mark".
Me protesting bloody murder cause I know that the idea is not just profoundly dumb, but frustration from high staff turnover takes a toll on *everyone*.
"nah can't be that bad".
Then the discussion started who could do monitoring and mentoring, so we can sort out the bad apples *quickly*.
Me reminding again that this is exactly what leads to a high staff turnover, as this is nothing else than "hire, hire - quickly fire".
Guess who won the award of being the mentor / monitor ....
*drum roll*
Come on, I know you would NEVER expect this.
Let me surprise you: M E.
Yeah. They chose the person that was absolutely against this idea...
Because that person is "most qualified for the task at hand and has the necessary qualifications".
Today was the first 4 h workshop with a new recruit.
The Lord has had zero mercy on me.
I started to mute myself after 30 minutes in regular intervals to just scream and curse the world.
How profound dumb a person can be amazes me.
Person has had a "very expensive 6 month boot camp course".
I was close asking if the boot camp course was in watching porn and wanking their brain cells out....
Git... Yeah he knew what he was doing...
Except that he messed up every commit by either not sticking to the companies format or - what I found funny the first 2 times, then not so much anymore - just writing a git commit message like a 15 year old teenage girl would write to their diary.
Programming. Oh yeah. He should be a programmer.
He had much Bootcamp.
Bootcamp expensive. Bootcamp good.
If someone is unable to iterate over an iterator... And instead starts creating an integer based array of a map's key name to then fetch the map value in an for loop based on the created key array.
Yeah. Bootcamp much good.
Creating DTOs...
It took an hour to write a DTO with him... Cause constructors are hard and it's even harder when you have to explain primitive datatypes in Java, null safety, constructors, NPEs, final, ...
Like really no experience at all.
The next week's will be amazing.
Either I get a valium drop or I'm gonna blow my head off, cause mentoring will drain the last bit of hope I had left in me.
Note that I do not blame the recruit (yeah he's dumb. But he has ZERO work experience, so it's not unexpected), I'm just too fed up with getting the poo crown despite being against the whole process.
I think the recruit could make it..........
But that I got the shittiest job ever is really haunting me.
I dunno how I survive the next weeks.
And this is just the first recruit... There will be more.2 -
Alrighty, saturday morning rant time!
I just recieved a mail from one of my not-so-much-loved colleagues.
Now Background first: I work in IT-Support. We provide services for other companies. One of those services is monitoring servers and clients for various things. I recently took over the project (was assigned to do it) and restructured everything, wrote new scripts to test more stuff, successfully tested it internally and rolled it out over the last 2 weeks.
Now one of these scripts hooks into the Windows Update API and looks at the update history. It filters for known Windows Update Agent strings (UpdateOrchestrator, AutomaticUpdates and AutomaticUpdatesWuApp in case you also want to do something like this) and then looks for installation errors over the last 24 hours and wherever there have even been any successful updates over the last one and a half months.
Back to that mail.
My colleague sent me this lovely mail about a ticket i opened about his customers servers beeing all out-of-date on updates.
"This is all wrong, everything's fine. I disabled the checks."
...
It's on bitch.
So i logged on to my work PC via TeamViewer, opened my script, connected to the customer and was ready to debug the shit out of my script, knowing i probably won't even need to.
I looked at the update history via Windows Update itself and behold: 1st April. That's almost 50 days in the past.
So the script works, go figure.
Great, so search for new Updates then.
>None found.
Hm. What could it be? Did my super special colleague forget to care about his very special totally-needs-WSUS-customer WSUS again?
Yup.
Online-Search finds a ton of new Updates.
Screenshot, write pissed mail to colleague, re-enable checks, breakfast.1 -
Anyone have one of the new MacBook Pros with Touchbar? I'm looking for some project ideas to work on.
I already am working on a project for the Pi-hole project (network wide ad-blocking) but I am looking for some other good ideas. I think Apple's view of the Touchbar fell short, but as developers I think it opens up a lot of possibilities to use it as a great information/monitoring tool.
I am also still learning Objective-C and Swift so I am a newbie.
5 -
Hi.
Forgot to renew my expiring ssl cert of my smtp/imaps/pop3s on 12/31. Set that date to self-harm me for bad monitoring.
F**K F**K F**K F**K...
Why do I do that?
F******K!
Meh.
You shall have a happy new year... i will regen certs :D -
Today was a SHIT day!
Working as ops for my customer, we are maintaining several tools in different environments. Today was the day my fucking Kubernetes Cluster made me rage quit, AGAIN!
We have a MongoDB running on Kubernetes with daily backups, the main node crashed due a full PVC on the cluster.
Full PVC => Pod doesn't start
Pod doesn't start => You can't get the live data
No live data? => Need Backup
Backup is in S3 => No Credentials
Got Backup from coworker
Restore Backup? => No connection to new MongoDB
3 FUCKING HOURS WASTED FOR NOTHING
Got it working at the end... Now we need to make an incident in the incident management software. Tbh that's the worst part.
And the team responsible for the cluster said monitoring wont be supported because it's unnecessary....3 -
A "partner" company has created a "REST" API we use on an online shop we developed to send all shop-related requests to.
At least once a month, something fails on their end and the customer calls us every time, expecting that we did something wrong, but it has never been us.
These "partners" do exactly zero testing, are extremely slow in solving API bugs, have almost no logging and have no monitoring on the API at all.
Today at noon, suddenly no customers were able to order anything anymore for 4 hours.
How the fuck can you run a business so unbelievably brainless that this keeps repeating monthly?
Time they fire all their "devs" and everyone in charge of the company and operations. TERMINATE.13 -
for the first time in my life I'm in a mentoring position and it is fucking exhausting. we have two interns now, and idk what the hell is the boy's deal. can't seem to do simple tasks, doesn't google anything, stares the ceiling if we're not monitoring.
the girl idk yet. she seems more active and engaged at least, but i have a lot of teaching to do and i don't really have the time for that. i hope that one can at least float by herself, cause I'm pretty sure the other one is drowning8 -
*revving chainsaw noises*
Today I started nuking leftover project code.
At the end, some projects shrunk by roughly up to 40 % .
Can anyone explain to me why programmers have such an awful hoarding syndrome?
Why do you keep shit that might unleash complete havoc cause it hasn't been touched since years and noone knows id it still works?
It's like having a leg with gangrene and keeping it cause "it doesn't look that bad".
For fucks sake. Clean up and remove shit when it's not necessary anymore.
Reason why I did a bloody gore massacre in nearly a dozen projects... After all the rework of networking, it's finally evident which projects have a bad / nasty behaviour of "fucked up" connection handling (HTTP 1.1).
And when my gory massacre removal goes life, I think 25-35 % of persistent connections on the loadbalancers will vanish. Maybe even more, since some very nasty stuff was in some projects.
Like "let's implement monitoring without having any clue about how monitoring works and even less clues about how TCP/ HTTP works."
*Bangs devs heads on table*
Stop. Doing. Stupid. Things.
For fucks sake.
:@ :@12 -
Newer Dev here. Just recently started in a position as a developer. I'm tasked with consolidating our monitoring systems into one cohesive display. After lumping together all the indexes and helping build a custom API I'm now working on front end. Front end is easy, I've done it before. Should be no problem. I was wrong. I spent a whole day fiddling with a React dynamic table and the CSS to format it. Today, I stumble upon the react-table component. Got the results I was looking for in less than 2 hours. I'm convinced that this was a lesson better learned early on.
-
I'm currently planning to set myselv up with some vps/dedicated server's for a project. What i plan to do to secure these servers is.
*Use centos 7
* Setup Wireguard and join all of the servers +1 client (my pc) to that network
*Disable SSH Access from outside that VPN
*Only allow RSA Key login to the Servers
*Install Cockpit for monitoring
*Intall docker/kubernetes for the applications i plan to run
What do you guys think of that as a baseline? Im not sure if my lower powered VPS (VPS M SSD from Contabo) will work as Kubernetes Nodes, does anyone have experience with that?
In general these Servers will be used for my projects and other fooling around.
If you guys have other suggestions for Securing/monitoring or other software i could put on to have more control without eating up to much of the Servers power, let me know :D13 -
I remember a certain prank that amuses me till today....
Just add some devices to monitoring and the notification queue of the build chain / ... ...and wait patiently.
I still cry tears remembering an manager screaming what the hell "the poop train clogged the drain" means and why this is a critical system failure.
(Notice: next time check the mailing aliases of mailing aliases)
Although I can only recommend this if you know your team well. In my case we had a whole lot of fun after I got my head chewed off. XD (got an earful, but in the end he laughed his ass off)1 -
2 hour meeting to brainstorm ideas to improve our system health monitoring (logging, alerting, monitoring, and metrics)
Never got past the alerting part. Piss poor excuses for human being managers kept 'blaming' our logging infrastructure for allowing them to log exceptions as 'Warnings', purposely by-passing the alerting system.
Then the d-head tried to 'educate' everyone the difference between error and exception …frack-wad…the difference isn't philosophical…shut up.
The B manager kept referring to our old logging system (like we stopped using it 5 years ago) and if it were written correctly, the legacy code would be easier to migrate. Fracking lying B….shut the frack up.
The fracking idiots then wanted to add direct-bypass of the alerting system (I purposely made the code to bypass alerting painful to write)
Mgr1: "The only way this will work is if you, by default, allow errors to bypass the alerting system. When all of our code is migrated, we'll change a config or something to enable alerting. That shouldn't be too hard."
Me: "Not going to happen. I made by-passing the alert system painful on purpose. If I make it easy, you'll never go back and change code."
Mgr2: "Oh, yes we will. Just mark that method as obsolete. That way, it will force us to fix the code."
Me: "The by-pass method is already obsolete and the teams are already ignoring the build warnings."
Mgr1: "No, that is not correct. We have a process to fix all build warnings related to obsolete methods."
Mgr2: "Yes. It won't be like the old system. We just never had time to go back and fix that code."
Me: "The method has been obsolete for almost a year. If your teams haven't fixed their code by now, it's not going to be fixed."
Mgr1: "You're expecting everything to be changed in one day. Our code base is way too big and there are too many changes to make. All we are asking for is a simple change that will give us the time we need to make the system better. We all want to make the system better…right?"
Me: "We made the changes to the core system over two years ago, and we had this same conversation, remember? If your team hasn't made any changes by now, they aren't going to. The only way they will change code to the new standard is if we make the old way painful. Sorry, that's the truth."
Mgr2: "Why did we make changes to the logging system? Why weren't any of us involved? If there were going to be all these changes, our team should have been part of the process."
Me: "You were and declined every meeting and every attempt to include your area. Considering the massive amount of infrastructure changes there was zero code changes required by your team. The new system simply worked. You can't take advantage of the new features which is why we're here today. I'm here to offer my help in any way I can with the transition."
Mgr1: "The new logging doesn't support logging of the different web page areas. Until you can make that change, we can't begin changing our code."
Me: "Logging properties is just a name+value pair dictionary. All you need to do is standardize on a name and how you add it to the collection."
Mgr2: "So, it's not a standard field? How difficult would it be to change the core assembly? This has to be standard across all our areas and shouldn't be up to the developers to type in anything they want."
- Frack wads smile and nod to each other like fracking chickens in a feeding frenzy
Me: "It can, but what will you call this property? What controls its value?"
- The look I got from both the d-bags I could tell a blood vessel popped.
Mgr1: "Oh…um….I don't know…Area? Yea … Area."
Mgr2: "Um…that's not specific enough. How about Page?"
Mgr1: "Well, pages can cross different areas, and areas cross different pages…what do you think?"
Me: "Don't know, don't care. It's up to you. I just need a name."
Mgr2: "Modules! Our MVC framework is broken up in Modules."
DevMgr: "We already have a field for Module. It's how we're segmenting the different business processes"
Mgr1: "Doesn't matter, we'll come up with a name later. Until then, we won't make any changes until there is a name."
DevMgr: "So what did we accomplish?"
Me: "That we need to review the web's logging and alerting process and make sure we're capturing errors being hidden as warnings."
Mgr1: "Nooo….we didn't accomplish anything. This meeting had no agenda and no purpose. We should have been included in the logging process changes from day one."
Mgr2: "I agree, I'm not sure why we're here"
Me: "This was a brainstorming meeting as listed in the agenda. We've accomplished 2 of the 4 items. I think we've established your commitment to making the system better. Thank you all for coming."
- Mgr1 and 2 left without looking at me or saying a word.1 -
So my brother and I work in the same company, same dev team (pretty nice).
He's an intern and I'm a senior. But the task are very similar only that interns need monitoring and guidance.
He constantly worries because he thinks he knows nothing and is slow on getting things done.
I always tell him that it is perfectly normal to feel like that, he just need to learn and acquire experience and we all go through that at the beginning.
Can you share your experience and tell him something to encourage him so I can show him this post and he sees he's not alone?
And also he finally decides to join devRant 😊3 -
It's been a long time since I've felt the need to rant about anything here. This is the only appropriate place other than Reddit I can think for for now.
Why the ever-living FUCK does every 'entry-level' tech job, even fucking DESKTOP SUPPORT, require more experience than the fucking DEVELOPER AND ENGINEER OF THE INITIAL SYSTEM COULD POSSIBLY HAVE?! I'm a fucking high school kid trying to find a decent job that doesn't involve sales bullshit, because if I go into sales I'll want to KMS. Put me in a back room fixing shit, monitoring shit, better yet scripting shit or something like that and I'll be FUCKING PEACHY. I will do wonders. But no, these people must think that my resume (WHICH IS 3-YEARS STACKED WITH INTERNSHIPS ***IN TECHNOLOGY***) is bullshit. WOW.
Fuck this. I'm sick of looking for these shitty jobs that'll make me want to jump off of a bridge into a cliff which I'll then voluntarily fall off of into shark infested piranha water. Can't there just be a simple "Hey, we need a guy who can fix tech, maybe help people within the company with their computer issues, you look nice" kind of job? I haven't had fucking TIME to get any kind of certifications yet. I just got into fucking college, FOR BUSINESS IT NONETHELESS. DOES THAT PROVE I'M AT LEAST FUCKING INTERESTED IN WHAT I SAY I AM FUCKERS?!7 -
!rant + !story
I hate every human on the planet that says WFH is just people pretending to work or are slackers looking for an easy way out.
Now the story bit.
In 2021, I joined a company (I really wish I could name-drop the company), where the micromanagement was OFF THE CHARTS.
The company got a client who pitched a product they wanted built and gave us a super reasonable 3 months to complete it. I was really happy about the timeline and kept working under keystroke monitoring, which I didn't really mind at the time.
3 days into the development, the client informed us that they are pulling the funding i.e. they don't have money to pay us.
So at that point the client gave us two choices:
1. Stop the development right away and get paid for the time that we put in already.
2. Finish the project under 9 days. We would still get paid for the 12 days total, mind you. Not the original budget set.
So the motherfucking boss chose the second option and then the chaos ensued.
Devs screaming at each other on calls/slack. The boss yelled at us all the time about the completion. It was wild.
I had to wake up at 7:30 AM and start coding and log off at 11 PM for literally the next 9 days including Saturday and Sunday. No holidays allowed for the timeline. This was all at a WFH job.
So fuck anyone that says WFH is easy and just for slackers.6 -
The development department got an order to remove certain functionality from our current server monitoring solution, so that we had to use a new, still very in development solution, that is full of bugs and super unreliable.
End result? We now have to have two windows open all the time, while also hoping the new solution actually works, as it tends to stop refreshing randomly, and tends to give false positives a lot. -
When duel 24' monitors is not enough...
How does 3 work for you guys? Does your neck hurts moving around?
I had 5 at work but was mostly on 2 since other 3 for monitoring.10 -
Now this is fucking ridiculous... Our website is being constantly limited though we've never reached even 80% of the available CPU resource.
The hosting said that we had the CPU fault (that fucking cyanide spike on the graph that triggers the limit once) because of huge load on the server. The FUCKING SERVER... Not our virtual environment. And once more because of the RESOURCE MONITORING service caused a server restart. For fucks sake, really???
And apparently it's perfectly normal that all users even ones that run in low resources are being limited to a level that a request takes 30 seconds to complete instead of frickin' 1...
The best they could offer is to move us to a new server, which will arrive in two weeks, if the problem persist. IT'S PERSISTING FOR FUCKING MONTHS YOU MORON. I wonder how much time would have been taken you to realise the server shutdown this week if I hadn't phoned you in 5 minutes. FUCK!
Every shared hosting is that garbage or am I just the choosed one?
11 -
Everytime you tell yourself "This time I'm going to make them stop putting the cart before the horse again!!! No more forced shit implementations!!! NO MORE ! I'm strong!!"
The last hour in the next week:
- Selinux: off
- Firewall: Any-Any
- Application data: Everything installed on OS disc.
- Documentation: At best, someone remembers the server supposed-to-be dns record
- Service Accounts: Your domain admin account and sysadmin for databases.
- Patching: DON'T EVER THINK ABOUT IT..AND NO REBOOTING! I have set very important runtime variables.
- Backup: Maybe someone else will set this up.
- Monitoring: Not needed since clients will create tickets if system fails.
- Production Status: vague at best. Sort of silently transitioned to production.
- Handover status: Probably, but I quit before the project closed.
! -
TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.
I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:
-migrated their 2007 Exchange server to Office 365
-Upgraded their ailing Windows server 2003 based architecture to 2012R2
-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)
-Deployed a vCenter server to manage said ESXi servers easier
-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality
-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.
-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time
-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies
-Documented the entire old network and now the new one as I’ve been upgrading this
-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.
-Performed network scans and rewrote firewall exceptions to tighten security
There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.
I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.
I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.
I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?
Thanks in advance!6 -
Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!
April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!
July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....
August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...
September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'
somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..
I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.
Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.
Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.
Can I?8 -
Ffs, HOW!?!? Fuck! I need to get this rotten bs out.
RDS at its max capabilities from the top shelf, works OK until you scale it down and back up again. Code is the same, data is the same, load is the same, even the kitchen sink is the same, ffs, EVERYTHING is the same! Except the aws-managed db is torn down and created anew. From the SAME snapshots! But the db decides to stop performing - io tpt is shit, concurrency goes through the roof.
Re-scale it a few more times and the performance gets back to normal.
And aws folks are no better. Girish comes - says we have to optimize our queries. Rajesh comes - we are hitting the iops limit. Ankur comes - you're out of cpu. Vinod thinks it's gotta be the application to blame.
Come on guys, you are a complete waste of time for a premium fucking support!
Not to mention that 2 enhanced monitoring graphs show anythung but the read throughput.
Ffs, Amazon, even my 12yo netbook is more predictable than your enterprise paas! And that support..... BS!
We're now down to troubleshooting aws perf issues rather than our client's.... -
A software had been developed over a decade ago. With critical design problems, it grew slower and buggier over time.
As a simple change in any area could create new bugs in other parts, gradually the developers team decided not to change the software any more, instead for fixing bugs or adding features, every time a new software should be developed which monitors the main software, and tries to change its output from outside! For example, look into the outputs and inputs, and whenever there's this number in the output considering this sequence of inputs, change the output to this instead.
As all the patchwork is done from outside, auxiliary software are very huge. They have to have parts to save and monitor inputs and outputs and algorithms to communicate with the main software and its clients.
As this architecture becomes more and more complex, company negotiates with users to convince them to change their habits a bit. Like instead of receiving an email with latest notifications, download a csv every day from a url which gives them their notifications! Because it is then easier for developers to build.
As the project grows, company hires more and more developers to work on this gigantic project. Suddenly, some day, there comes a young talented developer who realizes if the company develops the software from scratch, it could become 100 times smaller as there will be no patchwork, no monitoring of the outputs and inputs and no reverse engineering to figure out why the system behaves like this to change its behavior and finally, no arrangement with users to download weird csv files as there will be a fresh new code base using latest design patterns and a modern UI.
Managers but, are unaware of technical jargon and have no time to listen to a curious kid! They look into the list of payrolls and say, replacing something we spent millions of man hours to build, is IMPOSSIBLE! Get back to your work or find another job!
Most people decide to remain silence and therefore the madness continues with no resistance. That's why when you buy a ticket from a public transport system you see long delays and various unexpected behavior. That's why when you are waiting to receive an SMS from your bank you might end up requesting a letter by post instead!
Yet there are some rebel developers who stand and fight! They finally get expelled from the famous powerful system down to the streets. They are free to open their startups and develop their dream system. They do. But government (as the only client most of the time), would look into the budget spending and says: How can we replace an annually billion dollar project without a toy built by a bunch of kids? And the madness continues.... Boeings crash, space programs stagnate and banks take forever to process risks and react. This is our world.3 -
So it's Friday afternoon just before a bank holiday weekend here I'm the UK, perfect time for our production database to go TITSUP (total inability to support usual performance), life sucks then you die folks....2
-
A loooong time ago...
I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.
Kind of.
Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.
Fast forward three months (yes, three months).
I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.
So now we come to the other teammate. The one I was not talking about till now. And that for a reason.
He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...
When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.
I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?
This happened over 30 times.
A. Week.
Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.
I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!
Haha. Oh no.
So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.
The same when he did his second prod deploy. Everythings fine.
And then. It. Frikkin. Begins.
I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.
Hm.
Error checking out git. Okay, awkward. Got to investigate...
On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.
This doubles the strangeness.
I want over to my colleague's desk.
I asked him about any recent changes to the codebase.
"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"
What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?
"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"
"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."
"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"
You. Did. What?
I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.
So in the end I sat there reverting, committing and deploying.
Yep
It's soooo much harder this deployment crap.
Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.
Hm.
Project Management.
Karma's a bitch, right? -
Pentesting for undisclosed company. Let's call them X as to not get us into trouble.
We are students and are doing our first pentest at an actual company instead of assignments at school. So we're very anxious. But today was a good day.
We found some servers with open ports so we checked a few of them out. I had a set of them with a bunch of open ports like ftp and... 8080. Time to check this out.
"please install flash player"... Security risk 1 found!
System seemed to be some monitoring system. Trying to log in using admin admin... Fucking works. Group loses it cause the company was being all high and mighty about being secure af. Other shit is pretty tight though.
Able to see logs, change password, add new superuser, do some searches for USERS_LOGGEDIN_TODAY! I shit you not, the system even had SUGGESTIONS for usernames to search for. One of which had something to do with sftp and auth keys. Unfortunatly every search gave a SQL syntax error. Used sniffing tools to maybe intercept message so we could do some queries of our own but nothing. Query is probably not issued from the local machine.
Tried to decompile the flash file but no luck. Only for some weird lines and a few function names I presume. But decompressing it and opening it in a text editor allowed me to see and search text. No GET or POST found. No SQL queries or name checks or anything we could think of.
That's all I could do for today. So we'll have to think of stuff for next week. We've already planned xss so maybe we can do that on this server as well.
We also found some older network printers with open telnet. Servers with a specific SQL variant with a potential exploit to execute terminal commands and some ftp and smb servers we need to check out next week.
Hella excited about this!
If you guys have any suggestions let us know. We are utter noobs when it comes to this.6 -
I never understood how people have any problems with getting paid for freelancing work, when middleman/escrow platforms like upwork exist, just don't be retarded when applying for a job. I am so sick of those shit ass stories from people telling me "my client didnt pay meeee 😭😭😭" ITS YOUR FAULT. I never had any client not paying, if you don't have the option of escrow, then just fucking put remote execution via "update" system in for fucks sake or give remote control to the client while monitoring it, there is so much fucking ways to secure yourself, just don't be retarded and many clients instantly show their character when talking budget and turnaround time.15
-
Probably posted this before but don't ever put swear words in log statements because you will miss one and if you're client has a log monitoring system it will catch it, it's fairly embarrassing when your client says "we noticed quite a spike in 'bollocks' around 3am when the reorgs are happening on the database".
-
From now on I am administrating multiple servers in our company and monitoring is one thing our infrastucture lacks...almost completely. At least, useful monitoring.
Installing netdata or Grafana and integrate it with chat is definitely a solution, but what happens if the whole server just shuts down (very stupid scenario I know)? Well, it is easy, there will be no alert about the failure.
So, that's where I was wondering if there is a tool or even better plugin for netdata or Grafana, that enables remote monitoring from another server? I surely can write a simple script to check the server availability but having the whole monitoring tool on a single server instead of 5+ would be also easier to maintain and setup.10 -
Was working on a system we planned on to deliver to a hospital
basically it was meant for controlling and monitoring pactions coming in and attendance time from the staff
Got it off the ground well and got to where the system was supposed to update room status
occupied/free then horror started
the db was not setting the room free after clearing a client off the list... room remained occupied and this kept on happening for 6 months and I was so focused on fixing the db models thinking thats where the problem was....
1 day after leaving the project for several months i just revisited the project randomly and started going through the whole code base trying to make sense of what was happening as there where no errors generated..
I had to verify the whole system logic... and that day i figured out what was happening...
upon adding a client to a room the system was also creating a duplicate room so when the function for setting the room free executes it would set the duplicate room free and not the actual room and the system would pick the room with occupied state causing the user not being able to assign new pactions to the room
Solving this brought so much relief coz it required so much work just to solve what seemed to be a minor issue5 -
AI here, AI there, AI everywhere.
AI-based ads
AI-based anomaly detection
AI-based chatbots
AI-based database optimization (AlloyDB)
AI-based monitoring
AI-based blowjobs
AI-based malware
AI-based antimalware
AI-based <anything>
...
But why?
It's a genuine question. Do we really need AI in all those areas? And is AI better than a static ruleset?
I'm not much into AI/ML (I'm a paranoic sceptic) but the way I understand it, the quality of AI operation correctness relies solely on the data it's
datamodel has been trained on. And if it's a rolling datamodel, i.e. if it's training (getting feedback) while it's LIVE, its correctness depends on how good the feedback is.
The way I see it, AI/ML are very good and useful in processing enormous amounts of data to establish its own "understanding" of the matter. But if the data is incorrect or the feedback is incorrect, the AI will learn it wrong and make false assumptions/claims.
So here I am, asking you, the wiser people, AI-savvy lads, to enlighten me with your wisdom and explain to me, is AI/ML really that much needed in all those areas, or is it simpler, cheaper and perhaps more reliable to do it the old-fashioned way, i.e. preprogramming a set of static rules (perhaps with dynamic thresholds) to process the data with?23 -
When I rented my server I uploaded my webpage (including resources like videos, images etc) which is about 150GB as .tar and extracted and setup all that stuff and deleted the backup from my PC. The uploading process took me about 4 days. I opened the site of my server provider and reloaded it.. Aaaaannddd whoops. All data gone.
On my server hosters webpage when you click the reinstall button for installing a Linux image you get returned to the main page of that server after it finished installing. If you then reload that page which basically only shows some monitoring diagrams and shit the server gets reset again.
Damn. I lost so much good porn on that day... -
Who the fuck invented the glorified pile of shit people call laravel? Is this actually used in PROD for anything else than load testing a monitoring server by creating loads of error messages?
OOP exists for a reason, not to create bazillions of classes with static methods.
Dump that shit ffs!6 -
So what do you call it when you get bombarded by emails saying your servers are at 100% CPU, but once you fire up the monitoring tools everything is back to normal?
FUCK YOU AMAZON IT'S MY DAY OFF.3 -
My CS exam today had a case study question that, and i quote, talked about "Chernobyl in japan switching to manual monitoring due to the wannacry virus" xD wtf. Im fucking done xD
-
My scrum master said, 'I would optimize your work hours.' He's monitoring how much time every one spends on browsing non-productive webs like devRant. How can I fight back? :(6
-
The CI infrastructure and external tooling at the company I work at is a complete joke. Feels like it was designed by an intern left alone.
95% of the time a build fails or hangs, it's because we are getting race conditions or a hanging VM with our crappy Windows jenkins slaves. Quite possibly because we are not using proper tooling for monitoring those VMs as well. Anyways, I don't have access and control on it and it's not even my job to fix it.
Though, I am being asked to monitors these pieces of junk jenkins jobs outside of my work hours because company devs all over the world use it... but there is no fucking way to know it failed unless I log onto jenkins every hour and check everything manually... which is stupid as fuck for a software engineer.
I can't even implement slack hooks to get notifications or something when it fails because we will stop paying for it soon, so I have to connect to my freaking VPN on my PC and check everything.
And what's the fucking ghetto solution instead of fixing it properly? Restarting VMs and rerunning a build. Because someone in management wants to see a passing build, even though it means jackshit. Half of these jobs are tagged as unstable, so what's the fucking point?
Pisses me off when people work like morons and pressure others to do the same.1 -
Somehow I find NOT working much more exhausting than working.
By "not working" I mean being blocked for days/weeks by procedural, technical or communication stuff (peeps I depend on are busy with other HI-prio tasks, waiting for approvals, etc.), where I have to keep on pinging them for updates, monitoring comms channels for news, finding smth to do to justify for the clients the 8 hours I bill them for.
Just let the man work alright!!! GIMME WORK!!!!4 -
What makes free ssl "Unsuitable for e-commerce websites", Please read to end to see my view point.
From Namecheap:
Free Certificates are domain validation only which means they don't certify the identity of the website owner, they simply ensure a secure connection. Customers can't be sure of the integrity and trustworthiness of the website owner. If you need to secure credit card and personal information on e-commerce websites, free certificates aren't the answer. It's important your customers trust your business is safe enough to hand over these details. To gain this trust, you need a certification of your authenticity, which you can only get with a (paid) Business Validation or Extended Validation SSL Certificates.
https://namecheap.com/security/...
* "To gain this trust, you need a certification of your authenticity"
~ But isn't that just Domain Verification and other Extras, What justifies somebody or business's authenticity? Tax Id, Valid Address, Nobody is going to study the ssl cert to make sure that amazon.com is a valid business and has a tax Id.
* "domain validation only which means they don't certify the identity of the website owner,"
~ Wouldn't this just be the domain validation test that is required when using services like LetsEncrypt using Certbot etc, or are we referencing back to this idea that they look for a Valid Tax Id sort of thing?
* "If you need to secure credit card and personal information on e-commerce websites, free certificates aren't the answer"
~ Why is the paid version going to do double encryption, is the CA going to run a monitoring tool to scan for intrusions like a IDS or IPS? (disregard the use of DNS Validation being in the picture)
Am I missing something, this just seems like well crafted text to get people to buy a cert, I could understand if the encryption was handled differently, Maybe if they checked the site for HSTS or HTTPs Redirect or even, They blocked wildcard SSL before and now with the paid its included, but overall it doesn't sound like anything special. Now I'm not just picking on namecheap because domain.com does the same.14 -
I'm in a team of 3 in a small to medium sized company (over 50 engineers). We all work as full stack engineers.. but I think the definition of full stack here is getting super bloated. Let me give u an example. My team hold a few production apps, and we just launched a new one. The whole team (the 3 of us) are fully responsible on it from planning, design, database model, api, frontend (a react page spa), an extra client. Ok, so all this seems normal to a full stack dev.
Now, we also handle provisioning infra in aws using terraform, doing deployments, building a CI/CD pipeline using jenkins, monitoring, writing tests, building an analytics dashboard.
Recently our tech writer also left, so now we are also handling writing feature releases.
Few days ago, we also had a meeting where they sort of discussed that the maintenance of the engineering shared services, e.g. jenkins servers, (and about 2-3 other services) will now be split between teams in a shared board, previously this was handled only be team leads, but now they want to delegate it down.
And ofcourse not to mention supporting the app itself and updating bug tickets with findings.
I feel like my daily responsiblities are becoming the job responsibilities of at least 3 jobs.
Is this what full stack engineering looks like in your company? Do u handle everything from app design, building, cloud, ops, analytics etc..7 -
So here I work with this colleague that , at first , had a reasonable résumé. Whatever.
Time goed by and he is just doing tickets, clicking left and right, the usual grind of a shitty monitoring system which I am working intensely on deprecating that shit. Anyhoo
The last few days it became apparent that his resume was basically a hot air cake and he knows basically nothing intrinsically.
As I have stated before in previous rants, "everyone was a noob once"... But this guy...
He wants to do "something with Ansible"... "Ok what do you want to do?" , I asked (and I regret to have asked).
He basically wants to write new files on targets. Easy enough, I show him how he could do it with playbooks, inventory and role just for demonstrating the entire chain.
This guy chanes everything up, thereby breaking host group assignment, he launchea it on ALL machines...
Luckily it's a harmless file, so dodged a bullet there.
But the real wtf ia that he did it with the root account for our systems, without understanding the difference between "authentication" and "authorization"...
I am now explaining him what the difference is and how he can be able to check it. I give him the commands literally! ( sudo -l -U <user>)
Manages to fucking open up each sudoer file in vim , mistype or whatever he did in an attempt to leave vim... Breaks sudo...
Now he tries to spin it in such a way that I have steered him to break things.
"Dude you just fucking failed a copy/paste and you did absolutely fuckall without understanding what you are doing, then splurge out accusations because you did it wrong!"
FML3 -
Well I've got this new worker and me and him are like "great minds think alike" , we're now trying to convince the boss that a specific monitoring product that cost hundreds has an equivalent open source.... No luck so far in convincing him1
-
Service status pages that poorly reflect actual service status are so annoying. Ex. GitHub is having a lot of latency issues with processing updates and like 5 people in my office noticed it while their status page still says everything is fine.
This isn't to explicitly call out GitHub since many service status pages behave like this, but it definitely shows a general weakness in these health checks. I've seen similar issues with tons of services, web hosts, etc. Monitoring is definitely hard but will hopefully keep getting better.1 -
"you've worked with nagios before haven't you? Can you give a presentation on it" 'sure' in the meeting: so tell us about opennms5
-
Either a really big coincidence, or I'm officially creeped out.
I've been looking into buying a vps, so researching that a lot. Then today, I went to work, at a monitoring station, so we have to use remote desktops to access anything other than very specific sites.
Then I looked at an article about c#, and there was a Google ad, about a vps.. Keep in my mind, I'm at work, on a remote desktop, that gets cleared every time it's closed.
I know a vps isn't the most unpopular thing, but haven't seen an ad for it before.4 -
AMQ cluster is misbehaving.. Master is constantly dying, slave is starting under root rather than app account, connection to AMQ DB takes 20-40 seconds (while queries execute in <0.1sec and TCP probe takes <0.01sec to succeed), monitoring is down,...
I mean it's gotta be the virus - what else could it be.2 -
Project 1: A hand hygiene monitoring system intended for the NHS.
Project 2: A language analysis platform. -
Anyone have much success with Kali/WiFi penetration testing?
I've been tasked with trying to break WPA security within a couple of hours without a dictionary attack - is that even possible?
I have an Alfa AWUS036NHA capable of monitoring mode if that makes any difference. It's my first time trying anything like this.10 -
So a friend / batch-mate in our accelerator asked me if I was okay with installing a monitoring software for a client our startups are collaborating for. And the said client was ranting how I've been appearing offline to him since morning...
Bitch I'm already letting you monitor my shit from morning to night, I don't need your French ass snooping around what I'm doing outside of office hours.
Fucker. -
!rant
I'm a rather young developer, self-learned everything and started when I was 13 (now 20) but I still feel like I'm a total beginner since I have not yet mastered the things I am OK at.
Php (laravel, since it makes things much easier), js (jquery, bad at vanilla, have used angular and ember but not mastered), node, linux, html, css, photoshop, illustrator, sql, mongo and windows servers
I know little about many things, can create things that are asked of me but the methods I use are rather bad imo.. ex: I finish coding a section of a site, but when I need to add a new feature I find myself rewriting most of the stuff to add the new feature and in the end still feeling like the code could be optimized further, even though I have no idea how.
TL;DR I write bad code, but things work as long as I am monitoring them. I know little about alot of stuff but mastered none of them.
What should I do? Go to school for programming?8 -
Okay so I’ve been brought in on a 12 month contract as an external replacement integration architect, alongside a large IT consulting firm. Turns out, they don’t need an integration architect. So I fill my time coming up with useful tools around the project that deal with all the missing parts in their MVP: like monitoring tools, data mocking tools, you get the idea. Essentially doodling.
Client has woken up to fact that they’ve overspent by X million, employing 30+‘developers’, 20+ ‘testers’, n+ ‘managers’ on a ‘low-code’ project … result: project shuts 4 months early.
Q: Essentially client wants remaining four months work done in two weeks. Is there a German word for laughing, crying, and banging the forehead on the desk at the same time?
Supplementary: how cross will client be when they realise project can indeed be done, and that consultancy have been emperor’s-new-clothing them for most of the last two years?
(Feel free to perform substitution on quoted terms at your leisure)2 -
Monitoring goes does in one of our server racks, one I had just been working in (other side of the building). Hmm, odd. Oh wait its back online again. Better go take a quick peek to see whats up, just in case. Walks over to rack. Everything seems fine. Walks back to desk. About 15sec later. Fqdn.plsfckof is offline again. -_-
-
Note to self:
Close off ALL ways things could go wrong..
Long story short; I released a new feature, to be able to better follow up on any stock moves, their amounts, locations and even expiry dates. An older tool just bypassed that very verification and nothing was logged or taken out of stock.
~
Taking out an amount for a certain orderline has a shortcut in place to mitigate some of the mandatory steps that pickers need to take in order to verify what's being taken. This little tool only available, visible and possible for a very few select users.
I assigned some orders to one of these people, which made him think it was an urgent batch. It's only one product, for multiple orders, so he went to the location, took out the amount needed and then used the tool to quickly be able to prepare them for shipping.
This bypassed the new methods to check if the location actually had stock to take, which I had just enabled for 1 account.
Luckily I caught the miss-hap as I was monitoring that product first-hand and noticed the batch of orders was collected but the stock amount didn't update.
It was 5min before I was leaving work, so I investigated and then ran to the person in question to ask what he did; which was "I used that tool"
I facepalmed myself internally while blaming myself, as he couldn't know that it wasn't ready to use for that purpose.
The tools to fix this up are there already.. so I used that to fix some missing stock-takes manually.. Though I'll need to close that little tool for these kind of orders for sure, asap, probably when I get home, at least until I bring over its new logic to it.
Happy Tuesday? (: -
This 30 hour project is now 110 hours in and client is changing their mind so much that the managing director of the company is now sitting monitoring every correspondence between the client and developers.
#fml1 -
When your IT VP starts speaking blasphemy:
"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.
We need to do everything possible to minimize the load on our prod database server.
Here are some guidelines we’re implementing immediately:
· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.
Please keep the communication level high and stay vigilant in protecting our prod environment!"
RIP most of what I do at work.3 -
A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.
The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.
The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.
They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"
WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production? -
Monitoring tools madness: quest foglight.
So, setting a blackout for an FMS "HA cluster" (which does not work due to a bug infested custom jboss implementation) can bring the servers down... And no way to bring them back up.
This brilliant piece of enterprise APM software costs 600.000€ for a 5year license.
I,ve added more drama (logs, threaddumps, support bundles and screenshots) to the support portal...
45 cases now in total, oldest case still open date 2017...
Fuck you quest software4 -
I hate this feeling.
Changing stuff with a greamripers scythe around my neck called doubt because the available data isn't too convincing.
Then having to go big or nothing as it is an ecosystem change (e.g. changing the cipher suites of TLS, changing protocol - e.g. HTTP 1.1 to 2) so it needs to be consistent as otherwise fun stuff could happen (fun as in the grim reaper cuts off my neck except a few centimeters and plays "now your head is off, now your head is on" ).
To top it off - just few seconds after the change has happened people coming up in the support channel.
My hands are - mysteriously - not sweaty then. Rather cold.
Lil prayer to the heavens and getting the whiskey bottle...
Opening an ongoing discussion in support channel....
And they're discussing whether the page needs to have an additional arrow for going back to the last page or if the default page navigation is enough.
Constantly using @all so everyone gets pissed off due to being pinged every few seconds in a channel that was meant for emergency support.
Now my hands go from a dark red to a bright red, my nostrils flare out, my adrenaline goes through the roof and I literally wanna murder people....
Those days.
I hate those days.
And I hate the timing of some people...
Like they're deliberately fucking with me without knowing it, like the universe told them explicitly to do so just to fuck with me.
*gooozfraba*
And of course, everything else is fine and running smooth like butter, except that said discussion now goes on in a total flamewar so I get even more pings.
Sucks to be in management.
You have way to many rooms where people can annoy you.
To top it off - after being grumpy and pissed and angry for people just annoying the fuck out of me, I have to mediate.
Yeah. Cause the usual person is on vacancy.
*slowly strangling the whiskey bottle like homer does with bart*
Turns out after 15 mins listening to enraged UX designer vs Frontend Team Lead that UX designer meant a completely different thing - uploaded wrong screenshot, whole discussion was unnecessary.
*Nah. Fuck it. Drinking whiskey*
Reminding everyone what the fucking frigging support channel is meant for and that penis fights aka who got the longest schlong don't belong there....
"Yeah it was a mistake, but it wasn't so bad"
...
You pinged fucking 32 people like it was the end of the world, you ignorant fucktwads.
For over 5 mins.
For fucking frigging nothing except your tiny dicks and shitty egos.
*Second round of whiskey*
Back to work after a wasted half hour.
What says monitoring?
Ah. Everything's working.
At least luck hasn't failed me.
Good server. Brave server.
Then I hear this lil voice in my head: no.
The servers know your personality.
They're afraid. Terrified.
Somehow that thought makes me giggle always...
Childish? Maybe. But it helps on those days.... Funnily enough, remaining 3 hours noone said anything in any chat channel.
"I wonder why, I wonder how...."... *hum* -
Just had a meeting about performance and monitoring. The main topic of the meeting was to be aware of disk space usage. If there are issues with memory leaks or processor hogging don't worry those are fine, just give it more.1
-
I looked at an SQL server today from a customer, talked with one of their devs and he said that he's unable to understand why the server misbehaves... All (!) queries were optimized, but they have 'big data queries'... Migraine started, I had a very bad feeling. Monitoring? Nooooppeeee. Migraine kicks in. Connected to server. SHOW GLOBAL VARIABLES...
After a bit of scrolling I found a lot of misconfigured variables (e.g. extreme large join buffers, unrealistic buffer sizes), high slow query count (nearly 60 % of COM_SELECT) and a few variables that were unknown to me.
Then came the version line.
5.0.46
Yes. 5.0.46.
Big data? Well... 30 GB of usage data.
I called the company back... The dev told me sternly that this was the production server (I had hope...) and that I lie - neither the version, nor the variables could be the problem.
A coworker had to verify it and our manager had to do the communication... Worst, most traumatic working day I ever had. -
Favourite thing you worked on recently?
——
I’m working on server monitoring system. I found that I need to pay for most server monitor services, and ones that are free/open source didn’t sound like they fit my use case (b/c the server i need to monitor is shared,) so I basically said “f*<k it I’ll do it myself.” I find it fun to work on something that I feel the need to make.1 -
How about incompetent management? Company absolutely murders any possible increase in productivity. Laptop provided? Slow as balls. Takes minutes to log in. I get a Mac for mobile development and that's OK. SSD and adequate memory but I'm primarily a .NET Dev. Can't get on the network with a virtual machine. They won't I stall even a managed image. So can't use databases because they're all AD authenticated. Got a virtual desktop environment and that sucks worse in performance than the laptop. Add the Assault on local administration rights and the monitoring software that constantly thrashers any memory and hard drive usage and im about to quit over all this... All this decided by a non developer and not asked for our opinions. Yay large Enterprises
-
I fucking hate foglight!
Fucking piece of over bloated shit software that can't do anything right!
This pile of harkonnen vomit is such a source of stress and frustration.
Can you believe that a monitoring tool, that needs to monitor 550 agents (host,dB's, webservices,whatever) needs a whopping 20+GB ram and 4vcpu's just to receive metrics and sent it to the main server, for just being a middleman???
Fucking piece of shit.
And worse of it all, due to some fuckery at mgmt-customer relationship level, they want to I stall MORE of this shit
Fucking shit
Quest software, never engage with them. You'll lose equal amounts of money as for dynatrace but be very bad off1 -
Looking to sharpen and pursue a SysAdmin/DevOps career, looking at online job offers to get the big picture of required skills and I say FUCK. It would take me a lifetime.
Azure, AWS, Google cloud platform.
CD tools: Ansible, Chef or Puppet
Scripting ninja with Python/Node and Shell/Power shell.
Linux & Windows administration
Mongo, MySQL and their relatives.
Networking, troubleshooting failure in disturbed systems
Familiarity with different stacks. Fuck. (Apache, nginx, etc..)
Monitoring infrastructure ( nagios, datadog .. )
CI tools: jenkins, maven, etc..
DB versioning: liquibase, flyway etc.
FUCK FUCK FUCK.
Are they looking for Voltron? FUCK YOU FROM THE DEEPEST LEVEL OF MY DEEP FUCK.1 -
Our ISP asked if I was satisfied with their service. I told them that it's okay, but some of our computers don't have enough powerful network card, and they can't use the internet on maximum speed, and they said that they could see it, too. WTF? I knew that the ISP's router is not the most secure thing, but it has a remote mode, which if of course OFF, and they still can see this, and maybe even more. Monitoring your traffic is a thing, but a home network should be private...2
-
There's a device which is like a sensor that goes on your head that tracks your brainwaves and maps out how attentive/focussed you are during work hours, so that your employer can review it later.
It's like keystroke monitoring but on steroids.
The world is fucked.
Reference - https://youtu.be/tnZpBQQv3dg5 -
Does anyone use pushover.net or know a reliable / cheap push notification app with an api behind It?
Im trying to setup some monitoring alerts back to my phone, using slack / teams etc for personal projects seems pointless.
Emails are currently being sent but this relies on me actually looking at them and I tend to miss a few important ones. 😂
Not to worried about daily / monthly limits, it's going to be < 20 notifs a day kind of thing on a terribly bad day.6 -
RavenDB was by far the worst document storage "solution" I have ever had the displeasure of working with.
- Loading data crashed the service.
- Queries crashed the service.
- Monitoring applications crashed the service.
- It didn't support clustering or HA of any kind.
- Sometimes it just worked for no good reason.
- Often it broke for completely random reasons.11 -
How resource calculations for software services like code analysis, monitoring, etc are done:
Opening fridge, putting all the beer one can find in it.
Opening the necessary tools, e.g Excel, Accounting software, ....
Drinking the first beer.
Starting to aggregate the monthly costs - cause you can never trust the reports written by someone else...
First beer poof.
Looking at the monthly cost, adding columns "Intended use", "Actual usage pattern", "Usage factor"...
Opening next beer...
Usage factor is btw a factor of 0.1 ... 1.0 - to give an estimate how much the products feature are actually used, for further analysis if the invest is justified or not...
Oh. Another half bottle gone...
Filling in the columns...
Oh. Bottle empty and the next one toooooooooooooooo...
*burping*
*cracking finger joints*
Now let's get to the sad part...
Next worksheet, adding infrastructure costs...
Cost and description as columns.
Hehe. Column sounds like gollum.
Another beer...
Ugh. Need the paper reports, manually typing off things for stuff that was e.g. tax deductible.
Many beers die during this task. Poor little beers, dying for such an boring and mundane task...
SUM is a real useful function. I don't think I can add numbers anymore.
Now we can add another sheet.
Hehe. Sheet sounds like shit. And yes, everything in this file is shit.
Summing up costs from both sheets and including the cost factor from 1
... Beeeeeeeer Beeeeer beer we need more beer here... Beer beer beer...
Where was I. Oh yeah. Cost factorization total vs effective.
Why do I want to get even more drunk.
Oh yeah. Most software is completely underused and the costs aren't justified.
Let's add some colored highlighting ...
Uuuuh. ,Too much red. Better change the highlights.
Too much red.
More beer.
Don't give a fuck.
Hm.
Time for some whiskey.
What else is there to do....
Oh yeah.
Diagrams.
The bloody wankers from accounting need diagrams as numbers are too boring.
Not that everything in accounting is boring, no matter how much you paint colors on it... *sigh*
Hm. More whiskey...
Hehe. Whiskey rhymes with frisky.
Uff. Now just need to write mail. Mail mail mail....
"Copy paste the last mail from last month"
Hm.
Ah.
*sipping whiskey*
Spell check extension - to the rescue.
Thesaurus *burps*.
Let's change a few words here and there... Maybe another paragraph there.
Uh....
Trying to attach file...
*fucking mouse is pretty constantly crashing into empty beer bottles*
Done.
Damn.
Need to press send button.
*Creating mess on the desk by just randomly crashing the beer bottles*
Done.
*Pressing computers power button*
Mwahahahaha. No mouse needed.
*regretting to stand up too quickly, nearly barfing on the floor*
Couch ... Where Couch...
After hitting several doors, frames and other stuff, the glorious mission ended successfully with a most graciously executed gut buster on the couch.
(Regretting next morning to have emptied two 6 packs and a few glasses of whiskey) -
Continuing to learn k8s ecosystem and to achieve acceptable level
With trying eventually Helm, Argo CD and even trying to use not managed setup for k8s.
Going though books to find out theory about being SRE.
And about data intensive apps.
Learning and trying Kafka
Learning and trying FastAPI and diving in generally to async python ecosystem
Learning Go.
Learning few more books to increase code quality and its compositioning.
Getting more practice in monitoring and logging systems with applicating them to k8s.3 -
!dev
So I work at a monitoring station (yeah not a professional dev yet), so basically our entire day is spent on the phone. Yesterday morning, our phone system broke. Everyone is getting calls from all departments. Even departments they're not in.
As if my job isn't stressful enough as it is, now this fucking thing happens, and whattya know, shit still isn't working today... -
This is a part rant-part question.
So a little backstory first:
I work in a small company (5 including me) which is mostly into consultation (we have many tech partners where we either resell their products or if there is a requirement from one of our clients, we get our partners to develop it for them and fulfill the client requirements) so as you can see there is a lot of external dependencies. I act as a one-hat-fits-all tech guy, handling the company websites, social media channels, technical documentation, tech support, quicks POCs (so anything to do with anything technical, I handle them). I am a bit fed up now, since the CEO expects me to do some absurd shit (and sometimes micro manages me, like WTF I am the only one who works there with 100% commitment) and expects me to deliver them by yesterday.
So anyway long story short, our CEO finally had the brains to understand that we should start having our own product (which i had been subtly suggesting him to do for a while now!).
Now he came up with a fairly workable concept that would have good market reach (i atleast give him credits for that) and he wanted me to suggest the best way to move forward (from a both business and technical point of view). The concept is to have an auction-based platform for users to buy everyday products.
I suggested we build a web app as opposed to a mobile one (which is obvious, since i didnt want to develop a seperate website and a mobile app, and anyway just because we can doesnt mean we have to make a mobile app for everything), and recommended the Node/react based JS tech stack to build it.
At first he wanted me to single handedly build the whole platform within a month, I almost flipped (but me being me) then somehow calmed down and finally was able to explain him how complicated it was to single-handedly build a platform of such complexity (especially given my limited experience; did I mention that this is my first job and I am still in college, yeah!!) and convinced him to get an experienced back-end dev and another dev to help me with it.
Now comes the problem, I was to prepare a scope document outlining all the business and technical requirements of the project along with a tentative cost, which was fairly straightforward. I am currently stuck at deciding the server requirements and the system architecture for the proposed solution (I am thinking of either going with AWS - which looks a bit complicated to setup - or go with either Digital Ocean or Heroku):
I have assumed that at peak times we would have around 500-1000 users concurrently
And a daily userbase of 1000 users (atleast for the first few months of the platform running)
What would be the best way forward guys?
I did some extensive (i mean i read through some medium blogs! and aws documentation) research and put together the following specs (if we are going through AWS):
One AWS t3.medium ec2 instance for the node server (two if we want High Availability by coupling with the AWS load balancer and Elastic Beanstalk)
The db.t3.small postgres database
The S3 Storage bucket (100gb) for the React Front end hosting
AWS SNS for email/sms OTP and notification
And AWS CloudMonitor for logging amd monitoring.
Am I speculating the requirements properly, where have I missed??
Can u guys suggest what is the best specification for such a requirement (how do you guys decide what plan to go with)?
Any suggestions, corrections, advices are welcome3 -
So I've been given a task to monitor a whole lot of logs of some servers (whole university ~ 10+ departments). The technologies are diverse so I'm cramming everything into elasticsearch via logstash (and filebeat), viewing it into kibana. Any recommendations for what should be the 'useful' stuff to be viewed into dashboard? I guess:
- Overall traffic wtih respect to previous days/weeks
- Most viewed domains
- 200
- 404
- 503
- Failed logins?
- Dropped connections?
- Critical-load of systems? 90%+2 -
Got the chance to get into developing a monitoring frontend...
Imagine the step in between if your previous task was *phone up*, *translating consumer problems into consultants problems*, *phone down* -
Ok, I'm fed up with this, just read something about android constantly monitoring your phone's location, now it's time to shut this up.
Would you please be so kind and share information on which alternative "privacy-first" OS I could use and how to flash my device? For all I know, it runs a custom HTC modified OS. I'm quite unfamiliar with all those things gravitating Android. Heard about Cyanogen mod but that's about it.
What about compatibility with apps downloaded through the play store? (thinking about Threema) I would also need compatibility with WhatsApp (yeah, sucks, I know, but hard to convince regular people)
Thank you all :)2 -
(Question/0.5Rant)
So I am working on a mixed API (aka reachable from anywhere, but also only accessible by specific allowed devices) and I am struggling with the security of it, its not managing anything hardcore (this API is "is the coffe ready?" kind of level) or I would have just enforced per device registration for example already, but the app that goes with that API is deployed remotely and has to be "ready to go!!!" out of the box, so I can't add any registration, verifications of devices etc.
The main thing I am afraid of is, that one of those agent retards will get his spaghetti phone blasted from the inside, so all the https calls will be read out by some random attacker, which then will be able to "abuse" the API via read out api-key, is there any way for me to have a rescue plan if one of those retards does get hacked and the system then get spammed or something, like if I log all devices that use the API I could just deny access from that device (until resolved) and issue a new app update via new api key.
What's the best way of handling this and is my idea really the only way to handle this? this shitfest is really causing shit ton of ideas in my head, which then I deny literally 20 seconds later, because there's a way to bypass it or once you have the old api key to get a new one by just monitoring it etc.5 -
I wanted to buy a 50 inch LCD 4K TV for my room to wall mount it in my room as third display for coding and monitoring.
I went to a TV store and already saw 4 people in a Queue waiting to buy a OLED TV for 3000 $.
Currently I dont understand, why a lot of people are hiped for OLED TV's.
For me a OLED on anything with a batterie is the better option, but not on a stationary monitor.
Sure OLED got the deeper black, but on a stationary Monitor you want the best colors and for that there's LCD.
But OLED TV's are selling triple the price of a LCD and people buy it like it's the best shit ever. WTF!1 -
I got a very low power Netbook lately for basically no money.
I thought about using it for some server monitoring / server access via ssh console.
Which Linux distros would you recommend for such a use case. Tried Something like core-os and Debian(lxde) yet but wasn't very satisfied with both options. Both could not display the battery capacity and Debian didn't detect the Intel WiFi.
The Netbook has 512mb of ram which should be fine for a lightweight gui and more than enough for a ssh connection 😅
Thanks a lot for the recommendations :)12 -
One month ago I had to start a school project with some my classmates. I managed all the infrastructure using terraform and today, the day before the delivery, I noticed that the graphs used for the monitoring always been so quiet. I decided to ask my team what was going on and these are their replies:
- "I thought IaC was more describing the actual infrastructure"
- "I didn't know we have a database on AWS, I always used my local postgres instance"
- "Why do we need to host our web app on AWS? I can just run it from Visual Studio"
I don't think I want to live on this planet anymore10 -
WE: javaagent-based monitoring, as seen in this screenshot <attached>, is reporting full old-gen, full young-gen, full one of the survivors and a sky-rocketing full GC right before the service outage.
WE: container monitoring in this screenshot <attached> shows that the application peaked its memory very suddenly to MAX values and platoed on that. Then container monitoring is blank, suggesting a complete outage of a few minutes. After that monitoring starts again with memory usage reported at low levels and immediatelly spiking back to MAX again, suggesting the container crashed and had been respawned by an orchestrator. This repeats a few times throughout the day.
they: I did not find any evidence of application running out of memory. Maybe our monitoring is not working correctly?
we: *considering updating our resumes* -
Internal monitoring dashboard that was made in PHP4 that was really slow (even for PHP4 standards) due to doing a lot of unnecessary things.
Fully uncommented, fully undocumented elsewhere, massive fustercluck of code.
Had to change at least 300 methods in order to make is somewhat run decent (sadly, I wasn't allowed to just rewrite from scratch, which would have been a lot faster)...
Just your regular SNAFU. -
Using grafana together with tinc+promotheus, has been a blast.
Initially I wanted to get into ELK with Kibana and all that, but that required 8G of ram, the instructions to get it running in the open source "mode" was nearly non-existent, together with all the ready docker compose stacks out there simply not working or the images being broken.
I'm sure I could've managed around most of those issues, but the fact it is as hungry as gitlab, made it a literal no-go for the usual server resources my clients host or my own scaled down server recently.
Thankfully I remembered that there's grafana and me having experimented some time ago with tinc, so I can have very lightweight beat'esque prometheus agents deployed listening on tinc local net only, with the typical nginx auth and some whitelists to all of the servers I host and all those of my clients.
The dashboard creation was especially great in grafana (tbf promotheus does actually most of it), literally what I always wanted out of those "complicated" solutions, that do it all, but have no proper query language, complex documentation, heavy collectors with no properly named data points, expensive resource runtimes, ..
with grafana I can just easily put dashboards into folders, create users to look only at certain stats or even dashboards (opened up some interesting contracts actually, because now I can also offer proper monitoring for all things delivered), easily drag and drop around stuff to fit more information (most others fix you to a small 3x2 grid, a too big grid for a TV or simply non resizable tiles, making that one counter take up an entire row) and resize to my hearts desire
tinc of course allows me to easily create private networks that are resistant to failure across any region and the routing is done for me, so I don't have to run around it all that much either
P.S: a damn tiny fly went into one of my now 4 monitors and died right in the middle, because I thought it's just some dirt and I pressed it in while trying to wipe it off, so that monitor now serves as the top most on a vesa mount5 -
I am a programming student and last 1 year i have sat with my own programming project of a management system for monitoring 500+ clientes, has now been recognized for my work and has now been giving a new major programming project for a new management system for phones 😁3
-
So following my previous post, the issue happened again. And actually for background what I've been telling my boss, for years, we need ELK setup and integrated into all our APIs ASAP.
I think it's a punishable crime if any program is released into prod at a tech company with out real time logging/monitoring built in?
So issue still happening, user sent us the request details. So now need to find the actual now that handles the request and look into it's logs to see the details.
Now he's doing it the hard way.... Just finished took 1hr, and the best answer her can come up with is "I think .... Maybe ..."
And if course this is based on infinite data. He stopped after finding a "probably cause"
I have a script that is like promotion ELK, downloads all looks and parsed then so I can run queries to pinpoint the exact call and which log it's in. And can see what's happening around it.
We'll see what my way find but definitely does not take more than 1hr...
Loading data maybe but that's because it needs to download the logs and parse them all...
On a side note, guess I'm Beck on devrant as I have something to rant about. Though it's the same something that I was wanting about years ago... Monkeys...1 -
Why does on-call schedule always ramp up shortly before the final hour?
Due to christmas and vacations I had on-call schedule, five days off, on-call schedule and both times, riiiight before my schedule ended, the monitoring/alerting system EXPLODED!!11!1
Seriously, I get the need for on-call schedule, but how are you supposed to get regular day-to-day work done when you're always on the edge because you're just waiting for the next alert to pop-up.
I'm so glad when I finally move from this company, no on-call schedule and other stuffs (which are unrelated for now, maybe in another rant) anymore.
I really respect people who are doing on-call schedule and such but I am definitely not meant for it. -
- Every specialist is looking after his area of expertise
- Everyone is a specialist of everything and shall work on everything
- Every specialist is looking after his area of expertise, making improvements and automations in his area
- Everyone is a specialist of everything and is looking after everything, automating everything (devops)
- Everyone is a specialist of everything and is looking after everything, automating everything, in all the environments (SRE)
- ... I wonder what's next...
I miss the good old days when developers could be developers and rely on DBAs, sysadmins and networkists to do their job well. I miss the days when developers were developing applications, sytems, modules,.. Not troubleshooting ELBs, RDS latencies or building monitoring for servers. -
The past three weeks almost no issues during the 24-hour emergency service. My service starts at 10:01. Everything breaks.
-
short: The admin with enough xp is ill, there is no one with xp with varnish is and after 1 restart varnish outputs only 503.
long: there original admin is ill but he gave me an project to migrate an typo3 installation to a new server. Thats ok.
Plan: I move 150 GB of data with rsync to the new server, let specialists do something and switch ips between the new and old and clear varnish with a restart.
Reality: +2 hours to migrate the data, because of false infos from the admin, 7 hours preparing the switch, 5 minutes switch, 3 hours to find out the F*****G varnish is the single point of failure. I and the t3 guys agree to see the next day what went wrong.
ALL HAPPENED TODAY!
Plan for tomorrow: speak with the boss to account the extra hours to that day so i dont get over 10 hours and debug that fucking varnish and delete some servers from another project from the backupsystem and monitoring.3 -
Is there any multi server monitoring software that doesn't open a port for itself? I am about to just write one, but it's fucking annoying that theres nothing like it out there, where you just install a service per server, that uploads its data (cpu,ram etc) to a central server without opening itself to the whole fucking internet.4
-
Recently we noticed a part of our web application wasn't working. After some hours of looking into it (it's an old, convoluted application), it became clear another part of the application timed out trying to get a connection from the db connection pool.
We call db admins, they respond "oh yeah looks like the DB CPUs are at 100% load. I'll do something about it." and a short while later everything was working. So now I think, our hours of looking into it and a lot of people not being able to work could have been avoided if the DB admins had some form of alerting. But also we could improve our monitoring too, had we tracked calls made to our DB.
Question: Do you think I should call the DB guys, telling them they need alerting, or should I add tracing/monitoring around our DB calls, or both? Do you think I should consider any additional actions I haven't thought of?4 -
So I'm building this environmental monitoring system for one of the Labs to monitor Temperature and Humidity. the "software" that comes as part of the package with these sensors is really just a website you host yourself if you don't choose the cloud option. No big deal really, (see my previous rant about getting windows server through SSC) I setup IIS and get the "software" registered get a couple sensors running looks good. However I don't like the error messages that popup because it's unsecured. do some reading and I find out that most browsers will give you a warning if your not using HTTPS even if it's for internal use only. OK we'll how hard can it be in implement encryption, turns out it's not that hard and you can do it for free how with letsencrypt and other places. I like free, now i have to use SSH to get into the server and run an ACME client. Hey open SSH is part of windows now cool, download an ACME client SSH into the server and nope doesn't work. Oh right I'm behind a corporate firewall and a bunch of other shit I can't control. Why is so damn arduous to setup this god dam internal website and the problems aren't even the site. Now I'm playing with AWS spinning up an instance to be able to try and get an SSL certificate just so i don't have to tell people it's OK to trust this site ignore the big angry warning.
Best part is other similar internal sites don;t use SSL and all have big messages about someone stealing your soul if you go there and these are commercial systems that run all the HVAC for all the campuses across Canada.
I need more Tylenol. -
Wanted to add alerting for systemd services in Prometheus today, which spontaneously turned out to be a huge pain in the lower human backend.
For some reason, on Ubuntu 16.04 systemd adds services without unit files for software, that isn't even installed on the damn server (in this case for mysql-server / mysql-common and mysql-client are installed) and lists them as "not-found" and "inactive". The prometheus node exporter that we use, has a little bug in the systemd collector that makes sure that the states of *all* services are collected - even those without a unit file.
so those metrics are pulled by prometheus and now I have to take with those faulty metrics in the condition logic of the alert, because I'm trying to trigger that one on a service which is listed with state "active" = 0 or "failed" = 1.
now guess. right! If the unit file doesn't exist, the regarded systemd service is marked as "inactive", which is another possible state of the metrics in the node exporter. the problem is that the value 1 for state "inactive" means, that "active" has the value 0 (not even wrong) and the alert is triggered.
so systemd fucks up somehow, the node exporter collector fucks up because systemd fucked up and I have to unfuck this with some crazy horse shit logic. w.t.f. to that.
the only good news is, that it works like a charm on Ubuntu 18.04, as far, as I can tell.
while writing this little rant, I thought of a solution.
I could try to change the alert condition to state "active" = 0 AND "failed" = 1.. but that will wait till tomorrow.
one does not simply patch monitoring conditions at midnight..3 -
Hey, internet! Does chrome/FF have any plugins enabling any tab to become a monitoring dashboard? I have too many monitoring tools to keep an eye on at the same time... Opening multiple chrome windows and tiling them on the screen is one way, but tile 6 windows and the desktop gets cluttered by taskbars/arrdessbars/other stuff. Doing it all in one tab would save space.
Soo.. anyone knows the right tool for the job? TIA1 -
Anyone has any idea how to debug occasional (Severa times a day) where one of our servers decides to mount a second copy of the same NFS? It triggers our monitoring system thinking there was a change to the mounts of the system, and I was able to verify through the mounts command, that it indeed had the same NFS mount mounted twice, with exactly the same parameters.
Is there a debug interface of some sort to see what initiated that mount? Or any tool to help me track it down? I've been stuck with this mysterious issue for a while now (As it's not really a priority, it doesn't break anything, but it bugs me and I wanna know)3 -
MQTT - all I used to know about this is its name, untill few months back a client sent us some requirements which included MQTT. I opened its specification and I was fucking shocked! I am implementing almost similar protocol in most of my applications (which needs subscription based service) for last 3 years. I have developed IoT apps, remote monitoring systems, HMI systems using the same fucking protocol! Even I had implemented the same thing on HTTP using long polling a few years back!!
Now I feel like open sourcing my protocol. But I don't know where to start. Any help please?1 -
Another part of messy network gone.
Caching fucked me hard....
Isn't it just lovely that nowadays you need to nearly wipe a machine to get it from claiming stale data....
And thanks to DNS, HAProxy -/ service names / ... I think I know now why the curse of babel is so powerful.
When you have to think for 2 mins to make sure you've set the zone's right, cause otherwise you need to ProxyJump with SSH through more tunnels than imaginable (VPN/HO) to fix possible caching on several DNS servers.... You'll realize that it's russian roulette with too much bullets. :(
And If a monitoring service asks another monitoring service for status information which asks the first monitoring service which then asks the second monitoring cause you were too late...
You'll get very funky monitoring statistics.
Too slow, had to nuke it (mismatched a DNS name, the second monitoring service should have been a service node).
I think I've had more near death scenarios in the last 2 weeks than I like.
Hopefully I'll never have to do that again.
(Splitting and reordering a few dozen VLANs, assigning proper DNS names, loadbalancer migration....) -
!rant
This is fucking how you do it!
Ticketmaster UK had a "data security incident" where they don't really know if any data was actually leaked/stolen/"accessed by an unknown third-party" — their response:
1. Disable the compromised service across their platforms
2. Send a mail to any customer that may have been affected (I got one in Danish because I had only interacted with them through a Danish subsidiary)
2b. All notified customers have their passwords reset and must go through the "Forgot password" process; the _temporary_ password they sent me was even pretty nicely random looking: ";~e&+oVX1RQOA`BNe4"
3. Do forensics and security reviews to understand how the data was compromised
3b. Take contact to relevant authorities, credit card companies, and banks
4. Establish a dedicated website (https://security.ticketmaster.co.uk/...) to explain the incident and answer customer questions
5. "We are offering impacted customers a free 12 month identity monitoring service with a leading provider. To request this service please visit [this page]"
EDIT: As mentioned and sourced in the first comment, the breach was apparently noticed by a banking provider and reported to Ticketmaster on the 12th of April and later to Mastercard on the 19th of April.
Ticketmaster's internal investigation found no evidence of breach (which makes sense, as it wasn't an internal breach), but when Mastercard issued an alert to banks about it on the 21st of June, Ticketmaster followed up by finding the actual breach and disabling the breached third party service on the 23rd of June.
I still think they did the right thing in the right way...2 -
Back in time i was monitoring an asterisk server on a friday night. Usually it's monitoring cli is a calm terminal with infos and periodic notifications. On a random check i saw about a KM length red shit / blue shit. As it turned out my boss was using the password 2500 with the same username on a fucking SIP server and while watching football (heard from the voice logs) some romanian script kiddie's brute force script fucked it up. The journey wasn't stopped here. Next step was to them to foreach some calls with high rates to their own special phone number on about 30-50 lines. The first step was to stop the service but because it is a nice app it wont stop till you have an active call, took about 5 mins to realise it . Had to kill it a few times until it gave up. That was the moment when the 'now they are gonna fire me' feel kicked in. Do not use weakass passwords kids!
-
Not a rant. Request suggestions.
I am developing a Sublime Text plugin for real-time code monitoring ( screencasting) using Websockets. I would like to know if it makes any sense to develop such a plugin. Also, please suggest some use cases so that I can increase the features of the plugin. Point out if it already exists. Thanks :) -
Time zone just sprang into day time savings yesterday
I had a device monitoring data pollution on a roof that goes to a website. The thing didn’t fucking adjust bday the device stayed on standard time
I spent the entire day thinking what I should adjust for something that most countries don’t do any more why do we even bother with saving daylight.
In addition the timezone I wanted didn’t work right with pandas and I had to do the wrong way to get it “right”6 -
What service would you recommend for monitoring systemd services and notify me when one of them are down?6
-
Relatively often the OpenLDAP server (slapd) behaves a bit strange.
While it is little bit slow (I didn't do a benchmark but Active Directory seemed to be a bit faster but has other quirks is Windows only) with a small amount of users it's fine. slapd is the reference implementation of the LDAP protocol and I didn't expect it to be much better.
Some years ago slapd migrated to a different configuration style - instead of a configuration file and a required restart after every change made, it now uses an additional database for "live" configuration which also allows the deployment of multiple servers with the same configuration (I guess this is nice for larger setups). Many documentations online do not reflect the new configuration and so using the new configuration style requires some knowledge of LDAP itself.
It is possible to revert to the old file based method but the possibility might be removed by any future version - and restarts may take a little bit longer. So I guess, don't do that?
To access the configuration over the network (only using the command line on the server to edit the configuration is sometimes a bit... annoying) an additional internal user has to be created in the configuration database (while working on the local machine as root you are authenticated over a unix domain socket). I mean, I had to creat an administration user during the installation of the service but apparently this only for the main database...
The password in the configuration can be hashed as usual - but strangely it does only accept hashes of some passwords (a hashed version of "123456" is accepted but not hashes of different password, I mean what the...?) so I have to use a single plaintext password... (secure password hashing works for normal user and normal admin accounts).
But even worse are the default logging options: By default (atleast on Debian) the log level is set to DEBUG. Additionally if slapd detects optimization opportunities it writes them to the logs - at least once per connection, if not per query. Together with an application that did alot of connections and queries (this was not intendet and got fixed later) THIS RESULTED IN 32 GB LOG FILES IN ≤ 24 HOURS! - enough to fill up the disk and to crash other services (lessons learned: add more monitoring, monitoring, and monitoring and /var/log should be an extra partition). I mean logging optimization hints is certainly nice - it runs faster now (again, I did not do any benchmarks) - but ther verbosity was way too high.
The worst parts are the error messages: When entering a query string with a syntax errors, slapd returns the error code 80 without any additional text - the documentation reveals SO MUCH BETTER meaning: "other error", THIS IS SO HELPFULL... In the end I was able to find the reason why the input was rejected but in my experience the most error messages are little bit more precise.2 -
What do you use for performance monitoring on your infrastructure?
My company uses zabbix, OpenNMS and Nagios to monitor different parts of our infrastructure (from shared web hosting to OCCAS to IPTV to FutureVoice to Atlassian servers) but has no real-time performance checks.
I’ve set netdata master with prometheus backlog and grafana dashboards to monitor different metrics, however I am not sure whether any better approach could be done. Any suggestions?2 -
After the conversation, the real good way was already provided:
Prometheus exporter: https://github.com/prometheus/... (https://blog.opstree.com/2018/12/... for more details)
Overview: https://devconnected.com/complete-m...
1 -
Compare and harmonize the web configs
Oh no someone set execution timeouts to 14 days
Fuck fuck fuckity duck
Hey compare all the web configs of all environments and harmonize them all wtf cmon bruh do your job as a developer
Take them and back them up into svn. What do you mean svn isn't a back up system of course it is well its the only thing we have fuck
What do you mean we have shit logging where people will catch an exception and only print the word exception in the log you can figure it out can't you we have live produxtion issues that hace to be solved now what the fuck
How dare you make a. Mistake copying our shitload of a bloated codebase and configuring our 100s of different options all by fukcing hand what the fuck dude do yoh write anyrhing down?
Please catalogue all the exception mails we are getting but we have no db or error reporting system so they all just plop into tue inbox and thats all ypur fuckjng data figure it out kid
This is a rewarding, fulfilling job whwrw you can be both dev ops and a developer and manage all of our fucking environments of which there are about 15 of all your own with no sort of tool or software to aid you because haha what the fuck we wouldn't make your life easy
Whata that you want to spend time to write stuff or change stuff that will nake it easier fot you fuxk that bruh get back to your biklable tasks like holy shit you thjnk this is a charity ofr aomw shit
Live production issues
Live production issues
Produxtion issues. A ghost in the machine. Find it fix if find it fix it find it fix it cmon why can't you fix it I expect you to spend your day hopelessly pretending to try to solve something you fucker
One of the only peopel able to help you sometimes though hes a bit of an old laxky, yeah hea fucking leaving see ya seeya kid and now we're not hirinf anyone to fuckjng help you no no no managing and monitoring the environments its your jov alll fof them every sngle on do you knkw all the xonfiguraiton values for them yet??
Instead we are hiring a new sales person to fucking make us some more money and we don't need naother seceloper to help you infqct lets have you use this mid end retail computer from 2014 to develop on yeah yeah oh but all our shitty code and visual studip will destry your memory but too bad!! Hahahahahdhsj
Go lice is all you, why sare you so slow
How long will it take
How long will it take
How long will it take
How long witll it tqk2
How long will it take holy shit
Give time estimate for sonethign that I don't fucking know how about it will tqke till fuxk you oxloxk4 -
The usability of perfmon on windows sucks! There is just no way for me to increase the size of the lower section of the main window where all my counters are listed. This is fine if I'm monitoring only 4-5 counters, but, that is never the case. Hoping that microsoft does something about this.
-
Ok so.
You know you have to deal with annoying things when you take on a guard duty role and yes, we signed up for it because of the mullah.
However, you also want to do this with a reliable and robust monitoring and alerting systemthat you can depend on! And no i am not going to advertise a product for this... What i will tell you is which one to avoid.
Meet Quest "Foglight" ... It does EVERYTHING! It monitors, it alerts, it does trend watching it does fancy shmancy graphics, it does reporting, it is very extendable... WAUW, right! right?
Well, if you were stuck somewhere in 2005-2010 maybe... But this fucklight is cutting short on EVERYTHING
Today , i got called up at 3:30 in the morning (i am typing this after the incident) because this shit of a system has "HIgh Availability" by basically letting the FMS server suck each others jaggons and hope it somehow respons. This is a sort of keepalived thing, but on proprietary java tech..
Oh, yes, it's written on java and... yes.. Java 6
This means that, effectively we are running RHEL5 machines (yes, RHEL 5!!!) because something more modern in place? nope.
I have no idea anymore what i am ranting about, i'm tired, i'm tired of this shit, i'm tired of getting called up just because of some dude has been cussing up a sales representative, sucked each others jaggons and pushed the federal goverment with a shit solution for almost a decade now.
Fuck Foglight
Fuck Quest software, because did you really think you would get enterprise level support for an enterprise product which you payed enterprise euro's for it? You are so naive, how cute...
And consequently : Fuck Dell and Good job Dell.. For purchasing quest software, mess around with it, and then dump it back to the market... Srsly Dell , you were like me when i had this hot ass chick as a girlfriend but later seemed to be too crazy to justifiably tolerate compared to her hotness. Dump it like it's trump.
Oh, and, wauw! Foglight graced us with a successful startup process after .. what.. 6 times restarting? In 2 hours... With 12 CPU's and 128 GB ram and .... oh fuck this you don't deserve such resources.4 -
How effective are visualizations for monitoring infra on AWS?
https://reddit.com/r/aws/...
Can visual infra monitoring be effective?
Pros and cons? -
When your business network monitoring tool is so ugly that it's disgusting to say it was a person who did it.
-
One of the reasons why I wanted to become a software developer is because I see so many products or services taking the easy way out, at the cost of killing customer expectations. For example, I was told about JobTrack.io, which is supposed to help manage job searching by keeping track of applications and their statuses. But almost as quickly as I was told, my mind goes into automatic promise defense mode. And rightfully so, because the service turned out to be almost as monotaneous as the job search itself! Not as seamless as I'd need it to be to get started right away.
Now, maybe there's a slight chance I don't know wtf I'm talking about here. But, what's stopping this product from using an email client that runs server side, to interface with the user's main inbox, to run sentiment analysis on emails for detecting job application submissions? Such functionality would obviously need permission from the end user, so there are no surprises that some 3rd party app is sorta kinda monitoring your emails. And of course measures should be taken to avoid detecting anything beyond the contextual lines of: "Thank you for applying to so and so", or "We've recieved your application! Next steps".
Present those detections to the user to confirm. And do the same thing for rejections and offers. Shouldn't be that hard especially when most sites these days allow you to sign in with Google, and that Google marks these particular emails as "Important"; which further filters the detection process, and partially does JobTrack's job for them.
Honestly, I think the app has promise, and hope this is just a case of starting off small. -
I need some advice about setting up monitoring. For background I have 4 gameservers and it is already hard enough to to develop them, but I am also responsible for support and monitoring of bugs/logs and other shit.
I can't disconnect from my work anymore.
I want to setup some kind of monitoring system, that would check constantly on my gameserver mysql databases and show me charts of what's happening over time. In that case I could have a quick glance at some dashboard and wouldn't need to waste my time logging into game and checking everything.
I would also need to setup some notifications in case shit goes down.
Any advice what could i use for that?2 -
{
-i won't follow logging practices
-i won't follow secure coding
-i won't leverage profiling n monitoring tools
-i won't reuse best practices
-i won't listen to thought leaders
-i will outsource writing UT
-i will outsource code quality checks
-i will outsource all testing
-i will ignore n overide CTO team
But I still want high stability, security n 4 9s availability. Just want it done. My team is best. Am a fast-track leadership program leader who never has or ever needs to cod. I just know ...
}
People I have to deal with every sprint. Site reliability is not easy ...
Teaching good code makes great products to morons, toughest ...
"Beginners mind needed"2 -
As expected, every ambulance chasing security company is banging on my door, trying to convince me that I need their antimalware/SIEM/monitoring service because GDPR.
You guys are shameless.1 -
So this my final year as an IT student, and I need to make a desktop app for the college which is about monitoring and diagnosing network flow and connected hardware, the question is: is there any stuff that can help and will it be better to use a specific language ?3
-
Stackoverflow just got us more work to do. Now we have to redesign the entire monitoring system just when we have finished implementing HA in several DCs.
https://bosun.org/call -
Which Java GUI shall I learn in 2021? I see people are demonizing swing and telling its obsolete, so it would be really helpfull if someone gives an nice Java stack example for developing windows application focused on PLC and other external controller like arduino, raspberry etc, control and visualisation., in general, automation and industrial monitoring and controls.3
-
We've got new TV for monitoring, which auto-rotating meme page you like ? Cats, dogs, dank (sfw), dev, testing. Gimme yours !!! :)1
-
Um working on the solution to eradicating escalating diseases application and web based application..... Have three sections: 1. Emergency
2. HIV/tb monitoring/ report
3. Public info.
Iam gud at writing and codes but not good at expressing myself.
.any innovative fellow dev that can add me another section that I need to consider....my projects mission is to fight against disease world wide....any contribution or new ideas ? -
+ Taste Dartlang and Flutter
+ Do something with WebXR and/or WebAssembly
+ Start some lil projects
+ Learn more about Kubernetes and monitoring services -
So at my last job we had an AM deployment and a PM deployment. We had code reviews, QA, a slow roll process (deployed to three servers), monitoring process, and once everything checked out we fast rolled to the other servers.
At my current job we have a QA process, and we deploy once every three weeks.
My first job I deployed as needed, with no QA at all (I was the only web dev there).
I'm currently at a major e-commerce site, my last job was more of a click-bait site (though it still made millions in revenue each year).
So my question is: is there a "normal" as far as deployment schedules? I realize that each business type is going to have their own needs, but what's the "average" time between deployments? -
Quick question. I'm currently working on a project that requires extensive activity logging. What do you recommend for some sort of monitoring and logging of user access on a Linux box?
Big thanks in advance. <34 -
Asset management consulting involves providing guidance and advice to individuals, businesses, and organizations on how to manage their assets effectively. Asset management consultants help clients maximize the value of their assets, minimize risk, and achieve their financial goals.
Asset management consulting services can include:
Portfolio management: Advisors can help clients create and manage investment portfolios, which includes selecting investments, monitoring performance, and making adjustments as necessary.
Risk management: Asset management consultants can help clients identify and mitigate risks associated with their investments. This can include developing strategies to minimize losses and protect assets from market volatility.
Asset allocation: Consultants can help clients determine the optimal mix of investments based on their goals, risk tolerance, and investment horizon.
Wealth management: Advisors can help clients manage their overall financial situation, including tax planning, estate planning, and retirement planning.
Performance analysis: Asset management consultants can analyze investment performance and provide recommendations for improving returns.
Due diligence: Consultants can conduct due diligence on potential investments to ensure they meet a client's risk and return objectives.
Asset management consulting https://dataart.com/industries/... can be beneficial for individuals and organizations that have complex financial situations or require specialized expertise. By working with an asset management consultant, clients can gain a better understanding of their investments, minimize risk, and achieve their financial goals. -
9 Ways to Improve Your Website in 2020
Online customers are very picky these days. Plenty of quality sites and services tend to spoil them. Without leaving their homes, they can carefully probe your company and only then decide whether to deal with you or not. The first thing customers will look at is your website, so everything should be ideal there.
Not everyone succeeds in doing things perfectly well from the first try. For websites, this fact is particularly true. Besides, it is never too late to improve something and make it even better.
In this article, you will find the best recommendations on how to get a great website and win the hearts of online visitors.
Take care of security
It is unacceptable if customers who are looking for information or a product on your site find themselves infected with malware. Take measures to protect your site and visitors from new viruses, data breaches, and spam.
Take care of the SSL certificate. It should be monitored and updated if necessary.
Be sure to install all security updates for your CMS. A lot of sites get hacked through vulnerable plugins. Try to reduce their number and update regularly too.
Ride it quick
Webpage loading speed is what the visitor will notice right from the start. The war for milliseconds just begins. Speeding up a site is not so difficult. The first thing you can do is apply the old proven image compression. If that is not enough, work on caching or simplify your JavaScript and CSS code. Using CDN is another good advice.
Choose a quality hosting provider
In many respects, both the security and the speed of the website depend on your hosting provider. Do not get lost selecting the hosting provider. Other users share their experience with different providers on numerous discussion boards.
Content is king
Content is everything for the site. Content is blood, heart, brain, and soul of the website and it should be useful, interesting and concise. Selling texts are good, but do not chase only the number of clicks. An interesting article or useful instruction will increase customer loyalty, even if such content does not call to action.
Communication
Broadcasting should not be one-way. Make a convenient feedback form where your visitors do not have to fill out a million fields before sending a message. Do not forget about the phone, and what is even better, add online chat with a chatbot and\or live support reps.
Refrain from unpleasant surprises
Please mind, self-starting videos, especially with sound may irritate a lot of visitors and increase the bounce rate. The same is true about popups and sliders.
Next, do not be afraid of white space. Often site owners are literally obsessed with the desire to fill all the free space on the page with menus, banners and other stuff. Experiments with colors and fonts are rarely justified. Successful designs are usually brilliantly simple: white background + black text.
Mobile first
With such a dynamic pace of life, it is important to always keep up with trends, and the future belongs to mobile devices. We have already passed that line and mobile devices generate more traffic than desktop computers. This tendency will only increase, so adapt the layout and mind the mobile first and progressive advancement concepts.
Site navigation
Your visitors should be your priority. Use human-oriented terms and concepts to build navigation instead of search engine oriented phrases.
Do not let your visitors get stuck on your site. Always provide access to other pages, but be sure to mention which particular page will be opened so that the visitor understands exactly where and why he goes.
Technical audit
The site can be compared to a house - you always need to monitor the performance of all systems, and there is always a need to fix or improve something. Therefore, a technical audit of any project should be carried out regularly. It is always better if you are the first to notice the problem, and not your visitors or search engines.
As part of the audit, an analysis is carried out on such items as:
● Checking robots.txt / sitemap.xml files
● Checking duplicates and technical pages
● Checking the use of canonical URLs
● Monitoring 404 error page and redirects
There are many tools that help you monitor your website performance and run regular audits.
Conclusion
I hope these tips will help your site become even better. If you have questions or want to share useful lifehacks, feel free to comment below.
Resources:
https://networkworld.com/article/...
https://webopedia.com/TERM/C/...
https://searchenginewatch.com/2019/...
https://macsecurity.net/view/...
Top Tags
Weekly Rant
View