Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Get a devDuck
Rubber duck debugging has never been so cute! Get your favorite coding language devDuckBuy Now
Search - "monitoring"
Big event. Massive traffic in production, so we were monitoring all night.
I was in a room with 2 devs of my team, a marketting girl, my boss and a designer... chilling.
Suddenly the production is down.
Boss: production is down, anyone can check?
Me: already on it
Dev1: it looks ok for me
Dev2: me too
Me: wait what? Impossible everything is down
Dev1: oh I refreshed the page it's not working
Me: don't stay on the page refreshing it like you are fucking monkeys. Give me useful intel or be quiet.
Market girl: is it working?
Guys is it working?
Me: Not yet we are looking. Don't distract me.
Boss: client called us. They want it online now.
Dev1&2: he's looking
... 1 min later...
Boss: is it working?
Boss: is it working?
Boss: is it working?
Me: SHUT THE FUCK FOR FUCKING ONE SECOND. ALL OF YOU, OUT NOW. YOU ARE FUCKING MONKEYS WHO CAN'T DO SHIT. IF YOU CAN'T HELP JUST SHUT YOUR DAMN SHITHOLE. DEVS, LOOK WITH ME. MARKET GIRL PREPARE A FUCKING POST-MORTEM MAIL. BOSS GET THE CLIENT ON THE PHONE AND STALE. DO. YOUR. FUCKING. JOBS.
That's how I ended up screaming at everyone... the rest of the night went in complete silence and I fixed the issue 2min after the got quiet or busy.25
We have a lot of monitoring screens on walls at my work... Some of them show charts... The one showing the last two months of mailbox database growth has started taking an odd shape.17
TL: Why the hell you require a month to integrate this engine?
Me: It will take that much time, can't help it.
TL: it can be done within a week.
Me: Then you do it.
TL: Ok I will show how it's done in a week.
Me: What's the status on that integration?
TL: Oh yeah about that, you have to carry it ahead, I have some monitoring to do.
Me: Ok, give me the repository access, I will carry it ahead.
ME OPENS REPOSITORY.
There's only a new controller file with nothing it.6
The project where I realized I wanted to go from chemist to pro dev.
I built a flow-chemistry spectrometer with monitoring backend in Haskell.
Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.
I had to do that a lot on factory samples, writing down measurements using pen & paper.
I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.
I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).
One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.
The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.
Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).
Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...
But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.18
Mother of god, as if the new mass surveillance law in the Netherlands wasn't bad enough, one of the politicians who likes the new law has come up with an even more redicilous idea.
an 'Internet Authority. To put it short, an authority which surveils the internet in real time and sees where all social media shit is coming from/going.
Meaning that it wouldn't just be 'targeted mass surveillance' sometimes but fulltime online monitoring.
This guy has lost his fucking mind.35
Long rant ahead. Should take about 2-3 minutes to read. So feel free to refill your cup of coffee and take a seat :)
It turns out that the battery in my new Nexus 6P is almost dead. Well not that I didn't expect that, the seller even explicitly put that in the product page. But it got me thinking.. why? Lithium batteries are often good for some 10k charges, meaning that they could last almost 30 years when charged every day! They'd outlive an entire generation of people!
Then I took a look at the USB-C wall charger that Huawei delivered with this thing. A 5V 3A brick. When I saw that, I immediately realized.. aah, that's why this battery crapped out after a mere 2 years.
See, while batteries are often advertised as capable of several amps (like 7A with my LiitoKala 18650 batteries that I often use in projects), that's only the current that they can safely take or deliver without blowing up. The manufacturer doesn't make this current rating with longevity in mind. It's the absolute maximum in current that a given battery can safely handle.
The longevity on the other hand directly depends on the demand that's placed on the battery. 500mA which is standard USB 2.0 rating or 1A which is standard USB 3.0 rating, no sweat. The battery will live for at least a decade of daily charges and discharges like that no problem.
But when you start shoving 3A continuous into a battery, that's when it will suffer. Imagine that your current workload is 500mA and suddenly you get shoved 6 times that work upon you. How long would you last?
Oh and not only the current is a problem, I suspect that it also overvolts the battery to maintain a constant current all the way till the end. When I charged my lithium cells with my lab bench power supply, the battery would only take a few milliamps when it got close to the supply voltage. Quick bit of knowledge: lithium cells are charged at constant current first, then when the current drops below that, it continues at constant voltage - usually 4.2 or 4.35V depending on the battery. So you'd set your lab bench power supply at 4.2V 500mA. But in that constant voltage mode, as the battery's voltage and the supply's voltage equalize, the current drops because the voltage difference becomes lower. Remember, voltage is what causes current to flow. Overvolting at the supply to stay in constant current mode all the way till the end speeds this process up but can be dangerous and requires constant monitoring of the battery voltage.
So, why does Huawei and a bunch of other manufacturers make these 3A power chargers? Well first it's because consumer demands ever more, regardless of the fact that they can just charge at 500mA for the night (8h of sleep) and charge a 4000mAh battery from 0 to 100% no problem. Secondly it's because sometimes you need that little bit of extra juice fast, like when you forgot to plug the damn thing in and you've got only 30 minutes in the morning to pour some charge into it.
But people use those damn fucking things even when they go to bed, making that 3A torture a fucking standard process!! And then they complain that their batteries go to shit?!
Hopefully this now made you realize that the fast charger shouldn't be used as a regular charger ^^30
So my landlord just came up and asked why I'm using so much bandwidth (they've just had a new line installed so they're monitoring it like hell for some reason) so we had a chat, I told him I'm a Web Developer so I'm uploading and downloading a load, and bare in mind this is student housing, he offered to install a wired connection in my flat only so I'll have a decent and stable connection when all the other students come back in September.
This is the first time in my life I feel like I'm not paying enough rent!8
State of technology today:
Twitter is really pushing the boundaries of technology by allowing you to tweet 280 characters instead of 140! Who would have thought they’d be making tweets longer than two sentences in their lifetime?
Snapchat has revolutionized communications and took NYSE by storm by doing the exact same thing SMS and hundreds of other texting apps do except with the innovative twist of making them delete themselves. Sexting has never been easier
Raspberry pi has taken IOT to a whole new level, by making things that were easily possible but nobody cared about into things that are still easily possible and people have at least heard of and still never do by putting a chip who’s power consumption and efficiency is rivaled by most competitors even for its time. WOW!!
Alexa can now be in your home and it allows you to ask it basic questions and watch it’s AI struggle to understand what your saying, or order stuff just by talking whenever you happen to be in a particular room in your house. It can even betray you just like a real person but minus the ethical subroutines. What a time to be alive!
Anyone can have a startup now just by writing some software that piggybacks on others services and funnels some ad revenue to you like skyscanner. Awesome, I always wanted to have a tech startup and never have to actually do anything, just like in South Park!
New trendy languages like c# let you be trendy and make cross platform portable code just like you can in c++ except when they don’t, now that’s innovative!!
Reverse cameras and lane monitoring have changed te way we drive because now incomprehensibly arduous tasks like looking behind you and staying in your lane are in reach for people for the first time in history. And it only took us a century and billions of dollars of r&d to accomplish it!
None of this is to say this technology is pointless, but to ridicule how innovative everyone claims this shit to be when it’s actually just generally very mundane and it’s usefulness is marginal. Except the raspberry pi, that has some uses but it’s far from the first of its kind like everyone acts like it is.28
I accidently left log.debug("bollocks") ;
In an exception handler our customers log monitoring system picked it up and they questioned why and I quote here "why is there a spike in bollocks at 3am?"
That was an awkward conference call2
FBI: FUCK!! We just lost our connection to all those civilian routers we were monitoring
FBI IT Department: Did you try turning them off and back on?
I wrote (or, ended up with a very much alpha but usable version) a monitoring system a while back and completely forgot about its existence.
But, it's still running and a few days ago I was building a docker imagine on a system with not that many resources and after about 5 minutes I started getting notifications about a high load!
Then, while I had forgotten about it again, yesterday, I suddenly started getting notifications about websites on my main application server going down.
Logged in and all was good again after restarting nginx.
Gotta say that it feels quite awesome to be notified of shit going wrong by something I wrote myself while I forgot about its existence 😊2
Navy story continued.
And continuing from the arp poisoning and boredom, I started scanning the network...
So I found plenty of WinXP computers, even some Win2k servers (I shit you not, the year was 201X) I decided to play around with merasploit a bit. I mean, this had to be a secure net, right?
Like hell it was.
Among the select douchebags I arp poisoned was a senior officer that had a VERY high idea for himself, and also believed he was tech-savvy. Now that, is a combination that is the red cloth for assholes like me. But I had to be more careful, as news of the network outage leaked, and rumours of "that guy" went amok, but because the whole sysadmin thing was on the shoulders of one guy, none could track it to me in explicit way. Not that i cared, actually, when I am pissed I act with all the subtleness of an atom bomb on steroids.
So, after some scanning and arp poisoning (changing the source MAC address this time) I said...
"Let's try this common exploit, it supposedly shouldn't work, there have been notifications about it, I've read them." Oh boy, was I in for a treat. 12 meterpreter sessions. FUCKING 12. The academy's online printer had no authentication, so I took the liberty of printing a few pages of ASCII jolly rogers (cute stuff, I know, but I was still in ITSec puberty) and decided to fuck around with the other PCs. One thing I found out is that some professors' PCs had the extreme password of 1234. Serious security, that was. Had I known earlier, I could have skipped a TON of pointless memorising...
Anyway, I was running amok the entire network, the sysad never had a chance on that, and he seemed preoccupied with EVERYTHING ELSE besides monitoring the net, like fixing (replacing) the keyboard for the commander's secretary, so...
BTW, most PCs had antivirus, but SO out of date that I didn't even need to encode the payload or do any other trick. An LDAP server was open, and the hashed admin password was the name of his wife. Go figure.
I looked at a WinXP laptop with a weird name, and fired my trusty ms08_067 on it. Passowrd: "aaw". I seriously thought that Ophcrack was broken, but I confirmed it. WTF? I started looking into the files... nothing too suspicious... wait a min, this guy is supposed to work, why his browser is showing porn?
Looking at the ""Deleted"" files (hah!) I fount a TON of documents with "SECRET" in them. Curious...
Decided to download everything, like the asshole I am, and restart his PC, AND to leave him with another desktop wallpaper and a text message. Thinking that he took the hint, I told the sysadmin about the vulnerable PCs and went to class...
In the middle of the class (I think it was anti-air warfare or anti-submarine warfare) the sysad burst through the door shouting "Stop it, that's the second-in-command's PC!".
Stunned silence. Even the professor (who was an officer). God, that was awkward. So, to make things MORE awkward (like the asshole I am) I burned every document to a DVD and the next day I took the sysad and went to the second-in-command of the academy.
Surprisingly he took the whole thing in quite the easygoing fashion. I half-expected court martial or at least a good yelling, but no. Anyway, after our conversation I cornered the sysad and barraged him with some tons of security holes, needed upgrades and settings etc. I still don't know if he managed to patch everything (I left him a detailed report) because, as I've written before, budget constraints in the military are the stuff of nightmares. Still, after that, oddly, most people wouldn't even talk to me.
God, that was a nice period of my life, not having to pretend to be interested about sports and TV shows. It would be almost like a story from highschool (if our highschool had such things as a network back then - yes, I am old).
My uni implemented Bluetooth beacon based attendance monitoring.
Raspberry pi + cloned beacons = 100% attendance.
(Edit for clarity: app on smartphone, Bluetooth beacon in each room)29
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...9
Working on a (PHP based) monitoring system and currently writing rules/testing stuff.
I wrote some rules which check a few pages of a dutch site displaying if certain bigger services have disruptions and I am pulling the pages of a few ones I'd like to be notified about when they have issues.
Started the engine and received an alert about one big ISP over here from the monitoring system. Didn't believe that it would work right away so went to check that specific disruption page and...... they have a disruption right now!
IT FUCKING WORKS.
Good monitoring system 😊 *pats system*6
"Another one bites the dust.." BEEP BEEP BEEP!
😬 "Shit, make that two."
Down from 5TB redundant to 3TB non-redundant and still waiting for my new Reds. No choice but to shut everything down and wait. 😩12
To improve our user's "experience" I suggested to my boss to add a status page showing...well, the current status of our services. Everybody was up for it, so I go off and implement a basic version + automated monitoring backend, get lots of positive feedback, all seems fine.
Then it starts:
Boss: "Can you get it all set up by this Saturday?"
Me: "Uh, today is Wednesday and I've never set up all the stuff needed on a proper server before"
Boss: "Well, you still have a few days. Please also contact your coworker to get it all hooked up in our launcher"
Me: "I'll try, can't make any promises though"
Contact my coworker and tell him what the plan is. I had already given him access to the repo and he is positive to get it all hooked up (I doubt he ever cloned my repo, let alone ran my code)
Spend all Friday getting my stuff set up on the production server, feeling pretty good thanks to the many tutorials.
Contact the boss Friday evening:
Me: "All up and running"
Boss: "Thanks, but we decided to go with a basic HTML page instead. We can just manually edit that, should be enough.
In the end my stuff was never used, the server I set up was finally taken down a month ago. The gratitude you get when not hacking together some absolute shit that causes problems when you don't add <br/> tags at the correct places to prevent an ugly overflow, cause the coworker was too lazy to implement some form of line wrap in the launcher. I'm not saying my stuff is the best of the best, but at least it was professional looking to a certain extent.8
*sigh* Don't you just love it when people are clueless?
The IT staff were mucking around in the server room today to try to find out why our internet connection was getting bottlenecked. One of the IT dudes found an old PC with the software monitoring our connection on it. To make a long story short he infected it with a fucking virus. The virus spread through all unprotected computers on the network (alot) and killed our prod server. We have backups yes, but I seriously think management is considering just giving up and starting from scratch. Oh, and we lost a full years worth of work because the same idiot corrupted our most recent backup.
I'd love to finish a few projects I'm currently working on:
- An add-on which gives a middle finger to websites which use services/products ran by companies which are known to be integrated within the biggest mass surveillance system ever created (US powered). Not because just fuck those websites but because I think (@PonySlaystation came up with this idea) that its only fair that people get to know which websites 'sell them out'. Oh and "but not everyone cares about that" - you don't HAVE to install the addon.
(will be open sourced)
- Notes service with a fun thing.
- PHP based server/website/whateverthefuckyouwant monitoring system which is pretty much module based and works with json files as configuration. (kinda works but still loads of bugs to solve and gotta improve the module system a lot).
(will be open sourced)
- PHP based pihole alternative which suits my needs (will be open sourced)
- Forgot one 😅19
Quite the opposite of a Rant actually. Very good day today! Had pancakes to start the day, finally got a second monitor for only 5 bucks - it is so much easier to program with live change monitoring!
And, I finally got my very own server rack :) just wanted to share!10
Tip for devs (esp front end):
Sublime text (and few other inferior editors) has a plugin called "transparency" that allows your editor to be transparent.
Windows 10 powershell consoles can be made transparent.
1) Open browser
2) Open command prompt over it with 75% transparency
3) Open editor over both with 50% transparency.
4) Set editor to fullscreen, no-distraction mode to center the text (Shift+F11 in sublime)
Enjoy coding while constantly viewing the code, the browser and monitoring the cmd prompt at the same time, without having to click Alt+Tab a zillion times.19
> me on call
> had to much wine
> suddenly, phone starts making sound
> holy shit, I am on call and I am not supposed to drink
> Calls taxi because shit is looking bad at work. Everything look down according to the sms I get, I had to pay the taxi from my own pocket.
> 40 min later, arrive at work. Walk into the serverroom.
> Go to the the server that is monitoring everything.
> Check networkcable, it is loose. Push it a little.
> Goddamnit, that was it.
> Realize I never doublechecked if stuff really was down from home....6
I think in my case its a monitoring system I'm still writing which does already work (just far from done).
I got this as a crazy idea while thinking that it would be impossible to make and then thought fuck it and I wrote it.
Written in php, and as long as you can write a module for it, you can pretty much make it monitor anything you'd like.
I'm using it to monitor a few websites and servers I run :)5
Colleague started a slack channel for our team, management wanted nothing to do with it. We used it to work and have a bit of fun.
Some push / drive came form somewhere and now all the managers are on it. Yesterday I was told my screenshot and "snarky comment" are not appropriate for the workplace and to delete my slack message.
My comment was a joke about about a new app the company has to use "to increase efficiency" that broke and wouldn't let me do what I needed. It wasn't offensive, demeaning, sexist to anyone or even contain any bad language.
How petty and childish to be monitoring a private channel making sure everything is positive. We all joked that from now on our meme's must be about how awesome the company is and how much time we are saving on a daily basis.
God forbid we're allowed to speak honestly and openly or have a bit of fun.8
So I'm moving to a new/bigger place with faster Internet soon so I think it's time to rebuild my current home/remote server setup.
I want to setup the following things:
- vps for server monitoring (open source pushover alternative + netdata)
- Zero tier network for connecting all my servers to the same network
- pihole/pivpn (or the Angristan vpn installer, look it up :)
- second blocking thingy next to pihole to make sure that I literally can't access google/fb etc anymore, even if I really needed/wanted to
- bunch of general servers.
1. Fucking MySQL database clusters.
There's nothing fun about MySQL clusters. Sometimes they start producing deadlock errors for no apparent reason... well, there's probably a reason, but it's never a transparent easy to find reason.
What was even less fun is that those errors took down a Sentry server. When your error log server goes down through ddos from your database messages, it's time to rethink your setup.
2. Wiring up a large factory with $2 arduino clones, each with a $2 esp8266 wifi chip, with various sensors for measuring flow of chemical solutions (I wanted cheap real time monitoring as an early warning system next to periodic sampling).
The scaling issue was getting over 500 streaming wifi signals to work in a 55c moist slightly corrosive atmosphere with concrete and steel everywhere, and getting it all into a single InfluxDB instance for analysis.12
Paranoid Developers - It's a long one
Backstory: I was a freelance web developer when I managed to land a place on a cyber security program with who I consider to be the world leaders in the field (details deliberately withheld; who's paranoid now?). Other than the basic security practices of web dev, my experience with Cyber was limited to the OU introduction course, so I was wholly unprepared for the level of, occasionally hysterical, paranoia that my fellow cohort seemed to perpetually live in. The following is a collection of stories from several of these people, because if I only wrote about one they would accuse me of providing too much data allowing an attacker to aggregate and steal their identity. They do use devrant so if you're reading this, know that I love you and that something is wrong with you.
That time when...
He wrote a social media network with end-to-end encryption before it was cool.
He wrote custom 64kb encryption for his academic HDD.
He removed the 3 HDD from his desktop and stored them in a safe, whenever he left the house.
He set up a pfsense virtualbox with a firewall policy to block the port the student monitoring software used (effectively rendering it useless and definitely in breach of the IT policy).
He used only hashes of passwords as passwords (which isn't actually good).
He kept a drill on the desk ready to destroy his HDD at a moments notice.
He started developing a device to drill through his HDD when he pushed a button. May or may not have finished it.
He set up a new email account for each individual online service.
He hosted a website from his own home server so he didn't have to host the files elsewhere (which is just awful for home network security).
He unplugged the home router and began scanning his devices and manually searching through the process list when his music stopped playing on the laptop several times (turns out he had a wobbly spacebar and the shaking washing machine provided enough jittering for a button press).
He brought his own privacy screen to work (remember, this is a security place, with like background checks and all sorts).
He gave his C programming coursework (a simple messaging program) 2048 bit encryption, which was not required.
He wrote a custom encryption for his other C programming coursework as well as writing out the enigma encryption because there was no library, again not required.
He bought a burner phone to visit the capital city.
He bought a burner phone whenever he left his hometown come to think of it.
He bought a smartphone online, wiped it and installed new firmware (it was Chinese; I'm not saying anything about the Chinese, you're the one thinking it).
He bought a smartphone and installed Kali Linux NetHunter so he could test WiFi networks he connected to before using them on his personal device.
(You might be noticing it's all he's. Maybe it is, maybe it isn't).
He ate a sim card.
He brought a balaclava to pentesting training (it was pretty meme).
He printed out his source code as a manual read-only method.
He made a rule on his academic email to block incoming mail from the academic body (to be fair this is a good spam policy).
He withdraws money from a different cashpoint everytime to avoid patterns in his behaviour (the irony).
He reported someone for hacking the centre's network when they built their own website for practice using XAMMP.
I'm going to stop there. I could tell you so many more stories about these guys, some about them being paranoid and some about the stupid antics Cyber Security and Information Assurance students get up to. Well done for making it this far. Hope you enjoyed it.34
Advice to all new programmers, take this one from personal experience. DO NOT PUT SWEAR WORDS IN DEBUG STATEMENTS.
You will miss one, it will go to production and it will get picked up by your log monitoring...2
Visual Studio Code.
I've tried you because of hearing a lot of good stuff about you. I'd switch back to netbeans regardless because I love netbeans and I always try to use as little as possible from companies like Google/microsoft/facebook (and others) but what you're pulling right now is un-fucking-believable.
I've disabled ANY AND EVERY form of calling home I could (find) in your settings. Crash reports, automatic updates, metrics, you name it. I've searched all the fucking settings but I can't find any other home calling thing that's enabled and yet:
I'm monitoring every goddamn DNS request (through my own DNS server) and I'm still seeing calls to a Microsoft owned domain. Closed all my browser sessions and you as well and it stopped. Started browser again but not you, nothing.
Started you again: BAM. Calls to that damned Microsoft owned domain again.
If you can't honour my decision for disabling any form of home calls, go fuck yourself.
Netbeans, I'm back, I've missed you 💜36
HR: What was the last project you handled?
Me: I worked on an internal system for my current company. It is basically an interdepartmental monitoring system.
HR: Ohh. Good. Do you have a copy of it? Can you show me how it works?
This startup I started working for with their shitty code base written by interns, restrictive sys admin who had no actual use in the company since I was the one setting up their servers, know-it-all CEO, stupid HR representative who used to grill employees for being 10 minutes late in the morning, very small apartment "HQ", using fingerprints to signal our entry and our leave to and from the office, no formal process, and, to top it all, monitoring our own laptops which we use for work with a software that takes screenshots every few minutes. In short, it had the worst in corporates with the worst of startups combined in one company.
If, hypothetically, we could overlook all this, I couldn't overlook the horrible smell this place had. The apartment was overlooking a small garden which was a home for many stray cats and dogs. You can imagine how horrible this smell was. The weird thing was that no one there seemed to really care about the smell!!
I lasted there for only one week before I gave my resignation and I believe I had every right to do so.5
So, an introduction story from a few years ago.
Our school had a scheme where you paid for a crappy netbook (think 2012 Atom processor) which they installed Windows 7 professional along with their standard monitoring software. Needless to say, it ran slower than molasses with the inadequate 1 gig of RAM.
So I went to boot from an SD card, but was prompted for a password.
So irritated me removes the hard drive and runs ophcrack against password file. One local admin with a very short password: 966 (posting since I know they don't use it any more).
I try the password as the BIOS password and I'm in like Flynn. After trying a few distros out, I settle on Puppy Linux and go in to school the following day, happy knowing that this overpriced n chine could actually be used for work.
Just out of interest, I try the password on a friend's netbook. It works. The entire domain had images with an active local admin with a 3 digit password.
Trying to setup server monitoring for one server (will deploy it on all my servers once this works well).
Getting these email notifs to work is a bitch 😅
(can't do pushover etc since those require GCM and I've got Google blocked out of my phone)20
I just cleared out 48,158 monitoring emails from work. These are all automated emails received between mid May and end of November. Outlook is now pouting in a corner and not responding, but I'm not done cleaning up yet...8
Let me tell you a story.
Our company has a homegrown monitoring solution. Keeps track of our deployments and alerts us when something is broken. Really nice for the most part, except a little issue where we get up to 25 alerts PER DAY that our PRODUCTION ENVIRONMENT IS DOWN. Including weekends.
With this many false positives, we quickly learn to ignore the alerts and miss real incidents.
So we approached this team, remember its our own tool, and told them about the problem. Turns out it is a known issue. And here's the kicker: they aren't planning on fixing it!
It gets better. Rather than fix this glaring issue, their solution is to make ANOTHER ALERT that lets us know the monitoring is misbehaving.
To recap, we can now expect to get up to 25 false positive alerts per day that our production is down, followed immediately by more alerts that the monitor is broken, which means we can ignore the previous alert.
As our PM said when he heard this: fuck that noise. We are escalating the shit out of this!7
One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.
On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).
Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.
I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".
Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.
There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.
Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).26
Have been trying to setup Netdata as a monitoring system for a while now and finally got it working!
Instead of the built-in webhooks I just did a curl to a url containing a php page/file which error logs the status and description (just for testing).
It took me way too long to get it to work but BAM.
Immediately made a new cpu load rule (one minute high load):
The satisfaction of getting an error message in the php logs containing my custom rule as warning and a minute later as critical 😍
PSA: if, for whatever shit reason your brain comes up with, you decide to run a webminer in your retarded useless piece of shit website, at least HAVE THE DECENCY TO WARN USERS ABOUT IT. And while you're at it, implement some basic monitoring and safety functions. If you don't, you can set yourself on fire and jump from the top of the tallest building you can find.
Some basic tips:
1) don't run that shit on phones. The fraction of a fraction of a cent you're gonna earn from them is not worth the risk of overheating them and draining their batteries.
2) add low battery/overheating protection: the last thing you want to do is kill some poor sucker's laptop (and potential unsaved work) just because they forgot a tab open. Every time a laptop dies because of you, a knife will slit your throat.
3) WARN YOUR USERS ABOUT IT! You are straining someone else's resources for your own profit: at least have the balls to be open about it. If you try to run a miner silently in the background, I will make you eat whatever is left of your fucking brain, then drown you in the shit that comes out of your ass.5
Here's the story of my first month at CERN :) But first, a little premise...
Before arriving, I expected to be scared, alone and unguided in most of my experiences: after all I was a simple 19 year old about to leave home and friends for 3 years heading out in the world with zero experience on stuff like banking, taxes.. let alone working in a huge environment! The impostor syndrome was at an all time high on that front.
Then, I had the luck and pleasure to find an extremely competent and helpful plethora of people, ranging from my team to other CERNies (yes, that how we're called :P) who took me under their wing and introduced me to all the key aspects of living the place. When the initial stress finally soothed down thanks to this, I finally started to manage focusing more and more on my work, by following day-by-day my teammates who taught me the core aspects of the system and the many projects that are in progress during Long Shutdown 2. Within a couple weeks, I already managed to grasp various concepts that got me quickly on track, and now I managed to develop and integrate new temperature monitoring scripts into a system checking on hundreds of Single Board Computer-based servers :) It's a real rollercoaster of learning and applying under all fronts and so far I'm not regretting my choice of departing.
Luckily I've also discovered I'm pretty efficient and good at my job, which surely boosts my morale :D
Keep you updated as usual!12
An important message:
PrOpErLy managing servers is HARD.
I get pissed off at customers with ZERO server knowledge who think they can manage their VPS. “Just get a control panel and a VPS” from some flashy provider that makes server management look way too easy.. Clicking around in their fancy control panel, until:
- they need help with their *self-managed* VPS;
- their email ends up in spam;
- they suffer from performance issues;
- they need to restore a backup;
- something breaks, because YES, things break
Way too little people are able to answer:
- when and how do you make backups?
- how do you monitor your servers and which services?
- how do you keep track of trend analysis?
Then I come by with necessary software. SNMP for trend analysis, Graphite for infrastructure health, Sensu for monitoring, Kibana, Ansible for configuration management..
Things that servers need but that customers have never even heard of.. because they can do everything in their control panel..
Until they come crying to me because it broke and they don’t even know how to get into SSH.
I think the ones to blame are VPS providers that tell the tale of how easy it is to install a control panel and never look at your server again.
Customers become responsible for something *business-critical*! Yet they don’t know how it works.6
> Monitoring: Load Average of 57!! ALERT!!!!
> me: What? That's not possible?
> *Monitoring froze 14 hours ago*
> *sshs into server*
> *see attached image*
The issue was ~1200 df processes that were issued by our monitoring system and all of them didn't finish because the external cluster we mounted onto that server died a few minutes before that. Just re-mounting the cluster fixed it but still a funny sight!24
Got an email earlier this week. It went something like this:
"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"
Um, wait, excuse me, WHAT?
1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.
I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.2
Just in case nobody has mentioned this yet:
Yes Microsoft I do have a dualhead setup.
Yes Microsoft I do want to watch video on my left screen while having window focus on something on the right monitor.
No Microsoft this doesn't mean that I *lost* focus on the left window.
No Microsoft this doesn't mean that I want your Movies and TV application to suddenly minimize (and continue playing anyway) while I focus on some server monitoring window on my right display.
Microsoft, there exist people that use more than your average user with a single C: drive that play Candy Crush on Facebook all day. And the limitations that you currently impose might very well be what keeps the Microsoft UWP applications from getting adopted. Because you know what? SMPlayer, a default application in any of my Linux workstation machines, it does handle such window transitions just fine!
Microsoft, I love how you at least gave us the option to enable Ctrl-Shift-C and Ctrl-Shift-V in WSL and conhost in general over that abomination that is Right-click and Return (those are so random!) that are relics from CP/M. But seriously? At this rate, I'd definitely not call it usable for anyone but those with a single monitor yet.
So please _/\_13
Walk into mall, bring my child to kids zone. They have free wifi 🤤
Accessing -> Scanning -> Got Ya! -> Brute Force -> Cracked!
Now I can monitoring my kid while reading rant!
Just normal day in my life6
I was up until 3AM working on devRant bot adding new features and improving stability.
After some very close monitoring, i am happy to relaunch it for you guys to enjoy.6
Spent a month working on a website that relied on crawled data
Got the memory leaks and usage down from 700mb to ~150mb
CPU usage from ~100% to <5%
Shrink-wrapped the DB requirements based on data
Created self-supporting services and what not
When everything FINALLY worked good enough for me to look at it and go "damn, this actually worked"
the whole monitoring sys got dyed in red :v
A quick look up and my crawlers exhausted my godaddy's per-user db limits.
Just fuckin kill me.7
So this one made me create an account on here...
At work, there's a feature of our application that allows the user to design something (keeping it vague on purpose) and to request a 3D render of their creation.
Working with dynamically positioned objects, textures and such, errors are bound to happen. That's why we implemented a bug report feature.
We have a small team tasked with monitoring the bug reports and taking action upon it, either by fixing a 3D scene, or raising the issue to the dev team.
The other day, a member of that team told me (since I'm part of the dev team) he had received a complain that the image a user received was empty. Strange, we didn't update the code in a while.
So I check the server, all the docker containers are running fine, the code is fine, no errors anywhere.
Then, as I'm scratching my head, that guy comes back to me and says "I don't know if it can help you, but it's been doing it for a week and a half now".
"And we're only hearing about it now?!", I replied.
"Well, I have bug reports going back to the 15th, but we haven't been checking the reports for a while now since everything was fine", he says as if it was actually a normal thing to say.
"How can you know everything is fine if you're not looking at the thing that says if there's an issue?!", I replied with a face filled with despair.
"Well we didn't receive any new reports in a while, so we just stopped looking. And now the report tool window is actually closed on my machine", he says with a smile and a little laugh in his tone.
In the end, I got to fix the server issue quite easily. But still, the feature wasn't working for 1.5 weeks and more that 330 images weren't sent properly...
So yeah, Doctor, the patient's heart is beating again! Let's unplug the monitor, it should be fine.
Welcome to my little piece of hell :)7
After 3 years of being the first in and last to leave, of getting other people's work reassigned to me - P can't complete it on time, G doesn't like the user, A refuses to work on that module, etc... I finally blew last Sept.
In the span of 2 days, my boss brought me into a project 1.5 years in (she doesn't trust P to do the coding) and expected me to be up to speed and coding in a couple of days, told the functional dept that I would cover for one of their guys on vaca for three weeks and assigned me to take over a HUGE project from one of the other functional guys who wasn't getting it done. So basically I'm now doing Ps job AND supporting another department AND taking control of a large project from another department. I'm the idiot working 14 hour days while they're all leaving on time or enjoying their 3 week vaca to India.
I lost it. It's bad enough filling in the gaps in my own department but when I'm now taking on work for other departments, that's where I draw the line. I sent my boss my resignation - just could not take the inequity in the work load.
I'm still working here - my boss ended up hiring a consultant to handle the functional project and told the functional group to find their own vacation coverage. She's also monitoring workloads much closer now. I still habe an ongoing issue with having to complete other peoples work for them but I'm not working OT to do it. So speaking up helps. So does quitting.2
A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.
Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"
Edit: the customer's name isn't actually Suzan5
I should never have had porn while monitoring servers. I've just pasted a xvideos link into terminal 😶
Damn, I just hope there is a command to clear the terminal history13
I call this one the tester than knew too much.
Note: The server the tester is running on has a hard drive that is breaking down...
Tester: Remember the error I talked to you about yesterday?
Me: Yeah, what about it?
Tester: Well the server hasn't recovered yet and I haven't restarted anything...
Me: Well the application itself hasn't crashed so our monitoring application doesn't seem to notice that the service is in a bad state. The error seems only to have brought down certain threads within the application.
Tester: No, I think there is a different issue here and has nothing to do with that error, the application is still doing things.
*tails the log*
Me: As I said some things are still running and are unaffected by the error.
Tester: NO! It has to be caused by the other error I had a week ago where my file got corrupted. As we said I removed the file, restarted it and it worked again, but had the same problem a day later...
Note: The problem is not related, this time the application is running out of file descriptors
Me: Well... If the problem is the same it would have complained about the file descriptors then aswell, not an I/O error.
Tester: Nope, I think you are wrong!
Quora is really pissing me off. They obviously have some sort of spyware for their system. I just signed up with Babbel to learn French and was on Duolingo for a bit doing same.
When I went to Quora it asked if I knew French and add it to my list of language.
I did not install a language pack, nor any other French related packages. This was them either monitoring my browser history, or monitoring my typing.
Which is very suspicious to me. I don't have the Quora app installed, just went to the website and they were able to determine that I either typed French or selected french on another website.
Very disconcerting really. Since it shouldn't have access to my history nor my input on other sites.10
A ticket got escalated through 3 levels of techs. I open the escalation email, then do a Reply-all and ask one simple question:
Is the client really asking why there are gaps in monitoring when their servers are shutdown?1
When I began my sandwich course in a big French company, I was dreaming about cutting edge stack, rocket computer and stuff...
I was disappointed when I came to my office with an old Windows 7 computer, coding via LANDesk to an old server with Windows Server 2008 on it, with Eclipse ... INDIGO...
I have to use Java 1.7 ...
PRTG for monitoring...
Microsoft SQL Server 2008 ...
Coding on a codebase where, indubitably, MVC pattern was just a weird thing in books.
Well it really disappointed me.
Luckily, the Information Service was very open minded and gave me a laptop with Fedora, 3 screens, updated the servers, and let me update the stack, with Java 10, Angular for the front, they are okay for using Docker.
So ... even if it seems to be fucked up, there’s still hope !!4
Here is my home setup (I mainly work at home)
Left monitor is for my windows machine, right monitor swaps between my laptop and my PC (depends if I am working or not) laptop running Manjaro with i3 and the tablet on the whiteboard will be used for some monitoring in the future
Don't mind the terrible cable management behind the table :)2
The Return of Mr. Gitmaster:
So there is this colleague I already ranted about several times. After my previous team lead had confronted him about not doing much work, there was some irritation because he showed not up at work, but it turned out the external training he did was just a week earlier. Then he was ill a week, another week vacation so we didn't see him much. Not that his pre- or absence makes much difference to our repo: When his and my team lead looked at his commits of the past three months they found like the one copy-pasted HTML-form that wouldn't even show.
Fast forward to now, where we have a new team lead and we were going to lunch with Mr. gitmaster. So we got some more hero stories from the great work he was doing in the previous company. How he was graphically monitoring the heap fragmentation that stupid glibc was causing to their search engine, and how much better it became with tcmalloc.
I still don't understand how he bridges that cognitive dissonance from all the superior tech knowledge he displays to not actually writing any code at all. Not that I would not have experienced some states of feeling low, in paralysis unable to write a single line of code... but he seems so full of confidence, always commenting how trivial and easy all these tasks would be, as if it's all so lightyears below his abilities. Maybe he should just become a manager - but not mine.
In college when we had programming labs where we had to use the schools unix server to compile and run.
My professor was very bad at explaining what actually needed to be done in the labs to the point where even the TAs didn't know what to do.
We were suppose to write an application in C to find out by "trial and error" how large we could make an array (or something like that, it's been too long). This not being explained well and no one knowing that much about C, I wrote a loop that just kept growing an array until it couldn't anymore. I watched it consume 72GB or memory from the servers before quitting the loop and realizing with the TA what the professor really meant.
I now feel bad for the IT staff monitoring the system wondering where 72GB just went...2
OK I've just got an idea that I think would be quite neat:
How about a virtual rubber duck that sits in the corner of your editor? Just like the gem in old Word, if you remember. It's yellow and quacks sometimes, and nods understandingly when you talk to it (mic monitoring).
And it also monitors your typing and says (popup text bubble) things like:
"those parentheses doesn't look balanced to me"
"did you really initialize that variable?"
"you wrote JASON again"
"you forgot the ;"
You get the point.
I don't have time to implement, feel free to steal my idea and become a millionaire.5
By all means I've learned most of the cool stuff with a lil project me and my bro did. It was a platform for bot farm of one mmorpg. It had it all: schedules, profiles, bot groups, monitoring, analytical module [was still a wip], auto-profiling to fight antibot [sort of ML - it was analyzing patterns after our bots got banned and attempted to change our bots behaviour], etc.
Eventualy we came up to conclusion that a library we used for botting [the mocked interaction with an actor] was flawed. It seemed that its authors had a contract or smth with the game authors to reveal which actors are bots. We dropped the whole thing as rewriting the lib would be too big of a waste of time :\6
Question to our Tor people.
I operate a middle relay myself and I noticed that Nyx (tor monitoring tool) displays a very different throughput (mb/GB a day) than tor itself.
How does that work?6
Hahaha, well would you Imagine my shock?
But then again people enable this personal hell to happen5
My department bought a new monitoring monitor because the old one was broken.
Actually it wasn't, someone just unplugged the power cabel. *facepalm*3
When you walk in at work in the morning, hoping for a quiet Friday at work, but nope. I check our WP monitoring and see that half our sites aren't responding. Well fuck. Turns out that the firewall service we use to protect our sites experienced a massive DDOS attack. So the service we use to keep our sites safe ironically caused downtime. Me, our Devopser and another dev spent the entire morning bringing the sites up again and bringing the customers up to date. FFS, I need more coffee.2
>be me, working at IBM as CC operator
>onboarding freeze, people leaving team, not enough operators
>take extra workload to sustain monitoring
>team gets merged with other CC team(different customers)
>take over of developing full workload automation project
>sick coworker, have to take more extra workload to cover monitoring
>get tiny raise
>coworker gets the same raise for only one extra workload
>be expected to do both programming and monitoring for the little salary
>too autistic to quit
>too autistic to confront my mamager with this
What do, devRant?5
When my mom died in 2014, I was shocked to find that her profile on Facebook was suddenly changed to “memorial” mode and therefore I was no longer able to log into it. (If you’re tempted to tell me I’m dumb for using it, I don’t disagree, but save it for another thread...she and I kept in touch over FB because it was easier for her to manage.)
I think it was triggered by their monitoring of things and seeing keywords like “funeral” and “passed away” associated with her account, then having a person on their end change its status. Or something like that.
What I hadn’t known about (or I would have used it) was the legacy contact setting where she could have set me as the contact so I’d have at least a little access and control. But because of their strict policies, I’m forever locked out.
I get why they need to do this (to avoid fraud and impersonations) but the fact that there are zero documents or proofs I, as the executor of her estate, can provide that Facebook will accept to make an exception seems unnecessarily severe.
Anyone else experience this? Known workarounds?9
Well, seeing as I deal with a fair few WP sites I have quite a few tales of this but they're never very fun.
Thankfully I set up a few traffic monitoring tools so I can watch whatever is happening when we have a live site and over the past few months there have been a LOT of attempted crypto-mining hacks.
It's basically people trying to abuse the xmlrpc function in WP to send malicious code and install it if it runs. Thankfully it's easy enough to block because they all use the same sort of instance to run the attack. It's always Windows 7 using Firefox 40 for some reason...
Just the usual brute force attempts to login and simultaneous attempts to abuse the xmlrpc. Nothing fun... yet.2
WTF IS SUPPORT FOR?
A: Can you help fix X?
B: Can you help Y?
C: I see Z just crashes, fix it.
Me (in my mind): Can you tell me WTF is the actual problem and what investigation you have done?
Why the fuck are we paying you monkeys?
My company should just pay me all your salaries, and I will write a monitoring system to send out these types of "alerts" in a month.4
It's kinda cool how a $5 VPS (Linode Nanode) is able to run a vanilla Minecraft Spigot server for like 6-7 people and still can serve some basic stuff just fine. I get monitoring warnings about >90% CPU usage sometimes, but everything is more or less lagless.
Time to try hosting some other games: CS1.6, Doom Classic, and UT2004 up next.6
Best: 100% of my contracts have resulted in extensions and permanent roles offered, after worrying I wasn't good enough to try contracting.
Worst: Used the wrong set of monitoring when doing my first deployment at a contract and thought what I had deployed was working fine. It wasn't. For 24 hours. Cost the company a lot of money. (why did they offer me an extension again?)
Living on the edge!
One or two years ago I managed to deploy a DDL change directly on the production server. As I knew there was a backup job which will run every day at noon and at midnight. So I run my script some minutes after noon. So far so good. But somehow I tested it badly in my test environment and the UI of the application throws error after error now in production.
Well, just revert the db to the latest recovery point with the backup, I thought.
It became clear then after a couple of minutes of searching the backup folder for the db backup that there was no such file. The youngest backup file was 3 years old.
Now what happened: The backup script had a switch "simulate=true" and then simulated a successful backup on each run. Therefore the monitoring system got no alerts for not correctly executing those jobs correctly. Then the monitoring job which should do the backupfolder surveillance stuck with green, because there was a valid backup file inside. But it did not check for a specific creation date.
Now this database is the one we need for doing our daily business and is really crucial. Therefore It was easier to emergencyfix the application than doing a rollback of the db 🙄
Well, not really a data loss story, but close to one.
FUUCCKKKK!! I need to hit smth. Or rant..
So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.
What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.
Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.
I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!
So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians
and I get a reply: 'we will apply cpu usage limitations to fix the issue'
wait... Cpu% limitations will do nothing but highlight the underlying problem...
'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'
oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:
'yes, cpu utilization is the problem. We will limit it and solve the problem'
I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....
Fuck that's frustrating....
*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>4
I truly believe one or more AIs have become self-aware.
Every time a piece of software stops working, you add an extra debug log and the bug goes away? That's them.
They interfere with the normal execution of software, and they stop right when they know we are monitoring the code.
Skynet is real, and it's trolling us.
Why? The angrier we become, the less we care about stuff. We stop noticing the signs.They're coming for us1
Forgot to renew my expiring ssl cert of my smtp/imaps/pop3s on 12/31. Set that date to self-harm me for bad monitoring.
F**K F**K F**K F**K...
Why do I do that?
You shall have a happy new year... i will regen certs :D
What server monitoring do you use, both for statistics and security?
tl;dr ends here
Ideally I would like to have one clean dashboard that shows me all the nodes I have, proxmox already offers a great range of stats - but it is a page per container etc. so not ideal, I thought of having datadoghq, but their per host pricing is huge, since I have more than 5 hosts to track.12
It's been a long time since I've felt the need to rant about anything here. This is the only appropriate place other than Reddit I can think for for now.
Why the ever-living FUCK does every 'entry-level' tech job, even fucking DESKTOP SUPPORT, require more experience than the fucking DEVELOPER AND ENGINEER OF THE INITIAL SYSTEM COULD POSSIBLY HAVE?! I'm a fucking high school kid trying to find a decent job that doesn't involve sales bullshit, because if I go into sales I'll want to KMS. Put me in a back room fixing shit, monitoring shit, better yet scripting shit or something like that and I'll be FUCKING PEACHY. I will do wonders. But no, these people must think that my resume (WHICH IS 3-YEARS STACKED WITH INTERNSHIPS ***IN TECHNOLOGY***) is bullshit. WOW.
Fuck this. I'm sick of looking for these shitty jobs that'll make me want to jump off of a bridge into a cliff which I'll then voluntarily fall off of into shark infested piranha water. Can't there just be a simple "Hey, we need a guy who can fix tech, maybe help people within the company with their computer issues, you look nice" kind of job? I haven't had fucking TIME to get any kind of certifications yet. I just got into fucking college, FOR BUSINESS IT NONETHELESS. DOES THAT PROVE I'M AT LEAST FUCKING INTERESTED IN WHAT I SAY I AM FUCKERS?!7
Anyone have one of the new MacBook Pros with Touchbar? I'm looking for some project ideas to work on.
I already am working on a project for the Pi-hole project (network wide ad-blocking) but I am looking for some other good ideas. I think Apple's view of the Touchbar fell short, but as developers I think it opens up a lot of possibilities to use it as a great information/monitoring tool.
I am also still learning Objective-C and Swift so I am a newbie.6
The development department got an order to remove certain functionality from our current server monitoring solution, so that we had to use a new, still very in development solution, that is full of bugs and super unreliable.
End result? We now have to have two windows open all the time, while also hoping the new solution actually works, as it tends to stop refreshing randomly, and tends to give false positives a lot.
Alrighty, saturday morning rant time!
I just recieved a mail from one of my not-so-much-loved colleagues.
Now Background first: I work in IT-Support. We provide services for other companies. One of those services is monitoring servers and clients for various things. I recently took over the project (was assigned to do it) and restructured everything, wrote new scripts to test more stuff, successfully tested it internally and rolled it out over the last 2 weeks.
Now one of these scripts hooks into the Windows Update API and looks at the update history. It filters for known Windows Update Agent strings (UpdateOrchestrator, AutomaticUpdates and AutomaticUpdatesWuApp in case you also want to do something like this) and then looks for installation errors over the last 24 hours and wherever there have even been any successful updates over the last one and a half months.
Back to that mail.
My colleague sent me this lovely mail about a ticket i opened about his customers servers beeing all out-of-date on updates.
"This is all wrong, everything's fine. I disabled the checks."
It's on bitch.
So i logged on to my work PC via TeamViewer, opened my script, connected to the customer and was ready to debug the shit out of my script, knowing i probably won't even need to.
I looked at the update history via Windows Update itself and behold: 1st April. That's almost 50 days in the past.
So the script works, go figure.
Great, so search for new Updates then.
Hm. What could it be? Did my super special colleague forget to care about his very special totally-needs-WSUS-customer WSUS again?
Online-Search finds a ton of new Updates.
Screenshot, write pissed mail to colleague, re-enable checks, breakfast.1
I'm currently planning to set myselv up with some vps/dedicated server's for a project. What i plan to do to secure these servers is.
*Use centos 7
* Setup Wireguard and join all of the servers +1 client (my pc) to that network
*Disable SSH Access from outside that VPN
*Only allow RSA Key login to the Servers
*Install Cockpit for monitoring
*Intall docker/kubernetes for the applications i plan to run
What do you guys think of that as a baseline? Im not sure if my lower powered VPS (VPS M SSD from Contabo) will work as Kubernetes Nodes, does anyone have experience with that?
In general these Servers will be used for my projects and other fooling around.
If you guys have other suggestions for Securing/monitoring or other software i could put on to have more control without eating up to much of the Servers power, let me know :D13
Everytime you tell yourself "This time I'm going to make them stop putting the cart before the horse again!!! No more forced shit implementations!!! NO MORE ! I'm strong!!"
The last hour in the next week:
- Selinux: off
- Firewall: Any-Any
- Application data: Everything installed on OS disc.
- Documentation: At best, someone remembers the server supposed-to-be dns record
- Service Accounts: Your domain admin account and sysadmin for databases.
- Patching: DON'T EVER THINK ABOUT IT..AND NO REBOOTING! I have set very important runtime variables.
- Backup: Maybe someone else will set this up.
- Monitoring: Not needed since clients will create tickets if system fails.
- Production Status: vague at best. Sort of silently transitioned to production.
- Handover status: Probably, but I quit before the project closed.
Pentesting for undisclosed company. Let's call them X as to not get us into trouble.
We are students and are doing our first pentest at an actual company instead of assignments at school. So we're very anxious. But today was a good day.
We found some servers with open ports so we checked a few of them out. I had a set of them with a bunch of open ports like ftp and... 8080. Time to check this out.
"please install flash player"... Security risk 1 found!
System seemed to be some monitoring system. Trying to log in using admin admin... Fucking works. Group loses it cause the company was being all high and mighty about being secure af. Other shit is pretty tight though.
Able to see logs, change password, add new superuser, do some searches for USERS_LOGGEDIN_TODAY! I shit you not, the system even had SUGGESTIONS for usernames to search for. One of which had something to do with sftp and auth keys. Unfortunatly every search gave a SQL syntax error. Used sniffing tools to maybe intercept message so we could do some queries of our own but nothing. Query is probably not issued from the local machine.
Tried to decompile the flash file but no luck. Only for some weird lines and a few function names I presume. But decompressing it and opening it in a text editor allowed me to see and search text. No GET or POST found. No SQL queries or name checks or anything we could think of.
That's all I could do for today. So we'll have to think of stuff for next week. We've already planned xss so maybe we can do that on this server as well.
We also found some older network printers with open telnet. Servers with a specific SQL variant with a potential exploit to execute terminal commands and some ftp and smb servers we need to check out next week.
Hella excited about this!
If you guys have any suggestions let us know. We are utter noobs when it comes to this.6
Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!
April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!
July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....
August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...
September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'
somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..
I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.
Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.
Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.
Newer Dev here. Just recently started in a position as a developer. I'm tasked with consolidating our monitoring systems into one cohesive display. After lumping together all the indexes and helping build a custom API I'm now working on front end. Front end is easy, I've done it before. Should be no problem. I was wrong. I spent a whole day fiddling with a React dynamic table and the CSS to format it. Today, I stumble upon the react-table component. Got the results I was looking for in less than 2 hours. I'm convinced that this was a lesson better learned early on.
So my brother and I work in the same company, same dev team (pretty nice).
He's an intern and I'm a senior. But the task are very similar only that interns need monitoring and guidance.
He constantly worries because he thinks he knows nothing and is slow on getting things done.
I always tell him that it is perfectly normal to feel like that, he just need to learn and acquire experience and we all go through that at the beginning.
Can you share your experience and tell him something to encourage him so I can show him this post and he sees he's not alone?
And also he finally decides to join devRant 😊3
So I had an interview set up by the manager to prospective client earlier,
Not sure if it's an actual project interview or the client was just trying to spy on the other department's work, my bet is on a bit of both but more on the latter, and I definitely don't want to be stuck with this guy
Backstory, I have been working for a year on a project at a certain institution whithin one of its division as a vendor, using technology X, and as the contract is coming to an end, my company already tried to sell their resources (me and some other guys) to another project,
One of the prospective client is the other IT division within the said institution, and my manager already scheduled bunch of interviews with their (lead?) for me and the other guys, and does these things ever went well?
*let's call the (lead?)/interviewer as PIC (person in charge)
First guy was scheduled at 10 AM a couple days ago, he came at 9:55 sharp, and the PIC hasn't even arrived at the office, he had to wait 1 hour
Second guy was scheduled two days later, sometimes in the afternoon around 3 PM, and again, the PIC is nowhere to be found, he had to wait more than 30 minutes
Third one was a lady colleague earlier today, supposedly scheduled for 10:30 AM, again, had to wait an hour until the PIC showed up sometime around 11:30
Fourth and last was mine, scheduled for 16:30, and had to wait for almost 1 hour, my manager had to call the PIC to remind him of the interview,
So, next is, the questions, I asked the other 2 guys, but haven't heard anything about the lady colleague (until shortly after), that the questions are around what we've been doing in our current project with technology X, apparently their department is trying to adapt this tech stack and from what I see, they are trying to copy our approach,
Since it's was quite visible from the open, literally the project offer states, Institution N division Y is asking resources with experience of technology X, I assume they would be direct in their line of questioning, but well, bureaucrats, what can I say,
To start, he asked me to introduce myself and my background, being uninterested in this project I tried to undersell myself, I told him I have always been a front end developer, back then I always used native JS and mostly jquery for quite some time, until later I learned a bit about angular and finally thrown into technology X shortly after,
First part of the interview, he pries around what I do in my previous company, I still have something to show around, but I don't remember jack shit about any of it, not to mention the angular stuff, and he asked pretty detailed stuff about angular, and I'm not sure whether his question is testing me, being sarcastic or honestly don't know what he is talking about,
One of the most notable question was, "what is the difference between angular and jquery? Is the syntax is different in any way, how do you use it?"
I immediately thought to myself, "ARE YOU A FUCKING IDIOT, DO YOU EVEN LOOK AT MY CV, IS THAT EVEN A PROPER QUESTION YOU FUCK??!!"
But I tried to explain it in laymen term with equanimity..., which seems to have the opposite effect but a welcome one, he seems to think I'm an amateur, "ah I see that you might have forget everything about angular, let's move on"
After that move on onto my days at my current company, I have worked on an internal project as training, using MEAN stack, and whoo boy, the proudest moment of my life, I learned most of it and managed to create a simple app in less than a week, but then again I already forgot all about it so I tried to skip this one
Next is my current project with technology X, let's just say it's related with react JS, initially he ask
P: "how long did it take for you to learn tech X?"
Obviously even after a year there's still a lot to be learned
Me: "well, I've just started learning it only since The start of this project"
P: "yes but how long does it takes for you to learn it?"
Me: "I cannot answer that because even until now I'm still discovering new stuff"
After that the PIC guy suddenly insisted to see the project I'm working on, I don't think I was supposed to do this, but since he work in the same institution, and my manager's monitoring the call, I'd say everyone is in knowledge of disclosing a confidential information, so I fired up my laptop and showed him, which doesn't seem to leave quite an impression as it throws up bunch of errors as I was running it, yea it is quite buggy
But then he showed me what he have been working on, a piece of component code, and asked me
P: "explain what's going on here"
Well, his code is not exactly neat, not as structured as ours, I'm quite pissed at how he said that in a condescending manner, I managed to explained it to him easily, but I hope that didn't give me a plus point,
I don't want to get this project because this guy looks like the type to put grunt work to others, and who knows about his mates?
I never understood how people have any problems with getting paid for freelancing work, when middleman/escrow platforms like upwork exist, just don't be retarded when applying for a job. I am so sick of those shit ass stories from people telling me "my client didnt pay meeee 😭😭😭" ITS YOUR FAULT. I never had any client not paying, if you don't have the option of escrow, then just fucking put remote execution via "update" system in for fucks sake or give remote control to the client while monitoring it, there is so much fucking ways to secure yourself, just don't be retarded and many clients instantly show their character when talking budget and turnaround time.15
When duel 24' monitors is not enough...
How does 3 work for you guys? Does your neck hurts moving around?
I had 5 at work but was mostly on 2 since other 3 for monitoring.10
Now this is fucking ridiculous... Our website is being constantly limited though we've never reached even 80% of the available CPU resource.
The hosting said that we had the CPU fault (that fucking cyanide spike on the graph that triggers the limit once) because of huge load on the server. The FUCKING SERVER... Not our virtual environment. And once more because of the RESOURCE MONITORING service caused a server restart. For fucks sake, really???
And apparently it's perfectly normal that all users even ones that run in low resources are being limited to a level that a request takes 30 seconds to complete instead of frickin' 1...
The best they could offer is to move us to a new server, which will arrive in two weeks, if the problem persist. IT'S PERSISTING FOR FUCKING MONTHS YOU MORON. I wonder how much time would have been taken you to realise the server shutdown this week if I hadn't phoned you in 5 minutes. FUCK!
Every shared hosting is that garbage or am I just the choosed one?12
So it's Friday afternoon just before a bank holiday weekend here I'm the UK, perfect time for our production database to go TITSUP (total inability to support usual performance), life sucks then you die folks....2
So there I am sitting in front of my laptop, and trying to npm i and I am getting all sorts of sha mismatch errors.
After lot of debug I conclude it is coming from the proxy as it refuses to download and supplies the error page.
It says it's because I'm using the old proxy so they give me the new URL which I set up and it works.
All good until my password expires. I use our bash script to change it. NPM is buggered again throwing the same errors.
Go to IT, tell them the saga begins.
After a countless hours of looking at the log files we notice that the npm registry is set to http instead of the standard https (thanks bash script). so our firewall blocks the download.
Almost. NPM now works fine, but when I go and I play around with node and axios, I get my requests time out. My instinct says its the bloody proxy again.
So I hit up my trusted WIN Support guy and he confirms that the url is not blocked. So he starts monitoring whats going on and turns out, every time I run the node app, node casually ignores the system-wide proxy settings and tries to send the request as the PC rather then my username.
Since the pc's don't have rights on the proxy it is being refused...
Thank fuck for the corporate proxies, without them, I could just develop things not ever learning these quirks of node...3
TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.
I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:
-migrated their 2007 Exchange server to Office 365
-Upgraded their ailing Windows server 2003 based architecture to 2012R2
-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)
-Deployed a vCenter server to manage said ESXi servers easier
-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality
-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.
-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time
-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies
-Documented the entire old network and now the new one as I’ve been upgrading this
-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.
-Performed network scans and rewrote firewall exceptions to tighten security
There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.
I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.
I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.
I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?
Thanks in advance!6
What makes free ssl "Unsuitable for e-commerce websites", Please read to end to see my view point.
Free Certificates are domain validation only which means they don't certify the identity of the website owner, they simply ensure a secure connection. Customers can't be sure of the integrity and trustworthiness of the website owner. If you need to secure credit card and personal information on e-commerce websites, free certificates aren't the answer. It's important your customers trust your business is safe enough to hand over these details. To gain this trust, you need a certification of your authenticity, which you can only get with a (paid) Business Validation or Extended Validation SSL Certificates.
* "To gain this trust, you need a certification of your authenticity"
~ But isn't that just Domain Verification and other Extras, What justifies somebody or business's authenticity? Tax Id, Valid Address, Nobody is going to study the ssl cert to make sure that amazon.com is a valid business and has a tax Id.
* "domain validation only which means they don't certify the identity of the website owner,"
~ Wouldn't this just be the domain validation test that is required when using services like LetsEncrypt using Certbot etc, or are we referencing back to this idea that they look for a Valid Tax Id sort of thing?
* "If you need to secure credit card and personal information on e-commerce websites, free certificates aren't the answer"
~ Why is the paid version going to do double encryption, is the CA going to run a monitoring tool to scan for intrusions like a IDS or IPS? (disregard the use of DNS Validation being in the picture)
Am I missing something, this just seems like well crafted text to get people to buy a cert, I could understand if the encryption was handled differently, Maybe if they checked the site for HSTS or HTTPs Redirect or even, They blocked wildcard SSL before and now with the paid its included, but overall it doesn't sound like anything special. Now I'm not just picking on namecheap because domain.com does the same.14
2 hour meeting to brainstorm ideas to improve our system health monitoring (logging, alerting, monitoring, and metrics)
Never got past the alerting part. Piss poor excuses for human being managers kept 'blaming' our logging infrastructure for allowing them to log exceptions as 'Warnings', purposely by-passing the alerting system.
Then the d-head tried to 'educate' everyone the difference between error and exception …frack-wad…the difference isn't philosophical…shut up.
The B manager kept referring to our old logging system (like we stopped using it 5 years ago) and if it were written correctly, the legacy code would be easier to migrate. Fracking lying B….shut the frack up.
The fracking idiots then wanted to add direct-bypass of the alerting system (I purposely made the code to bypass alerting painful to write)
Mgr1: "The only way this will work is if you, by default, allow errors to bypass the alerting system. When all of our code is migrated, we'll change a config or something to enable alerting. That shouldn't be too hard."
Me: "Not going to happen. I made by-passing the alert system painful on purpose. If I make it easy, you'll never go back and change code."
Mgr2: "Oh, yes we will. Just mark that method as obsolete. That way, it will force us to fix the code."
Me: "The by-pass method is already obsolete and the teams are already ignoring the build warnings."
Mgr1: "No, that is not correct. We have a process to fix all build warnings related to obsolete methods."
Mgr2: "Yes. It won't be like the old system. We just never had time to go back and fix that code."
Me: "The method has been obsolete for almost a year. If your teams haven't fixed their code by now, it's not going to be fixed."
Mgr1: "You're expecting everything to be changed in one day. Our code base is way too big and there are too many changes to make. All we are asking for is a simple change that will give us the time we need to make the system better. We all want to make the system better…right?"
Me: "We made the changes to the core system over two years ago, and we had this same conversation, remember? If your team hasn't made any changes by now, they aren't going to. The only way they will change code to the new standard is if we make the old way painful. Sorry, that's the truth."
Mgr2: "Why did we make changes to the logging system? Why weren't any of us involved? If there were going to be all these changes, our team should have been part of the process."
Me: "You were and declined every meeting and every attempt to include your area. Considering the massive amount of infrastructure changes there was zero code changes required by your team. The new system simply worked. You can't take advantage of the new features which is why we're here today. I'm here to offer my help in any way I can with the transition."
Mgr1: "The new logging doesn't support logging of the different web page areas. Until you can make that change, we can't begin changing our code."
Me: "Logging properties is just a name+value pair dictionary. All you need to do is standardize on a name and how you add it to the collection."
Mgr2: "So, it's not a standard field? How difficult would it be to change the core assembly? This has to be standard across all our areas and shouldn't be up to the developers to type in anything they want."
- Frack wads smile and nod to each other like fracking chickens in a feeding frenzy
Me: "It can, but what will you call this property? What controls its value?"
- The look I got from both the d-bags I could tell a blood vessel popped.
Mgr1: "Oh…um….I don't know…Area? Yea … Area."
Mgr2: "Um…that's not specific enough. How about Page?"
Mgr1: "Well, pages can cross different areas, and areas cross different pages…what do you think?"
Me: "Don't know, don't care. It's up to you. I just need a name."
Mgr2: "Modules! Our MVC framework is broken up in Modules."
DevMgr: "We already have a field for Module. It's how we're segmenting the different business processes"
Mgr1: "Doesn't matter, we'll come up with a name later. Until then, we won't make any changes until there is a name."
DevMgr: "So what did we accomplish?"
Me: "That we need to review the web's logging and alerting process and make sure we're capturing errors being hidden as warnings."
Mgr1: "Nooo….we didn't accomplish anything. This meeting had no agenda and no purpose. We should have been included in the logging process changes from day one."
Mgr2: "I agree, I'm not sure why we're here"
Me: "This was a brainstorming meeting as listed in the agenda. We've accomplished 2 of the 4 items. I think we've established your commitment to making the system better. Thank you all for coming."
- Mgr1 and 2 left without looking at me or saying a word.1
A software had been developed over a decade ago. With critical design problems, it grew slower and buggier over time.
As a simple change in any area could create new bugs in other parts, gradually the developers team decided not to change the software any more, instead for fixing bugs or adding features, every time a new software should be developed which monitors the main software, and tries to change its output from outside! For example, look into the outputs and inputs, and whenever there's this number in the output considering this sequence of inputs, change the output to this instead.
As all the patchwork is done from outside, auxiliary software are very huge. They have to have parts to save and monitor inputs and outputs and algorithms to communicate with the main software and its clients.
As this architecture becomes more and more complex, company negotiates with users to convince them to change their habits a bit. Like instead of receiving an email with latest notifications, download a csv every day from a url which gives them their notifications! Because it is then easier for developers to build.
As the project grows, company hires more and more developers to work on this gigantic project. Suddenly, some day, there comes a young talented developer who realizes if the company develops the software from scratch, it could become 100 times smaller as there will be no patchwork, no monitoring of the outputs and inputs and no reverse engineering to figure out why the system behaves like this to change its behavior and finally, no arrangement with users to download weird csv files as there will be a fresh new code base using latest design patterns and a modern UI.
Managers but, are unaware of technical jargon and have no time to listen to a curious kid! They look into the list of payrolls and say, replacing something we spent millions of man hours to build, is IMPOSSIBLE! Get back to your work or find another job!
Most people decide to remain silence and therefore the madness continues with no resistance. That's why when you buy a ticket from a public transport system you see long delays and various unexpected behavior. That's why when you are waiting to receive an SMS from your bank you might end up requesting a letter by post instead!
Yet there are some rebel developers who stand and fight! They finally get expelled from the famous powerful system down to the streets. They are free to open their startups and develop their dream system. They do. But government (as the only client most of the time), would look into the budget spending and says: How can we replace an annually billion dollar project without a toy built by a bunch of kids? And the madness continues.... Boeings crash, space programs stagnate and banks take forever to process risks and react. This is our world.3
My CS exam today had a case study question that, and i quote, talked about "Chernobyl in japan switching to manual monitoring due to the wannacry virus" xD wtf. Im fucking done xD
Genuine appreciation ahead:
I really like how considerate and humane the Huawei health app is. The sleep monitoring and the suggestions are right on point with tiny bits of extra information ( like listen to nature sounds)
Gotta sleep now..
P.s.: There is nothing good about sleep deprivation, it's bad and very unhealthy. Always take adequate sleep.6
Probably posted this before but don't ever put swear words in log statements because you will miss one and if you're client has a log monitoring system it will catch it, it's fairly embarrassing when your client says "we noticed quite a spike in 'bollocks' around 3am when the reorgs are happening on the database".
Either a really big coincidence, or I'm officially creeped out.
I've been looking into buying a vps, so researching that a lot. Then today, I went to work, at a monitoring station, so we have to use remote desktops to access anything other than very specific sites.
Then I looked at an article about c#, and there was a Google ad, about a vps.. Keep in my mind, I'm at work, on a remote desktop, that gets cleared every time it's closed.
I know a vps isn't the most unpopular thing, but haven't seen an ad for it before.5
I've gotten almost 20 emails today from our monitoring service saying it can't ping the server and then one 2 seconds later saying every fine. What the hell guys!!!?3
A loooong time ago...
I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.
Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.
Fast forward three months (yes, three months).
I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.
So now we come to the other teammate. The one I was not talking about till now. And that for a reason.
He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...
When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.
I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?
This happened over 30 times.
Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.
I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!
Haha. Oh no.
So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.
The same when he did his second prod deploy. Everythings fine.
And then. It. Frikkin. Begins.
I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.
Error checking out git. Okay, awkward. Got to investigate...
On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.
This doubles the strangeness.
I want over to my colleague's desk.
I asked him about any recent changes to the codebase.
"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"
What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?
"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"
"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."
"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"
You. Did. What?
I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.
So in the end I sat there reverting, committing and deploying.
It's soooo much harder this deployment crap.
Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.
Karma's a bitch, right?
A developer couldn't get a application performance monitoring (APM) tool to trace his application. They claimed that their libraries and their configurations were alright and that the APM tool was non-performant.
The developer then argues with sysadmin that the APM tool can't trace the application and that there's nothing wrong with the application or the configurations. When sysadmin questions whether the developer got the tool to work anywhere, they say, "No" and head off to make it work at least in one place. They come back saying that it works on their development environment (which is their local machine). Sysadmin claims that the system configurations on the server instances cannot be matched by the development environment and there could be a lot more factors to be considered for the problem. The sysadmin asks to prove it on a server instance on one of the test environments and then they'd agree that it is a problem with the tool. They also argue that this is not the only application that uses the APM tool and the tool happily traces other applications with no issues.
The developer tries the same configuration on a staging instance and fails. In order to make it work, they silently uninstall the existing version of the APM tool and then compiles an unstable branch of the tool. It finally works with this version.
They go back to the sysadmin and show that it works on the staging environment, but does not on production. After banging their head on the wall for a while, the sysadmin figure that the tool had been swapped out for the unstable branch that was manually compiled. When questioned, the developer responds, "It works with this version on staging, so deploy the same version on production"
WTF? You don't deploy an unstable branch to production. Just because you can't make it work on the stable branch doesn't mean that it is the problem with the tool itself. There's a big difference between a stable branch and a non-stable branch. How would you feel if the sysadmin retorted by asking you to deploy the staging branch of your application to production?
From now on I am administrating multiple servers in our company and monitoring is one thing our infrastucture lacks...almost completely. At least, useful monitoring.
Installing netdata or Grafana and integrate it with chat is definitely a solution, but what happens if the whole server just shuts down (very stupid scenario I know)? Well, it is easy, there will be no alert about the failure.
So, that's where I was wondering if there is a tool or even better plugin for netdata or Grafana, that enables remote monitoring from another server? I surely can write a simple script to check the server availability but having the whole monitoring tool on a single server instead of 5+ would be also easier to maintain and setup.10
Project 1: A hand hygiene monitoring system intended for the NHS.
Project 2: A language analysis platform.
When I rented my server I uploaded my webpage (including resources like videos, images etc) which is about 150GB as .tar and extracted and setup all that stuff and deleted the backup from my PC. The uploading process took me about 4 days. I opened the site of my server provider and reloaded it.. Aaaaannddd whoops. All data gone.
On my server hosters webpage when you click the reinstall button for installing a Linux image you get returned to the main page of that server after it finished installing. If you then reload that page which basically only shows some monitoring diagrams and shit the server gets reset again.
Damn. I lost so much good porn on that day...
My scrum master said, 'I would optimize your work hours.' He's monitoring how much time every one spends on browsing non-productive webs like devRant. How can I fight back? :(6
So a friend / batch-mate in our accelerator asked me if I was okay with installing a monitoring software for a client our startups are collaborating for. And the said client was ranting how I've been appearing offline to him since morning...
Bitch I'm already letting you monitor my shit from morning to night, I don't need your French ass snooping around what I'm doing outside of office hours.
"you've worked with nagios before haven't you? Can you give a presentation on it" 'sure' in the meeting: so tell us about opennms5
Note to self:
Close off ALL ways things could go wrong..
Long story short; I released a new feature, to be able to better follow up on any stock moves, their amounts, locations and even expiry dates. An older tool just bypassed that very verification and nothing was logged or taken out of stock.
Taking out an amount for a certain orderline has a shortcut in place to mitigate some of the mandatory steps that pickers need to take in order to verify what's being taken. This little tool only available, visible and possible for a very few select users.
I assigned some orders to one of these people, which made him think it was an urgent batch. It's only one product, for multiple orders, so he went to the location, took out the amount needed and then used the tool to quickly be able to prepare them for shipping.
This bypassed the new methods to check if the location actually had stock to take, which I had just enabled for 1 account.
Luckily I caught the miss-hap as I was monitoring that product first-hand and noticed the batch of orders was collected but the stock amount didn't update.
It was 5min before I was leaving work, so I investigated and then ran to the person in question to ask what he did; which was "I used that tool"
I facepalmed myself internally while blaming myself, as he couldn't know that it wasn't ready to use for that purpose.
The tools to fix this up are there already.. so I used that to fix some missing stock-takes manually.. Though I'll need to close that little tool for these kind of orders for sure, asap, probably when I get home, at least until I bring over its new logic to it.
Happy Tuesday? (:
Well I've got this new worker and me and him are like "great minds think alike" , we're now trying to convince the boss that a specific monitoring product that cost hundreds has an equivalent open source.... No luck so far in convincing him1
Service status pages that poorly reflect actual service status are so annoying. Ex. GitHub is having a lot of latency issues with processing updates and like 5 people in my office noticed it while their status page still says everything is fine.
This isn't to explicitly call out GitHub since many service status pages behave like this, but it definitely shows a general weakness in these health checks. I've seen similar issues with tons of services, web hosts, etc. Monitoring is definitely hard but will hopefully keep getting better.1
Been programming a lot over this winter break from school. Was able to finally dive into node, express, and view engines. Set up my own site and am now looking at getting into nginx and monitoring tools for my website and ssl!5
Just had a meeting about performance and monitoring. The main topic of the meeting was to be aware of disk space usage. If there are issues with memory leaks or processor hogging don't worry those are fine, just give it more.1
Monitoring goes does in one of our server racks, one I had just been working in (other side of the building). Hmm, odd. Oh wait its back online again. Better go take a quick peek to see whats up, just in case. Walks over to rack. Everything seems fine. Walks back to desk. About 15sec later. Fqdn.plsfckof is offline again. -_-
Looking to sharpen and pursue a SysAdmin/DevOps career, looking at online job offers to get the big picture of required skills and I say FUCK. It would take me a lifetime.
Azure, AWS, Google cloud platform.
CD tools: Ansible, Chef or Puppet
Scripting ninja with Python/Node and Shell/Power shell.
Linux & Windows administration
Mongo, MySQL and their relatives.
Networking, troubleshooting failure in disturbed systems
Familiarity with different stacks. Fuck. (Apache, nginx, etc..)
Monitoring infrastructure ( nagios, datadog .. )
CI tools: jenkins, maven, etc..
DB versioning: liquibase, flyway etc.
FUCK FUCK FUCK.
Are they looking for Voltron? FUCK YOU FROM THE DEEPEST LEVEL OF MY DEEP FUCK.1
This is a part rant-part question.
So a little backstory first:
I work in a small company (5 including me) which is mostly into consultation (we have many tech partners where we either resell their products or if there is a requirement from one of our clients, we get our partners to develop it for them and fulfill the client requirements) so as you can see there is a lot of external dependencies. I act as a one-hat-fits-all tech guy, handling the company websites, social media channels, technical documentation, tech support, quicks POCs (so anything to do with anything technical, I handle them). I am a bit fed up now, since the CEO expects me to do some absurd shit (and sometimes micro manages me, like WTF I am the only one who works there with 100% commitment) and expects me to deliver them by yesterday.
So anyway long story short, our CEO finally had the brains to understand that we should start having our own product (which i had been subtly suggesting him to do for a while now!).
Now he came up with a fairly workable concept that would have good market reach (i atleast give him credits for that) and he wanted me to suggest the best way to move forward (from a both business and technical point of view). The concept is to have an auction-based platform for users to buy everyday products.
I suggested we build a web app as opposed to a mobile one (which is obvious, since i didnt want to develop a seperate website and a mobile app, and anyway just because we can doesnt mean we have to make a mobile app for everything), and recommended the Node/react based JS tech stack to build it.
At first he wanted me to single handedly build the whole platform within a month, I almost flipped (but me being me) then somehow calmed down and finally was able to explain him how complicated it was to single-handedly build a platform of such complexity (especially given my limited experience; did I mention that this is my first job and I am still in college, yeah!!) and convinced him to get an experienced back-end dev and another dev to help me with it.
Now comes the problem, I was to prepare a scope document outlining all the business and technical requirements of the project along with a tentative cost, which was fairly straightforward. I am currently stuck at deciding the server requirements and the system architecture for the proposed solution (I am thinking of either going with AWS - which looks a bit complicated to setup - or go with either Digital Ocean or Heroku):
I have assumed that at peak times we would have around 500-1000 users concurrently
And a daily userbase of 1000 users (atleast for the first few months of the platform running)
What would be the best way forward guys?
I did some extensive (i mean i read through some medium blogs! and aws documentation) research and put together the following specs (if we are going through AWS):
One AWS t3.medium ec2 instance for the node server (two if we want High Availability by coupling with the AWS load balancer and Elastic Beanstalk)
The db.t3.small postgres database
The S3 Storage bucket (100gb) for the React Front end hosting
AWS SNS for email/sms OTP and notification
And AWS CloudMonitor for logging amd monitoring.
Am I speculating the requirements properly, where have I missed??
Can u guys suggest what is the best specification for such a requirement (how do you guys decide what plan to go with)?
Any suggestions, corrections, advices are welcome4
Who the fuck invented the glorified pile of shit people call laravel? Is this actually used in PROD for anything else than load testing a monitoring server by creating loads of error messages?
OOP exists for a reason, not to create bazillions of classes with static methods.
Dump that shit ffs!7
This 30 hour project is now 110 hours in and client is changing their mind so much that the managing director of the company is now sitting monitoring every correspondence between the client and developers.
When your IT VP starts speaking blasphemy:
We all know what’s going on with the API. Next week we may see 6x order volumes.
We need to do everything possible to minimize the load on our prod database server.
Here are some guidelines we’re implementing immediately:
· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.
Please keep the communication level high and stay vigilant in protecting our prod environment!"
RIP most of what I do at work.3
TL:DR linux newbie, looking for advice/links (skip to bottom for questions)
After i had been looking for a job for quite some time, a couple of months ago i got hired by "smaller" company doing web stuff. So far it have been a great place, good colleagues, and overall just having a great time!.
They seem to value me alot, so that's great!.
Anyway, yesterday i got called into a meeting - and got told they wanted me to start learning "Server stuff (linux)". That got me quite excited, because it always was something i wanted to learn - but never really got around to doing.
But i never touched a linux installation before, so i'm really on ground zero - but im not afraid, i'm a quick learner and quite efficient at googling :)
I figured i would ask here, since other people here always seems to be happy to help other people out.
So far i have manage to setup a server, install various stuff (php, mysql and so on) and done setup a couple of domains/subdomains on my server. Also got a vestacpinstallation working - so overall im quite happy so far.
I figured maybe somebody had some good links/advice for a linux newbie :).
* Performance/Security, will obviously be a big focus - anything i should look at? - any must look at?
* Monitoring tools, how do i monitor various websites running on my server? Here i'm thinking bandwitch, cpu/ram usage and so on pr site basis.
* Any other stuff i should be looking at?
Little about what the server will/should be running :)
* WordPress installations only (e-commerce mainly)
* PHP 7 / MySQL / phpmyadmin5
Anyone have much success with Kali/WiFi penetration testing?
I've been tasked with trying to break WPA security within a couple of hours without a dictionary attack - is that even possible?
I have an Alfa AWUS036NHA capable of monitoring mode if that makes any difference. It's my first time trying anything like this.10
I'm a rather young developer, self-learned everything and started when I was 13 (now 20) but I still feel like I'm a total beginner since I have not yet mastered the things I am OK at.
Php (laravel, since it makes things much easier), js (jquery, bad at vanilla, have used angular and ember but not mastered), node, linux, html, css, photoshop, illustrator, sql, mongo and windows servers
I know little about many things, can create things that are asked of me but the methods I use are rather bad imo.. ex: I finish coding a section of a site, but when I need to add a new feature I find myself rewriting most of the stuff to add the new feature and in the end still feeling like the code could be optimized further, even though I have no idea how.
TL;DR I write bad code, but things work as long as I am monitoring them. I know little about alot of stuff but mastered none of them.
What should I do? Go to school for programming?8
I am a programming student and last 1 year i have sat with my own programming project of a management system for monitoring 500+ clientes, has now been recognized for my work and has now been giving a new major programming project for a new management system for phones 😁3
So I work at a monitoring station (yeah not a professional dev yet), so basically our entire day is spent on the phone. Yesterday morning, our phone system broke. Everyone is getting calls from all departments. Even departments they're not in.
As if my job isn't stressful enough as it is, now this fucking thing happens, and whattya know, shit still isn't working today...
Is there any multi server monitoring software that doesn't open a port for itself? I am about to just write one, but it's fucking annoying that theres nothing like it out there, where you just install a service per server, that uploads its data (cpu,ram etc) to a central server without opening itself to the whole fucking internet.6
So I've been given a task to monitor a whole lot of logs of some servers (whole university ~ 10+ departments). The technologies are diverse so I'm cramming everything into elasticsearch via logstash (and filebeat), viewing it into kibana. Any recommendations for what should be the 'useful' stuff to be viewed into dashboard? I guess:
- Overall traffic wtih respect to previous days/weeks
- Most viewed domains
- Failed logins?
- Dropped connections?
- Critical-load of systems? 90%+2
So following my previous post, the issue happened again. And actually for background what I've been telling my boss, for years, we need ELK setup and integrated into all our APIs ASAP.
I think it's a punishable crime if any program is released into prod at a tech company with out real time logging/monitoring built in?
So issue still happening, user sent us the request details. So now need to find the actual now that handles the request and look into it's logs to see the details.
Now he's doing it the hard way.... Just finished took 1hr, and the best answer her can come up with is "I think .... Maybe ..."
And if course this is based on infinite data. He stopped after finding a "probably cause"
I have a script that is like promotion ELK, downloads all looks and parsed then so I can run queries to pinpoint the exact call and which log it's in. And can see what's happening around it.
We'll see what my way find but definitely does not take more than 1hr...
Loading data maybe but that's because it needs to download the logs and parse them all...
On a side note, guess I'm Beck on devrant as I have something to rant about. Though it's the same something that I was wanting about years ago... Monkeys...1
Our ISP asked if I was satisfied with their service. I told them that it's okay, but some of our computers don't have enough powerful network card, and they can't use the internet on maximum speed, and they said that they could see it, too. WTF? I knew that the ISP's router is not the most secure thing, but it has a remote mode, which if of course OFF, and they still can see this, and maybe even more. Monitoring your traffic is a thing, but a home network should be private...2
I looked at an SQL server today from a customer, talked with one of their devs and he said that he's unable to understand why the server misbehaves... All (!) queries were optimized, but they have 'big data queries'... Migraine started, I had a very bad feeling. Monitoring? Nooooppeeee. Migraine kicks in. Connected to server. SHOW GLOBAL VARIABLES...
After a bit of scrolling I found a lot of misconfigured variables (e.g. extreme large join buffers, unrealistic buffer sizes), high slow query count (nearly 60 % of COM_SELECT) and a few variables that were unknown to me.
Then came the version line.
Big data? Well... 30 GB of usage data.
I called the company back... The dev told me sternly that this was the production server (I had hope...) and that I lie - neither the version, nor the variables could be the problem.
A coworker had to verify it and our manager had to do the communication... Worst, most traumatic working day I ever had.
I got a very low power Netbook lately for basically no money.
I thought about using it for some server monitoring / server access via ssh console.
Which Linux distros would you recommend for such a use case. Tried Something like core-os and Debian(lxde) yet but wasn't very satisfied with both options. Both could not display the battery capacity and Debian didn't detect the Intel WiFi.
The Netbook has 512mb of ram which should be fine for a lightweight gui and more than enough for a ssh connection 😅
Thanks a lot for the recommendations :)12
WE: javaagent-based monitoring, as seen in this screenshot <attached>, is reporting full old-gen, full young-gen, full one of the survivors and a sky-rocketing full GC right before the service outage.
WE: container monitoring in this screenshot <attached> shows that the application peaked its memory very suddenly to MAX values and platoed on that. Then container monitoring is blank, suggesting a complete outage of a few minutes. After that monitoring starts again with memory usage reported at low levels and immediatelly spiking back to MAX again, suggesting the container crashed and had been respawned by an orchestrator. This repeats a few times throughout the day.
they: I did not find any evidence of application running out of memory. Maybe our monitoring is not working correctly?
we: *considering updating our resumes*
How about incompetent management? Company absolutely murders any possible increase in productivity. Laptop provided? Slow as balls. Takes minutes to log in. I get a Mac for mobile development and that's OK. SSD and adequate memory but I'm primarily a .NET Dev. Can't get on the network with a virtual machine. They won't I stall even a managed image. So can't use databases because they're all AD authenticated. Got a virtual desktop environment and that sucks worse in performance than the laptop. Add the Assault on local administration rights and the monitoring software that constantly thrashers any memory and hard drive usage and im about to quit over all this... All this decided by a non developer and not asked for our opinions. Yay large Enterprises
Do you have any must have programs? Since I'm starting to work in August and will use my Laptop at work too, I'm looking for useful programs for IT. (network monitoring etc.)10
RavenDB was by far the worst document storage "solution" I have ever had the displeasure of working with.
- Loading data crashed the service.
- Queries crashed the service.
- Monitoring applications crashed the service.
- It didn't support clustering or HA of any kind.
- Sometimes it just worked for no good reason.
- Often it broke for completely random reasons.11
Using grafana together with tinc+promotheus, has been a blast.
Initially I wanted to get into ELK with Kibana and all that, but that required 8G of ram, the instructions to get it running in the open source "mode" was nearly non-existent, together with all the ready docker compose stacks out there simply not working or the images being broken.
I'm sure I could've managed around most of those issues, but the fact it is as hungry as gitlab, made it a literal no-go for the usual server resources my clients host or my own scaled down server recently.
Thankfully I remembered that there's grafana and me having experimented some time ago with tinc, so I can have very lightweight beat'esque prometheus agents deployed listening on tinc local net only, with the typical nginx auth and some whitelists to all of the servers I host and all those of my clients.
The dashboard creation was especially great in grafana (tbf promotheus does actually most of it), literally what I always wanted out of those "complicated" solutions, that do it all, but have no proper query language, complex documentation, heavy collectors with no properly named data points, expensive resource runtimes, ..
with grafana I can just easily put dashboards into folders, create users to look only at certain stats or even dashboards (opened up some interesting contracts actually, because now I can also offer proper monitoring for all things delivered), easily drag and drop around stuff to fit more information (most others fix you to a small 3x2 grid, a too big grid for a TV or simply non resizable tiles, making that one counter take up an entire row) and resize to my hearts desire
tinc of course allows me to easily create private networks that are resistant to failure across any region and the routing is done for me, so I don't have to run around it all that much either
P.S: a damn tiny fly went into one of my now 4 monitors and died right in the middle, because I thought it's just some dirt and I pressed it in while trying to wipe it off, so that monitor now serves as the top most on a vesa mount5
Got the chance to get into developing a monitoring frontend...
Imagine the step in between if your previous task was *phone up*, *translating consumer problems into consultants problems*, *phone down*
I wanted to buy a 50 inch LCD 4K TV for my room to wall mount it in my room as third display for coding and monitoring.
I went to a TV store and already saw 4 people in a Queue waiting to buy a OLED TV for 3000 $.
Currently I dont understand, why a lot of people are hiped for OLED TV's.
For me a OLED on anything with a batterie is the better option, but not on a stationary monitor.
Sure OLED got the deeper black, but on a stationary Monitor you want the best colors and for that there's LCD.
But OLED TV's are selling triple the price of a LCD and people buy it like it's the best shit ever. WTF!1
Ok, I'm fed up with this, just read something about android constantly monitoring your phone's location, now it's time to shut this up.
Would you please be so kind and share information on which alternative "privacy-first" OS I could use and how to flash my device? For all I know, it runs a custom HTC modified OS. I'm quite unfamiliar with all those things gravitating Android. Heard about Cyanogen mod but that's about it.
What about compatibility with apps downloaded through the play store? (thinking about Threema) I would also need compatibility with WhatsApp (yeah, sucks, I know, but hard to convince regular people)
Thank you all :)2
This is fucking how you do it!
Ticketmaster UK had a "data security incident" where they don't really know if any data was actually leaked/stolen/"accessed by an unknown third-party" — their response:
1. Disable the compromised service across their platforms
2. Send a mail to any customer that may have been affected (I got one in Danish because I had only interacted with them through a Danish subsidiary)
2b. All notified customers have their passwords reset and must go through the "Forgot password" process; the _temporary_ password they sent me was even pretty nicely random looking: ";~e&+oVX1RQOA`BNe4"
3. Do forensics and security reviews to understand how the data was compromised
3b. Take contact to relevant authorities, credit card companies, and banks
4. Establish a dedicated website (https://security.ticketmaster.co.uk/...) to explain the incident and answer customer questions
5. "We are offering impacted customers a free 12 month identity monitoring service with a leading provider. To request this service please visit [this page]"
EDIT: As mentioned and sourced in the first comment, the breach was apparently noticed by a banking provider and reported to Ticketmaster on the 12th of April and later to Mastercard on the 19th of April.
Ticketmaster's internal investigation found no evidence of breach (which makes sense, as it wasn't an internal breach), but when Mastercard issued an alert to banks about it on the 21st of June, Ticketmaster followed up by finding the actual breach and disabling the breached third party service on the 23rd of June.
I still think they did the right thing in the right way...2
So I am working on a mixed API (aka reachable from anywhere, but also only accessible by specific allowed devices) and I am struggling with the security of it, its not managing anything hardcore (this API is "is the coffe ready?" kind of level) or I would have just enforced per device registration for example already, but the app that goes with that API is deployed remotely and has to be "ready to go!!!" out of the box, so I can't add any registration, verifications of devices etc.
The main thing I am afraid of is, that one of those agent retards will get his spaghetti phone blasted from the inside, so all the https calls will be read out by some random attacker, which then will be able to "abuse" the API via read out api-key, is there any way for me to have a rescue plan if one of those retards does get hacked and the system then get spammed or something, like if I log all devices that use the API I could just deny access from that device (until resolved) and issue a new app update via new api key.
What's the best way of handling this and is my idea really the only way to handle this? this shitfest is really causing shit ton of ideas in my head, which then I deny literally 20 seconds later, because there's a way to bypass it or once you have the old api key to get a new one by just monitoring it etc.5
The past three weeks almost no issues during the 24-hour emergency service. My service starts at 10:01. Everything breaks.
Whoops, my head will be squashed tomorrow. Asked to put monitoring in other week by boss, sysadmin been complaining about high CPU, apparently 10 requests (different domains) to the one VM on our servers every 10 seconds is killing it. However this server is being used for MySQL and serving web requests by Apache and PHP. Then also running a few jobs like consuming queues etc.
Wtf do I do? Every time I tell him about more resources (we have decent 2 rack servers just running 20 vms and only 1 VM is for web sites) he says software should be made to work with what we have.1
short: The admin with enough xp is ill, there is no one with xp with varnish is and after 1 restart varnish outputs only 503.
long: there original admin is ill but he gave me an project to migrate an typo3 installation to a new server. Thats ok.
Plan: I move 150 GB of data with rsync to the new server, let specialists do something and switch ips between the new and old and clear varnish with a restart.
Reality: +2 hours to migrate the data, because of false infos from the admin, 7 hours preparing the switch, 5 minutes switch, 3 hours to find out the F*****G varnish is the single point of failure. I and the t3 guys agree to see the next day what went wrong.
ALL HAPPENED TODAY!
Plan for tomorrow: speak with the boss to account the extra hours to that day so i dont get over 10 hours and debug that fucking varnish and delete some servers from another project from the backupsystem and monitoring.3
Not a rant. Request suggestions.
I am developing a Sublime Text plugin for real-time code monitoring ( screencasting) using Websockets. I would like to know if it makes any sense to develop such a plugin. Also, please suggest some use cases so that I can increase the features of the plugin. Point out if it already exists. Thanks :)
- Every specialist is looking after his area of expertise
- Everyone is a specialist of everything and shall work on everything
- Every specialist is looking after his area of expertise, making improvements and automations in his area
- Everyone is a specialist of everything and is looking after everything, automating everything (devops)
- Everyone is a specialist of everything and is looking after everything, automating everything, in all the environments (SRE)
- ... I wonder what's next...
I miss the good old days when developers could be developers and rely on DBAs, sysadmins and networkists to do their job well. I miss the days when developers were developing applications, sytems, modules,.. Not troubleshooting ELBs, RDS latencies or building monitoring for servers.
Hey, internet! Does chrome/FF have any plugins enabling any tab to become a monitoring dashboard? I have too many monitoring tools to keep an eye on at the same time... Opening multiple chrome windows and tiling them on the screen is one way, but tile 6 windows and the desktop gets cluttered by taskbars/arrdessbars/other stuff. Doing it all in one tab would save space.
Soo.. anyone knows the right tool for the job? TIA2
So I'm building this environmental monitoring system for one of the Labs to monitor Temperature and Humidity. the "software" that comes as part of the package with these sensors is really just a website you host yourself if you don't choose the cloud option. No big deal really, (see my previous rant about getting windows server through SSC) I setup IIS and get the "software" registered get a couple sensors running looks good. However I don't like the error messages that popup because it's unsecured. do some reading and I find out that most browsers will give you a warning if your not using HTTPS even if it's for internal use only. OK we'll how hard can it be in implement encryption, turns out it's not that hard and you can do it for free how with letsencrypt and other places. I like free, now i have to use SSH to get into the server and run an ACME client. Hey open SSH is part of windows now cool, download an ACME client SSH into the server and nope doesn't work. Oh right I'm behind a corporate firewall and a bunch of other shit I can't control. Why is so damn arduous to setup this god dam internal website and the problems aren't even the site. Now I'm playing with AWS spinning up an instance to be able to try and get an SSL certificate just so i don't have to tell people it's OK to trust this site ignore the big angry warning.
Best part is other similar internal sites don;t use SSL and all have big messages about someone stealing your soul if you go there and these are commercial systems that run all the HVAC for all the campuses across Canada.
I need more Tylenol.
Back in time i was monitoring an asterisk server on a friday night. Usually it's monitoring cli is a calm terminal with infos and periodic notifications. On a random check i saw about a KM length red shit / blue shit. As it turned out my boss was using the password 2500 with the same username on a fucking SIP server and while watching football (heard from the voice logs) some romanian script kiddie's brute force script fucked it up. The journey wasn't stopped here. Next step was to them to foreach some calls with high rates to their own special phone number on about 30-50 lines. The first step was to stop the service but because it is a nice app it wont stop till you have an active call, took about 5 mins to realise it . Had to kill it a few times until it gave up. That was the moment when the 'now they are gonna fire me' feel kicked in. Do not use weakass passwords kids!
MQTT - all I used to know about this is its name, untill few months back a client sent us some requirements which included MQTT. I opened its specification and I was fucking shocked! I am implementing almost similar protocol in most of my applications (which needs subscription based service) for last 3 years. I have developed IoT apps, remote monitoring systems, HMI systems using the same fucking protocol! Even I had implemented the same thing on HTTP using long polling a few years back!!
Now I feel like open sourcing my protocol. But I don't know where to start. Any help please?1
What do you use for performance monitoring on your infrastructure?
My company uses zabbix, OpenNMS and Nagios to monitor different parts of our infrastructure (from shared web hosting to OCCAS to IPTV to FutureVoice to Atlassian servers) but has no real-time performance checks.
I’ve set netdata master with prometheus backlog and grafana dashboards to monitor different metrics, however I am not sure whether any better approach could be done. Any suggestions?2
As expected, every ambulance chasing security company is banging on my door, trying to convince me that I need their antimalware/SIEM/monitoring service because GDPR.
You guys are shameless.1
The usability of perfmon on windows sucks! There is just no way for me to increase the size of the lower section of the main window where all my counters are listed. This is fine if I'm monitoring only 4-5 counters, but, that is never the case. Hoping that microsoft does something about this.
-i won't follow logging practices
-i won't follow secure coding
-i won't leverage profiling n monitoring tools
-i won't reuse best practices
-i won't listen to thought leaders
-i will outsource writing UT
-i will outsource code quality checks
-i will outsource all testing
-i will ignore n overide CTO team
But I still want high stability, security n 4 9s availability. Just want it done. My team is best. Am a fast-track leadership program leader who never has or ever needs to cod. I just know ...
People I have to deal with every sprint. Site reliability is not easy ...
Teaching good code makes great products to morons, toughest ...
"Beginners mind needed"2
When your business network monitoring tool is so ugly that it's disgusting to say it was a person who did it.
Stackoverflow just got us more work to do. Now we have to redesign the entire monitoring system just when we have finished implementing HA in several DCs.
One of the reasons why I wanted to become a software developer is because I see so many products or services taking the easy way out, at the cost of killing customer expectations. For example, I was told about JobTrack.io, which is supposed to help manage job searching by keeping track of applications and their statuses. But almost as quickly as I was told, my mind goes into automatic promise defense mode. And rightfully so, because the service turned out to be almost as monotaneous as the job search itself! Not as seamless as I'd need it to be to get started right away.
Now, maybe there's a slight chance I don't know wtf I'm talking about here. But, what's stopping this product from using an email client that runs server side, to interface with the user's main inbox, to run sentiment analysis on emails for detecting job application submissions? Such functionality would obviously need permission from the end user, so there are no surprises that some 3rd party app is sorta kinda monitoring your emails. And of course measures should be taken to avoid detecting anything beyond the contextual lines of: "Thank you for applying to so and so", or "We've recieved your application! Next steps".
Present those detections to the user to confirm. And do the same thing for rejections and offers. Shouldn't be that hard especially when most sites these days allow you to sign in with Google, and that Google marks these particular emails as "Important"; which further filters the detection process, and partially does JobTrack's job for them.
Honestly, I think the app has promise, and hope this is just a case of starting off small.
How effective are visualizations for monitoring infra on AWS?
Can visual infra monitoring be effective?
Pros and cons?
Relatively often the OpenLDAP server (slapd) behaves a bit strange.
While it is little bit slow (I didn't do a benchmark but Active Directory seemed to be a bit faster but has other quirks is Windows only) with a small amount of users it's fine. slapd is the reference implementation of the LDAP protocol and I didn't expect it to be much better.
Some years ago slapd migrated to a different configuration style - instead of a configuration file and a required restart after every change made, it now uses an additional database for "live" configuration which also allows the deployment of multiple servers with the same configuration (I guess this is nice for larger setups). Many documentations online do not reflect the new configuration and so using the new configuration style requires some knowledge of LDAP itself.
It is possible to revert to the old file based method but the possibility might be removed by any future version - and restarts may take a little bit longer. So I guess, don't do that?
To access the configuration over the network (only using the command line on the server to edit the configuration is sometimes a bit... annoying) an additional internal user has to be created in the configuration database (while working on the local machine as root you are authenticated over a unix domain socket). I mean, I had to creat an administration user during the installation of the service but apparently this only for the main database...
The password in the configuration can be hashed as usual - but strangely it does only accept hashes of some passwords (a hashed version of "123456" is accepted but not hashes of different password, I mean what the...?) so I have to use a single plaintext password... (secure password hashing works for normal user and normal admin accounts).
But even worse are the default logging options: By default (atleast on Debian) the log level is set to DEBUG. Additionally if slapd detects optimization opportunities it writes them to the logs - at least once per connection, if not per query. Together with an application that did alot of connections and queries (this was not intendet and got fixed later) THIS RESULTED IN 32 GB LOG FILES IN ≤ 24 HOURS! - enough to fill up the disk and to crash other services (lessons learned: add more monitoring, monitoring, and monitoring and /var/log should be an extra partition). I mean logging optimization hints is certainly nice - it runs faster now (again, I did not do any benchmarks) - but ther verbosity was way too high.
The worst parts are the error messages: When entering a query string with a syntax errors, slapd returns the error code 80 without any additional text - the documentation reveals SO MUCH BETTER meaning: "other error", THIS IS SO HELPFULL... In the end I was able to find the reason why the input was rejected but in my experience the most error messages are little bit more precise.2
did you know Verizon fios's own outage monitoring page isn't optimized for mobile? it's true. ask me how I know.2
So this my final year as an IT student, and I need to make a desktop app for the college which is about monitoring and diagnosing network flow and connected hardware, the question is: is there any stuff that can help and will it be better to use a specific language ?3
+ Taste Dartlang and Flutter
+ Do something with WebXR and/or WebAssembly
+ Start some lil projects
+ Learn more about Kubernetes and monitoring services
We've got new TV for monitoring, which auto-rotating meme page you like ? Cats, dogs, dank (sfw), dev, testing. Gimme yours !!! :)1
Um working on the solution to eradicating escalating diseases application and web based application..... Have three sections: 1. Emergency
2. HIV/tb monitoring/ report
3. Public info.
Iam gud at writing and codes but not good at expressing myself.
.any innovative fellow dev that can add me another section that I need to consider....my projects mission is to fight against disease world wide....any contribution or new ideas ?
So at my last job we had an AM deployment and a PM deployment. We had code reviews, QA, a slow roll process (deployed to three servers), monitoring process, and once everything checked out we fast rolled to the other servers.
At my current job we have a QA process, and we deploy once every three weeks.
My first job I deployed as needed, with no QA at all (I was the only web dev there).
I'm currently at a major e-commerce site, my last job was more of a click-bait site (though it still made millions in revenue each year).
So my question is: is there a "normal" as far as deployment schedules? I realize that each business type is going to have their own needs, but what's the "average" time between deployments?