AboutSysadmin of humble beginnings gradually becoming a reputed entrepreneur.
SkillsPuppet, lua, python, c++, bash, LinuxWizardry, documentationEnforcerDominator, pixelfucker
Joined devRant on 5/14/2016
Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Monitoring tools madness: quest foglight.
So, setting a blackout for an FMS "HA cluster" (which does not work due to a bug infested custom jboss implementation) can bring the servers down... And no way to bring them back up.
This brilliant piece of enterprise APM software costs 600.000€ for a 5year license.
I,ve added more drama (logs, threaddumps, support bundles and screenshots) to the support portal...
45 cases now in total, oldest case still open date 2017...
Fuck you quest software4
Worst enterprise software must be 'foglight' from quest.
I have ranted about this before...
Whenever you see the name 'quest software' or foglight, NOPE the fuck outta there.
You know you have to deal with annoying things when you take on a guard duty role and yes, we signed up for it because of the mullah.
However, you also want to do this with a reliable and robust monitoring and alerting systemthat you can depend on! And no i am not going to advertise a product for this... What i will tell you is which one to avoid.
Meet Quest "Foglight" ... It does EVERYTHING! It monitors, it alerts, it does trend watching it does fancy shmancy graphics, it does reporting, it is very extendable... WAUW, right! right?
Well, if you were stuck somewhere in 2005-2010 maybe... But this fucklight is cutting short on EVERYTHING
Today , i got called up at 3:30 in the morning (i am typing this after the incident) because this shit of a system has "HIgh Availability" by basically letting the FMS server suck each others jaggons and hope it somehow respons. This is a sort of keepalived thing, but on proprietary java tech..
Oh, yes, it's written on java and... yes.. Java 6
This means that, effectively we are running RHEL5 machines (yes, RHEL 5!!!) because something more modern in place? nope.
I have no idea anymore what i am ranting about, i'm tired, i'm tired of this shit, i'm tired of getting called up just because of some dude has been cussing up a sales representative, sucked each others jaggons and pushed the federal goverment with a shit solution for almost a decade now.
Fuck Quest software, because did you really think you would get enterprise level support for an enterprise product which you payed enterprise euro's for it? You are so naive, how cute...
And consequently : Fuck Dell and Good job Dell.. For purchasing quest software, mess around with it, and then dump it back to the market... Srsly Dell , you were like me when i had this hot ass chick as a girlfriend but later seemed to be too crazy to justifiably tolerate compared to her hotness. Dump it like it's trump.
Oh, and, wauw! Foglight graced us with a successful startup process after .. what.. 6 times restarting? In 2 hours... With 12 CPU's and 128 GB ram and .... oh fuck this you don't deserve such resources.3
During these interesting times it has certainly been a productive one for me. But after this fuckup i need to take a break. Also came to the reallisation i rely too much on Ctrl-r in terminal. I just needed to find that one long weird rsync thingy that i use once a quarter year...
:~$ history -c | grep rsync | grep...
I need a break. I royally fucked up now and i cannot be bothered right now to type that 25 lines of escaped backslashed one-liner rsync thing...3
Oh boy, this is gonna be good:
TL;DR: Digital bailiffs are vulnerable as fuck
So, apparently some debt has come back haunting me, it's a somewhat hefty clai and for the average employee this means a lot, it means a lot to me as well but currently things are looking better so i can pay it jsut like that. However, and this is where it's gonna get good:
The Bailiff sent their first contact by mail, on my company address instead of my personal one (its's important since the debt is on a personal record, not company's) but okay, whatever. So they send me a copy of their court appeal, claiming that "according to our data, you are debtor of this debt". with a URL to their portal with a USERNAME and a PASSWORD in cleartext to the message.
Okay, i thought we were passed sending creds in plaintext to people and use tokenized URL's for initiating a login (siilar to email verification links) but okay! Let's pretend we're a dumbfuck average joe sweating already from the bailiff claims and sweating already by attempting to use the computer for something useful instead of just social media junk, vidya and porn.
So i click on the link (of course with noscript and network graph enabled and general security precautions) and UHOH, already a first red flag: The link redirects to a plain http site with NOT username and password: But other fields called OGM and dossiernumer AND it requires you to fill in your age???
Filling in the received username and password obviously does not work and when inspecting the page... oh boy!
I ask for the seemingly "new" credentials and i receive new credentials again, no tokenized URL. okay.
Now Once i log in i get a horrible looking screen still made in the 90's or early 2000's which just contains: the claimaint, a pie chart in big red for amount unpaid, a box which allows you to write an - i suspect unsanitized - text block input field and... NO DATA! The bailiff STILL cannot show what the documents are as evidence for the claim!
Now we stop being the pretending dumbassery and inspect what's going on: A 'customer portal' that does not redirect to a secure webpage, credentials in plaintext and not even working, and the portal seems to have various calls to various domains i hardly seem to think they can be associated with bailiff operations, but more marketing and such... The portal does not show any of the - required by law - data supporting the claim, and it contains nothing in the user interface showing as such.
The portal is being developed by some company claiming to be "specialized in bailiff software" and oh boy oh boy..they're fucked because...
The GDPR requirements.. .they comply to none of them. And there is no way to request support nor to file a complaint nor to request access to the actual data. No DPO, no dedicated email addresses, nothing.
But this is really the ham: The amount on their portal as claimed debt is completely different from the one they came for today, for the sae benefactor! In Belgium, this is considered illegal and is reason enough to completely make the claim void. the siple reason is that it's unjust for the debtor to assess which amount he has to pay, and obviously bailiffs want to make the people pay the highest amount.
So, i sent the bailiff a business proposal to hire me as an expert to tackle these issues and even sent him a commercial bonus of a reduction of my consultancy fees with the amount of the bailiff claim! Not being sneery or angry, but a polite constructive proposal (which will be entirely to my benefit)
So, basically what i want to say is, when life gives you lemons, use your brain and start making lemonade, and with the rest create fertilizer and whatnot and sent it to the lemonthrower, and make him drink it and tell to you it was "yummy yummy i got my own lemons in my tummy"
So, instead of ranting and being angry and such... i simply sent an email to the bailiff, pointing out various issues (the ones6
TL;DR: work at the reference hospitals, we got precautions, no panic, we got this.
Well, currently my client is one of the reference hospitals in belgium in regards to coronavirus so they receive a lot of the infected patients. Although the general public is 'uneasy' to put it mildly, the IT department is not scared. We take our precautions, we already have safe working distances from each other. If the federal minister of health announces it, all non-medical or non-essential employees from the hospital will be either put on leave or work remote. Bottom line is: no panic. we got this!
'Every variable is a static constant once declared'
All variables in PuppetDSL, really hard to wrap the head around if you are 'used to' re-assign a variable a new value.5
In today's episode of "how i got almost to the point of insanity for hours and the sudden realization and relief"
When you have ssh error saying your private key is an invalid format in your CI, you probably just missed an EOL.
MCP says EOL.
That is the realization i made after half a day wasting on debugging this.4
Most satisfying was reducing the time my ci/cd did to build,test,verify complance and deploy of virtually anything i want in lrss then 10 minutes. From code to running appliance fully configured and being absolute certain it will work without any other modificatio . it used to be an hour.
Achieved this to do lots of caching and parallell test runs.
The downside is that my development server is feeling like a unvoluntary black person from ghana moving to the newfound united states 400 years ago...
I just got this linked from a friend of mine:
I believe this js widget can prove quite useful to many...3
Today we learned:
Don't run a backup integrity check on terabytes of data in multiple jobs your cpu can't handle.
And i cannot abort the process...
Guess i have to go outside, in the cold, brrrr.1
You know what is a nice phrase to write in your documentation right before you leave the job?
"I leave this to the evaluation and practice of the reader."
Such a delight to write that down.2
TL;DR: idiot 'team leader' does mindless merge to master. Precious time wasted in a high pressure deadline environment.
So, i work currently at one of Belgiums largest consulting company's at brussels airport, we are moving their analytics platform to the cloud.
We use puppet to manage the systems.
When i started i noticed immediately that their 'development workflow' is hardly to be named as such, because they simply change stuff directly on server , manual 'temporary' fixes everywhere, hardcoded stuff, non validated code... Basically the way one would develop in their garage, not in a consulting company as this one. But that is just the beginning.
A month ago i did a major effort to equalize all the discrepancies between the codebase and the server. Ensured entire codebase to be validated, syntax checked, parsed, tested... It works. A 'great codebase overhaul' commit was PR'ed to master and got merged.
Yesterday the team lead, i'll call him 'B-tard' from here on, has also 'equalized the discrepancies between codebase, server and the restnof the stale branches on the repo' . i was doing my other work on my branch so no fucks given. This is where i should have given some fucks.
Anyways, today. The day starts every day with merging the master branch into your working branh because you need the latest working codebase, right?
This fucking dipshit smug b-tard has done a mindless merge of the entire codebase, effectively removing ALL validated working code for provisioning servers. Control blocks, lookup functions, lambda's... Basically everything he did not understand.
At the same time the project is already way beyond the allotted budget in pkney and time, so there is a huge pressure to have a working 'production' environment TODAY!
THIS MOTHERFUCKING B-TARD JUST MADE THAT IMPOSSIBLE.
i'm loving this assignment, i'm loving the PM, the collegues, the environment, the location... everything. All but this fuckibg b-tard that somehow got his position by sucking dick or licking ass or both...
I wanna get out asap.
Oh... While typing this and arriving at the room of the office... It is locked, i have no key.
My own colo server. My own cloud. My own infra.
Fuck all of the CSP's and their fucking broken TOS and their data privacy violations!
Why do you think Amazon is so cheap? Because when they discover a product/service/software on AWS makes money, they WILL Reverse engineer it, make it and price you out of the market. It's their business model!5
The most efficient meeting you can have is the one that you don't have to go to.
Basically if a meeting is proposed and it's a pure waste of time. Just don't show up, spend time wisely.
My current job at the release & deploy mgmt team:
Basically this is the "theoretically sound flow":
* devs shit code and build stuff => if all tests in pipeline are green, it's eligible for promotion
* devs fill in desired version number build inside an excel sheet, we take this version number and deploy said version into a higher environment
* we deploy all the thingies and we just do ONE spec run for the entire environment
* we validate, and then go home
In the real world however:
* devs build shit and the tests are failed/unstable ===> disable test in the pipeline
* devs write down a version umber but since they disabled the tests they realize it's not working because they forgot thing XYZ, and want us to deploy another version of said application after code-freeze deadline
* deployments fail because said developers don't know jack shit about flyway database migrations, they always fail, we have to point them out where they'd go wrong, we even gave them the tooling to use to check such schema's, but they never use it
* a deploy fails, we send feedback, they request a NEW version, with the same bug still in it, because working with git is waaaaay too progressive
* We enable all the tests again (we basically regenerate all the pipeline jobs) And it turns out some devs have manually modified the pipelines, causing the build/deploy process to fail. We urged Mgmt to seal off the jenkins for devs since we're dealing with this fucking nonsense the whole time, but noooooo , devs are "smart persons that are supposed to have sense of responsibility"...yeah FUCK THAT
* Even after new versions received after deadline, the application still ain't green... What happens is basically doing it all over again the next day...
This is basically what happens when you:=
* have nos tandards and rules inr egards to conventions
* have very poor solution-ed work flow processes that have "grown organically"
* have management that is way too permissive in allowing breaking stuff and pleasing other "team leader" asscracks...
* have a very bad user/rights mgmt on LDAP side (which unfortunately we cannot do anything about it, because that is in the ownership of some dinosaur fossil that strangely enough is alive and walks around in here... If you ask/propose solutions that person goes into sulking mode. He (correctly) fears his only reason for existence (LDAP) will be gone if someone dares to touch it...
This is a government agency mind you!
More and more thinking daily that i really don't want to go to office and make a ton of money.
So the only motivation right now is..the money, which i find abhorrent.
And also more stuff, but now that i am writing this down makes me really really sad. I don't want to feel sad, so i stop being sad and feel awesome instead.1
My own server infra without configmgmt. Thanks to puppet i have been able to finally give my brain some more compute resources for other things in life, because managing them all by hand is almost the equivalent of a medieval monk copying the bible over and over again.
Now i can manage in the tens of thousands at relative ease.
My life could get worse, but it's really shitty now.
Suffering from a serious back injury since last year, my health has been not so positieve lately.
It put a toll on my mood, which in turn asked it's price regarding my relationship. Needless to say that did not go well. Already a fe months single but we kept in touch.
Three days ago my back injury returned, and was unable to lead a normal life. Constant pain, coyld not even move in the house. Even going to the toilet was a terrible experience because when you move, you're in a world of pain.
I asked my ex girlfriend to help me, since she was the only one having a key to my house.
When she arrived i hoped to have some moral support and to help me mive around, ensuring i would not injure myself any more.
Instead i received the cold shoulder. When she wanted to help pe up she did it a bit too hard and the pain sheered thrpughout my body. Screaming in pain.
She promptly left, leaving keys behind.
The hardest part is that she just left without me being able to explain clearly why i screamed. She thought i was yelling at her while in reality i was yelling due to the immense pain.
After that i had to cut ties forever. Tabula rasa. So i removed everything that is related to that time and locked it in my vault.
Since then i can hardly focus, my mibd is numb and i cannot think straight. The alcohol and other sedatives are probably also involved, but still i feel my life is a mountain of depressing shit.
Needed to vent. And yes i post this because i have a need for some understanding, yes for now i crave for some attention and some encouraging, supportive words. I'm left With no other options since the person i wanted it from the most has simply left... And the fact i am unable to actually be social outside...
Fuck friends and relationships, right?13
The only keyboard i will ever need...
* Steep learning curve
* where the fuck are my key combo's? Ah there they are!
* so comfy!
* much wow!
* such openness
* da blingbling
* wood finish!28
Dear diary, today was a good day.
1: i got the confirmation of promotion.
2: i solved a task using newly introduced tech and it works. Which has lots of implications on future work, a lot faster too. Also everyone is happy and supportive.
3: i felt good at the progress made with my kinesitherapy, my spine is starting to cooperate again.
Overall a good day.
Oh, and also i got payed :D1
Unfortunately it isnt everyday, but iy's nice from time to time to work outdoors while visiting family.2
Dont fuck with firewall rules when intoxicated.
I was on a weekend, my mailserver was acting weird again.
I do my shizzle, git commit, push.... And it broke
And i was too far gone tp notice on time where the forward rules were broken... That made it stop completely
At least it was not an open firewall
These two 'new' servers need hostnames.
Names must be within the following constraints:
* Names must be female
* Names must be pronouncable and writable in ASCII standard set of characters.
* Names are preferred to reference to duo's, like sisters, twins, dynamic duo's like chip and dale, but female.
Previous servers were name 'tairu' & 'mairu' (heroic age) and 'karen' & 'tsukihi' (nisemonogatori)
Let's see if the devrant community can surprise me.
Entries will be closed within 48 hours of this post.82
My best mentoring experience was the teacher that taught me puppet. The way he explained it, the way he walked you through a very intense course of three days consecutive puppet knowlegde rammed into your head...and still manage to actually learn almost all of it. Great attitude, and meanwhile, he is a esteemed collegue and friend...
Without you Johan, i would not be an independent entrepreneur and making bank at my current client
Johan the white, to me one of the greatest sysadmins ever
And this, ladies and gentlemen, is why you need properly tested backups!
TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.
So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken
Fuck my life5
TL;DR: fear of bricking my laptop due to typo pinning.
The worst nightmare i am living in right now...
I was noticing i did need some software in sid so i decided to use apt pinning for said software...
I configure the system, ok test looks good... I push it to production, run it on the system....and the nightmare starts.
Lits of packages get updated, and i am screaming 'noooooooo' since debian sid softwarz can sometimes break everything! I discovered that i did test my apt pinning config for the presence of the amount of numbers, but not at their value... Sooo, by accident swapping pin numbers for stable and unstable you get... Your worst apt-get update nightmare...
I hope it does not become a brick.1