I hired a woman for senior quality assurance two weeks ago. Impressive resume, great interview, but I was met with some pseudo-sexist puzzled looks in the dev team.

Meeting today. Boss: "Why is the database cluster not working properly?"

Team devs: "We've tried diagnosing the problem, but we can't really find it. It keeps being under high load."

New QA: "It might have something to do with the way you developers write queries".

She pulls up a bunch of code examples with dozens of joins and orderings on unindexed columns, explains that you shouldn't call queries from within looping constructs, that it's smart to limit the data with constraints and aggregations, hints at where to actually place indexes, how not to drag the whole DB to the frontend and process it in VueJS, etc...

New QA: "I've already put the tasks for refactoring the queries in Asana"

I'm grinning, because finally... finally I'm not alone in my crusade anymore.

Boss: "Yeah but that's just that code quality nonsense Bittersweet always keeps nagging about. Why is the database not working? Can't we just add more thingies to the cluster? That would be easier than rewriting the code, right?"

Dev team: "Yes... yes. We could try a few more of these aws rds db.m4.10xlarge thingies. That will solve it."

QA looks pissed off, stands up: "No. These queries... they touch the database in so many places, and so violently, that it has to go to therapy. That's why it's down. It just can't take the abuse anymore. You could add more little brothers and sisters to the equation, but damn that would be cruel right? Not to mention that therapy isn't exactly cheap!"

Dev team looks annoyed at me. My boss looks even more annoyed at me. "You hired this one?"

I keep grinning, and I nod.

"I might have offered her a permanent contract"

Hey everyone!

devRant will be going down on Friday, July 7th around 10:30pm EDT so we can do some database maintenance and restructuring of our cluster. It hopefully won't be down for more than about 30 minutes or so, and during that time you should see our "down for maintenance" message.

If you usually use devRant while you're on the toilet (we know many do!), we apologize and suggest you try to schedule around this!

Please let me know if you have any questions and apologies for the inconvenience.

rant? rant!

I work for a company that develops a variety of software solutions for companies of varying sizes. The company has three people in charge, and small teams that each worked on a certain project. 9 months ago I joined the company as a junior developer, and coincidentally, we also started working on our biggest project so far - an online platform for buying groceries from a variety of vendors/merchants and having them be delivered to your doorstep on the same day (hadn't been done to this scale in Estonia yet). One of the people from management joined the team working on that. The company that ordered this is coincidentally being run by one of the richest men in Estonia. The platform included both the actual website for customers to use, a logistics system for routing between the merchants, the warehouse, and the customers, as well as a bunch of mobile apps for the couriers, warehouse personnel, etc. It was built on Node.js with Hapi (for the backend stuff), Angular 2 (for all the UIs, including the apps which are run through a WebView wrapper), and PostgreSQL (for the database). The deadline for the MVP we (read: the management) gave them, but we finished it in about 7 months in a team of five.

The hours were insane, from 10 AM to 10 PM if lucky. When we weren't lucky (which was half of the time, if not more), we had to work until anywhere from 12 PM to 3 AM, sometimes even the whole night. The weekends weren't any better, for the majority of the time we had to put in even more extra hours on the weekends. Luckily, we were paid extra for them, but the salary was no way near fair (the majority of the team earned about 1000€/mo after taxes in a country where junior developers usually earn 1500€/month). Also because of the short deadline given to us, we skipped all the important parts like writing tests, doing CI, code reviews, feature branching/PR's, etc. I tried pushing the team and the management to at least write tests and make feature branches/PRs, but the management always told me that there wasn't enough time to coordinate and work on all that, that we'll do that after launching the MVP, etc. We basically just wrote features, tested them by hand, and pushed into the "test" branch which would later get tested and merged into master.

During development, one of the other juniors managed to write the worst kind of Angular code you could imagine - enormous amounts of duplication, no reusable components (every view contained the everything used in the view, so popups and other parts that should logically be reusable were in every view separately), fuck - even the HTML was broken (the most memorable for me were the "table > tr > div > td" ones, but that's barely scratching the surface). He left a few months into the project, and we had to build upon his shit, ever so slightly trying to fix the shit he produced. This could have definitely been avoided if we did code reviews.

A month after launching the MVP for internal testing, the guy working on the logistics system had burned out and left the company (he's earning more than twice the salary he got here, happy for him, he is a great coder and an even better team player). This could have been avoided if this project had been planned better, but I can't really blame them, since it was the first project they had at this scale (even though they had given longer deadlines for projects way smaller than this).

After we finished and launched the MVP, the second guy from management joined, because he saw we needed extra help. Again I tried to push us into investing the time to write tests for the system (because at this point we had created an unstable cluster fuck of a codebase), but again to no avail. The same "no time, just test it manually for now, we'll do that later when we have time" bullshit from management.

Now, a few weeks ago, the third guy from management joined. He saw what a disaster our whole project was. Him joining was simply a blessing from the skies. He started off by writing migrations using sequelize. I talked to him about writing tests and everything, and he actually listened. He told me that I'm gonna be the one writing them, and also talked to the rest of management about it. I was overjoyed. I could actually hear the bitterness in the voices of the rest of management when they told me how to write the tests, what to test, etc. But I didn't give a flying rat's ass, I was hapi.

I was told to start off by writing a smoke test for the whole client flow using Puppeteer. I got even happier, since I was finally able to again learn new things (this stopped at about 4 or 5 months into the project).

I'm using jest as the framework and started writing the tests in TypeScript. Later I found a library called jest-extended, but it didn't have type defs, so I decided to write them and, for the first time in my life, contribute to the open source community.

Finish my Raspberry Pi Zero Cluster... complete with kubernetes, etc.

... I still don't know what I'm going to run on this. suggestions in the comments welcome.

Never have I been so furious whilst at work as yesterday, I am still super pissed about going back today but knowing it's only for another few weeks makes it baerable.

I have been the lead developer on a project for the last 3~ months and our CTO is the product owner. So every now and then he decides to just work on a feature he is interested in- fair enough I guess. But everything I have to go and clean up his horrendous code. Everything he writes is an absolute joke, it's like he is constantly in Hackathon mode "let's just copy and paste some code here, hardcoded shit there and forgot about separation of code- it all goes in 1 file".

So yesterday he added a application to the project and instead of reusing a shared data access layer he added an entirely new ORM, which is near identical to the existing ORM in use, for this one application.

Being anal about these things, the first thing I did was delete his shit and simply reference the shared library then refactor a little code to make it compatible.

WELL!! I certainly hit a nerve, he went crazy spamming messages on Slack demanding I revert as it broke ONE SINGLE QUERY that he hadn't checked in (he does 1 huge commit for 10 of everyone else's). I stuck to my principals and explained both ORM's are similar and that we only needed one, the second would cause a fragmented codebase for no benefit whatsoever.

The lead Dev was then forced to come and convince me to revert, again I refused and called out the shit quality of their code. The battle raged on via the public slack group and I could hear colleagues enjoying the heated debate, new users even started joining the group just to get in on mine and the cto's difference of opinion.

I even offered to fix his code for him if he were to commit it, obviously that was not taken well ;).

Once I finally got a luck at the cluster fuck of shit he had written it took me around 5 minutes to fix and I ever improved performance. Regardless he was having none of it. Still the demands to revert continued.

I left the office steaming after long discussions with the lead Dev caught in the middle.

Fortunately my day was salvages with a positive technical discussion that evening at a company with whome I had a job offer from.

I really hate burning bridges and have never left a company under bad terms but this dictator is making me look forward to breaking the news today I will be gone in 4 weeks.

!rant

So last weekend I started collecting hardware for a small scale cluster at home to test scalability of my software. Making some decent progress.

Tomorrow I will replace the switch and this weekend I will set up storage so I can start my first application

So, some time ago, I was working for a complete puckered anus of a cosmetics company on their ecommerce product. Won't name names, but they're shitty and known for MLM. If you're clever, go you ;)

Anyways, over the course of years they brought in a competent firm to implement their service layer. I'd even worked with them in the past and it was designed to handle a frankly ridiculous-scale load. After they got the 1.0 released, the manager was replaced with some absolutely talentless, chauvinist cuntrag from a phone company that is well known for having 99% indian devs and not being able to heard now. He of course brought in his number two, worked on making life miserable and running everyone on the team off; inside of a year the entire team was ex-said-phone-company.

Watching the decay of this product was a sheer joy. They cratered the database numerous times during peak-load periods, caused $20M in redis-cluster cost overrun, ended up submitting hundreds of erroneous and duplicate orders, and mailed almost $40K worth of product to a random guy in outer mongolia who is , we can only hope, now enjoying his new life as an instagram influencer. They even terminally broke the automatic metadata, and hired THIRTY PEOPLE to sit there and do nothing but edit swagger. And it was still both wrong and unusable.

Over the course of two years, I ended up rewriting large portions of their infra surrounding the centralized service cancer to do things like, "implement security," as well as cut memory usage and runtimes down by quite literally 100x in the worst cases.

It was during this time I discovered a rather critical flaw. This is the story of what, how and how can you fucking even be that stupid. The issue relates to users and their reports and their ability to order.

I first found this issue looking at some erroneous data for a low value order and went, "There's no fucking way, they're fucking stupid, but this is borderline criminal." It was easy to miss, but someone in a top down reporting chain had submitted an order for someone else in a different org. Shouldn't be possible, but here was that order staring me in the face.

So I set to work seeing if we'd pwned ourselves as an org. I spend a few hours poring over logs from the log service and dynatrace trying to recreate what happened. I first tested to see if I could get a user, not something that was usually done because auth identity was pervasive. I discover the users are INCREMENTAL int values they used for ids in the database when requesting from the API, so naturally I have a full list of users and their title and relative position, as well as reports and descendants in about 10 minutes.

I try the happy path of setting values for random, known payment methods and org structures similar to the impossible order, and submitting as a normal user, no dice. Several more tries and I'm confident this isn't the vector.

Exhausting that option, I look at the protocol for a type of order in the system that allowed higher level people to impersonate people below them and use their own payment info for descendant report orders. I see that all of the data for this transaction is stored in a cookie. Few tests later, I discover the UI has no forgery checks, hashing, etc, and just fucking trusts whatever is present in that cookie.

An hour of tweaking later, I'm impersonating a director as a bottom rung employee. Score. So I fill a cart with a bunch of test items and proceed to checkout. There, in all its glory are the director's payment options. I select one and am presented with:

"please reenter card number to validate."

Bupkiss. Dead end.

OR SO YOU WOULD THINK.

One unimportant detail I noticed during my log investigations that the shit slinging GUI monkeys who butchered the system didn't was, on a failed attempt to submit payment in the DB, the logs were filled with messages like:

"Failed to submit order for [userid] with credit card id [id], number [FULL CREDIT CARD NUMBER]"

One submit click later and the user's credit card number drops into lnav like a gatcha prize. I dutifully rerun the checkout and got an email send notification in the logs for successful transfer to fulfillment. Order placed. Some continued experimentation later and the truth is evident:

With an authenticated user or any privilege, you could place any order, as anyone, using anyon's payment methods and have it sent anywhere.

So naturally, I pack the crucifixion-worthy body of evidence up and walk it into the IT director's office. I show him the defect, and he turns sheet fucking white. He knows there's no recovering from it, and there's no way his shitstick service team can handle fixing it. Somewhere in his tiny little grinchly manager's heart he knew they'd caused it, and he was to blame for being a shit captain to the SS Failboat. He replies quietly, "You will never speak of this to anyone, fix this discretely." Straight up hitler's bunker meme rage.

We are moving to kubernetes.
Nothing much has changed except we now get to say
it works on my cluster.
instead of
it work on my machine.

Firefox is running in our universe and are some bits of data

A Developer is desperate: his java application servers are unresponsive, thousand of dead zombie threads are sucking all cpus, memory is leaking everywhere, garbage collector has gone crazy, the cluster sessions are fucked....

The Developer goes to the closest bridge, ties a stone to his neck and gets ready to jump.

Suddenly a bearded old man with a fiery look runs toward him, yelling:
- stop stop!!!! Your application is not scaling and misconfigured, your servers are melting, cpu usage is not sustainable anymore, but don't despair

The Developer, puzzled, looks at him:
-I've never seen you...how do you know...
- Hey, man, I'm the Devil. I know everything. All your problems are solved. I'll give you magic functions. They are called Lambda.
You'll never have to worry about your servers, scalability, security, configuration and shit.

The Developer seems astonished but relieved:
- Ok, sounds great! let's try it - suddenly suspicion creeps in - hmmmm but you are the Devil....so...you want something back, don't you?
(the Devil nods lightly with a diabolic smile)
- ...and...you want my soul, I guess...
- your soul??? come on!!! - the Devil burst in a laugh - we are in 2019. I don't care about your soul. I want your ass.
- What!???!!!?
- yes, I want to fuck your ass

The Developer, evaluates quickly the situation.
Few moments of pain or slight discomfort (?) in exchange for magic lambda. It could be worth. He accepts.

After a while of rough anal fucking, the devil asks
- Hey, how old are you anyway?
- 45, why?
- Oh jeeez...45!!!??? and you still believe in the devil?

Story time!!!

(Disclaimer: I did not actually make a bomb)

So I had this amazing teacher in grade 7 (he gave me an Arduino for free) and he also let me and the annoying kid from one of my other rants use his storage room as a makeshift makerspace.. we had a raspberry pi cluster and more Arduino and that we could dream of having.. also cool stuff like leap motion controllers...

One day he was out sick and had this strict supply teacher in. The two of us went in to the storage room at lunch break and started working on a doorbell for the room.. I was doing some circuitry..

The bell rang and we left..

The next class, the supply teacher and the principal walk in and ask me to come out into the Hall.. I am like "fuck" ..

She and the principal start yelling at me about making a bomb and burning down the school and other outrageous things.. she started telling me about kids that got swatted at school for making bombs and that I was in big trouble..

The whole time I am like.... Wtf are these people talking about???

Then eventually they start asking me questions that I didn't understand.. and they ask if Dan (other kid) was doing it too.. I said yes because I was hoping that they would bring him out.. he might know what they are talking abt..

He then comes out into the hallway and they start yelling at him..

Then I realize..

I think that I left our makeshift battery pack on..

So I go and check.. sure enough, the battery had just shorted out and melted the plastic (which was giving off the smell that she was talking abt) so I unplugged everything (and electrocuted myself)

So ya.... I made a mistake and I got a unfairly harsh lecture about it..

Lesson learned.. don't let stupid people see you do anything computer or circuitry related....

My weekend project. Anyone else built a Raspberry Pi 3 cluster (for Kubernetes)?

Some days I just want to shoot myself.

I get why... someone might do this, but sweet mother of god!

Rare footage of me waiting for my Kubernetes cluster to start.(2018BC,colorized)

This how every cluster-fuck begins..

CEO - So... We'll have a new side project, it's a small thing, I spoke with a guy, he needs just a small thing...
Me - Ok....... So, what do you need me for?
CEO - Not much, this guy started a project but wants your help on a small part, and you already did something similar for us, so it should be fast, just copy and paste and change a little bit. You probably know better than me.
Me - *Sigh* ... So `friend`, what do you need from me?
Friend - So, I made a crawler that is storing some information on a local file, I just need to add multiprocessing and multithreading, a producer-consumer system with a queue so I can automatically add new links every now and then, a failback system, so when a process doesn't finish, it should be re-queued and crawled later, store all the information on a database cluster (that is not set up), [......]
Me - And when is this supposed to be up and running?
Friend - Your CEO told me you could do this by the weekend. Can we finish this by Friday?
Me - *facepalm* FML

I am running a small - but growing - ceph-cluster at work. Since it is fun and our storage demand is growing each day.

Today, it was time to bring another node online and add another 12TB to the cluster.
Installation of the OS went fine, network settings fine, drives looks fine.

Now, time to add it into the cluster.... BAM
Every Dell machine in the Cluster - Dead.
The two HP-machines is online and running. But the Dell-machines just died.

WAT!?

Yep. So the dev teams boss says it's fine to run a production environment on a single Windows instance with the db on that same instance, which they already totally lost once from a reboot after an auto update before I came along tasked with fixing the cluster fuck they created.

This from a man who somehow runs a dev team while using gmail via the web because he can't use an email client, uses email to track tasks but can't because they get lost amongst his 3000+ unread emails, has a screen dirtier than a hookers vag on half priced Tuesday, and got a new laptop but had to get his daughter to set it up and transfer his data because he couldn't.

But ok... you have a degree, You must know what you're doing.
It's ok though, I'll keep covering your incompetent ass while you keep raping the company because no one listens.
Peoples ignorance and arrogance astounds me.

Inception.
Today I needed to check something in a remote server: this was the easiest way:
1: teamviewer to my home pc from university
2: started a vm on that machine with vpn connection to my work office
3: rdp to a windows server vm
4: ssh to a vm on our hosting cluster
5: from there, ssh to the server that I needed access to

Manager: "Can we get an accurate report on how many containers we have on the Kubernetes cluster?"

Me: "Well not really since Kubernetes is designed to be dynamic and agile with the number of resources and containers being created and deleted being subject to change at a moment's notice."

Manager: "I want numbers"

Me: "Okay well if we look at a simple moving average over time, we can see how the number of containers changes and then grab a rough answer from that"

Manager: "These numbers look a little round, are you sure these are exact?"

I'm going to throw myself into a pile of used heroin needles and hope i get stuck with whatever the hell this guy has to somehow be a manager while also being this retarded.

Went in to the therapist today. Their impression is I'm likely okay. Wait until I show them the cluster fuck that is Java Enterprise. 🙃

"Let's create a docker-swarm cluster thingy for this application with horizontal scaling to learn docker and run this application better!"

This stuff is so overwhelming and I don't understand half of how to possibly set this up 😅

> Monitoring: Load Average of 57!! ALERT!!!!
> me: What? That's not possible?
> *Monitoring froze 14 hours ago*
> *sshs into server*
> *see attached image*

The issue was ~1200 df processes that were issued by our monitoring system and all of them didn't finish because the external cluster we mounted onto that server died a few minutes before that. Just re-mounting the cluster fixed it but still a funny sight!

We are building this big-data engine for a client's product for which we were using a cluster on GCP and they were billed ~1100$ for the last month's usage.
The CTO - the CHIEFFUCKING TECHNOLOGY OFFICER told us to hook up 5-6 laptops in our server room and create our own cluster because they cannot afford so much bill.

Nice Windows Defender :) I should buy a cluster next time to use this antivirus :P

Sometimes I daydream about just selling out and doing the whole diversity and women who code kind of thing. Apply for the big tech companies and work my way towards becoming one of the faces of their bullshit company. I'm gonna get a long wig, tan myself brown, and wear subtle native shit like a carabao brooch or some Cheongsam-inspired work suit. And when there's a big work event or party, I'm gonna go full on White Rose from Mr Robot, smiling and shaking people's hands like I give a shit. Gonna try to look wholesome and innocent, all I really wanted was a chance to get into tech as a humble being.

I'm gonna practice a strong accent and I would tell humble stories about where I came from, that god sent me here, and about all the struggles I had getting into the field as a woman. I'm gonna step up to the podium and say shit like, "Women can do it too, we can kuh-hode!" Then I'm gonna stir up some shit like, "Why is the terminal black?" Gonna make my voice and hands shake from intense emotion (visible anger, almost crying), "Why is something that we use to do our bidding black? Are we not over the years of slavery in the past? What does a terminal represent? Slavery. You know what else is terminal? Cancer. Slavery and racism are the cancers of our society. We must stop this. Terminals should be in all colors and it shouldn't be black by default."

1. Point out a non-malicious thing and add malice to it. The more racist and sexist, the better.
2. Mention something bad that is not related to it at all.
3. Make them seem related in an attempt to make a point.
4. Fail to make a point.
5. Say "we must stop this".

I'm gonna look straight into people's eyes with a serious face while cry-laugh-cringing internally. "As a brown woman, this makes me sad. I have Kubernetes skills, you know and it hurts me." If I catch some people laughing even though I'm also about to fart from internal laughter, I will point at the whitest male of them all. "You, do you think this is funny? Because this is my life, as well as millions of other people like ME!"

Then I'm gonna curse all of the white men who obviously only got into this field through violence. Yes, Mike, I know you pointed a gun at the HR to get in. What else could it be? All you know is violence because testosterone! You don't really know Kubernetes, I do, I set it up all the time for my thousands of hello world applications and yet here you are, just walking in with your white boy genes and your guns and taking all our jobs away. "What do you have to say for yourself? Shame."

Then of course, some hardcore social media "journalists" are gonna capture this on video and people are gonna ruin whoever man I pointed out's life and say shit like, "This big white man laughed at this little brown chick while she was talking. Sexism and racism in one video. This is why tech is so toxic." I'm gonna allow myself to get caught with a boner (big plastic white bottle) and people are gonna talk about how they didn't use a real woman as the face of their women who code thing. "I admit, I was born in a body that did not align with my desires but just because I have a penis doesn't mean I can't wear a Cheongsam. Also, the fact that you're pointing this out is discrimination and I had enough of that in my life. Shame. This is why tech is so toxic. You accept black terminals but not Cheongsam dicks. Fuck you."

I'm gonna keep ranting about big bad white men while dating one. Of course, people will find this out and point it out. I would flip it and say, "See? This is the problem. You look down on people like me because you're all racist. Just because I'm brown doesn't mean I can't date someone who is white. Sexist. Just because I'm a woman doesn't mean I don't have a choice who to date."

I'll keep ranting about all the ridiculous shit, just calling people names every time they speak. Just shame them for no reason then after several years of collecting my diversity money, I'm gonna leak a porn video of me. A compilation of sucking white man dick with voice overs just saying "big bad white men" over and over again. Some really degrading porn kinda shit. "You know what else scales? My ass cluster. Yeah, baby, fill up my containers with your sweet, sweet, love juice."

God bless whoever hires me and make me the face of their company. They will be forever ruined.

My company is providing cloud infrastructure to our customers. For research purposes we are running a little openstack cloud in our laboratory datacenter were we can test stuff before implementing it in the productive environment.

Last week the manager asked me to shut down the cluster over night and only power on the servers when we need it. (about twice a week)

The reason: it produces too much heat.

My answer was: No.
First off thats not how cloud infrastructure works, and how about a proper climate control?

Sometimes i ask myself in which parallel universum our managers live 😑

Finally got ceph up and running, so far it does look really great!
At the moment we have 45 TB in the cluster, but we are able to expand it extremely easy.

Two servers with 256 GB RAM and 24 cores, three more with 32 GB and 12 cores.
Dedicated gigabit WAN connection.

The project is going forward strong, I'll let you know when it is time for beta testing.

When your co-worker asks for a more memorable hostname for the next dev cluster...

IT is so violent and disturbing when it comes to technical terms in relation to clusters:

- master / slave
- master / master / witness (crime scene?)
- splitbrain (how does that look in reality?)
- stonith (shoot the other node in the head)
- deadtime (no heartbeat)
- slicing (wtf?)
- shard (do they kill with that?)
- active / passive (sex scene?)
- single point of failure (at least they know who is to blame)
- share nothing / shared all (dictators vs communism)

😯

OMFG! Who’s bright fucking horrible stupid ass idea was it to mix Ajax with php (php deciding the ajax paths) with random js outputting HTML inside random fucking static divs found no where near the logical route of content.

Trying to add a simple fucking status to a gigantic cluster fuck of a legacy project is just FUCK.

If I could I would burn this bitch to the ground and start again I would, But no, it’s needed.

Someone kill me before I break the shit out of this thing, I would take a wordpress project right now instead.

Personally the coolest was the program I built for my fathers use on his job.

It was my first to be used commercially in the real.

That was a very big thing, I was 17 at the time an used turbo pascal 5.5 and he used it to compute how well all machinery was doing, they rented out diggers and other construction equipment to construction sites and manually compute this with a calculator took up to three days. (This was 1987 so there was not very many ready made programs for business, you often had to build your own)

With this program he had it done in around 30 minutes.

The next best was recently when I got my raft distributed consensus cluster server working. Its a little bit like zookeeper.
Building that purely from the research paper was rewarding but a bit of a challenge.

So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?

I received a job offer as web-app developer and, in order to access the interview, I had to do an online QI test.

I login in the website and I see an ugly cluster of links, missing informations, errors at receiving the data I just inserted, a lot of bugs, broken links, images not availables, blank pages and so on.

I'm not sure I want to work there anymore

Between containers, cluster managers and virtual machines we've lost track of where our code even is.

I once found a MongoDB cluster open to the internet with no authentication with nearly a terabyte of data that backed a CRM service whose customers included Microsoft and Adobe to name a few.

We have a 15-machine cluster that went down last night because one machine in the cluster went down. Apparently having a cluster for redundancy is just a nice idea and doesnt actually work in practice.

Also I shouldnt have to go to a vendor's forums to find out the bug that is causing my cluster to go down is fixed in a future version. It should be in the goddamn patch notes!!!

Should’ve posted this after it happened, but it requires a bit of background anyway.

There’s this guy that oversees our OpenStack environment. My team often make jokes and groan about him in private because he’s so overbearing. A few months back, he had to take us to our data center to show us our new racks, and he kept saying stupid stuff like “you break this and it costs me $30,000” as if he owns everything. He’s just... one of THOSE people. Always speaks in such a condescending way. We make jokes that he is our “best friend”.

Our company is shifting most of our products to the cloud in response to the coronavirus (trying to make it an opportunity for “innovation”). This has involved some structural and responsibility changes in our department, and long story short, I’m now heading the OpenStack environment alongside other projects.

This means going through grueling 1-on-1 meetings with our “best friend”. It’s not too bad, I can be pretty patient with people, so I didn’t mind too much at first. Then a few things happened.

1. He sent a shared folder that he owned containing info related to the environments. Several documents were outdated and incomplete, so I downloaded them, corrected them, and then uploaded the documents to my teams file share, as I was supposed to since we now own the projects.

2. Several files were missing, and when I asked about them, he said “Oh, did you refresh the browser?”. I told him no, that I downloaded them locally and republished them to my teams server, because he was supposed to hand everything off to us at once. He says “Well, silly, how are you going to get updates if you’re looking at them locally?” and kind of chuckles at me like I’m stupid.

3. He insists on training me how to remote into one of the servers to check on cluster space, which in itself is fine. I understand others wanting to make sure things will be done right by the people who come after them. But he tells me to download SuperPutty. I tell him, “oh no, that’s alright. I don’t need putty”. He says “oh cool, what tool do you use for ssh?”. I answer him “Just Git. If I want to I can use a CentOs bash terminal too, because we have WSL installed”. He responds “You can’t ssh through Git”.
I was actually a little shocked. I didn’t know if he was serious or not so I was silent for a few seconds before hesitantly saying “yes you can”. He says “this is news to me” and I so I tell him “every single one of our build jobs fetches code from Git with ssh” and he seemed genuinely shocked and surprised by that.... so then it occurs to me to show him that you can ssh in Powershell and that REALLY blew his mind. He would not shut up about it for several minutes. I was amused until it just got annoying.

Needless to say, my team had been previously teasing me about having to work with him, so they found it hilarious when I told them afterwards.

"four million dollars"

TL;DR. Seriously, It's way too long.

That's all the management really cares about, apparently.

It all started when there were heated, war faced discussions with a major client this weekend (coonts, I tell ye) and it was decided that a stupid, out of context customisation POC had that was hacked together by the "customisation and delivery " (they know to do neither) team needed to be merged with the product (a hot, lumpy cluster fuck, made in a technology so old that even the great creators (namely Goo-fucking-gle) decided that it was their worst mistake ever and stopped supporting it (or even considering its existence at this point)).

Today morning, I my manager calls me and announces that I'm the lucky fuck who gets to do this shit.

Now being the defacto got admin to our team (after the last lead left, I was the only one with adequate experience), I suggested to my manager "boss, here's a light bulb. Why don't we just create a new branch for the fuckers and ask them to merge their shite with our shite and then all we'll have to do it build the mixed up shite to create an even smellier pile of shite and feed it to the customer".

"I agree with you mahaDev (when haven't you said that, coont), but the thing is <insert random manger talk here> so we're the ones who'll have to do it (again, when haven't you said that, coont)"

I said fine. Send me the details. He forwarded me a mail, which contained context not amounting to half a syllable of the word "context". I pinged the guy who developed the hack. He gave me nothing but a link to his code repo. I said give me details. He simply said "I've sent the repo details, what else do you require?"

1st motherfucker.

Dafuq? Dude, gimme some spice. Dafuq you done? Dafuq libraries you used? Dafuq APIs you used? Where Dafuq did you get this old ass checkout on which you've made these changes? AND DAFUQ IS THIS TOOL SUPPOSED TO DO AND HOW DOES IT AFFECT MY PRODUCT?

Anyway, since I didn't get a lot of info, I set about trying to just merge the code blindly and fix all conflicts, assuming that no new libraries/APIs have been used and the code is compatible with our master code base.

Enter delivery head. 2nd motherfucker.

This coont neither has technical knowledge nor the common sense to ask someone who knows his shit to help out with the technical stuff.

I find out that this was the half assed moron who agreed to a 3 day timeline (and our build takes around 13 hours to complete, end to end). Because fuck testing. They validated the their tool, we've tested our product. There's no way it can fail when we make a hybrid cocktail that will make the elephants foot look like a frikkin mojito!

Anywho, he comes by every half-mother fucking-hour and asks whether the build has been triggered.

Bitch. I have no clue what is going on and your people apparently don't have the time to give a fuck. How in the world do you expect me to finish this in 5 minutes?

Anyway, after I compile for the first time after merging, I see enough compilations to last a frikkin life time. I kid you not, I scrolled for a complete minute before reaching the last one.

Again, my assumption was that there are no library or dependency changes, neither did I know the fact that the dude implemented using completely different libraries altogether in some places.

Now I know it's my fault for not checking myself, but I was already having a bad day.

I then proceeded to have a little tantrum. In the middle of the floor, because I DIDN'T HAVE A CLUE WHAT CHANGES WERE MADE AND NOBODY CARED ENOUGH TO GIVE A FUCKING FUCK ABOUT THE DAMN FUCK.

Lo and behold, everyone's at my service now. I get all things clarified, takes around an hour and a half of my time (could have been done in 20 minutes had someone given me the complete info) to find out all I need to know and proceed to remove all compilation problems.

Hurrah. In my frustration, I forgot to push some changes, and because of some weird shit in our build framework, the build failed in Jenkins. Multiple times. Even though the exact same code was working on my local setup (cliche, I know).

In any case, it was sometime during sorting out this mess did I come to know that the reason why the 2nd motherfucker accepted the 3 day deadline was because the total bill being slapped to the customer is four fucking million USD.

Greed. Wow. The fucker just sacrificed everyone's day and night (his team and the next) for 4mil. And my manager and director agreed. Four fucking million dollars. I don't get to see a penny of it, I work for peanut shells, for 15 hours, you'll get bonuses and commissions, the fucking junior Dev earns more than me, but my manager says I'm the MVP of the team, all I get is a thanks and a bad rating for this hike cycle.

4mil usd, I learnt today, is enough to make you lick the smelly, hairy balls of a Neanderthal even though the money isn't truly yours.

Once upon a time, one or two jobs ago, a really awesome engineer specced out a distributed search application in response to a business need. This company was managed pretty oldschool and required a ton of paperwork and approvals.

The engineer spent many weeks running tests and optimizing the hell out of this app cluster. It flew, and he had the data to prove it could handle production workloads (think hundreds of terabytes of data being processed every single day)

Part of the way he achieved this was having RAID0 on all of the servers to maximize I/O throughput. He didn't care much about data loss, since the application itself was fault tolerant on a much more granular level.

Management, hearing about this, absolutely flipped their shit and demanded RAID6 instead. This despite the conclusive data that the engineer had that proved RAID6 couldn't keep up.

He more or less got told to STFU.

Even this despite the fact that a RAID restripe would actually take many times longer than rebuilding the failed node from scratch (a process that took about 30 minutes by hand, and could probably be automated to be done in less than five), causing a longer exposure to actual data loss throughout the length of the days-long array rebuild time.

The ill-thought-out requirement added about 50% to the cost of the project (*many* more hard drives now required), beyond the original budget, and the subsequent bureaucratic wrangling resulted in a late product launch.

6 months or so later, after real customers were using this product, the app was buckling under around half of its expected workload. A friend of the engineer suggested to management to try RAID0. Sure enough, that resolved the I/O bottleneck.

This rage-inducing story has a happy ending, though! Said engineer left the company not long after this incident, citing it as a reason for his departure. He was immediately hired by another company, making integer multiples of his prior salary.

The product the company botched the launch of by ignoring his spec? It died a few months later. Maybe the poor customer experience was to blame? Maybe the late launch? Maybe it was another reason entirely.

Either way, millions of dollars of hardware now sat fallow. This was a black eye on the company all the way up to the C-level.

tl;dr: Listen to your engineers. You hired them for their expertise.

Too many night shifts.

But it's done.

After the last migrations my emotional state is... Questionable.

VM migrations between different CPU vendors and generations leading to segfaults because of unsupported X86 extensions.... Thx for doing that at 23 o'clock after 8 hours of work....

Forgetting a left over NIC in a virtual machine, creating a routing loop, leading to very erratic behaviour and fun things.

Someone forgot to check the '"Unique" box, mass spawning a cluster of VMs with same MAC adresses....

DNS fuckery since someone thought that reboot would flush the cache of an DNS server.... Nope most DNS servers have persistent caches. You'll have to flush manually.

And let's not forget the joy of the 12 plus pages of when and where to move VMs, harddrives and VLAN configuration.

Oh migrations are such a festival of joy.

Finally done with that shit -.-

So we have an API that my team is supposed send messages to in a fire and forget kind of style.

We are dependent on it. If it fails there is some annoying manual labor involved to clean that mess up. (If it even can be cleaned up, as sometimes it is also time-sensitive.)

Yet once in a while, that endpoint just crashes by letting the request vanish. No response, no error, nothing, it is just gone.

Digging through the log files of that API nothing pops up. Yet then I realize the size of the log files. About ~30GB on good old plain text log files.

It turns out that that API has taken the LOG EVERYTHING approach so much too heart that it logs to the point of its own death.

Is circular logging such a bleeding edge technology? It's not like there are external solutions for it like loggly or kibana. But oh, one might have to pay for them. Just dump it to the disk :/

This is again a combination of developers thinking "I don't need to care about space! It's cheap!" and managers thinking "100 GB should be enough for that server cluster. Let's restrict its HDD to 100GB, save some money!"

And then, here I stand trying to keep my sanity :/

So I was wondering, what is you guys' 'goal' in the long run? What do you really want to be able to do or what do you really want to achieve in 5 to 10 years? Are you working primarily for an income or are there more prominent reasons for doing what you do?

I'm in my last years of university and have a job at a software development company, not because I need the money but because of all the amazing technologies I come in touch with and the amazing things I learn, mostly about servers and devops tools.

I currently have 2 goals in mind: Creating an .io game together with a frontend developer and setting up a kubernetes cluster and using it. I personally consider kubernetes an end-game thing when it comes to running apps and am experimenting with it on three servers that I got from school.

So I was wondering, what ahout you? 😀

This begs for a rant... [too bad I can't post actual screenshots :/ ]

Me: He k8s team! We're having trouble with our k8s cluster. After scaling up and running h/c and Sanity tests environment was confirmed as Healthy and Stable. But once we'd started our load tests k8s cluster went out for a walk: most of the replicas got stoped and restarted and I cannot find in events' log WHY that happened. Could you please have a look?

k8s team [india]: Hello, thank you for reaching out to k8s support. We will check and let you know.

Me: Oh, you're welcome! I'll be just sitting here quietly and eagerly waiting for your reply. TIA! :slightly_smiling_face:

<5 minutes later>

k8s team India: Hi. Could you give me a list of replicas that were failing?

Me: I gave you a Grafana link with a timeframe filter. Look there -- almost all apps show instability at k8s layer. For instance APP_1 and APP_2 were OK. But APP_3, APP_4 and APP_5 were crashing all over the place

k8s team India: ok I will check.

<My shift has ended. k8s team works in different timezone. I've opened up Slack this morning>

k8s team India: HI. APP_1 and APP_2 are fine. I don't even see any errors from logs, no restarts. All response codes are 200.

Me: 🤦‍♂️ .... Man, isn't that what I've said? ... 🤦‍♂️

Was forced to do some work on Windows this week (CAD tools that runs only on Windows). I spent a few days just setting up the tools. There were quite a few things I realized I forgot about Windows (as compared to Linux).

1) Installation times are down right horrific. What exactly are the installer doing for 10 minutes?
2) .NET is a cluster fuck. Not even Microsofts repair tool can fix it, but rather just hangs. I ended up using another tool to nuke it and reinstall.
3) Windows binary installs are insanely huge, thus, takes forever to download.
4) The registry is a pointless database that must have been written in hell with the single intent of destroying users will to live. The sole existence of the registry is another proof that completely incompetent engineers designed Windows.
5) Rebooting is the only way to solve many problems. This is another sure sign of a fundamentally fucked up OS design.
6) What the heck is wrong with the GUIs designers? The control panel must be the worst design ever. There are so many levels to get to a particular setting I'm getting dizzy. Nothing gets better by the illogical organisation.
7) Windows networking. A perversion of the tcp/ip stack that makes it virtually impossible to understand a damn thing about the current network configuration. There are at least 3 different places that effects the settings.
8) Windows command prompt. Why did they even bother to leave it in? The interpreter is as intelligent as retarded donut. You can't do anything with it, except typing "exit" and Google for another solution.
8) Updates. Why does it takes hundreds of updates per month to keep that thing safe?
9) Despite all updates that is flying out of Redmond like confetti, it is still necessary to install antivirus to keep the damn thing safe. That cost extra money, and further cost you by degrading performance of your hardware.
10) Window performance. Software runs like it was swimming in molasses. The final stab in the back on your hardware investment, and pretty much sends performance on your hardware back a few hundred bucks more.
11) Closed source is evil. If something crash consistently, you might find a forum that address the issues you have. Otherwise you're out of luck. On the other hand, it might be for the better. I imagine reading the code for Windows can lead to severe depression.

I'm lucky to be a Linux dev, and should probably not complain too much... But really, Windows, go get yourself hit by a truck and die. I won't miss you.

I had spent the last year working on a online store power by woocommerce with over 100k products from various suppliers. This online store utilized a custom API that would take the various formats that suppliers offer their inventory in and made them consistent. Now everything was going swimmingly initially, but then I began adding more and more products using a plug-in called WP all import. I reached around 100k products and the site would take up to an entire minute to load sometimes timing out. I got desperate so I installed several caching plugins, but to no avail this did not help me. The site was originally only supposed to take three to four months but ended up taking an entire year. Then, just yesterday I found out what went wrong and why this woocommerce website with all of these optimizations was still taking anywhere from 60 to 90 seconds to load, or just timing out entirely. I had initially thought that I needed a beefier server so I moved it to a high CPU digitalocean VM. While this did help a little bit, the site was still very slow and now I had very high CPU usage RAM usage and high disk IO. I was seriously stumped the Apache process was using a high amount of CPU and IO along with MYSQL as well. It wasn't until I started digging deeper into the database that I actually found out what the issue was. As I was loading the site I would run 'show process list' in the SQL terminal, I began to notice a very significant load time for one of the tables, so I went to go and check it out. What I did was I ran a select all query on that particular table just to see how full it was and SQL returned a error saying that I had exceeded the maximum packet size. So I was like okay what the fuck...
So I exited my SQL and re-entered it this time with a higher packet size. I ran a query that would count how many rows were in this particular table and the number came out to being in the millions. I was surprised, and what's worse is that this table belong to a plugin that I had attempted to use early in the development process to cache the site. The plugin was deactivated but apparently it had left PHP files within the wp content directory outside of the actual plugin directory, so it's still executing scripts even though the plugin itself was disabled. Basically every time I would change anything on the site, it would recache the whole thing, and it didn't delete any old records. So 100k+ products caching on saves with no garbage collection... You do the math, it's gonna be a heavy ass database. Not only that but it was serialized data, so when it did pull this metric shit ton of spaghetti from the database, PHP then had to deserialize it. Hence the high ass CPU load. I had caching enabled on the MySQL end of things so that ate the ram. I was really desperate to get this thing running.
Honest to God the main reason why this website took so long was because the load times made it miserable to work on. I just thought that the hardware that I had the site on was inadequate. I had initially started the development on a small Linux VM which apparently wasn't enough, which is why I moved it to digitalocean which also seemed to not be enough, so from there I moved to a dedicated server which still didn't seem to be enough. I was probably a few more 60-second wait times or timeouts from recommending a server cluster to my client who I know would not be willing to purchase it. The client who I promised this site to have completed in 3 months and has waited a year. Seriously, I would tell people the struggles that I would go through with this particular site and they would just tell me to just drop the site; just take the money, just take the loss. I refused to, this was really the only thing that was kicking my ass. I present myself as this high-and-mighty developer like I'm just really good at what I do but then I have this WordPress site that's just beating the shit out of me for a year. It was a very big learning experience and it was also very humbling as well, it made me realize that I really don't know as much as I think I might. It was evidence that there is still so much more to learn out there, I did learn a lot from that experience especially about optimizing websites the different types of methods to do that particular lonely on the server side and I'll be able to utilize this knowledge in the future.
I guess the moral of the story is, never really give up. Ultimately things might get so bad that you're running on hopes and dreams. Those experiences are generally the most humbling. Now I can finally present the site that I am basically a year late on to the client who will be so happy that I did not give up on the project entirely. I'll have experienced this feeling of pure euphoria, and help the small business significantly grow their revenue. Helping others is very fulfilling for me, even at my own expense.

Anyways, gonna stop ranting. Running out of characters. If you're still here... Ty for reading :')

I do like Windows, it is a quite good OS nowadays, but for FUCK SAKE, what does it take to fix that CLUSTER FUCK that you call search? You don't have enough people MS or what? Just show me the BLOODY ITEMS that actually contain the words that I typed in!!! While you are at it WOULD YOU MIND LEAVING ME THE FUCK ALONE WITH THE FUCKING WEB RESULTS???

5 of us working for a larger team were tasked with doing some R&D, we blew everyone away and were given funding to start a new team and hire people to make the project come to life.

One of the high level sales / product managers we were reporting to, secretly had another team work on a similar idea because he needed it quicker (i.e. no time for research, just build it).

After forming new team, we were asked to work on his project instead because it was further along. 4 months later, big knob comes to a meeting and basically says "You know what, this doesn't look like we have enough features, we need more, but I don't know what".

Project blew up 2 months later, head of the unit kicked up a shit storm saying how badly everything was planned and canned everything. Now one of our clients is building nearly the same thing we were originally working on, the team no longer exists and i'm back on the R&D team.

Don't get me wrong, I LOVE the R&D team, actually didn't want to leave in the first place but was told I had to. But the sheer anger and frustration to see that walking cluster fuck strutting around like his shit doesn't stink, derailing entire teams, meanwhile we can't hire new staff due to lack of funding.

Heres an idea, fire the fucktards bleeding us dry ... then we'll have lots of funding.

Things that make me feel like a child:

* Getting access to company computational cluster.

Want to make someone's life a misery? Here's how.

Don't base your tech stack on any prior knowledge or what's relevant to the problem.

Instead design it around all the latest trends and badges you want to put on your resume because they're frequent key words on job postings.

Once your data goes in, you'll never get it out again. At best you'll be teased with little crumbs of data but never the whole.

I know, here's a genius idea, instead of putting data into a normal data base then using a cache, lets put it all into the cache and by the way it's a volatile cache.

Here's an idea. For something as simple as a single log lets make it use a queue that goes into a queue that goes into another queue that goes into another queue all of which are black boxes. No rhyme of reason, queues are all the rage.

Have you tried: Lets use a new fangled tangle, trust me it's safe, INSERT BIG NAME HERE uses it.

Finally it all gets flushed down into this subterranean cunt of a sewerage system and good luck getting it all out again. It's like hell except it's all shitty instead of all fiery.

All I want is to export one table, a simple log table with a few GB to CSV or heck whatever generic format it supports, that's it.

So I run the export table to file command and off it goes only less than a minute later for timeout commands to start piling up until it aborts. WTF. So then I set the most obvious timeout setting in the client, no change, then another timeout setting on the client, no change, then i try to put it in the client configuration file, no change, then I set the timeout on the export query, no change, then finally I bump the timeouts in the server config, no change, then I find someone has downloaded it from both tucows and apt, but they're using the tucows version so its real config is in /dev/database.xml (don't even ask). I increase that from seconds to a minute, it's still timing out after a minute.

In the end I have to make my own and this involves working out how to parse non-standard binary formatted data structures. It's the umpteenth time I have had to do this.

These aren't some no name solutions and it really terrifies me. All this is doing is taking some access logs, store them in one place then index by timestamp. These things are all meant to be blazing fast but grep is often faster. How the hell is such a trivial thing turned into a series of one nightmare after another? Things that should take a few minutes take days of screwing around. I don't have access logs any more because I can't access them anymore.

The terror of this isn't that it's so awful, it's that all the little kiddies doing all this jazz for the first time and using all these shit wipe buzzword driven approaches have no fucking clue it's not meant to be this difficult. I'm replacing entire tens of thousands to million line enterprise systems with a few hundred lines of code that's faster, more reliable and better in virtually every measurable way time and time again.

This is constant. It's not one offender, it's not one project, it's not one company, it's not one developer, it's the industry standard. It's all over open source software and all over dev shops. Everything is exponentially becoming more bloated and difficult than it needs to be. I'm seeing people pull up a hundred cloud instances for things that'll be happy at home with a few minutes to a week's optimisation efforts. Queries that are N*N and only take a few minutes to turn to LOG(N) but instead people renting out a fucking off huge ass SQL cluster instead that not only costs gobs of money but takes a ton of time maintaining and configuring which isn't going to be done right either.

I think most people are bullshitting when they say they have impostor syndrome but when the trend in technology is to make every fucking little trivial thing a thousand times more complex than it has to be I can see how they'd feel that way. There's so bloody much you need to do that you don't need to do these days that you either can't get anything done right or the smallest thing takes an age.

I have no idea why some people put up with some of these appliances. If you bought a dish washer that made washing dishes even harder than it was before you'd return it to the store.

Every time I see the terms enterprise, fast, big data, scalable, cloud or anything of the like I bang my head on the table. One of these days I'm going to lose my fucking tits.

Building my own router was a great idea. It solved almost all of my problems.

Almost.

Just recently have I started to build a GL CI pipeline for my project. >100 jobs for each commit - quite a bundle. Naturally, I have used up all my free runners' time after a few commits, so I had to build myself a runner. "My old i7 should do well" - I thought to myself and deployed the GL runner on my local k8s cluster.

And my router is my k8s master.

And this is the ping to my router (via wifi) every time after I push to GL :)

DAMN IT!

P.S. at least I have Noctua all over that PC - I can't hear a sound out of it while all the CPUs are at 100%

This just happened....

Tester: My cluster is not working properly!!!
Me: What's wrong?
Tester: I don't know. I've checked all the logs available on the entire cluster. All i know is that node 1 and 7 is broken.
*ssh into the cluster*
node1
# less /var/log/<affected application log>.log
*no errors here everything is working properly*
node7
# less /var/log/<affected application log>.log
*goes down to the bottom and scrolls up a few lines*
<insert massive error here>

Checked all the logs eh?

So, it's 22:40 here and I'm sat on a bench staring out at a pond because my stress and anxiety is at an all time high after a couple of weeks of hellish arguments with work and my personal life so as were all developers here to some degree let me convey my fucking thoughts here.
If you care more about maintaining your fucking superiority complex over writing good clean efficient code then get the fuck out of the industry.
I don't give two fucks whether you use Linux or Windows. I couldn't give two fucks about whether you use sublime, Emacs or VIM. I couldn't give two fucks about the framework you spend more time defending than coding in, because absolutely none of it matters if you code like a retard on bath salts you pretentious cunts.
Stop feeding you fucking ego. Absolute cluster fuck of an industry.

Hey guys,

@devDNS will most likely be down for the next three hours, as I am installing a new node in my raspberry pi cluster, and need to do some networking work.

I will post an update when the bot is operational again.

Holy fcuk! Can anyone here help me understand how this domain is possible?

WARNING: obviously its a spam site. Take necessary security precautions if you are going to visit.

the following domain opens a cluster fuck domain name! >> secret.ɢoogle.com

That ɢ is not what it looks like. How is such domains possible to exist? Even more surprising, how is this sub domain -ception possible?

I'm a python fanboy, not gonna lie.
I love everything about it. It's clean syntax, ready to use out of the box-ness, convenient built-in functions.

The one thing I hate is the official documentation. It's ugly, hard to navigate and a cluster fuck.

But it has proper information, so it's fine I guess. tsch

Creating a cluster with shared storage in Proxmox

Once you've learned how to configure a single Proxmox host and Linux and Windows guests, the next step is to expand...

Want to continue reading this article? Register here with your corporate email address. Because your private email address isn't good enough, we need your corporate one.

No TechTarget, how about you go fuck yourself? As if anyone is going to register just to read one fucking article on your goddamn shitty site. Fucking wanketeering dickheads.

>Client complains about a 30 minute downtime around midnight
>Client also pays only for a single VM on a HV that they don't even own themselves

>Replies with an offer of how to make the setup more resilient, going from 1 VM to 2 LBs/FE loadbalanced through BGP, and distributing traffic through HaProxy onto 2 BE machines that in turn talk to a Postgres Cluster with RepMgr for dynamic failover.

>No reply so far

Hmmm :^)

Me: The IP Address on your public mail server cluster has been blacklisted.

Supporter: What is the IP? You can get it from whatismyip.com

Me: *has left the chat*

@dfox Would you make it possible to cluster push notifications that occur on the same post? When I receive a lot of notifs at once it clogs up my notification center.

Additionally would it be possible so that I can completely turn off seeing certain types of notifs even in the app not just for push notifications (from what I gather that is the only thing I have control over)

2 weeks and i leave my job.

Pro: no more dealing-with-math-msc-students

Con: i want to keep that juicy cluster to run my own calculations ;_;

I kept piling on hard drives at home and you know... One day I'll setup some redundancy.
Then a 5 months old full 4TB drive gave up and I lost that data.
After that I've upgraded to having a ceph cluster storing everything.
At the beginning of this year one hard drive in the cluster gave up. I didn't notice until I wondered why available storage was low. Cluster had already rebalance itself and were running flawless.

Yesterday I shutdown a database cluster after 2 years of running it, what a ride it was... Felt a little bad when I removed the servers...learned a lot ...

My boss did not care about making things secure in our early development stage, even though I told him several times.
After 1 day our elastic search cluster was filled with random crappy data.

Fix: Apply security schemes provided by AWS

- Launch the new version of the system I have been refactoring for 2 years and counting, then ceremoniously burn (literally) the legacy code as well as the cluster fuck of hardware it runs on.

- Decrease my stress + bus factor by bringing another up to speed on my code & the new version (his cluster fuck now).

- Pay attention to & take better care of health, my wrists in patricular.

- Find a mentor and mentor someone else.

- Get out of crisis management mode and find the time to write tuts, experiment and live a little.

- Find & join a local dev meetup, maybe make a local dev friend.

- Book leave and actually take it, preferabbly without having to take my laptop to the beach - actually, preferabbly at least have the choice to take a offline vacation.

- Sort through the drives containing ALL the code I have ever written, migrate the usefull interesting bits to Github.

Phew, that bit of self reflection was intense! I'm adding a cron to my server to sms & email me this rant in a year to remind me what hope looks like.

update : love my new job... but, holly crap the sharepoint environment is a cluster fuck... guess who has to fix it... -!-

Monday is the deadline for a big project in my master, I worked together with 11 people for one whole year. Our repository is on a server from university and we use the uni cluster to evaluate our results.
And today the uni spontaneous turns of its electricyty for the weekend to test some fallback systems...

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

Today marks the day that i finally get to do stuff on a production server.
Its just installing the elasticsearch cluster. But i still feel honored by the trust im given even tho im still an apprentice.

I could kill someone. My boss occupied the whole cluster for 24h yesterday, so that I had to wait until today to see that I had a small bug in my code. I wasted a whole day waiting around for something I could have fixed in 5 minutes yesterday if I only had 1 free node on the cluster 😠

Worst of all, if anybody else had occupied the whole cluster for so long without asking, he would have sent an angry e-mail to the whole institute 😠

Debugging distributed systems in a nutshell.

Here is another Tester story for you!

Tester: We have a problem with the application and it says that we cant process the queue to store the objects in the database.
Me: Can we reproduce it easily?
Tester: Yes..
Me: Ok, I'll add some debug for you so that you can see the error and the worker status more easily.
Tester: Great!

<10 minutes later>
After adding the debug output and installing it on his test cluster this happens...
Me: The debug version is installed.
Tester: Ok, great!
Me: How fast does it take to reproduce?
Tester: Well... It happened this morning at 9
Me: Ok..? But are you able to reproduce the issue at all?
Tester: No... i don't think so...

Good thing the maintenance is at night here, so I won't miss out on scrolling rants :)
Don't do the GitLab and do changes on the wrong db cluster :P

There is this dude called Richard Eng which is sort of famous for 2 things:
First: he is known as *the* Smalltall evangelist of mothern times. And he constantly writes about it. Which is fine since he tries to attract new users to this beautiful and simple little language.

Second: his constant bashing of other technologies, mainly Javascript stating that it is the most harmful tech known to man.

The thing is, saying "use this because that is shit" is never going to convince a community, specifically one as potent as that of the JS community. And to make it worse...the dude links his reasoning about bad languages to articles he wrote. As in "this is shit, look at my completely biased article regarding why its shit"

Once he is confronted about it he links back to his own writings. Much like christian fanatics do
"good is real because it says so in the bible"
"but how can you trust that resource?"
"Because the bible is the word of God"
"and how do you know?"
"Because it is in the bible"

Circular arguments like that cannot be taken seriously. And what this guy does for the Smalltalk community hurts more than it helps really.

Claims like those are all around us. If we were to believe or consider them depending on who said what then we would never have the amazing cluster of tech choices that we have.

Take c++. It is absolutely powerful and gives you the ability to do pretty much anything. If we were to take Linus Torvalds's word about it being shit and only having subpar development we would miss on absolutely powerful tools.

The same came to me from Evee, writer of "PHP a fractal of bad design" or the "Node.js is cancer" article.

You are never going to please anyone with anything. I go by live and let live, and whilst I don't like some technologies I certainly don't look down on those that do.

Four and half months,
Hundreds of hundreds PRs and one additional product cluster
By 6 Engineers..
To 500+ micro services
Which has no timezone or currency context,
Created by 250+ engineers,

To launch in a new country...

It didn't make me happy.

But the feedback from customers and drivers is priceless and #heartwhelming

It seems that every Pi day, I end up with a new Raspberry Pi that I got sometime before it. First, a 1, then a 2+3, now I have a Zero as well. Next year, I'll have a cluster of 3s. No reason, just want to mess with it.

Hey guys i wanna build a Pi-like Cluster to try out kunernetes locally and I would like some parts advice for the boards, casing, cooling, and the rack itself.

so far I have a Cisco E1200 Router I would use for the cluster for SSHing and connecting to

The Matrix. Then use the brains of the people plugged in like a computer cluster for large calculations.

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

I'm very short tempered at the moment.

A lot like Dr Cox in Scrubs.

And really ... You mother fucking stupid idiotic developers with your tendency to discuss absolutely everything just to not have to work for a dozen more minutes...

But ok. Let's discuss.

But even that seems to be absolutely impossible for you little shitheads.

Instead of discussing solutions, nooooooooo....

We're grown up developers so we discuss how the baddy manager hurt our lil feelings by saying that we're morons for wasting all the fucking time without coming up with a solution.

Now my lil cry babies, once the baddy manager got your pacifiers so at least once in an hour my migraine finally calms down for not hearing your bitching pathetic lil whiny noises...

Face it. Over the years you collected a huge ton of mother fucking tech debt because no one of you actually took a bit of time to use that empty space in your head to think at least a mu further than the dumb jira task you were given.

And yes. That ends badly.

And yes. As it is now in a state of cluster fuck, guess what. You have to work. You get money for it, remember?

And yes. if you would stop moping and bitching and crying and being a pathetic lil piece of shit, you'd realize we could come up with solutions very fast.

But nooo... Let's talk about our feelings.

And how we are over worked.

And how nothing works.

Cause yes. That will be the hail mary that saves us all.

Let me give u a hint: it's a mother fucking waste of time bitches.

I think it's time I put a pacifier not only in your mouth, but arse too. Maybe it helps overcoming the anal and oral phase of childhood so we can at least have something close to adult talk.

*breathes in*

Gooozfraba.

What if people, life, humanity, the universe is just a cluster of CPUs running a giant Recurrent Neural Network algorithm? 🤔

-Sun and food == power source
-People == semiconductors
-Earth/a Galaxy == a single CPU
-Universe == a local grouping of nearby nodes, so far the ones we've discovered are dead or not what same data transport protocol/port as us
-Universal Expansion == the search algorithm
-Blackholes: sector failures
-Big Bang == God turns on his PC, starts the program
-Big Crunch == rm -rf

Currently learning Kubernetes Cluster. Wow so much to learn.

Much easier to setup any app for a Cluster. Only building the Image takes time 🙄

Containers are the future ✌🏻

Just finished dumping all ethereum tranasctions into one big 30 GB csv.

Only thing left is to configure Apache Spark cluster.

At night is when my creativity starts flowing like a motherfucker. The moment when all the tasks are done for the day and you can start working on your own projects and just lay back and smoke (or drink, whichever you prefer) and zone the fuck out with some good music. Oh and if I've gotten a good work out in that day, then there's no stopping me.

I had no plans to even create an admin panel for my own small project but last night I made one just to make it look professional. After that, I got an idea for a separate project which I started working on. I usually write my ideas down so that I don't get into a complete project cluster fuck with 50 half completed projects, but sometimes you get that golden one that you have to start (currently those are the only two unfinished projects I got).

What the fuck this fucking shit of an app supposedly went through five rounds of QA and I broke it in 5 minutes? Holy fucking shit How this is the main point of your job. You shitty shits take forever getting this PR to me and now it’s buggier than a haunted house on Halloween?

After this is after wanting to cuss Apple for making me update the os then then xcode then iTunes just to recognize the goddamn development phone. It’s an app built off Cordova it works for like 14 versions of iOS just run the bugger you over priced aluminum chassis dildo riding fruit Fuckers.

Now back to the goddamn cluster fuck I just got delivered. What the goddamn fuck. This level of bullshittery I have not seen before. And apparently Cordova is only partly de-sandboxed? I don’t know for sure because I don’t have time to test it because I’m running the entire technological stack of this company on a junior dev salary!

When you tell me it goes though all this QA and you spend 2 weeks on just QA tickets for fucks sake the first operation I do, the most common fucking operation on the app, the shit we have been building around all this fucking time, should not fucking error out. For fucks sake at least try to get the main fucking thing working. I recognize you did a lot of work and implemented a number of features but what the fuck good are they if I can’t even run them for one fucking time.

Don't use your senior software engineer title or years of experience as reasons in a debate or argument about software

My manager was asking me what steps needs to be done to perform a disaster recovery for our cluster( on production). I will be honest here, I have not maintained this type of cluster(kubernetes) in production before. However, I have enough understand of the system to answer my boss question. I basically told him there are A, B, C you have to do.

My senior developer jumped in and said "No you should do A,C, B because C is more critical than B. " I then replied to him: "I understand your point, I notice that too, but .." Before I can even finish my sentence, this dude has already rolled his eyes and interrupted me very loudly: "Have you worked with these systems on production before? I did". The asshole knows I haven't maintained Kubernetes on production yet of course.

I got super pissed at him and pretty much shouted back to him and my manager: "Just because I haven't worked on this system on production yet, does not mean my argument is wrong" .

I then dragged my managers, that asshole, and other engineers in a room and settle this out. In the end, people agreed with my steps over that asshole senior engineer dude because I gave them rational reasons.

The conclusion is: Your senior title is given by the company, It doesn't mean anything to me. Also, it doesn't make you more right than another person just because he has a "lower" title than you.

Developing and deploying in Xcode is some Requiem for a Dream level bullshit.

I literally just de selected everything for managing automatic signing, and re-selected the EXACT SAME GODDAMN THING. And it worked. It’s literally some fucking shit you do when you are first learning how to code or learning a language and you keep flipping something but you don’t get exactly how it works.

But this is YOUR FUCKING FLAGSHIP development product. I shouldn’t have to check my goddamn inception totem to see if I’m dreaming or not because this kind of bullshit can’t be real life.

That being fucking said your bullshit forced shutdown also FUCKED MY ANDROID STUDIO INSTALLATION AND FUCKED MY $PATH. Thanks. Now NOTHING WORKS. Fuck you Apple. Between slowing my phone and the cluster or problems your shit is causing that are just random as hell and are plenty common because thank god people smarter than me have fixed them in SO by now, I am SO READY TO LEAVE THE APPLE ECO SYSTEM. If I didn’t have to use one of the boxes to push iPhone app updates I doubt I would touch one again.

Apple stuff looks good but at this point that’s about it.

So today I got fucked by myself. And it hurts.

In my Company, we started thinking how we could organize some kind of think tank for the next 5/ years goals.

As CTO I got designated to be in these workshops... And stupid me, I started to organize the "workshops". Using " agile" and "facilitator" artifacts (serious games, ideation sessions, open space sessions, etc).

Today we finished the roadmap and proto processes to make bottom to top inclusive process, and co-create with everyone what should be the Company roadmap.

We went to dinner. And I was happily talking with my smartphone when they decided that I should preside the committee in charge of the roadmap creation!!!!!

Useless mother fuckers!!!!! Go kill yourselves!!!! I have some interesting stuff to do!!!

Why didn't I just shut it!!! Now I'll have to make sure the old cluster fuck part of the Company do real shit!!! They are fucking useless and don't give a shit about the future!!!!

FUCK THEM!!!!!
FUCK ME!!!!!!

Today I learned that some external devs one of our projects is working with have DB tables where they store references to specific dates, and not only that, but every minute of those dates, and the day of the week, and what season its in. Im not joking.
Hmm should I use the local datetime libs or should I go through a firewall, load balancer and DB cluster just to find out what day it is?

Here it goes,

So there I was a Linux enthusiast stuck in a windows job for about 3 years. I would spend my weekends doing Linux related tasks for my personal amusement, while I spent my week doing windows maintenance and development (partially) professionally.
It was about 2014 I started building an openstack cluster at home and i was so stoked! I searched for openstack summits or meetups and for my surprise there was an openstack meetup in my town. Holly 🐄 I said.
The date of the event came and I left work earlier to attend the meetup.
There , I had a talk with the meetup organizer/speaker and he told he was interested in what I was doing and that they were going to open a job in the next months.

A few months later still at my boring job I got an email from him for an interview.
Everything went just about right...and here I am a Linux systems engineer doing everything I love for a living...

Avoid ACPICA if at all possible. It's one garbage tier cluster fuck of bad design, horrible documentation and downright misleading and wrong code

It's meant to consist of an ASL compiler, disassembler, debugger, dumper, various user space utitilies and a kernel resident OSPM implementation *if* you can figure out what belongs to what. Even just compiling this pile of trash is a mystery in itself. Think you need the source files in source/common? EEEEH, wrong. Well, at least partially since most of them seem to be for the user space stuff..? Other ones *are* needed on the other hand. At least the disassembler and/or debugger and/or dumper components seem to reference them. Not that I could figure out how to compile those anyways. The real path to your goal seems to be to ignore a seemingly arbitrary subset of source and header files until your linker stops complaining

There's also a bunch of configuration defines, some of which *you* define, some defined *for* you, based on again others. Of course most of them do stupid shit. Enabling the debugger automatically enables debug logging. Enabling the disassembler force enables debug allocation tracking... What?

The code itself isn't of much help either. Looking in "os_specific/service_layers" you find what looks to be reference implementations of acpica functions in certain os' like windows and unix. Of course I had a look because AcpiOsReadMemory is supposed to read physical memory and I don't know how I would even implement that. But hey, osunixxf.c (xf for interface... of course) should tell me. I'll let you see for yourself in the attached image. Apparently it does fuck all and just returns AE_OK. No error, no logging, no nothing. Just ok. As you can imagine, AcpiOsWriteMemory doesn't do much more either.
...okay so maybe physical memory accesses aren't actually used and these functions are some sort of relic from past times? Nope! They are absolutely necessary for doing low level device interaction. WTF. So finally I went to the linux source and checked how *they* implemented them, and just as I thought, these functions are anything but no-ops...
...So for what fucking reason do these stupid interface implementations even exist but to purposefully mislead you?? They aren't used for fucking anything! As far as I know Windows doesn't even *use* ACPICA and Linux have their own fork with working implementations... They just sit there, just to tell you how to NOT do it

So that's some of my thoughts about ACPICA. Note that I haven't even used it as a library yet, I just got it to compile and link and it already fucked with me this much.
There's also so much more I didn't mention like that you *have* to modify the acpica source in order to get your own platform header working (else #error) eventhough the docs explicitely instruct you not too but you get the point

Don't use ACPICA if you don't have to. Save your sanity for something that's worth it

So for a while I have wanted to build a raspberry pi cluster. In the spirit of shia labeouf I got started last saturday.

I had two pies lying around so I figured I'd run some experiments before I invested in a lot of hardware. After about a day I had turned the two pies into a shared cluster when disaster struck....

I had completely ignored the fact that you cannot run 32 or 64bit software on an arm processor (I know... I'm a java developer). So when I booted my service and the load balancer, I found that nothing worked. So pretty bumbed out, I quit the project.

Later that day I found a crazy guy who had bought a batch of 400 small form factor PSUs (300W) and internally I laughed at him a little. I mean, who's gonna sell 300W irregular power supplies. Then, just as I was about to go to bed I found this guy, he was selling from a batch of CPU-onboard motherboard for 10 bucks each and everything clicked!

I did some quick calculations and decided I could probably gather enough cash to get: 10 motherboards, 10 2GB ram dimms, 10 Sata disks and 14 PSU (in case some fail) and some misc hardware for networking and such.

So... Long story short, I am going to build a cluster computer, the first version is going to have 10 nodes and I am waiting for delivery right now!

1. Exporting fat jar
2. Transferring to cluster using WinSCP
3. Running it in the cluster.
4. Find a small bug.
5. Repeat

I HATE IT

Stupid pipeline bullshit.

Yeah i get it, it speeds up development/deployment time, but debugging this shit with secret variables/generated config and only viewable inside kubernetes after everything has been entered into the helm charts through Key Vaults in the pipeline just to see the docker image fail with "no such file found" or similar errors...

This means, a new commit, a new commit message, waiting for the docker build and push to finish, waiting for the release pipeline to trigger, a new helm chart release, waiting for kubernetes deployment and taking a look at the logs...

And another error which shouldn't happen.

Docker, fixes "it runs on my machine"
Kubernetes, fixes "it runs on my docker image"
Helm, fixes "it runs in my kubernetes cluster"

Why is this stuff always so unnecessarily hard to debug?!

I sure hope the devs appreciate my struggle with this... well guess what, they won't.

Anyways, weekend is near and my last day in this company is only four months away.

Was wondering if there are any open source or free for non-commercial (would like to tinker with it / learn about it as my weekend adventure first) use projects (or approaches how to solve this) that I might not have heard of, related to what I'm about to describe....

Usually, when you got 'n' ammount of machines (be it VM's, baremetal or just a bunch of Raspberry Pis) the usual approach is to either:
1)Run them as independent machines, each serving it's own purpose;
2)Run them as independent machines but "tied together" with software configured in a cluster and featuring some sort of replication;
3)Pool them together into a "swarm"/resource pool and provision VM's across them, with resources spread across the machines;

But I here's where I started wondering: What if I had 3 identical desktop PCs with relatively decent computing power and resources allocated to each... stay with me here - I can already feel some of you writing this off completely or already preparing to answer how impossible this is... what if I was ready to sacrifice portion of the combined I/O or computing power so I could set these 3 computers up in some form of "grid computing" (2-3 machines working as one)? Are there any solutions out there that could take care of the related problems (I/O error/fault handling, fragmentation, CPU governing, timing, synchronization.... all the stuff that sane and normal people don't have to worry about because it's already been taken care of on the lowest levels of programming/computing)?

I'm also thinking about the average consumer grade x86/x86_64 and ARM architecture here, not some exclusive spacegrade architectures normally used in supercomputers.

I'm currently between jobs and have a few rants about my previous job (naturally). In retrospect, it's somewhat therapeutic to range about the sheer brainfuckery that has taken place. Enjoy!

First, let me set the scene: legacy B2B web app made with LEMP stack and sencha ext.js 3 + 4 (don't ask) and a lot of madness. Let's call that app "Alpha".

Alpha is a self made CMS build for typical ERP stuff. Yes, a self made CMS: entities are containers, containers have types and fields and values. Like so many legacy PHP apps, it does not have a dedicated FE: the HTML is rendered on the server and then spewed out to the browser.

Easy right? Coding like it's 1999! But there was a twist: Because everything is basically a container, the HTML-templates are saved in the DB. Along with the nessary JS and the CSS. And the translation variables. Why? Because fuck you! That's why. Who needs a git history anyways.

For some reason, Alpha was kinda slow.

There was also an editor, that allowed you to modify templates (web, mail, pdf) on the fly in prod. Because templates contain repeating data (header/footer), one template could contain additional templates. Much confusion. You could change templates via migration (slow, boring) or just ctrl-c/ctrl-v that sucker (fast, much excitement).

Did I mention Alpha was slow?

On with the rant: e-mails! How do they work? Noone knows. How to send mails asynchronous in PHP? Witchcraft is the only possible answer to that riddle. Here is your enterprise™ solution:
1. create mail
2. insert mail into DB
3. WAIT UP TO 59 SECONDS FOR A FUCKING CRON TO SEND MAIL

Why? "Because that way, we can resend mails in case the network is down :)"

Same procedure for the SOAP-API (db-queue + cron). You read that right: all requests to various other systems are processed once a minute.

Alpha slow.

Alpha was only one of several systems. Imagine a bunch of monolithic php apps, interconnected via SOAP, REST and GraphQL like a godamn intergalactic orgy. Image having to debug that cluster fuck.

Let's say there is a bad request. These things happen. No biggie. Remember the db-queue? Let's try to send the bad request a second time! And a third time! Still no luck? How odd. Let's create a specific file in a specific directory: a LOCK-file. Now, "the db-queue is on hold and no request gets processed :)"

Golly gee thanks Alpha.

Anyhow, did you know that MySQL has a join limit of 61 tables?

You know what I had to deal with
A bunch of these shit
try{
//Shitty cluster fuck excuse for java
//code
}catch(Exception e){

}

Got handed a CentOS 7 cluster, previous admin made kernel command line changes in grub.cfg instead of default/grub.

Ah, thank you.

I really want to build a raspberry pi cluster but I don't know what to do with it...
Has anyone of you ever built a raspi cluster?
What could I do with it?

I wanted to install this new software in our cluster but it didn't have Cmake on it, so I scp Cmake source files and then I started looking for a CMakeLists.txt file for compile it when I suddenly realized what I wanted to accomplish in the first place 😅

Day 1:
Optimizing huge problems for the company. Get mail. *sigh*; Why is your script using up half the CPU on our thin clients? *place in complaints folder and go on*

Day 2:
Boss asks about it during scrum meeting. *Oh shits*
Need a cluster. Been asking for it for months...

Day 3:
Start runs on all thin clients. *Thou shall feel my wrath*
Complaints folder floods.

Day 4:
Expect rage from boss.
"IT seems to have found a cluster for you at last."

Finally!

I'm quite pleased to show off the initial rendering of my CaaB (Cluster as a Box)!

This is a project that I've put not enough time into, and them finally I decided to get a move on with it. The estimated cost of the whole device/s ranges somewhere around $600. It's gonna be fun!

P.S. Suggestions welcome, particularly if it's about a substitute for the ODroid HC1.

Completely fucked up replication of MySQL servers.

Remote: 2 different database Servers
--> made sense.

Except the misconfiguration. Or better: No configuration at all.

So how to solve the massiv delays and make everything even more crazy?

2 remote servers - 2 readonly slaves for reading data remote (master - slave)

2 local (internal) servers.

Remote - Local Master Master.

Unfucking this cluster fuck was a real nightmare.

It had to be done at night, cause everything needed to be ripped apart.

And the servers were the backend of a warehouse with supply chain and multiple selling channels (Amazon, eBay etcetera).

So. It had to run the next day at 05.00 clock so the incoming orders could be packaged / prepared for shipping.

That was fun. Not.

And the clusterfuck died spectaculously on my first work day - the old DBA was gone (fired....)

:)

Set-up a 5 node EC2 cluster in AWS; Install my dependencies on all; Add private keys between all for handshake; Submit my Hadoop job for processing; AWS closes my instance within 10 minutes of starting a job that took me almost half an hour to set-up because the master node's spot costs have reached more than 15 USD :(

One of my colleagues who thinks he knows all about "big data": "I try to put everything in hadoop, that is my philosophy".

We don't even have a hadoop cluster.

Setting up a single node Hadoop cluster. Then installing intellij idea, to find that it doesn't detect any installed jdk. Then uninstalled all jdks, and then reinstalled one, then Hadoop won't work. Now everything seems fine.

There is a drawing competition for my currently most played game.
I'm on vacation and the deadline is when I get back. So what did I do?

I made an inspiration cluster with the character and drawings having this arrogant face (laptop, gimp).
I sketched on my phone and put it on my laptop (the sketch). I have no desk here so I'm drawing on my bed. I have no drawing pad so I'm drawing with my mouse. Then I draw it in gimp with colors and everything (the stroke in another program on my gfs laptop), put each layer in inkscape to svg-ify them and to hq-render them back in Gimp. Corrected a few things in Gimp. Added more detail, effects (glow, gradient instead of flat color ...).

~6 hours over two days. That was fun. And fucking unprofessional.

Debugging your application on a mesosphere cluster for the first time.

## Learning k8s

Okay, that's kind of obvious, I just have no idea why I didn't think of it..

I've made a cluster out of a rpi, a i7 PC and a dell xps lappy. Lappy is a master and the other two are worker nodes.

I've noticed that the rpi tends to hardly ever run any of my pods. It's only got 3 of them assigned and neither of them work. They all say: "Back-off restarting failed container" as a sole message in pod's description and the log only says 'standard_init_linux.go:211: exec user process caused "exec format error"' - also the only entry.

Tried running the same image locally on the XPS, via docker run -- works flawlessly (apart from being detached from the cluster of other instances).

Tried to redeploy k8s.yaml -- still raspberry keeps failing.

wtf...

And then it came to me. Wait.. You idiot.. Now ssh to that rpi and run that container manually. Et voila! "docker: no matching manifest for linux/arm/v7 in the manifest list entries."

IDK whether it's lack of sleep or what, but I have missed the obvious -- while docker IS cross-platform, it's not a VM and it does not change the instructions' set supported by the node's cpu. Effectively meaning that the dockerized app is not guaranteed to work on any platform there is!

Shit. I'll have to assemble my own image I guess. It sucks, since I'll have to use CentOS, which is oh-so-heavy compared to Alpine :( Since one of the dependencies does not run well there..

Shit.

Learning k8s is sometimes so frustrating :)

A few weeks back we ported our PHP Rest API into a couple of Go micro-services.

Incredibly _satisfying_ job.

Requests went from 20+ seconds to ~100-300ms.

There was still one bottle-neck, though, because we had to use most of the old cluster-fork of a database (because no way I'll be able to fix all that in a week).

And ooh, next we're thinking of switching to gRPC. Man, we have the best jobs.

Clustering high dimensional data - SOS!

I have to cluster documents. I computed the tf*idf, but I have ~45K docs with 28K words.
I did minibatch kmeans which works alright, but everything else runs forever. I need to compare 2 clustering algorithm, so I need at least another one that works. In this life time.
Suggestions?

L1 support requested to terminate an EC2 instance on which one of our apps seemed to be misbehaving. The node was terminated after few min.

L1 later realized that that instance didn't belong to that app, but instead it was one of the RabbitMQ nodes.

Then, after some panicking we remembered that HA was enabled, so nothing should've been lost.

Later, we realized that the recent RMQ upgrade necessitated a new cluster on which HA was NOT enabled!

Got my networking for cluster emulation using virtualbox right after 6 days.

Thanks @sbiewald

Just got handed a dozen servers. Documentation shows a (Linux) database cluster is using ldap authentication. I try logging in with my creds. No joy. I look up the root password and log in.

Not only is it not configured to use ldap, it's also not clustered.

I need more coffee.

I am really having a hard time keeping an "open" mindset with my team...

So when I email you, the junior at your first job and an almost 6 months work experience, with a set of instructions, I do not want inline comments and feedback to the tune of "Yes, I agree. I think we should go ahead" or debates on why, in your opinion.. xyz, because you tested it yourself..

WHERE THE FUCK DID YOU SETUP AND TEST BILLIONS OF ROWS OF DATA ON YOUR MANY MANY MULTI BOX CLUSTER? You live in a data center? Oh, and say "Serverless" one more fucking time....

And so begins the painful process of performance managing you out the door. (You cant fire anybody in South Africa for being a prick. Or useless.)

I am sure there once was a time where you could beat guys like this to within an inch of his life with his own keyboard. If it wasn't wireless I could have strangled him with the cord. Ah, I miss model M's....

I am so sick of this place. Production deployment shouldn't be such a massive cluster fuck of different departments and sometimes redundant configurations. Knowing who to talk to about a problem shouldn't change daily, and it shouldn't be hard to figure out.

My jvm doesn't appear to be running in production. I hunt down the fucker who is supposed to be handling this problem today...but he's out of office and no one knows who his stand in is...GODDAMNIT!

Why do some people feel the need to prove their stupidity and utter lack of skill in the face of the world?!?!
Yesterday I learned that a sister company is hiring an intern civil engineer to code some application plugins connected to our IS ?!?!? How the fuck do you think he can only understand what the fuck we do?

To put it in context, I'm kind of the CDO of a French medium group (a little cluster of companies), as the group is in the construction industry I'm the CTO for all Computer things. Inside the group, I'm the CTO of the digital factory. So the group IS is a microservice decentralized API REST-based architecture.

Next Monday we'll have a meeting, so I can explain to them why it's a FUCKING STUPID IDEA!!!! The only good thing is that any application programming done outside of the Digital Factory will be handled as an External Company Application, so it's not my problem to secure it, debug it, or simply make it work. And they already know that I'll enforce this ruling!!!

But WHY the fuck do they still think any mother fucker can professionally program!!!!!! Every time I have to deal with them It's horrendous!!!! I had to prove them why using a not encrypted external drive for a high security mission It's stupid!!!, and why having the same password for every account is FUCKING STUPID!!!

The most ridiculous part is they have a guy who really believe he has some IT skills!! Saying things like "SVN" it's a today tool (WTF), firewall are useless, etc....

WHY!!!! WHY!!!!

i found playing with arduino is never-ending fun. and then i tried Raspi and feeling the same. now i buy a bunch of raspi and trying to cluster them up, just for another random project.

yeah this is clearly nonsense. this is myself when my boredom gets high enough. No, vacation is not helping, trust me i've been camping in a jungle for 5 days and the next week i go to the beach. this is bad.

making things worse, My way of life is like any ordinary people (or maybe a developers) but turned upside down for example I program, learning for fun, and going vacation is just another medical need that i need to follow.

I am an university student, studying IT. I've been programming since 6, and geeking anything around technology and astronomy till now. this is my third semester and i don't get challanged enough for this kind of curricullum. i need some kind of project that even i don't think about any number of money anymore, but just enough to challange me up. just need to get rid my boredom.

sorry for ranting from a to z, this is just buzzing around my mind.

code reviewed a "senior" developers code (guy has been at the companyny since noah and his clay computer tablets), he replied with my comments against it stating he has more experience than i have and that i shouldnt do anything except just sign off the stuff to production, stuff goes to production and breaks a massive financial cluster fuck of a mess, who gets the blame for releasing the code to production ? Junior me !!! gotta love it when management needs scapegoats of FIFO people.

Posted previously about our codebase being a monolithic, poorly-written, pain-to-maintain gigantic cluster-fuck. And the efforts made to rewrite it.
Well, we made huge success the previous year in this regard. I rewrote the entire API while my other team mates worked on two different UI apps one of which is now in production and the other soon to be released in alpha.
Processes have being put in place for our team and are being improved.
We still have some technical debts though.
Dev goal for 2020,
- Pay most of the technical debt.
- Dive deeper into Flutter and finish the app I wanted.
- Play with ML, AI and Game dev.

Give me a second to get my todo list.

- get a office/server room built
- setup a home kubernetes cluster
- create an open-source ActivityPub whatsapp clone
- unify existing ActivityPub implementations where an account on one can be used on others
- finish dockerssh
- create an irc bridge for signal messenger.
- find a way to fully provision linux workstations fully unattended

The most difficult part about learning/working with new technology is the lack of online support and a really small community! So every time you're stuck with an error you got to open each and every configuration file to see which little value was throwing that page long error!!!!

The same happened with me while working with the new RedHat CEPH Storage while I was configuring my nodes using ceph-ansible. A new error pops up and it was like reaching milestones when I found the error halting up the execution of the playbook!

kubectl scale tears --replicas=infinity

I just deployed my first AWS Kubernetes cluster. Fuck yeah!

Finally set up my 5-node Raspberry Pi Kubernetes cluster.

Showing off this PC cluster rendering these gigantic 3D frames, a customer shakes his head: "Why is this so slow?" I asked why he thought it was slow and he said, "I have an old Windows laptop but if I double click on an image on the desktop, it opens immediately!"

Let's see what's on the menu today:

* Web Application Catastrophe Special *

Includes, but not limited to:
- Orphaned server processes in the configuration management cluster
- Microservice back-end architecture with no API documentation
- Poorly implemented cache microservice with no documentation
- Stale data causing everything to be shown as down in production, despite everything running fine

Cost: 1 developer's sanity

AMQ cluster is misbehaving.. Master is constantly dying, slave is starting under root rather than app account, connection to AMQ DB takes 20-40 seconds (while queries execute in <0.1sec and TCP probe takes <0.01sec to succeed), monitoring is down,...

I mean it's gotta be the virus - what else could it be.

If you would have a cluster made of 10 RPi, what would you do? I'm looking for fun project to make and I'm willing to learn new things.

Screw DataStage
Screw Saleforce
Screw the $25k/week Vendor who finishes work maybe 60% complete, if I’m being generous.
Screw the business rushing to us to fix their shadow IT cluster.

Where can i get good second hand servers, am planning to homelab cluster. Will appreciate any pointers ...

"shiny shiny" time... Lets standup a kubernetes cluster and then find a problem to solve! #BuzzWordDrivenDev

In three days my team closes the book on this monolithic cluster of a project that I thought would never end! I feel like Christmas is coming. Hopefully the PM doesn’t let the business throw us into another scope-less pit like this again! 😩

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

Parsing logs to create conditional insert statements cause expert morons fucked up production database cluster.

Database is partially corrupted and cannot be used and they don’t know how to fix it.

Just succesfully setup database cluster , now I need balancer!

## Learning k8s

Sooo yeah, 2 days have been wasted only because I did not reset my cluster correctly the first time. Prolly some iptables rules were left that prevented me from using DNS. Nothing worked...

2 fucking days..

2 FUCKING DAYS!!! F!!!

My final year taking a B.Sc. I'm writing up my Distributed Systems project, the day before handing it in. It's on top of Transis, and source code is "stored" in RCS (yes, I'm that old). The project is a reliable system administration tool, that performs the same action across a cluster with guaranteed semantics.
I'm very proud of the semantics, but cannot figure out why the subdirectory installation stuff works almost but not quite. Here's my sequence of actions:
1. Install across all machines.
2. Manually see it's broken.
3. "rm -rf *".
4. Repeat.

What in to discover is that the subdirectory installation always finishes off in a current directory 1 level higher than where it started. Oh, and the entire cluster sees my NFS home directory. Oh, and I'm running each cluster member in a deep subdirectory of my dev directory. Oh, and my RCS files live in a subdirectory of my dev directory.

All of a sudden, my 5 concurrent "rm -rf *"s were printing weird error messages about ENOENT and not being able to find some inodes. In a belated flash of brilliance, I figure out all the above, and also that I've just deleted my dev directory. 5 times, concurrently. And the RCS files.

That was the day a kindly sysadmin taught me than NetApps have these .snapshot directories.

Having a Node.js app ready for production is so satisfying with PM2.

## Learning k8s

Interesting. So sometimes k8s network goes down. Apparently it's a pitfall that has been logged with vendor but not yet fixed. If on either of the nodes networking service is restarted (i.e. you connect to VPN, plug in an USB wifi dongle, etc..) -- you will lose the flannel.1 interface. As a result you will NOT be able to use kube-dns (because it's unreachable) not will you access ClusterIPs on other nodes. Deleting flannel and allowing it to restart on control place brings it back to operational.

And yet another note.. If you're making a k8s cluster at home and you are planning to control it via your lappy -- DO NOT set up control plane on your lappy :) If you are away from home you'll have a hard time connecting back to your cluster.
A raspberry pi ir perfectly enough for a control place. And when you are away with your lappy, ssh'ing home and setting up a few iptables DNATs will do the trick

netikras@netikras-xps:~/skriptai/bin$ cat fw_kubeadm
#!/bin/bash

FW_LOCAL_IP=127.0.0.15
FW_PORT=6443
FW_PORT_INTERMED=16443
MASTER_IP=192.168.1.15
MASTER_USER=pi

FW_RULE="OUTPUT -d ${MASTER_IP} -p tcp -j DNAT --to-destination ${FW_LOCAL_IP}"

sudo iptables -t nat -A ${FW_RULE}

ssh home -p 4522 -l netikras -tt \
-L ${FW_LOCAL_IP}:${FW_PORT}:${FW_LOCAL_IP}:${FW_PORT_INTERMED} \
ssh ${MASTER_IP} -l ${MASTER_USER} -tt \
-L ${FW_LOCAL_IP}:${FW_PORT_INTERMED}:${FW_LOCAL_IP}:${FW_PORT} \
/bin/bash
# 'echo "Tunnel is open. Disconnect from this SSH session to close the tunnel and remove NAT rules" ; bash'

sudo iptables -t nat -D ${FW_RULE}

And ofc copy control plane's ~/.kube to your lappy :)

This is probably the worst place to start my Rant saga but this is recent (this is one of the last few episodes of a 3 series cluster fuck of a job so you're missing out on all the straws that go into breaking the camels back and making him unaccommodating)

TL;DR I do good work, management dont like me and go out their way to try and fuck up my days

So, lets start, I'm a contractor, got funeral Tuesday, book leave, book WFH for day after.

I leave in 3 weeks, woman who is the CIO's right hand bitch takes me into a room the next day or so in the morning to discuss my WFH day. Leave on tuesday is cool but this WFH day...there's only so long until I'm gone so they want me to stay in for more face-to-face time blah blah blah (considering this woman isn't even part of the project I'm working on anymore because she decided to deflect it onto a underqualified junior with no PM experience)

So I sit there, thinking of all the blood and sweat that I have shed, the mountains I've moved just to be told to move the mountain somewhere else and whether coming in would kill me (in other words im fucking burnt out!!! I have built their GDPR database and app backend single-handedly with no requirements, project managers who can't plan and being chastised for asking for documentation/plan/anything written down and having the CIO who is also the fucking DPO ignore any emails/slack I send him relating to the project and having to keep up with a team of devs....).

So because there was a momentary silence, she decided to fill the gap

"Oh, you've done some good work so far and I wouldn't want you to ruin it all in these last 3 weeks. So just come in on the Wednesday so that we can have you here."

Hmm....yeah...i didn't notice what she had ACTUALLY said there, still thinking about can i be fucked? So she decides to add

"...there's only 3 weeks left, wouldn't want you to burn any bridges. Remember, we still have to give you a reference"

....Okay....shots fired. So i respond

"You saying, if I take a WFH day, you'll give me a bad reference?"

"Noooo no no no, not saying that, just that you've done good work and we wouldn't want you to ruin it"

"With one wfh day?"

"We just want you to come in because the developers might be coming here that week"

"Oh... I hear that...what day?"

"I dunno, it's not been booked yet"

".............................I'll think about it"

"There's nothing to consider"

*Start leaving room* "I'll think about it...."

So cool, obviously, had a think, decide to shoot over an email (or more accurately, a collection of bullets). Which basically said, in devRant translation, "Fuck y'all, I'm WFH on that day, I wish a motherfucker would fuck up my reference, we can go that way if you want it. *snaps fingers* I. WISH. YOU. WOULD! "

Woman says "I wasn't threatening you, was just saying...dont ruin your last 3 weeks, wouldn't want you to burn any bridges and that we still have to give you a reference"

What kind of Godfather comment is that?

Come in today, the CIO, who is a prick who don't like me for whatever reason, sends me long email trying to disrespect me and in the midst says "I’m sorry that you have chosen to react like this, I’m sure that [my bitch] was conveying a position that your last three weeks of contract are crucial for a smooth handover. I have made the decision to not require you to work from home on Wednesday. I understand you are on leave on Tuesday and therefore this is now extended to include Wednesday. I look forward to seeing you back in the office on Thursday. I hope this will make the situation better for all parties."

.................................thought you lot needed me in the office to ensure a smooth handover................logic..........people.............where the fuck do you get yours from!?!?!?!? All this just so they can say "We made the decision at the end :cool:"

I hope my boss learned his lesson: dd if=/dev/zero of=[hdd storing DB about VM cluster]
- is a very very bad idea...

Dear AWS, your Elasticsearch service is a bogus pile of shit-engorged horse fly larvae. Not only do you give no useful visibility into what's happening with the cluster (making diagnosis a sadistic guessing game), you lock down the fucking settings API, making it impossible to debug!! But your excellent support is on it! I wonder if I'll hear back from them this week with another inane suggestion like "increase the node count". Meanwhile the rest of my system is limping along, sometimes getting data where it's supposed to be while I keep fake-smiling and reassuring management and customers that "I'm working on it". If you're going to offer a service either make sure it works or get the fuck out of my way. I'll be moving my cluster back to EC2 and you can go do a back flip off a skyscraper. I need a drink

Things I say to my clients when I know that a reboot is required to fix their issue but I don't have enough evidence to prove it to them :

"... On any computing platform, we noted that the only solution to infinite loops (and similar behaviors) under cooperative preemption is to reboot the machine. While you may scoff at this hack, researchers have shown that reboot (or in general, starting over some piece of software) can be a hugely useful tool in building robust systems.

Specifically, reboot is useful because it moves software back to a known and likely more tested state. Reboots also reclaim stale or leaked resources (e.g., memory) which may otherwise be hard to handle. Finally, reboots are easy to automate. For all of these reasons, it is not uncommon in large-scale cluster Internet services for system management software to periodically reboot sets of machines in order to reset them and thus obtain the advantages listed above.
Thus, when you indeed perform a reboot, you are not just enacting some ugly hack. Rather, you are using a time-tested approach to improving the behavior of a computer system."
😎

My works funny man.

Over 12 years we've built out a crazy amount of services utilizing several languages, paradigms and deployment methods.

Apparently we use to have a massive dev team or something but as of now its like barely 9 (that might include QA so like 6).

We have a deployment repo that manages the entanglement of everything on a kubernetes cluster. There you will be forced to work with python, bash scripts, ruby, some lua, and node.js. And of course alot of yaml files.

Move along to one of our many extensions or servers and you will find yourself needing to know angular 1.x, angular 9.x (right big jump), angular 7.x, Php (no frameworks), node.js (express spaghetti code, likely written when it first got hot), ruby on rails, meteor.js.

Sense theres very few of us and we're over customer engineering (fixing bugs primarily to prevent churn), that means Im crossing each of these domains and needing to understand each enough to understand a mess of code that on average 9+ developers have had their hands on.

Im exhausted. Even coming from an agency environment this is still brutal.

It seems he is trying to replace HTML with svg and canvas when basic html and css can be used to make beautiful dashboards.

Like i anticipated it is gonna be a ux cluster fuck, where most of the website is an ugly bootstrap contraception and some parts gonna look like futuristic.

Is it normal for "enterprise" software to have 14+ pages of known issues in the release notes, including issue descriptions that use phrases like "may lead to data corruption" and "may cause the cluster to crash"??

Monitoring tools madness: quest foglight.

So, setting a blackout for an FMS "HA cluster" (which does not work due to a bug infested custom jboss implementation) can bring the servers down... And no way to bring them back up.

This brilliant piece of enterprise APM software costs 600.000€ for a 5year license.

I,ve added more drama (logs, threaddumps, support bundles and screenshots) to the support portal...

45 cases now in total, oldest case still open date 2017...

Fuck you quest software

From HTML CSS javascript to some voodoo WPF xaml and c#, mvvm and databinding dependency inject cluster bomb to my brain god Microsoft just put a tutorial out like a normal group of people. where you do examples of projects. I'm 2 weeks in and this still makes no sense

launched major version today which has been in development nearly 2 years. daily bugfix builds for the next month or so until it's stable as we slowly rollout one cluster at a time. going to be my first time on production bugfix duty. man it feels cool to be a full time software engineer.

Make your code available for your team members, please.

So we're working on this robotics project using ROS, a framework that enables multiple nodes in a network exchange their functionality among each other through tcp connections. Each node can be implemented and executed on your own machine, and tested with dummy inputs, but in collaboration they make a robot do fancy stuff.

The knowledgebase needs data from the image processing unit, providing this data to others with semantic context to high level planning, which uses this semantic data for decision making and calling the robot manipulation node with meaningful input, to navigate the robot's components in the environment. We use a dedicated machine, which pulls the corresponding repositories and is always kept configured correctly, to run each node, such that everybody has access to each other's work when needed.

So far so good. We tried to convince the manipulation guy (let's call him John) to run his code on our central machine, not a week, but since the first day, 5 months ago. Our cluster classification has been unavailable for 2 months, but my collegue fixed that. We still can't run the whole project without John's computer. If his machine blows up we're fucked.

Each milestone feels like a big-bang-test, fixing issues in interfaces last-minute. We see the whole demo just moments before our supervisors arrive at the door.

I just hope he doesn't get hit by a truck.

*The one where he breaks ssh*

TL;DR: Minikube's dick is too big, and my ass wasnt ready.

So there was a time about 2 weeks ago where i wanted to try and set up a minikube cluster using SOP, and that actually went okay, aside from having to move over to a completely different server after discovering that my processor doesn't support virtualization.

So i set it up on my other server, and everything immediately starts going to shit; i can no longer run commands without processor latency. Also top shows 200% CPU usage. Maybe i should stop... NAHHH... so i continue on, and the biggest fuck up was starting up the nginx pods. I have 6 of them, and the moment i try and stand up my custom container which was the WHOLE POINT of this whole exercise, i lose ssh access and cant get back in. I go over to the server and kill the minikube and virtualbox processes, and everything's back to normal.

Four nested tmux sessions:
My mac: ctrl+L
Remote: ctrl+G
Cluster: crtl+B
Server: ctrl+T

LGBT

Spend all day debugging simple post request. Like really what is going on. Super simple. Eyes start to bleed. Check spelling on everything. Finally find out the access-control-origin isn't set right, other dev said it was whatever so glad I'm moving on. Nope. Same error running the app from Visual Studio. Check code again. Everything works in a browser. Windows, VS, or the emulator is blocking just POST requests. I can do get requests all day.

What hell. I'm so critical of my code I spend hours pouring over something I knew was right instead of looking for network errors. I just need to trust myself I guess.

Oh and Windows Cordova apps don't support ES6 lol.

I like the people I work with although they are very shit, I get paid a lot and I mostly enjoy the company but..

Our scrum implementation is incredibly fucked so much so that it is not even close to scrum but our scrum master doesn't know scrum and no one else cares so we do everything fucked.

Our prs are roughly 60 file hangers at a time, we only complete 50% of our work each sprint because the stories are so fucked up, we have no testers at all, team lead insists on creating sql table designs but doesn't understand normalisation so our tables often hold 3 or 4 sets of data types just jammed in.

Our software sits broken for months on end until someone notices (pre release), our architecture is garbage or practically non existent. Our front end apps that only I know the technology have approaches dictated by team lead that has no clue of the language or framework.

Our front end app is now about 50% tech debt because project management is so ineffectual and approaches are constantly changing. For instance we used to use view models for domain transfer objects... Now we use database entities, so there is no commonality between models but the system used to have shared features relying on that..sour roles and permissions are fucked since a role is a page regardless of the pages functionality so there is no ability to toggle features, but even though I know the design is fucked I still had to implement after hours of trying to convince team lead of it. Fast forward a few months and it's a huge cluster fuck to enforce.

We have no automated testing of any sort or manual testing in place.

I know of a few security vulnerabilities I can nuke our databases with but it got ignored.

Pr reviews are obviously a nightmare since they're so big.

I just tried to talk to scrum master again about story creation since any story involving front end ui as an aspect of it is crammed in under one pointed story as sub tasks, essentially throwing away any ability to calculate velocity. Been here a year now and the scrum master doesn't know what I mean by velocity... Her entire job is scrum master.

So anyway I am thinking about leaving because I like being a developer and it is slowly making me give up on doing things to a high standard and I have no chance of improving things, but at the same time the pay is great and I like the people.

That feeling when the cluster your ovh vps runs on goes down the exact moment your application is supposed to go live, and everyone is blaming you :|

Fix desynchronized Redis cluster

I'm not sure whats saddest that'll likely be doing gdpr for another two months. Or that the pm thinks we'll be ready by the deadline.

A very long rant.. but I'm looking to share some experiences, maybe a different perspective.. huge changes at the company.

So my company is starting our microservices journey (we have a 359 retail websites at this moment)

First question was: What to build first?

The first thing we had to do was to decide what we wanted to build as our first microservice. We went looking for a microservice that can be used read only, consumers could easily implement without overhauling production software and is isolated from other processes.

We’ve ended up with building a catalog service as our first microservice. That catalog service provides consumers of the microservice information of our catalog and its most essential information about items in the catalog.

By starting with building the catalog service the team could focus on building the microservice without any time pressure. The initial functionalities of the catalog service were being created to replace existing functionality which were working fine.

Because we choose such an isolated functionality we were able to introduce the new catalog service into production step by step. Instead of replacing the search functionality of the webshops using a big-bang approach, we choose A/B split testing to measure our changes and gradually increase the load of the microservice.

Next step: Choosing a datastore

The search engine that was in production when we started this project was making user of Solr. Due to the use of Lucene it was performing very well as a search engine, but from engineering perspective it lacked some functionalities. It came short if you wanted to run it in a cluster environment, configuring it was hard and not user friendly and last but not least, development of Solr seemed to be grinded to a halt.

Elasticsearch started entering the scene as a competitor for Solr and brought interesting features. Still using Lucene, which we were happy with, it was build with clustering in mind and being provided out of the box. Managing Elasticsearch was easy since there are REST APIs for configuration and as a fallback there are YAML configurations available.

We decided to use Elasticsearch since it provides us the strengths and capabilities of Lucene with the added joy of easy configuration, clustering and a lively community driving the project.

Even bigger challenge? Which programming language will we use

The team responsible for developing this first microservice consists out of a group web developers. So when looking for a programming language for the microservice, we went searching for a language close to their hearts and expertise. At that time a typical web developer at least had knowledge of PHP and Javascript.

What we’ve noticed during researching various languages is that almost all actions done by the catalog service will boil down to the following paradigm:

- Execute a HTTP call to fetch some JSON

- Transform JSON to a desired output

- Respond with the transformed JSON

Actions that easily can be done in a parallel and asynchronous manner and mainly consists out of transforming JSON from the source to a desired output. The programming language used for the catalog service should hold strong qualifications for those kind of actions.

Another thing to notice is that some functionalities that will be built using the catalog service will result into a high level of concurrent requests. For example the type-ahead functionality will trigger several requests to the catalog service per usage of a user.

To us, PHP and .NET at that time weren’t sufficient enough to us for building the catalog service based on the requirements we’ve set. Eventually we’ve decided to use Node.js which is better suited for the things we are looking for as described earlier. Node.js provides a non-blocking I/O model and being event driven helps us developing a high performance microservice.

The leap to start programming Node.js is relatively small since it basically is Javascript. A language that is familiar for the developers around that time. While Node.js is displaying some new concepts it is relatively easy for a developer to start using it.

The beauty of microservices and the isolation it provides, is that you can choose the best tool for that particular microservice. Not all microservices will be developed using Node.js and Elasticsearch. All kinds of combinations might arise and this is what makes the microservices architecture so flexible.

Even when Node.js or Elasticsearch turns out to be a bad choice for the catalog service it is relatively easy to switch that choice for magic ‘X’ or component ‘Z’. By focussing on creating a solid API the components that are driving that API don’t matter that much. It should do what you ask of it and when it is lacking you just replace it.

Many more headaches to come later this year ;)