I hired a woman for senior quality assurance two weeks ago. Impressive resume, great interview, but I was met with some pseudo-sexist puzzled looks in the dev team.

Meeting today. Boss: "Why is the database cluster not working properly?"

Team devs: "We've tried diagnosing the problem, but we can't really find it. It keeps being under high load."

New QA: "It might have something to do with the way you developers write queries".

She pulls up a bunch of code examples with dozens of joins and orderings on unindexed columns, explains that you shouldn't call queries from within looping constructs, that it's smart to limit the data with constraints and aggregations, hints at where to actually place indexes, how not to drag the whole DB to the frontend and process it in VueJS, etc...

New QA: "I've already put the tasks for refactoring the queries in Asana"

I'm grinning, because finally... finally I'm not alone in my crusade anymore.

Boss: "Yeah but that's just that code quality nonsense Bittersweet always keeps nagging about. Why is the database not working? Can't we just add more thingies to the cluster? That would be easier than rewriting the code, right?"

Dev team: "Yes... yes. We could try a few more of these aws rds db.m4.10xlarge thingies. That will solve it."

QA looks pissed off, stands up: "No. These queries... they touch the database in so many places, and so violently, that it has to go to therapy. That's why it's down. It just can't take the abuse anymore. You could add more little brothers and sisters to the equation, but damn that would be cruel right? Not to mention that therapy isn't exactly cheap!"

Dev team looks annoyed at me. My boss looks even more annoyed at me. "You hired this one?"

I keep grinning, and I nod.

"I might have offered her a permanent contract"

Hey everyone!

devRant will be going down on Friday, July 7th around 10:30pm EDT so we can do some database maintenance and restructuring of our cluster. It hopefully won't be down for more than about 30 minutes or so, and during that time you should see our "down for maintenance" message.

If you usually use devRant while you're on the toilet (we know many do!), we apologize and suggest you try to schedule around this!

Please let me know if you have any questions and apologies for the inconvenience.

rant? rant!

I work for a company that develops a variety of software solutions for companies of varying sizes. The company has three people in charge, and small teams that each worked on a certain project. 9 months ago I joined the company as a junior developer, and coincidentally, we also started working on our biggest project so far - an online platform for buying groceries from a variety of vendors/merchants and having them be delivered to your doorstep on the same day (hadn't been done to this scale in Estonia yet). One of the people from management joined the team working on that. The company that ordered this is coincidentally being run by one of the richest men in Estonia. The platform included both the actual website for customers to use, a logistics system for routing between the merchants, the warehouse, and the customers, as well as a bunch of mobile apps for the couriers, warehouse personnel, etc. It was built on Node.js with Hapi (for the backend stuff), Angular 2 (for all the UIs, including the apps which are run through a WebView wrapper), and PostgreSQL (for the database). The deadline for the MVP we (read: the management) gave them, but we finished it in about 7 months in a team of five.

The hours were insane, from 10 AM to 10 PM if lucky. When we weren't lucky (which was half of the time, if not more), we had to work until anywhere from 12 PM to 3 AM, sometimes even the whole night. The weekends weren't any better, for the majority of the time we had to put in even more extra hours on the weekends. Luckily, we were paid extra for them, but the salary was no way near fair (the majority of the team earned about 1000€/mo after taxes in a country where junior developers usually earn 1500€/month). Also because of the short deadline given to us, we skipped all the important parts like writing tests, doing CI, code reviews, feature branching/PR's, etc. I tried pushing the team and the management to at least write tests and make feature branches/PRs, but the management always told me that there wasn't enough time to coordinate and work on all that, that we'll do that after launching the MVP, etc. We basically just wrote features, tested them by hand, and pushed into the "test" branch which would later get tested and merged into master.

During development, one of the other juniors managed to write the worst kind of Angular code you could imagine - enormous amounts of duplication, no reusable components (every view contained the everything used in the view, so popups and other parts that should logically be reusable were in every view separately), fuck - even the HTML was broken (the most memorable for me were the "table > tr > div > td" ones, but that's barely scratching the surface). He left a few months into the project, and we had to build upon his shit, ever so slightly trying to fix the shit he produced. This could have definitely been avoided if we did code reviews.

A month after launching the MVP for internal testing, the guy working on the logistics system had burned out and left the company (he's earning more than twice the salary he got here, happy for him, he is a great coder and an even better team player). This could have been avoided if this project had been planned better, but I can't really blame them, since it was the first project they had at this scale (even though they had given longer deadlines for projects way smaller than this).

After we finished and launched the MVP, the second guy from management joined, because he saw we needed extra help. Again I tried to push us into investing the time to write tests for the system (because at this point we had created an unstable cluster fuck of a codebase), but again to no avail. The same "no time, just test it manually for now, we'll do that later when we have time" bullshit from management.

Now, a few weeks ago, the third guy from management joined. He saw what a disaster our whole project was. Him joining was simply a blessing from the skies. He started off by writing migrations using sequelize. I talked to him about writing tests and everything, and he actually listened. He told me that I'm gonna be the one writing them, and also talked to the rest of management about it. I was overjoyed. I could actually hear the bitterness in the voices of the rest of management when they told me how to write the tests, what to test, etc. But I didn't give a flying rat's ass, I was hapi.

I was told to start off by writing a smoke test for the whole client flow using Puppeteer. I got even happier, since I was finally able to again learn new things (this stopped at about 4 or 5 months into the project).

I'm using jest as the framework and started writing the tests in TypeScript. Later I found a library called jest-extended, but it didn't have type defs, so I decided to write them and, for the first time in my life, contribute to the open source community.

Finish my Raspberry Pi Zero Cluster... complete with kubernetes, etc.

... I still don't know what I'm going to run on this. suggestions in the comments welcome.

Never have I been so furious whilst at work as yesterday, I am still super pissed about going back today but knowing it's only for another few weeks makes it baerable.

I have been the lead developer on a project for the last 3~ months and our CTO is the product owner. So every now and then he decides to just work on a feature he is interested in- fair enough I guess. But everything I have to go and clean up his horrendous code. Everything he writes is an absolute joke, it's like he is constantly in Hackathon mode "let's just copy and paste some code here, hardcoded shit there and forgot about separation of code- it all goes in 1 file".

So yesterday he added a application to the project and instead of reusing a shared data access layer he added an entirely new ORM, which is near identical to the existing ORM in use, for this one application.

Being anal about these things, the first thing I did was delete his shit and simply reference the shared library then refactor a little code to make it compatible.

WELL!! I certainly hit a nerve, he went crazy spamming messages on Slack demanding I revert as it broke ONE SINGLE QUERY that he hadn't checked in (he does 1 huge commit for 10 of everyone else's). I stuck to my principals and explained both ORM's are similar and that we only needed one, the second would cause a fragmented codebase for no benefit whatsoever.

The lead Dev was then forced to come and convince me to revert, again I refused and called out the shit quality of their code. The battle raged on via the public slack group and I could hear colleagues enjoying the heated debate, new users even started joining the group just to get in on mine and the cto's difference of opinion.

I even offered to fix his code for him if he were to commit it, obviously that was not taken well ;).

Once I finally got a luck at the cluster fuck of shit he had written it took me around 5 minutes to fix and I ever improved performance. Regardless he was having none of it. Still the demands to revert continued.

I left the office steaming after long discussions with the lead Dev caught in the middle.

Fortunately my day was salvages with a positive technical discussion that evening at a company with whome I had a job offer from.

I really hate burning bridges and have never left a company under bad terms but this dictator is making me look forward to breaking the news today I will be gone in 4 weeks.

!rant

So last weekend I started collecting hardware for a small scale cluster at home to test scalability of my software. Making some decent progress.

Tomorrow I will replace the switch and this weekend I will set up storage so I can start my first application

Data scientist: we need to whitelist a pod to connect to a database
Me: Whitelist? We don't use whitelists on private databases
DS: It's the new data warehouse database
Me: is it on <X> VPC?
DS: I'm not sure what that means but its ip is <real world ipv4>
Me: Are you hosting a publicly accessible database with all our end users information?!
DS: ...
Me: There goes our SOC2 audit controls...
DS: how long until you can white list it?
Me: I won't be whitelisting it. You need to put it on a private VPC and peer with the cluster, you'll have to rebuild all the Terraform and redeploy
DS: We didn't use Terraform because it takes too long, just white list the pods IP.
Me: No. I'm contacting the CISO and CTO...

So, some time ago, I was working for a complete puckered anus of a cosmetics company on their ecommerce product. Won't name names, but they're shitty and known for MLM. If you're clever, go you ;)

Anyways, over the course of years they brought in a competent firm to implement their service layer. I'd even worked with them in the past and it was designed to handle a frankly ridiculous-scale load. After they got the 1.0 released, the manager was replaced with some absolutely talentless, chauvinist cuntrag from a phone company that is well known for having 99% indian devs and not being able to heard now. He of course brought in his number two, worked on making life miserable and running everyone on the team off; inside of a year the entire team was ex-said-phone-company.

Watching the decay of this product was a sheer joy. They cratered the database numerous times during peak-load periods, caused $20M in redis-cluster cost overrun, ended up submitting hundreds of erroneous and duplicate orders, and mailed almost $40K worth of product to a random guy in outer mongolia who is , we can only hope, now enjoying his new life as an instagram influencer. They even terminally broke the automatic metadata, and hired THIRTY PEOPLE to sit there and do nothing but edit swagger. And it was still both wrong and unusable.

Over the course of two years, I ended up rewriting large portions of their infra surrounding the centralized service cancer to do things like, "implement security," as well as cut memory usage and runtimes down by quite literally 100x in the worst cases.

It was during this time I discovered a rather critical flaw. This is the story of what, how and how can you fucking even be that stupid. The issue relates to users and their reports and their ability to order.

I first found this issue looking at some erroneous data for a low value order and went, "There's no fucking way, they're fucking stupid, but this is borderline criminal." It was easy to miss, but someone in a top down reporting chain had submitted an order for someone else in a different org. Shouldn't be possible, but here was that order staring me in the face.

So I set to work seeing if we'd pwned ourselves as an org. I spend a few hours poring over logs from the log service and dynatrace trying to recreate what happened. I first tested to see if I could get a user, not something that was usually done because auth identity was pervasive. I discover the users are INCREMENTAL int values they used for ids in the database when requesting from the API, so naturally I have a full list of users and their title and relative position, as well as reports and descendants in about 10 minutes.

I try the happy path of setting values for random, known payment methods and org structures similar to the impossible order, and submitting as a normal user, no dice. Several more tries and I'm confident this isn't the vector.

Exhausting that option, I look at the protocol for a type of order in the system that allowed higher level people to impersonate people below them and use their own payment info for descendant report orders. I see that all of the data for this transaction is stored in a cookie. Few tests later, I discover the UI has no forgery checks, hashing, etc, and just fucking trusts whatever is present in that cookie.

An hour of tweaking later, I'm impersonating a director as a bottom rung employee. Score. So I fill a cart with a bunch of test items and proceed to checkout. There, in all its glory are the director's payment options. I select one and am presented with:

"please reenter card number to validate."

Bupkiss. Dead end.

OR SO YOU WOULD THINK.

One unimportant detail I noticed during my log investigations that the shit slinging GUI monkeys who butchered the system didn't was, on a failed attempt to submit payment in the DB, the logs were filled with messages like:

"Failed to submit order for [userid] with credit card id [id], number [FULL CREDIT CARD NUMBER]"

One submit click later and the user's credit card number drops into lnav like a gatcha prize. I dutifully rerun the checkout and got an email send notification in the logs for successful transfer to fulfillment. Order placed. Some continued experimentation later and the truth is evident:

With an authenticated user or any privilege, you could place any order, as anyone, using anyon's payment methods and have it sent anywhere.

So naturally, I pack the crucifixion-worthy body of evidence up and walk it into the IT director's office. I show him the defect, and he turns sheet fucking white. He knows there's no recovering from it, and there's no way his shitstick service team can handle fixing it. Somewhere in his tiny little grinchly manager's heart he knew they'd caused it, and he was to blame for being a shit captain to the SS Failboat. He replies quietly, "You will never speak of this to anyone, fix this discretely." Straight up hitler's bunker meme rage.

Firefox is running in our universe and are some bits of data

We are moving to kubernetes.
Nothing much has changed except we now get to say
it works on my cluster.
instead of
it work on my machine.

A Developer is desperate: his java application servers are unresponsive, thousand of dead zombie threads are sucking all cpus, memory is leaking everywhere, garbage collector has gone crazy, the cluster sessions are fucked....

The Developer goes to the closest bridge, ties a stone to his neck and gets ready to jump.

Suddenly a bearded old man with a fiery look runs toward him, yelling:
- stop stop!!!! Your application is not scaling and misconfigured, your servers are melting, cpu usage is not sustainable anymore, but don't despair

The Developer, puzzled, looks at him:
-I've never seen you...how do you know...
- Hey, man, I'm the Devil. I know everything. All your problems are solved. I'll give you magic functions. They are called Lambda.
You'll never have to worry about your servers, scalability, security, configuration and shit.

The Developer seems astonished but relieved:
- Ok, sounds great! let's try it - suddenly suspicion creeps in - hmmmm but you are the Devil....so...you want something back, don't you?
(the Devil nods lightly with a diabolic smile)
- ...and...you want my soul, I guess...
- your soul??? come on!!! - the Devil burst in a laugh - we are in 2019. I don't care about your soul. I want your ass.
- What!???!!!?
- yes, I want to fuck your ass

The Developer, evaluates quickly the situation.
Few moments of pain or slight discomfort (?) in exchange for magic lambda. It could be worth. He accepts.

After a while of rough anal fucking, the devil asks
- Hey, how old are you anyway?
- 45, why?
- Oh jeeez...45!!!??? and you still believe in the devil?

Reasons I hate the US

1. It's fucking 2 in the morning there and folks are checking their Slack when they wake up for pee breaks or done after their sex sessions.

Nearly 90% of the team is on and off checking Slack.

2. Culture fucks have only 10 holidays and hence they align rest of the world to their calendar and only give 10 holidays. India, Europe, and entire world can easily get 15 holidays per year outside their leave quota.

What a cluster fuck of a country it is.

a dude just DROPPED the whole Fu**ing mongodb cluster. like Right Now.

multiple databases, expanding multiple projects.

fortunately in dev. but dunno how much data is recoverable.

My weekend project. Anyone else built a Raspberry Pi 3 cluster (for Kubernetes)?

Some days I just want to shoot myself.

I get why... someone might do this, but sweet mother of god!

Rare footage of me waiting for my Kubernetes cluster to start.(2018BC,colorized)

CEO - So... We'll have a new side project, it's a small thing, I spoke with a guy, he needs just a small thing...
Me - Ok....... So, what do you need me for?
CEO - Not much, this guy started a project but wants your help on a small part, and you already did something similar for us, so it should be fast, just copy and paste and change a little bit. You probably know better than me.
Me - *Sigh* ... So `friend`, what do you need from me?
Friend - So, I made a crawler that is storing some information on a local file, I just need to add multiprocessing and multithreading, a producer-consumer system with a queue so I can automatically add new links every now and then, a failback system, so when a process doesn't finish, it should be re-queued and crawled later, store all the information on a database cluster (that is not set up), [......]
Me - And when is this supposed to be up and running?
Friend - Your CEO told me you could do this by the weekend. Can we finish this by Friday?
Me - *facepalm* FML

I deployed my first Kubernetes cluster today, just wanted to share a win! 💪

This how every cluster-fuck begins..

Inception.
Today I needed to check something in a remote server: this was the easiest way:
1: teamviewer to my home pc from university
2: started a vm on that machine with vpn connection to my work office
3: rdp to a windows server vm
4: ssh to a vm on our hosting cluster
5: from there, ssh to the server that I needed access to

I am running a small - but growing - ceph-cluster at work. Since it is fun and our storage demand is growing each day.

Today, it was time to bring another node online and add another 12TB to the cluster.
Installation of the OS went fine, network settings fine, drives looks fine.

Now, time to add it into the cluster.... BAM
Every Dell machine in the Cluster - Dead.
The two HP-machines is online and running. But the Dell-machines just died.

WAT!?

Yep. So the dev teams boss says it's fine to run a production environment on a single Windows instance with the db on that same instance, which they already totally lost once from a reboot after an auto update before I came along tasked with fixing the cluster fuck they created.

This from a man who somehow runs a dev team while using gmail via the web because he can't use an email client, uses email to track tasks but can't because they get lost amongst his 3000+ unread emails, has a screen dirtier than a hookers vag on half priced Tuesday, and got a new laptop but had to get his daughter to set it up and transfer his data because he couldn't.

But ok... you have a degree, You must know what you're doing.
It's ok though, I'll keep covering your incompetent ass while you keep raping the company because no one listens.
Peoples ignorance and arrogance astounds me.

Went in to the therapist today. Their impression is I'm likely okay. Wait until I show them the cluster fuck that is Java Enterprise. 🙃

Manager: "Can we get an accurate report on how many containers we have on the Kubernetes cluster?"

Me: "Well not really since Kubernetes is designed to be dynamic and agile with the number of resources and containers being created and deleted being subject to change at a moment's notice."

Manager: "I want numbers"

Me: "Okay well if we look at a simple moving average over time, we can see how the number of containers changes and then grab a rough answer from that"

Manager: "These numbers look a little round, are you sure these are exact?"

I'm going to throw myself into a pile of used heroin needles and hope i get stuck with whatever the hell this guy has to somehow be a manager while also being this retarded.

We are building this big-data engine for a client's product for which we were using a cluster on GCP and they were billed ~1100$ for the last month's usage.
The CTO - the CHIEFFUCKING TECHNOLOGY OFFICER told us to hook up 5-6 laptops in our server room and create our own cluster because they cannot afford so much bill.

"Let's create a docker-swarm cluster thingy for this application with horizontal scaling to learn docker and run this application better!"

This stuff is so overwhelming and I don't understand half of how to possibly set this up 😅

Got laid off last week with the rest of the dev team, except one full stack Laravel dev. Investors money drying up, and the clowns can't figure out how to sell what we have.

I was all of devops and cloud infra. Had a nice k8s cluster, all terraform and gitops. The only dev left is being asked to migrate all of it to Laravel forge. 7 ML microservices, monolith web app, hashicorp vault, perfect, mlflow, kubecost, rancher, some other random services.

The genius asked the dev to move everything to a single aws account and deploy publicly with Laravel forge... While adding more features. The VP of engineering just finished his 3 year plan for the 5 months of runway they have left.

I already have another job offer for 50k more a year. I'm out of here!

> Monitoring: Load Average of 57!! ALERT!!!!
> me: What? That's not possible?
> *Monitoring froze 14 hours ago*
> *sshs into server*
> *see attached image*

The issue was ~1200 df processes that were issued by our monitoring system and all of them didn't finish because the external cluster we mounted onto that server died a few minutes before that. Just re-mounting the cluster fixed it but still a funny sight!

Nice Windows Defender :) I should buy a cluster next time to use this antivirus :P

When your co-worker asks for a more memorable hostname for the next dev cluster...

So I'm sitting on the swings, minding my own business, seing how best I could destroy this cluster of servers, when suddenly I notice SOMEONE IS COMING FOR MY COFFEE

"hi neighbour! What you've got there"

IT is so violent and disturbing when it comes to technical terms in relation to clusters:

- master / slave
- master / master / witness (crime scene?)
- splitbrain (how does that look in reality?)
- stonith (shoot the other node in the head)
- deadtime (no heartbeat)
- slicing (wtf?)
- shard (do they kill with that?)
- active / passive (sex scene?)
- single point of failure (at least they know who is to blame)
- share nothing / shared all (dictators vs communism)

😯

Finally got ceph up and running, so far it does look really great!
At the moment we have 45 TB in the cluster, but we are able to expand it extremely easy.

Two servers with 256 GB RAM and 24 cores, three more with 32 GB and 12 cores.
Dedicated gigabit WAN connection.

The project is going forward strong, I'll let you know when it is time for beta testing.

Between containers, cluster managers and virtual machines we've lost track of where our code even is.

Personally the coolest was the program I built for my fathers use on his job.

It was my first to be used commercially in the real.

That was a very big thing, I was 17 at the time an used turbo pascal 5.5 and he used it to compute how well all machinery was doing, they rented out diggers and other construction equipment to construction sites and manually compute this with a calculator took up to three days. (This was 1987 so there was not very many ready made programs for business, you often had to build your own)

With this program he had it done in around 30 minutes.

The next best was recently when I got my raft distributed consensus cluster server working. Its a little bit like zookeeper.
Building that purely from the research paper was rewarding but a bit of a challenge.

My company is providing cloud infrastructure to our customers. For research purposes we are running a little openstack cloud in our laboratory datacenter were we can test stuff before implementing it in the productive environment.

Last week the manager asked me to shut down the cluster over night and only power on the servers when we need it. (about twice a week)

The reason: it produces too much heat.

My answer was: No.
First off thats not how cloud infrastructure works, and how about a proper climate control?

Sometimes i ask myself in which parallel universum our managers live 😑

OMFG! Who’s bright fucking horrible stupid ass idea was it to mix Ajax with php (php deciding the ajax paths) with random js outputting HTML inside random fucking static divs found no where near the logical route of content.

Trying to add a simple fucking status to a gigantic cluster fuck of a legacy project is just FUCK.

If I could I would burn this bitch to the ground and start again I would, But no, it’s needed.

Someone kill me before I break the shit out of this thing, I would take a wordpress project right now instead.

I once found a MongoDB cluster open to the internet with no authentication with nearly a terabyte of data that backed a CRM service whose customers included Microsoft and Adobe to name a few.

So you want full stack engineers to: design, do UX, create front end, build backend and deploy it in your mono repo stupid manual deployment "kubernetes cluster", add monitoring alerting manually, review others PR, QA our own apps and features, manually sync to Production, use VPN otherwise we cannot connect to anything, 2factor auth, do SRE, architecture diagrams, demo, run agile ceremonies, and learn a legacy coding language which was never mentioned in the job description. Did I miss anything?

I received a job offer as web-app developer and, in order to access the interview, I had to do an online QI test.

I login in the website and I see an ugly cluster of links, missing informations, errors at receiving the data I just inserted, a lot of bugs, broken links, images not availables, blank pages and so on.

I'm not sure I want to work there anymore

We have a 15-machine cluster that went down last night because one machine in the cluster went down. Apparently having a cluster for redundancy is just a nice idea and doesnt actually work in practice.

Also I shouldnt have to go to a vendor's forums to find out the bug that is causing my cluster to go down is fixed in a future version. It should be in the goddamn patch notes!!!

Should’ve posted this after it happened, but it requires a bit of background anyway.

There’s this guy that oversees our OpenStack environment. My team often make jokes and groan about him in private because he’s so overbearing. A few months back, he had to take us to our data center to show us our new racks, and he kept saying stupid stuff like “you break this and it costs me $30,000” as if he owns everything. He’s just... one of THOSE people. Always speaks in such a condescending way. We make jokes that he is our “best friend”.

Our company is shifting most of our products to the cloud in response to the coronavirus (trying to make it an opportunity for “innovation”). This has involved some structural and responsibility changes in our department, and long story short, I’m now heading the OpenStack environment alongside other projects.

This means going through grueling 1-on-1 meetings with our “best friend”. It’s not too bad, I can be pretty patient with people, so I didn’t mind too much at first. Then a few things happened.

1. He sent a shared folder that he owned containing info related to the environments. Several documents were outdated and incomplete, so I downloaded them, corrected them, and then uploaded the documents to my teams file share, as I was supposed to since we now own the projects.

2. Several files were missing, and when I asked about them, he said “Oh, did you refresh the browser?”. I told him no, that I downloaded them locally and republished them to my teams server, because he was supposed to hand everything off to us at once. He says “Well, silly, how are you going to get updates if you’re looking at them locally?” and kind of chuckles at me like I’m stupid.

3. He insists on training me how to remote into one of the servers to check on cluster space, which in itself is fine. I understand others wanting to make sure things will be done right by the people who come after them. But he tells me to download SuperPutty. I tell him, “oh no, that’s alright. I don’t need putty”. He says “oh cool, what tool do you use for ssh?”. I answer him “Just Git. If I want to I can use a CentOs bash terminal too, because we have WSL installed”. He responds “You can’t ssh through Git”.
I was actually a little shocked. I didn’t know if he was serious or not so I was silent for a few seconds before hesitantly saying “yes you can”. He says “this is news to me” and I so I tell him “every single one of our build jobs fetches code from Git with ssh” and he seemed genuinely shocked and surprised by that.... so then it occurs to me to show him that you can ssh in Powershell and that REALLY blew his mind. He would not shut up about it for several minutes. I was amused until it just got annoying.

Needless to say, my team had been previously teasing me about having to work with him, so they found it hilarious when I told them afterwards.

So we have an API that my team is supposed send messages to in a fire and forget kind of style.

We are dependent on it. If it fails there is some annoying manual labor involved to clean that mess up. (If it even can be cleaned up, as sometimes it is also time-sensitive.)

Yet once in a while, that endpoint just crashes by letting the request vanish. No response, no error, nothing, it is just gone.

Digging through the log files of that API nothing pops up. Yet then I realize the size of the log files. About ~30GB on good old plain text log files.

It turns out that that API has taken the LOG EVERYTHING approach so much too heart that it logs to the point of its own death.

Is circular logging such a bleeding edge technology? It's not like there are external solutions for it like loggly or kibana. But oh, one might have to pay for them. Just dump it to the disk :/

This is again a combination of developers thinking "I don't need to care about space! It's cheap!" and managers thinking "100 GB should be enough for that server cluster. Let's restrict its HDD to 100GB, save some money!"

And then, here I stand trying to keep my sanity :/

"four million dollars"

TL;DR. Seriously, It's way too long.

That's all the management really cares about, apparently.

It all started when there were heated, war faced discussions with a major client this weekend (coonts, I tell ye) and it was decided that a stupid, out of context customisation POC had that was hacked together by the "customisation and delivery " (they know to do neither) team needed to be merged with the product (a hot, lumpy cluster fuck, made in a technology so old that even the great creators (namely Goo-fucking-gle) decided that it was their worst mistake ever and stopped supporting it (or even considering its existence at this point)).

Today morning, I my manager calls me and announces that I'm the lucky fuck who gets to do this shit.

Now being the defacto got admin to our team (after the last lead left, I was the only one with adequate experience), I suggested to my manager "boss, here's a light bulb. Why don't we just create a new branch for the fuckers and ask them to merge their shite with our shite and then all we'll have to do it build the mixed up shite to create an even smellier pile of shite and feed it to the customer".

"I agree with you mahaDev (when haven't you said that, coont), but the thing is <insert random manger talk here> so we're the ones who'll have to do it (again, when haven't you said that, coont)"

I said fine. Send me the details. He forwarded me a mail, which contained context not amounting to half a syllable of the word "context". I pinged the guy who developed the hack. He gave me nothing but a link to his code repo. I said give me details. He simply said "I've sent the repo details, what else do you require?"

1st motherfucker.

Dafuq? Dude, gimme some spice. Dafuq you done? Dafuq libraries you used? Dafuq APIs you used? Where Dafuq did you get this old ass checkout on which you've made these changes? AND DAFUQ IS THIS TOOL SUPPOSED TO DO AND HOW DOES IT AFFECT MY PRODUCT?

Anyway, since I didn't get a lot of info, I set about trying to just merge the code blindly and fix all conflicts, assuming that no new libraries/APIs have been used and the code is compatible with our master code base.

Enter delivery head. 2nd motherfucker.

This coont neither has technical knowledge nor the common sense to ask someone who knows his shit to help out with the technical stuff.

I find out that this was the half assed moron who agreed to a 3 day timeline (and our build takes around 13 hours to complete, end to end). Because fuck testing. They validated the their tool, we've tested our product. There's no way it can fail when we make a hybrid cocktail that will make the elephants foot look like a frikkin mojito!

Anywho, he comes by every half-mother fucking-hour and asks whether the build has been triggered.

Bitch. I have no clue what is going on and your people apparently don't have the time to give a fuck. How in the world do you expect me to finish this in 5 minutes?

Anyway, after I compile for the first time after merging, I see enough compilations to last a frikkin life time. I kid you not, I scrolled for a complete minute before reaching the last one.

Again, my assumption was that there are no library or dependency changes, neither did I know the fact that the dude implemented using completely different libraries altogether in some places.

Now I know it's my fault for not checking myself, but I was already having a bad day.

I then proceeded to have a little tantrum. In the middle of the floor, because I DIDN'T HAVE A CLUE WHAT CHANGES WERE MADE AND NOBODY CARED ENOUGH TO GIVE A FUCKING FUCK ABOUT THE DAMN FUCK.

Lo and behold, everyone's at my service now. I get all things clarified, takes around an hour and a half of my time (could have been done in 20 minutes had someone given me the complete info) to find out all I need to know and proceed to remove all compilation problems.

Hurrah. In my frustration, I forgot to push some changes, and because of some weird shit in our build framework, the build failed in Jenkins. Multiple times. Even though the exact same code was working on my local setup (cliche, I know).

In any case, it was sometime during sorting out this mess did I come to know that the reason why the 2nd motherfucker accepted the 3 day deadline was because the total bill being slapped to the customer is four fucking million USD.

Greed. Wow. The fucker just sacrificed everyone's day and night (his team and the next) for 4mil. And my manager and director agreed. Four fucking million dollars. I don't get to see a penny of it, I work for peanut shells, for 15 hours, you'll get bonuses and commissions, the fucking junior Dev earns more than me, but my manager says I'm the MVP of the team, all I get is a thanks and a bad rating for this hike cycle.

4mil usd, I learnt today, is enough to make you lick the smelly, hairy balls of a Neanderthal even though the money isn't truly yours.

Once upon a time, one or two jobs ago, a really awesome engineer specced out a distributed search application in response to a business need. This company was managed pretty oldschool and required a ton of paperwork and approvals.

The engineer spent many weeks running tests and optimizing the hell out of this app cluster. It flew, and he had the data to prove it could handle production workloads (think hundreds of terabytes of data being processed every single day)

Part of the way he achieved this was having RAID0 on all of the servers to maximize I/O throughput. He didn't care much about data loss, since the application itself was fault tolerant on a much more granular level.

Management, hearing about this, absolutely flipped their shit and demanded RAID6 instead. This despite the conclusive data that the engineer had that proved RAID6 couldn't keep up.

He more or less got told to STFU.

Even this despite the fact that a RAID restripe would actually take many times longer than rebuilding the failed node from scratch (a process that took about 30 minutes by hand, and could probably be automated to be done in less than five), causing a longer exposure to actual data loss throughout the length of the days-long array rebuild time.

The ill-thought-out requirement added about 50% to the cost of the project (*many* more hard drives now required), beyond the original budget, and the subsequent bureaucratic wrangling resulted in a late product launch.

6 months or so later, after real customers were using this product, the app was buckling under around half of its expected workload. A friend of the engineer suggested to management to try RAID0. Sure enough, that resolved the I/O bottleneck.

This rage-inducing story has a happy ending, though! Said engineer left the company not long after this incident, citing it as a reason for his departure. He was immediately hired by another company, making integer multiples of his prior salary.

The product the company botched the launch of by ignoring his spec? It died a few months later. Maybe the poor customer experience was to blame? Maybe the late launch? Maybe it was another reason entirely.

Either way, millions of dollars of hardware now sat fallow. This was a black eye on the company all the way up to the C-level.

tl;dr: Listen to your engineers. You hired them for their expertise.

Was forced to do some work on Windows this week (CAD tools that runs only on Windows). I spent a few days just setting up the tools. There were quite a few things I realized I forgot about Windows (as compared to Linux).

1) Installation times are down right horrific. What exactly are the installer doing for 10 minutes?
2) .NET is a cluster fuck. Not even Microsofts repair tool can fix it, but rather just hangs. I ended up using another tool to nuke it and reinstall.
3) Windows binary installs are insanely huge, thus, takes forever to download.
4) The registry is a pointless database that must have been written in hell with the single intent of destroying users will to live. The sole existence of the registry is another proof that completely incompetent engineers designed Windows.
5) Rebooting is the only way to solve many problems. This is another sure sign of a fundamentally fucked up OS design.
6) What the heck is wrong with the GUIs designers? The control panel must be the worst design ever. There are so many levels to get to a particular setting I'm getting dizzy. Nothing gets better by the illogical organisation.
7) Windows networking. A perversion of the tcp/ip stack that makes it virtually impossible to understand a damn thing about the current network configuration. There are at least 3 different places that effects the settings.
8) Windows command prompt. Why did they even bother to leave it in? The interpreter is as intelligent as retarded donut. You can't do anything with it, except typing "exit" and Google for another solution.
8) Updates. Why does it takes hundreds of updates per month to keep that thing safe?
9) Despite all updates that is flying out of Redmond like confetti, it is still necessary to install antivirus to keep the damn thing safe. That cost extra money, and further cost you by degrading performance of your hardware.
10) Window performance. Software runs like it was swimming in molasses. The final stab in the back on your hardware investment, and pretty much sends performance on your hardware back a few hundred bucks more.
11) Closed source is evil. If something crash consistently, you might find a forum that address the issues you have. Otherwise you're out of luck. On the other hand, it might be for the better. I imagine reading the code for Windows can lead to severe depression.

I'm lucky to be a Linux dev, and should probably not complain too much... But really, Windows, go get yourself hit by a truck and die. I won't miss you.

Just let me be a programmer. Why do I have to learn yaml and deploy an app with 100 lines of code to a kubernetes cluster with literally 0 users?
Scale sucks.

5 of us working for a larger team were tasked with doing some R&D, we blew everyone away and were given funding to start a new team and hire people to make the project come to life.

One of the high level sales / product managers we were reporting to, secretly had another team work on a similar idea because he needed it quicker (i.e. no time for research, just build it).

After forming new team, we were asked to work on his project instead because it was further along. 4 months later, big knob comes to a meeting and basically says "You know what, this doesn't look like we have enough features, we need more, but I don't know what".

Project blew up 2 months later, head of the unit kicked up a shit storm saying how badly everything was planned and canned everything. Now one of our clients is building nearly the same thing we were originally working on, the team no longer exists and i'm back on the R&D team.

Don't get me wrong, I LOVE the R&D team, actually didn't want to leave in the first place but was told I had to. But the sheer anger and frustration to see that walking cluster fuck strutting around like his shit doesn't stink, derailing entire teams, meanwhile we can't hire new staff due to lack of funding.

Heres an idea, fire the fucktards bleeding us dry ... then we'll have lots of funding.

REDIS: Great for cloud, will fuck up your local disk if too many write operations per second.

DynamoDB: WTF 10Mb should not be "too large for a single record"!!

SPARK: NEVER CONNECT IT TO A DATABASE! Wasted A LOT of cluster time. Also, can you be LESS specific on exactly what are the bugs in my code? 'cause I don't think it's possible.

NPM: can't install a package for shit. tried it waaaay to many times.

Makefiles: Just fuck you.

WSL1: breaks more often than a glass hammer.

Python >= 3.6: FUCK ENCODINGS!!

Jupyter: STOP MESSING UP WHILE SAVING!

Living is to collet bugs, it seems.

I do like Windows, it is a quite good OS nowadays, but for FUCK SAKE, what does it take to fix that CLUSTER FUCK that you call search? You don't have enough people MS or what? Just show me the BLOODY ITEMS that actually contain the words that I typed in!!! While you are at it WOULD YOU MIND LEAVING ME THE FUCK ALONE WITH THE FUCKING WEB RESULTS???

So I was wondering, what is you guys' 'goal' in the long run? What do you really want to be able to do or what do you really want to achieve in 5 to 10 years? Are you working primarily for an income or are there more prominent reasons for doing what you do?

I'm in my last years of university and have a job at a software development company, not because I need the money but because of all the amazing technologies I come in touch with and the amazing things I learn, mostly about servers and devops tools.

I currently have 2 goals in mind: Creating an .io game together with a frontend developer and setting up a kubernetes cluster and using it. I personally consider kubernetes an end-game thing when it comes to running apps and am experimenting with it on three servers that I got from school.

So I was wondering, what ahout you? 😀

I had spent the last year working on a online store power by woocommerce with over 100k products from various suppliers. This online store utilized a custom API that would take the various formats that suppliers offer their inventory in and made them consistent. Now everything was going swimmingly initially, but then I began adding more and more products using a plug-in called WP all import. I reached around 100k products and the site would take up to an entire minute to load sometimes timing out. I got desperate so I installed several caching plugins, but to no avail this did not help me. The site was originally only supposed to take three to four months but ended up taking an entire year. Then, just yesterday I found out what went wrong and why this woocommerce website with all of these optimizations was still taking anywhere from 60 to 90 seconds to load, or just timing out entirely. I had initially thought that I needed a beefier server so I moved it to a high CPU digitalocean VM. While this did help a little bit, the site was still very slow and now I had very high CPU usage RAM usage and high disk IO. I was seriously stumped the Apache process was using a high amount of CPU and IO along with MYSQL as well. It wasn't until I started digging deeper into the database that I actually found out what the issue was. As I was loading the site I would run 'show process list' in the SQL terminal, I began to notice a very significant load time for one of the tables, so I went to go and check it out. What I did was I ran a select all query on that particular table just to see how full it was and SQL returned a error saying that I had exceeded the maximum packet size. So I was like okay what the fuck...
So I exited my SQL and re-entered it this time with a higher packet size. I ran a query that would count how many rows were in this particular table and the number came out to being in the millions. I was surprised, and what's worse is that this table belong to a plugin that I had attempted to use early in the development process to cache the site. The plugin was deactivated but apparently it had left PHP files within the wp content directory outside of the actual plugin directory, so it's still executing scripts even though the plugin itself was disabled. Basically every time I would change anything on the site, it would recache the whole thing, and it didn't delete any old records. So 100k+ products caching on saves with no garbage collection... You do the math, it's gonna be a heavy ass database. Not only that but it was serialized data, so when it did pull this metric shit ton of spaghetti from the database, PHP then had to deserialize it. Hence the high ass CPU load. I had caching enabled on the MySQL end of things so that ate the ram. I was really desperate to get this thing running.
Honest to God the main reason why this website took so long was because the load times made it miserable to work on. I just thought that the hardware that I had the site on was inadequate. I had initially started the development on a small Linux VM which apparently wasn't enough, which is why I moved it to digitalocean which also seemed to not be enough, so from there I moved to a dedicated server which still didn't seem to be enough. I was probably a few more 60-second wait times or timeouts from recommending a server cluster to my client who I know would not be willing to purchase it. The client who I promised this site to have completed in 3 months and has waited a year. Seriously, I would tell people the struggles that I would go through with this particular site and they would just tell me to just drop the site; just take the money, just take the loss. I refused to, this was really the only thing that was kicking my ass. I present myself as this high-and-mighty developer like I'm just really good at what I do but then I have this WordPress site that's just beating the shit out of me for a year. It was a very big learning experience and it was also very humbling as well, it made me realize that I really don't know as much as I think I might. It was evidence that there is still so much more to learn out there, I did learn a lot from that experience especially about optimizing websites the different types of methods to do that particular lonely on the server side and I'll be able to utilize this knowledge in the future.
I guess the moral of the story is, never really give up. Ultimately things might get so bad that you're running on hopes and dreams. Those experiences are generally the most humbling. Now I can finally present the site that I am basically a year late on to the client who will be so happy that I did not give up on the project entirely. I'll have experienced this feeling of pure euphoria, and help the small business significantly grow their revenue. Helping others is very fulfilling for me, even at my own expense.

Anyways, gonna stop ranting. Running out of characters. If you're still here... Ty for reading :')

A new head of operations joins a small company.
— Okay guys, I’m planning for the long run. I need 500 warehouses across the country — we might need that capacity. We will build them rather than renting them — Amazon does the same thing, so we should too. We also need our own shipping fleet — FedEx has that too, so it’s a battle-tested approach. We might need that capacity. We need a future-proof solution.
— Uh… That’s kind of dumb. Are you kidding me?

A new head of engineering joins a small company.
— Okay guys, I’m planning for the long run. I need an AWS cluster running Kubernetes deploying microservices built with Docker. We might need autoscaling. Frontend should be Next.js + TypeScript — everyone does that now, plus we can develop a React Native app more easily if need be. We need a future-proof solution.
— Wow! That’s what I call a good manager. You really know what you’re talking about. You’re promoted!

Building my own router was a great idea. It solved almost all of my problems.

Almost.

Just recently have I started to build a GL CI pipeline for my project. >100 jobs for each commit - quite a bundle. Naturally, I have used up all my free runners' time after a few commits, so I had to build myself a runner. "My old i7 should do well" - I thought to myself and deployed the GL runner on my local k8s cluster.

And my router is my k8s master.

And this is the ping to my router (via wifi) every time after I push to GL :)

DAMN IT!

P.S. at least I have Noctua all over that PC - I can't hear a sound out of it while all the CPUs are at 100%

This begs for a rant... [too bad I can't post actual screenshots :/ ]

Me: He k8s team! We're having trouble with our k8s cluster. After scaling up and running h/c and Sanity tests environment was confirmed as Healthy and Stable. But once we'd started our load tests k8s cluster went out for a walk: most of the replicas got stoped and restarted and I cannot find in events' log WHY that happened. Could you please have a look?

k8s team [india]: Hello, thank you for reaching out to k8s support. We will check and let you know.

Me: Oh, you're welcome! I'll be just sitting here quietly and eagerly waiting for your reply. TIA! :slightly_smiling_face:

<5 minutes later>

k8s team India: Hi. Could you give me a list of replicas that were failing?

Me: I gave you a Grafana link with a timeframe filter. Look there -- almost all apps show instability at k8s layer. For instance APP_1 and APP_2 were OK. But APP_3, APP_4 and APP_5 were crashing all over the place

k8s team India: ok I will check.

<My shift has ended. k8s team works in different timezone. I've opened up Slack this morning>

k8s team India: HI. APP_1 and APP_2 are fine. I don't even see any errors from logs, no restarts. All response codes are 200.

Me: 🤦‍♂️ .... Man, isn't that what I've said? ... 🤦‍♂️

Things that make me feel like a child:

* Getting access to company computational cluster.

Too many night shifts.

But it's done.

After the last migrations my emotional state is... Questionable.

VM migrations between different CPU vendors and generations leading to segfaults because of unsupported X86 extensions.... Thx for doing that at 23 o'clock after 8 hours of work....

Forgetting a left over NIC in a virtual machine, creating a routing loop, leading to very erratic behaviour and fun things.

Someone forgot to check the '"Unique" box, mass spawning a cluster of VMs with same MAC adresses....

DNS fuckery since someone thought that reboot would flush the cache of an DNS server.... Nope most DNS servers have persistent caches. You'll have to flush manually.

And let's not forget the joy of the 12 plus pages of when and where to move VMs, harddrives and VLAN configuration.

Oh migrations are such a festival of joy.

Finally done with that shit -.-

Want to make someone's life a misery? Here's how.

Don't base your tech stack on any prior knowledge or what's relevant to the problem.

Instead design it around all the latest trends and badges you want to put on your resume because they're frequent key words on job postings.

Once your data goes in, you'll never get it out again. At best you'll be teased with little crumbs of data but never the whole.

I know, here's a genius idea, instead of putting data into a normal data base then using a cache, lets put it all into the cache and by the way it's a volatile cache.

Here's an idea. For something as simple as a single log lets make it use a queue that goes into a queue that goes into another queue that goes into another queue all of which are black boxes. No rhyme of reason, queues are all the rage.

Have you tried: Lets use a new fangled tangle, trust me it's safe, INSERT BIG NAME HERE uses it.

Finally it all gets flushed down into this subterranean cunt of a sewerage system and good luck getting it all out again. It's like hell except it's all shitty instead of all fiery.

All I want is to export one table, a simple log table with a few GB to CSV or heck whatever generic format it supports, that's it.

So I run the export table to file command and off it goes only less than a minute later for timeout commands to start piling up until it aborts. WTF. So then I set the most obvious timeout setting in the client, no change, then another timeout setting on the client, no change, then i try to put it in the client configuration file, no change, then I set the timeout on the export query, no change, then finally I bump the timeouts in the server config, no change, then I find someone has downloaded it from both tucows and apt, but they're using the tucows version so its real config is in /dev/database.xml (don't even ask). I increase that from seconds to a minute, it's still timing out after a minute.

In the end I have to make my own and this involves working out how to parse non-standard binary formatted data structures. It's the umpteenth time I have had to do this.

These aren't some no name solutions and it really terrifies me. All this is doing is taking some access logs, store them in one place then index by timestamp. These things are all meant to be blazing fast but grep is often faster. How the hell is such a trivial thing turned into a series of one nightmare after another? Things that should take a few minutes take days of screwing around. I don't have access logs any more because I can't access them anymore.

The terror of this isn't that it's so awful, it's that all the little kiddies doing all this jazz for the first time and using all these shit wipe buzzword driven approaches have no fucking clue it's not meant to be this difficult. I'm replacing entire tens of thousands to million line enterprise systems with a few hundred lines of code that's faster, more reliable and better in virtually every measurable way time and time again.

This is constant. It's not one offender, it's not one project, it's not one company, it's not one developer, it's the industry standard. It's all over open source software and all over dev shops. Everything is exponentially becoming more bloated and difficult than it needs to be. I'm seeing people pull up a hundred cloud instances for things that'll be happy at home with a few minutes to a week's optimisation efforts. Queries that are N*N and only take a few minutes to turn to LOG(N) but instead people renting out a fucking off huge ass SQL cluster instead that not only costs gobs of money but takes a ton of time maintaining and configuring which isn't going to be done right either.

I think most people are bullshitting when they say they have impostor syndrome but when the trend in technology is to make every fucking little trivial thing a thousand times more complex than it has to be I can see how they'd feel that way. There's so bloody much you need to do that you don't need to do these days that you either can't get anything done right or the smallest thing takes an age.

I have no idea why some people put up with some of these appliances. If you bought a dish washer that made washing dishes even harder than it was before you'd return it to the store.

Every time I see the terms enterprise, fast, big data, scalable, cloud or anything of the like I bang my head on the table. One of these days I'm going to lose my fucking tits.

Holy fcuk! Can anyone here help me understand how this domain is possible?

WARNING: obviously its a spam site. Take necessary security precautions if you are going to visit.

the following domain opens a cluster fuck domain name! >> secret.ɢoogle.com

That ɢ is not what it looks like. How is such domains possible to exist? Even more surprising, how is this sub domain -ception possible?

Some "engineers" entire jobs seems to only consist of enforcing ridiculous bureaucracy in multinational companies.

I'm not going to get specific, the flow is basically:

- Developer that has to actually write code and build functionality gets given a task, engineer needs X to do it - a jenkins job, a small k8s cluster, etc.
- Developer needs to get permission from some highly placed "engineer" who hasn't touched a docker image or opened a PR in the last 2 years
- Sends concise documentation on what needs to be built, why X is needed, etc.

Now we enter the land of needless bureaucracy. Everything gets questioned by people who put near 0 effort into actually understanding why X is needed.

They are already so much more experienced than you - so why would they need to fucking read anything you send them.

They want to arrange public meetings where they can flaunt their "knowledge" and beat on whatever you're building publicly while they still have nearly 0 grasp of what it actually is.

I hold a strong suspicion that they use these meetings simply as a way to publicly show their "impact", as they'll always make sure enough important people are invited. X will 99% of the time get approved eventually anyway, and the people approving it just know the boxes are being ticked while still not understanding it.

Just sick of dealing with people like this. Engineers that don't code can be great, reasonable people. I've had brilliant Product Owners, Architects, etc. But some of them are a fucking nightmare to deal with.

I'm a python fanboy, not gonna lie.
I love everything about it. It's clean syntax, ready to use out of the box-ness, convenient built-in functions.

The one thing I hate is the official documentation. It's ugly, hard to navigate and a cluster fuck.

But it has proper information, so it's fine I guess. tsch

This just happened....

Tester: My cluster is not working properly!!!
Me: What's wrong?
Tester: I don't know. I've checked all the logs available on the entire cluster. All i know is that node 1 and 7 is broken.
*ssh into the cluster*
node1
# less /var/log/<affected application log>.log
*no errors here everything is working properly*
node7
# less /var/log/<affected application log>.log
*goes down to the bottom and scrolls up a few lines*
<insert massive error here>

Checked all the logs eh?

So, it's 22:40 here and I'm sat on a bench staring out at a pond because my stress and anxiety is at an all time high after a couple of weeks of hellish arguments with work and my personal life so as were all developers here to some degree let me convey my fucking thoughts here.
If you care more about maintaining your fucking superiority complex over writing good clean efficient code then get the fuck out of the industry.
I don't give two fucks whether you use Linux or Windows. I couldn't give two fucks about whether you use sublime, Emacs or VIM. I couldn't give two fucks about the framework you spend more time defending than coding in, because absolutely none of it matters if you code like a retard on bath salts you pretentious cunts.
Stop feeding you fucking ego. Absolute cluster fuck of an industry.

>Client complains about a 30 minute downtime around midnight
>Client also pays only for a single VM on a HV that they don't even own themselves

>Replies with an offer of how to make the setup more resilient, going from 1 VM to 2 LBs/FE loadbalanced through BGP, and distributing traffic through HaProxy onto 2 BE machines that in turn talk to a Postgres Cluster with RepMgr for dynamic failover.

>No reply so far

Hmmm :^)

Boss activates encryption on dashboard
we installed the software
2 machines get locked out coz drive got encrypted with bitlocker
No one received the 48 bit key from bitlocker
I loose all my work coz the only way to use my laptop was to format the drive
Me as the technical guy and knowing how encryption works i just formatted the drive

Boss blames me for the cluster fuck

Everyone and their dog is making a game, so why can't I?
1. open world (check)

2. taking inspiration from metro and fallout (check)

3. on a map roughly the size of the u.s. (check)

So I thought what I'd do is pretend to be one of those deaf mutes. While also pretending to be a programmer. Sometimes you make believe
so hard that it comes true apparently.

For the main map I thought I'd automate laying down the base map before hand tweaking it. It's been a bit of a slog. Roughly 1 pixel per mile. (okay, 1973 by 1067). The u.s. is 3.1 million miles, this would work out to 2.1 million miles instead. Eh.

Wrote the script to filter out all the ocean pixels, based on the elevation map, and output the difference. Still had to edit around the shoreline but it sped things up a lot. Just attached the elevation map, because the actual one is an ugly cluster of death magenta to represent the ocean.

Consequence of filtering is, the shoreline is messy and not entirely representative of the u.s.

The preprocessing step also added a lot of in-land 'lakes' that don't exist in some areas, like death valley. Already expected that.

But the plus side is I now have map layers for both elevation and ecology biomes. Aligning them close enough so that the heightmap wasn't displaced, and didn't cut off the shoreline in the ecology layer (at export), was a royal pain, and as super finicky. But thankfully thats done.

Next step is to go through the ecology map, copy each key color, and write down the biome id, courtesy of the 2017 ecoregions project.

From there, I write down the primary landscape features (water, plants, trees, terrain roughness, etc), anything easy to convey.
Main thing I'm interested in is tree types, because those, as tiles, convey a lot more information about the hex terrain than anything else.

Once the biomes are marked, and the tree types are written, the next step is to assign a tile to each tree type, and each density level of mountains (flat, hills, mountains, snowcapped peaks, etc).
The reference ids, colors, and numbers on the map will simplify the process.

After that, I'll write an exporter with python, and dump to csv or another format.

Next steps are laying out the instances in the level editor, that'll act as the tiles in question.
Theres a few naive approaches:

Spawn all the relevant instances at startup, and load the corresponding tiles.

Or setup chunks of instances, enough to cover the camera, and a buffer surrounding the camera. As the camera moves, reconfigure the instances to match the streamed in tile data.

Instances here make sense, because if theres any simulation going on (and I'd like there to be), they can detect in event code, when they are in the invisible buffer around the camera but not yet visible, and be activated by the camera, or deactive themselves after leaving the camera and buffer's area.

The alternative is to let a global controller stream the data in, as a series of tile IDs, corresponding to the various tile sprites, and code global interaction like tile picking into a single event, which seems unwieldy and not at all manageable. I can see it turning into a giant switch case already.

So instances it is.

Actually, if I do 16^2 pixel chunks, it only works out to 124x68 chunks in all. A few thousand, mostly inactive chunks is pretty trivial, and simplifies spawning and serializing/deserializing.

All of this doesn't account for
* putting lakes back in that aren't present

* lots of islands and parts of shores that would typically have bays and parts that jut out, need reworked.

* great lakes need refinement and corrections

* elevation key map too blocky. Need a higher resolution one while reducing color count
This can be solved by introducing some noise into the elevations, varying say, within one standard div.

* mountains will still require refinement to individual state geography. Thats for later on

* shoreline is too smooth, and needs to be less straight-line and less blocky. less corners.

* rivers need added, not just large ones but smaller ones too

* available tree assets need to be matched, as best and fully as possible, to types of trees represented in biome data, so that even if I don't have an exact match, I can still place *something* thats native or looks close enough to what you would expect in a given biome.
Ponderosa pines vs white pines for example.

This also doesn't account for 1. major and minor roads, 2. artificial and natural attractions, 3. other major features people in any given state are familiar with. 4. named places, 5. infrastructure, 6. cities and buildings and towns.

Also I'm pretty sure I cut off part of florida.
Woops, sorry everglades.

Guess I'll just make it a death-zone from nuclear fallout.

Take that gators!

Aaaand I soon as I leave, the colleague crashed the nutanix cluster

Amazing work

Me: do you have monitorin enabled for your cluster?
They: no

me: I recommend enabling it
they: naah, we're good

error: *occurs*
they: *try to guess at which hop which limits were or were not hit and/or which nw links could've dropped a packet with trial-and-error approach for days*

me: telemetry would give you an answer in under a minute....
They: ok cool. We're still good

My boss did not care about making things secure in our early development stage, even though I told him several times.
After 1 day our elastic search cluster was filled with random crappy data.

Fix: Apply security schemes provided by AWS

Monday is the deadline for a big project in my master, I worked together with 11 people for one whole year. Our repository is on a server from university and we use the uni cluster to evaluate our results.
And today the uni spontaneous turns of its electricyty for the weekend to test some fallback systems...

Just got a new job at an old school hardware company. The codebase is giving me heart attack. They don't care about dev experience or code navigation at all. Every attempts to modernize the codebase is so half assed. All patches are so bloated that make the codebase even worse.

Frontend is migrated from prototype-oop-jquery cluster fuck to AngularJS, then finally angular. Holy moly, all business logics are baked into UI "classes" using prototype chain. When they migrated to AngularJS, someone simply added a wrapper to that jQuery cluster fuck class and overwrote all the prototype with a 10k +lines file. Since all the methods are hidden in either prototype, JS object, or callback function, it's impossible to trace the data pipeline using IDE when "go to definition" on update() method gives you all the update methods/string in all objects/classes. And they don't care about immutability. References are taken out, renamed, and mutated everywhere. Finding the source of a bug is fucking guessing game.

I don't know what trick they use that makes cLion static analyzer fail.

And there is no unit test or spec doc.

Fuck me dead

Creating a cluster with shared storage in Proxmox

Once you've learned how to configure a single Proxmox host and Linux and Windows guests, the next step is to expand...

Want to continue reading this article? Register here with your corporate email address. Because your private email address isn't good enough, we need your corporate one.

No TechTarget, how about you go fuck yourself? As if anyone is going to register just to read one fucking article on your goddamn shitty site. Fucking wanketeering dickheads.

Me: The IP Address on your public mail server cluster has been blacklisted.

Supporter: What is the IP? You can get it from whatismyip.com

Me: *has left the chat*

Yesterday I shutdown a database cluster after 2 years of running it, what a ride it was... Felt a little bad when I removed the servers...learned a lot ...

2 weeks and i leave my job.

Pro: no more dealing-with-math-msc-students

Con: i want to keep that juicy cluster to run my own calculations ;_;

update : love my new job... but, holly crap the sharepoint environment is a cluster fuck... guess who has to fix it... -!-

- Launch the new version of the system I have been refactoring for 2 years and counting, then ceremoniously burn (literally) the legacy code as well as the cluster fuck of hardware it runs on.

- Decrease my stress + bus factor by bringing another up to speed on my code & the new version (his cluster fuck now).

- Pay attention to & take better care of health, my wrists in patricular.

- Find a mentor and mentor someone else.

- Get out of crisis management mode and find the time to write tuts, experiment and live a little.

- Find & join a local dev meetup, maybe make a local dev friend.

- Book leave and actually take it, preferabbly without having to take my laptop to the beach - actually, preferabbly at least have the choice to take a offline vacation.

- Sort through the drives containing ALL the code I have ever written, migrate the usefull interesting bits to Github.

Phew, that bit of self reflection was intense! I'm adding a cron to my server to sms & email me this rant in a year to remind me what hope looks like.

@dfox Would you make it possible to cluster push notifications that occur on the same post? When I receive a lot of notifs at once it clogs up my notification center.

Additionally would it be possible so that I can completely turn off seeing certain types of notifs even in the app not just for push notifications (from what I gather that is the only thing I have control over)

At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).

At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.

I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.

A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.

First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.

Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.

So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.

Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care.

I'm so fucking frustrated with my ex company CEO, this motherfucker made everyone move to Bangalore costing is employees a good chunk of their salary and this delusional ASSHOLE knew that only half of the expected funding was coming in January 2023 itself and they'd be out of funds by July/August, they let go few folks from the team, fired the entire marketing team and expected to make the product profitable. The only reason I had stayed at that time was because the product was interesting to build and the scale I was working with was crazy like 100k request per minute peaks and avg of 10k rpm. I left the company in August...

This MOTHERFUCKER hasn't paid out final settlement after leaving for most employees and he openly says to the folks who are still working there that paying us is not his priority.

I hope your Atlas cluster gets fucking deleted, accesses revoked and entire AWS setup goes down forever, bitch.

We can't goto courts because the company law tribunal needs atleast 1 crore (1.2 million usd) of unpaid dues to declare it insolvent in a years time..

This asshole deducted taxes from our salaried but didn't pay them to the income tax department for an entire fucking financial year.

What a cheating, delusional, sick bastard. And he's still not willing to sell off the company to pay off the debts and call it a day.

Aarrghhhh on top of losing 2-3k USD I might have to pay my taxes approx 5-6k USD to the govt to keep my records well maintained.

What a grade A delusional asshole 😡
If he won't clear the dues till December, I'm gonna launch a mass of social media posts and destroy his reputation so that he doesn't get one penny of VC funds in the future I'm gonna make sure of that...

Here is another Tester story for you!

Tester: We have a problem with the application and it says that we cant process the queue to store the objects in the database.
Me: Can we reproduce it easily?
Tester: Yes..
Me: Ok, I'll add some debug for you so that you can see the error and the worker status more easily.
Tester: Great!

<10 minutes later>
After adding the debug output and installing it on his test cluster this happens...
Me: The debug version is installed.
Tester: Ok, great!
Me: How fast does it take to reproduce?
Tester: Well... It happened this morning at 9
Me: Ok..? But are you able to reproduce the issue at all?
Tester: No... i don't think so...

I kept piling on hard drives at home and you know... One day I'll setup some redundancy.
Then a 5 months old full 4TB drive gave up and I lost that data.
After that I've upgraded to having a ceph cluster storing everything.
At the beginning of this year one hard drive in the cluster gave up. I didn't notice until I wondered why available storage was low. Cluster had already rebalance itself and were running flawless.

Debugging distributed systems in a nutshell.

Don't use your senior software engineer title or years of experience as reasons in a debate or argument about software

My manager was asking me what steps needs to be done to perform a disaster recovery for our cluster( on production). I will be honest here, I have not maintained this type of cluster(kubernetes) in production before. However, I have enough understand of the system to answer my boss question. I basically told him there are A, B, C you have to do.

My senior developer jumped in and said "No you should do A,C, B because C is more critical than B. " I then replied to him: "I understand your point, I notice that too, but .." Before I can even finish my sentence, this dude has already rolled his eyes and interrupted me very loudly: "Have you worked with these systems on production before? I did". The asshole knows I haven't maintained Kubernetes on production yet of course.

I got super pissed at him and pretty much shouted back to him and my manager: "Just because I haven't worked on this system on production yet, does not mean my argument is wrong" .

I then dragged my managers, that asshole, and other engineers in a room and settle this out. In the end, people agreed with my steps over that asshole senior engineer dude because I gave them rational reasons.

The conclusion is: Your senior title is given by the company, It doesn't mean anything to me. Also, it doesn't make you more right than another person just because he has a "lower" title than you.

The Matrix. Then use the brains of the people plugged in like a computer cluster for large calculations.

I could kill someone. My boss occupied the whole cluster for 24h yesterday, so that I had to wait until today to see that I had a small bug in my code. I wasted a whole day waiting around for something I could have fixed in 5 minutes yesterday if I only had 1 free node on the cluster 😠

Worst of all, if anybody else had occupied the whole cluster for so long without asking, he would have sent an angry e-mail to the whole institute 😠

Today was a SHIT day!

Working as ops for my customer, we are maintaining several tools in different environments. Today was the day my fucking Kubernetes Cluster made me rage quit, AGAIN!

We have a MongoDB running on Kubernetes with daily backups, the main node crashed due a full PVC on the cluster.

Full PVC => Pod doesn't start
Pod doesn't start => You can't get the live data
No live data? => Need Backup
Backup is in S3 => No Credentials
Got Backup from coworker
Restore Backup? => No connection to new MongoDB

3 FUCKING HOURS WASTED FOR NOTHING

Got it working at the end... Now we need to make an incident in the incident management software. Tbh that's the worst part.

And the team responsible for the cluster said monitoring wont be supported because it's unnecessary....

OMG SharePoint is such a royal cluster fuck.

Random interface with some controls here, some controls there, sometimes shit is in one place, another time it is in another.

It is like a fucking retarded toddler glued a bunch of different animal parts together to make a pet monkey-horse-fish. Frankinware 🧟

Whomever runs that at Microsoft needs a spanking. Hell, their mother needs a spanking for birthing them.

It seems that every Pi day, I end up with a new Raspberry Pi that I got sometime before it. First, a 1, then a 2+3, now I have a Zero as well. Next year, I'll have a cluster of 3s. No reason, just want to mess with it.

What if people, life, humanity, the universe is just a cluster of CPUs running a giant Recurrent Neural Network algorithm? 🤔

-Sun and food == power source
-People == semiconductors
-Earth/a Galaxy == a single CPU
-Universe == a local grouping of nearby nodes, so far the ones we've discovered are dead or not what same data transport protocol/port as us
-Universal Expansion == the search algorithm
-Blackholes: sector failures
-Big Bang == God turns on his PC, starts the program
-Big Crunch == rm -rf

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

Today marks the day that i finally get to do stuff on a production server.
Its just installing the elasticsearch cluster. But i still feel honored by the trust im given even tho im still an apprentice.

For just 26 easy payments of $79.99, our AI cluster that consumes as much electricity as Guatemala will take photos of the hottest event (out of unconsenting people's phones), inpaint you into them next to the celebrities and post it to your Instagram and TikTok! For $19.99/mo you'll get at least 30 comments saying “mood af” and “🍆🍑💦”, and for $39.99/mo we will highlight other people's posts that were generated by our AI!

Four and half months,
Hundreds of hundreds PRs and one additional product cluster
By 6 Engineers..
To 500+ micro services
Which has no timezone or currency context,
Created by 250+ engineers,

To launch in a new country...

It didn't make me happy.

But the feedback from customers and drivers is priceless and #heartwhelming

Here it goes,

So there I was a Linux enthusiast stuck in a windows job for about 3 years. I would spend my weekends doing Linux related tasks for my personal amusement, while I spent my week doing windows maintenance and development (partially) professionally.
It was about 2014 I started building an openstack cluster at home and i was so stoked! I searched for openstack summits or meetups and for my surprise there was an openstack meetup in my town. Holly 🐄 I said.
The date of the event came and I left work earlier to attend the meetup.
There , I had a talk with the meetup organizer/speaker and he told he was interested in what I was doing and that they were going to open a job in the next months.

A few months later still at my boring job I got an email from him for an interview.
Everything went just about right...and here I am a Linux systems engineer doing everything I love for a living...

Stupid pipeline bullshit.

Yeah i get it, it speeds up development/deployment time, but debugging this shit with secret variables/generated config and only viewable inside kubernetes after everything has been entered into the helm charts through Key Vaults in the pipeline just to see the docker image fail with "no such file found" or similar errors...

This means, a new commit, a new commit message, waiting for the docker build and push to finish, waiting for the release pipeline to trigger, a new helm chart release, waiting for kubernetes deployment and taking a look at the logs...

And another error which shouldn't happen.

Docker, fixes "it runs on my machine"
Kubernetes, fixes "it runs on my docker image"
Helm, fixes "it runs in my kubernetes cluster"

Why is this stuff always so unnecessarily hard to debug?!

I sure hope the devs appreciate my struggle with this... well guess what, they won't.

Anyways, weekend is near and my last day in this company is only four months away.

Avoid ACPICA if at all possible. It's one garbage tier cluster fuck of bad design, horrible documentation and downright misleading and wrong code

It's meant to consist of an ASL compiler, disassembler, debugger, dumper, various user space utitilies and a kernel resident OSPM implementation *if* you can figure out what belongs to what. Even just compiling this pile of trash is a mystery in itself. Think you need the source files in source/common? EEEEH, wrong. Well, at least partially since most of them seem to be for the user space stuff..? Other ones *are* needed on the other hand. At least the disassembler and/or debugger and/or dumper components seem to reference them. Not that I could figure out how to compile those anyways. The real path to your goal seems to be to ignore a seemingly arbitrary subset of source and header files until your linker stops complaining

There's also a bunch of configuration defines, some of which *you* define, some defined *for* you, based on again others. Of course most of them do stupid shit. Enabling the debugger automatically enables debug logging. Enabling the disassembler force enables debug allocation tracking... What?

The code itself isn't of much help either. Looking in "os_specific/service_layers" you find what looks to be reference implementations of acpica functions in certain os' like windows and unix. Of course I had a look because AcpiOsReadMemory is supposed to read physical memory and I don't know how I would even implement that. But hey, osunixxf.c (xf for interface... of course) should tell me. I'll let you see for yourself in the attached image. Apparently it does fuck all and just returns AE_OK. No error, no logging, no nothing. Just ok. As you can imagine, AcpiOsWriteMemory doesn't do much more either.
...okay so maybe physical memory accesses aren't actually used and these functions are some sort of relic from past times? Nope! They are absolutely necessary for doing low level device interaction. WTF. So finally I went to the linux source and checked how *they* implemented them, and just as I thought, these functions are anything but no-ops...
...So for what fucking reason do these stupid interface implementations even exist but to purposefully mislead you?? They aren't used for fucking anything! As far as I know Windows doesn't even *use* ACPICA and Linux have their own fork with working implementations... They just sit there, just to tell you how to NOT do it

So that's some of my thoughts about ACPICA. Note that I haven't even used it as a library yet, I just got it to compile and link and it already fucked with me this much.
There's also so much more I didn't mention like that you *have* to modify the acpica source in order to get your own platform header working (else #error) eventhough the docs explicitely instruct you not too but you get the point

Don't use ACPICA if you don't have to. Save your sanity for something that's worth it

There is this dude called Richard Eng which is sort of famous for 2 things:
First: he is known as *the* Smalltall evangelist of mothern times. And he constantly writes about it. Which is fine since he tries to attract new users to this beautiful and simple little language.

Second: his constant bashing of other technologies, mainly Javascript stating that it is the most harmful tech known to man.

The thing is, saying "use this because that is shit" is never going to convince a community, specifically one as potent as that of the JS community. And to make it worse...the dude links his reasoning about bad languages to articles he wrote. As in "this is shit, look at my completely biased article regarding why its shit"

Once he is confronted about it he links back to his own writings. Much like christian fanatics do
"good is real because it says so in the bible"
"but how can you trust that resource?"
"Because the bible is the word of God"
"and how do you know?"
"Because it is in the bible"

Circular arguments like that cannot be taken seriously. And what this guy does for the Smalltalk community hurts more than it helps really.

Claims like those are all around us. If we were to believe or consider them depending on who said what then we would never have the amazing cluster of tech choices that we have.

Take c++. It is absolutely powerful and gives you the ability to do pretty much anything. If we were to take Linus Torvalds's word about it being shit and only having subpar development we would miss on absolutely powerful tools.

The same came to me from Evee, writer of "PHP a fractal of bad design" or the "Node.js is cancer" article.

You are never going to please anyone with anything. I go by live and let live, and whilst I don't like some technologies I certainly don't look down on those that do.

I'm very short tempered at the moment.

A lot like Dr Cox in Scrubs.

And really ... You mother fucking stupid idiotic developers with your tendency to discuss absolutely everything just to not have to work for a dozen more minutes...

But ok. Let's discuss.

But even that seems to be absolutely impossible for you little shitheads.

Instead of discussing solutions, nooooooooo....

We're grown up developers so we discuss how the baddy manager hurt our lil feelings by saying that we're morons for wasting all the fucking time without coming up with a solution.

Now my lil cry babies, once the baddy manager got your pacifiers so at least once in an hour my migraine finally calms down for not hearing your bitching pathetic lil whiny noises...

Face it. Over the years you collected a huge ton of mother fucking tech debt because no one of you actually took a bit of time to use that empty space in your head to think at least a mu further than the dumb jira task you were given.

And yes. That ends badly.

And yes. As it is now in a state of cluster fuck, guess what. You have to work. You get money for it, remember?

And yes. if you would stop moping and bitching and crying and being a pathetic lil piece of shit, you'd realize we could come up with solutions very fast.

But nooo... Let's talk about our feelings.

And how we are over worked.

And how nothing works.

Cause yes. That will be the hail mary that saves us all.

Let me give u a hint: it's a mother fucking waste of time bitches.

I think it's time I put a pacifier not only in your mouth, but arse too. Maybe it helps overcoming the anal and oral phase of childhood so we can at least have something close to adult talk.

*breathes in*

Gooozfraba.

What the fuck this fucking shit of an app supposedly went through five rounds of QA and I broke it in 5 minutes? Holy fucking shit How this is the main point of your job. You shitty shits take forever getting this PR to me and now it’s buggier than a haunted house on Halloween?

After this is after wanting to cuss Apple for making me update the os then then xcode then iTunes just to recognize the goddamn development phone. It’s an app built off Cordova it works for like 14 versions of iOS just run the bugger you over priced aluminum chassis dildo riding fruit Fuckers.

Now back to the goddamn cluster fuck I just got delivered. What the goddamn fuck. This level of bullshittery I have not seen before. And apparently Cordova is only partly de-sandboxed? I don’t know for sure because I don’t have time to test it because I’m running the entire technological stack of this company on a junior dev salary!

When you tell me it goes though all this QA and you spend 2 weeks on just QA tickets for fucks sake the first operation I do, the most common fucking operation on the app, the shit we have been building around all this fucking time, should not fucking error out. For fucks sake at least try to get the main fucking thing working. I recognize you did a lot of work and implemented a number of features but what the fuck good are they if I can’t even run them for one fucking time.

Just finished dumping all ethereum tranasctions into one big 30 GB csv.

Only thing left is to configure Apache Spark cluster.

Who the hell deploys PROD environments in a STAGING cluster.....?

Who the hell deploys PROD environments in a PROD cluster and (deliberately) configures them to use a STAGING environment's DB cluster....?

At night is when my creativity starts flowing like a motherfucker. The moment when all the tasks are done for the day and you can start working on your own projects and just lay back and smoke (or drink, whichever you prefer) and zone the fuck out with some good music. Oh and if I've gotten a good work out in that day, then there's no stopping me.

I had no plans to even create an admin panel for my own small project but last night I made one just to make it look professional. After that, I got an idea for a separate project which I started working on. I usually write my ideas down so that I don't get into a complete project cluster fuck with 50 half completed projects, but sometimes you get that golden one that you have to start (currently those are the only two unfinished projects I got).

So for a while I have wanted to build a raspberry pi cluster. In the spirit of shia labeouf I got started last saturday.

I had two pies lying around so I figured I'd run some experiments before I invested in a lot of hardware. After about a day I had turned the two pies into a shared cluster when disaster struck....

I had completely ignored the fact that you cannot run 32 or 64bit software on an arm processor (I know... I'm a java developer). So when I booted my service and the load balancer, I found that nothing worked. So pretty bumbed out, I quit the project.

Later that day I found a crazy guy who had bought a batch of 400 small form factor PSUs (300W) and internally I laughed at him a little. I mean, who's gonna sell 300W irregular power supplies. Then, just as I was about to go to bed I found this guy, he was selling from a batch of CPU-onboard motherboard for 10 bucks each and everything clicked!

I did some quick calculations and decided I could probably gather enough cash to get: 10 motherboards, 10 2GB ram dimms, 10 Sata disks and 14 PSU (in case some fail) and some misc hardware for networking and such.

So... Long story short, I am going to build a cluster computer, the first version is going to have 10 nodes and I am waiting for delivery right now!

Today I learned that some external devs one of our projects is working with have DB tables where they store references to specific dates, and not only that, but every minute of those dates, and the day of the week, and what season its in. Im not joking.
Hmm should I use the local datetime libs or should I go through a firewall, load balancer and DB cluster just to find out what day it is?

Developing and deploying in Xcode is some Requiem for a Dream level bullshit.

I literally just de selected everything for managing automatic signing, and re-selected the EXACT SAME GODDAMN THING. And it worked. It’s literally some fucking shit you do when you are first learning how to code or learning a language and you keep flipping something but you don’t get exactly how it works.

But this is YOUR FUCKING FLAGSHIP development product. I shouldn’t have to check my goddamn inception totem to see if I’m dreaming or not because this kind of bullshit can’t be real life.

That being fucking said your bullshit forced shutdown also FUCKED MY ANDROID STUDIO INSTALLATION AND FUCKED MY $PATH. Thanks. Now NOTHING WORKS. Fuck you Apple. Between slowing my phone and the cluster or problems your shit is causing that are just random as hell and are plenty common because thank god people smarter than me have fixed them in SO by now, I am SO READY TO LEAVE THE APPLE ECO SYSTEM. If I didn’t have to use one of the boxes to push iPhone app updates I doubt I would touch one again.

Apple stuff looks good but at this point that’s about it.

Set-up a 5 node EC2 cluster in AWS; Install my dependencies on all; Add private keys between all for handshake; Submit my Hadoop job for processing; AWS closes my instance within 10 minutes of starting a job that took me almost half an hour to set-up because the master node's spot costs have reached more than 15 USD :(

Day 1:
Optimizing huge problems for the company. Get mail. *sigh*; Why is your script using up half the CPU on our thin clients? *place in complaints folder and go on*

Day 2:
Boss asks about it during scrum meeting. *Oh shits*
Need a cluster. Been asking for it for months...

Day 3:
Start runs on all thin clients. *Thou shall feel my wrath*
Complaints folder floods.

Day 4:
Expect rage from boss.
"IT seems to have found a cluster for you at last."

Finally!

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

I wanted to install this new software in our cluster but it didn't have Cmake on it, so I scp Cmake source files and then I started looking for a CMakeLists.txt file for compile it when I suddenly realized what I wanted to accomplish in the first place 😅

Things I love today.

Totally love. Like kick in the balls with testicle torsion love. Picking my eyeballs out with a spoon... I think you got the idea.

Getting updates of other managers, as I'm busy with other stuff.

More or less goes like this:

Flaky tests. Since weeks...
Ain't nobody got time for that.
🤬

I don't wanna upgrade that version to the next major version, cause then I'd have to do tests... And the tests are flaky.

🤬🤬

I wanna have shiny new thing XY, but NOONE wants to upgrade to next major version so we cannot have that

🤬🤬🤬

Oh we just crushed the live cluster cause there's this PR everyone constantly ignores cause the tests are flaky....

🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬🤬

Good thing I'm busy and just getting all the updates via the gossip mill...

I'm just prolonging my current tasks as I really don't wanna have to fix that mess.

My fix would be probably eye for eye, tooth for tooth.

...

Problem is.... I'm slowly getting into trouble because some of these fixes would be much needed for my task...

Why do I have always to be the bad cop in the company -.

I think I'm gonna ask HR what applying electro shocks would cost me, cause I think that would solve a lot of problems.

10 kV for every stupid answer.

Smells like bacon!

Currently learning Kubernetes Cluster. Wow so much to learn.

Much easier to setup any app for a Cluster. Only building the Image takes time 🙄

Containers are the future ✌🏻

There is a drawing competition for my currently most played game.
I'm on vacation and the deadline is when I get back. So what did I do?

I made an inspiration cluster with the character and drawings having this arrogant face (laptop, gimp).
I sketched on my phone and put it on my laptop (the sketch). I have no desk here so I'm drawing on my bed. I have no drawing pad so I'm drawing with my mouse. Then I draw it in gimp with colors and everything (the stroke in another program on my gfs laptop), put each layer in inkscape to svg-ify them and to hq-render them back in Gimp. Corrected a few things in Gimp. Added more detail, effects (glow, gradient instead of flat color ...).

~6 hours over two days. That was fun. And fucking unprofessional.

1. Exporting fat jar
2. Transferring to cluster using WinSCP
3. Running it in the cluster.
4. Find a small bug.
5. Repeat

I HATE IT

Hey guys i wanna build a Pi-like Cluster to try out kunernetes locally and I would like some parts advice for the boards, casing, cooling, and the rack itself.

so far I have a Cisco E1200 Router I would use for the cluster for SSHing and connecting to

I'm currently between jobs and have a few rants about my previous job (naturally). In retrospect, it's somewhat therapeutic to range about the sheer brainfuckery that has taken place. Enjoy!

First, let me set the scene: legacy B2B web app made with LEMP stack and sencha ext.js 3 + 4 (don't ask) and a lot of madness. Let's call that app "Alpha".

Alpha is a self made CMS build for typical ERP stuff. Yes, a self made CMS: entities are containers, containers have types and fields and values. Like so many legacy PHP apps, it does not have a dedicated FE: the HTML is rendered on the server and then spewed out to the browser.

Easy right? Coding like it's 1999! But there was a twist: Because everything is basically a container, the HTML-templates are saved in the DB. Along with the nessary JS and the CSS. And the translation variables. Why? Because fuck you! That's why. Who needs a git history anyways.

For some reason, Alpha was kinda slow.

There was also an editor, that allowed you to modify templates (web, mail, pdf) on the fly in prod. Because templates contain repeating data (header/footer), one template could contain additional templates. Much confusion. You could change templates via migration (slow, boring) or just ctrl-c/ctrl-v that sucker (fast, much excitement).

Did I mention Alpha was slow?

On with the rant: e-mails! How do they work? Noone knows. How to send mails asynchronous in PHP? Witchcraft is the only possible answer to that riddle. Here is your enterprise™ solution:
1. create mail
2. insert mail into DB
3. WAIT UP TO 59 SECONDS FOR A FUCKING CRON TO SEND MAIL

Why? "Because that way, we can resend mails in case the network is down :)"

Same procedure for the SOAP-API (db-queue + cron). You read that right: all requests to various other systems are processed once a minute.

Alpha slow.

Alpha was only one of several systems. Imagine a bunch of monolithic php apps, interconnected via SOAP, REST and GraphQL like a godamn intergalactic orgy. Image having to debug that cluster fuck.

Let's say there is a bad request. These things happen. No biggie. Remember the db-queue? Let's try to send the bad request a second time! And a third time! Still no luck? How odd. Let's create a specific file in a specific directory: a LOCK-file. Now, "the db-queue is on hold and no request gets processed :)"

Golly gee thanks Alpha.

Anyhow, did you know that MySQL has a join limit of 61 tables?

I just deployed my first AWS Kubernetes cluster. Fuck yeah!

One of my colleagues who thinks he knows all about "big data": "I try to put everything in hadoop, that is my philosophy".

We don't even have a hadoop cluster.

Debugging your application on a mesosphere cluster for the first time.

Just got handed a dozen servers. Documentation shows a (Linux) database cluster is using ldap authentication. I try logging in with my creds. No joy. I look up the root password and log in.

Not only is it not configured to use ldap, it's also not clustered.

I need more coffee.

Setting up a single node Hadoop cluster. Then installing intellij idea, to find that it doesn't detect any installed jdk. Then uninstalled all jdks, and then reinstalled one, then Hadoop won't work. Now everything seems fine.

Got handed a CentOS 7 cluster, previous admin made kernel command line changes in grub.cfg instead of default/grub.

Ah, thank you.

Low budget companies be charging insane prices to clients and let the junior frontend dev setup a Azure cluster.. then wonder why it suddenly falls apart at the end of the project

Completely fucked up replication of MySQL servers.

Remote: 2 different database Servers
--> made sense.

Except the misconfiguration. Or better: No configuration at all.

So how to solve the massiv delays and make everything even more crazy?

2 remote servers - 2 readonly slaves for reading data remote (master - slave)

2 local (internal) servers.

Remote - Local Master Master.

Unfucking this cluster fuck was a real nightmare.

It had to be done at night, cause everything needed to be ripped apart.

And the servers were the backend of a warehouse with supply chain and multiple selling channels (Amazon, eBay etcetera).

So. It had to run the next day at 05.00 clock so the incoming orders could be packaged / prepared for shipping.

That was fun. Not.

And the clusterfuck died spectaculously on my first work day - the old DBA was gone (fired....)

:)

Hello devRant!
Man its been a while, i havent logged in here in like 4 years.

Recently ive been getting into home-labbing, and i thought to myself

"all of these people on youtube/reddit run Plex on pre-built NASs that have awful celerons and whatnot, we can do much better!"

And by "much better" i meant a bare metal k8s cluster.

My hybris knows no bounds apparently.

Turns out this shit is quite hard.
Really gives u an appreciation of just how much stuff cloud providers magically abstract away....

My final goal is to run stableDiffusion on this thing, even know i know full-well the moment i try Nvidia will fuck me raw with some hidden enterprise subscrition :)

Task: Deploy MinIO in k8s cluster
Me: deploys the first docker image found on google: bitnami/minio
MinIO: starts

Me: log in
MinIO: Fuck you! There's a cryptic error: Expected element type <Error> but have <HTML>

Me: spends half a day trying out different vendors, different versions, different environments (works on local BTW)

Me: got tired, restored the manifest to what it was at the beginning. Gave it the last try before signing off

MinIO: works 100%

wtf... So switching it back and forth fixed the problem, whatever it was. Oh well, yet another day.

I really want to build a raspberry pi cluster but I don't know what to do with it...
Has anyone of you ever built a raspi cluster?
What could I do with it?

Also, holy FUCKING SHIT. DEPLOYING A KUBERNETES CLUSTER JENKINS AND SONARQUBE EC2s, TAKES SO MUCH FUCKING RESOURCES AND COSTS SO FUCKING MUCH THAT THIS HURTS ME FUCKING PHYSICALLY.

HOW DO I PRACTICE WORKING DEVOPS WITH KUBERNETES ON AWS IF THE COSTS ARE FUCKING OUTRAGEOUS?????

I am so sick of this place. Production deployment shouldn't be such a massive cluster fuck of different departments and sometimes redundant configurations. Knowing who to talk to about a problem shouldn't change daily, and it shouldn't be hard to figure out.

My jvm doesn't appear to be running in production. I hunt down the fucker who is supposed to be handling this problem today...but he's out of office and no one knows who his stand in is...GODDAMNIT!

Posted previously about our codebase being a monolithic, poorly-written, pain-to-maintain gigantic cluster-fuck. And the efforts made to rewrite it.
Well, we made huge success the previous year in this regard. I rewrote the entire API while my other team mates worked on two different UI apps one of which is now in production and the other soon to be released in alpha.
Processes have being put in place for our team and are being improved.
We still have some technical debts though.
Dev goal for 2020,
- Pay most of the technical debt.
- Dive deeper into Flutter and finish the app I wanted.
- Play with ML, AI and Game dev.

Got my networking for cluster emulation using virtualbox right after 6 days.

Thanks @sbiewald

You know what I had to deal with
A bunch of these shit
try{
//Shitty cluster fuck excuse for java
//code
}catch(Exception e){

}

code reviewed a "senior" developers code (guy has been at the companyny since noah and his clay computer tablets), he replied with my comments against it stating he has more experience than i have and that i shouldnt do anything except just sign off the stuff to production, stuff goes to production and breaks a massive financial cluster fuck of a mess, who gets the blame for releasing the code to production ? Junior me !!! gotta love it when management needs scapegoats of FIFO people.

Let's see what's on the menu today:

* Web Application Catastrophe Special *

Includes, but not limited to:
- Orphaned server processes in the configuration management cluster
- Microservice back-end architecture with no API documentation
- Poorly implemented cache microservice with no documentation
- Stale data causing everything to be shown as down in production, despite everything running fine

Cost: 1 developer's sanity

The most difficult part about learning/working with new technology is the lack of online support and a really small community! So every time you're stuck with an error you got to open each and every configuration file to see which little value was throwing that page long error!!!!

The same happened with me while working with the new RedHat CEPH Storage while I was configuring my nodes using ceph-ansible. A new error pops up and it was like reaching milestones when I found the error halting up the execution of the playbook!

L1 support requested to terminate an EC2 instance on which one of our apps seemed to be misbehaving. The node was terminated after few min.

L1 later realized that that instance didn't belong to that app, but instead it was one of the RabbitMQ nodes.

Then, after some panicking we remembered that HA was enabled, so nothing should've been lost.

Later, we realized that the recent RMQ upgrade necessitated a new cluster on which HA was NOT enabled!

A few weeks back we ported our PHP Rest API into a couple of Go micro-services.

Incredibly _satisfying_ job.

Requests went from 20+ seconds to ~100-300ms.

There was still one bottle-neck, though, because we had to use most of the old cluster-fork of a database (because no way I'll be able to fix all that in a week).

And ooh, next we're thinking of switching to gRPC. Man, we have the best jobs.

I'm quite pleased to show off the initial rendering of my CaaB (Cluster as a Box)!

This is a project that I've put not enough time into, and them finally I decided to get a move on with it. The estimated cost of the whole device/s ranges somewhere around $600. It's gonna be fun!

P.S. Suggestions welcome, particularly if it's about a substitute for the ODroid HC1.

Monitoring tools madness: quest foglight.

So, setting a blackout for an FMS "HA cluster" (which does not work due to a bug infested custom jboss implementation) can bring the servers down... And no way to bring them back up.

This brilliant piece of enterprise APM software costs 600.000€ for a 5year license.

I,ve added more drama (logs, threaddumps, support bundles and screenshots) to the support portal...

45 cases now in total, oldest case still open date 2017...

Fuck you quest software

i found playing with arduino is never-ending fun. and then i tried Raspi and feeling the same. now i buy a bunch of raspi and trying to cluster them up, just for another random project.

yeah this is clearly nonsense. this is myself when my boredom gets high enough. No, vacation is not helping, trust me i've been camping in a jungle for 5 days and the next week i go to the beach. this is bad.

making things worse, My way of life is like any ordinary people (or maybe a developers) but turned upside down for example I program, learning for fun, and going vacation is just another medical need that i need to follow.

I am an university student, studying IT. I've been programming since 6, and geeking anything around technology and astronomy till now. this is my third semester and i don't get challanged enough for this kind of curricullum. i need some kind of project that even i don't think about any number of money anymore, but just enough to challange me up. just need to get rid my boredom.

sorry for ranting from a to z, this is just buzzing around my mind.

fuck.. FUCK FUCK FUCK!!!
I'mma fakin EXPLODE!

It was supposed to be a week, maybe two weeks long gig MAX. Now I'm on my 3rd (or 4th) week and still got plenty on my plate. I'm freaking STRESSED. Yelling at people for no reason, just because they interrupt my train of thought, raise a hand, walk by, breathe, stay quiet or simply are.

FUCK!

Pressure from all the fronts, and no time to rest. Sleeping 3-5 hours, falling asleep with this nonsense and breaking the day with it too.

And now I'm fucking FINALLY CLOSE, I can see the light at the end of the tunne<<<<<TTTOOOOOOOOOOOOOTTTTT>>>>>>>

All that was left was to finish up configuring a firewall and set up alerting. I got storage sorted out, customized a CSI provider to make it work across the cluster, raised, idk, a gazillion issues in GH in various repositories I depend on, practically debugged their issues and reported them.

Today I'm on firewall. Liason with the client is pressured by the client bcz I'm already overdue. He propagates that pressure on to me. I have work. I have family, I have this side gig. I have people nagging me to rest. I have other commitments (you know.. eating (I practically finish my meal in under 3 minutes; incl. the 2min in the µ-wave), shitting (I plan it ahead so I could google issues on my phone while there), etc.)

A fucking firewall was left... I configured it as it should be, and... the cluster stopped...clustering. inter-node comms stopped. `lsof` shows that for some reason nodes are accessing LAN IPs through their WAN NIC (go figure!!!) -- that's why they don't work!!

Sooo.. my colleagues suggest me to make it faster/quicker and more secure -- disable public IPs and use a private LB. I spent this whole day trying to implement it. I set up bastion hosts, managed to hack private SSH key into them upon setup, FINALLY managed to make ssh work and the user_data script to trigger, only to find out that...

~]# ping 1.1.1.1
ping: connect: Network is unreachable
~]#

... there's no nat.
THERE"S NO FUCKING NAT!!!

HOW CAN THERE BE NO NAT!?!?!????? MY HOME LAPTOP HAS A NAT, MY PHONE HAS A NAT, EVEN MY CAT HAS A MOTHER HUGGING NAT, AND THIS FUCKING INFRA HAS NO FUCKING NAT???????????????????????

ALready under loads of pressure, and the whole day is wasted. And now I'll be spending time to fucking UNDO everything I did today. Not try something new. But UNDO. And hour or more for just that...

I don't usually drink, but recently that bottom shelf bottle of Captain Morgan that smells and tastes like a bottle of medical spirit starts to feel very tempting.

Soo.. how's your day

## Learning k8s

Okay, that's kind of obvious, I just have no idea why I didn't think of it..

I've made a cluster out of a rpi, a i7 PC and a dell xps lappy. Lappy is a master and the other two are worker nodes.

I've noticed that the rpi tends to hardly ever run any of my pods. It's only got 3 of them assigned and neither of them work. They all say: "Back-off restarting failed container" as a sole message in pod's description and the log only says 'standard_init_linux.go:211: exec user process caused "exec format error"' - also the only entry.

Tried running the same image locally on the XPS, via docker run -- works flawlessly (apart from being detached from the cluster of other instances).

Tried to redeploy k8s.yaml -- still raspberry keeps failing.

wtf...

And then it came to me. Wait.. You idiot.. Now ssh to that rpi and run that container manually. Et voila! "docker: no matching manifest for linux/arm/v7 in the manifest list entries."

IDK whether it's lack of sleep or what, but I have missed the obvious -- while docker IS cross-platform, it's not a VM and it does not change the instructions' set supported by the node's cpu. Effectively meaning that the dockerized app is not guaranteed to work on any platform there is!

Shit. I'll have to assemble my own image I guess. It sucks, since I'll have to use CentOS, which is oh-so-heavy compared to Alpine :( Since one of the dependencies does not run well there..

Shit.

Learning k8s is sometimes so frustrating :)

I don't like when client decide which tech use in the project. I got some weird tech request like:
1. Move existing database from postgresql to Hadoop because hadoop is Big Data (is kinda move from amazon rds to amazon s3 just why? have you index, cluster your postgresql table?)
2. Move from mysql to postgresql because mysql cause deadlock (maybe their previous developer just fucking moron)
In this situation we just explain why we don't use that and propose alternative solution. If they insist with their solution either ignore it or decide not continuing the project.

Give me a second to get my todo list.

- get a office/server room built
- setup a home kubernetes cluster
- create an open-source ActivityPub whatsapp clone
- unify existing ActivityPub implementations where an account on one can be used on others
- finish dockerssh
- create an irc bridge for signal messenger.
- find a way to fully provision linux workstations fully unattended

Oh my.. I think I'm enjoying molesting kubernetes :)

A while ago I got pissed at k8s because with 1.24 they brought backward-incompatible changes, ruling my cluster broken. Then I thought to myself: "why not create a Docker image that would run kubernetes inside? Separate images for control plane, agent and client"

Took me a while, but I think tonight I've had a breakthrough (I love how linux works...)!! The control-plane is spinning up!! Running on containerd

Still needs some work and polishing, but hey! Ephemeral k8s installation with a single docker-run command sure sounds tempting!

P.S. Yes, I know there is `kind` and 'kinder', but I'm reluctant to install a separate tool that installs a set of tools for me. Kind of... too shady. Too many moving parts. Too deeply hidden parts I may have to fix. Having a dumb-simple Dockerfile gives me the openness, flexibility and simplicity I want. + I can always use it as a base image to add my customizations later on! Reinstalling a cluster would be a breeeeeeze

kubectl scale tears --replicas=infinity

Finally set up my 5-node Raspberry Pi Kubernetes cluster.

They say that runing the same command over and over again is a sign of insanity.

LIKE HELL IT IS!!!

I've been running `terraform apply` for the last hour (trying to dump an EKS token in plain-text, because my k8s-related providers failed to auth to the cluster), and miraculously the problem went away. Now the error is no more.

Insanity?

I beg to differ!

Narf!

Screw DataStage
Screw Saleforce
Screw the $25k/week Vendor who finishes work maybe 60% complete, if I’m being generous.
Screw the business rushing to us to fix their shadow IT cluster.

Where can i get good second hand servers, am planning to homelab cluster. Will appreciate any pointers ...

Just succesfully setup database cluster , now I need balancer!

Showing off this PC cluster rendering these gigantic 3D frames, a customer shakes his head: "Why is this so slow?" I asked why he thought it was slow and he said, "I have an old Windows laptop but if I double click on an image on the desktop, it opens immediately!"

In three days my team closes the book on this monolithic cluster of a project that I thought would never end! I feel like Christmas is coming. Hopefully the PM doesn’t let the business throw us into another scope-less pit like this again! 😩

"shiny shiny" time... Lets standup a kubernetes cluster and then find a problem to solve! #BuzzWordDrivenDev

I've never been a big fan of the "Cloud hype".

Take today for example. What decent persistent storage options do I have for my EKS cluster?

- EBS -- does not support ReadWriteMany, meaning all the pods mounting that volume will have to be physically running on the same server. No HA, no HP. Bummer

- EFS -- expensive. On top of that, its performance is utter shit. Sure, I could buy more IOPS, but then again.. even more expensive.

S3 -- half-assed filesystem. Does not support O_APPEND, so basically any file modifications will have to be in a
`createFile(file+"_new", readAll(file) + new_data); removeFile(file); renameFile(file + "_new", file);`
way.
ON TOP of that, the s3 CSI has even more limitations, limiting my ability to cross-mount volumes across different applications (permission issues)

I'm running out of options. And this does not help my distrust in cloud infras...

I hope my boss learned his lesson: dd if=/dev/zero of=[hdd storing DB about VM cluster]
- is a very very bad idea...

## Learning k8s

Sooo yeah, 2 days have been wasted only because I did not reset my cluster correctly the first time. Prolly some iptables rules were left that prevented me from using DNS. Nothing worked...

2 fucking days..

2 FUCKING DAYS!!! F!!!

Having a Node.js app ready for production is so satisfying with PM2.