Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "cluster@home"
-
!rant
So last weekend I started collecting hardware for a small scale cluster at home to test scalability of my software. Making some decent progress.
Tomorrow I will replace the switch and this weekend I will set up storage so I can start my first application20 -
Inception.
Today I needed to check something in a remote server: this was the easiest way:
1: teamviewer to my home pc from university
2: started a vm on that machine with vpn connection to my work office
3: rdp to a windows server vm
4: ssh to a vm on our hosting cluster
5: from there, ssh to the server that I needed access to7 -
Want to make someone's life a misery? Here's how.
Don't base your tech stack on any prior knowledge or what's relevant to the problem.
Instead design it around all the latest trends and badges you want to put on your resume because they're frequent key words on job postings.
Once your data goes in, you'll never get it out again. At best you'll be teased with little crumbs of data but never the whole.
I know, here's a genius idea, instead of putting data into a normal data base then using a cache, lets put it all into the cache and by the way it's a volatile cache.
Here's an idea. For something as simple as a single log lets make it use a queue that goes into a queue that goes into another queue that goes into another queue all of which are black boxes. No rhyme of reason, queues are all the rage.
Have you tried: Lets use a new fangled tangle, trust me it's safe, INSERT BIG NAME HERE uses it.
Finally it all gets flushed down into this subterranean cunt of a sewerage system and good luck getting it all out again. It's like hell except it's all shitty instead of all fiery.
All I want is to export one table, a simple log table with a few GB to CSV or heck whatever generic format it supports, that's it.
So I run the export table to file command and off it goes only less than a minute later for timeout commands to start piling up until it aborts. WTF. So then I set the most obvious timeout setting in the client, no change, then another timeout setting on the client, no change, then i try to put it in the client configuration file, no change, then I set the timeout on the export query, no change, then finally I bump the timeouts in the server config, no change, then I find someone has downloaded it from both tucows and apt, but they're using the tucows version so its real config is in /dev/database.xml (don't even ask). I increase that from seconds to a minute, it's still timing out after a minute.
In the end I have to make my own and this involves working out how to parse non-standard binary formatted data structures. It's the umpteenth time I have had to do this.
These aren't some no name solutions and it really terrifies me. All this is doing is taking some access logs, store them in one place then index by timestamp. These things are all meant to be blazing fast but grep is often faster. How the hell is such a trivial thing turned into a series of one nightmare after another? Things that should take a few minutes take days of screwing around. I don't have access logs any more because I can't access them anymore.
The terror of this isn't that it's so awful, it's that all the little kiddies doing all this jazz for the first time and using all these shit wipe buzzword driven approaches have no fucking clue it's not meant to be this difficult. I'm replacing entire tens of thousands to million line enterprise systems with a few hundred lines of code that's faster, more reliable and better in virtually every measurable way time and time again.
This is constant. It's not one offender, it's not one project, it's not one company, it's not one developer, it's the industry standard. It's all over open source software and all over dev shops. Everything is exponentially becoming more bloated and difficult than it needs to be. I'm seeing people pull up a hundred cloud instances for things that'll be happy at home with a few minutes to a week's optimisation efforts. Queries that are N*N and only take a few minutes to turn to LOG(N) but instead people renting out a fucking off huge ass SQL cluster instead that not only costs gobs of money but takes a ton of time maintaining and configuring which isn't going to be done right either.
I think most people are bullshitting when they say they have impostor syndrome but when the trend in technology is to make every fucking little trivial thing a thousand times more complex than it has to be I can see how they'd feel that way. There's so bloody much you need to do that you don't need to do these days that you either can't get anything done right or the smallest thing takes an age.
I have no idea why some people put up with some of these appliances. If you bought a dish washer that made washing dishes even harder than it was before you'd return it to the store.
Every time I see the terms enterprise, fast, big data, scalable, cloud or anything of the like I bang my head on the table. One of these days I'm going to lose my fucking tits.10 -
I kept piling on hard drives at home and you know... One day I'll setup some redundancy.
Then a 5 months old full 4TB drive gave up and I lost that data.
After that I've upgraded to having a ceph cluster storing everything.
At the beginning of this year one hard drive in the cluster gave up. I didn't notice until I wondered why available storage was low. Cluster had already rebalance itself and were running flawless.2 -
This is the third part of my ongoing series "The Ballad of the Six Witchers and the Undocumented Java Tool".
In this part, we have the massive Battle of Sparks and Storms.
The first part is here: https://devrant.com/rants/5009817/...
The second part is here: https://devrant.com/rants/5054467/...
Over the last couple sprints and then some, The Witcher Who Writes and the Butchers of Jarfile had studied the decompiled guts of the Undocumented Java Beast and finally derived (most of) the process by which the data was transformed. They even built a model to replicate the results in small scale.
But when such process was presented to the Priests of Accounting at the Temple of Cash-Flow, chaos ensued.
This cannot be! - cried the priests - You must be wrong!
Wrong, the Witchers were not. In every single test case the Priests of Accounting threw at the Witchers, their model predicted perfectly what would be registered by the Undocumented Java Tool at the very end.
It was not the Witchers. The process was corrupted at its essence.
The Witchers reconvened at their fortress of Sprint. In the dark room of Standup, the leader of their order, wise beyond his years (and there were plenty of those), in a deep and solemn voice, there declared:
"Guys, we must not fuck this up." (actual quote)
For the leader of the witchers had just returned from a war council at the capitol of the province. There, heading a table boarding the Archpriest of Accounting, the Augur of Economics, the Marketing Spymaster and Admiral of the Fleet, was the Ciefoh Seat himself.
They had heard rumors about the Order of the Witchers' battles and operations. They wanted to know more.
It was quiet that night in the flat and cloudy plains of Cluster of Sparks and Storms. The Ciefoh Seat had ordered the thunder to stay silent, so that the forces of whole cluster would be available for the Witchers.
The cluster had solid ground for Hive and Parquet turf, and extended from the Connection River to farther than the horizon.
The Witcher Who Writes, seated high atop his war-elephant, looked at the massive battle formations behind.
The frontline were all war-elephants of Hadoop, their mahouts the Witchers themselves.
For the right flank, the Red Port of Redis had sent their best connectors - currency conversions would happen by the hundreds, instantly and always updated.
The left flank had the first and second army of Coroutine Jugglers, trained by the Witchers. Their swift catapults would be able to move data to and from the JIRA cities. No data point will be left behind.
At the center were thousands of Sparks mounting their RDD warhorses. Organized in formations designed by the Witchers and the Priestesses of Accounting, those armoured and strong units were native to this cloudy landscape. This was their home, and they were ready to defend it.
For the enemy could be seen in the horizon.
There were terabytes of data crossing the Stony Event Bridge. Hundreds of millions of datapoints, eager to flood the memory of every system and devour the processing time of every node on sight.
For the Ciefoh Seat, in his fury about the wrong calculations of the processes of the past, had ruled that the Witchers would not simply reshape the data from now on.
The Witchers were to process the entire historical ledger of transactions. And be done before the end of the month.
The metrics rumbled under the weight of terabytes of data crossing the Event Bridge. With fire in their eyes, the war-elephants in the frontline advanced.
Hundreds of data points would be impaled by their tusks and trampled by their feet, pressed into the parquet and hive grounds. But hundreds more would take their place. There were too many data points for the Hadoop war-elephants alone.
But the dawn will come.
When the night seemed darker, the Witchers heard a thunder, and the skies turned red. The Sparks were on the move.
Riding into the parquet and hive turf, impaling scores of data points with their long SIMD lances and chopping data off with their Scala swords, the Sparks burned through the enemy like fire.
The second line of the sparks would pick data off to be sent by the Coroutine Jugglers to JIRA. That would provoke even more data to cross the Event Bridge, but the third line of Sparks were ready for it - those data would be pierced by the rounds provided by the Red Port of Redis, and sent back to JIRA - for good.
They fought for six days and six nights, taking turns so that the battles would not stop. And then, silence. The day was won, all the data crushed into hive and parquet.
Short-lived was the relief. The Witchers knew that the enemy in combat is but a shadow of the troubles that approach. Politics and greed and grudge are all next in line. Are the Witchers heroes or marauders? The aftermath is to come, and I will keep you posted.4 -
Here it goes,
So there I was a Linux enthusiast stuck in a windows job for about 3 years. I would spend my weekends doing Linux related tasks for my personal amusement, while I spent my week doing windows maintenance and development (partially) professionally.
It was about 2014 I started building an openstack cluster at home and i was so stoked! I searched for openstack summits or meetups and for my surprise there was an openstack meetup in my town. Holly 🐄 I said.
The date of the event came and I left work earlier to attend the meetup.
There , I had a talk with the meetup organizer/speaker and he told he was interested in what I was doing and that they were going to open a job in the next months.
A few months later still at my boring job I got an email from him for an interview.
Everything went just about right...and here I am a Linux systems engineer doing everything I love for a living... -
Hello devRant!
Man its been a while, i havent logged in here in like 4 years.
Recently ive been getting into home-labbing, and i thought to myself
"all of these people on youtube/reddit run Plex on pre-built NASs that have awful celerons and whatnot, we can do much better!"
And by "much better" i meant a bare metal k8s cluster.
My hybris knows no bounds apparently.
Turns out this shit is quite hard.
Really gives u an appreciation of just how much stuff cloud providers magically abstract away....
My final goal is to run stableDiffusion on this thing, even know i know full-well the moment i try Nvidia will fuck me raw with some hidden enterprise subscrition :) -
Give me a second to get my todo list.
- get a office/server room built
- setup a home kubernetes cluster
- create an open-source ActivityPub whatsapp clone
- unify existing ActivityPub implementations where an account on one can be used on others
- finish dockerssh
- create an irc bridge for signal messenger.
- find a way to fully provision linux workstations fully unattended2 -
fuck.. FUCK FUCK FUCK!!!
I'mma fakin EXPLODE!
It was supposed to be a week, maybe two weeks long gig MAX. Now I'm on my 3rd (or 4th) week and still got plenty on my plate. I'm freaking STRESSED. Yelling at people for no reason, just because they interrupt my train of thought, raise a hand, walk by, breathe, stay quiet or simply are.
FUCK!
Pressure from all the fronts, and no time to rest. Sleeping 3-5 hours, falling asleep with this nonsense and breaking the day with it too.
And now I'm fucking FINALLY CLOSE, I can see the light at the end of the tunne<<<<<TTTOOOOOOOOOOOOOTTTTT>>>>>>>
All that was left was to finish up configuring a firewall and set up alerting. I got storage sorted out, customized a CSI provider to make it work across the cluster, raised, idk, a gazillion issues in GH in various repositories I depend on, practically debugged their issues and reported them.
Today I'm on firewall. Liason with the client is pressured by the client bcz I'm already overdue. He propagates that pressure on to me. I have work. I have family, I have this side gig. I have people nagging me to rest. I have other commitments (you know.. eating (I practically finish my meal in under 3 minutes; incl. the 2min in the µ-wave), shitting (I plan it ahead so I could google issues on my phone while there), etc.)
A fucking firewall was left... I configured it as it should be, and... the cluster stopped...clustering. inter-node comms stopped. `lsof` shows that for some reason nodes are accessing LAN IPs through their WAN NIC (go figure!!!) -- that's why they don't work!!
Sooo.. my colleagues suggest me to make it faster/quicker and more secure -- disable public IPs and use a private LB. I spent this whole day trying to implement it. I set up bastion hosts, managed to hack private SSH key into them upon setup, FINALLY managed to make ssh work and the user_data script to trigger, only to find out that...
~]# ping 1.1.1.1
ping: connect: Network is unreachable
~]#
... there's no nat.
THERE"S NO FUCKING NAT!!!
HOW CAN THERE BE NO NAT!?!?!????? MY HOME LAPTOP HAS A NAT, MY PHONE HAS A NAT, EVEN MY CAT HAS A MOTHER HUGGING NAT, AND THIS FUCKING INFRA HAS NO FUCKING NAT???????????????????????
ALready under loads of pressure, and the whole day is wasted. And now I'll be spending time to fucking UNDO everything I did today. Not try something new. But UNDO. And hour or more for just that...
I don't usually drink, but recently that bottom shelf bottle of Captain Morgan that smells and tastes like a bottle of medical spirit starts to feel very tempting.
Soo.. how's your dayrant overdue tired no nat hcloud why there's no nat???? fuck frustrated waiting for concrete to settle angry hetzner need an outlet2 -
## Learning k8s
Interesting. So sometimes k8s network goes down. Apparently it's a pitfall that has been logged with vendor but not yet fixed. If on either of the nodes networking service is restarted (i.e. you connect to VPN, plug in an USB wifi dongle, etc..) -- you will lose the flannel.1 interface. As a result you will NOT be able to use kube-dns (because it's unreachable) not will you access ClusterIPs on other nodes. Deleting flannel and allowing it to restart on control place brings it back to operational.
And yet another note.. If you're making a k8s cluster at home and you are planning to control it via your lappy -- DO NOT set up control plane on your lappy :) If you are away from home you'll have a hard time connecting back to your cluster.
A raspberry pi ir perfectly enough for a control place. And when you are away with your lappy, ssh'ing home and setting up a few iptables DNATs will do the trick
netikras@netikras-xps:~/skriptai/bin$ cat fw_kubeadm
#!/bin/bash
FW_LOCAL_IP=127.0.0.15
FW_PORT=6443
FW_PORT_INTERMED=16443
MASTER_IP=192.168.1.15
MASTER_USER=pi
FW_RULE="OUTPUT -d ${MASTER_IP} -p tcp -j DNAT --to-destination ${FW_LOCAL_IP}"
sudo iptables -t nat -A ${FW_RULE}
ssh home -p 4522 -l netikras -tt \
-L ${FW_LOCAL_IP}:${FW_PORT}:${FW_LOCAL_IP}:${FW_PORT_INTERMED} \
ssh ${MASTER_IP} -l ${MASTER_USER} -tt \
-L ${FW_LOCAL_IP}:${FW_PORT_INTERMED}:${FW_LOCAL_IP}:${FW_PORT} \
/bin/bash
# 'echo "Tunnel is open. Disconnect from this SSH session to close the tunnel and remove NAT rules" ; bash'
sudo iptables -t nat -D ${FW_RULE}
And ofc copy control plane's ~/.kube to your lappy :)3 -
My final year taking a B.Sc. I'm writing up my Distributed Systems project, the day before handing it in. It's on top of Transis, and source code is "stored" in RCS (yes, I'm that old). The project is a reliable system administration tool, that performs the same action across a cluster with guaranteed semantics.
I'm very proud of the semantics, but cannot figure out why the subdirectory installation stuff works almost but not quite. Here's my sequence of actions:
1. Install across all machines.
2. Manually see it's broken.
3. "rm -rf *".
4. Repeat.
What in to discover is that the subdirectory installation always finishes off in a current directory 1 level higher than where it started. Oh, and the entire cluster sees my NFS home directory. Oh, and I'm running each cluster member in a deep subdirectory of my dev directory. Oh, and my RCS files live in a subdirectory of my dev directory.
All of a sudden, my 5 concurrent "rm -rf *"s were printing weird error messages about ENOENT and not being able to find some inodes. In a belated flash of brilliance, I figure out all the above, and also that I've just deleted my dev directory. 5 times, concurrently. And the RCS files.
That was the day a kindly sysadmin taught me than NetApps have these .snapshot directories. -
This is probably the worst place to start my Rant saga but this is recent (this is one of the last few episodes of a 3 series cluster fuck of a job so you're missing out on all the straws that go into breaking the camels back and making him unaccommodating)
TL;DR I do good work, management dont like me and go out their way to try and fuck up my days
So, lets start, I'm a contractor, got funeral Tuesday, book leave, book WFH for day after.
I leave in 3 weeks, woman who is the CIO's right hand bitch takes me into a room the next day or so in the morning to discuss my WFH day. Leave on tuesday is cool but this WFH day...there's only so long until I'm gone so they want me to stay in for more face-to-face time blah blah blah (considering this woman isn't even part of the project I'm working on anymore because she decided to deflect it onto a underqualified junior with no PM experience)
So I sit there, thinking of all the blood and sweat that I have shed, the mountains I've moved just to be told to move the mountain somewhere else and whether coming in would kill me (in other words im fucking burnt out!!! I have built their GDPR database and app backend single-handedly with no requirements, project managers who can't plan and being chastised for asking for documentation/plan/anything written down and having the CIO who is also the fucking DPO ignore any emails/slack I send him relating to the project and having to keep up with a team of devs....).
So because there was a momentary silence, she decided to fill the gap
"Oh, you've done some good work so far and I wouldn't want you to ruin it all in these last 3 weeks. So just come in on the Wednesday so that we can have you here."
Hmm....yeah...i didn't notice what she had ACTUALLY said there, still thinking about can i be fucked? So she decides to add
"...there's only 3 weeks left, wouldn't want you to burn any bridges. Remember, we still have to give you a reference"
....Okay....shots fired. So i respond
"You saying, if I take a WFH day, you'll give me a bad reference?"
"Noooo no no no, not saying that, just that you've done good work and we wouldn't want you to ruin it"
"With one wfh day?"
"We just want you to come in because the developers might be coming here that week"
"Oh... I hear that...what day?"
"I dunno, it's not been booked yet"
".............................I'll think about it"
"There's nothing to consider"
*Start leaving room* "I'll think about it...."
So cool, obviously, had a think, decide to shoot over an email (or more accurately, a collection of bullets). Which basically said, in devRant translation, "Fuck y'all, I'm WFH on that day, I wish a motherfucker would fuck up my reference, we can go that way if you want it. *snaps fingers* I. WISH. YOU. WOULD! "
Woman says "I wasn't threatening you, was just saying...dont ruin your last 3 weeks, wouldn't want you to burn any bridges and that we still have to give you a reference"
What kind of Godfather comment is that?
Come in today, the CIO, who is a prick who don't like me for whatever reason, sends me long email trying to disrespect me and in the midst says "I’m sorry that you have chosen to react like this, I’m sure that [my bitch] was conveying a position that your last three weeks of contract are crucial for a smooth handover. I have made the decision to not require you to work from home on Wednesday. I understand you are on leave on Tuesday and therefore this is now extended to include Wednesday. I look forward to seeing you back in the office on Thursday. I hope this will make the situation better for all parties."
.................................thought you lot needed me in the office to ensure a smooth handover................logic..........people.............where the fuck do you get yours from!?!?!?!? All this just so they can say "We made the decision at the end :cool:" -
Bought two hp z230 and one hp z210 to setup as a kubernetes cluster at home.
The first two worked as expected to install Ubuntu 18.04 but the z210 just fails installation just at the end of.
I've updated the bios, I've tried different hard drive, (obvious I've turned off secure boot), I've downgraded the bios, I've cursed, spoken harch language at it and sprinkled it with holy water, still it fails.
A Google search the problem, one hit similar to my problem but it did not help me.
Currently I'm on my 5:the glass of wine, if not solved tomorrow I'm hiding it at work until the next "downsizing" and it will have an accedent from the 9:the floor.
I've spent 150$ on it but I have the economy to nurture my mental health... Not all the time but this time it feels worth it!!!3