I'm a DevOps engineer. It's my job to understand why this type of shit is broken, and when I finally figure it out, I get so mad at bullish players like AWS.

It's simple. Install Python3 from apt.

`apt-get update && apt-get install -y python3-dev`

I've done this thousands of times, and it just works.

Docker? Yup.
AWS AMI? Yup.
Automation? Nope.

WTF? Let's waste 2.5 hours and figure out why this morning.

In docker: `apt-cache policy python3-dev` shows us:
python3-dev:
http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

But in AWS instance, we see we're reading from "http://us-east-1.ec2.archive.ubuntu.com/... focal/main" instead!

Ah, but why does it fail? AWS is just using a mirror, right? Not quite.

When the automation script is running, it's beating AWS to the apt mirror update! My instance, running on AWS is trying to access the same archive.ubuntu.com that the Docker container tried to use. "python3-dev" was not a candidate for installation! WTF Amazon? Shouldn't that just work, even if I'm not using your mirror?

So I try again, and again, and again. It works, on average, 1 out of every 5 times. I'm assuming this means we're seeing some strange shit configuration between EC2 racks where some are configured to redirect archive.ubuntu.com to the ec2 mirror, and others are configured to block. I haven't dug this far into the issue yet, because by the time I can SSH into the machine after automation, the apt list has already received it's blessed update from EC2.

Now I have to build a graceful delay into my automation while I wait for AWS to mangle, I mean "fix up" my apt sources list to their whim.

After completely blowing my allotted time on this task, I just shipped a "sleep" statement in my code. I feel so dirty. I'm going to go brew some more coffee to be okay with my life. Then figure out a proper wait statement.

EoS1: This is the continuation of my previous rant, "The Ballad of The Six Witchers and The Undocumented Java Tool". Catch the first part here: https://devrant.com/rants/5009817/...

The Undocumented Java Tool, created by Those Who Came Before to fight the great battles of the past, is a swift beast. It reaches systems unknown and impacts many processes, unbeknownst even to said processes' masters. All from within it's lair, a foggy Windows Server swamp of moldy data streams and boggy flows.

One of The Six Witchers, the Wild One, scouted ahead to map the input and output data streams of the Unmapped Data Swamp. Accompanied only by his animal familiars, NetCat and WireShark.

Two others, bold and adventurous, raised their decompiling blades against the Undocumented Java Tool beast itself, to uncover it's data processing secrets.

Another of the witchers, of dark complexion and smooth speak, followed the data upstream to find where the fuck the limited excel sheets that feeds The Beast comes from, since it's handlers only know that "every other day a new one appears on this shared active directory location". WTF do people often have NPC-levels of unawareness about their own fucking jobs?!?!

The other witchers left to tend to the Burn-Rate Bonfire, for The Sprint is dark and full of terrors, and some bigwigs always manage to shoehorn their whims/unrelated stories into a otherwise lean sprint.

At the dawn of the new year, the witchers reconvened. "The Beast breathes a currency conversion API" - said The Wild One - "And it's claws and fangs strike mostly at two independent JIRA clusters, sometimes upserting issues. It uses a company-deprecated API to send emails. We're in deep shit."

"I've found The Source of Fucking Excel Sheets" - said the smooth witcher - "It is The Temple of Cash-Flow, where the priests weave the Tapestry of Transactions. Our Fucking Excel Sheets are but a snapshot of the latest updates on the balance of some billing accounts. I spoke with one of the priestesses, and she told me that The Oracle (DB) would be able to provide us with The Data directly, if we were to learn the way of the ODBC and the Query"

"We stroke at the beast" - said the bold and adventurous witchers, now deserving of the bragging rights to be called The Butchers of Jarfile - "It is actually fewer than twenty classes and modules. Most are API-drivers. And less than 40% of the code is ever even fucking used! We found fucking JIRA API tokens and URIs hard-coded. And it is all synchronous and monolithic - no wonder it takes almost 20 hours to run a single fucking excel sheet".

Together, the witchers figured out that each new billing account were morphed by The Beast into a new JIRA issue, if none was open yet for it. Transactions were used to update the outstanding balance on the issues regarding the billing accounts. The currency conversion API was used too often, and it's purpose was only to give a rough estimate of the total balance in each Jira issue in USD, since each issue could have transactions in several currencies. The Beast would consume the Excel sheet, do some cryptic transformations on it, and for each resulting line access the currency API and upsert a JIRA issue. The secrets of those transformations were still hidden from the witchers. When and why would The Beast send emails, was still a mistery.

As the Witchers Council approached an end and all were armed with knowledge and information, they decided on the next steps.
The Wild Witcher, known in every tavern in the land and by the sea, would create a connector to The Red Port of Redis, where every currency conversion is already updated by other processes and can be quickly retrieved inside the VPC. The Greenhorn Witcher is to follow him and build an offline process to update balances in JIRA issues.
The Butchers of Jarfile were to build The Juggler, an automation that should be able to receive a parquet file with an insertion plan and asynchronously update the JIRA API with scores of concurrent requests.
The Smooth Witcher, proud of his new lead, was to build The Oracle Watch, an order that would guard the Oracle (DB) at the Temple of Cash-Flow and report every qualifying transaction to parquet files in AWS S3. The Data would then be pushed to cross The Event Bridge into The Cluster of Sparks and Storms.
This Witcher Who Writes is to ride the Elephant of Hadoop into The Cluster of Sparks an Storms, to weave the signs of Map and Reduce and with speed and precision transform The Data into The Insertion Plan.
However, how exactly is The Data to be transformed is not yet known.

Will the Witchers be able to build The Data's New Path? Will they figure out the mysterious transformation? Will they discover the Undocumented Java Tool's secrets on notifying customers and aggregating data?

This story is still afoot. Only the future will tell, and I will keep you posted.

FUCK YOU GITKRAKEN
After all the suggestions in https://devrant.com/rants/1540091 I decided to give Gitkraken a try.

Here's the shitty experience you can expect:
1) It doesn't even ask you where to install it. Turns out, it spontaneously installs itself in "%LOCALAPPDATA%\gitkraken" - who the fuck installs software there??

2) It is "seamlessly integrated with GitLab", except the first time you open it you can only log in with your GitKraken or GitHub account, and NOT with a GitHub one. Just brilliant

3) After logging in, it spontaneously changes your global git username and email config, because fuck you that's why

4) If you have a repo on AWS CodeCommit with an remote that looks like "ssh://git-codecommit.us-east-2.amazonaws.com/...", *after the first push* it will spontaneously change it to "<user>@git-codecommit.us-east-2.amazonaws.com/bla/bla", causing future actions to fail. Because FUCK YOU, THAT'S WHY.

And they expect people to pay for this shit, just to be able to manage more than one account at a time (and some "additional features" that are not even listed on the site)?
FUCK OFF, AND FUCK YOU FOR WASTING MY FUCKING TIME, HOW ABOUT I CHANGE YOUR FUCKING SETTINGS TO FUCK YOU

https://techcrunch.com/2020/06/...

Yet another tool to "empower" management into thinking they are able to do in days what takes engineers years to accomplish.

All this is going to do is create technical debt for developers to consume when management is promoted for a "job well done".

Fuck.

So what's with the whole Elastic open source licenses thing. Seems like a spat btwn then and Amazon?

https://elastic.co/blog/...

Amazon now argues that doing this means Elasticsearch and Kibana will no longer be open sourced and that the Elastic License limits how the code can be used while the Server Side Public License makes it unacceptable to the open-source community.

!rant !dev

So, following up my last rant.
https://devrant.com/rants/2433162

I quit on Friday, this is what I said to my bosses.

"In the last week I had, 2 panic attacks, and I have 2 theories for this, one is that I have underlying psychological problems, the other theory is that we are under an impossible task, I choose to say now that I have to quit because I have psychological issues, but if you are willing to hear my other theory, that involves saying that meeting the deadline is not viable, then I can tell you that, so do want to listen that part?.

Bosses: No, we heard enough, we are going to have your contract terminated in order, and we will let you know when you can come and pick your paycheck."

So, that's them. Now about me and how I re-discovered GTD, or more precisely how I organized my whole weekend using taskwarrior with GTD, and why I think is going to be useful as a freelancer.

Before I feel good about telling you about my weekend I have to tell you a few things about myself.

I am a very impulsive person, I have a lot of energy in short surges, so I have to be able to maximize my activity when I'm in a surge, and I have to maximize my rest when I am not.

That's hard to do, it requires a balanced lifestyle, I am also very prone to being neurotic, and overwhelmed by the amount of stuff that I want to do.

And on top of that, when I am resting, I have surges of things that I want to have, do, or implement, it could be software related, as "Doing an app that will be the Uber of home services", to house improvements like, "I have to fix that leaking roof", and all the sort of stuff that happens in between hardware and software. That surge of consciousness doesn't allow me to have the proper rest that I need before I engage with activities again.

Because of this I have a very cyclic rhythm, with whole weeks burning my energy into doing stuff, and weeks resting doing very little and thinking too much.

Now about my weekend. Friday night I was browsing the web, and a thought came to my head. "The way you use your terminal, says a lot about your personality", and I got curious, so I searched for, "Show me your terminal", and found a post in dev.to to see all kind of nice terminal setups, from the very minimalist to very feature rich oh-my-zsh themes with plugins for git, aws and what not. One of these pictures really got my attention, a guy had set up his terminal to show him, how many task has he done in the day, and how many cups of coffee has he had.

So by investigating how he set up his terminal to show in the prompt the number of successfully completed tasks in the day, I found out that he was using taskwarrior, he was also kind enough to share the source code of his prompt setup, which I bookmarked to later incorporate that into my oh-my-zsh config.

After reading about taskwarrior, I also got a reference to GTD, I don't remember if this was one of those thoughts that I have and follow immediately, or if I read something that led me to a YouTube video summarizing GTD.

In the end, after watching that GTD video, I decided to give it a try to organize my life, and help me find a remote job, keep my house in order, plan my social activities as "hang out with friends", "visit mom and dad", and give the proper amount of attention to my GF, with whom I am deeply in love, and willing to spend the remaining of my years with her.

So my fist task was.

task add Ask for GF's parents blessing.

Which of course I have no intention of doing right now, but is one of the things that I will eventually have to do.

Then it started, I started adding tasks, and things to do, and go through the whole Capture phase of GTD.

Now it is a good time to write a small summary of what I think GTD is.

GTD is a life habit of organizing your life in todo-lists. And it was a very specific core method, that in the video summary that I watched was called CPR.

Capture, Process and Review.

Capture:
When you capture you just add your tasks to a bucket list.
So I took a notebook and started writing down everything that I wanted to have done. I also started to capture ideas as they came up to me, I did this by writing a telegram saved message in my phone, or directly adding it as a task in TW.

Process:
I read my telegram messages and put them into my task warrior list, then I started to organize my tasks into projects, breaking down every task that was not an atomic unit.
* And different projects started to emerge from this. One of them was project:Housekeeping.

And here's my screenshot of what I did this weekend, also the number of projects that I have, and all the things that I have to do in order to have what I think would be a very balanced, fun, and productive life.

You'll be able to see in the screenshot, that there's a blocked task, yes, tw allows you to organize dependencies too, so one task is delegated, and blocked by the delegation task.

// Rant 1
---
Im literally laughing and crying rn

I tried to deploy a backend on aws Fargate for the first time. Never used Fargate until now

After several days of brainwreck of trial and error

After Fucking around to find out

After Multiple failures to deploy the backend app on AWS Fargate

After Multiple times of deleting the whole infrastructure and redoing everything again

After trying to create the infrastructure through terraform, where 60% of it has worked but the remaining parts have failed

After then scraping off terraform and doing everything manually via AWS ui dashboard because im that much desperate now and just want to see my fucking backend work on aws and i dont care how it will be done anymore

I have finally deployed the backend, successfully

I am yet unsure of what the fuck is going on. I followed an article. Basically i deployed the backend using:

- RDS
- ECS
- ECR
- VPC
- ALB

You may wonder am i fucking retarded to fail this hard for just deploying a backend to aws?

No. Its much deeper than you think. I deployed it on a real world production ready app way.

- VPC with 2 public and 2 private subnets. Private subnets used only for RDS. Public for ALB.

- Everything is very well done and secure. 3 security groups: 1 for ALB (port 80), 1 for Fargate (port 8080, the one the backend is running on), 1 for RDS postgres (port 5432). Each one stacked on top and chained

- custom domain name + SSL certificate so i can have a clean version of the fully working backend such as https://api.shitstain.com

- custom ECS cluster

- custom target groups

- task definitions

Etc.

Right now im unsure how all of this is glued together. I have no idea why this works and why my backend is secure and reachable. Well i do know to some extent but not everything.

To know everything, I'll now ask some dumbass questions:

1. What is ECS used for?

2. What is a task definition and why do i need it?

3. What does Fargate do exactly? As far as i understood its a on-demand use of a backend. Almost like serverless backend? Like i get billed only when the backend is used by someone?

4. What is a target group and why do i need it?

5. Ive read somewhere theres a difference between using Fargate and... ECS (or is it something else)? Whats the difference?

Everything else i understand well enough.

In the meantime I'll now start analyzing researching and understanding deeply what happened here and why this works. I'll also turn all of this in terraform. I'll also build a custom gitlab CI/CD to automate all of this shit and deploy to fargate prod app

// Rant 2
---
Im pissing and shitting a lot today. I piss so much and i only drink coffee. But the bigger problem is i can barely manage to hold my piss. It feels like i need to piss asap or im gonna piss myself. I used to be able to easily hold it for hours now i can barely do it for seconds. While i was sleeping with my gf @retoor i woke up by pissing on myself on her bed right next to her! the heavy warmness of my piss woke me up. It was so embarrassing. But she was hardcore sleeping and didnt notice. I immediately got out of bed to take a shower like a walking dead. I thought i was dreaming. I was half conscious and could barely see only to find out it wasnt a dream and i really did piss on myself in her bed! What the fuck! Whats next, to uncontrollably shit on her bed while sleeping?! Hopefully i didnt get some infection. I feel healthy. But maybe all of this is one giant dream im having and all of u are not real