502 Bad Gateway

FUCK this startup mentality of implementing all these external services and APIs for absolutely fucking everything.

I get that your vacuous fresh-mint-tea-soaked hipster brains are all cheering about these "only $10/month/seat" services, because you imbeciles with your nodejs-sticker-plastered macbooks have never done anything but knot the work of other dimwits together.

I don't even care about the subscription costs. That shit is more trouble to maintain than writing it yourself, and there's no guarantee that visualizemyballs.com & lintmycock.io still work tomorrow.

I'm getting so sick of being barraged with 502 bad gateway errors because you halfassed yet another API implementation. Stop advertising your crossfit stats, your meditation-app records and your vegan protein bars for a minute, and maybe start writing some fucking code of your own, something with a higher shelf-life than your iPhone screen...

You know... something which actually fucking adds value to the world.

Tl;Dr - It started as an escape, carried on as fun, then as a way to be lazy, and finally as a way of life. Coding has defined and shaped my entire life from the age of nine.

When I was nine I was playing a game on my ZX spectrum and accidentally knocked the keyboard as I reached over to adjust my TV. Incredibly parts of it actually made a little sense to me and got my curiosity. I spent hours reading through that code, afraid to turn the Spectrum off in case I couldn't get back to it. Weeks later I got hold of a book of example code to copy out to do various things like making patterns on the screen. I was amazed by it. You told it what to do, and it did it! (don't you miss the days when coding worked like that?) I was bitten by the coding bug (excuse the pun) and I'd got it bad! I spent many late nights on that thing, escaping from a difficult home life. People (especially adults) were confusing, and in my experience unpredictable. When you did things wrong they shouted at you and threatened to take you away, or ignored you completely. Code never did that. If you did something wrong, it quietly let you know and often told you exactly what was wrong. It wasn't because of shifting expectations or a change of mood or anything like that. It was just clean logic, simple cause and effect.

I get my first computer a year later: an IBM XT that had been discarded by a company and was fitted with a key on the side to turn it on. With the impressive noise it made it really was like starting an engine. Whole most kids would have played with the games, I spent my time playing with batch scripts and writing very simple text adventures. And discovering what "format c:" does. With some abuse and threatened violence I managed to get windows running on it. Windows 2.1 I think it was.

At 12 I got a Gateway 75 running Windows 95. Over the next few years I do covered many amazing games: ROTT, Doom, Hexen, and so on. Aside from the games themselves, I was fascinated by the way computers could be linked together to play together (this was still early days for the Web and computers networked in a home was very unusual). I also got into making levels for Doom, Heretic, and years later Duke Nukem 3D (pretty sure it was heretic; all I remember is the nightmare of trying to write levels entirely by code!). I enjoyed re-scripting some of the weapons and monsters to behave differently. About this time I also got into HTML (I still call this coding, but not programming), C, and java. I had trouble with C as none of the examples and tutorial code seemed to run properly under a Windows environment. Similar for my very short stint with assembly. At some point I got a TI-83 programmable calculator and started rewriting my old batch script games on it, including one "Gangster Lord" game that had the same mechanics as a lot of the Facebook games that appeared later (do things, earn money, spend money to buy stuff to do more things). Worried about upcoming exams, I also made a number of maths helper apps, including a quadratic equation solver that gave the steps, and a fake calculator reset to smuggle them into my exams. When the day came I panicked and did a proper reset for fear of being caught.

At 18 I was convinced I was going to be a professional coder as I started a degree in Computer Science. Three months later I dropped out after a bunch of lectures teaching what input and output devices were and realising we were only going to be taught Java and no C++. I started a job on the call centre of a big company, but was frustrated with many of the boring and repetitive tasks we had to do. So I put my previous knowledge to use, and quickly learned VBA to automate tasks. It wasn't long before I ended up promoted to Business Analyst where I worked on a great team building small systems in Office, SAS, and a few other tools.

I decided to retrain in psychology, so left the job I was in and started another degree. During my work and placements my skills came in use a number of times to simplify and automate tasks. I finished my degree, then took a job as a teaching assistant while I worked out what I wanted to do next and how to pay for it. Three years later I've ended up IT technican at the school, responsible for the website, teaching a number of Computing lessons each week, and unofficial co-coordinator for Computing as a subject. I also run a team of ten year old Digital Leaders who I am training in online safety and as technical experts; I am hoping to inspire them to a future in coding. In September I'll be starting teacher training with a view to becoming a Computing specialist teacher. Oh, and I'm currently doing a course in Android Development in my free time.

And this all started with an accidental knock on the keyboard of a ZX Spectrum.

Roses are red,
Violets are blue,
Bad gateway on port five hundred two.

Sometimes I got one of these:
403 Forbidden
404 Not found
502 Bad gateway

Most of time I got a 702 Incompatible user.

You know what, let me jump in on the "I hate PHP" bandwagon.

A couple months ago I upgraded my mail servers unattended. Roundcube got fucked for a couple of months, and I figured.. fuck it, I can still use Dovecot for authenticating with desktop mail clients like K-9.
Recently I unfucked it, turns out that it was an issue with the sock file in php-fpm. That's also when I noticed that PHP apparently hardcodes in its current version in the bloody socket file. Because why the fuck wouldn't you? It makes upgrades so much fucking easier!!! Said no fucking sysadmin ever!!!

And today I upgraded one of my mail servers to Ubuntu Server 18.04, finally, after a lot of hesitation. Bad decision, because now PHP got fucked YET AGAIN.
Again an issue with socket files? I have no fucking idea. systemctl shows no failed services (because you know PHP, why would you fail your service with an error message instead of throwing a meaningless 502 Bad Gateway, right?!!) and looking at the config files, well the socket file got its new php-fpm 7.2 file (still got the fucking version number hardcoded in) and thus I changed that socket file location in /etc/php/7.0...

devRant may just have been my rubber duck.

WHY THE FUCK DO YOU STINKING FUCKING PILE OF SHIT CALLED FUCKING PHP KEEP THE FUCKING 7.0 DIRECTORY OUT THERE WHEN YOU'VE UPGRADED, WITHOUT EVEN HAVING THE FUCKING BALLS TO RENAME THE MOTHERFUCKING DIRECTORY TO 7.2, IF YOU'RE GOING TO HARDCODE IN YOUR VERSION NUMBERS ANYWAY?!!!!!

Bloody fucking pile of fucking junk!!!!

```js
const app = require('express')()
app.get('/myname', (req, res) => {
res.status(502).send('Bad Gateway.\nUnknown header in brain response: "Hello world!".')
})
app.listen(80)
```

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

Related to the project in my last rant...

Project got delayed for about a month in total because the API for the payment gateway wasn’t allowing charges against stored cards. Could save, modify, and delete them, but no charges.

After a week of trying to get things working based on the documentation, I get in touch with the vendor (great people) who file a support request with the people running the processor so we can see what’s up. Long story short, that amounted to 3 weeks of getting ignored until the vendor raised hell on my behalf, only to get the following reply back:

“You’ve been using the dev credentials, try it on live transactions instead!”

Thankfully, we’re able to move the customer to another processor under the same vendor, where I already have all the requests figured out...

Zomato is down! No dinner for me, I guess! XD

502 bad gateway on AWS!! OMFG!!! So I cant deploy to my newly created VPC of awesomeness!