Wow, what a fucking mess this sunday was.
My boss wrote me an email that one route of a RESTful API we wrote for a customer was not working anymore and puking back a status 500 with some error mentioning invalid UTF-8 characters.
Not one single person has had touched nor changed the code on production in some 6 months, so what the fuck could it be?

Phpunit did not give any errors (running only locally), the code had no syntax errors and the DB dump did not contain any invalid bytes (tested with a hex editor).

WHAT THE FUCK?!

OK so I started to comment out lines (all tested directly on production of course) until the error vanished.

Guess what was the culprit?
.
.
.
.
.
.
In the code (PHP) we used strftime(...) to get nice time strings. Of course we set the correct locale on the server, thus having months and days formatted in German.
So, in Geman there is this one mysterious month called "März" which contains an umlaut character.
Calling strftime generated the date with März in it, but the server locale was de_CH.iso-8859-1 and not fucking de_CH.utf8, so the "ä" was returned as 0xE4 instead of 0xC3A4 (valid UTF-8), which json_encode(...) did not want to swallow but instead threw an exception.

*wrestling commentator voice*
"In this weeks episode of encoding hell:
The iiiinnnfamous UTF-8 Byte Order Mark veeeersus PHP!"

For an online shop we developed, there is currently a CSV upload feature in review by our client. Before we developed this feature, we created together with the client a very precise specification, including the file format and encoding (UTF-8).

After the first test day, the client informed us, that there were invalid characters after processing the uploaded file.
We checked the code and compared the customer's file with our template.
The file was encoded in ISO-8859-1 and NOT as specified UTF-8.
But what ever, we had to add an encoding check, thus allowing both encodings from now on.

Well well well welly welly fucking well...

Test day 2: We receive an email from said client, that the CSV is not working, again.
This time: UTF-8 encoding, but some fields had more colums with different values than specified.
Fucking hell.
We tell the customer that.
(I was about to write a nice death threat novel to them, but my boss held me back)

Testing day 3, today:
"The uploading feature is not working with our file, please fix it."
I tried to debug it, but only got misleading errors. After about 30 minutes, at 20 stacks of hatered, I finally had an idea to check the file in a hex editor:

God fucking what!?!!?!11?!1!!!?2!!

The encoding was valid UTF-8, all columns and fields were correct, but this time the file contained somthing different.
Something the world does not need.
Something nearly as wasteful as driving a monster truck in first gear from NYC to LA.

It was the UTF-8 Byte Order Mark.
3 bytes of pure hell.
Fucking 0xEFBBBF.
The archenemy of PHP and sane people.

If the devil had sex with the ethernet port of a rusty Mac OS X Server, then 9 microseconds later a UTF-8 BOM would have been born.

OK, maybe if PHP would actually cope with these bytes of death without crashing, that would be great.

FUCKING ENCODING SHITHEADS!
WHY ISN'T UTF-8 THE FUCKING DEFAULT EVERYWHERE!
§¥~@#&•…≈!

Finally Spend two fucking days debugging shit until I figured it it. Freaking stupid shit encoding problems and old data combined isn't fun. Dafuq why can't everybody use UTF-8 or Unicode or something else but PLEASE stop using some old school IBM shit codepages.
Leckt mich doch am arsch mit diesem scheiß man -_-

What kind of rusty asshole develops an FTP client which seemingly treats uppercase and lowercase filenames as exactly the same and is not able to fucking understant UTF-8 filenames!?
OK or maybe it was the shitty ass server to which I had to deploy the website to.
I've never been so pissed in my life.
It's already an asshole torture to upload 2.3 giggle bytes of pixel jizz, but 5 hours later, when the site has been made public, you find out that 25% of these images' filenames were automatically renamed during the extraction because some asshole dev thought it was a great idea to not even inform the user about this behaviour.
Fixing filenames in production while your boss is really pissed next to you the hole time is not a great feeling. Especially when you accidentally purge the whole image cache and the PHP image transform task then blocks thus making the whole site not loading any more images for 40 minutes.

WHAT AN ASSRAPE!

Please don't comment. I'm still too pissed to read comments. Thanks.

Fucking gnu gettext, fucking .po and .mo files and fuck the fucking poedit. Why in the fucking hell a damn i18n solution used by so many languages doesn't support correctly fucking utf 8 characters in strings inside the code, I mean, WHAT THE FUCK, this is my second day trying to make the fucking Python gettext module to translate the fucking word "está" because I've a fucking big code base programming in Spanish that need to be translated. This is the fucking 2016 and we still have problems with fucking special characters? I mean CMMON

That one fucking time my text editor so damn high, that it choose to encode my file utf-8 BOM.
I was using Notepad++ because I thought it would just provide a good syntax highlighting, without stabbing me in the back

Seemingly nothing wrong with the code, but it took me, a friend and two teacher almost half an hour to figure out why the css simply wasn't work, even though it was clearly used and worker as intended when embeded in the file.

This was some years ago, so please don't judge me for my editor of choice at that time

Other than that, i simply suck at css and gladly use css frameworks 😅

Writing hebrew in Latex using a template that doesn't support UTF-8 is the most archaic shit I've done lately. I feel like some sort of a caveman.
This fucking encoding inverts all letters so it can support right to left. 😓

*laughing maniacally*

Okidoky you lil fucker where you've been hiding...

*streaming tcpdump via SSH to other box, feeding tshark with input filters*

Finally finding a request with an ominous dissector warning about headers...

Not finding anything with silversearcher / ag in the project...

*getting even more pissed causr I've been looking for lil fucker since 2 days*

*generating possible splits of the header name, piping to silversearcher*

*I/O looks like clusterfuck*

Common, it are just dozen gigabytes of text, don't choke just because you have to suck on all the sucking projects this company owns... Don't drown now, lil bukkake princess.

*half an hour later*

Oh... Interesting. Bukkake princess survived and even spilled the tea.

Someone was trying to be overly "eager" to avoid magic numbers...

They concatenated a header name out of several const vars which stem from a static class with like... 300? 400? vars of which I can make no fucking sense at all.

Class literally looks like the most braindamaged thing one could imagine.

And yes... Coming back to the network error I'm debugging since 2 days as it is occuring at erratic intervals and noone knew of course why...

One of the devs changed the const value of one of the variables to have UTF 8 characters. For "cleaner meaning".

Sometimes I just want to electrocute people ...

The reason this didn't pop up all the time was because the test system triggered one call with the header - whenever said dev pushed changes...

And yeah. Test failures can be ignored.
Why bother? Just continue meddling in shit.

I'm glad for the dev that I'm in home office... :@

TLDR: Dev changed const value without thinking, ignoring test failures and I had the fun of debunking for 2 days a mysterious HAProxy failure due to HTTP header validation...