UTF-8 gone wrong

I got 99 problems but encoding ain’t one

I have got 99 problems and emoji encoding ain't one of them.

P.S. /r/softwaregore

string excuses[]={
"it's not a bug it's a feature",
"it worked on my machine",
"i tested it and it worked",
"its production ready",
"your browser must be caching the old content",
"that error means it was successful",
"the client fucked it up",
"the systems crashed and the code got lost" ,
"this code wont go into the final version",
"It's a compiler issue",
"it's only a minor issue",
"this will take two weeks max",
"my code is flawless must be someone else's mistake",
"it worked a minute ago",
"that was not in the original specification",
"i will fix this",
"I was told to stop working on that when something important came up",
"You must have the wrong version",
"that's way beyond my pay grade",
"that's just an unlucky coincidence",
"i saw the new guy screw around with the systems",
"our servers must've been hacked",
"i wasn't given enough time",
"its the designers fault",
"it probably won't happen again",
"your expectations were unrealistic",
"everything's great on my end",
"that's not my code",
"it's a hardware problem",
"it's a firewall issue",
"it's a character encoding issue",
"a third party API isn't responding",
"that was only supposed to be a placeholder",
"The third party documentation is wrong",
"that was just a temporary fix.",
"We outsourced that months ago.","
"that value is only wrong half of the time.",
"the person responsible for that does not work here anymore",
"That was literally a one in a million error",
"our servers couldn't handle the traffic the app was receiving",
"your machines processors must be too slow",
"your pc is too outdated",
"that is a known issue with the programming language",
"it would take too much time and resources to rebuild from scratch",
"this is historically grown",
"users will hardly notice that",
"i will fix it" };

Ｔｈｅ　ｔｈｉｎｇ　ａｂｏｕｔ　ＵＮＩＣＯＤＥ　ｉｓ　ｔｈｅ　ａｂｉｌｉｔｙ　ｔｏ　ｍａｋｅ　ｉｔ　ｈａｒｄ　ｔｏ　ｐａｒｓｅ　ｗｈａｔ　ｈｕｍａｎｓ　ａｕｔｏｍａｔｉｃａｌｌｙ　ｓｅｅ　ａｓ　ＡＳＣＩＩ

"Ok, the site looks fine. Now let's move the style tag into it's own file."

*makes css file*

"WHY DOES IT HARDLY EVER LOAD!?!?, I checked the syntax trice"

*Spends 20 min. Asking friends for help, but none of them knows a reason*

"Time to ask the teacher, I guess"

*Teacher comes over, but has no clue either*

Teacher: "Give me the files, let's test it on my laptop"

*Css doesn't load there either*

*Teachers pair programming and trying some serious debugging technics. No progress*

*I decided to look at the sourcecode while refreshing the site*

1. Refresh: Css is loaded properly
2. Refresh: Css is gone, and source turned into various asian symbols.

Looks at the (default) file encoding: UCS-2
WTF NOTEPAD++, I SPEND 2 HOURS OF MY LIFE BECAUSE YOU DECIDED THIS WAS A PROPER ENCODING!

Web programming seems fun.

Not sure if it just doesn't have the right character encoding scheme, or if my existence is being questioned...

Notepad finally adds supports Unix-encoding

Now THAT is good news!

stolen from somewhere...

Intelligence and ability cannot be measured by education.

I have a client who asked a Master in Computer Science to develop a small system, for querying product title and their code. The guy used python, vanilla js, and... Txt file for the "database". Then my client asked me to integrated this in... WordPress.

This was in 2016. And idiot as I'm, I agreed and adapted his code to use php and a database.

April this year, my client said they are still using the python system to add new products all this time, in parallel. And wanted to update the WordPress with the data.

- No problem! - I said. Just send me the SQL file.

So the Master in CS sent me a SQL coded in ANSI. I asked for the SQL again, but with a more appropriate encoding. He took 1 month to reply back, and said it would be better if I get rid of the database and just use the txt file for querying.

This is outrageous.

I really hate people who are educated but completely useless.

Encoding issues always give me �headache�.

Me: I have an input stream!
Library: I want a file.
Me: I can see your code, you will convert it to an input stream! Don't any of your constructors take an input stream?
Library: No. I want a file. Okay, you can also include an encoding.
Me: I don't care about encoding.
Library: Fine, just give me the file then.
Me: You mean I have to somehow convert my input stream into an actual file so I can give it to you?
Library: You're gradually catching on, yes.
Me: Can I add a new constructor myself that takes an input stream?
Library: Sure! Good luck in getting it approved by the maintainers and the new jar distributed everywhere, including Maven Central before your deadline.
Me: Fine, I'll just rearchitect everything so I can give you an actual file.
Library: And then everything will be fine.
Me:

My stickers just arrived. And I especially love the encoding of “ß” within the street name.

German. It's hard.

Next time, just use upper letter and replaced “ß” with “SS” or just use a “B”. The B is especially fine to drive typographers mad.

*wrestling commentator voice*
"In this weeks episode of encoding hell:
The iiiinnnfamous UTF-8 Byte Order Mark veeeersus PHP!"

For an online shop we developed, there is currently a CSV upload feature in review by our client. Before we developed this feature, we created together with the client a very precise specification, including the file format and encoding (UTF-8).

After the first test day, the client informed us, that there were invalid characters after processing the uploaded file.
We checked the code and compared the customer's file with our template.
The file was encoded in ISO-8859-1 and NOT as specified UTF-8.
But what ever, we had to add an encoding check, thus allowing both encodings from now on.

Well well well welly welly fucking well...

Test day 2: We receive an email from said client, that the CSV is not working, again.
This time: UTF-8 encoding, but some fields had more colums with different values than specified.
Fucking hell.
We tell the customer that.
(I was about to write a nice death threat novel to them, but my boss held me back)

Testing day 3, today:
"The uploading feature is not working with our file, please fix it."
I tried to debug it, but only got misleading errors. After about 30 minutes, at 20 stacks of hatered, I finally had an idea to check the file in a hex editor:

God fucking what!?!!?!11?!1!!!?2!!

The encoding was valid UTF-8, all columns and fields were correct, but this time the file contained somthing different.
Something the world does not need.
Something nearly as wasteful as driving a monster truck in first gear from NYC to LA.

It was the UTF-8 Byte Order Mark.
3 bytes of pure hell.
Fucking 0xEFBBBF.
The archenemy of PHP and sane people.

If the devil had sex with the ethernet port of a rusty Mac OS X Server, then 9 microseconds later a UTF-8 BOM would have been born.

OK, maybe if PHP would actually cope with these bytes of death without crashing, that would be great.

I think this is so far one of the most priceless WTF moments I encountered at my current work:

A coworker of mine came up to me explaining the problem he had with russian characters in the filename. He explained in detail that everything works ok (the other part of the code he was fixing) if he changes the name of the file to test1.xlsx for example which doesn't use russian characters. OK great.

Then he goes on to show me how he fixed the other stuff and of course everything blows up. The file he used for demonstration was of course the original file our cusotomer provided, he just deleted the obvious russian chars and left the rest.

МТС != MTC

I cracked up: but you still have russian chars in the name.
The guy: no way, I deleted them all.
Me: but what about that МТС in the name?! Guy: what about it?
Me: did you actually typed that in or you left it there?! Those are russian chars that are fucking things up for you.
Guy: no way, it's MTC.
Me: checked the logs, you have ??? In the filename instead of МТС..don't you find that at least a little bit suspicious?!
Guy: but it looks the same. How does it (the computer) know it is in russian?!? //Why doesn't it understand?!

O.o I still can't believe it.. Is it just me & my high standards, or should it be normal for coders to know things such as character encoding & stuff?!?

I almost died of laughter, he and some other guy had problems finding customers in the software due to not being able to type the russian chars << happened more then once before, even after I told them about a quick hack on how to use google translate onboard keyboard & other stuff to make proper chars so they can get a match..

I think when they bury me, I'll still be facepalming and laughing over this incident. 🤣🤣🤣🤣🤣🤣🤣

Today I'm trying to study how to encode data in idx-ubyte format for my machine learning project.
Professors I'm going to astonish you!

Good day and good coding to all of you! :)

EXCEL YOU FUCKING PIECE OF SHIT! don't get me wrong, it's usefull and kt works, usually... Buckle up, your i for a ride. SO HERE WE FUCKING GO: TRANSLATED FORMULA NAMES? SUCKS BUT MANAGABLE. WHATS REALLY FUCKED UP IS HTHE GERMAN VERSION!
DID YOU HEAR ABOUT .csv? It stands for MOTHERFUCKING COMMA SEPERATED VALUES! GUESS WHAT SOME GENIUS AT MICROSOFT FIGURED? Hey guys let's use a FUCKING SEMICOLON INSTEAD OF A COMMA IN THE GERMAN VERSION! LET'S JUST FUCK EVERY ONE EXPORTING ANY DATA FROM ANY WEBSITE!

The workaround is to go to your computer settings, YOU CAN'T FUCKING ADJUST THIS IN EXCEL!, change the language of the OS to English, open the file and change it back to German. I mean, come on guys, what is this shit?

AND DON'T GET ME STARTED ON ENCODING! äöü and that stuff usually works, but in Switzerland we also use French stuff, that then usually breaks the encoding for Excel if the OS language is set to German (both on Windows and Mac, at least they are consistent...)

To whoever approved, implemented or tested it: FUCK YOU, YOU STUPID SHITFUCK, with love: me

I sometimes remember the time when I wrote a Email-inbox-exporter-PHP-script-type of application that collects all the emails from an inbox, "copied" it to a database with the attachements and stuff and moves it to a folder..

I just started at the company for like a couple of months, had no privileges to create mailboxes and such and I didn't want to interrupt our programmer to do this for me, so... I decided.. to save time and resources.. to test run it on our global, live 'support' mailbox.. :D Well.. You might guess what happened.. Apparently I mistyped the name of the move-destination folder (because imap-weird-things) that resulted in a completly empty mailbox and an empty database because the inserts failed due to bad encoding and mime-type issues..

The moment I refreshed my Outlook and noticed that all our mails where gone.. I swear, I can't describe that feeling of fear, cold sweat, intense heartbeat... I just stood up, asked if anyone wanted coffee, and just walked out of the office.. When in the hallway, I heard my collegues ask to one another "do you have any issues with outlook, all my mails are gone?". Everyone was stressing out, the chief was stressing out "what happened?!", nobody knew what happened.. :D

They could partially resolve it via one collegue who hadn't refreshed the mailbox and he could forward all the mails back to our support mailbox..

I dropped the project idea and learned to work with dev environments :D A couple of months later, I accidentially forgot a where condition in my SQL UPDATE statement, but that was the last time I seriously f*cked up.. :D Got to learn the hard way I guess.. Now everything I do runs in dev environments, I test everything before publishing,.. When I look back.. I don't even recognize the (inexperienced) guy I was back then ! :D

Ps. No one still knows what happened that day and they blamed it on server issues :D

FUCKING ENCODING SHITHEADS!
WHY ISN'T UTF-8 THE FUCKING DEFAULT EVERYWHERE!
§¥~@#&•…≈!

Story: Password hashing and UTF-8

Context: PHP 5.6, 270kloc 15+ years legacy project. ~3 years ago. tl;dr at bottom.

Password hashing & verification was done with an obsolete way of hashing passwords. I was given the task to update our password handler to from now on generate passwords with PHP's good built-in password hashing function.

It was decided that old passwords still needed to work, instead of prompting users to set a new password. The old password verification still had to function in conjunction with the new.

The previous password handler was split into multiple classes, due to (I assume) poor structuring and shoehorning in an object oriented approach. Furthermore, it abused global variables.

A new password handler had to be created.

I implemented the new password verification and creation methods (which now used PHP internal password functions), and it worked perfectly. Then to get the old password verification to work.

I removed all obsolete methods from the old handler, and was left with a hashing function which took in a password, salt, and a secret key. I copied this code into the new handler.

It failed. It returned "Password does not match" for old passwords. I was unsure what had happened here. I did all sorts of shotgun debugging. I ended up with two versions of the login page next to each other, which used the old and new code respectively. I started modifying the original code, extracting variables, logging, you name it. I ended up with exactly the same snippets of code in both password handlers, and yet it failed.

The culprit? The character encoding.

Because this project was over a decade old, the .php-file had the encoding 'windows-1252'. When I created the new password handler, my IDE set the file encoding to 'UTF-8'. Then when I copied the secret, my IDE converted the string to 'UTF-8', effectively changing the value of the secret and causing any password verifications to fail. The solution was to manually create a string using the byte values in the old secret.

<rant>
It is these extreme, obscene, scenarios which makes working with legacy projects a living hell. In this scenario, it was my IDE at fault for changing the character encoding.

But my IDE is not the root problem. No, I blame it on the lack of maintenance from previous developers. Not keeping the codebase up to standard causes problems like this in the long run.
</rant>

tl;dr: copied hash secret to a file with another encoding. IDE changed the byte values for those characters, causing password verification to fail. fml.

Encoding strikes yet another victim

Well it seems that i broke something..

So, California, land of no rain whatsoever.

Until today, where it actually rained!

Also today, a weather applet for tmux started misbehaving. Suspicious.

Coincidence, I think not! Sure enough, stopping that applet from running fixed everything. Turns out, the raining icon breaks the terminal, for some strange reason.

And why had this not been discovered earlier? This had been the first time it had rained for at least 8 months, and I added this only 4 months ago. Thus, it had never had its time to shine.

Yay for strange encoding!

Anyway, I now have a new excuse; blame bad code on global warming.

Testing hell.

I'm working on a ticket that touches a lot of areas of the codebase, and impacts everything that creates a ... really common kind of object.

This means changes throughout the codebase and lots of failing specs. Ofc sometimes the code needs changing, and sometimes the specs do. it's tedious.

What makes this incredibly challenging is that different specs fail depend on how i run them. If I use Jenkins, i'm currently at 160 failing tests. If I run the same specs from the terminal, Iget 132. If I run them from RubyMine... well, I can't run them all at once because RubyMine sucks, but I'm guessing it's around 90 failures based on spot-checking some of the files.

But seriously, how can I determine what "fixed" even means if the issues arbitrarily pass or fail in different environments? I don't even know how cli and rubymine *can* differ, if I'm being honest.

I asked my boss about this and he said he's never seen the issue in the ten years he's worked there. so now i'm doubly confused.

Update: I used a copy of his db (the same one Jenkins is using), and now rspec reports 137 failures from the terminal, and a similar ~90 (again, a guess) from rubymine based on more spot-checking. I am so confused. The db dump has the same structure, and rspec clears the actual data between tests, so wtf is even going on? Maybe the encoding differs? but the failing specs are mostly testing logic?

none of this makes any sense.
i'm so confused.

It feels like i'm being asked to build a machine when the laws of physics change with locality. I can make it work here just fine, but it misbehaves a little at my neighbor's house, and outright explodes at the testing ground.

One of the things I love the most about programming is that sometimes you feel like you're taking a step forward when you're taking a step back... and sometimes you feel like you're taking a step back when you take a step forward.

Or maybe that's just me. XD

My teacher thinks encryption and encoding are same

Woke up this morning to a fucking giant snowstorm and my first reaction was 'fml' , poured some coffee , lit a smoke and started checking my work mail 'Issue xxxx response : Not solvable '...what the...I go through the files on my phone , look at what that issue was : lack of proper validation , filtering and encoding of input thus enabling xss . Not solvable my ass ...simply adding literally 3 more characters to that fucking retarded filter would stop all the bypasses . This issue is a showstopper for their project and that is what they answer ?
Sorry to indians out here but some of your colleagues are as stupid and unimaginative as they can possibly ever come .

Uggg..... I'm trying to encode a binary file in Python which may be an image or may be an executable, and then decode it back into a file (I plan on editing it in the middle, but baby steps for now..) but nothing is working!!

My plan is to:

Open binary file.

Decode as base64, or something else that could easily handle binary.

Convert byte data to string (for editory perpousos - I won't be editing bytes, I'll be doing custom encoding but that's irrelevant for this test)

Convert back to a byte string/array (with .encode(), probably)

Write to file.

I do this, yet the output has been altered... Though I haven't touched anything..

It's so enfuriating.. x.x

Because nothing says "security" like some good ol' Base64 encoding. Bet whoever wrote that code was wearing mirror shades.

Yesterday a strange bug appeared in Chrome: In a small web app we have some umlaut characters like äöü. Strangely said characters were displayed as cyrillic, but only on my pc... On every other device it worked. I spent about 5 hours of checking encodings (everything was in UTF-8), reading posts in the almighty stackoverflow.
Finally i figured out, the font was broken. After reinstalling it, everything was peaceful again in my head.

Just imagine the world where we have CAPITALIZED DIGITS.

So someone here had the job to merge the current version of our software into another older repo.
(That older repo has some features for future versions, that should be only in there ... don´t ask ... not my decision)

Took him long enough. But he forgot to check one thing while merging: The encoding of the files.

And we´re german...

ALL umlauts, and all other special chars in the code were replaced with this: �
No global replace, because all chars were replaced with the same char.
(Why the fuck do we have special chars in the code in the first place???)

So to not need him to start all over again I compiled a list of common german words containing umlauts and did a global search and replace.

I think I got 90% of the errors like that.
Now he´s going to correct the rest of the errors.

Fuck the comments in the code though.
Just a waste of time...

I was working on this mustachio fork called Morestachio and changed the Default Encoding from "Encoding.Default" to UTF8 while creating a .NetStandard version of it. After i have created everything i thought "Hey just for the lolz run my tests again" ...

The senior developer swore the server was responding with a 500 because I, the co-op, was definitely sending the 'wrong encoding' whenever I post special characters. He said, "I'm only taking your data and putting it straight into the database!!!"

I found out it was breaking because he was converting the JSON payload to a STRING and simplistically separating the keys and values by colons and double quotes. Yuuup. Nothing can go wrong there...

TL:DR
Why do so shitty "API"s exist that are even harder to write than proper ones? D:

Trying to hack my venilation at home.
This API is so horrible D:
The API is only based on POST requests no matter if you want to write values or get values and the response only contains XML with cryptic values like:

<?xml version="1.0" encoding="UTF-8"?>
<PARAMETER>
<LANG>de</LANG>
<ID>v01306</ID>
<VA>00011100000000000000000010000001</VA>
<ID>v00024</ID>
<VA>0</VA>
<ID>v00033</ID>
<VA>2</VA>
<ID>v00037</ID>
<VA>0</VA>

Also there are multiple API routes like
POST /data/werte1.xml
POST /data/werte2.xml
POST /data/werte3.xml
POST /data/werte4.xml
And actually the real API route is only given in the request body and not in the path.

Why is this so shitty? D:<

Btw in terms of security this is also top notch. It just globally saves if one computer sends the login password.
I mean why even ask for a password then? D:
That made me end up with a cronjob to send a login request so I don't have to login on any device.

PS:
You see, great piece of German engineering.

On one hand, emoji are an example of the miracle of encoding, and I should really appreciate them more from a technical perspective.

On the other, anyone that sends 14 "100" symbols and an OK hand sign should be legally prohibited from breeding.

Just had to do this. Homemade. Sorry :D

When you encounter a character encoding issue.

UTF-8 without BOM.
{ ä, ö, ü } are not going to be decoded correctly.
Why the fuck does BOMless UTF-8 still exist? Such a pain in the ass.

Currently working with @Kreischo and another good friend to create a secured, encoded container to store files in it.

I am currently working on the frontend of things, thinking it's quite beautiful.. (Done with Electron)

Your opinion?

How deep does the rabbit hole go?

Problem: Convert numpy array containing an audio time series to a .wav file and save on disk

Error 1:
Me: pip install "stupid package"
Console: Can't pip, behind a proxy

Me: Finds workaround after several minutes

Error 2:
Conversion works, but audio file on disk doesn't work
Encoding Error only works with array of ints not floats
BUT I NEED IT TO BE FLOATS

Looks for another library
scikits.audiolab <- should work
Me: pip --proxy=myproxy:port install "this shit"
Command Line *spits back huge error*

Googles error <- You need to install this package with a .whl file
Me: Downloads .whl file <- pip install "filename".whl
Command Line: ERROR: scikits.audiolab-0.11.0-cp27-cp27m-win32.whl is not a supported wheel on this platform.

Googles Error <- Need to see supported file formats
Me: python -c "import pip; print(pip.pep425tags.get_supported())"
Console: AttributeError: module 'pip' has no attribute 'pep425tags'

Googles Error <- Use another command for pip v10
Me: python -c "import pip._internal; print(pip._internal.pep425tags.get_supported())"
Console: complies

Me: pip install "filename".whl
Console: complies

Me: *spends 30 minutes to find directory where I should paste .dll file*
Finds Directory (was hidden btw), pastes file

Me: Runs .py file
Console: from version import version as _version ModuleNotFoundError: No module named 'version'

Googles Error <- Fix is: "just comment out the import statement"
Me: HAHAHAHAHAHA
Console: HAHAHAHAHA

Unfortunately this shit still didn't work after two hours of debugging, lmao fuck this

Hmm...recently I've seen an increase in the idea of raising security awareness at a user level...but really now , it gets me thinking , why not raise security awareness at a coding level ? Just having one guy do encryption and encoding most certainly isn't enough for an app to be considered secure . In this day an age where most apps are web based and even open source some of them , I think that first of all it should be our duty to protect the customer/consumer rather than make him protect himself . Most of everyone knows how to get user input from the UI but how many out here actually think that the normal dummy user might actually type unintentional malicious code which would break the app or give him access to something he shouldn't be allowed into ? I've seen very few developers/software architects/engineers actually take the blame for insecure code . I've seen people build apps starting on an unacceptable idea security wise and then in the end thinking of patching in filters , encryptions , encodings , tokens and days before release realise that their app is half broken because they didn't start the whole project in a more secure way for the user .

Just my two cents...we as devs should be more aware of coding in a way that makes apps more secure from and for the user rather than saying that we had some epic mythical hackers pull all the user tables that also contained unhashed unencrypted passwords by using magix . It certainly isn't magic , it's just our bad coding that lets outside code interact with our own code .

Today I learned about binary encoding formats alternative to JSON such as Google Protocol Butter.
I like these binary formats.
Just thought I would share this here so others would benefit as well (and please share your experience if it is relevant)

When you Render something and the name is just perfect!

Translation: Catched encoding buying Toilet Paper

Guess I'll fuckin try again tomorrow.

Building a cross platform c program. On Linux side, just using a makefile. Today I tried using visual studios "clone" feature for git. It just downloads the files and makes them available to the editor, it doesn't make a project, obviously.

But this has some disadvantages. For one, you can't build, or run. Two, you don't get any project properties. My project needs to set the character encoding to Unicode. Can't do that without a project.

So I use their tool to create a project from existing code. It didn't really work. The build profiles were janky at best and I still couldn't set the character encoding.

Ended up just deleting the whole thing.

Encoding problems on live TV in portugal...

The only way I would ever use a recovery question.
BTW nice encoding there...

All it takes is one jackass somewhere in the world with no understanding of character encoding to ruin things for everyone else.

So for those of you keeping track, I've become a bit of a data munger of late, something that is both interesting and somewhat frustrating.

I work with a variety of enterprise data sources. Those of you who have done enterprise work will know what I mean. Forget lovely Web APIs with proper authentication and JSON fed by well-known open source libraries. No, I've got the output from an AS/400 to deal with (For the youngsters amongst you, AS/400 is a 1980s IBM mainframe-ish operating system that oriiganlly ran on 48-bit computers). I've got EDIFACT to deal with (for the youngsters amongst you: EDIFACT is the 1980s precursor to XML. It's all cryptic codes, + delimited fields and ' delimited lines) and I've got legacy databases to massage into newer formats, all for what is laughably called my "data warehouse".

But of course, the one system that actually gives me serious problems is the most modern one. It's web-based, on internal servers. It's got all the late-naughties buzzowrds in web development, such as AJAX and JQuery. And it now has a "Web Service" interface at the request of the bosses, that I have to use.

The programmers of this system have based it on that very well-known database: Intersystems Caché. This is an Object Database, and doesn't have an SQL driver by default, so I'm basically required to use this "Web Service".

Let's put aside the poor security. I basically pass a hard-coded human readable string as password in a password field in the GET parameters. This is a step up from no security, to be fair, though not much.

It's the fact that the thing lies. All the files it spits out start with that fateful string: '<?xml version="1.0" encoding="ISO-8859-1"?>' and it lies.

It's all UTF-8, which has made some of my parsers choke, when they're expecting latin-1.

But no, the real lie is the fact that IT IS NOT WELL-FORMED XML. Let alone Valid.

THERE IS NO ROOT ELEMENT!

So now, I have to waste my time writing a proxy for this "web service" that rewrites the XML encoding string on these files, and adds a root element, just so I can spit it at an XML parser. This means added infrastructure for my data munging, and more potential bugs introduced or points of failure.

Let's just say that the developers of this system don't really cope with people wanting to integrate with them. It's amazing that they manage to integrate with third parties at all...

Writing hebrew in Latex using a template that doesn't support UTF-8 is the most archaic shit I've done lately. I feel like some sort of a caveman.
This fucking encoding inverts all letters so it can support right to left. 😓

Over the summer I was recruited to be a supplement instructor for a data structures course. As a result of that I was asked (separately by the professor) to be a grader for the course. Because of pay limitations I've mostly been grading homework project assignments. In any case, it's a great job to get my foot into the department and get recognized.

Over the course of the semester I've had this one person, OSX, named after their operating system of choice, who has been giving me awkward submissions. On the first assignment they asked the professor for extra time for some reason or the other, and that's perfectly fine.

So I finally receive OSX's submission, and it's a .py file as per course of the course. So I pop up a terminal in the working directory and type "python OSX_hw1.py". Get some error spit out about the file not being the right encoding. I know that I can tell python to read it in a different encoding, so I open it up in a text editor. To my surprise it's totally not a text file, but rather a .zip file!

I've seen weirder things done before, so no big deal. I rename the file extension, and open it up to extract the files when I see that there's no python files. "Okay, what's goin on here OSX..." I think to myself.

Poking around in the files it appears to be some sort of meta-data. To what, I had no clue, but what I did find was picture files containing what appeared to be some auto-generated screenshots of incomplete code. Since I'm one to give people the benefit of doubt even when they've long exhausted other peoples', I thought that it must be some fluke, and emailed OSX along with the professor detailing my issue.

I got back a rather standard reply, one of which was so un-notable I could not remember it if my life depended on it. However, that also meant I didn't have to worry about that anymore. Which when you're juggling 50 bazillion things is quite a relief. Tragically, this relief was short lived with the introduction of assignment 2.

Assignment 2 comes around, and I get the same type of submission from OSX. At this time I also notice that all their submissions are *very* close to the due time of 11:59pm (which I don't care about as long as it's in before people start waking up the next morning). I email OSX and the professor again, and receive a similar response. I also get an email from OSX worried about points being deducted. I reply, "No issue. You know what's wrong. Go and submit the right file on $CentralGradingCenter. Just submit over your old assignment".

To my frustration OSX claimed to not know how to do this. I write up a quick response explaining the process, and email it. In response OSX then asks if I can show them if they comes to my supplemental lesson. I tell OSX that if they are the only person, sure, otherwise no because it would not be a fair use of time to the other students.

OSX ends up showing up before anyone else, so I guide them through the process. It's pretty easy, so I'm surprised that they were having issues. Another person then shows up, so I go through relevant material and ask them if they have any questions about recent material in class. That said, afterwards OSX was being somewhat awkward and pushy trying to shake my hand a lot to the point of making me uncomfortable and telling them that there's no reason to be so formal.

Despite that chat, I still did not see a resubmission of either of those two assignments, and assignment 3 began to show it's head. Obviously, this time, as one might expect after all those conversations, I get another broken submission in the same format. Finally pissed off, I document exactly how everything looks on my end, how the file fails to run, how it's actually a zip file, etc, all with screenshots. That then gets emailed to the professor and OSX.

In response, I get an email from OSX panicking asking me how to submit it right, etc, etc. However, they also removed the professor from the CC field. In response I state that I do not know how to use whatever editor they are using, and that they should refer to the documentation in order to get a proper runnable file. I also re-CC the professor, making sure OSX's email to me is included in my reply.

OSX then shows up for one of my lessons, and since no one had shown up yet, I reiterate through what I had sent in the email. OSX's response was astonished that they could ever screw up that bad, but also admits that they had yet to install python(!!!). Obviously, the next thing that comes from my mouth is asking OSX how they write their code. Their response was that they use a website that lets them run python code.

At this point I'm honestly baffled and explain that a lot of websites like those can have limitations which might make code run differently then it should (maybe it's a simple interpreter written on JavaScript, or maybe it is real python, but how are you supposed to do file I/O?) .

After that I finally get a submission for assignment 1!

Superhuman capabilities.

Friend - Hey is this file correctly encoded ?

Me - Hell Yes. I can decode visually from an IMAGE and make sense of underlying data encoded.

Some people think I am so awesome coder I can look at Huffman encoded result and understand if it's correct or not.

Someone building the iPad software for this beer museum did Not think about the encoding xD

i wrote a website, a server in go, a small os in c, a game in js, a game and server and web scraper and other desktop apps in java, mobile apps with flutter, a website with php also, implemented aes in go, wrote a parser in java. done sysadmin stuff on my vps and pihole/openvpn/nextcloud on my rpi. learn about c vulnerabilities and used metasploit. attempted to write an interpreted language. did some led displays with arduino. currently learning tensorflow.

i have never...
- written a driver
- made a game with a game engine
- created a file encoding
- implemented an oauth2 server
- made an api
- worked with vr

what am i missing? i want to be a very well rounded dev.

API provider: include a signature based on these fields in this order. DO NOT ENCODE IT!

Implementation works a while, then..

*a wild apostrophe appears*

Signature no longer works.

API Provider: "oh, yeah we escape those."

Arrghhghghghhhghvhxmchsoxnsoxnwl

Not only is it a poor design for signing payloads, the documentation is shockingly poor in it.
Even the implementation example (which is supposedly from their code) doesn't account for any type of escaping or encoding.

Before anyone asks, I can't into details about the implementation.

Go fmt yourself? I guess. Taken from Golang's documentation on encoding/json.

Let's focus/master computer architectures, coding paradigms, datastructures. Everything comes after that...the problem with todays academics is that they are more focused on immediately deploying students to industry; theyre more focused on teaching specific frameworks and specific language instead of teaching how things work...i bet most students (at least in my country) are having troubles with endianess or encoding or even byte manipulation or what a thread is....If im going to be the teacher for example of an oop subject, ill let the student choose the language they want as long as the oop paradigm is intact ,it will be fine.. i dont friggin care whether you know vue or angular or swing if you dont even know what a callback is..

Just found out the API of our zentracloud sensors is sending the units with a space before the actual unit. Couldn't figure out for half an hour why Doctrine is not finding the unit in the database. Encoding? JSON decoding? Character itself? Screw you. Screw you...

Working with a vendor's api. Every call is coming back error. Emails back and forth for two weeks, sending logs, changing settings, encoding urls...

From them: oh, this field needs to be unique.

Me: how are you just now catching that?!

The doc showed it to be a specific value. We must be their first client or something.

OK what the actual fuck is going on within this company.

TL;DR: Spaghetti Copy/Pasted code that made me mad because it's just a mess

I just looked into a code file to search for a specific procedure regarding the creation of invoices.
I thought "Oh this is gonna be a quick look-through of like 1000 lines MAX" turns out this script is 11317 fucking lines long and most of it's logic is written there multiple (up to 6-7 times). And I'm not talking about a simple 10 lines or something. No! Logic of over 300 lines.. copy & pasted over .. and over .. and over?! I mean what the fuck did this guy drink when he wrote this.

Alsooo 10000 of those 11317 lines is ONE FUNCTION.. I kid you not! It's just a gigantic if / else if construct that, as I said before, contains copy-pasted code all over the place.
Sadly my TL thinks that code cleanup / optimization is "not necessary as long as it works" like wtf dude. If anyone wants to ever fix something in this mess or add a new feature they take a few hours longer just to "adjust" to this fucking shit.
This is a nightmare. The worst part: This is not the only script that has shit like this. We got over 150 "modules" (Yeah, we ATTEMPTED something OOP-ish but failed miserably) that sometimes have over 15000 lines which could be easily cut down to 1/3 and/or splitted into multiple files.
Let's not start about centralization of methods or encoding handling or coding standards or work code review or .. you get the point because there's a character limit for one rant and I guess I'd overshoot that by a lot if I'd start with that. Holy shit I can't wait until my internship is over and I can leave this code-hell!!

Use .editorconfig files when working with multiple developers. Otherwise your file encoding and code style will get fucked up eventually.

So the guy who i mentioned previously with his shitty csv made webservice which i needed to use.

It returned utf-8 encoded xml:
- in file the header contained central europe encoding
- the xml had more roots
- basically he only put his csv format inside xml tags
- csv contained html tags but there wasnt cdata

Now work with this lol

I neded to cut this shit with string functions and also some numeric data get with regexp from htm table td's

Whyyyyyyyyyyyy

Every flipping time when receiving file with win1251 charset

Trying to work out an encoding bug after upload, I ask them to send me the original version of the file that is currently uploaded. I look at the files and they seem strangely similar, but strangely different.

I reply to them, "To confirm, this is the version you uploaded?"

"Nope."

If I were to write a de/encoding algorithm, that nobody but me knows about, would it be safe?

Been working on this project for the last 9 months, it's 1000 times bigger than the original spec, but the worst part of the whole experience is the god damn Microsoft Access database and migrating data from it.

Wanky encoding, references that aren't actually linked up, ID's that aren't unique, meaningless columns and table names.

Pushed me to the edge.

Just getting a pizza, can't help but get annoyed by the jittery as fuck newsticker, they didn't even get the encoding right....

(Yes our POS software has 60fps jitterless hardware accelerated fully utf-8 encoded newstickers before anyone asks me to do a better job)

First experience with utf-16 and all I can think is "whatthefuckwhatthefuckwhatthefuckwhatthefuck"

During a code review I was doing the dev wrote a function that takes a tweet, iterates over each letter, if the letter is not in a specified unicode range the letter is removed from the sentence. So now you have a sentence with missing letters... Wtf

a lot of dev have a miss concept about Unicode/utf8 including me but I believe my understanding get Better and this my last version.
For a project i was developing a rest api for mobile app
when an ios dev asked me
"I send you Unicode string but it appears as ????? in admin web panel "
OMG!!!😨😨😨
Unicode is not an encoding nor an algorithm. it's a standerd which just map a glyph to a codepiont .
but utf8 is the encoding of Unicode and how it's stored or transferred ,
the string you send must be a utf-8 encoded string as the rest of the json you sent .

Sooo. My team and I have module we're supposed to be porting to async code and aiohttp will not work. The server keeps rejecting the byte payload, but if we use synch code like the requests library, it works fine. The code is like identical, the only difference is async. It's been really frustrating because another drop in async version of requests (httpx) works just fine! I don't want to use httpx, the rest of our codebase is already using aiohttp! We think the problem is with gzip encoding being handled incorrectly by aiohttp. I've reported the issue.

"So we know you came from a past environment of API development, healthcare encoding and micro-service architecture. With that, we have a shopify site that needs built in 2 weeks. Have fun (finger quotes) DEV".

Is there any program that can be used to cut out parts of a video file without re-encoding.

The way I'm thinking is if you already have a video file, you should be able to just copy the data from 1 file to another like a file splitter.

BigBit standard for numeric format and character encoding

A shitty platform that, although open source, there is no clearly documented way of setting a development environment for it. This pile of crap states clearly that it does NOT support RTL languages. One of the core business requirements is Arabic support. What to do? Look for other platforms? WRONG!

Base the fucking business on it and ask ME to see why the SQL database is not encoding the Arabic characters correctly and to look into the logs that back-end puked. My expertise is mobile development anyways damnit. Sure the backend code is Java code (Java jokers and haters, not the appropriate place) and I know it but there is no fucking way to test that motherfucker or to build it! No fucking testing server can be made! Only instructions to get a Docker image pulled and set up.

FML.

"This company is a fucking م."

I cannot believe I am so frustrated that I am ending this rant with a fun puzzle.

Hints to help you decipher the quoted sentence:

Hint 1: That Arabic letter is the perfect letter.
Hint 2: You don't need to be an Arab to understand what it means.

Utf-string encoding in windows has been constantly fucking me up... It's been 4 hours and I'm still in the office alone.

Finally got around to some real video encoding work on my new computer but noticing it's not blazing fast...

And more work is still handled by the CPU... But I thought video processing is handled by the GPU, which seems to be barely used at all. I'm using Handbrake but I thought the whole point of dedicated GPU was for intensive graphics and video processing?

Never again will I use eclipses egit extension. First eclipse thought that my plain text java source code should be encoded in some bizarre occult way which made eclipse think its binary what made me try pretty much anything one can do with a .gitattributes file before a colleague suggested to not trust eclipse eith the encoding it was explicitly told to use, then I fetched another branch to merge them which somehow killed my .project file and forced me to delete and refetch the whole thing which led to eclipse not longer recognizing it as a java project. May it be because I'm to stupid to use my tools? Yeah, probably. But I'm done with egit, it's all console gitting from now on, fuck suggested practice.

Please, before exporting anything in whatever editor you use, check if it is in UTF-8. Today I didn't knew why my new font wasn't working in certain places and I later discovered that more than 9000 characters were replaced by the replacement character...

Fuck encoding and fuck PHP!!!
I'm programming a little vocab trainer to get used to php and MySQL. From an old VB vocab trainer I had ca. 2000 txt-files with words and converted them to sql-queries with a simple python script. When SELECTING words with special characters they become encoded properly. But if I UPDATE words their encoding is just fucked up... The table is utf-8 encoded all the columns are utf-8 encoded. The php mysqli connection is utf-8 encoded. My HTML header is utf-8... WTF?

Come on, how hard can it be?

On every fucking TLV data structure I get to handle, the hobo who defined the structure obviously stopped reading the TLV specification after the second sentence.
Fucked up tags, misuse of length encoding, and as a result no real TLV parser can handle that crap. Workarounds and manual parsing all over the place for *every* *single* interface.

Get your shit together, and if you don't want to handle the complex parts, then at least make the simple types right.

me Looks at beta 1.0 of encoding/encryption program

program Looks back

me :>

announcement bet1.0 is here

Hello, community, how do you relax between encoding, when you take a pause after hard task?)

3 hours later I discover why my unset destroyed our json.

"When encoding an array, if the keys are not a continuous numeric sequence starting from 0, all keys are encoded as strings, and specified explicitly for each key-value pair."

Thanks obama.

It could be seen as n bug, when i was busy with file exports from a query on our web system i mannaged to skrew up the bitstream encoding for excel files and with a sysout the code started beeping everytime i exported the file. Managed to fix it by changing it to csv format. So no more beeping :D

Fuck those weird encoding issues with Python! I've read the HowTo Unicode 10 or 20 times and I still got those 'ordinal not in range error'!!!

I was in my IDE writing up a new program, but my compiler didn't accept the encoding format of the files. Because I used special characters, it flipped a lot of encodings around. Then I changed my projects master encoding, then all of the source files they all messed up. The all the files were composed of "????????" I went to a backup and they were messed up too. I exploded... Control Z couldn't even be my hero this time.

NTSC vs PAL

First time would be when I had two teachers working of a bug that resultet in the CSS not loading / updating properly. It took me around 1 hour as well to realize that the text file was encoded with some obscure windows encoding, that doesn't load chrome (and maybe other browsers) can't load properly.

Thanks notepad++, will never use you again for web development.

Second time would be when I presented an app I made too the same teachers in order to boost my mark. Bugs that never appeared before every where and the basic functionality of the app was questionable. I hadn't worked on that app for half a year and had to fix as much as possible in a short amount of time, which might have made it worse.

Don't ask why but thanks to my knowledge about the france language I got a better mark, even though it was spanish

OK, so I've been working on processing a Japanese dictionary file and things are going smoothly for the most part. Out of ~185,000 entries, I've got 35 that are still causing problems.

The error I'm getting is "Incorrect string value '\xF0\xA4\xAD\xAF' for column...". I've checked all of my encoding and collation settings, and I'm pretty sure I've got it set to properly implement all of Unicode (as well as it does, anyway), as shown in the image attached. My suspicion is the problem characters are likely among the JIS X 0213 character set; in either case we're clearly dealing with a 4-byte character encoding issue here.

If needed I can attach a flag in the database and base64 encode these particular entries so the data isn't lost, but I'd like to just get it to handle the data properly in the first place if possible.

Anyone have any ideas on other items I can check to resolve the error?

How's that character encoding going McDonalds?

Is it just me or is encoding in 15.9.3 broken AGAIN when using .editorconfig to set the charset?.......

GOD DAMN THAT OLD BROKEN DATABASE.

Having to work with a legacy old database system running MySQL 4.0 is a pain. Especially when even finding a frikin manual for the thing is hard af.

And a cherry on top is dealing with encoding and collation in a system, that didn't really have a wide support for it yet.

10/10. At least I am only dealing with it so that we could later shut it down for good.

when you find a single invisible character in each of your sql files in visual studio that causes a sql implosion saying

invalid syntax near ''

Sublime Text 3 and other tools didn't help. required manually finding this time, in 20 files..... with an overdue project.

I wanted to burn visual studio to the ground

i can never understand the theme behind kotlin.

THEY DON'T HAVE A FUCKING TERNARY OPERATOR!!![?:]

Like before realizing this, i thought yeah jetbrains has decided to make android development a privileged hobby and non beginner friendly , so its now creating an encoding like language, in the false theme of " reducing code size"

But now they remove WORLDWIDE KNOWN, OPTIMIZED , EASY TO READ AND USE AND UNDERSTAND FEATURE of ternary operator and replacing it with less powerful but same looking elvis operator.( and stating that using if else for that is a better option)

Like why? if your goal is to make a shitty encoding language that makes everything shorter and most of the things optional, why remove the already efficient if else encoder?

God knows when this stupid language is going to stop my brain from getting blasted

Fucking encoding types everywhere, spending nearly half of the day with guessing which encoding type is correct for some freaking files 😑

Fuck powershell.

How in the world can one lang have so much problems with date conversion, and chars like üöä. Its so funny how they get through just fine sometime when u specify encoding utf8 but other times it has no ides wha5 these chars are.
Honestly fuck powershell

For those who hate to like PHP. I have an issue. Character encoding.

A semicolon ' is replaced by ’

A fullstop . is replaced by ' Â

Any ideas?

Python needs a tool called 2to23.

Also, 2to3 does NOT take into account things like byte encoding changes. Kind of annoying.