UTF-8 gone wrong

I got 99 problems but encoding ain’t one

I have got 99 problems and emoji encoding ain't one of them.

P.S. /r/softwaregore

string excuses[]={
"it's not a bug it's a feature",
"it worked on my machine",
"i tested it and it worked",
"its production ready",
"your browser must be caching the old content",
"that error means it was successful",
"the client fucked it up",
"the systems crashed and the code got lost" ,
"this code wont go into the final version",
"It's a compiler issue",
"it's only a minor issue",
"this will take two weeks max",
"my code is flawless must be someone else's mistake",
"it worked a minute ago",
"that was not in the original specification",
"i will fix this",
"I was told to stop working on that when something important came up",
"You must have the wrong version",
"that's way beyond my pay grade",
"that's just an unlucky coincidence",
"i saw the new guy screw around with the systems",
"our servers must've been hacked",
"i wasn't given enough time",
"its the designers fault",
"it probably won't happen again",
"your expectations were unrealistic",
"everything's great on my end",
"that's not my code",
"it's a hardware problem",
"it's a firewall issue",
"it's a character encoding issue",
"a third party API isn't responding",
"that was only supposed to be a placeholder",
"The third party documentation is wrong",
"that was just a temporary fix.",
"We outsourced that months ago.","
"that value is only wrong half of the time.",
"the person responsible for that does not work here anymore",
"That was literally a one in a million error",
"our servers couldn't handle the traffic the app was receiving",
"your machines processors must be too slow",
"your pc is too outdated",
"that is a known issue with the programming language",
"it would take too much time and resources to rebuild from scratch",
"this is historically grown",
"users will hardly notice that",
"i will fix it" };

Ｔｈｅ　ｔｈｉｎｇ　ａｂｏｕｔ　ＵＮＩＣＯＤＥ　ｉｓ　ｔｈｅ　ａｂｉｌｉｔｙ　ｔｏ　ｍａｋｅ　ｉｔ　ｈａｒｄ　ｔｏ　ｐａｒｓｅ　ｗｈａｔ　ｈｕｍａｎｓ　ａｕｔｏｍａｔｉｃａｌｌｙ　ｓｅｅ　ａｓ　ＡＳＣＩＩ

"Ok, the site looks fine. Now let's move the style tag into it's own file."

*makes css file*

"WHY DOES IT HARDLY EVER LOAD!?!?, I checked the syntax trice"

*Spends 20 min. Asking friends for help, but none of them knows a reason*

"Time to ask the teacher, I guess"

*Teacher comes over, but has no clue either*

Teacher: "Give me the files, let's test it on my laptop"

*Css doesn't load there either*

*Teachers pair programming and trying some serious debugging technics. No progress*

*I decided to look at the sourcecode while refreshing the site*

1. Refresh: Css is loaded properly
2. Refresh: Css is gone, and source turned into various asian symbols.

Looks at the (default) file encoding: UCS-2
WTF NOTEPAD++, I SPEND 2 HOURS OF MY LIFE BECAUSE YOU DECIDED THIS WAS A PROPER ENCODING!

Web programming seems fun.

Not sure if it just doesn't have the right character encoding scheme, or if my existence is being questioned...

Notepad finally adds supports Unix-encoding

Now THAT is good news!

stolen from somewhere...

Intelligence and ability cannot be measured by education.

I have a client who asked a Master in Computer Science to develop a small system, for querying product title and their code. The guy used python, vanilla js, and... Txt file for the "database". Then my client asked me to integrated this in... WordPress.

This was in 2016. And idiot as I'm, I agreed and adapted his code to use php and a database.

April this year, my client said they are still using the python system to add new products all this time, in parallel. And wanted to update the WordPress with the data.

- No problem! - I said. Just send me the SQL file.

So the Master in CS sent me a SQL coded in ANSI. I asked for the SQL again, but with a more appropriate encoding. He took 1 month to reply back, and said it would be better if I get rid of the database and just use the txt file for querying.

This is outrageous.

I really hate people who are educated but completely useless.

Encoding issues always give me �headache�.

Me: I have an input stream!
Library: I want a file.
Me: I can see your code, you will convert it to an input stream! Don't any of your constructors take an input stream?
Library: No. I want a file. Okay, you can also include an encoding.
Me: I don't care about encoding.
Library: Fine, just give me the file then.
Me: You mean I have to somehow convert my input stream into an actual file so I can give it to you?
Library: You're gradually catching on, yes.
Me: Can I add a new constructor myself that takes an input stream?
Library: Sure! Good luck in getting it approved by the maintainers and the new jar distributed everywhere, including Maven Central before your deadline.
Me: Fine, I'll just rearchitect everything so I can give you an actual file.
Library: And then everything will be fine.
Me:

My stickers just arrived. And I especially love the encoding of “ß” within the street name.

German. It's hard.

Next time, just use upper letter and replaced “ß” with “SS” or just use a “B”. The B is especially fine to drive typographers mad.

*wrestling commentator voice*
"In this weeks episode of encoding hell:
The iiiinnnfamous UTF-8 Byte Order Mark veeeersus PHP!"

For an online shop we developed, there is currently a CSV upload feature in review by our client. Before we developed this feature, we created together with the client a very precise specification, including the file format and encoding (UTF-8).

After the first test day, the client informed us, that there were invalid characters after processing the uploaded file.
We checked the code and compared the customer's file with our template.
The file was encoded in ISO-8859-1 and NOT as specified UTF-8.
But what ever, we had to add an encoding check, thus allowing both encodings from now on.

Well well well welly welly fucking well...

Test day 2: We receive an email from said client, that the CSV is not working, again.
This time: UTF-8 encoding, but some fields had more colums with different values than specified.
Fucking hell.
We tell the customer that.
(I was about to write a nice death threat novel to them, but my boss held me back)

Testing day 3, today:
"The uploading feature is not working with our file, please fix it."
I tried to debug it, but only got misleading errors. After about 30 minutes, at 20 stacks of hatered, I finally had an idea to check the file in a hex editor:

God fucking what!?!!?!11?!1!!!?2!!

The encoding was valid UTF-8, all columns and fields were correct, but this time the file contained somthing different.
Something the world does not need.
Something nearly as wasteful as driving a monster truck in first gear from NYC to LA.

It was the UTF-8 Byte Order Mark.
3 bytes of pure hell.
Fucking 0xEFBBBF.
The archenemy of PHP and sane people.

If the devil had sex with the ethernet port of a rusty Mac OS X Server, then 9 microseconds later a UTF-8 BOM would have been born.

OK, maybe if PHP would actually cope with these bytes of death without crashing, that would be great.

I think this is so far one of the most priceless WTF moments I encountered at my current work:

A coworker of mine came up to me explaining the problem he had with russian characters in the filename. He explained in detail that everything works ok (the other part of the code he was fixing) if he changes the name of the file to test1.xlsx for example which doesn't use russian characters. OK great.

Then he goes on to show me how he fixed the other stuff and of course everything blows up. The file he used for demonstration was of course the original file our cusotomer provided, he just deleted the obvious russian chars and left the rest.

МТС != MTC

I cracked up: but you still have russian chars in the name.
The guy: no way, I deleted them all.
Me: but what about that МТС in the name?! Guy: what about it?
Me: did you actually typed that in or you left it there?! Those are russian chars that are fucking things up for you.
Guy: no way, it's MTC.
Me: checked the logs, you have ??? In the filename instead of МТС..don't you find that at least a little bit suspicious?!
Guy: but it looks the same. How does it (the computer) know it is in russian?!? //Why doesn't it understand?!

O.o I still can't believe it.. Is it just me & my high standards, or should it be normal for coders to know things such as character encoding & stuff?!?

I almost died of laughter, he and some other guy had problems finding customers in the software due to not being able to type the russian chars << happened more then once before, even after I told them about a quick hack on how to use google translate onboard keyboard & other stuff to make proper chars so they can get a match..

I think when they bury me, I'll still be facepalming and laughing over this incident. 🤣🤣🤣🤣🤣🤣🤣

EXCEL YOU FUCKING PIECE OF SHIT! don't get me wrong, it's usefull and kt works, usually... Buckle up, your i for a ride. SO HERE WE FUCKING GO: TRANSLATED FORMULA NAMES? SUCKS BUT MANAGABLE. WHATS REALLY FUCKED UP IS HTHE GERMAN VERSION!
DID YOU HEAR ABOUT .csv? It stands for MOTHERFUCKING COMMA SEPERATED VALUES! GUESS WHAT SOME GENIUS AT MICROSOFT FIGURED? Hey guys let's use a FUCKING SEMICOLON INSTEAD OF A COMMA IN THE GERMAN VERSION! LET'S JUST FUCK EVERY ONE EXPORTING ANY DATA FROM ANY WEBSITE!

The workaround is to go to your computer settings, YOU CAN'T FUCKING ADJUST THIS IN EXCEL!, change the language of the OS to English, open the file and change it back to German. I mean, come on guys, what is this shit?

AND DON'T GET ME STARTED ON ENCODING! äöü and that stuff usually works, but in Switzerland we also use French stuff, that then usually breaks the encoding for Excel if the OS language is set to German (both on Windows and Mac, at least they are consistent...)

To whoever approved, implemented or tested it: FUCK YOU, YOU STUPID SHITFUCK, with love: me

I sometimes remember the time when I wrote a Email-inbox-exporter-PHP-script-type of application that collects all the emails from an inbox, "copied" it to a database with the attachements and stuff and moves it to a folder..

I just started at the company for like a couple of months, had no privileges to create mailboxes and such and I didn't want to interrupt our programmer to do this for me, so... I decided.. to save time and resources.. to test run it on our global, live 'support' mailbox.. :D Well.. You might guess what happened.. Apparently I mistyped the name of the move-destination folder (because imap-weird-things) that resulted in a completly empty mailbox and an empty database because the inserts failed due to bad encoding and mime-type issues..

The moment I refreshed my Outlook and noticed that all our mails where gone.. I swear, I can't describe that feeling of fear, cold sweat, intense heartbeat... I just stood up, asked if anyone wanted coffee, and just walked out of the office.. When in the hallway, I heard my collegues ask to one another "do you have any issues with outlook, all my mails are gone?". Everyone was stressing out, the chief was stressing out "what happened?!", nobody knew what happened.. :D

They could partially resolve it via one collegue who hadn't refreshed the mailbox and he could forward all the mails back to our support mailbox..

I dropped the project idea and learned to work with dev environments :D A couple of months later, I accidentially forgot a where condition in my SQL UPDATE statement, but that was the last time I seriously f*cked up.. :D Got to learn the hard way I guess.. Now everything I do runs in dev environments, I test everything before publishing,.. When I look back.. I don't even recognize the (inexperienced) guy I was back then ! :D

Ps. No one still knows what happened that day and they blamed it on server issues :D

Today I'm trying to study how to encode data in idx-ubyte format for my machine learning project.
Professors I'm going to astonish you!

Good day and good coding to all of you! :)

FUCKING ENCODING SHITHEADS!
WHY ISN'T UTF-8 THE FUCKING DEFAULT EVERYWHERE!
§¥~@#&•…≈!

Well it seems that i broke something..

Encoding strikes yet another victim

So, California, land of no rain whatsoever.

Until today, where it actually rained!

Also today, a weather applet for tmux started misbehaving. Suspicious.

Coincidence, I think not! Sure enough, stopping that applet from running fixed everything. Turns out, the raining icon breaks the terminal, for some strange reason.

And why had this not been discovered earlier? This had been the first time it had rained for at least 8 months, and I added this only 4 months ago. Thus, it had never had its time to shine.

Yay for strange encoding!

Anyway, I now have a new excuse; blame bad code on global warming.

!dev - cybersecurity related.

This is a semi hypothetical situation. I walked into this ad today and I know I'd have a conversation like this about this ad but I didn't this time, I had convo's like this, though.

*le me walking through the city centre with a friend*
*advertisement about a hearing aid which can be updated through remote connection (satellite according to the ad) pops up on screen*

Friend: Ohh that looks usefu.....
Me: Oh damn, what protocol would that use?
Does it use an encrypted connection?
How'd the receiving end parse the incoming data?
What kinda authentication might the receiving end use?
Friend: wha..........
Me: What system would the hearing aid have?
Would it be easy to gain RCE (Remote Code Execution) to that system through the satellite connection and is this managed centrally?
Could you do mitm's maybe?
What data encoding would the transmissions/applications use?
Friend: nevermind.... ._________.

Cybersecurity mindset much...!

Finally Spend two fucking days debugging shit until I figured it it. Freaking stupid shit encoding problems and old data combined isn't fun. Dafuq why can't everybody use UTF-8 or Unicode or something else but PLEASE stop using some old school IBM shit codepages.
Leckt mich doch am arsch mit diesem scheiß man -_-

Testing hell.

I'm working on a ticket that touches a lot of areas of the codebase, and impacts everything that creates a ... really common kind of object.

This means changes throughout the codebase and lots of failing specs. Ofc sometimes the code needs changing, and sometimes the specs do. it's tedious.

What makes this incredibly challenging is that different specs fail depend on how i run them. If I use Jenkins, i'm currently at 160 failing tests. If I run the same specs from the terminal, Iget 132. If I run them from RubyMine... well, I can't run them all at once because RubyMine sucks, but I'm guessing it's around 90 failures based on spot-checking some of the files.

But seriously, how can I determine what "fixed" even means if the issues arbitrarily pass or fail in different environments? I don't even know how cli and rubymine *can* differ, if I'm being honest.

I asked my boss about this and he said he's never seen the issue in the ten years he's worked there. so now i'm doubly confused.

Update: I used a copy of his db (the same one Jenkins is using), and now rspec reports 137 failures from the terminal, and a similar ~90 (again, a guess) from rubymine based on more spot-checking. I am so confused. The db dump has the same structure, and rspec clears the actual data between tests, so wtf is even going on? Maybe the encoding differs? but the failing specs are mostly testing logic?

none of this makes any sense.
i'm so confused.

It feels like i'm being asked to build a machine when the laws of physics change with locality. I can make it work here just fine, but it misbehaves a little at my neighbor's house, and outright explodes at the testing ground.

One of the things I love the most about programming is that sometimes you feel like you're taking a step forward when you're taking a step back... and sometimes you feel like you're taking a step back when you take a step forward.

Or maybe that's just me. XD

My teacher thinks encryption and encoding are same

Uggg..... I'm trying to encode a binary file in Python which may be an image or may be an executable, and then decode it back into a file (I plan on editing it in the middle, but baby steps for now..) but nothing is working!!

My plan is to:

Open binary file.

Decode as base64, or something else that could easily handle binary.

Convert byte data to string (for editory perpousos - I won't be editing bytes, I'll be doing custom encoding but that's irrelevant for this test)

Convert back to a byte string/array (with .encode(), probably)

Write to file.

I do this, yet the output has been altered... Though I haven't touched anything..

It's so enfuriating.. x.x

Woke up this morning to a fucking giant snowstorm and my first reaction was 'fml' , poured some coffee , lit a smoke and started checking my work mail 'Issue xxxx response : Not solvable '...what the...I go through the files on my phone , look at what that issue was : lack of proper validation , filtering and encoding of input thus enabling xss . Not solvable my ass ...simply adding literally 3 more characters to that fucking retarded filter would stop all the bypasses . This issue is a showstopper for their project and that is what they answer ?
Sorry to indians out here but some of your colleagues are as stupid and unimaginative as they can possibly ever come .

Because nothing says "security" like some good ol' Base64 encoding. Bet whoever wrote that code was wearing mirror shades.

Yesterday a strange bug appeared in Chrome: In a small web app we have some umlaut characters like äöü. Strangely said characters were displayed as cyrillic, but only on my pc... On every other device it worked. I spent about 5 hours of checking encodings (everything was in UTF-8), reading posts in the almighty stackoverflow.
Finally i figured out, the font was broken. After reinstalling it, everything was peaceful again in my head.

So someone here had the job to merge the current version of our software into another older repo.
(That older repo has some features for future versions, that should be only in there ... don´t ask ... not my decision)

Took him long enough. But he forgot to check one thing while merging: The encoding of the files.

And we´re german...

ALL umlauts, and all other special chars in the code were replaced with this: �
No global replace, because all chars were replaced with the same char.
(Why the fuck do we have special chars in the code in the first place???)

So to not need him to start all over again I compiled a list of common german words containing umlauts and did a global search and replace.

I think I got 90% of the errors like that.
Now he´s going to correct the rest of the errors.

Fuck the comments in the code though.
Just a waste of time...

I was working on this mustachio fork called Morestachio and changed the Default Encoding from "Encoding.Default" to UTF8 while creating a .NetStandard version of it. After i have created everything i thought "Hey just for the lolz run my tests again" ...

Just imagine the world where we have CAPITALIZED DIGITS.

TL:DR
Why do so shitty "API"s exist that are even harder to write than proper ones? D:

Trying to hack my venilation at home.
This API is so horrible D:
The API is only based on POST requests no matter if you want to write values or get values and the response only contains XML with cryptic values like:

<?xml version="1.0" encoding="UTF-8"?>
<PARAMETER>
<LANG>de</LANG>
<ID>v01306</ID>
<VA>00011100000000000000000010000001</VA>
<ID>v00024</ID>
<VA>0</VA>
<ID>v00033</ID>
<VA>2</VA>
<ID>v00037</ID>
<VA>0</VA>

Also there are multiple API routes like
POST /data/werte1.xml
POST /data/werte2.xml
POST /data/werte3.xml
POST /data/werte4.xml
And actually the real API route is only given in the request body and not in the path.

Why is this so shitty? D:<

Btw in terms of security this is also top notch. It just globally saves if one computer sends the login password.
I mean why even ask for a password then? D:
That made me end up with a cronjob to send a login request so I don't have to login on any device.

PS:
You see, great piece of German engineering.

The senior developer swore the server was responding with a 500 because I, the co-op, was definitely sending the 'wrong encoding' whenever I post special characters. He said, "I'm only taking your data and putting it straight into the database!!!"

I found out it was breaking because he was converting the JSON payload to a STRING and simplistically separating the keys and values by colons and double quotes. Yuuup. Nothing can go wrong there...

Just had to do this. Homemade. Sorry :D

When you encounter a character encoding issue.

UTF-8 without BOM.
{ ä, ö, ü } are not going to be decoded correctly.
Why the fuck does BOMless UTF-8 still exist? Such a pain in the ass.

Guess I'll fuckin try again tomorrow.

Building a cross platform c program. On Linux side, just using a makefile. Today I tried using visual studios "clone" feature for git. It just downloads the files and makes them available to the editor, it doesn't make a project, obviously.

But this has some disadvantages. For one, you can't build, or run. Two, you don't get any project properties. My project needs to set the character encoding to Unicode. Can't do that without a project.

So I use their tool to create a project from existing code. It didn't really work. The build profiles were janky at best and I still couldn't set the character encoding.

Ended up just deleting the whole thing.

Currently working with @Kreischo and another good friend to create a secured, encoded container to store files in it.

I am currently working on the frontend of things, thinking it's quite beautiful.. (Done with Electron)

Your opinion?

Hmm...recently I've seen an increase in the idea of raising security awareness at a user level...but really now , it gets me thinking , why not raise security awareness at a coding level ? Just having one guy do encryption and encoding most certainly isn't enough for an app to be considered secure . In this day an age where most apps are web based and even open source some of them , I think that first of all it should be our duty to protect the customer/consumer rather than make him protect himself . Most of everyone knows how to get user input from the UI but how many out here actually think that the normal dummy user might actually type unintentional malicious code which would break the app or give him access to something he shouldn't be allowed into ? I've seen very few developers/software architects/engineers actually take the blame for insecure code . I've seen people build apps starting on an unacceptable idea security wise and then in the end thinking of patching in filters , encryptions , encodings , tokens and days before release realise that their app is half broken because they didn't start the whole project in a more secure way for the user .

Just my two cents...we as devs should be more aware of coding in a way that makes apps more secure from and for the user rather than saying that we had some epic mythical hackers pull all the user tables that also contained unhashed unencrypted passwords by using magix . It certainly isn't magic , it's just our bad coding that lets outside code interact with our own code .

Today I learned about binary encoding formats alternative to JSON such as Google Protocol Butter.
I like these binary formats.
Just thought I would share this here so others would benefit as well (and please share your experience if it is relevant)

🎼Up and down
🎶 And all the way around
🎶 Crossentropy is a bitch
🎶 Epoch and loss are still the same
🎵

Edit: I left this rant open for half an hour. Finally figured that I should do one hot encoding and labeling doesn't exactly work out the way I was hoping it works out.

But hey, half an hour ago I was angry and desperate. So... ¯\_(ツ)_/¯

How deep does the rabbit hole go?

Problem: Convert numpy array containing an audio time series to a .wav file and save on disk

Error 1:
Me: pip install "stupid package"
Console: Can't pip, behind a proxy

Me: Finds workaround after several minutes

Error 2:
Conversion works, but audio file on disk doesn't work
Encoding Error only works with array of ints not floats
BUT I NEED IT TO BE FLOATS

Looks for another library
scikits.audiolab <- should work
Me: pip --proxy=myproxy:port install "this shit"
Command Line *spits back huge error*

Googles error <- You need to install this package with a .whl file
Me: Downloads .whl file <- pip install "filename".whl
Command Line: ERROR: scikits.audiolab-0.11.0-cp27-cp27m-win32.whl is not a supported wheel on this platform.

Googles Error <- Need to see supported file formats
Me: python -c "import pip; print(pip.pep425tags.get_supported())"
Console: AttributeError: module 'pip' has no attribute 'pep425tags'

Googles Error <- Use another command for pip v10
Me: python -c "import pip._internal; print(pip._internal.pep425tags.get_supported())"
Console: complies

Me: pip install "filename".whl
Console: complies

Me: *spends 30 minutes to find directory where I should paste .dll file*
Finds Directory (was hidden btw), pastes file

Me: Runs .py file
Console: from version import version as _version ModuleNotFoundError: No module named 'version'

Googles Error <- Fix is: "just comment out the import statement"
Me: HAHAHAHAHAHA
Console: HAHAHAHAHA

Unfortunately this shit still didn't work after two hours of debugging, lmao fuck this

The only way I would ever use a recovery question.
BTW nice encoding there...

When you Render something and the name is just perfect!

Translation: Catched encoding buying Toilet Paper

All it takes is one jackass somewhere in the world with no understanding of character encoding to ruin things for everyone else.

So for those of you keeping track, I've become a bit of a data munger of late, something that is both interesting and somewhat frustrating.

I work with a variety of enterprise data sources. Those of you who have done enterprise work will know what I mean. Forget lovely Web APIs with proper authentication and JSON fed by well-known open source libraries. No, I've got the output from an AS/400 to deal with (For the youngsters amongst you, AS/400 is a 1980s IBM mainframe-ish operating system that oriiganlly ran on 48-bit computers). I've got EDIFACT to deal with (for the youngsters amongst you: EDIFACT is the 1980s precursor to XML. It's all cryptic codes, + delimited fields and ' delimited lines) and I've got legacy databases to massage into newer formats, all for what is laughably called my "data warehouse".

But of course, the one system that actually gives me serious problems is the most modern one. It's web-based, on internal servers. It's got all the late-naughties buzzowrds in web development, such as AJAX and JQuery. And it now has a "Web Service" interface at the request of the bosses, that I have to use.

The programmers of this system have based it on that very well-known database: Intersystems Caché. This is an Object Database, and doesn't have an SQL driver by default, so I'm basically required to use this "Web Service".

Let's put aside the poor security. I basically pass a hard-coded human readable string as password in a password field in the GET parameters. This is a step up from no security, to be fair, though not much.

It's the fact that the thing lies. All the files it spits out start with that fateful string: '<?xml version="1.0" encoding="ISO-8859-1"?>' and it lies.

It's all UTF-8, which has made some of my parsers choke, when they're expecting latin-1.

But no, the real lie is the fact that IT IS NOT WELL-FORMED XML. Let alone Valid.

THERE IS NO ROOT ELEMENT!

So now, I have to waste my time writing a proxy for this "web service" that rewrites the XML encoding string on these files, and adds a root element, just so I can spit it at an XML parser. This means added infrastructure for my data munging, and more potential bugs introduced or points of failure.

Let's just say that the developers of this system don't really cope with people wanting to integrate with them. It's amazing that they manage to integrate with third parties at all...

Encoding problems on live TV in portugal...

Writing hebrew in Latex using a template that doesn't support UTF-8 is the most archaic shit I've done lately. I feel like some sort of a caveman.
This fucking encoding inverts all letters so it can support right to left. 😓

Oh man, I found the names for the different number bases past 36! I should expand my base converter!

That doesn’t... look right...

Ah, because these ones require the use of encoding algorithms.

NOPE NOPE NOPE NOPE

Someone building the iPad software for this beer museum did Not think about the encoding xD

i wrote a website, a server in go, a small os in c, a game in js, a game and server and web scraper and other desktop apps in java, mobile apps with flutter, a website with php also, implemented aes in go, wrote a parser in java. done sysadmin stuff on my vps and pihole/openvpn/nextcloud on my rpi. learn about c vulnerabilities and used metasploit. attempted to write an interpreted language. did some led displays with arduino. currently learning tensorflow.

i have never...
- written a driver
- made a game with a game engine
- created a file encoding
- implemented an oauth2 server
- made an api
- worked with vr

what am i missing? i want to be a very well rounded dev.

Over the summer I was recruited to be a supplement instructor for a data structures course. As a result of that I was asked (separately by the professor) to be a grader for the course. Because of pay limitations I've mostly been grading homework project assignments. In any case, it's a great job to get my foot into the department and get recognized.

Over the course of the semester I've had this one person, OSX, named after their operating system of choice, who has been giving me awkward submissions. On the first assignment they asked the professor for extra time for some reason or the other, and that's perfectly fine.

So I finally receive OSX's submission, and it's a .py file as per course of the course. So I pop up a terminal in the working directory and type "python OSX_hw1.py". Get some error spit out about the file not being the right encoding. I know that I can tell python to read it in a different encoding, so I open it up in a text editor. To my surprise it's totally not a text file, but rather a .zip file!

I've seen weirder things done before, so no big deal. I rename the file extension, and open it up to extract the files when I see that there's no python files. "Okay, what's goin on here OSX..." I think to myself.

Poking around in the files it appears to be some sort of meta-data. To what, I had no clue, but what I did find was picture files containing what appeared to be some auto-generated screenshots of incomplete code. Since I'm one to give people the benefit of doubt even when they've long exhausted other peoples', I thought that it must be some fluke, and emailed OSX along with the professor detailing my issue.

I got back a rather standard reply, one of which was so un-notable I could not remember it if my life depended on it. However, that also meant I didn't have to worry about that anymore. Which when you're juggling 50 bazillion things is quite a relief. Tragically, this relief was short lived with the introduction of assignment 2.

Assignment 2 comes around, and I get the same type of submission from OSX. At this time I also notice that all their submissions are *very* close to the due time of 11:59pm (which I don't care about as long as it's in before people start waking up the next morning). I email OSX and the professor again, and receive a similar response. I also get an email from OSX worried about points being deducted. I reply, "No issue. You know what's wrong. Go and submit the right file on $CentralGradingCenter. Just submit over your old assignment".

To my frustration OSX claimed to not know how to do this. I write up a quick response explaining the process, and email it. In response OSX then asks if I can show them if they comes to my supplemental lesson. I tell OSX that if they are the only person, sure, otherwise no because it would not be a fair use of time to the other students.

OSX ends up showing up before anyone else, so I guide them through the process. It's pretty easy, so I'm surprised that they were having issues. Another person then shows up, so I go through relevant material and ask them if they have any questions about recent material in class. That said, afterwards OSX was being somewhat awkward and pushy trying to shake my hand a lot to the point of making me uncomfortable and telling them that there's no reason to be so formal.

Despite that chat, I still did not see a resubmission of either of those two assignments, and assignment 3 began to show it's head. Obviously, this time, as one might expect after all those conversations, I get another broken submission in the same format. Finally pissed off, I document exactly how everything looks on my end, how the file fails to run, how it's actually a zip file, etc, all with screenshots. That then gets emailed to the professor and OSX.

In response, I get an email from OSX panicking asking me how to submit it right, etc, etc. However, they also removed the professor from the CC field. In response I state that I do not know how to use whatever editor they are using, and that they should refer to the documentation in order to get a proper runnable file. I also re-CC the professor, making sure OSX's email to me is included in my reply.

OSX then shows up for one of my lessons, and since no one had shown up yet, I reiterate through what I had sent in the email. OSX's response was astonished that they could ever screw up that bad, but also admits that they had yet to install python(!!!). Obviously, the next thing that comes from my mouth is asking OSX how they write their code. Their response was that they use a website that lets them run python code.

At this point I'm honestly baffled and explain that a lot of websites like those can have limitations which might make code run differently then it should (maybe it's a simple interpreter written on JavaScript, or maybe it is real python, but how are you supposed to do file I/O?) .

After that I finally get a submission for assignment 1!

Superhuman capabilities.

Friend - Hey is this file correctly encoded ?

Me - Hell Yes. I can decode visually from an IMAGE and make sense of underlying data encoded.

Some people think I am so awesome coder I can look at Huffman encoded result and understand if it's correct or not.

Go fmt yourself? I guess. Taken from Golang's documentation on encoding/json.

API provider: include a signature based on these fields in this order. DO NOT ENCODE IT!

Implementation works a while, then..

*a wild apostrophe appears*

Signature no longer works.

API Provider: "oh, yeah we escape those."

Arrghhghghghhhghvhxmchsoxnsoxnwl

Not only is it a poor design for signing payloads, the documentation is shockingly poor in it.
Even the implementation example (which is supposedly from their code) doesn't account for any type of escaping or encoding.

Before anyone asks, I can't into details about the implementation.

Just found out the API of our zentracloud sensors is sending the units with a space before the actual unit. Couldn't figure out for half an hour why Doctrine is not finding the unit in the database. Encoding? JSON decoding? Character itself? Screw you. Screw you...

Working with a vendor's api. Every call is coming back error. Emails back and forth for two weeks, sending logs, changing settings, encoding urls...

From them: oh, this field needs to be unique.

Me: how are you just now catching that?!

The doc showed it to be a specific value. We must be their first client or something.

Let's focus/master computer architectures, coding paradigms, datastructures. Everything comes after that...the problem with todays academics is that they are more focused on immediately deploying students to industry; theyre more focused on teaching specific frameworks and specific language instead of teaching how things work...i bet most students (at least in my country) are having troubles with endianess or encoding or even byte manipulation or what a thread is....If im going to be the teacher for example of an oop subject, ill let the student choose the language they want as long as the oop paradigm is intact ,it will be fine.. i dont friggin care whether you know vue or angular or swing if you dont even know what a callback is..

Why in the everliving freaking fuck does Java have a class called URLEncoder which DOES NOT DO URL ENCODING??? The shit-spraying piece of a mouldy footgun does formencoding?? Which sadistic maniac thought that was a good idea?

Every flipping time when receiving file with win1251 charset

I've seen some of the videos that I downloaded from the internet are only playable through one particular application. How did they do it? How can I implement the same thing with my application? What technology should I research on to know more about this?

If I were to write a de/encoding algorithm, that nobody but me knows about, would it be safe?

OK what the actual fuck is going on within this company.

TL;DR: Spaghetti Copy/Pasted code that made me mad because it's just a mess

I just looked into a code file to search for a specific procedure regarding the creation of invoices.
I thought "Oh this is gonna be a quick look-through of like 1000 lines MAX" turns out this script is 11317 fucking lines long and most of it's logic is written there multiple (up to 6-7 times). And I'm not talking about a simple 10 lines or something. No! Logic of over 300 lines.. copy & pasted over .. and over .. and over?! I mean what the fuck did this guy drink when he wrote this.

Alsooo 10000 of those 11317 lines is ONE FUNCTION.. I kid you not! It's just a gigantic if / else if construct that, as I said before, contains copy-pasted code all over the place.
Sadly my TL thinks that code cleanup / optimization is "not necessary as long as it works" like wtf dude. If anyone wants to ever fix something in this mess or add a new feature they take a few hours longer just to "adjust" to this fucking shit.
This is a nightmare. The worst part: This is not the only script that has shit like this. We got over 150 "modules" (Yeah, we ATTEMPTED something OOP-ish but failed miserably) that sometimes have over 15000 lines which could be easily cut down to 1/3 and/or splitted into multiple files.
Let's not start about centralization of methods or encoding handling or coding standards or work code review or .. you get the point because there's a character limit for one rant and I guess I'd overshoot that by a lot if I'd start with that. Holy shit I can't wait until my internship is over and I can leave this code-hell!!

I know, I said it before. But Python is sooo shit. Sooo shit! Latest experience: the standard json library has it's syntax broken going from 3.8 to 3.9. A service just crashed after update, complaining it does not know what encoding means. Real languages log a warning that syntax changes are coming in next full version. Python just goes bang and burns after a minor version update. Not a tool, just a toy. Is this what they learned after fucking up so epically royally going from 2 to 3?

Trying to work out an encoding bug after upload, I ask them to send me the original version of the file that is currently uploaded. I look at the files and they seem strangely similar, but strangely different.

I reply to them, "To confirm, this is the version you uploaded?"

"Nope."

So the guy who i mentioned previously with his shitty csv made webservice which i needed to use.

It returned utf-8 encoded xml:
- in file the header contained central europe encoding
- the xml had more roots
- basically he only put his csv format inside xml tags
- csv contained html tags but there wasnt cdata

Now work with this lol

I neded to cut this shit with string functions and also some numeric data get with regexp from htm table td's

Whyyyyyyyyyyyy

Use .editorconfig files when working with multiple developers. Otherwise your file encoding and code style will get fucked up eventually.

Just getting a pizza, can't help but get annoyed by the jittery as fuck newsticker, they didn't even get the encoding right....

(Yes our POS software has 60fps jitterless hardware accelerated fully utf-8 encoded newstickers before anyone asks me to do a better job)

Been working on this project for the last 9 months, it's 1000 times bigger than the original spec, but the worst part of the whole experience is the god damn Microsoft Access database and migrating data from it.

Wanky encoding, references that aren't actually linked up, ID's that aren't unique, meaningless columns and table names.

Pushed me to the edge.

First experience with utf-16 and all I can think is "whatthefuckwhatthefuckwhatthefuckwhatthefuck"

alright so there's ICWH which means what it means; I call it releasing the dragon, if you catch my meaning.

Dopesmoking aside, I did some wonky Y'UV fiddle to encode RGBA data in that state of mind and because this augmented the compression ratio by a holy fuckton I said great, I'm amazing let's party.

But there was a problem: normal maps. They're not really color data, but rather directions in 3D space *encoded* as color. So my 420 encoder is great for an albedo or diffuse, but not normals.

So, what to do? Well, if we remember fucking vector math, then we know that the magnitude of a unit vector is exactly one -- normalized.

Therefore, we can encode only two elements and only the __sign__ of the third, because if given

d = 1 - ( pow(x, 2) + pow(y, 2) );

then

if(d > 0) { z = sqrt(d) * sign; };

and of course

sign = (z<0) ? -1 : 1;

You lost yet? Good. Normalpacking out of the way, we have the question of ambient occlusion, roughness, emission, metalness, edges, curvature, height, displacement... lots of grayscale maps, so why not pack them together? One map per RGBA channel and you have 2 images instead of 8.

BUT: these maps are ALSO not real color data, they're sort of these funky grayscale masks that we encode as color for convenience. So why not treat them as such?

We can have the code swap out encoding methods depending on the type of image, isn't knowing half a shit about the data convenient? So let's do that and give each individual mask less bits to begin with, depending on how much precision we need. Because if we have a

fac = {some pow of 2 <= 256};

and a

step = 1.0f/fac;

then why not just just fucking

v = clampf(v, -maxval, (maxval - step));

and then

(char) (roundf(v * step);

to force those pesky floats into byte-sized, rich-in-fucking-redundancy fixed point static ranges of values that'll fuck the entropy right in it's highly improbable ass?

Bitch you don't need millions of colors, matter of fact we could get away with barely two or even single bit for stuff like metallic or emission. If you had any idea how much smaller the file size is getting then you'd KNOW I'm going for the lossy compression overkill to get that certified, fucking authentic LOFI vibes.

Post processing, you can suck my mother fucking dick; I get them retro aesthethics for real. Since we're already reducing the color count by enormous magnitudes, let's step it up some more: about time we get into palettes, no?

Also because we're dealing with less than 8 bits per channel for every last value and this is the same for every image, then why not just go all out and pack them together -- accompany the increase in redundancy with increased density of data to make the added size of a dictionary be fucking worth it. And why not just put similar data together and pass THAT through a goddamned deflate?

Who could've thought! Knowing SOMETHING about the DATA makes DEALING with that DATA so very much EASIER! Does that make SENSE to YOU, you FUCKING rectal bleeding, fifty-gigs in size day-one update, anal bleaching lord of object oriented faggotry?

NO 64x64 _grunge-plasttr-wall_albedo06448(Copy).png in my drive shall remain untouched; I am become $:GT0x24;>, master of the pixel!

a lot of dev have a miss concept about Unicode/utf8 including me but I believe my understanding get Better and this my last version.
For a project i was developing a rest api for mobile app
when an ios dev asked me
"I send you Unicode string but it appears as ????? in admin web panel "
OMG!!!😨😨😨
Unicode is not an encoding nor an algorithm. it's a standerd which just map a glyph to a codepiont .
but utf8 is the encoding of Unicode and how it's stored or transferred ,
the string you send must be a utf-8 encoded string as the rest of the json you sent .

During a code review I was doing the dev wrote a function that takes a tweet, iterates over each letter, if the letter is not in a specified unicode range the letter is removed from the sentence. So now you have a sentence with missing letters... Wtf

Utf-string encoding in windows has been constantly fucking me up... It's been 4 hours and I'm still in the office alone.

Is there any program that can be used to cut out parts of a video file without re-encoding.

The way I'm thinking is if you already have a video file, you should be able to just copy the data from 1 file to another like a file splitter.

More from my big black book of ai and neuroscience:

I think if trace theory is true to any degree it would go some distance in explaining phenomenal consciousness, assuming I haven't misunderstood anything.

In fuzzy trace theory (FTT) it is posited that people form two types of mental representations about a past event:

*verbatim traces: detailed representations of a past event.

*gist traces: fuzzy representations of a past event.

People can reason with verbatim *and* gist traces but prefer gists.

*vision was suggested to work similarly in 1999. With human vision, two processes could be used: one that aggregates local receptive fields and one that parses the local receptive spatial field. It was suggested that people used prior experience, gists, to decide which dominates a perceptual decision.

Gist processes form representations of events, semantic details, where verbatim reinstates the context found in the surface details of an event.

__notes__

Parallel storage: asserts encoding/storage of verbatim/gist traces operate in *parallel*, not in serial.

I like to think of verbatim traces as databases, and gists as queries constructed by recognition.

Several studies have found that the meaning (gist) of an item is encoded even *before* the surface details (verbatim).

This might be important as a survival mechanism but should not be taken to mean strictly that gists are formed wholly *without* details or important and recognizable features of the item in question. It may well be for high level el processing and classification efficiency this may be an important reprocessing step, in the same way that many functions of the brain are duplicated throughout.

Finally got around to some real video encoding work on my new computer but noticing it's not blazing fast...

And more work is still handled by the CPU... But I thought video processing is handled by the GPU, which seems to be barely used at all. I'm using Handbrake but I thought the whole point of dedicated GPU was for intensive graphics and video processing?

A shitty platform that, although open source, there is no clearly documented way of setting a development environment for it. This pile of crap states clearly that it does NOT support RTL languages. One of the core business requirements is Arabic support. What to do? Look for other platforms? WRONG!

Base the fucking business on it and ask ME to see why the SQL database is not encoding the Arabic characters correctly and to look into the logs that back-end puked. My expertise is mobile development anyways damnit. Sure the backend code is Java code (Java jokers and haters, not the appropriate place) and I know it but there is no fucking way to test that motherfucker or to build it! No fucking testing server can be made! Only instructions to get a Docker image pulled and set up.

FML.

"This company is a fucking م."

I cannot believe I am so frustrated that I am ending this rant with a fun puzzle.

Hints to help you decipher the quoted sentence:

Hint 1: That Arabic letter is the perfect letter.
Hint 2: You don't need to be an Arab to understand what it means.

"So we know you came from a past environment of API development, healthcare encoding and micro-service architecture. With that, we have a shopify site that needs built in 2 weeks. Have fun (finger quotes) DEV".

BigBit standard for numeric format and character encoding

Never again will I use eclipses egit extension. First eclipse thought that my plain text java source code should be encoded in some bizarre occult way which made eclipse think its binary what made me try pretty much anything one can do with a .gitattributes file before a colleague suggested to not trust eclipse eith the encoding it was explicitly told to use, then I fetched another branch to merge them which somehow killed my .project file and forced me to delete and refetch the whole thing which led to eclipse not longer recognizing it as a java project. May it be because I'm to stupid to use my tools? Yeah, probably. But I'm done with egit, it's all console gitting from now on, fuck suggested practice.

Fuck encoding and fuck PHP!!!
I'm programming a little vocab trainer to get used to php and MySQL. From an old VB vocab trainer I had ca. 2000 txt-files with words and converted them to sql-queries with a simple python script. When SELECTING words with special characters they become encoded properly. But if I UPDATE words their encoding is just fucked up... The table is utf-8 encoded all the columns are utf-8 encoded. The php mysqli connection is utf-8 encoded. My HTML header is utf-8... WTF?

I was in my IDE writing up a new program, but my compiler didn't accept the encoding format of the files. Because I used special characters, it flipped a lot of encodings around. Then I changed my projects master encoding, then all of the source files they all messed up. The all the files were composed of "????????" I went to a backup and they were messed up too. I exploded... Control Z couldn't even be my hero this time.

Please, before exporting anything in whatever editor you use, check if it is in UTF-8. Today I didn't knew why my new font wasn't working in certain places and I later discovered that more than 9000 characters were replaced by the replacement character...

me Looks at beta 1.0 of encoding/encryption program

program Looks back

me :>

announcement bet1.0 is here

Fuck those weird encoding issues with Python! I've read the HowTo Unicode 10 or 20 times and I still got those 'ordinal not in range error'!!!

Sooo. My team and I have module we're supposed to be porting to async code and aiohttp will not work. The server keeps rejecting the byte payload, but if we use synch code like the requests library, it works fine. The code is like identical, the only difference is async. It's been really frustrating because another drop in async version of requests (httpx) works just fine! I don't want to use httpx, the rest of our codebase is already using aiohttp! We think the problem is with gzip encoding being handled incorrectly by aiohttp. I've reported the issue.

It could be seen as n bug, when i was busy with file exports from a query on our web system i mannaged to skrew up the bitstream encoding for excel files and with a sysout the code started beeping everytime i exported the file. Managed to fix it by changing it to csv format. So no more beeping :D

3 hours later I discover why my unset destroyed our json.

"When encoding an array, if the keys are not a continuous numeric sequence starting from 0, all keys are encoded as strings, and specified explicitly for each key-value pair."

Thanks obama.

Come on, how hard can it be?

On every fucking TLV data structure I get to handle, the hobo who defined the structure obviously stopped reading the TLV specification after the second sentence.
Fucked up tags, misuse of length encoding, and as a result no real TLV parser can handle that crap. Workarounds and manual parsing all over the place for *every* *single* interface.

Get your shit together, and if you don't want to handle the complex parts, then at least make the simple types right.

Has anyone maybe a link to HTTP security topics in general?

I find often breadcrumbs, like in several different attack possibilities, but nothing comprehensive.

Mostly regarding HTTP 1.1 / HTTP 2 (h2c) and proxying.

I'm currently unclogging an whole ecosystem of proxies, endpoints, edge nodes and so on...

My knowledge is limited and it's frustrating to Google cause seemingly I get always just pieces of the puzzles but not a collection -.-

(Looking for specific information, e.g. regarding attacks like H2C Smuggling, HPACK attacks, stuff regarding Cookies / Headers / Encoding... But please not spread over several dozen pages where it becomes frustrating to read the same shit over and over again without learning something new :( )

Hello, community, how do you relax between encoding, when you take a pause after hard task?)

NTSC vs PAL

Is it just me or is encoding in 15.9.3 broken AGAIN when using .editorconfig to set the charset?.......

One-hot encoding is fucking garbage. Everyone loves using this useless stupid shit that doesn’t work.

OK, so I've been working on processing a Japanese dictionary file and things are going smoothly for the most part. Out of ~185,000 entries, I've got 35 that are still causing problems.

The error I'm getting is "Incorrect string value '\xF0\xA4\xAD\xAF' for column...". I've checked all of my encoding and collation settings, and I'm pretty sure I've got it set to properly implement all of Unicode (as well as it does, anyway), as shown in the image attached. My suspicion is the problem characters are likely among the JIS X 0213 character set; in either case we're clearly dealing with a 4-byte character encoding issue here.

If needed I can attach a flag in the database and base64 encode these particular entries so the data isn't lost, but I'd like to just get it to handle the data properly in the first place if possible.

Anyone have any ideas on other items I can check to resolve the error?

First time would be when I had two teachers working of a bug that resultet in the CSS not loading / updating properly. It took me around 1 hour as well to realize that the text file was encoded with some obscure windows encoding, that doesn't load chrome (and maybe other browsers) can't load properly.

Thanks notepad++, will never use you again for web development.

Second time would be when I presented an app I made too the same teachers in order to boost my mark. Bugs that never appeared before every where and the basic functionality of the app was questionable. I hadn't worked on that app for half a year and had to fix as much as possible in a short amount of time, which might have made it worse.

Don't ask why but thanks to my knowledge about the france language I got a better mark, even though it was spanish

when you find a single invisible character in each of your sql files in visual studio that causes a sql implosion saying

invalid syntax near ''

Sublime Text 3 and other tools didn't help. required manually finding this time, in 20 files..... with an overdue project.

I wanted to burn visual studio to the ground

GOD DAMN THAT OLD BROKEN DATABASE.

Having to work with a legacy old database system running MySQL 4.0 is a pain. Especially when even finding a frikin manual for the thing is hard af.

And a cherry on top is dealing with encoding and collation in a system, that didn't really have a wide support for it yet.

10/10. At least I am only dealing with it so that we could later shut it down for good.

How's that character encoding going McDonalds?

Fucking encoding types everywhere, spending nearly half of the day with guessing which encoding type is correct for some freaking files 😑

Python needs a tool called 2to23.

Also, 2to3 does NOT take into account things like byte encoding changes. Kind of annoying.

Let's say I take a matrix of high entropy random numbers (call it matrix J), and encode a problem into those numbers (represented as some integers which in turn represent some operations and data).

And then I generate *another* high entropy random number matrix (call it matrix K). As I do this I measure the Pearson's correlation coefficient between J (before encoding the problem into it, call Jb), and K, and the correlation between J (after encoding, let's call it Ja) and K.

I stop at some predetermined satisfactory correlation level, let's say > 0.5 or < -0.5

I do this till Ja is highly correlated with some sample of K, and Jb's correlation with K is close to 0.

Would the random numbers in K then represent, in some way, the data/problem encoded in Ja? Or is it merely a correlation?

Keep in mind K has no direct connection to J, Ja, or Jb, we're only looking for a matrix of high entropy random numbers that indicated a correlation to J and its data.

I say "high entropy", it would be trivial to generate random numbers with a PRNG that are highly correlated simply by virtue of the algorithm that generated them.

Fuck powershell.

How in the world can one lang have so much problems with date conversion, and chars like üöä. Its so funny how they get through just fine sometime when u specify encoding utf8 but other times it has no ides wha5 these chars are.
Honestly fuck powershell

For those who hate to like PHP. I have an issue. Character encoding.

A semicolon ' is replaced by ’

A fullstop . is replaced by ' Â

Any ideas?

i can never understand the theme behind kotlin.

THEY DON'T HAVE A FUCKING TERNARY OPERATOR!!![?:]

Like before realizing this, i thought yeah jetbrains has decided to make android development a privileged hobby and non beginner friendly , so its now creating an encoding like language, in the false theme of " reducing code size"

But now they remove WORLDWIDE KNOWN, OPTIMIZED , EASY TO READ AND USE AND UNDERSTAND FEATURE of ternary operator and replacing it with less powerful but same looking elvis operator.( and stating that using if else for that is a better option)

Like why? if your goal is to make a shitty encoding language that makes everything shorter and most of the things optional, why remove the already efficient if else encoder?

God knows when this stupid language is going to stop my brain from getting blasted