UTF-8 gone wrong

I got 99 problems but encoding ain’t one

I have got 99 problems and emoji encoding ain't one of them.

P.S. /r/softwaregore

string excuses[]={
"it's not a bug it's a feature",
"it worked on my machine",
"i tested it and it worked",
"its production ready",
"your browser must be caching the old content",
"that error means it was successful",
"the client fucked it up",
"the systems crashed and the code got lost" ,
"this code wont go into the final version",
"It's a compiler issue",
"it's only a minor issue",
"this will take two weeks max",
"my code is flawless must be someone else's mistake",
"it worked a minute ago",
"that was not in the original specification",
"i will fix this",
"I was told to stop working on that when something important came up",
"You must have the wrong version",
"that's way beyond my pay grade",
"that's just an unlucky coincidence",
"i saw the new guy screw around with the systems",
"our servers must've been hacked",
"i wasn't given enough time",
"its the designers fault",
"it probably won't happen again",
"your expectations were unrealistic",
"everything's great on my end",
"that's not my code",
"it's a hardware problem",
"it's a firewall issue",
"it's a character encoding issue",
"a third party API isn't responding",
"that was only supposed to be a placeholder",
"The third party documentation is wrong",
"that was just a temporary fix.",
"We outsourced that months ago.","
"that value is only wrong half of the time.",
"the person responsible for that does not work here anymore",
"That was literally a one in a million error",
"our servers couldn't handle the traffic the app was receiving",
"your machines processors must be too slow",
"your pc is too outdated",
"that is a known issue with the programming language",
"it would take too much time and resources to rebuild from scratch",
"this is historically grown",
"users will hardly notice that",
"i will fix it" };

Ｔｈｅ　ｔｈｉｎｇ　ａｂｏｕｔ　ＵＮＩＣＯＤＥ　ｉｓ　ｔｈｅ　ａｂｉｌｉｔｙ　ｔｏ　ｍａｋｅ　ｉｔ　ｈａｒｄ　ｔｏ　ｐａｒｓｅ　ｗｈａｔ　ｈｕｍａｎｓ　ａｕｔｏｍａｔｉｃａｌｌｙ　ｓｅｅ　ａｓ　ＡＳＣＩＩ

"Ok, the site looks fine. Now let's move the style tag into it's own file."

*makes css file*

"WHY DOES IT HARDLY EVER LOAD!?!?, I checked the syntax trice"

*Spends 20 min. Asking friends for help, but none of them knows a reason*

"Time to ask the teacher, I guess"

*Teacher comes over, but has no clue either*

Teacher: "Give me the files, let's test it on my laptop"

*Css doesn't load there either*

*Teachers pair programming and trying some serious debugging technics. No progress*

*I decided to look at the sourcecode while refreshing the site*

1. Refresh: Css is loaded properly
2. Refresh: Css is gone, and source turned into various asian symbols.

Looks at the (default) file encoding: UCS-2
WTF NOTEPAD++, I SPEND 2 HOURS OF MY LIFE BECAUSE YOU DECIDED THIS WAS A PROPER ENCODING!

Web programming seems fun.

Not sure if it just doesn't have the right character encoding scheme, or if my existence is being questioned...

Notepad finally adds supports Unix-encoding

Now THAT is good news!

stolen from somewhere...

Intelligence and ability cannot be measured by education.

I have a client who asked a Master in Computer Science to develop a small system, for querying product title and their code. The guy used python, vanilla js, and... Txt file for the "database". Then my client asked me to integrated this in... WordPress.

This was in 2016. And idiot as I'm, I agreed and adapted his code to use php and a database.

April this year, my client said they are still using the python system to add new products all this time, in parallel. And wanted to update the WordPress with the data.

- No problem! - I said. Just send me the SQL file.

So the Master in CS sent me a SQL coded in ANSI. I asked for the SQL again, but with a more appropriate encoding. He took 1 month to reply back, and said it would be better if I get rid of the database and just use the txt file for querying.

This is outrageous.

I really hate people who are educated but completely useless.

Encoding issues always give me �headache�.

Me: I have an input stream!
Library: I want a file.
Me: I can see your code, you will convert it to an input stream! Don't any of your constructors take an input stream?
Library: No. I want a file. Okay, you can also include an encoding.
Me: I don't care about encoding.
Library: Fine, just give me the file then.
Me: You mean I have to somehow convert my input stream into an actual file so I can give it to you?
Library: You're gradually catching on, yes.
Me: Can I add a new constructor myself that takes an input stream?
Library: Sure! Good luck in getting it approved by the maintainers and the new jar distributed everywhere, including Maven Central before your deadline.
Me: Fine, I'll just rearchitect everything so I can give you an actual file.
Library: And then everything will be fine.
Me:

My stickers just arrived. And I especially love the encoding of “ß” within the street name.

German. It's hard.

Next time, just use upper letter and replaced “ß” with “SS” or just use a “B”. The B is especially fine to drive typographers mad.

*wrestling commentator voice*
"In this weeks episode of encoding hell:
The iiiinnnfamous UTF-8 Byte Order Mark veeeersus PHP!"

For an online shop we developed, there is currently a CSV upload feature in review by our client. Before we developed this feature, we created together with the client a very precise specification, including the file format and encoding (UTF-8).

After the first test day, the client informed us, that there were invalid characters after processing the uploaded file.
We checked the code and compared the customer's file with our template.
The file was encoded in ISO-8859-1 and NOT as specified UTF-8.
But what ever, we had to add an encoding check, thus allowing both encodings from now on.

Well well well welly welly fucking well...

Test day 2: We receive an email from said client, that the CSV is not working, again.
This time: UTF-8 encoding, but some fields had more colums with different values than specified.
Fucking hell.
We tell the customer that.
(I was about to write a nice death threat novel to them, but my boss held me back)

Testing day 3, today:
"The uploading feature is not working with our file, please fix it."
I tried to debug it, but only got misleading errors. After about 30 minutes, at 20 stacks of hatered, I finally had an idea to check the file in a hex editor:

God fucking what!?!!?!11?!1!!!?2!!

The encoding was valid UTF-8, all columns and fields were correct, but this time the file contained somthing different.
Something the world does not need.
Something nearly as wasteful as driving a monster truck in first gear from NYC to LA.

It was the UTF-8 Byte Order Mark.
3 bytes of pure hell.
Fucking 0xEFBBBF.
The archenemy of PHP and sane people.

If the devil had sex with the ethernet port of a rusty Mac OS X Server, then 9 microseconds later a UTF-8 BOM would have been born.

OK, maybe if PHP would actually cope with these bytes of death without crashing, that would be great.

I think this is so far one of the most priceless WTF moments I encountered at my current work:

A coworker of mine came up to me explaining the problem he had with russian characters in the filename. He explained in detail that everything works ok (the other part of the code he was fixing) if he changes the name of the file to test1.xlsx for example which doesn't use russian characters. OK great.

Then he goes on to show me how he fixed the other stuff and of course everything blows up. The file he used for demonstration was of course the original file our cusotomer provided, he just deleted the obvious russian chars and left the rest.

МТС != MTC

I cracked up: but you still have russian chars in the name.
The guy: no way, I deleted them all.
Me: but what about that МТС in the name?! Guy: what about it?
Me: did you actually typed that in or you left it there?! Those are russian chars that are fucking things up for you.
Guy: no way, it's MTC.
Me: checked the logs, you have ??? In the filename instead of МТС..don't you find that at least a little bit suspicious?!
Guy: but it looks the same. How does it (the computer) know it is in russian?!? //Why doesn't it understand?!

O.o I still can't believe it.. Is it just me & my high standards, or should it be normal for coders to know things such as character encoding & stuff?!?

I almost died of laughter, he and some other guy had problems finding customers in the software due to not being able to type the russian chars << happened more then once before, even after I told them about a quick hack on how to use google translate onboard keyboard & other stuff to make proper chars so they can get a match..

I think when they bury me, I'll still be facepalming and laughing over this incident. 🤣🤣🤣🤣🤣🤣🤣

Today I'm trying to study how to encode data in idx-ubyte format for my machine learning project.
Professors I'm going to astonish you!

Good day and good coding to all of you! :)

EXCEL YOU FUCKING PIECE OF SHIT! don't get me wrong, it's usefull and kt works, usually... Buckle up, your i for a ride. SO HERE WE FUCKING GO: TRANSLATED FORMULA NAMES? SUCKS BUT MANAGABLE. WHATS REALLY FUCKED UP IS HTHE GERMAN VERSION!
DID YOU HEAR ABOUT .csv? It stands for MOTHERFUCKING COMMA SEPERATED VALUES! GUESS WHAT SOME GENIUS AT MICROSOFT FIGURED? Hey guys let's use a FUCKING SEMICOLON INSTEAD OF A COMMA IN THE GERMAN VERSION! LET'S JUST FUCK EVERY ONE EXPORTING ANY DATA FROM ANY WEBSITE!

The workaround is to go to your computer settings, YOU CAN'T FUCKING ADJUST THIS IN EXCEL!, change the language of the OS to English, open the file and change it back to German. I mean, come on guys, what is this shit?

AND DON'T GET ME STARTED ON ENCODING! äöü and that stuff usually works, but in Switzerland we also use French stuff, that then usually breaks the encoding for Excel if the OS language is set to German (both on Windows and Mac, at least they are consistent...)

To whoever approved, implemented or tested it: FUCK YOU, YOU STUPID SHITFUCK, with love: me

I sometimes remember the time when I wrote a Email-inbox-exporter-PHP-script-type of application that collects all the emails from an inbox, "copied" it to a database with the attachements and stuff and moves it to a folder..

I just started at the company for like a couple of months, had no privileges to create mailboxes and such and I didn't want to interrupt our programmer to do this for me, so... I decided.. to save time and resources.. to test run it on our global, live 'support' mailbox.. :D Well.. You might guess what happened.. Apparently I mistyped the name of the move-destination folder (because imap-weird-things) that resulted in a completly empty mailbox and an empty database because the inserts failed due to bad encoding and mime-type issues..

The moment I refreshed my Outlook and noticed that all our mails where gone.. I swear, I can't describe that feeling of fear, cold sweat, intense heartbeat... I just stood up, asked if anyone wanted coffee, and just walked out of the office.. When in the hallway, I heard my collegues ask to one another "do you have any issues with outlook, all my mails are gone?". Everyone was stressing out, the chief was stressing out "what happened?!", nobody knew what happened.. :D

They could partially resolve it via one collegue who hadn't refreshed the mailbox and he could forward all the mails back to our support mailbox..

I dropped the project idea and learned to work with dev environments :D A couple of months later, I accidentially forgot a where condition in my SQL UPDATE statement, but that was the last time I seriously f*cked up.. :D Got to learn the hard way I guess.. Now everything I do runs in dev environments, I test everything before publishing,.. When I look back.. I don't even recognize the (inexperienced) guy I was back then ! :D

Ps. No one still knows what happened that day and they blamed it on server issues :D

FUCKING ENCODING SHITHEADS!
WHY ISN'T UTF-8 THE FUCKING DEFAULT EVERYWHERE!
§¥~@#&•…≈!

Encoding strikes yet another victim

So, California, land of no rain whatsoever.

Until today, where it actually rained!

Also today, a weather applet for tmux started misbehaving. Suspicious.

Coincidence, I think not! Sure enough, stopping that applet from running fixed everything. Turns out, the raining icon breaks the terminal, for some strange reason.

And why had this not been discovered earlier? This had been the first time it had rained for at least 8 months, and I added this only 4 months ago. Thus, it had never had its time to shine.

Yay for strange encoding!

Anyway, I now have a new excuse; blame bad code on global warming.

Finally Spend two fucking days debugging shit until I figured it it. Freaking stupid shit encoding problems and old data combined isn't fun. Dafuq why can't everybody use UTF-8 or Unicode or something else but PLEASE stop using some old school IBM shit codepages.
Leckt mich doch am arsch mit diesem scheiß man -_-

!dev - cybersecurity related.

This is a semi hypothetical situation. I walked into this ad today and I know I'd have a conversation like this about this ad but I didn't this time, I had convo's like this, though.

*le me walking through the city centre with a friend*
*advertisement about a hearing aid which can be updated through remote connection (satellite according to the ad) pops up on screen*

Friend: Ohh that looks usefu.....
Me: Oh damn, what protocol would that use?
Does it use an encrypted connection?
How'd the receiving end parse the incoming data?
What kinda authentication might the receiving end use?
Friend: wha..........
Me: What system would the hearing aid have?
Would it be easy to gain RCE (Remote Code Execution) to that system through the satellite connection and is this managed centrally?
Could you do mitm's maybe?
What data encoding would the transmissions/applications use?
Friend: nevermind.... ._________.

Cybersecurity mindset much...!

My teacher thinks encryption and encoding are same

Testing hell.

I'm working on a ticket that touches a lot of areas of the codebase, and impacts everything that creates a ... really common kind of object.

This means changes throughout the codebase and lots of failing specs. Ofc sometimes the code needs changing, and sometimes the specs do. it's tedious.

What makes this incredibly challenging is that different specs fail depend on how i run them. If I use Jenkins, i'm currently at 160 failing tests. If I run the same specs from the terminal, Iget 132. If I run them from RubyMine... well, I can't run them all at once because RubyMine sucks, but I'm guessing it's around 90 failures based on spot-checking some of the files.

But seriously, how can I determine what "fixed" even means if the issues arbitrarily pass or fail in different environments? I don't even know how cli and rubymine *can* differ, if I'm being honest.

I asked my boss about this and he said he's never seen the issue in the ten years he's worked there. so now i'm doubly confused.

Update: I used a copy of his db (the same one Jenkins is using), and now rspec reports 137 failures from the terminal, and a similar ~90 (again, a guess) from rubymine based on more spot-checking. I am so confused. The db dump has the same structure, and rspec clears the actual data between tests, so wtf is even going on? Maybe the encoding differs? but the failing specs are mostly testing logic?

none of this makes any sense.
i'm so confused.

It feels like i'm being asked to build a machine when the laws of physics change with locality. I can make it work here just fine, but it misbehaves a little at my neighbor's house, and outright explodes at the testing ground.

One of the things I love the most about programming is that sometimes you feel like you're taking a step forward when you're taking a step back... and sometimes you feel like you're taking a step back when you take a step forward.

Or maybe that's just me. XD

Woke up this morning to a fucking giant snowstorm and my first reaction was 'fml' , poured some coffee , lit a smoke and started checking my work mail 'Issue xxxx response : Not solvable '...what the...I go through the files on my phone , look at what that issue was : lack of proper validation , filtering and encoding of input thus enabling xss . Not solvable my ass ...simply adding literally 3 more characters to that fucking retarded filter would stop all the bypasses . This issue is a showstopper for their project and that is what they answer ?
Sorry to indians out here but some of your colleagues are as stupid and unimaginative as they can possibly ever come .

Uggg..... I'm trying to encode a binary file in Python which may be an image or may be an executable, and then decode it back into a file (I plan on editing it in the middle, but baby steps for now..) but nothing is working!!

My plan is to:

Open binary file.

Decode as base64, or something else that could easily handle binary.

Convert byte data to string (for editory perpousos - I won't be editing bytes, I'll be doing custom encoding but that's irrelevant for this test)

Convert back to a byte string/array (with .encode(), probably)

Write to file.

I do this, yet the output has been altered... Though I haven't touched anything..

It's so enfuriating.. x.x

Because nothing says "security" like some good ol' Base64 encoding. Bet whoever wrote that code was wearing mirror shades.

Yesterday a strange bug appeared in Chrome: In a small web app we have some umlaut characters like äöü. Strangely said characters were displayed as cyrillic, but only on my pc... On every other device it worked. I spent about 5 hours of checking encodings (everything was in UTF-8), reading posts in the almighty stackoverflow.
Finally i figured out, the font was broken. After reinstalling it, everything was peaceful again in my head.

So someone here had the job to merge the current version of our software into another older repo.
(That older repo has some features for future versions, that should be only in there ... don´t ask ... not my decision)

Took him long enough. But he forgot to check one thing while merging: The encoding of the files.

And we´re german...

ALL umlauts, and all other special chars in the code were replaced with this: �
No global replace, because all chars were replaced with the same char.
(Why the fuck do we have special chars in the code in the first place???)

So to not need him to start all over again I compiled a list of common german words containing umlauts and did a global search and replace.

I think I got 90% of the errors like that.
Now he´s going to correct the rest of the errors.

Fuck the comments in the code though.
Just a waste of time...

I was working on this mustachio fork called Morestachio and changed the Default Encoding from "Encoding.Default" to UTF8 while creating a .NetStandard version of it. After i have created everything i thought "Hey just for the lolz run my tests again" ...

Just imagine the world where we have CAPITALIZED DIGITS.

TL:DR
Why do so shitty "API"s exist that are even harder to write than proper ones? D:

Trying to hack my venilation at home.
This API is so horrible D:
The API is only based on POST requests no matter if you want to write values or get values and the response only contains XML with cryptic values like:

<?xml version="1.0" encoding="UTF-8"?>
<PARAMETER>
<LANG>de</LANG>
<ID>v01306</ID>
<VA>00011100000000000000000010000001</VA>
<ID>v00024</ID>
<VA>0</VA>
<ID>v00033</ID>
<VA>2</VA>
<ID>v00037</ID>
<VA>0</VA>

Also there are multiple API routes like
POST /data/werte1.xml
POST /data/werte2.xml
POST /data/werte3.xml
POST /data/werte4.xml
And actually the real API route is only given in the request body and not in the path.

Why is this so shitty? D:<

Btw in terms of security this is also top notch. It just globally saves if one computer sends the login password.
I mean why even ask for a password then? D:
That made me end up with a cronjob to send a login request so I don't have to login on any device.

PS:
You see, great piece of German engineering.

When you encounter a character encoding issue.

The senior developer swore the server was responding with a 500 because I, the co-op, was definitely sending the 'wrong encoding' whenever I post special characters. He said, "I'm only taking your data and putting it straight into the database!!!"

I found out it was breaking because he was converting the JSON payload to a STRING and simplistically separating the keys and values by colons and double quotes. Yuuup. Nothing can go wrong there...

Hmm...recently I've seen an increase in the idea of raising security awareness at a user level...but really now , it gets me thinking , why not raise security awareness at a coding level ? Just having one guy do encryption and encoding most certainly isn't enough for an app to be considered secure . In this day an age where most apps are web based and even open source some of them , I think that first of all it should be our duty to protect the customer/consumer rather than make him protect himself . Most of everyone knows how to get user input from the UI but how many out here actually think that the normal dummy user might actually type unintentional malicious code which would break the app or give him access to something he shouldn't be allowed into ? I've seen very few developers/software architects/engineers actually take the blame for insecure code . I've seen people build apps starting on an unacceptable idea security wise and then in the end thinking of patching in filters , encryptions , encodings , tokens and days before release realise that their app is half broken because they didn't start the whole project in a more secure way for the user .

Just my two cents...we as devs should be more aware of coding in a way that makes apps more secure from and for the user rather than saying that we had some epic mythical hackers pull all the user tables that also contained unhashed unencrypted passwords by using magix . It certainly isn't magic , it's just our bad coding that lets outside code interact with our own code .

Currently working with @Kreischo and another good friend to create a secured, encoded container to store files in it.

I am currently working on the frontend of things, thinking it's quite beautiful.. (Done with Electron)

Your opinion?

IPAY88 is the worst payment integration. They parse html data and encoded it into xml for return the data, it is not even singlet or server to server communication , tey called it the ADVANCED BACKEND SYSTEM (My arse!) For security, they ENCODE THE STRING into BASE64 and called it ENCRYPTION ! WHAT THE FUCK?
Encoding is not encryption! I qas expecting they used diffie hellman or AES or RSA etc. THEY TOLD BE ENCODING IS ENCRYPTION? WHAT THE FUCK?

Today I learned about binary encoding formats alternative to JSON such as Google Protocol Butter.
I like these binary formats.
Just thought I would share this here so others would benefit as well (and please share your experience if it is relevant)

Guess I'll fuckin try again tomorrow.

Building a cross platform c program. On Linux side, just using a makefile. Today I tried using visual studios "clone" feature for git. It just downloads the files and makes them available to the editor, it doesn't make a project, obviously.

But this has some disadvantages. For one, you can't build, or run. Two, you don't get any project properties. My project needs to set the character encoding to Unicode. Can't do that without a project.

So I use their tool to create a project from existing code. It didn't really work. The build profiles were janky at best and I still couldn't set the character encoding.

Ended up just deleting the whole thing.

Just had to do this. Homemade. Sorry :D

How deep does the rabbit hole go?

Problem: Convert numpy array containing an audio time series to a .wav file and save on disk

Error 1:
Me: pip install "stupid package"
Console: Can't pip, behind a proxy

Me: Finds workaround after several minutes

Error 2:
Conversion works, but audio file on disk doesn't work
Encoding Error only works with array of ints not floats
BUT I NEED IT TO BE FLOATS

Looks for another library
scikits.audiolab <- should work
Me: pip --proxy=myproxy:port install "this shit"
Command Line *spits back huge error*

Googles error <- You need to install this package with a .whl file
Me: Downloads .whl file <- pip install "filename".whl
Command Line: ERROR: scikits.audiolab-0.11.0-cp27-cp27m-win32.whl is not a supported wheel on this platform.

Googles Error <- Need to see supported file formats
Me: python -c "import pip; print(pip.pep425tags.get_supported())"
Console: AttributeError: module 'pip' has no attribute 'pep425tags'

Googles Error <- Use another command for pip v10
Me: python -c "import pip._internal; print(pip._internal.pep425tags.get_supported())"
Console: complies

Me: pip install "filename".whl
Console: complies

Me: *spends 30 minutes to find directory where I should paste .dll file*
Finds Directory (was hidden btw), pastes file

Me: Runs .py file
Console: from version import version as _version ModuleNotFoundError: No module named 'version'

Googles Error <- Fix is: "just comment out the import statement"
Me: HAHAHAHAHAHA
Console: HAHAHAHAHA

Unfortunately this shit still didn't work after two hours of debugging, lmao fuck this

Hey, been gone a hot minute from devrant, so I thought I'd say hi to Demolishun, atheist, Lensflare, Root, kobenz, score, jestdotty, figoore, cafecortado, typosaurus, and the raft of other people I've met along the way and got to know somewhat.

All of you have been really good.

And while I'm here its time for maaaaaaaaath.

So I decided to horribly mutilate the concept of bloom filters.

If you don't know what that is, you take two random numbers, m, and p, both prime, where m < p, and it generate two numbers a and b, that output a function. That function is a hash.

Normally you'd have say five to ten different hashes.

A bloom filter lets you probabilistic-ally say whether you've seen something before, with no false negatives.
It lets you do this very space efficiently, with some caveats.

Each hash function should be uniformly distributed (any value input to it is likely to be mapped to any other value).

Then you interpret these output values as bit indexes.

So Hi might output [0, 1, 0, 0, 0]
while Hj outputs [0, 0, 0, 1, 0]
and Hk outputs [1, 0, 0, 0, 0]
producing [1, 1, 0, 1, 0]

And if your bloom filter has bits set in all those places, congratulations, you've seen that number before.

It's used by big companies like google to prevent re-indexing pages they've already seen, among other things.

Well I thought, what if instead of using it as a has-been-seen-before filter, we mangled its purpose until a square peg fit in a round hole?

Not long after I went and wrote a script that 1. generates data, 2. generates a hash function to encode it. 3. finds a hash function that reverses the encoding.

And it just works. Reversible hashes.

Of course you can't use it for compression strictly, not under normal circumstances, but these aren't normal circumstances.

The first thing I tried was finding a hash function h0, that predicts each subsequent value in a list given the previous value. This doesn't work because of hash collisions by default. A value like 731 might map to 64 in one place, and a later value might map to 453, so trying to invert the output to get the original sequence out would lead to branching. It occurs to me just now we might use a checkpointing system, with lookahead to see if a branch is the correct one, but I digress, I tried some other things first.

The next problem was 1. long sequences are slow to generate. I solved this by tuning the amount of iterations of the outer and inner loop. We find h0 first, and then h1 and put all the inputs through h0 to generate an intermediate list, and then put them through h1, and see if the output of h1 matches the original input. If it does, we return h0, and h1. It turns out it can take inordinate amounts of time if h0 lands on a hash function that doesn't play well with h1, so the next step was 2. adding an error margin. It turns out something fun happens, where if you allow a sequence generated by h1 (the decoder) to match *within* an error margin, under a certain error value, it'll find potential hash functions hn such that the outputs of h1 are *always* the same distance from their parent values in the original input to h0. This becomes our salt value k.
So our hash-function generate called encoder_decoder() or 'ed' (lol two letter functions), also calculates the k value and outputs that along with the hash functions for our data.

This is all well and good but what if we want to go further? With a few tweaks, along with taking output values, converting to binary, and left-padding each value with 0s, we can then calculate shannon entropy in its most essential form.

Turns out with tens of thousands of values (and tens of thousands of bits), the output of h1 with the salt, has a higher entropy than the original input. Meaning finding an h1 and h0 hash function for your data is equivalent to compression below the known shannon limit.
By how much?

Approximately 0.15%
Of course this doesn't factor in the five numbers you need, a0, and b0 to define h0, a1, and b1 to define h1, and the salt value, so it probably works out to the same. I'd like to see what the savings are with even larger sets though.

Next I said, well what if we COULD compress our data further?

What if all we needed were the numbers to define our hash functions, a starting value, a salt, and a number to represent 'depth'?

What if we could rearrange this system so we *could* use the starting value to represent n subsequent elements of our input x?

And thats what I did.

We break the input into blocks of 15-25 items, b/c thats the fastest to work with and find hashes for.

We then follow the math, to get a block which is
H0, H1, H2, H3, depth (how many items our 1st item will reproduce), & a starting value or 1stitem in this slice of our input.

x goes into h0, giving us y. y goes into h1 -> z, z into h2 -> y, y into h3, giving us back x.

The rest is in the image.
Anyway good to see you all again.

All it takes is one jackass somewhere in the world with no understanding of character encoding to ruin things for everyone else.

Encoding problems on live TV in portugal...

So for those of you keeping track, I've become a bit of a data munger of late, something that is both interesting and somewhat frustrating.

I work with a variety of enterprise data sources. Those of you who have done enterprise work will know what I mean. Forget lovely Web APIs with proper authentication and JSON fed by well-known open source libraries. No, I've got the output from an AS/400 to deal with (For the youngsters amongst you, AS/400 is a 1980s IBM mainframe-ish operating system that oriiganlly ran on 48-bit computers). I've got EDIFACT to deal with (for the youngsters amongst you: EDIFACT is the 1980s precursor to XML. It's all cryptic codes, + delimited fields and ' delimited lines) and I've got legacy databases to massage into newer formats, all for what is laughably called my "data warehouse".

But of course, the one system that actually gives me serious problems is the most modern one. It's web-based, on internal servers. It's got all the late-naughties buzzowrds in web development, such as AJAX and JQuery. And it now has a "Web Service" interface at the request of the bosses, that I have to use.

The programmers of this system have based it on that very well-known database: Intersystems Caché. This is an Object Database, and doesn't have an SQL driver by default, so I'm basically required to use this "Web Service".

Let's put aside the poor security. I basically pass a hard-coded human readable string as password in a password field in the GET parameters. This is a step up from no security, to be fair, though not much.

It's the fact that the thing lies. All the files it spits out start with that fateful string: '<?xml version="1.0" encoding="ISO-8859-1"?>' and it lies.

It's all UTF-8, which has made some of my parsers choke, when they're expecting latin-1.

But no, the real lie is the fact that IT IS NOT WELL-FORMED XML. Let alone Valid.

THERE IS NO ROOT ELEMENT!

So now, I have to waste my time writing a proxy for this "web service" that rewrites the XML encoding string on these files, and adds a root element, just so I can spit it at an XML parser. This means added infrastructure for my data munging, and more potential bugs introduced or points of failure.

Let's just say that the developers of this system don't really cope with people wanting to integrate with them. It's amazing that they manage to integrate with third parties at all...

I am sick of misrotated videos.

Sometimes, the phone camera software saves a video vertically because the user hits "record" before the software has detected that the user is holding the smartphone horizontally, because the software stupidly launches in vertical orientation by default.

So the software wants the user to wait until it has finally detected horizontal orientation, which causes the user to miss out on a moment.

How about the camera software actually saves the video in the orientation it was recorded in for the most time, rather than only the beginning of the video?

If I can think of this idea, billion-dollar companies surely can.

In the meantime, misrotated videos can be fixed using this ffmpeg command on Linux or Windows:

ffmpeg -i input_file.mp4 -metadata:s:v rotate="0" -c copy output_file.mp4

And if the phone was held with the home button to the left side:

ffmpeg -i input_file.mp4 -metadata:s:v rotate="180" -c copy output_file.mp4

This solution is superior compared to using -vf (video filters) because it only touches the metadata of the video. No re-encoding. This means no quality loss and no CPU/GPU power needed to process the video again. It just passes through.

Why in the everliving freaking fuck does Java have a class called URLEncoder which DOES NOT DO URL ENCODING??? The shit-spraying piece of a mouldy footgun does formencoding?? Which sadistic maniac thought that was a good idea?

The only way I would ever use a recovery question.
BTW nice encoding there...

🎼Up and down
🎶 And all the way around
🎶 Crossentropy is a bitch
🎶 Epoch and loss are still the same
🎵

Edit: I left this rant open for half an hour. Finally figured that I should do one hot encoding and labeling doesn't exactly work out the way I was hoping it works out.

But hey, half an hour ago I was angry and desperate. So... ¯\_(ツ)_/¯

Over the summer I was recruited to be a supplement instructor for a data structures course. As a result of that I was asked (separately by the professor) to be a grader for the course. Because of pay limitations I've mostly been grading homework project assignments. In any case, it's a great job to get my foot into the department and get recognized.

Over the course of the semester I've had this one person, OSX, named after their operating system of choice, who has been giving me awkward submissions. On the first assignment they asked the professor for extra time for some reason or the other, and that's perfectly fine.

So I finally receive OSX's submission, and it's a .py file as per course of the course. So I pop up a terminal in the working directory and type "python OSX_hw1.py". Get some error spit out about the file not being the right encoding. I know that I can tell python to read it in a different encoding, so I open it up in a text editor. To my surprise it's totally not a text file, but rather a .zip file!

I've seen weirder things done before, so no big deal. I rename the file extension, and open it up to extract the files when I see that there's no python files. "Okay, what's goin on here OSX..." I think to myself.

Poking around in the files it appears to be some sort of meta-data. To what, I had no clue, but what I did find was picture files containing what appeared to be some auto-generated screenshots of incomplete code. Since I'm one to give people the benefit of doubt even when they've long exhausted other peoples', I thought that it must be some fluke, and emailed OSX along with the professor detailing my issue.

I got back a rather standard reply, one of which was so un-notable I could not remember it if my life depended on it. However, that also meant I didn't have to worry about that anymore. Which when you're juggling 50 bazillion things is quite a relief. Tragically, this relief was short lived with the introduction of assignment 2.

Assignment 2 comes around, and I get the same type of submission from OSX. At this time I also notice that all their submissions are *very* close to the due time of 11:59pm (which I don't care about as long as it's in before people start waking up the next morning). I email OSX and the professor again, and receive a similar response. I also get an email from OSX worried about points being deducted. I reply, "No issue. You know what's wrong. Go and submit the right file on $CentralGradingCenter. Just submit over your old assignment".

To my frustration OSX claimed to not know how to do this. I write up a quick response explaining the process, and email it. In response OSX then asks if I can show them if they comes to my supplemental lesson. I tell OSX that if they are the only person, sure, otherwise no because it would not be a fair use of time to the other students.

OSX ends up showing up before anyone else, so I guide them through the process. It's pretty easy, so I'm surprised that they were having issues. Another person then shows up, so I go through relevant material and ask them if they have any questions about recent material in class. That said, afterwards OSX was being somewhat awkward and pushy trying to shake my hand a lot to the point of making me uncomfortable and telling them that there's no reason to be so formal.

Despite that chat, I still did not see a resubmission of either of those two assignments, and assignment 3 began to show it's head. Obviously, this time, as one might expect after all those conversations, I get another broken submission in the same format. Finally pissed off, I document exactly how everything looks on my end, how the file fails to run, how it's actually a zip file, etc, all with screenshots. That then gets emailed to the professor and OSX.

In response, I get an email from OSX panicking asking me how to submit it right, etc, etc. However, they also removed the professor from the CC field. In response I state that I do not know how to use whatever editor they are using, and that they should refer to the documentation in order to get a proper runnable file. I also re-CC the professor, making sure OSX's email to me is included in my reply.

OSX then shows up for one of my lessons, and since no one had shown up yet, I reiterate through what I had sent in the email. OSX's response was astonished that they could ever screw up that bad, but also admits that they had yet to install python(!!!). Obviously, the next thing that comes from my mouth is asking OSX how they write their code. Their response was that they use a website that lets them run python code.

At this point I'm honestly baffled and explain that a lot of websites like those can have limitations which might make code run differently then it should (maybe it's a simple interpreter written on JavaScript, or maybe it is real python, but how are you supposed to do file I/O?) .

After that I finally get a submission for assignment 1!

When you Render something and the name is just perfect!

Translation: Catched encoding buying Toilet Paper

list of things that Microsoft fucked up :

1. Windows-1250 encoding

Writing hebrew in Latex using a template that doesn't support UTF-8 is the most archaic shit I've done lately. I feel like some sort of a caveman.
This fucking encoding inverts all letters so it can support right to left. 😓

Go fmt yourself? I guess. Taken from Golang's documentation on encoding/json.

Oh man, I found the names for the different number bases past 36! I should expand my base converter!

That doesn’t... look right...

Ah, because these ones require the use of encoding algorithms.

NOPE NOPE NOPE NOPE

Someone building the iPad software for this beer museum did Not think about the encoding xD

This is gonna get someone illogically upset, but idc about that.

I know it's ignorance of semantics but I'm tired of propagated ignorance changing the meaning of things.

Non-binary is NOT a legitimate term for whatever 'gender' you are!

I get what *whoever-started-it* was going for, but it's NOT valid. If you want to say that youre not male or female, fine... just don't abuse binary systems to do it. Just say youre non-bool/anti-boolean or identify with one of the, apparently 50, shades of gray.

I keep getting into logical loops to nowhere about this nonsense. No one is even defining what's supposed to be the 1 vs the 0. Which then makes me think '1 must be male... genitalia=1 in many ways...' which then sources back to the historic validity of males vs undervalued/less than human interpretations of females...

Then <brake>.
Ofc these people aren't going into the historical significance... they don't even realise how binary works! Ofc they'd have no clue that all 0s= no data... and 0s only have significance when viewed in placement to the 1s.

Let's all start using proper terminologies, like non-boolean. Maybe i can start a trend by paying people pennies to learn/teach wtf a boolean value is, and that binary can represent anything. With proper encoding the array is limitless... so being binary is actually a giant spectrum... therefore makes no sense to be "non-binary".

Ok... im done. It had to be said.

Who wants to start identifying as non, or educating wtf is, boolean with me???

Hardly anything in tech aged better than H.264.

The H.264 video format, also known as AVC (Advanced Video Coding), was made in the 2000s, is still widely in use in the 2020s, and only slowly being usurped by H.265, which was made in 2013.

Working with a vendor's api. Every call is coming back error. Emails back and forth for two weeks, sending logs, changing settings, encoding urls...

From them: oh, this field needs to be unique.

Me: how are you just now catching that?!

The doc showed it to be a specific value. We must be their first client or something.

Trying to work out an encoding bug after upload, I ask them to send me the original version of the file that is currently uploaded. I look at the files and they seem strangely similar, but strangely different.

I reply to them, "To confirm, this is the version you uploaded?"

"Nope."

i wrote a website, a server in go, a small os in c, a game in js, a game and server and web scraper and other desktop apps in java, mobile apps with flutter, a website with php also, implemented aes in go, wrote a parser in java. done sysadmin stuff on my vps and pihole/openvpn/nextcloud on my rpi. learn about c vulnerabilities and used metasploit. attempted to write an interpreted language. did some led displays with arduino. currently learning tensorflow.

i have never...
- written a driver
- made a game with a game engine
- created a file encoding
- implemented an oauth2 server
- made an api
- worked with vr

what am i missing? i want to be a very well rounded dev.

Let's focus/master computer architectures, coding paradigms, datastructures. Everything comes after that...the problem with todays academics is that they are more focused on immediately deploying students to industry; theyre more focused on teaching specific frameworks and specific language instead of teaching how things work...i bet most students (at least in my country) are having troubles with endianess or encoding or even byte manipulation or what a thread is....If im going to be the teacher for example of an oop subject, ill let the student choose the language they want as long as the oop paradigm is intact ,it will be fine.. i dont friggin care whether you know vue or angular or swing if you dont even know what a callback is..

API provider: include a signature based on these fields in this order. DO NOT ENCODE IT!

Implementation works a while, then..

*a wild apostrophe appears*

Signature no longer works.

API Provider: "oh, yeah we escape those."

Arrghhghghghhhghvhxmchsoxnsoxnwl

Not only is it a poor design for signing payloads, the documentation is shockingly poor in it.
Even the implementation example (which is supposedly from their code) doesn't account for any type of escaping or encoding.

Before anyone asks, I can't into details about the implementation.

Use .editorconfig files when working with multiple developers. Otherwise your file encoding and code style will get fucked up eventually.

I've seen some of the videos that I downloaded from the internet are only playable through one particular application. How did they do it? How can I implement the same thing with my application? What technology should I research on to know more about this?

Just found out the API of our zentracloud sensors is sending the units with a space before the actual unit. Couldn't figure out for half an hour why Doctrine is not finding the unit in the database. Encoding? JSON decoding? Character itself? Screw you. Screw you...

Every flipping time when receiving file with win1251 charset

If I were to write a de/encoding algorithm, that nobody but me knows about, would it be safe?

The next step for improving large language models (if not diffusion) is hot-encoding.

The idea is pretty straightforward:
Generate many prompts, or take many prompts as a training and validation set. Do partial inference, and find the intersection of best overall performance with least computation.

Then save the state of the network during partial inference, and use that for all subsequent inferences. Sort of like LoRa, but for inference, instead of fine-tuning.

Inference, after-all, is what matters. And there has to be some subset of prompt-based initializations of a network, that perform, regardless of the prompt, (generally) as well as a full inference step.

Likewise with diffusion, there likely exists some priors (based on the training data) that speed up reconstruction or lower the network loss, allowing us to substitute a 'snapshot' that has the correct distribution, without necessarily performing a full generation.

Another idea I had was 'semantic centering' instead of regional image labelling. The idea is to find some patch of an object within an image, and ask, for all such patches that belong to an object, what best describes the object? if it were a dog, what patch of the image is "most dog-like" etc. I could see it as being much closer to how the human brain quickly identifies objects by short-cuts. The size of such patches could be adjusted to minimize the cross-entropy of classification relative to the tested size of each patch (pixel-sized patches for example might lead to too high a training loss). Of course it might allow us to do a scattershot 'at a glance' type lookup of potential image contents, even if you get multiple categories for a single pixel, it greatly narrows the total span of categories you need to do subsequent searches for.

In other news I'm starting a new ML blackbook for various ideas. Old one is mostly outdated now, and I think I scanned it (and since buried it somewhere amongst my ten thousand other files like a digital hoarder) and lost it.

I have some other 'low-hanging fruit' type ideas for improving existing and emerging models but I'll save those for another time.

Just getting a pizza, can't help but get annoyed by the jittery as fuck newsticker, they didn't even get the encoding right....

(Yes our POS software has 60fps jitterless hardware accelerated fully utf-8 encoded newstickers before anyone asks me to do a better job)

OK what the actual fuck is going on within this company.

TL;DR: Spaghetti Copy/Pasted code that made me mad because it's just a mess

I just looked into a code file to search for a specific procedure regarding the creation of invoices.
I thought "Oh this is gonna be a quick look-through of like 1000 lines MAX" turns out this script is 11317 fucking lines long and most of it's logic is written there multiple (up to 6-7 times). And I'm not talking about a simple 10 lines or something. No! Logic of over 300 lines.. copy & pasted over .. and over .. and over?! I mean what the fuck did this guy drink when he wrote this.

Alsooo 10000 of those 11317 lines is ONE FUNCTION.. I kid you not! It's just a gigantic if / else if construct that, as I said before, contains copy-pasted code all over the place.
Sadly my TL thinks that code cleanup / optimization is "not necessary as long as it works" like wtf dude. If anyone wants to ever fix something in this mess or add a new feature they take a few hours longer just to "adjust" to this fucking shit.
This is a nightmare. The worst part: This is not the only script that has shit like this. We got over 150 "modules" (Yeah, we ATTEMPTED something OOP-ish but failed miserably) that sometimes have over 15000 lines which could be easily cut down to 1/3 and/or splitted into multiple files.
Let's not start about centralization of methods or encoding handling or coding standards or work code review or .. you get the point because there's a character limit for one rant and I guess I'd overshoot that by a lot if I'd start with that. Holy shit I can't wait until my internship is over and I can leave this code-hell!!

So the guy who i mentioned previously with his shitty csv made webservice which i needed to use.

It returned utf-8 encoded xml:
- in file the header contained central europe encoding
- the xml had more roots
- basically he only put his csv format inside xml tags
- csv contained html tags but there wasnt cdata

Now work with this lol

I neded to cut this shit with string functions and also some numeric data get with regexp from htm table td's

Whyyyyyyyyyyyy

Superhuman capabilities.

Friend - Hey is this file correctly encoded ?

Me - Hell Yes. I can decode visually from an IMAGE and make sense of underlying data encoded.

Some people think I am so awesome coder I can look at Huffman encoded result and understand if it's correct or not.

First experience with utf-16 and all I can think is "whatthefuckwhatthefuckwhatthefuckwhatthefuck"

Why the hell do languages like Kotlin (Java) and C# handle dates and datetimes so needlessly complicated?
There are multiple types with different implementations and concepts like local time or time zones represented by those types. Some of them have capabilities like serialization, some of them don’t.
Parsing and encoding is tied to the types.

Why? Take Swift as an example:
It has one single Date type (including time) which represents a point in time independent of any calendar, time zone, encoding or format.
There is a DateFormatter to parse from APIs from iso or timestamps or whatever and to format to UI as a string in any language (localization), for any region, in any format.
If you just want a container for the date time components themselves (which the concept of local date time seems to be in those languages), you can use the DateComponents type. If you are interested in dates from the perspective of a calendar, there is a Calendar type.

Everything makes sense and the different concepts are decoupled from each other as they should be.

Damn! My memory about C# is a bit hazy but Kotlin, I’m disappointed in you! Date handling is a horrible mess!
Ok, I guess I can blame it on Java and JVM.

a lot of dev have a miss concept about Unicode/utf8 including me but I believe my understanding get Better and this my last version.
For a project i was developing a rest api for mobile app
when an ios dev asked me
"I send you Unicode string but it appears as ????? in admin web panel "
OMG!!!😨😨😨
Unicode is not an encoding nor an algorithm. it's a standerd which just map a glyph to a codepiont .
but utf8 is the encoding of Unicode and how it's stored or transferred ,
the string you send must be a utf-8 encoded string as the rest of the json you sent .

Swift does some things really well, but then falls flat in others. Why is polymorphic JSON decoding/encoding such a pain?
In Kotlin, it's a breeze to support multiple object types.

During a code review I was doing the dev wrote a function that takes a tweet, iterates over each letter, if the letter is not in a specified unicode range the letter is removed from the sentence. So now you have a sentence with missing letters... Wtf

120fps and 240fps filming isn't just for slow motion playback, but recent smartphones have 120 Hz screens so those videos can finally be watched as ultra-smooth motion with audio.

If only all smartphones encoded high-framerate videos in real-time with the same framerate recorded from the image sensor instead of stupidly slowing down when encoding.

Granted, this is a thing Apple has always done right: they encoded their "slow motion" videos in real-time and let the user select the slowed-down portions during playback!

Let the user set their preferred playback speed in the video editor, don't dictate that 1× playback speed is 1/4 of real-life speed. 1× playback speed must be 1× real-life speed to clear up all confusion.

Besides, laptops with 120 Hz screens existed as early as 2011 (Samsung 700G7A)!.

More from my big black book of ai and neuroscience:

I think if trace theory is true to any degree it would go some distance in explaining phenomenal consciousness, assuming I haven't misunderstood anything.

In fuzzy trace theory (FTT) it is posited that people form two types of mental representations about a past event:

*verbatim traces: detailed representations of a past event.

*gist traces: fuzzy representations of a past event.

People can reason with verbatim *and* gist traces but prefer gists.

*vision was suggested to work similarly in 1999. With human vision, two processes could be used: one that aggregates local receptive fields and one that parses the local receptive spatial field. It was suggested that people used prior experience, gists, to decide which dominates a perceptual decision.

Gist processes form representations of events, semantic details, where verbatim reinstates the context found in the surface details of an event.

__notes__

Parallel storage: asserts encoding/storage of verbatim/gist traces operate in *parallel*, not in serial.

I like to think of verbatim traces as databases, and gists as queries constructed by recognition.

Several studies have found that the meaning (gist) of an item is encoded even *before* the surface details (verbatim).

This might be important as a survival mechanism but should not be taken to mean strictly that gists are formed wholly *without* details or important and recognizable features of the item in question. It may well be for high level el processing and classification efficiency this may be an important reprocessing step, in the same way that many functions of the brain are duplicated throughout.

A shitty platform that, although open source, there is no clearly documented way of setting a development environment for it. This pile of crap states clearly that it does NOT support RTL languages. One of the core business requirements is Arabic support. What to do? Look for other platforms? WRONG!

Base the fucking business on it and ask ME to see why the SQL database is not encoding the Arabic characters correctly and to look into the logs that back-end puked. My expertise is mobile development anyways damnit. Sure the backend code is Java code (Java jokers and haters, not the appropriate place) and I know it but there is no fucking way to test that motherfucker or to build it! No fucking testing server can be made! Only instructions to get a Docker image pulled and set up.

FML.

"This company is a fucking م."

I cannot believe I am so frustrated that I am ending this rant with a fun puzzle.

Hints to help you decipher the quoted sentence:

Hint 1: That Arabic letter is the perfect letter.
Hint 2: You don't need to be an Arab to understand what it means.

The rear ducking continues. We've built a reliable translator in the dumbest fucking way possible, it's just lovely. I simply reused the structure for feeding data to the VM assembler, an array of arrays, where there's one array of (ins [args]) per node in the parse tree.

It's nice because nodes can be solved out of order without affecting the actual sequence in which the instructions are output. And if one statement (node) equals multiple instructions, you just push multiple entries to the corresponding array, or push nothing if you need to output nothing. Easy as goblin pie.

This is enough to convert an input language to the assembly-like intermediate representation we use for the virtual machine. So then there's doing it backwards: walk the same array of arrays, and map those virtual instructions to a physical architechture. I guess I could do the encoding to native binary myself, it'd certainly be interesting to try, but I'm burnt-out already so I'll just use fasm for now.

Initial test: wrote a test program in my own stupid language, ran the translator, dump output to file, assemble that with fasm, run with r2 -d.

Crashes? No.
Runs fine? Yes and no.

For fuck's sake, I don't have syscalls. Mainly because the VM doesn't have an operating system, lmao. I was testing virtual programs by just freezing state, terminating, then dumping the fucking registers and stack to the console, we have no I/O to speak of. Not even a real 'exit', VM handles that by reading a return value every step like a mentally damaged son of a bitch.

So anyway, I manually paste the linux mambo, you know:

mov rax,60
mov rdi,0
syscall

And NOW our program can end execution without crashing.
Okay then, so does the test code work correctly?

** DRUM ROLL **

Yes.

Ladies and gentlemen, mother fucking PESO is now a compiled language, and going forward I will be expectantly receiving your marriage proposals for reviewing. Oh, but not so fast, we still need a frontend...

Well, we'll handle that in the next few days. I'm just glad to be *nearly* finished with this fucking compiler, I want nothing to do with anything else ever, but we know that's not going to happen, so Lord please end my pain.

No sponsor as this rant has been paid for by tax evasion.

Is there any program that can be used to cut out parts of a video file without re-encoding.

The way I'm thinking is if you already have a video file, you should be able to just copy the data from 1 file to another like a file splitter.

Been working on this project for the last 9 months, it's 1000 times bigger than the original spec, but the worst part of the whole experience is the god damn Microsoft Access database and migrating data from it.

Wanky encoding, references that aren't actually linked up, ID's that aren't unique, meaningless columns and table names.

Pushed me to the edge.

Finally got around to some real video encoding work on my new computer but noticing it's not blazing fast...

And more work is still handled by the CPU... But I thought video processing is handled by the GPU, which seems to be barely used at all. I'm using Handbrake but I thought the whole point of dedicated GPU was for intensive graphics and video processing?

Utf-string encoding in windows has been constantly fucking me up... It's been 4 hours and I'm still in the office alone.

It could be seen as n bug, when i was busy with file exports from a query on our web system i mannaged to skrew up the bitstream encoding for excel files and with a sysout the code started beeping everytime i exported the file. Managed to fix it by changing it to csv format. So no more beeping :D

Once again, the next build after hitting Ctrl + C while building a project surprised me with `java.lang.IllegalArgumentException: Malformed \uxxxx encoding`.

In which file? Is it so difficult to provide users with helpful information? Your `-X` option still does not show anything.

(Not to mention it must not corrupt ~/.m2/repository on SIGINT).

Fuck encoding and fuck PHP!!!
I'm programming a little vocab trainer to get used to php and MySQL. From an old VB vocab trainer I had ca. 2000 txt-files with words and converted them to sql-queries with a simple python script. When SELECTING words with special characters they become encoded properly. But if I UPDATE words their encoding is just fucked up... The table is utf-8 encoded all the columns are utf-8 encoded. The php mysqli connection is utf-8 encoded. My HTML header is utf-8... WTF?

Please, before exporting anything in whatever editor you use, check if it is in UTF-8. Today I didn't knew why my new font wasn't working in certain places and I later discovered that more than 9000 characters were replaced by the replacement character...

damn you LDAC....
just give me aptX

3 hours later I discover why my unset destroyed our json.

"When encoding an array, if the keys are not a continuous numeric sequence starting from 0, all keys are encoded as strings, and specified explicitly for each key-value pair."

Thanks obama.

Fuck those weird encoding issues with Python! I've read the HowTo Unicode 10 or 20 times and I still got those 'ordinal not in range error'!!!

I was in my IDE writing up a new program, but my compiler didn't accept the encoding format of the files. Because I used special characters, it flipped a lot of encodings around. Then I changed my projects master encoding, then all of the source files they all messed up. The all the files were composed of "????????" I went to a backup and they were messed up too. I exploded... Control Z couldn't even be my hero this time.

Never again will I use eclipses egit extension. First eclipse thought that my plain text java source code should be encoded in some bizarre occult way which made eclipse think its binary what made me try pretty much anything one can do with a .gitattributes file before a colleague suggested to not trust eclipse eith the encoding it was explicitly told to use, then I fetched another branch to merge them which somehow killed my .project file and forced me to delete and refetch the whole thing which led to eclipse not longer recognizing it as a java project. May it be because I'm to stupid to use my tools? Yeah, probably. But I'm done with egit, it's all console gitting from now on, fuck suggested practice.

BigBit standard for numeric format and character encoding

One-hot encoding is fucking garbage. Everyone loves using this useless stupid shit that doesn’t work.

Hello, community, how do you relax between encoding, when you take a pause after hard task?)

me Looks at beta 1.0 of encoding/encryption program

program Looks back

me :>

announcement bet1.0 is here

Is it just me or is encoding in 15.9.3 broken AGAIN when using .editorconfig to set the charset?.......

OK, so I've been working on processing a Japanese dictionary file and things are going smoothly for the most part. Out of ~185,000 entries, I've got 35 that are still causing problems.

The error I'm getting is "Incorrect string value '\xF0\xA4\xAD\xAF' for column...". I've checked all of my encoding and collation settings, and I'm pretty sure I've got it set to properly implement all of Unicode (as well as it does, anyway), as shown in the image attached. My suspicion is the problem characters are likely among the JIS X 0213 character set; in either case we're clearly dealing with a 4-byte character encoding issue here.

If needed I can attach a flag in the database and base64 encode these particular entries so the data isn't lost, but I'd like to just get it to handle the data properly in the first place if possible.

Anyone have any ideas on other items I can check to resolve the error?

Come on, how hard can it be?

On every fucking TLV data structure I get to handle, the hobo who defined the structure obviously stopped reading the TLV specification after the second sentence.
Fucked up tags, misuse of length encoding, and as a result no real TLV parser can handle that crap. Workarounds and manual parsing all over the place for *every* *single* interface.

Get your shit together, and if you don't want to handle the complex parts, then at least make the simple types right.

Sooo. My team and I have module we're supposed to be porting to async code and aiohttp will not work. The server keeps rejecting the byte payload, but if we use synch code like the requests library, it works fine. The code is like identical, the only difference is async. It's been really frustrating because another drop in async version of requests (httpx) works just fine! I don't want to use httpx, the rest of our codebase is already using aiohttp! We think the problem is with gzip encoding being handled incorrectly by aiohttp. I've reported the issue.

How's that character encoding going McDonalds?

Portrait of Me, Writting Documentation -- a short french film:

The processes applied to any section of memory utilized for a given purpose should be strictly limited to those declared by the associated type that encapsulates the purpose in question until release or mutation.

That is to say, improperly encoding the intended usage of such a block by utilizing an identical type or alias thereof for a multitude of incompatible situations, giving place to guesswork to arise, constitutes the prostitution of an abstraction.

Such heinous acts of symbolical pimping have received strong condemnation from multiple digital rights organizations, as well as our own, prestigious office. Let it be made Crystal, Alizé and Hennessy clear, that we will not stand for this kind of degenerate practice, and that any heretical sects and cabals built around worship of the strange creatures that arise every eleventh night from the depths of the Black Mausoleum will be prosecuted with the full force of the law.

As a young, corageous man once said at the peak of his career: "it is only through the self-inflicted, hyperbolic discharge of smouldered, comminute perennial anadenanthera colubrina spermatic fluid that the cannonical transfiguration of our collective rectosigmoid junction can be brought to fruition". He was immediately violated with might and ire far beyond our wildest, most profligately depraved fantasies, yet his message lives on.

I leave you now to be ritually and figuratively blown by a posssessed mortician that is to become concubine to our dark master; the long journey to the old graveyard will be perilous, and my destination most assuredly fatal, as I depart to give my firstborn to our Lord Berzchjanzad -- a blood sacrifice meant to appease him from peeling off my skin and refashioning it into a bloodied scarf to be worn around his thumping, grandemonic cock.

And in this moment, as I stare blankly at this teleprompter, the president wishes to reassure you of his sacred vows of stalwart and promethean gayhood, and may __these__ nuts bounce on chins forevermore. Here's to *not* bleeding to death in retribution for this unending litany of sins...

Yet all predictions come to pass.

««««««««««« finẽ »»»»»»»»»»»

when you find a single invisible character in each of your sql files in visual studio that causes a sql implosion saying

invalid syntax near ''

Sublime Text 3 and other tools didn't help. required manually finding this time, in 20 files..... with an overdue project.

I wanted to burn visual studio to the ground

Has anyone maybe a link to HTTP security topics in general?

I find often breadcrumbs, like in several different attack possibilities, but nothing comprehensive.

Mostly regarding HTTP 1.1 / HTTP 2 (h2c) and proxying.

I'm currently unclogging an whole ecosystem of proxies, endpoints, edge nodes and so on...

My knowledge is limited and it's frustrating to Google cause seemingly I get always just pieces of the puzzles but not a collection -.-

(Looking for specific information, e.g. regarding attacks like H2C Smuggling, HPACK attacks, stuff regarding Cookies / Headers / Encoding... But please not spread over several dozen pages where it becomes frustrating to read the same shit over and over again without learning something new :( )

GOD DAMN THAT OLD BROKEN DATABASE.

Having to work with a legacy old database system running MySQL 4.0 is a pain. Especially when even finding a frikin manual for the thing is hard af.

And a cherry on top is dealing with encoding and collation in a system, that didn't really have a wide support for it yet.

10/10. At least I am only dealing with it so that we could later shut it down for good.

For those who hate to like PHP. I have an issue. Character encoding.

A semicolon ' is replaced by ’

A fullstop . is replaced by ' Â

Any ideas?

First time would be when I had two teachers working of a bug that resultet in the CSS not loading / updating properly. It took me around 1 hour as well to realize that the text file was encoded with some obscure windows encoding, that doesn't load chrome (and maybe other browsers) can't load properly.

Thanks notepad++, will never use you again for web development.

Second time would be when I presented an app I made too the same teachers in order to boost my mark. Bugs that never appeared before every where and the basic functionality of the app was questionable. I hadn't worked on that app for half a year and had to fix as much as possible in a short amount of time, which might have made it worse.

Don't ask why but thanks to my knowledge about the france language I got a better mark, even though it was spanish

Fucking encoding types everywhere, spending nearly half of the day with guessing which encoding type is correct for some freaking files 😑

i can never understand the theme behind kotlin.

THEY DON'T HAVE A FUCKING TERNARY OPERATOR!!![?:]

Like before realizing this, i thought yeah jetbrains has decided to make android development a privileged hobby and non beginner friendly , so its now creating an encoding like language, in the false theme of " reducing code size"

But now they remove WORLDWIDE KNOWN, OPTIMIZED , EASY TO READ AND USE AND UNDERSTAND FEATURE of ternary operator and replacing it with less powerful but same looking elvis operator.( and stating that using if else for that is a better option)

Like why? if your goal is to make a shitty encoding language that makes everything shorter and most of the things optional, why remove the already efficient if else encoder?

God knows when this stupid language is going to stop my brain from getting blasted

Fuck powershell.

How in the world can one lang have so much problems with date conversion, and chars like üöä. Its so funny how they get through just fine sometime when u specify encoding utf8 but other times it has no ides wha5 these chars are.
Honestly fuck powershell

Let's say I take a matrix of high entropy random numbers (call it matrix J), and encode a problem into those numbers (represented as some integers which in turn represent some operations and data).

And then I generate *another* high entropy random number matrix (call it matrix K). As I do this I measure the Pearson's correlation coefficient between J (before encoding the problem into it, call Jb), and K, and the correlation between J (after encoding, let's call it Ja) and K.

I stop at some predetermined satisfactory correlation level, let's say > 0.5 or < -0.5

I do this till Ja is highly correlated with some sample of K, and Jb's correlation with K is close to 0.

Would the random numbers in K then represent, in some way, the data/problem encoded in Ja? Or is it merely a correlation?

Keep in mind K has no direct connection to J, Ja, or Jb, we're only looking for a matrix of high entropy random numbers that indicated a correlation to J and its data.

I say "high entropy", it would be trivial to generate random numbers with a PRNG that are highly correlated simply by virtue of the algorithm that generated them.