Let’s hope the compression won’t kill this image.

Why do SAMBA network drives have to suck this much? Yeah I understand that compiling to a network drive is probably a bad idea just for performance reasons alone but can't you at least not fuck with my git repo?

$ git gc
Enumerating objects: 330, done.
Counting objects: 100% (330/330), done.
Delta compression using up to 24 threads
Compressing objects: 100% (165/165), done.
Writing objects: 100% (330/330), done.
Total 330 (delta 177), reused 281 (delta 151), pack-reused 0
error: unable to open .git/objects/7e: Not a directory
error: unable to open .git/objects/7e: Not a directory
fatal: unable to mark recent objects
fatal: failed to run prune

$ git gc
error: unable to open .git/objects/00: Not a directory
fatal: unable to add recent objects
fatal: failed to run repack

$ git gc
Enumerating objects: 330, done.
Counting objects: 100% (330/330), done.
Delta compression using up to 24 threads
Compressing objects: 100% (139/139), done.
Writing objects: 100% (330/330), done.
Total 330 (delta 177), reused 330 (delta 177), pack-reused 0
Removing duplicate objects: 100% (256/256), done.
error: unable to open .git/objects/05: Not a directory
error: unable to open .git/objects/05: Not a directory

Oh boyyy, I just had to work with Asterisk again. And holy shit it is still the clusterfuck it was many years ago.
We got:
- Inconsequent documentation that is mixed through all versions.
- The config sprinkled over what feels like 20 gazillion files.
- AEL being a half assed attempt at a "pRoGRamMinG LanGuAgE"
- The fuck you mean with extensions, endpoints and AOR's?
- Inconsistent config parameter naming. Some are snake case, some camel case some are just everything smushed into a single word.
- queue_log determines wheter to write a log to a file. queue_log_to_file Says to do so independent of you having a realtime backend. Whatever the fuck that is.
- Log compression is done by executing a gzip command after a rotation??!!?!!

So a couple of months ago I had some stability issues which seems to have caused Baloo go crazy and create an 1.7 exabyte index file. It was apparently mainly empty as zfs compressed it down to 535MB

Today I spent some time trying to reproduce the "issue" and turns out that wasn't that hard.

So this little program running on FreeBSD with a compressed (lz4) zfs dataset creates an 1.9 Exabyte large file, nicely compressed down to 45KB :)

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/limits.h>

int main(int argc, char** argv) {
int fd = open("bigfile.lge", O_RDWR|O_CREAT, 0644);
for (int i = 0 ; i < 1000000000; i++) {
lseek(fd, INT_MAX, SEEK_CUR);
}
write(fd, " ",1);
close(fd);
}

Alright, I sometimes... Alright often... almost every night while trying to fall asleep... imagine applications on an NP computer. No, I don't claim there is a NP computer. But still...

Alright, if you don't wanna think of NP computer... Think of non-deterministic turing machines, which are NP computers....

Quick recap about the NP rules:
- If you have a problem with a realm of solutions, your computer will guess the right answer in O(1), if an answer exists.
- After guessing an answer you have to confirm it using a normal deterministic approach that the answer is correct. No unconfirmed answers, no ambiguity.

Anyway... Data compression in an NP computer. I will make a claim that I don't wanna look up or calculate, but think it is correct:
1. There is a number n. If we have any number of bits smaller or equal to n, we cannot find two combination of bits, so that combination 1 and combination 2 both evaluate to the same md5 hash and have the same length.
2. The given number n is really large, so that at least a few gigabytes, if not terabytes can be described by it. (Hash collisions are generally allowed, just not between two hashes with the same amount of bits within the bit amount of n)

Now it is possible to send a whole file by just sending it's md5hash and how many bits are in the file (as long as the file is smaller or equal to n, otherwise slice it). Because the other side can just decompress it by guessing the right program and confirming it by hashing it again.

This would be compressed in O(n) and decompressed in O(n). So it would be extremely fast.

I mean, sometimes it is a pity that we don't have NP computers, but given that with enormous amounts of calculation powers and or enough memory space, every NP program can be run on a P computer, we can conclude that technically md5 is compression. Even though our computers are far too slow to actually use it as such.

Obviously not limited to md5. True for other hashes. Just n changes.

Do we need compression on api level? say I have a rest api sending json data on requests. So if compression is needed then should it be in the server when returning the json response or in the client side when receiving it? which one is ideal?

Would you consider compression (gzip static files) as a prepare step of the deploy stage or as a part of the build stage?

It's somewhat irrelevant but its bugging me