4
slar
3y

So a couple of months ago I had some stability issues which seems to have caused Baloo go crazy and create an 1.7 exabyte index file. It was apparently mainly empty as zfs compressed it down to 535MB

Today I spent some time trying to reproduce the "issue" and turns out that wasn't that hard.

So this little program running on FreeBSD with a compressed (lz4) zfs dataset creates an 1.9 Exabyte large file, nicely compressed down to 45KB :)

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/limits.h>

int main(int argc, char** argv) {
int fd = open("bigfile.lge", O_RDWR|O_CREAT, 0644);
for (int i = 0 ; i < 1000000000; i++) {
lseek(fd, INT_MAX, SEEK_CUR);
}
write(fd, " ",1);
close(fd);
}

Comments
  • 6
    This is called an sparse file. The file looks bigger than it really is.some programs use this to preallocate space.
  • 1
    You are absolutely correct and my assumption that zfs compression was doing most of the work here was completely incorrect. The compression ratio of this dataset is still 1, it's just a sparse file.

    It would have been quicker just to run truncate 2E...
  • 0
    @slar not all files can be truncated. Truncating an iso for example breaks the file.
Add Comment