I've got a Linux Server running with 2 NVMe disks in RAID 1 configuration using mdadm. But if I want to create a big file, say 4GB, the whole system starts to hang.
I found out it's because of the journaling process which gives the CPU a long IOWAIT.

My problem is, I want to unzip a huge file, but this results in an immense server hang. Is there any way to do this without the server hang?
I unzip it inside of a docker container, if that is of any help.

How did you find that journaling process is to blame? Which FS?

I've extracted 2 10+ GB files on my nvme-equipped lappy today (non-RAID tho) w/o any issues.

IDK, I'd suspect SW-RAID...

@netikras
iotop command displayed jbd2/dm-0-8 as longest io blocking, that's where I now it from
Fs is ext4

The system is blocking for half a minute even after I killed the unzip programm, which makes sense because the process above does its io stuff after that, I guess.

@EaZyCode sorry, I'm out. Never had a chance to deal with sw-raid systems, dunno how to debug them

Unzip to tmpfs, split, copy parts, concat?

@netikras thanks anyway
@theKarlisK huh, I guess I'll try @gronostaj idea, that sounds like a reasonable approach

@netikras @theKarlisK @gronostaj
I solved the issue, if you want to know what the problem was read on.

Basically, it was the fault of caching, nothing new there. Just that this server has 64GB of RAM and the cache, before actually writing to disk, is 10% of that, 6.4GB. In normal usage that buffer will never be full.

There are 2 ways the buffer will be written to disk:
1. The buffer is full
2. The timer exceeded

Usually, after 30 seconds the buffer, with only a small amount of data, will be flushed.

But when unzipping, huge amounts of data is written into the buffer, it gets full and the buffer will be flushed. That is, 6.4GB have to be written almost instantly, because in the meantime, other processes hang.

The solution was to either disable caching while unzipping/writing a huge file or lowering the buffer.
I decided to lower the cache buffer to 64MB.

Yes this makes writing a huge file a lot slower but the system doesn't hang. Writing 64MB isn't that much of a deal.

Just for completeness, this was the command:

sysctl vm.dirty_bytes=64000000