Dear Python,

Seriously fuck your horrible compression APIs. What if I want to make a normal stream a tarball stream?

- Me

Wait what? What are you trying to do?

@py2js Imagine you're sending large backups in the form of .tar.xz files to AWS. The existing tarfile library doesn't allow for creating proper tar streams - Each file must be serialized into a tar entry (header + data pair) in one shot in order to get the padding right. This means that each file in the tar stream must be loaded into memory. Even if you send each entry over the wire and empty your buffers, a single big file could be problematic. Also, AFAIK the lzma library doesn't allow uncompressed filesize to be specified upfront - causing the whole archive to be loaded into memory. Alternatively, the end of section char could be used, but binary data might break the archive. The APIs could be a lot less abstract.