Whenever stuff like this is brought up, I'm reminded of how silly it is that we collect things with tar before compressing. The idea of separating file bundling and compression is fine, but tar is such a bad fit for modern usecases, no? Or at least non-streaming usecases. Whenever I write an NPZ file I find myself wondering whether I should start using foo.zip.lz or whatever instead. With the inner zip being uncompressed, of course.
Tar is short for tape archive. It is specifically intended for streaming because the intended use case is to take data and create a stream that can be written to the tape. Adding in compression is composable in exactly the same way.
As I recall, both zip and tar put their dictionaries at the end of the file, because that's when you have already written them (PDF does the same - write all of the objects, then write the dictionary at the end).
Compressing an uncompressed zip file has the same problems as compressing a tar archive: you need to decompress it entirely to be able to get any file out. This is why zip does per-file compression: the dictionary is uncompressed (but is tiny), each file can be pulled out one at a time and used. This is why zip works nicely as a read-only filesystem (each file can be decompressed on demand as it's used) but tar.gz does not (you must decompress the whole thing to read one file).
Modern compressors such as zstd have a dictionary mode, where you give it a load of data and it generates something roughly like a Huffman encoding of the common sequences across the entire data set, then you can individually compress files referencing that dictionary, so they're independent but the common parts are in the dictionary. I'd like to see a ZIP-like format that used this, with a shared dictionary but the ability to decode individual files (and, ideally, individual blocks of files, with something like a 2 MiB granularity) independently.
I wish there was a standarized "tar but with a table-of-contents" format.
Having to scan the entire archive just to find a file is awful, and it does cause issues in practice. My "favorite" one is how this means that tarsnap, when reading backups, has to do a roundtrip for every file in the archive because, after all the fancy deduplication stuff, it stores archives as plain tarballs. It's a very elegant design, limited by tar being crap (and there being no real alternatives for *nix).
(unless I'm wrong and zips can store xattrs etc)
You could even stay backwards-compatible with the current tar format if you wanted to.
Pretty sure xattrs work, BeOS and Haiku used ZIP in order to preserve BFS attributes.
"Each entry stored in a ZIP archive is introduced by a local file header with information about the file such as the comment, file size and file name, followed by optional "extra" data fields, and then the possibly compressed, possibly encrypted file data. The "Extra" data fields are the key to the extensibility of the ZIP format. "Extra" fields are exploited to support the ZIP64 format, WinZip-compatible AES encryption, file attributes, and higher-resolution NTFS or Unix file timestamps. Other extensions are possible via the "Extra" field. ZIP tools are required by the specification to ignore Extra fields they do not recognize."
Let's not forget the ultimate Unix hack of recursively copying a directory by tarring to standard input, piping that to a subshell that does cd and then untarring the standard output...
It's also a useful way of copying a directory tree over something like ssh: tar | gzip | ssh on one side and then gunzip | tar on the other side. Or even over nc if you trust the network...
Last week I needed to copy a directory with a ton of small files from one Mac to another on the same LAN. I did a simple Finder drag and drop, then saw it was taking forever to even start (it wants to enumerate all the files). Tried regular cp, saw that the actual copy crawled too (presumably SMB per-file overhead). Finally, I did an ssh tar | tar instead, for an, IDK, 10x speed increase?
I’ve been doing this exact same thing for 30-plus years on every OS. At some point you’d think file sharing protocol designers would realize remote enumeration of files and reading multiple files with a stream is a necessary thing.
I didn't know TAR did this before, and that got me into a rabbit hole exploring NAR. Archiving is simple enough that TAR would stay relevant. Compression is better angle for size decreases. Though NAR seems to be experimenting with compressed blocks of NAR for more parallelism.
By the way, what are the non-streaming use cases? Aren't archive formats like these usually made for transfer?
I’m working on a project that generates multiple .tar.gz archives, and I need to combine them into one final file. I thought I could just cat the bytes together, but that doesn’t work.
You could just tar the compressed files into one tape archive and later select out of it whichever .tar.gz file you need for further processing. However, doing that would not have taken you into this journey.
I accidentally stumbled upon the 'shar' command a few years ago and I had completely forgotten about it until you just mentioned it! I'd be curious to know what that command's limitations are as opposed to doing what the original author is facing.
I recently stumbled on chapter-tgz, which makes specially crafted tar files that let you skip over “chapters” without having to gunzip the inner blocks (and lets you decompress in parallel) which is a neat trick.
This is just weird to me - i cant think of any case beyond plain text where i would believe concatenating two files of some type would produce a single file that would appear to contain both?
There’s a lot of fluff in this article, but this has nothing to do with gzip or tar specifically: Concatenation to join files is the outlier. Even plain text formats often can’t be joined through concatenation.
Not encouraging it, but doesn't deflate have a flush mode which is basically "reset this stream"? Feels like a little bitstream twiddling at the start/end of each tarball could join the streams without recompressing
Zlib streams also have an “end of stream” signal so you can just feed the decompressor data until it tells you to stop. Git uses that property in its packfiles so it only needs to store the decompressed size of an entry.
However working with these raw streams at the shell is pretty obnoxious.
ISTM that the issue isn't that the author lacks understanding of tar and gzip, but that the lacks understanding of cat: it was intended for simple text files, and won't work properly if- as a simple example- the first file is terminated by ^Z and subsequent software honours it as an EOF.
This is not true, cat is binary safe and not limited to text files. It will copy the "^Z" (0x1a) just as it does as any other byte. There is no EOF character in Unix, ^Z is a CP/M-ism that DOS copied.
I think the commenter you're replying to understands that. They're giving the hypothetical of a utility that respects ^Z as an EOF, which cat will preserve causing additional file content to be ignored.
(In this case the hypothetical utility is tar, which has the concept of an EOF (EOA?) in the form of two zero blocks)
I could be misunderstanding it but still, after re-reading, the comment I replied to seems to suggest that since cat is meant for text files it wouldn't just pass everything through (including a 0x1a) as it is.
I'm sorry, but you do misunderstand my intent: cat /does/ pass everything through verbatim, including any header or footer (and I /did/ propose EOF as an explicitly simple example of the latter in certain cases). So basically, cat will mess up any file format where whatever reads it terminates at the first trailer (either because it recognises the record type or because there is a length in the header), rather than looking to see whether the trailer is followed by another header.
gspr | a day ago
Whenever stuff like this is brought up, I'm reminded of how silly it is that we collect things with tar before compressing. The idea of separating file bundling and compression is fine, but tar is such a bad fit for modern usecases, no? Or at least non-streaming usecases. Whenever I write an NPZ file I find myself wondering whether I should start using foo.zip.lz or whatever instead. With the inner zip being uncompressed, of course.
david_chisnall | 21 hours ago
Tar is short for tape archive. It is specifically intended for streaming because the intended use case is to take data and create a stream that can be written to the tape. Adding in compression is composable in exactly the same way.
As I recall, both zip and tar put their dictionaries at the end of the file, because that's when you have already written them (PDF does the same - write all of the objects, then write the dictionary at the end).
Compressing an uncompressed zip file has the same problems as compressing a tar archive: you need to decompress it entirely to be able to get any file out. This is why zip does per-file compression: the dictionary is uncompressed (but is tiny), each file can be pulled out one at a time and used. This is why zip works nicely as a read-only filesystem (each file can be decompressed on demand as it's used) but tar.gz does not (you must decompress the whole thing to read one file).
Modern compressors such as zstd have a dictionary mode, where you give it a load of data and it generates something roughly like a Huffman encoding of the common sequences across the entire data set, then you can individually compress files referencing that dictionary, so they're independent but the common parts are in the dictionary. I'd like to see a ZIP-like format that used this, with a shared dictionary but the ability to decode individual files (and, ideally, individual blocks of files, with something like a 2 MiB granularity) independently.
fanf | 19 hours ago
Tar doesn’t have any index at all, so even an uncompressed tar file needs a linear scan of the whole archive to find a particular file within.
(This article surprised me because I had forgotten that tar has an explicit EOF marker so I thought tar files are cattable.)
david_chisnall | 17 hours ago
Huh, I have also learned something today.
dzwdz | 10 hours ago
I wish there was a standarized "tar but with a table-of-contents" format.
Having to scan the entire archive just to find a file is awful, and it does cause issues in practice. My "favorite" one is how this means that tarsnap, when reading backups, has to do a roundtrip for every file in the archive because, after all the fancy deduplication stuff, it stores archives as plain tarballs. It's a very elegant design, limited by tar being crap (and there being no real alternatives for *nix).
(unless I'm wrong and zips can store xattrs etc)
You could even stay backwards-compatible with the current tar format if you wanted to.
jessicah | 3 hours ago
Pretty sure xattrs work, BeOS and Haiku used ZIP in order to preserve BFS attributes.
"Each entry stored in a ZIP archive is introduced by a local file header with information about the file such as the comment, file size and file name, followed by optional "extra" data fields, and then the possibly compressed, possibly encrypted file data. The "Extra" data fields are the key to the extensibility of the ZIP format. "Extra" fields are exploited to support the ZIP64 format, WinZip-compatible AES encryption, file attributes, and higher-resolution NTFS or Unix file timestamps. Other extensions are possible via the "Extra" field. ZIP tools are required by the specification to ignore Extra fields they do not recognize."
gerikson | 21 hours ago
Let's not forget the ultimate Unix hack of recursively copying a directory by tarring to standard input, piping that to a subshell that does
cdand then untarring the standard output...david_chisnall | 20 hours ago
It's also a useful way of copying a directory tree over something like ssh: tar | gzip | ssh on one side and then gunzip | tar on the other side. Or even over nc if you trust the network...
wrs | 16 hours ago
Last week I needed to copy a directory with a ton of small files from one Mac to another on the same LAN. I did a simple Finder drag and drop, then saw it was taking forever to even start (it wants to enumerate all the files). Tried regular
cp, saw that the actual copy crawled too (presumably SMB per-file overhead). Finally, I did anssh tar | tarinstead, for an, IDK, 10x speed increase?I’ve been doing this exact same thing for 30-plus years on every OS. At some point you’d think file sharing protocol designers would realize remote enumeration of files and reading multiple files with a stream is a necessary thing.
dseum | 23 hours ago
I didn't know TAR did this before, and that got me into a rabbit hole exploring NAR. Archiving is simple enough that TAR would stay relevant. Compression is better angle for size decreases. Though NAR seems to be experimenting with compressed blocks of NAR for more parallelism.
By the way, what are the non-streaming use cases? Aren't archive formats like these usually made for transfer?
JulianSildenLanglo | 23 hours ago
A non-streaming use case is that you sometimes just want one of the files from an archive.
adamo | 22 hours ago
shar enters the chat... Although
You could just
tarthe compressed files into one tape archive and later select out of it whichever.tar.gzfile you need for further processing. However, doing that would not have taken you into this journey.usrme | 20 hours ago
I accidentally stumbled upon the 'shar' command a few years ago and I had completely forgotten about it until you just mentioned it! I'd be curious to know what that command's limitations are as opposed to doing what the original author is facing.
jcspencer | 22 hours ago
I recently stumbled on chapter-tgz, which makes specially crafted tar files that let you skip over “chapters” without having to gunzip the inner blocks (and lets you decompress in parallel) which is a neat trick.
olliej | 11 hours ago
This is just weird to me - i cant think of any case beyond plain text where i would believe concatenating two files of some type would produce a single file that would appear to contain both?
There’s a lot of fluff in this article, but this has nothing to do with gzip or tar specifically: Concatenation to join files is the outlier. Even plain text formats often can’t be joined through concatenation.
xq | 2 hours ago
there's a fairly common thing to glue a filesystem of sorts to an application:
PE/ELF is read from the head, while ZIP is read from the back.
This means you get a ZIP archive AND an executable in a single file.
With libraries like physfs, you can then basically just open
argv[0], and get your shipped assets available.riking | 2 hours ago
Google's Protobuf has this property as an explicit design goal.
dw | 22 hours ago
Not encouraging it, but doesn't deflate have a flush mode which is basically "reset this stream"? Feels like a little bitstream twiddling at the start/end of each tarball could join the streams without recompressing
masklinn | 14 hours ago
Zlib streams also have an “end of stream” signal so you can just feed the decompressor data until it tells you to stop. Git uses that property in its packfiles so it only needs to store the decompressed size of an entry.
However working with these raw streams at the shell is pretty obnoxious.
MarkMLl | a day ago
ISTM that the issue isn't that the author lacks understanding of tar and gzip, but that the lacks understanding of cat: it was intended for simple text files, and won't work properly if- as a simple example- the first file is terminated by ^Z and subsequent software honours it as an EOF.
donio | a day ago
This is not true, cat is binary safe and not limited to text files. It will copy the "^Z" (0x1a) just as it does as any other byte. There is no EOF character in Unix,
^Zis a CP/M-ism that DOS copied.syncsynchalt | 14 hours ago
I think the commenter you're replying to understands that. They're giving the hypothetical of a utility that respects ^Z as an EOF, which
catwill preserve causing additional file content to be ignored.(In this case the hypothetical utility is
tar, which has the concept of an EOF (EOA?) in the form of two zero blocks)donio | 12 hours ago
I could be misunderstanding it but still, after re-reading, the comment I replied to seems to suggest that since cat is meant for text files it wouldn't just pass everything through (including a 0x1a) as it is.
MarkMLl | 2 hours ago
I'm sorry, but you do misunderstand my intent: cat /does/ pass everything through verbatim, including any header or footer (and I /did/ propose EOF as an explicitly simple example of the latter in certain cases). So basically, cat will mess up any file format where whatever reads it terminates at the first trailer (either because it recognises the record type or because there is a length in the header), rather than looking to see whether the trailer is followed by another header.
bedrovelsen | an hour ago
Also zgrep / zcat