I pretty much implement the same thing in every codebase which needs to handle text properly, but the main issue is usually libc pulling you back towards null terminated strings. Also I really dislike that ".*s" requires the length of the string to be an int, but that might change soon.
Yes I also found myself doing this over and over which is why I decided to do it properly once and just reuse it whenever I need it. I'll admit, the printing part is really annoying (what you mention with the length having to be an int), bit what can you do, it's C. We love C even with all its rough edges, am I right?
I'm concerned about the STR and LENGTH_OF macros. Both
... STR("Foo");
and
const char* str = "Foo";
....
STR(str);
work, but in the latter LENGTH_OF returns sizeof(void*), and if you're lucky you will get a warning.
The author does state that this is for string literals, but that nothing in the construction makes enforces that, and it's easy enough for this to just work: the overwhelming majority of APIs that are used will be passed the data pointer.
The only place that easily gets the correct behavior if printf, and that only works because the %.*s format specifier does support providing a length. There's no equivalent for most string APIs, and there's no way to avoid lowering to just passing the data field, which renders the stored length irrelevant, and so the behavior appears correct.
This works because sizeof("" s) will only compile if s is a string literal, since C allows concatenating string literals like ("" "hello"), but not char* pointers like ("" str). Unfortunately, the compiler error is not very descriptive. Here's a godbolt example.
The problem here is that the author assumes too much. He assumes he understands the problem — more or less invents one — and sets about solving it. Then, as he approaches the very first hurdle, I/O, he cops out and says, well, you know, it really is NUL-terminated.
Among things more or less obviously missing is encoding. He makes a point about how important u8 is (debatable). Why in 2026 does “string” not countenance Unicode?
Was it tested on a machine other than with 8-bit characters? If not, whatever is the point? Any benefit or problem with the choice would be purely the author’s conjecture. There is a reason C chose char to be machine-dependent.
The feature set of the supplied functions is idiosyncratic. There are dozens of string libraries to compare with. How does it measure against std::string in C++ for example or, for that matter, against string.h in C?
The million-character string — that has no NUL! — is exceedingly rare. The need to find the terminator before acting on the data is virtually nonexistent. To compare for equality, for example, it is not necessary to know either length. Simply compare byte-for-byte until one doesn’t match. Not matching the NUL terminator is just one case.
Sorry to rain on the parade. The question of string-handling could fill a book and probably has. A handful of functions and a pointer-length pair just isn’t very interesting.
Was it tested on a machine other than with 8-bit characters? If not, whatever is the point? Any benefit or problem with the choice would be purely the author’s conjecture. There is a reason C chose char to be machine-dependent.
If you only care about systems you use (and POSIX mandates 8 bit char) whatever is the point of supporting those other "strange" systems? Like SQLite: https://sqlite.org/lts.html
(...) SQLite runs on any platform with an 8-bit byte (...)
He makes a point about how important u8 is (debatable). Why in 2026 does “string” not countenance Unicode?
Most modern languages use utf8 as an internal encoding, and that relies on u8.
Using something wider brings you almost nothing (in retrospect “wide strings” are generally considered a mistake) outside of very specialised use cases of mostly space savings e.g. utf16 can make sense if you have a massive corpus of low-to-no markup east asian text but that’s about it, and even then compression will recoup much of the difference, the raw content is about 30% larger in UTF8 but 0~10 once compression is applied to both.
utf32 makes pretty much no sense for general purpose representation, it blows up the storage for basically no reason (vectorised decoding works at line speed so you’re probably losing more in wasted cache and io traffic than gaining in skipping decode) and the savings in complexity are close to nil if not negative as generally speaking Unicode processing is inherently variable-width (because grapheme clusters), and depending on what you’re doing you can optimise down to processing at the CU so undecided UTF8 bytes can be sufficient for the job.
Most modern languages use utf8 as an internal encoding, and that relies on u8.
Really? Java and JavaScript both have assumptions in their String types that effectively mandate UTF-16. I guess they aren’t really ‘modern’ since they date from a time when Unicode has 16-bit code points.
Most other application languages entirely separate string storage from character access because this can lead to huge (factor of two) speedups in processing. For example, being able to special-case 7-bit ASCII strings can significantly improve cache usage, as can storing short strings embedded in pointers (which has the nice property that string comparison can be a single instruction, which is an enormous benefit in string-keyed maps). A huge proportion of strings in a typical program are things like path components or dictionary (e.g. JSON) keys, where storing them as short ASCII (or Huffman-encoded 6-bit character set) strings embedded in a pointer can make comparison cheap and avoid dynamic allocations.
In measurement I’ve done on desktop workloads, these optimisations reduce the total number of heap allocations by 10-20% (not just strings, all objects). Apple reached the same conclusion, which is why they rolled out the same thing for their APIs.
UTF-16 is less clear. If you’re doing rich text processing, you typically don’t store markup inline, you have either a separate table mapping attributes to ranges, or you have something like a DOM. For some Asian languages, the cache benefits of using UTF-16 for the currently edited / rendered text show up, especially when you start doing things like regex matching over an entire document, though they’re often dwarfed by font rendering time (less these days, as more of that is offloaded to the GPU). Compression matters for storage, but text is rarely a large enough proportion of storage to matter. You don’t want to be doing text compression in the cache because that makes anything involving random access very slow and it’s the size in cache that matters for performance.
I can't think of a language from the last 10~15 years which bothers with something else.
Java and JavaScript both have assumptions in their String types that effectively mandate UTF-16. I guess they aren’t really ‘modern’ since they date from a time when Unicode has 16-bit code points.
They're 30 years old, and from the maximally awkward era of the early 90s.
Most other application languages entirely separate string storage from character access because this can lead to huge (factor of two) speedups in processing.
Some do that in the background, and generally from the same era, but not necessarily because that's a good idea, more because they couldn't break their guarantees of O(1) "character" access (that's certainly why Python did it).
For example, being able to special-case 7-bit ASCII strings can significantly improve cache usage, as can storing short strings embedded in pointers (which has the nice property that string comparison can be a single instruction, which is an enormous benefit in string-keyed maps). A huge proportion of strings in a typical program are things like path components or dictionary (e.g. JSON) keys, where storing them as short ASCII (or Huffman-encoded 6-bit character set) strings embedded in a pointer can make comparison cheap and avoid dynamic allocations.
In UTF8 the ascii range is represented as ascii, all of that just works out the box (except huffman encoding but that's a hyper specific use case which you'd implement in the application anyway), at most you need the ability to bail to a slow path if you encounter a high bit set.
Apple reached the same conclusion, which is why they rolled out the same thing for their APIs.
Are you talking about Swift? Because that switched to UTF-8 with swift 5 back in 2019, it still has multiple storage classes and opaque bridging for objc compatibility, but native strings are in UTF8, the ascii/utf16 shape shifting is gone. It does still have various content-oriented flag for optimizations, like isASCII (or did last time I checked), but that's just that.
You don’t want to be doing text compression in the cache because that makes anything involving random access very slow
Any time you're doing random access in text it's probably broken, so that's not much of a concern.
Raku (formerly Perl 6) uses a really cool string representation that stores text as either length-prefixed ASCII byte arrays, length-prefixed 32-bit grapheme cluster arrays, or a concatenation of multiple strings. This allows for intuitive length ("🤦🏼♂️" has length 1) and fast/intuitive indexing or slicing. For multi-codepoint grapheme clusters, Raku creates a "synthetic grapheme" at runtime so it can be represented as a single negative integer. In this case, "x🤦🏼♂️y" would be stored as the 32-bit integers {120, -1, 122}, with a program-wide lookup table that maps -1 back to 🤦🏼♂️ for printing. Operations like string comparison, replacing text, substring searching, etc. can operate without needing to use the lookup table at all, since the mapping of -1 to 🤦🏼♂️ is canonical for that program run.
In my own programming language, I've extended this concept to improve on storage size using simple text compression. It stores chunks that use up to 256 unique grapheme clusters using a per-chunk lookup table and encoding each grapheme cluster as a single byte index into that table. This actually achieves significantly better compression than either UTF32 or UTF8 in most cases. For example, text written in Greek uses the Greek alphabet (which has fewer than 256 letters), but the UTF8 representation needs 2 or more bytes for each letter. With my language's text compression, you build a lookup table that lets you encode each letter as a single-byte index into a translation table that holds 32-bit grapheme clusters. This saves a lot of memory and preserves constant-time indexing, but at the cost of doing some pointer chasing in the lookup table (which is stored contiguously to mitigate that cost).
As a practical example, Euclid's Elements is encoded in UTF8 using 1.91 bytes per grapheme (1 byte for spaces and most punctuation, 2 bytes for most Greek letters), but my language's text compression can encode it with 1.001 bytes per grapheme (using a small lookup table plus exactly 1 byte for each of the 796k graphemes in the text). Plus, my language can access euclids_elements[400000] in constant time, rather than linear time for UTF8.
Was it tested on a machine other than with 8-bit characters? If not, whatever is the point? Any benefit or problem with the choice would be purely the author’s conjecture. There is a reason C chose char to be machine-dependent.
The saner solution to that conundrum is to let vendors of non-8-bit char platforms stew in their own self made misery and not pay them any heed. It's unlikely that they will want to build your software anyway, because if they cared about portability they surely wouldn't have been that contrarian.
I'm sorry you're not a fan of it. Of course, this is heavily tailored towards my use case for my (rather simple) string processing. I just though I would share it. Maybe calling this a library is not quite accurate, it's just some code I wrote that I thought others could find useful, so I thought to share it.
The only minor point is history. i.e. There have been many, many, many proposals to do similar things. Since none of them made the C standard, everyone is still inventing their own solutions.
And yes, I've done something similar in my code. Welcome to C. It's firmly stuck in the 1970s.
That's just... Not true. If n is the number of characters in my string my function is not O(n²) if anything it is O(n×m) where m is the length of the substring (which is usually very short) but even then it's almost always faster than that since it breaks as soon as one character does not match. That scenario almost never happens, it's almost always very close to lineare time because m << n and also for most characters you don't even begin checking for a substring match.
lor_louis | a day ago
I pretty much implement the same thing in every codebase which needs to handle text properly, but the main issue is usually libc pulling you back towards null terminated strings. Also I really dislike that
".*s"requires the length of the string to be an int, but that might change soon.lonjil | 11 hours ago
I would suggest never using libc other than where necessary.
[OP] eliasebner | 11 hours ago
Sometimes it's hard to go without it. For example sockets and file I/O in general.
[OP] eliasebner | 22 hours ago
Yes I also found myself doing this over and over which is why I decided to do it properly once and just reuse it whenever I need it. I'll admit, the printing part is really annoying (what you mention with the length having to be an int), bit what can you do, it's C. We love C even with all its rough edges, am I right?
olliej | 15 hours ago
I'm concerned about the
STRandLENGTH_OFmacros. Bothand
work, but in the latter
LENGTH_OFreturnssizeof(void*), and if you're lucky you will get a warning.The author does state that this is for string literals, but that nothing in the construction makes enforces that, and it's easy enough for this to just work: the overwhelming majority of APIs that are used will be passed the data pointer.
The only place that easily gets the correct behavior if printf, and that only works because the
%.*sformat specifier does support providing a length. There's no equivalent for most string APIs, and there's no way to avoid lowering to just passing thedatafield, which renders the stored length irrelevant, and so the behavior appears correct.spillybones | 5 hours ago
This is a bit of an ugly hack, but you can make
STR("hi")work whileSTR(str)gives a compiler error if you define it this way:This works because
sizeof("" s)will only compile ifsis a string literal, since C allows concatenating string literals like("" "hello"), but notchar*pointers like("" str). Unfortunately, the compiler error is not very descriptive. Here's a godbolt example.olliej | 5 hours ago
Oh nice! that's actually hilariously great :D
ManuLinares | 15 hours ago
Apart from C23
[[nodiscard]], this is standard C99.Languages like C3 already solve this natively with built-in slices.
jklowden | 18 hours ago
The problem here is that the author assumes too much. He assumes he understands the problem — more or less invents one — and sets about solving it. Then, as he approaches the very first hurdle, I/O, he cops out and says, well, you know, it really is NUL-terminated.
Among things more or less obviously missing is encoding. He makes a point about how important
u8is (debatable). Why in 2026 does “string” not countenance Unicode?Was it tested on a machine other than with 8-bit characters? If not, whatever is the point? Any benefit or problem with the choice would be purely the author’s conjecture. There is a reason C chose
charto be machine-dependent.The feature set of the supplied functions is idiosyncratic. There are dozens of string libraries to compare with. How does it measure against
std::stringin C++ for example or, for that matter, againststring.hin C?The million-character string — that has no NUL! — is exceedingly rare. The need to find the terminator before acting on the data is virtually nonexistent. To compare for equality, for example, it is not necessary to know either length. Simply compare byte-for-byte until one doesn’t match. Not matching the NUL terminator is just one case.
Sorry to rain on the parade. The question of string-handling could fill a book and probably has. A handful of functions and a pointer-length pair just isn’t very interesting.
hawski | 16 hours ago
If you only care about systems you use (and POSIX mandates 8 bit char) whatever is the point of supporting those other "strange" systems? Like SQLite: https://sqlite.org/lts.html
masklinn | 15 hours ago
Most modern languages use utf8 as an internal encoding, and that relies on u8.
Using something wider brings you almost nothing (in retrospect “wide strings” are generally considered a mistake) outside of very specialised use cases of mostly space savings e.g. utf16 can make sense if you have a massive corpus of low-to-no markup east asian text but that’s about it, and even then compression will recoup much of the difference, the raw content is about 30% larger in UTF8 but 0~10 once compression is applied to both.
utf32 makes pretty much no sense for general purpose representation, it blows up the storage for basically no reason (vectorised decoding works at line speed so you’re probably losing more in wasted cache and io traffic than gaining in skipping decode) and the savings in complexity are close to nil if not negative as generally speaking Unicode processing is inherently variable-width (because grapheme clusters), and depending on what you’re doing you can optimise down to processing at the CU so undecided UTF8 bytes can be sufficient for the job.
david_chisnall | 13 hours ago
Really? Java and JavaScript both have assumptions in their String types that effectively mandate UTF-16. I guess they aren’t really ‘modern’ since they date from a time when Unicode has 16-bit code points.
Most other application languages entirely separate string storage from character access because this can lead to huge (factor of two) speedups in processing. For example, being able to special-case 7-bit ASCII strings can significantly improve cache usage, as can storing short strings embedded in pointers (which has the nice property that string comparison can be a single instruction, which is an enormous benefit in string-keyed maps). A huge proportion of strings in a typical program are things like path components or dictionary (e.g. JSON) keys, where storing them as short ASCII (or Huffman-encoded 6-bit character set) strings embedded in a pointer can make comparison cheap and avoid dynamic allocations.
In measurement I’ve done on desktop workloads, these optimisations reduce the total number of heap allocations by 10-20% (not just strings, all objects). Apple reached the same conclusion, which is why they rolled out the same thing for their APIs.
UTF-16 is less clear. If you’re doing rich text processing, you typically don’t store markup inline, you have either a separate table mapping attributes to ranges, or you have something like a DOM. For some Asian languages, the cache benefits of using UTF-16 for the currently edited / rendered text show up, especially when you start doing things like regex matching over an entire document, though they’re often dwarfed by font rendering time (less these days, as more of that is offloaded to the GPU). Compression matters for storage, but text is rarely a large enough proportion of storage to matter. You don’t want to be doing text compression in the cache because that makes anything involving random access very slow and it’s the size in cache that matters for performance.
masklinn | 6 hours ago
I can't think of a language from the last 10~15 years which bothers with something else.
They're 30 years old, and from the maximally awkward era of the early 90s.
Some do that in the background, and generally from the same era, but not necessarily because that's a good idea, more because they couldn't break their guarantees of O(1) "character" access (that's certainly why Python did it).
In UTF8 the ascii range is represented as ascii, all of that just works out the box (except huffman encoding but that's a hyper specific use case which you'd implement in the application anyway), at most you need the ability to bail to a slow path if you encounter a high bit set.
Are you talking about Swift? Because that switched to UTF-8 with swift 5 back in 2019, it still has multiple storage classes and opaque bridging for objc compatibility, but native strings are in UTF8, the ascii/utf16 shape shifting is gone. It does still have various content-oriented flag for optimizations, like
isASCII(or did last time I checked), but that's just that.Any time you're doing random access in text it's probably broken, so that's not much of a concern.
spillybones | 4 hours ago
Raku (formerly Perl 6) uses a really cool string representation that stores text as either length-prefixed ASCII byte arrays, length-prefixed 32-bit grapheme cluster arrays, or a concatenation of multiple strings. This allows for intuitive length (
"🤦🏼♂️"has length 1) and fast/intuitive indexing or slicing. For multi-codepoint grapheme clusters, Raku creates a "synthetic grapheme" at runtime so it can be represented as a single negative integer. In this case,"x🤦🏼♂️y"would be stored as the 32-bit integers{120, -1, 122}, with a program-wide lookup table that maps-1back to🤦🏼♂️for printing. Operations like string comparison, replacing text, substring searching, etc. can operate without needing to use the lookup table at all, since the mapping of-1to🤦🏼♂️is canonical for that program run.In my own programming language, I've extended this concept to improve on storage size using simple text compression. It stores chunks that use up to 256 unique grapheme clusters using a per-chunk lookup table and encoding each grapheme cluster as a single byte index into that table. This actually achieves significantly better compression than either UTF32 or UTF8 in most cases. For example, text written in Greek uses the Greek alphabet (which has fewer than 256 letters), but the UTF8 representation needs 2 or more bytes for each letter. With my language's text compression, you build a lookup table that lets you encode each letter as a single-byte index into a translation table that holds 32-bit grapheme clusters. This saves a lot of memory and preserves constant-time indexing, but at the cost of doing some pointer chasing in the lookup table (which is stored contiguously to mitigate that cost).
As a practical example, Euclid's Elements is encoded in UTF8 using 1.91 bytes per grapheme (1 byte for spaces and most punctuation, 2 bytes for most Greek letters), but my language's text compression can encode it with 1.001 bytes per grapheme (using a small lookup table plus exactly 1 byte for each of the 796k graphemes in the text). Plus, my language can access
euclids_elements[400000]in constant time, rather than linear time for UTF8.kryptiskt | 12 hours ago
The saner solution to that conundrum is to let vendors of non-8-bit char platforms stew in their own self made misery and not pay them any heed. It's unlikely that they will want to build your software anyway, because if they cared about portability they surely wouldn't have been that contrarian.
[OP] eliasebner | 12 hours ago
I'm sorry you're not a fan of it. Of course, this is heavily tailored towards my use case for my (rather simple) string processing. I just though I would share it. Maybe calling this a library is not quite accurate, it's just some code I wrote that I thought others could find useful, so I thought to share it.
Cheers!
alandekok | 12 hours ago
It's interesting and useful.
The only minor point is history. i.e. There have been many, many, many proposals to do similar things. Since none of them made the C standard, everyone is still inventing their own solutions.
And yes, I've done something similar in my code. Welcome to C. It's firmly stuck in the 1970s.
izabera | 10 hours ago
your heavily tailored strstr/memmem takes quadratic time
[OP] eliasebner | 7 hours ago
That's just... Not true. If
nis the number of characters in my string my function is notO(n²)if anything it isO(n×m)wheremis the length of the substring (which is usually very short) but even then it's almost always faster than that since it breaks as soon as one character does not match. That scenario almost never happens, it's almost always very close to lineare time because m << n and also for most characters you don't even begin checking for a substring match.izabera | 6 hours ago