I couldn’t see in the examples how can a program report an error when the user passes an unknown option?
The short option syntax doesn’t follow the traditional unix / POSIX syntax which treats -f arg or -farg the same. (Sadly this is a fairly common bug.) The standard syntax requires that the option parser is told whether a flag takes an argument before it continues parsing after the flag, so that it can tell the difference between grouped flags and a flag with argument. (It’s possible to tell a lexopt-style parser that an argument is expected immediately after it parses the flag, but that’s the latest possible moment. Traditionally the parser gets told at the earliest possible moment, before it starts.)
Great points, thanks for engaging with it. To your first question, you'd have to inspect the data you get back when you call positionals. I've an example (well, example approach in prose) of that in https://codeberg.org/hgrsd/zopt#api-surface but I should also add it to the positionals.zig example file. I've been toying with the idea of considering anything before the -- terminator that starts with - or -- an unknown option, but so far zopt doesn't assume whether an 'unclaimed' option is an unknown option or a positional argument.
Your other points show up the limitations of the schemaless design. I'll have to think about them a bit to see whether they can be addressed, but I suspect they might be fundamental limitations that this type of approach has.
One thing I just realised is that, because it is a schemaless parser and the way you call the functions determines the behaviour, I can just add an unknownOptions() function that does exactly that: if any unclaimed arguments prior to the -- terminater is prefixed with - or --, they can be considered unknowns. And it's up to the user of the library to decide whether or not to use that functionality, based on how their CLI works.
No informed opinion on the right Unix way to parse CLI arguments, but, from the language design perspective, I want to highlight that Zig lends itself exceptionally well to declarative parsing. Eg, at TigerBeetle we do something like this:
The fn Flags.parse(T: type) T then uses comptime to map runtime args to T in a way which is significantly more straightforward than what you'd have in Rust (zig version vs rust version)
Something along these lines might get into stdlib soon.
I extracted the TigerBeetle argparser (with a couple of tweaks) and updated it for Zig 0.16 compatibility not too long ago: https://git.sr.ht/~zmitchell/argparser
That'd be very cool! I enjoyed watching your videos on Tigerbeetle's arg parsing. The struct-based declarative approach with a schema upfront is obviously superior for many use cases than what zopt does, which tries to fill a different niche.
Flags parsing in the standard library would be sweet. Are there any actual threads discussing this yet that you know of or is this just a goal that has been expressed fleshing out soon?
Congratulations on releasing this, and thank you for sharing.
I have been doing some design work on command-line argument processing (in Zig and other languages,) and I've actually come to the conclusion that every command-line parsing library has adopted the wrong abstraction.
While I can appreciate the convenience and terseness provided by declarative interfaces, since good command-line interfaces usually form an inconsistent grammar, they cannot be correctly parsed in a declarative manner (and specifying the grammar is usually too clumsy.) However, they also usually just simple enough to support ad hoc parsing. Therefore, the ideal is actually a switch inside a while loop, and, consequently, the parsing library should be designed to support this.
Along the way, I've discovered some missing pieces of command-line interface and parsing design that may be of interest—e.g., auto-canonicalisation. At its simplest, this feature is easy to provide by both a declarative and imperative parsing API; however, in the imperative API, it seems easier to provide the ad hoc canonical variations that are most useful.
My command-line interfaces can (usu. to one or two fixed-degrees) auto-canonicalise: given a command-line invocation, they can emit the canonical form. e.g., given command --canonicalise -a -b -c x y z they can emit a prescribed canonical form of command --ayy --bee --see ecks --dee default y z.
The idea is to avoid a problem we see very frequently in qemu invocations. There are at least four ways to indicate that you want to use KVM: -enable-kvm, -machine accel=kvm, -M accel=kvm, -accel kvm. Most tutorials and blog posts about qemu randomly pick one of these for their examples. Even if you read through the man page very careful, it is quite difficult to distinguish which of these is the one you want. You actually have to read through the qemu command-line parsing code (and version control commit messages) to figure out which you want. qemu is full of these “conveniences” that inevitably become “legacy” approaches. Providing the conveniences is the right thing to do for the users, but the tool (by virtue of its authors knowing what they consider canonical) should also be able to provide guidance on canonical forms. This way, we can bridge interactive, iterative, or exploratory use and automated, scripted use.
(Similarly, it is quite surprising to me how few tools support self-quoting, despite having glaring encoding ambiguities—e.g., socat unix-listen:${path} …—but this isn't something where better command-line parsing libraries would any difference.)
ad hoc parsing. Therefore, the ideal is actually a switch inside a while loop
By the way, the design of this tool is actually much closer to the ad hoc switch inside a while, but I'm not sure how you would handle conjoined options like -abc for -a-b-c (e.g., tar -xavf)† or ambiguities where you have to peek ahead a fix amount like flags with optional qualifying arguments. The former is somewhat controversial, but I don't think the latter is really that wild, and both of these are fairly easy to implement with an ad hoc switch inside a while.
(† Actually, for this, you would just do multiple scans, but this means your argument parser has to either take ownership of the arguments or rely on someone else having ownership of the arguments. In other words, you'll have to allocate if you want to parse arguments from, e.g., stdin, which it's not as unusual as it may sound…)
Similarly, .flag doesn't afford any modalities—it just scans from .args[0...flag_opt_iteration_end]—so embedded commands (e.g., ssh) will probably require some special designated terminator (e.g., --) to set the upper bound on the range, but that probably leads to other problems.
I suspect embedded commands are actually a fairly large subset of actual command-line utilities people want to write (and they typically write these as shell scripts, but I think deploying a Zig statically linked binary may sometimes be a better choice overall.) They want to “wrap” some common sequence of operations with a fit-for-purpose interface.
My perspective is essentially this: corporate internal tool CLIs are usually terrible, because they're designed with very little care. (LLM CLI design recommendations are also often terrible, because they are trained on terrible inputs.)
What makes them terrible is not mere sloppiness—e.g., using stdout instead of stderr for error reporting—or lack of care for fluency—e.g., failing to support piping to stdin or from stdout to support composable automation—but simple lack of care for design. (I, perhaps unfairly, have in previous comments blamed the incredibly primitive design of Go's flags module for the carelessly poor design of a lot of automation tooling like Docker.)
The command-line interface is the primary thing that the users will see and interact with. While it is undoubtedly true that they care most about the consequences of the command they're running (i.e., what it does,) they will also care about how convenience and intuitive it was to achieve those ends (assuming, of course, they wrote the command-line invocation by hand.)
Thus, for a certain (common, well-circumscribed) class of command-line scripts, there's benefit in properly designing the command-line interface. In these cases, I often first write the usage text by hand as a single multiline string and think through what a user would want to do, how they might go about doing it, what their assumptions and intuitions will be, &c. Then I go and implement that UI.
Of course, what makes makes for a convenient, intuitive tool doesn't necessarily mean that the grammar that the UI presents is readily expressible in, say, EBNF. In fact, it may be the case that the grammar may be unambiguous but may be context-sensitive, and, consequently, resistant to common declarative parsing techniques. Of course, it's also usually the case that an ad hoc, hand-written, imperative parser is not only more than sufficient to handle these grammars, but the resulting code is very easy for someone (i.e., most programmers who are accustomed to reading through imperative code,) to read through and understand.
Also, as a practical matter, programme configuration is usually a (pure?) function of the command-line arguments, environmental variables, and configuration file contents. If a programme is misbehaving, it should be trivial for someone to read through the code to see exactly how these sources interact, especially considering that these interactions may introduce modalities and may have complex prioritisation. (By the way, it would be very nice if command-line interfaces routinely provided some variation of --config-file /dev/null and --ignore-env so that users can safely use them without having to bwrap … --tmpfs ~ --clearenv … or env -i … to get reliable, repeatable state. One of the benefits of auto-canonicalisation is that it can reveal these hidden dependencies in the iterative prototyping enviroment. One auto-canonicalisation format I like to support is where you get the canonical command-line arguments in shell-quoted form with attached shell comments on why they were set—e.g., --dee default # ~/.config/app.conf line 3.)
In fact, I would argue that there are a number of very common patterns that arise in command line interfaces that become incredibly unwieldy to handle in a declarative manner yet nearly trivial to write and read in an imperative style…
Hmm, my experience is almost exactly the opposite. I've also been exposed to plenty of corporate internal CLIs, and the more ad-hoc and inconsistent their grammar is, the worse.
Usually, the developer creates the ad-hoc parser based around their own intuition, and tests it with command lines that they write using their own intuition. Then when I try it, since I'm a different person, I have a different intuition and run into weird/broken cases immediately. The worst is when the CLI is accidentally sensitive to flag ordering, because the ad-hoc parser does imperative stuff directly as it encounters the flags.
So for a corporate CLI, which I agree is likely written with less care, I ask people to please just use some boring declarative library. The CLI may end up being slightly less ergonomic but that's fine, I much prefer that to debugging your broken parser.
I think the switch inside a while loop (which, again, may actually be very similar to what you have created,) is the dominant style for old C programmes. A lot of GNU utilities look like this. A lot of Shell scripts look this way, too (and I have previously made the mistake of using tools like zparseopts which are convenient but quickly become a huge mess.)
Zig's switch and pattern matching are actually quite good, and (comptime-defined) iterators are a surprisingly good fit for the language, so we can do this (many details elided):
When trying to write some command-line tooling recently, I found that I became very picky about certain capabilities, based on irritations I faced in very useful but unfortunately designed (yet common) tools like ssh, rsync, socat, qemu, &c.
The ArgsIterator gives you a non-owning, non-allocating iterator over some []const []const u8 transformed into something like a union(enum) { long: []const u8, short: []const u8, ...} that lets you split conjoined -abc → -a, -b, -c options and pattern match against how the pattern-matched switch target was discovered (so, e.g., you can distinguish -f from --f while pattern matching against just the "f" part turned into an enum { f, ... })
This (and your library) are definitely huge improvements over scattered std.mem.eql(u8, ..., ...) (as I've seen a few times in Zig projects,) but I think using a StaticStringMap to turn these into pattern-matchable enum values is quite slick, especially since I also have some facilities for transforming those enum values into enum values that make nested switches more compact. e.g., if you had a chown/chgrp-like that allows --user UID or --user USERNAME or --group GID or --group GROUPNAME, you would want an outer switch on .u, .user, .g, .group and maybe a terser inner switch on enum { user, group } as well as an inner switch on enum { uid, name }.
These comments combined should be a blog post themselves :)
Just some ramblings to add:
One thing that's been on my mind lately is; how we might go about creating extensible cli applications from an API standpoint. In games we're often so dynamic that it's pretty common to ship the executable which probably does most of the cli parsing and providing the API, but then have most of the cli arguments and options way down stream in multiple dynamic libraries (executable=engine, dlls=games).
Including even potentially top-level "subcommands". For example, let's say you have something like a simplified git; with init, clone, etc. how might this work (well!) if each one of those sub-commands is a dll and the primary executable has absolutely no idea about "clone" not to mention the arguments to clone ?
Similarly we might even go further and do something like engine.exe --project "myproject", where the project flag would load myproject; but NOT myotherproject. How might our help output look in the various cases of engine.exe --help vs engine.exe --help --project "myproject" etc? What might our error handling look like if just the project name is wrong but you have a bunch of flags that project defined/uses? What do definition collisions look like when you're gathering all these subcommands / flags from different sources?
Note that obviously this is a pretty niche usecase, and not every command line API needs to support this, but once we're in multi-million LOC codebases with 100s of developers some conventions have to change.
Historically game engines have had very poor CLI support as a result of this. In my own engine there's currently no way to list all options or arguments for example, but it's absolutely been on my mind as something to fix. It's entirely possible that maybe there's entirely different expectations for how this works for a true CLI application, a GUI application, and maybe even a service / daemon type application too. Ideally I'd like a similar API for them though.
My understanding is that games programming is still very Windows-focused. If that is the case, isn't the general expectation that the command-line environment will be very weak (and, in any case, won't be very POSIX/Linux-like)?
extensible cli applications from an API standpoint
each one of those sub-commands is a dll and the primary executable has absolutely no idea
multi-million LOC codebases with 100s of developers
Presumably, you're still able to coördinate in some fashion—via an API or some kind of convention.
Of course, while we may readily assume historical technologies would be insufficient for automatically deriving a command-line grammar from --help output, I suspect this might not be beyond what a language model can accomplish.
pretty niche usecase
I suspect the unaffiliated “subcommand” use-case is actually quite common in practice. Consider that a typical shell script will invoke multiple subordinate commands to accomplish its task. It will expose flags on some of these commands but not on others. Inevitably, there will be some modality in a subordinate command that is necessary but cannot be exposed.
If the shell script cannot be rewritten, the user will either try to trigger that modality via alternate means (an environmental variable or configuration file entry,) or they will try to intercept the invocation of the command and perform some kind of rewriting (e.g., write a stub entrypoint, define a shell function with the same name, and source the original script, or write a shell script with same name somewhere eariler in $PATH.)
Some overeager script authors will actually expose these parameters in some fashion, like a command-line --subcommand-flags argument or an environmental variable SUBCOMMAND_FLAGS. rsync actually has two flags for this: --rsh or -e and --rsync-path (the latter of which can intercept the rsync invocation on the remote side.) I have an unpublished blog post where I use this to implement remote⇋remote rsync transfers (intermediated through the local machine, like in scp.)
each one of those sub-commands is a dll and the primary executable
If the subordinate units are actually DLLs rather than separate executables, that they are written by authors willing to follow a convention, and that they are very similar in their development (e.g., all written in the same programming language targeting the same platform,) I suspect that these simplifications will afford you a lot of freedom. If the command-line parsing is guaranteed to always nicely fit into a declarative model, then you just have to expose that in some fashion. If the command-line parsing cannot fit into a declarative model, then you would have to expose the parser function.
Without these simplifications, I think the problem becomes very hard. In fact, my interest in Zig for command-line parsing is a result of realising that the N×M problem of commands × shell completions is largely unavoidable. Every command needs to have a shell completion written for every shell. If there were a single, uniform declarative model that would work for all commands, then we would ask authors to write these, and have shells parse them to create completions; but, as I hope to have demonstrated previously, I don't think this is actually feasible.
Instead, I have been investigating whether completion can be internalised into a very fast command-line parser itself. If the parser is fast enough, then the completion can be a thin(ner) wrapper against invoking the program itself. (It is, after all, the foremost authority on its own completion.)
Furthermore, if the parser is written against a non-allocating iterator that can be safely copied, then we can serialise this state between completions to support incremental parsing. We may even be able to do parsing, completion, and auto-canonicalisation within the same body of code.
If you couple this with some sort of Zig plugin per shell that can rapidly parse some highly structured completion return value, then you could probably get very good quality completion across all shells with the vast majority of the code sitting in the command itself.
Actually, if you squint, this is not that different than the problem you describe, is it?
fanf | 17 hours ago
I couldn’t see in the examples how can a program report an error when the user passes an unknown option?
The short option syntax doesn’t follow the traditional unix / POSIX syntax which treats
-f argor-fargthe same. (Sadly this is a fairly common bug.) The standard syntax requires that the option parser is told whether a flag takes an argument before it continues parsing after the flag, so that it can tell the difference between grouped flags and a flag with argument. (It’s possible to tell a lexopt-style parser that an argument is expected immediately after it parses the flag, but that’s the latest possible moment. Traditionally the parser gets told at the earliest possible moment, before it starts.)[OP] hgrsd | 17 hours ago
Great points, thanks for engaging with it. To your first question, you'd have to inspect the data you get back when you call positionals. I've an example (well, example approach in prose) of that in https://codeberg.org/hgrsd/zopt#api-surface but I should also add it to the positionals.zig example file. I've been toying with the idea of considering anything before the -- terminator that starts with - or -- an unknown option, but so far zopt doesn't assume whether an 'unclaimed' option is an unknown option or a positional argument.
Your other points show up the limitations of the schemaless design. I'll have to think about them a bit to see whether they can be addressed, but I suspect they might be fundamental limitations that this type of approach has.
[OP] hgrsd | 15 hours ago
One thing I just realised is that, because it is a schemaless parser and the way you call the functions determines the behaviour, I can just add an
unknownOptions()function that does exactly that: if any unclaimed arguments prior to the--terminater is prefixed with-or--, they can be considered unknowns. And it's up to the user of the library to decide whether or not to use that functionality, based on how their CLI works.fanf | 15 hours ago
Yeah, that’s the kind of thing I was looking for :-)
[OP] hgrsd | 13 hours ago
I spent a bit of time hacking on this feature this evening: https://codeberg.org/hgrsd/zopt/src/commit/21fb5152fe1f829e7e11f9f47a895c95ac471d69/examples/01-positional.zig#L19 :)
Thanks again for your feedback
matklad | 11 hours ago
No informed opinion on the right Unix way to parse CLI arguments, but, from the language design perspective, I want to highlight that Zig lends itself exceptionally well to declarative parsing. Eg, at TigerBeetle we do something like this:
The
fn Flags.parse(T: type) Tthen uses comptime to map runtime args toTin a way which is significantly more straightforward than what you'd have in Rust (zig version vs rust version)Something along these lines might get into stdlib soon.
zmitchell | 10 hours ago
I extracted the TigerBeetle argparser (with a couple of tweaks) and updated it for Zig 0.16 compatibility not too long ago: https://git.sr.ht/~zmitchell/argparser
[OP] hgrsd | 2 hours ago
That'd be very cool! I enjoyed watching your videos on Tigerbeetle's arg parsing. The struct-based declarative approach with a schema upfront is obviously superior for many use cases than what zopt does, which tries to fill a different niche.
jesseb34r | 11 hours ago
Flags parsing in the standard library would be sweet. Are there any actual threads discussing this yet that you know of or is this just a goal that has been expressed fleshing out soon?
liberty | 6 hours ago
https://codeberg.org/ziglang/zig/issues/30677
dutc | 16 hours ago
Congratulations on releasing this, and thank you for sharing.
I have been doing some design work on command-line argument processing (in Zig and other languages,) and I've actually come to the conclusion that every command-line parsing library has adopted the wrong abstraction.
While I can appreciate the convenience and terseness provided by declarative interfaces, since good command-line interfaces usually form an inconsistent grammar, they cannot be correctly parsed in a declarative manner (and specifying the grammar is usually too clumsy.) However, they also usually just simple enough to support ad hoc parsing. Therefore, the ideal is actually a
switchinside awhileloop, and, consequently, the parsing library should be designed to support this.Along the way, I've discovered some missing pieces of command-line interface and parsing design that may be of interest—e.g., auto-canonicalisation. At its simplest, this feature is easy to provide by both a declarative and imperative parsing API; however, in the imperative API, it seems easier to provide the ad hoc canonical variations that are most useful.
My command-line interfaces can (usu. to one or two fixed-degrees) auto-canonicalise: given a command-line invocation, they can emit the canonical form. e.g., given
command --canonicalise -a -b -c x y zthey can emit a prescribed canonical form ofcommand --ayy --bee --see ecks --dee default y z.The idea is to avoid a problem we see very frequently in
qemuinvocations. There are at least four ways to indicate that you want to use KVM:-enable-kvm,-machine accel=kvm,-M accel=kvm,-accel kvm. Most tutorials and blog posts aboutqemurandomly pick one of these for their examples. Even if you read through themanpage very careful, it is quite difficult to distinguish which of these is the one you want. You actually have to read through theqemucommand-line parsing code (and version control commit messages) to figure out which you want.qemuis full of these “conveniences” that inevitably become “legacy” approaches. Providing the conveniences is the right thing to do for the users, but the tool (by virtue of its authors knowing what they consider canonical) should also be able to provide guidance on canonical forms. This way, we can bridge interactive, iterative, or exploratory use and automated, scripted use.(Similarly, it is quite surprising to me how few tools support self-quoting, despite having glaring encoding ambiguities—e.g.,
socat unix-listen:${path} …—but this isn't something where better command-line parsing libraries would any difference.)dutc | 16 hours ago
By the way, the design of this tool is actually much closer to the ad hoc
switchinside awhile, but I'm not sure how you would handle conjoined options like-abcfor-a-b-c(e.g.,tar -xavf)† or ambiguities where you have to peek ahead a fix amount like flags with optional qualifying arguments. The former is somewhat controversial, but I don't think the latter is really that wild, and both of these are fairly easy to implement with an ad hocswitchinside awhile.(† Actually, for this, you would just do multiple scans, but this means your argument parser has to either take ownership of the arguments or rely on someone else having ownership of the arguments. In other words, you'll have to allocate if you want to parse arguments from, e.g.,
stdin, which it's not as unusual as it may sound…)Similarly,
.flagdoesn't afford any modalities—it just scans from.args[0...flag_opt_iteration_end]—so embedded commands (e.g.,ssh) will probably require some special designated terminator (e.g.,--) to set the upper bound on the range, but that probably leads to other problems.I suspect embedded commands are actually a fairly large subset of actual command-line utilities people want to write (and they typically write these as shell scripts, but I think deploying a Zig statically linked binary may sometimes be a better choice overall.) They want to “wrap” some common sequence of operations with a fit-for-purpose interface.
[OP] hgrsd | 16 hours ago
That's a very interesting perspective. Are you thinking of something like https://github.com/blyxxyz/lexopt?
dutc | 15 hours ago
My perspective is essentially this: corporate internal tool CLIs are usually terrible, because they're designed with very little care. (LLM CLI design recommendations are also often terrible, because they are trained on terrible inputs.)
What makes them terrible is not mere sloppiness—e.g., using stdout instead of stderr for error reporting—or lack of care for fluency—e.g., failing to support piping to stdin or from stdout to support composable automation—but simple lack of care for design. (I, perhaps unfairly, have in previous comments blamed the incredibly primitive design of Go's
flagsmodule for the carelessly poor design of a lot of automation tooling like Docker.)The command-line interface is the primary thing that the users will see and interact with. While it is undoubtedly true that they care most about the consequences of the command they're running (i.e., what it does,) they will also care about how convenience and intuitive it was to achieve those ends (assuming, of course, they wrote the command-line invocation by hand.)
Thus, for a certain (common, well-circumscribed) class of command-line scripts, there's benefit in properly designing the command-line interface. In these cases, I often first write the usage text by hand as a single multiline string and think through what a user would want to do, how they might go about doing it, what their assumptions and intuitions will be, &c. Then I go and implement that UI.
Of course, what makes makes for a convenient, intuitive tool doesn't necessarily mean that the grammar that the UI presents is readily expressible in, say, EBNF. In fact, it may be the case that the grammar may be unambiguous but may be context-sensitive, and, consequently, resistant to common declarative parsing techniques. Of course, it's also usually the case that an ad hoc, hand-written, imperative parser is not only more than sufficient to handle these grammars, but the resulting code is very easy for someone (i.e., most programmers who are accustomed to reading through imperative code,) to read through and understand.
Also, as a practical matter, programme configuration is usually a (pure?) function of the command-line arguments, environmental variables, and configuration file contents. If a programme is misbehaving, it should be trivial for someone to read through the code to see exactly how these sources interact, especially considering that these interactions may introduce modalities and may have complex prioritisation. (By the way, it would be very nice if command-line interfaces routinely provided some variation of
--config-file /dev/nulland--ignore-envso that users can safely use them without having tobwrap … --tmpfs ~ --clearenv …orenv -i …to get reliable, repeatable state. One of the benefits of auto-canonicalisation is that it can reveal these hidden dependencies in the iterative prototyping enviroment. One auto-canonicalisation format I like to support is where you get the canonical command-line arguments in shell-quoted form with attached shell comments on why they were set—e.g.,--dee default # ~/.config/app.conf line 3.)In fact, I would argue that there are a number of very common patterns that arise in command line interfaces that become incredibly unwieldy to handle in a declarative manner yet nearly trivial to write and read in an imperative style…
muvlon | 2 hours ago
Hmm, my experience is almost exactly the opposite. I've also been exposed to plenty of corporate internal CLIs, and the more ad-hoc and inconsistent their grammar is, the worse.
Usually, the developer creates the ad-hoc parser based around their own intuition, and tests it with command lines that they write using their own intuition. Then when I try it, since I'm a different person, I have a different intuition and run into weird/broken cases immediately. The worst is when the CLI is accidentally sensitive to flag ordering, because the ad-hoc parser does imperative stuff directly as it encounters the flags.
So for a corporate CLI, which I agree is likely written with less care, I ask people to please just use some boring declarative library. The CLI may end up being slightly less ergonomic but that's fine, I much prefer that to debugging your broken parser.
dutc | 16 hours ago
Very similar.
I think the
switchinside awhileloop (which, again, may actually be very similar to what you have created,) is the dominant style for old C programmes. A lot of GNU utilities look like this. A lot of Shell scripts look this way, too (and I have previously made the mistake of using tools likezparseoptswhich are convenient but quickly become a huge mess.)Zig's
switchand pattern matching are actually quite good, and (comptime-defined) iterators are a surprisingly good fit for the language, so we can do this (many details elided):When trying to write some command-line tooling recently, I found that I became very picky about certain capabilities, based on irritations I faced in very useful but unfortunately designed (yet common) tools like
ssh,rsync,socat,qemu, &c.The
ArgsIteratorgives you a non-owning, non-allocating iterator over some[]const []const u8transformed into something like aunion(enum) { long: []const u8, short: []const u8, ...}that lets you split conjoined-abc→-a,-b,-coptions and pattern match against how the pattern-matchedswitchtarget was discovered (so, e.g., you can distinguish-ffrom--fwhile pattern matching against just the"f"part turned into anenum { f, ... })This (and your library) are definitely huge improvements over scattered
std.mem.eql(u8, ..., ...)(as I've seen a few times in Zig projects,) but I think using aStaticStringMapto turn these into pattern-matchableenumvalues is quite slick, especially since I also have some facilities for transforming thoseenumvalues intoenumvalues that make nestedswitches more compact. e.g., if you had achown/chgrp-like that allows--user UIDor--user USERNAMEor--group GIDor--group GROUPNAME, you would want an outer switch on.u, .user, .g, .groupand maybe a terser inner switch onenum { user, group }as well as an inner switch onenum { uid, name }.rileylabrecque | 8 hours ago
These comments combined should be a blog post themselves :)
Just some ramblings to add:
One thing that's been on my mind lately is; how we might go about creating extensible cli applications from an API standpoint. In games we're often so dynamic that it's pretty common to ship the executable which probably does most of the cli parsing and providing the API, but then have most of the cli arguments and options way down stream in multiple dynamic libraries (executable=engine, dlls=games). Including even potentially top-level "subcommands". For example, let's say you have something like a simplified git; with init, clone, etc. how might this work (well!) if each one of those sub-commands is a dll and the primary executable has absolutely no idea about "clone" not to mention the arguments to clone ? Similarly we might even go further and do something like
engine.exe --project "myproject", where the project flag would load myproject; but NOT myotherproject. How might our help output look in the various cases ofengine.exe --helpvsengine.exe --help --project "myproject"etc? What might our error handling look like if just the project name is wrong but you have a bunch of flags that project defined/uses? What do definition collisions look like when you're gathering all these subcommands / flags from different sources?Note that obviously this is a pretty niche usecase, and not every command line API needs to support this, but once we're in multi-million LOC codebases with 100s of developers some conventions have to change. Historically game engines have had very poor CLI support as a result of this. In my own engine there's currently no way to list all options or arguments for example, but it's absolutely been on my mind as something to fix. It's entirely possible that maybe there's entirely different expectations for how this works for a true CLI application, a GUI application, and maybe even a service / daemon type application too. Ideally I'd like a similar API for them though.
dutc | 5 hours ago
My understanding is that games programming is still very Windows-focused. If that is the case, isn't the general expectation that the command-line environment will be very weak (and, in any case, won't be very POSIX/Linux-like)?
Presumably, you're still able to coördinate in some fashion—via an API or some kind of convention.
Of course, while we may readily assume historical technologies would be insufficient for automatically deriving a command-line grammar from
--helpoutput, I suspect this might not be beyond what a language model can accomplish.I suspect the unaffiliated “subcommand” use-case is actually quite common in practice. Consider that a typical shell script will invoke multiple subordinate commands to accomplish its task. It will expose flags on some of these commands but not on others. Inevitably, there will be some modality in a subordinate command that is necessary but cannot be exposed.
If the shell script cannot be rewritten, the user will either try to trigger that modality via alternate means (an environmental variable or configuration file entry,) or they will try to intercept the invocation of the command and perform some kind of rewriting (e.g., write a stub entrypoint, define a shell function with the same name, and source the original script, or write a shell script with same name somewhere eariler in
$PATH.)Some overeager script authors will actually expose these parameters in some fashion, like a command-line
--subcommand-flagsargument or an environmental variable SUBCOMMAND_FLAGS.rsyncactually has two flags for this:--rshor-eand--rsync-path(the latter of which can intercept thersyncinvocation on the remote side.) I have an unpublished blog post where I use this to implement remote⇋remotersynctransfers (intermediated through the local machine, like inscp.)If the subordinate units are actually DLLs rather than separate executables, that they are written by authors willing to follow a convention, and that they are very similar in their development (e.g., all written in the same programming language targeting the same platform,) I suspect that these simplifications will afford you a lot of freedom. If the command-line parsing is guaranteed to always nicely fit into a declarative model, then you just have to expose that in some fashion. If the command-line parsing cannot fit into a declarative model, then you would have to expose the parser function.
Without these simplifications, I think the problem becomes very hard. In fact, my interest in Zig for command-line parsing is a result of realising that the N×M problem of commands × shell completions is largely unavoidable. Every command needs to have a shell completion written for every shell. If there were a single, uniform declarative model that would work for all commands, then we would ask authors to write these, and have shells parse them to create completions; but, as I hope to have demonstrated previously, I don't think this is actually feasible.
Instead, I have been investigating whether completion can be internalised into a very fast command-line parser itself. If the parser is fast enough, then the completion can be a thin(ner) wrapper against invoking the program itself. (It is, after all, the foremost authority on its own completion.)
Furthermore, if the parser is written against a non-allocating iterator that can be safely copied, then we can serialise this state between completions to support incremental parsing. We may even be able to do parsing, completion, and auto-canonicalisation within the same body of code.
e.g.,
If you couple this with some sort of Zig plugin per shell that can rapidly parse some highly structured completion return value, then you could probably get very good quality completion across all shells with the vast majority of the code sitting in the command itself.
Actually, if you squint, this is not that different than the problem you describe, is it?