I dislike interfaces that try too hard to guess what I want. Each parameter that goes into the guess is a bit of context that I need to remember if I want to predict the behavior of the UI.
I'd rather understand my interface than get used to it.
The .NET team had a really nice thing for this. I’m not sure if it was ever released, but they had a CLI parser that came with a grammar description and embedded the grammar in a special section of the executable. PowerShell would then pull that out and so could parse your command line exactly as the program would, including understanding dependent arguments.
OpenStep (and Cocoa) have an interesting model where settings are provided as key-value pairs (where the values can be strings, numbers, dates, binary blobs, dictionaries, or arrays). These are stored as user defaults in domains, where each layer could override settings from the layer below. This let you have app-bundled defaults, system-wide settings, and per-user settings all exposed via the same interface. The final layer was the arguments domain. When you start an OpenStep / Cocoa app, it passes argc and argv to NSApplicationMain, which manages the run loop. This also populates the arguments domain in user defaults with things from the command line. A lot of Apple’s first-party CLI tools work this way: their command-line flags are handled via the CoreFoundation user defaults later and you can set default values for them with the defaults tool (which inspects and modifies the persistent user defaults domains). This doesn’t provide any introspection to the shell though and it would be great if they included a schema in the tool’s plist.
TOPS-20 actually went a step further (IIRC*): the command’s process was launched as soon as you typed its name, and then it used system calls to communicate with the shell (EXEC) while you typed the command line. All subsequent shells I know of parsed the command line themselves (with varying levels of sophistication) and then launched the command.
So the more fundamental way we’re warped by Unix is the assumption that command arguments should be there at all when you launch the process! TOPS-20 didn’t have argc/argv.
* It may be more correct to say that EXEC handed the terminal over to the command, which then used the COMND library to process the user’s typing — I can’t quite remember.
For the record, even DEC's own (Open)VMS decided to offer nearly no completion on DCL. But yes, seeing TOPS-20 in a PDP-10 emulator was mind-blowing to me — I couldn't believe how backwards even many later systems were by comparison.
There’s really no reason we couldn’t do this today. Just have the shell pass a socket to the command and do some IPC. It’s trendy to build three different completion script generators into the command anyway!
It's kinda sad that we're talking about reintroducing a feature from the late 70s and performance is a concern. If you can't load the completion routine that fast, maybe just skip it.
The second bullet was why I pointed out we're already writing commands to support three completion standards. And those are even (mostly) redundant! (Thanks to everyone who's added fish support, btw.)
Are there Unix shells that approximate TOPS-20’s completion behavior? I can only think of transient which, yes, isn’t a shell nor an extension for one, but as it’s applied by Magit, it wraps git the way an elaborate completion script would in a shell, and it produces a kind of completion behavior that understands dependent arguments.
The completion features in all three of the most popular shells (bash, zsh, and fish) are comparable to those in TOPS-20. The difference is they do this with completion scripts that are independent of the actual command and have to be installed separately.
As I mentioned, some commands generate their own completion scripts, e.g. source <(mycommand completion bash) would install mycommand's completions into a bash session, and some argument parsing libraries include that functionality (e.g., Swift ArgumentParser). So in that sense completion definition is integrated into the command, but it's pre-baked, not as interactive as the TOPS-20 method.
Magit doesn't really just wrap git commands; it's a complex elisp application that uses git commands in its implementation. The transient elisp library is a way of giving sophisticated arguments to elisp code, not building Unix command lines.
You could combine these approaches and have a TUI tool that generated a magit-ish UI for the completion script. Macintosh Programmers Workshop had a thing called commando that read argument definitions from a resource in the tool binary and would automatically build a GUI dialog with checkboxes, dropdowns, etc.
I gather that some of the interactive usability features in csh/tcsh were copied from TOPS 10 or 20, which is why tcsh was the popular unix shell through most of the 1980s – until the rise of bash in the 1990s. previously
A lot of Apple’s first-party CLI tools work this way: their command-line flags are handled via the CoreFoundation user defaults later and you can set default values for them with the defaults tool (which inspects and modifies the persistent user defaults domains).
Can you list a few examples? This would be really helpful for something I'm looking into.
Most of the GUI administrative tools are Something Utility. There’s usually a somethingutil that accompanies it. For example diskutil, hdiutil, and so on.
Yeah, I think this is a false dichotomy..I want to understand how it works but I still want it to be easy to use and not give me completions that won't work.
ZSH's completion system does this automatically for not repeating options (see the _arguments function and the explanation of *optspec), and it has a mechanism to define mutually exclusive arguments (see the paragraph starting with "Each of the forms above").
But I also think that encoding these very specific behaviors in a completion script is a lot of work tied to the definition of the CLI interface with questionable benefit. If the CLI interface changes regularly, it's going to be a lot of work to keep the actual behavior in line with the completion behavior.
The zsh kubectl completion goes the other way, and just calls kubectl __complete with the provided arguments to get the completions. This keeps the implementation in sync with the completion, but it's very slow (at least for me).
I'm quite happy with fish's implementation. It seems to mostly do when I just type in a few of the words and it matches them based on what I typed in the past as I type with a tab option to complete.
Fish completion scripts use this. Both at the shell level (don’t suggest an option that can only appear once again) and at the completion scripts use logic level (only suggest option X if option Y has been entered or only if option Z hasn’t).
I’ve always found things trying to autocomplete git options to be annoying- it makes tab completion obnoxiously slow when you’re simply trying to tab complete a directory name, but also I’ve never found the need to tab complete branch names but maybe that’s a byproduct of how slow it is?
I wrote a Javascript package that uses parser combinators to parse command-line input. It also supports interacting with the web page during parsing, e.g. so items referenced in the command can be highlighted as it is typed.
orib | 22 hours ago
I dislike interfaces that try too hard to guess what I want. Each parameter that goes into the guess is a bit of context that I need to remember if I want to predict the behavior of the UI.
I'd rather understand my interface than get used to it.
david_chisnall | 19 hours ago
The .NET team had a really nice thing for this. I’m not sure if it was ever released, but they had a CLI parser that came with a grammar description and embedded the grammar in a special section of the executable. PowerShell would then pull that out and so could parse your command line exactly as the program would, including understanding dependent arguments.
OpenStep (and Cocoa) have an interesting model where settings are provided as key-value pairs (where the values can be strings, numbers, dates, binary blobs, dictionaries, or arrays). These are stored as user defaults in domains, where each layer could override settings from the layer below. This let you have app-bundled defaults, system-wide settings, and per-user settings all exposed via the same interface. The final layer was the arguments domain. When you start an OpenStep / Cocoa app, it passes argc and argv to NSApplicationMain, which manages the run loop. This also populates the arguments domain in user defaults with things from the command line. A lot of Apple’s first-party CLI tools work this way: their command-line flags are handled via the CoreFoundation user defaults later and you can set default values for them with the
defaultstool (which inspects and modifies the persistent user defaults domains). This doesn’t provide any introspection to the shell though and it would be great if they included a schema in the tool’s plist.jmc | 10 hours ago
Not quite as cool, but Abseil's flags module exposes an
--helpxmlflag that spits out structured data you can parse: https://abseil.io/docs/python/guides/flags#special-flagsaag | 11 hours ago
I believe that it was the DEC TOPS-20 operating system that did this in the 1970s. Unix has set our expectations so low.
https://en.wikipedia.org/wiki/TOPS-20
wrs | 10 hours ago
TOPS-20 actually went a step further (IIRC*): the command’s process was launched as soon as you typed its name, and then it used system calls to communicate with the shell (EXEC) while you typed the command line. All subsequent shells I know of parsed the command line themselves (with varying levels of sophistication) and then launched the command.
So the more fundamental way we’re warped by Unix is the assumption that command arguments should be there at all when you launch the process! TOPS-20 didn’t have argc/argv.
* It may be more correct to say that EXEC handed the terminal over to the command, which then used the COMND library to process the user’s typing — I can’t quite remember.
Edit: my footnote is correct, and here’s a tutorial with a good explanation of how it works.
dmbaturin | 7 hours ago
For the record, even DEC's own (Open)VMS decided to offer nearly no completion on DCL. But yes, seeing TOPS-20 in a PDP-10 emulator was mind-blowing to me — I couldn't believe how backwards even many later systems were by comparison.
wrs | 6 hours ago
There’s really no reason we couldn’t do this today. Just have the shell pass a socket to the command and do some IPC. It’s trendy to build three different completion script generators into the command anyway!
Uh-oh, did I just nerd snipe myself?
dsr | 6 hours ago
Two things come to mind:
oops, that wasn't the command I wanted to run. And it takes three seconds to start up the one I typed, so...
great, now we have to rewrite every command to a new standard
wrs | 5 hours ago
It's kinda sad that we're talking about reintroducing a feature from the late 70s and performance is a concern. If you can't load the completion routine that fast, maybe just skip it.
The second bullet was why I pointed out we're already writing commands to support three completion standards. And those are even (mostly) redundant! (Thanks to everyone who's added
fishsupport, btw.)gnyeki | 5 hours ago
Is Emacs’s transient UI the closest we’ve come to this on Unix?
wrs | 4 hours ago
I wouldn't have described
transientas being "on Unix", but rather "in Emacs". Doesn't it only work with elisp code, not commands in general?gnyeki | 3 hours ago
I guess we’re having different things in mind.
Are there Unix shells that approximate TOPS-20’s completion behavior? I can only think of transient which, yes, isn’t a shell nor an extension for one, but as it’s applied by Magit, it wraps git the way an elaborate completion script would in a shell, and it produces a kind of completion behavior that understands dependent arguments.
wrs | 3 hours ago
The completion features in all three of the most popular shells (bash, zsh, and fish) are comparable to those in TOPS-20. The difference is they do this with completion scripts that are independent of the actual command and have to be installed separately.
As I mentioned, some commands generate their own completion scripts, e.g.
source <(mycommand completion bash)would installmycommand's completions into a bash session, and some argument parsing libraries include that functionality (e.g., SwiftArgumentParser). So in that sense completion definition is integrated into the command, but it's pre-baked, not as interactive as the TOPS-20 method.Magit doesn't really just wrap git commands; it's a complex elisp application that uses git commands in its implementation. The
transientelisp library is a way of giving sophisticated arguments to elisp code, not building Unix command lines.You could combine these approaches and have a TUI tool that generated a magit-ish UI for the completion script. Macintosh Programmers Workshop had a thing called
commandothat read argument definitions from a resource in the tool binary and would automatically build a GUI dialog with checkboxes, dropdowns, etc.fanf | 8 hours ago
I gather that some of the interactive usability features in csh/tcsh were copied from TOPS 10 or 20, which is why tcsh was the popular unix shell through most of the 1980s – until the rise of bash in the 1990s. previously
habibalamin | 8 hours ago
Can you list a few examples? This would be really helpful for something I'm looking into.
david_chisnall | 7 hours ago
Most of the GUI administrative tools are Something Utility. There’s usually a somethingutil that accompanies it. For example diskutil, hdiutil, and so on.
habibalamin | 6 hours ago
Ah, I should have said, I was hoping for something CLI-only. This doesn't really change my current findings.
icholy | 11 hours ago
It's not guessing what you want, it's showing you what's allowed given your input so far.
apromixately | 4 hours ago
Yeah, I think this is a false dichotomy..I want to understand how it works but I still want it to be easy to use and not give me completions that won't work.
jeeger | 18 hours ago
ZSH's completion system does this automatically for not repeating options (see the _arguments function and the explanation of
*optspec), and it has a mechanism to define mutually exclusive arguments (see the paragraph starting with "Each of the forms above"). But I also think that encoding these very specific behaviors in a completion script is a lot of work tied to the definition of the CLI interface with questionable benefit. If the CLI interface changes regularly, it's going to be a lot of work to keep the actual behavior in line with the completion behavior.The zsh kubectl completion goes the other way, and just calls
kubectl __completewith the provided arguments to get the completions. This keeps the implementation in sync with the completion, but it's very slow (at least for me).dmytrish | 15 hours ago
There is an edge case where repetition of the same option with different values is a valid way to pass a list of values.
Edit: that's actually about dependencies of options, for which it is a good suggestion.
invlpg | 13 hours ago
Is it even an edge case? The '-Xlinker' option of gcc and clang, and the
-Hof curl immediately sprung to mind when I read the title.Boojum | 8 hours ago
I've also seen some CLIs with things like
-v -v -vfor maximum verbosity.symgryph | 8 hours ago
I'm quite happy with fish's implementation. It seems to mostly do when I just type in a few of the words and it matches them based on what I typed in the past as I type with a tab option to complete.
mayli | 7 hours ago
Same here, good enough without making it over complicated
mqudsi | 5 hours ago
Fish completion scripts use this. Both at the shell level (don’t suggest an option that can only appear once again) and at the completion scripts use logic level (only suggest option X if option Y has been entered or only if option Z hasn’t).
Fun sidebar: the git completion script is probably our most complex.
olliej | 3 hours ago
I’ve always found things trying to autocomplete git options to be annoying- it makes tab completion obnoxiously slow when you’re simply trying to tab complete a directory name, but also I’ve never found the need to tab complete branch names but maybe that’s a byproduct of how slow it is?
aag | 11 hours ago
I wrote a Javascript package that uses parser combinators to parse command-line input. It also supports interacting with the web page during parsing, e.g. so items referenced in the command can be highlighted as it is typed.
https://github.com/arthurgleckler/command.js
coxley | 13 hours ago
I have a library that makes contextual, self-completion easy to manage and test: https://github.com/coxley/complete
I’ve used it for:
Like the author, I wish it was more common. It’s such a subtlety, but makes a big difference for production tooling you use a lot at work.