RFC: Clap, from scratch re-write and redesign discussion

[WildCryptoFox]

[Drafting; @kbknapp @sru please feel free to update this OP as you see fit. Please "create a lock" by posting a comment saying that you're editing - and then delete when done]

From: #259 (comment)

• List all current clap features and capabilities in the OP (EDIT: This is already mostly done, but not quite complete. There are a few features missing, or some that should be broken down into further sub-sections. EDIT 2: Also, I'm speaking more about separating current features, from proposed features, from implementation details...currently it's all mixed into one list)
• Determine which of those capabilities will be changed (i.e. implemented but in a slightly different manner from the clap consumer standpoint...breaking changes)
• Implement all current capabilities in clap 2.x tracked by individual 2.x issues with this issue being the overarching 2.x discussion forum and overall tracking issue
• Determine which new features will be present in 2.x (i.e. custom Matches struct allowing things like matches.some_arg fields instead of the current matches.value_of("some_arg"))
• Implement all new features in individual 2.x tracking issues, again with this issue being the overarching 2.x discussion forum
• Test / Bench the 2.x branch to ensure we didn't lose ground
• Once all features / bugs / benches have been worked from the 2.x branch we'll ensure docs are good to go and release

Current (ripped from the README) feature

• Auto-generated Help, Version, and Usage information
• Flags
  • Short version (i.e. -f)
  • Long version (i.e. --flag)
  • Combining short versions (i.e. -fBgoZ is the same as -f -B -g -o -Z)
  • Multiple occurrences (i.e. -vvv or -v -v -v)
• Positional Arguments (i.e. those which are based off an index from the program name)
  • Multiple values (i.e. myprog <file>...)
  • Supports the unix -- meaning, only positional arguments follow
  • Optionally sets value parameters
    • Minimum values
    • Maximum values
    • Exact number of values
• Option Arguments (i.e. those that take values as options)
  • Short version (i.e. -o value)
  • Long version
    • --option value
    • --option=value
  • Multiple values
  • Named values
  • Value parameters
    • Minimum values
    • Maximum values
    • Exact number of values
• Sub-Commands (i.e. git add <file> where add is a sub-command of git)
  • Their own sub-arguments, and sub-sub-commands independent of the parent
  • Get their own auto-generated Help, Version, and Usage independent of parent
• Requirement Rules
  • Required by default
  • Required only if certain arguments are present
  • Can require other arguments to be present
• Exclusion/Confliction Rules
  • Can be disallowed when certain arguments are present
  • Can disallow use of other arguments when present
• Groups
  • Fully compatible with other relational rules (requirements and exclusions) which allows things like requiring the use of a group, or denying the use of a group conditionally
• Specific Value Sets
  • For positional
  • For option
• Default Values: Although not specifically provided by clap you can achieve this exact functionality from Rust's Option<&str>.unwrap_or("some default") method (or Result<T,String>.unwrap_or(T) when using typed values)
• Automatic Version from Cargo.toml
• Typed Values
• Suggestions
• Colorized (Red) Errors (Non Windows OS only)
• Global Arguments
• Custom Validations
• POSIX Compatible Conflicts
• Support for building CLIs from YAML

Current Implementation Details

(Something about the current design that needs attention)

Proposed

• Flags
  • Short version (i.e. -f)
  • Long version (i.e. --flag)
  • Combining short versions (i.e. -fBgoZ is the same as -f -B -g -o -Z)
  • Multiple occurrences (i.e. -vvv or -v -v -v)
• Positional Arguments (i.e. those which are based off an index from the program name)
  • Multiple values (i.e. myprog <file>...)
  • Supports the unix -- meaning, only positional arguments follow
  • Optionally sets value parameters
    • Minimum values (needs post parser validation test)
    • Maximum values
    • Exact number of values
• Option Arguments (i.e. those that take values as options)
  • Short version (i.e. -o value)
  • Long version
    • --option value
    • --option=value
  • Multiple values
    • -o value -o other_value abbreviated to -o value other_value
      • Instead abbreviated to -oo value other_value. The one instance of -o is too easily problematic.
    • --option=val1,val2,val3
  • Named values
  • Value parameters
    • Minimum values (needs post parser validation test)
    • Maximum values
    • Exact number of values
• Specific Value Sets (possible values?)
  • For positional
  • For option
• Sub-Commands (i.e. git add <file> where add is a sub-command of git)
  • Their own sub-arguments, and sub-sub-commands independent of the parent
  • Get their own auto-generated Help, Version, and Usage independent of parent
• Requirement Rules
  • Required by default Required through consumer decision. No default. Macro uses [] for optional and <> for required.
  • Required only if certain arguments are present
  • Can require other arguments to be present
• Exclusion/Confliction Rules
  • Can be disallowed when certain arguments are present
  • Can disallow use of other arguments when present
• Groups
  • Fully compatible with other relational rules (requirements and exclusions) which allows things like requiring the use of a group, or denying the use of a group conditionally
• Global Arguments
• Default Values
• Automatic Version from Cargo.toml
• Suggestions
• Custom Validations
  • Partial
• Auto-generated Help, Version, and Usage information
• Support for building CLIs from YAML
• Typed Values
• POSIX Compatible Conflicts
• Colorized (Red) Errors (Non Windows OS only [why not, it has colors too?])

Proposed Implementation Details

RFCs

Naming change
Settings

[WildCryptoFox]

impl ArgRules; fn matcher should really be impl ArgMatcher; fn with_rules

[WildCryptoFox]

ArgMatcher should be renamed to ArgMatches and possibly an enum respecting the ArgType and used as a tuple aside the rules (ArgRules, ArgMatches); dropping this down to the matches when returning the final HashMap for further handling.

enum ArgMatches {
    Flag(u8), // occurrences of the flag
    Argument(Vec<CowStr>), // arguments provided
        // doesn't need 'occurrences' as we have vec.len() already
        // assert!(vec.len() % self.rules.values_collected == 0)
//    Positional(Vec<CowStr>), // Just use Argument at this level
    // SubCommand?
}

[sru]

I think there is over usage of CowStr. If we know that we need to format or own the str, then we should use String and vice versa for &str. Otherwise, CowStr is good idea as we might allocate more String when the user provided String.

We can easily provide macro for impl From<ArgBuilder> for MyApp.

I am not sure about -oo foo foo expanding to -o foo -o foo because it may be confusing for the user (of this library and of the program that uses this library). Maybe allow comma separated? (--option=foo,foo)

I think subcommands require more special treatment than "just a positional argument that eats the rest of the iterator" (what if two subcommands are specified?)

[WildCryptoFox]

@sru I find the current -o foo foo as confusing itself. And yes, comma separated does look nice.

• -oo foo foo is common in other places. One notable source being IRC channel/user modes.
• -o foo foo is confusing and I've only seen in clap
• -ofoo, compilers? -lsomelib
• What should be the expected result of -abc 1 2 3 where a takes 1 argument and c takes 2? I'd expect -a 1 -b -c 2 3. In any case, this is shorthand and is the most flexible option such far, also easier to implement.

---

As for two subcommands. Say like cmd --foo subcmd --bar subsubcmd --baz ? Nothing wrong with this.

[WildCryptoFox]

Updated gist, slightly less Cow

I might solve the 'short' vs 'long' issue with a map of shorts/longs to u8 indexes of a vector containing the matchers.

[sru]

@james-darkfox Yes, -o foo foo is quite confusing too. Should cba 1 2 3 be -c 1 2 -b -a then?

[WildCryptoFox]

Yes. For implementing this, it is simply for each short we see. Call the handle function (lending it the iterator); then it may grab as many arguments as it needs. See the commented out code for the general gist. (Commented out because verbose may be the key, but verbose as the long "works" but v is not verbose; I'll likely solve this with a map for long/short to vector index)

[WildCryptoFox]

Noticed two typos in your question. -cba 1 2 3 should be -c 1 2 -b -a 3. Notably the -cba vs cba and -a takes 3

[sru]

@james-darkfox Ah yes, that was what I meant.

[WildCryptoFox]

@sru What is your preference for the handling of shorts and their arguments?

[WildCryptoFox]

Updated gist

@sru The last test -babcb 1 2 3 results in Foo { a: vec![1], b: 3, c: vec![2, 3] } (Not yet doing the struct)

[WildCryptoFox]

I need to flatten a few things (easier split for quick code): I've flattened a number of things down.

• enum Arg into struct Arg
• ArgMeta and Rules are both used in the builder, and matcher. Neither are for direct consumer usage. Might as well be flattened too.
• max_occurrences None is equal to Some(1), should drop the Option and default to 1
• Currently not using Cow at all. So much for that optimization. 😉

Updated checklist, and gist

[WildCryptoFox]

• Basic positional arguments
• Required arguments

Updated gist

[WildCryptoFox]

Hacky

https://gist.github.com/james-darkfox/c318258835114f3be4c9/revisions#diff-096d8044f81d02e61e0c93b463725feeR72

[WildCryptoFox]

• Possible values. I think this should just be a validation function?
• Overrides... Practical? Even re-setting arguments is fairly useless (unless you have lazy shell aliases calling each other) ls --color=auto --color=never, uses never.
• ArgRule will need to also have description
• ArgCollection will likely be renamed to App
• Should subcommands just be RDT (Recursive Data Types), paired aside normal arguments?
  • If so, they must be the last positional argument in the Vec<ArgRule>

type App(AppMeta, Vec<Foo>);
enum Foo {
    Arg(ArgRule),
    SubCommand(App),
}

[sru]

need @kbknapp...

[WildCryptoFox]

@sru Chained commands; I was thinking much like make, and it wouldn't be difficult for any consumer to adopt that structure by just collecting multiple positional values.

[sru]

@james-darkfox Yes, I was also thinking of make as a prime example. Since it wouldn't be too difficult for any consumer to adopt the structure by just collecting multiple positional values (and check the values with validate), I am unsure if there are enough use cases for this particular structure to actually include in the library. (Is it actually useful for majority of people vs. is it just a code bloat)

[WildCryptoFox]

@sru :)

[kbknapp]

@james-darkfox Are you talking about auto-completion as in the shell auto-autocompletion? If that's the case, I've been thinking about a way to easily take a clap App definition and generate a bash, or zsh auto-completion file for some time (on the back burner of my mind at least). I thought you were talking about someone typing $ myapp --ver and clap interpreting that as $ myapp --version and continuing execution.

---

Going forward to keep discussion and implementation focused I'm proposing we do the following:

• List all current clap features and capabilities in the OP (EDIT: This is already mostly done, but not quite complete. There are a few features missing, or some that should be broken down into further sub-sections. EDIT 2: Also, I'm speaking more about separating current features, from proposed features, from implementation details...currently it's all mixed into one list)
• Determine which of those capabilities will be changed (i.e. implemented but in a slightly different manner from the clap consumer standpoint...breaking changes)
• Implement all current capabilities in clap 2.x tracked by individual 2.x issues with this issue being the overarching 2.x discussion forum and overall tracking issue
• Determine which new features will be present in 2.x (i.e. custom Matches struct allowing things like matches.some_arg fields instead of the current matches.value_of("some_arg"))
• Implement all new features in individual 2.x tracking issues, again with this issue being the overarching 2.x discussion forum
• Test / Bench the 2.x branch to ensure we didn't lose ground
• Once all features / bugs / benches have been worked from the 2.x branch we'll ensure docs are good to go and release

I'm proposing we do these in that order because it appears to me we need some structure to this re-design and rebuild. EDIT 3: This means if box 1 isn't checked, we shouldn't be working on box 2. It's very easy to get side-tracked working on the latest and greatest new idea without implementing the "must haves" or doing so with an incomplete feature set in mind...which is how clap 1.x go to where it is today.

Most importantly if we don't have a full picture of the feature set 2.x will include we cannot make sound design decisions. Meaning certain choices are dictated by other features that must be present. An example of this is validators. And this is not picking on @sru , it's just a an example 😄 But validators as currently implemented in the 2.x branch will probably not work with global arguments (because of the need to clone args down through child subcommands). If we go at this re-design without first fully listing the features and capabilities of clap 2.x we have the potential of not making fully informed design decisions which we'll have to fix, adjust, duct-tape, band-aid fix throughout the process...which is exactly how clap 1.x became a less friendly mess (or at least one of the reasons 😜).

I just want to be clear that all current capability must be present in 2.x although some of it may be tweaked/modified/combined/etc.

---

I'm not afraid to scrap any 2.x designs we come up with if we determine it's no better than 1.x. I want 2.x to be rock solid and something we're all extremely proud of.

So let's not be afraid to break things and change things in 2.x. This is our chance to get things right, and set ourselves up for the future...but we need to do so correctly and not back step at all, unless a current feature is truly a mistake that should never have been 😄

See EDIT 1, 2, and 3 above

[WildCryptoFox]

@kbknapp I was talking about completion for --ver to --version. This is completion. While --verion -> --version would be correction. As for shell completions, one common method is to just provide a list separated by newlines '--help' '--version' '-h' '-V' ... Alternatively, more complicated shell completions could exist too.

As for the structure. Please free free to create the respective issue/threads. I also think we should have an IRC channel for private discussion and interactive design decisions. Dumping ideas and responding after often-hours via GitHub isn't quite so ideal. I've just created irc.mozilla.org 6697 +ssl #clap-rs.

As I've mentioned in one of @sru's PRs, I have a number of things that I need to do and so I won't be able to contribute as much as I'd like; I should be freed up in a couple of months.

2.0 does not need to be released anytime too soon

@kbknapp rock solid may not necessarily be correctly implemented. See git vs darcs and the suckless philosophy. In either case, the balance of science and engineering is what determines the result.

---

Clap1.x is pretty solid, covering many edge cases; while clap2 avoids the need for edge cases through a stricter design.

---

I mentioned git vs darcs because git is very rock solid, but not so well designed. While darcs is very well designed, and not yet as solid. I happen to be working on implementing the patch theory from darcs in rust for one of my projects.

[kbknapp]

@james-darkfox 👍 to everything you said, I agree very much. I'll start idling in the orc when I can too, that's a good idea.

[WildCryptoFox]

irc.mozilla.org 6697 +ssl #clap-rs

• @james-darkfox
• @sru
• @kbknapp
• @Vinatorul

[kbknapp]

I think yo-bot has some irc pinging ability as well, it's just not enabled right now.

[sru]

@kbknapp I am not sure how this will work out. Since we are virtually rewriting the library, it is difficult to separate breaking changes from non-breaking changes. I agree that clap2 should have almost all the functionality that current clap has and that we should be more wary of what features will be in the library. However, I think that categorizing by feature is not a good idea because as of right now,

I think it should be divided into three big categories: implementation, tests, and documentations. tests and documentations will be fine with just checklists,and we could divide implementations by... implementation.

[Vinatorul]

@james-darkfox wow, irc channel, just saw.
I will join when come home

[kbknapp]

@sru to be clear, I'm ok with losing some functionality for a clean implementation so long as that's not functionality we want anyways. There are certain parts of clap that I'd pull out and not re-implement in 2.x, but most of 1.x I do want to keep.

My main goal that I'm trying to avoid is pulling features out for a clean implementation, only to add them back in later putting us right back where we started or in the same boat as 1.x.

Once I get some time I'll sit down and look at the current feature set and decide what my personal priorities are and rank them similar to issues (i.e. "required", "want to have", "nice to have", "remove", etc.). This doesn't have to be the final list, just my own thoughts on the matter 😄

[nagisa]

What should be the expected result of -abc 1 2 3 where a takes 1 argument and c takes 2? I'd expect -a 1 -b -c 2 3. In any case, this is shorthand and is the most flexible option such far, also easier to implement.

This kind of flag usually fails to parse with most arg parsers. I’d probably expect the same because its ambiguous somewhat, not really intuitive, doesn’t match previous parsers etc.

Short flags should only take arguments if they’re last in a concatenation of arguments (in your case -ba 1 and -bc 1 2 both work, but not -ac 1 2 or -ca 1 or -ab … or -cb …)

[kbknapp]

@nagisa 👍 I agreee

[WildCryptoFox]

This shorthand was a side affect for the method that I was using to quickly parse shorts. It would take more effort to prevent the shorthand. There is nothing really wrong with the shorthand, and will likely not be the advertised method from developers of some project. They would split things up naturally.

[sru]

@james-darkfox Some users of clap library might think it as "unexpected behavior" though.

[WildCryptoFox]

@sru Maybe so, but if clearly explained there isn't anything wrong with it?

Speaking of behavior.. Anyone have any thoughts to either an environment variable, feature-gate, or simple function to dump the argument parsing result. Would be nice for troubleshooting parsing and clearly demonstrate how the configuration actually parses what the developer or users provide the program.

[kamalmarhubi]

@kbknapp

I've been thinking about a way to easily take a clap App definition and generate a bash, or zsh auto-completion file for some time (on the back burner of my mind at least).

Is this in the feature list? I just started using clap, and the structure seems very amenable to allowing completion. Though rather than generating an auto-completion file, I had been thinking of a special completion mode enabled by flag or env var. Completion mode would take a partial command line and offer a list of completions. The benefit is that the program can use runtime information to complete values for some args, eg drawing values out of a database.

Either way, I think it's a really worthwhile feature.

[kbknapp]

@kamalmarhubi

Yes it is....although I have created anew issue to track it since it's just been on the back burner. I've thought about doing like you said but what I'm leary of is including that functionality unconditionally in programs that don't need or care about it. Perhaps only compiling in that functionality only in debug mode? I need to think on this and am open to all suggestions :)

[sru]

• formats
  • short
    • flag -f
    • option -o val1
    • combining multiple shorts -fBgoZ == -f -B -g -o -Z
  • long --f
    • flag --flag
    • option --option value
    • option --option=value
  • multiple occurences
    • short -v -v -v
    • long --verbose --verbose --verbose
  • multiple values
    • short -o val1 val2
    • long --option val1 val2
    • long --option=val1,val2
  • unix -- meaning
• values
  • default values
  • multiple values
    • mininum (at least)
    • maximum (at most)
    • exact
  • possible values
  • typed values
  • validation
  • required / optional
• rules (the relationships between arguments and state of arguments)
  • depends
  • conflicts with
  • mutually exclusive
  • groups
• subcommands
  • global argument
  • external subcommands
• meta data
  • automatically generate help, version, and usage
    • get version from Cargo.toml
• builder
  • YAML parsing
  • macro
• settings
  • SubcommandsNegateReqs
  • SubcommandRequired
  • ArgRequiredElseHelp
  • GlobalVersion
  • VersionlessSubcommands
  • UnifiedHelpMessage
  • WaitOnError
  • SubcommandRequiredElseHelp
  • Hidden
  • TrailingVarArg
  • NoBinaryName
• misc
  • POSIX-ly currect compatible conflicts
  • suggestion
  • colors
• default behaviors
  • ????
  • very important

I made this list according to what I think clap can be divided into, from the list of features above.

First, I'd like to point out that "multiple occurence" and "multiple values" are separate things and should be emphasized so. Consider --option=val1,val2,val3 --option=val4,val5,val6. It has an occurence of 2 and values of 3 for each occurence. Of course, this separation loses meaning in positional arguments, but for options (and flags), it could be a great way to separate things.

I think separating the meta data (authors, versions, and any extra information) from the actual parsing and match of argument is good idea because meta data isn't needed to parse arguments.

If done right, I believe that clap can be modularized. For example, parser doing parser thing, meta data doing meta data thing and so on. This could give the consumer customization and control in the intermediate steps of clap. Of course, it would reduce the number of AppSettings as well.

It is also important that these customizations have to come with sane defaults such that clap can run out of the box without much configuration.

I typed this up at a whim, so I am not sure if deviates too much from what clap currently is. The details has to be ironed out before actually doing anything (as mentioned by @kbknapp), but I think this is a good start to look into which will be changed, same, or new.

It was rather lengthy, I apologize :P

[kbknapp]

@sru
No problem with the length :)

And you're very correct about multiple occurrences being different from multiple values. Your example is spot on. I also totally agree with your comments.

The one thing I'd add is, I'm all for (and encourage) modularizing clap internally but don't necessarily want to modularize it too much externally (beyond App, Arg, Setting, and SubCommand). Because while making things modular does help conceptualize it once you're familiar with how everything works and fits together, when you're new it just adds confusion. This was on of my main complaints with getopts (and really just libraries in general). I think something that resonates with people is that clap is very intuitive (or close to), where you can almost use it without documentation...just an auto-completer.

To be clear, i do want to modularize things internally as best we can, and that's something I've started to slowly refactor in. And I am good with modularizing certain things on the public facing API when it's intuitive enough, and provides a significant advantage in one way or another. My goal is to keep the external public facing API as simple as possible.

[sru]

@kbknapp
I see. So, the goal of 2.0 should be introduce breaking changes, but as little as possible, and overhaul the internal workings of clap.

I agree that exposing something to public API should provide significant advantage, but I am ambivalent about the intuitiveness. For example, let's say there are three possible ways to print help (this is a silly example):

1. By print_help() function
2. By print_help_with(some_string: &str) that takes custom string
3. By exposing the metadata through some means (say get_metadata())
   The third method may or may not be intuitive (some would say why would you need third method while others are delighted to get the metadata). However, it theoretically gives consumer total control of what should be done when printing help (again, this is a silly example). Consumers should default to first method or second method if they do not care much about help message, and clap should encourage them to use the first method. That's why I say default behaviors are very important.

On to breaking changes. The most obvious breaking changes would be:

• getting rid of deprecated methods
• reduce number of get_matches_* variants (by returning Result<...> as default)
• SubCommand -> Subcommand (not sure...)

These would not be complete list of breaking changes because we do not have implementation yet, but they are guaranteed breaking changes, yes?

On the sidenote, since, I am not able to go on IRC often and I don't have IRC bouncer, it is not an effective way to communicate for me. I noticed clap has gitter. Is it viable platform for communication?

[kamalmarhubi]

Have a path for localization and translation #380

[kbknapp]

@sru You can see in #372 what you listed is almost exactly what's happening (minus the SubCommand->Subcommand....which I agree with, but I'm trying to keep breaking changes minor-ish and it's more of a bike-sheddding thing anyways). But with the addition of decreasing the lifetime annotations in public types App, Arg, et. al.

So everyone is tracking, I'll be putting out a 2x release shortly with these minor breaking changes, but ergonomic wins and a chance to clean up some cruft.

This full re-design can still be discussed, and once some solid implementation happens it'll just be the next major release. I'm not scared of large version numbers 😄 Bottom line, continue with discussion and implementation, just cause 2x is coming out does not mean this thread is done! 😄

[kbknapp]

I think conversation has staled a bit on this. I'm preferring individual tracking issues related to re-designing components. So I'm closing this for now.

[kamalmarhubi]

👍 I feel like this thread served its purpose quite well :-)