REP: 110 Title: SCM-like rosinstall command structure Author: Thibault Kruse <[email protected]> Status: Draft Type: Informational Content-Type: text/x-rst Created: 11-Jun-2011 Post-History: 11-Jun-2011, 10-Aug-2011, 03-Mar-2012
- Abstract
- Specification
- Use-cases
- Analysis
- Rationale
- Motivation
- Compatibility
- Design-Alternatives
- Reference-implementation
- References
- Copyright
This REP presents an alternative to the rosinstall [1] command-line
interface to a command-based syntax. The command-based syntax is
inspired by SCM tools like svn, git, and hg as well as
several other ros commands like rosnode or rosservice, and is intended
to improve the usability of rosinstall as well as expose more
functionality. Common use cases of rosinstall are described and
compared with the proposed changes.
The name "rosinstall" does not fit into an SCM like tool as it contains a verb "install" in its name, which is confusing. The name rosws for rosworkspace is a suggestion based also on other activities around rosinstall. (see https://code.ros.org/lurker/message/20110711.160222.666ecfe4.en.html)
The command by default uses the context to determine which folder to use as target of the operations. Two location types are relevant, the workspace folder, and folders of entries.
The workspace folder must contain a .rosinstall file to be recognized as such. The rosws command will consider the first such folder in the chain of the current directory and its parents. The rosinstall command may also rely on the ROS_WORKSPACE environment variable to specify a workspace. If both approaches detect a suitable folder, the command will fail with an error message indicating the ambiguity. The user may specify a target workspace folder with the same option for all commands. This is the preferred approach for scripting with rosws.
To determine what entry a user wants to work on for certain commands,
the concept of a localname is used in what follows. The localname is
historically an absolute filepath or a relative path relative to the
workspace folder. Commands which accept a localname argument will
first attempt to resolve that argument against the list of known
localnames. If that fails, the commands will fall back to choosing a
workspace entry which has the same path in the filesystem, even if the
localname does not match. As an example, rosws diff . In a tree with
localname foo is equivalent to rosws diff foo. Finally, some
commands may fall back to interpret such an argument, if it is the
only one, and in the absence of a specifying option, as a workspace
folder if the command also accepts a workspace folder.
The command creates an initial environment for the other commands to work. This is analogous to scm syntax. Its main purpose is to explain the tool to the user, who understands that the location used here is going to be a special location further on.
The command also generates three initial files, setup.sh, setup.bash and
setup.zsh, which when sourced set up environment variables according to
the .rosinstall file in the same folder. Those files wll only be overwritten
by the rosws regenerate command.
--catkin,
--cmake-prefix-path
Options retain semantics as in rosinstall
This command processes several entries from a given URI source
The command merges new entries (command line or changed in
.rosinstall) with existing entries (if a .rosinstall is
present). The strategy is to replace existing entries rather than
adding whenever possible. This is the case when an existing and a new
entry have the same target path, the same SCM type, and a similar SCM
configuration (e.g. same URI root).
The strategy fails whenever there are two entries with the same target path but replacing is impossible with the strategy above.
When successfull, the command will append the new entries to the end
of the .rosinstall file.
URI can be a path to a rosinstall file, to a local directory
containing a .rosinstall.
The command will fail when there are entries with duplicate target paths.
The command calculates the change to the configuration. If the change merely consists of adding entries, this is executed. Else, the user is presented with an overview of planned changes, roughly similar to what the linux aptitude tool may present. The user can then interrupt the operation. An option exists to not ask for confirmation.
Options:
The merge options apply whenever two elements have equivalent localnames (pointing to the same directory)
| --merge-keep | Existing entries will remain unchanged |
| --merge-replace | |
| Existing entries will be overwritten in place | |
| --merge-kill-append | |
| Exiting entries will be removed, new ones appended to the end | |
| --confirm-all | Do not ask for confirmation |
-- A single dash can be used for piping stdout into the merge command
The command serves to add or modify a single entry of the config and is in particular aimed at users entering the entry information themselves rather than relying on some other source of information.
For adding an entry, the common case is to add an SCM entry. The syntax would then be
rosws set localname uri --scmtype --version=version
Options:
| --svn | Subversion SCM |
| --hg | Mercurial SCM |
| --git | git SCM |
| --bzr | bazaar SCM |
| --tar | tar as source (unofficial experimental feature) |
| --detached | no SCM |
| --version-new | the version specification to use |
- single dash means read from stdin (similar to tar command)
Option -version would go against the GNU standard of a global --version
option giving the version of a command line tool.
Examples:
$ rosws set robot_model --hg https://kforge.ros.org/robotmodel/robot_model $ rosws set robot_model --version robot_model-1.7.1 $ rosws set robot_model --detached
The command checks out or updates the given subtrees or all subtrees from their respective SCM provider and URL.
| --continue-on-error | |
| Continue despite checkout errors | |
| --delete-changed-uris | |
| Delete the local copy of a directory before changing uri. | |
| --abort-changed-uris | |
| Abort if changed uri detected | |
| --backup-changed-uris=BACKUP_CHANGED | |
| backup the local copy of a directory before changing uri to this directory. | |
Options retain their semantics from current rosinstall.
Examples:
$ rosws update -t ~/fuerte $ rosws update robot_model geometry
This serves to explain the current environment to the novice user, and
provide an overview of it. It is supposed to give an overview of the
environment and its state. The reference implementation lists the
interpreted ROS_ROOT and the ordered paths in the
ROS_PACKAGE_PATH. It also provides status information about the
local directories, such as whether they exist and whether their
current version matches the .rosinstall.
Options:
| --data-only | just the data, no headers or text |
| --no-pkg-path | does not print ROS_PACKAGE_PATH |
| --pkg-path-only | |
Shows only ROS_PACKAGE_PATH separated by :. Supercedes all other options. | |
| --only | Shows only given comma-separated attribute(s) separated by , |
| --yaml | prints rosinstall format yaml for backing up the current config |
These options are mainly intended to allow scripting based on rosinstall.
Examples:
$ rosws info -t ~/ros/fuerte $ rosws info robot_model $ rosws info --yaml $ rosws info --only=path,cur_uri,cur_revision robot_model geometry
removes entries with given localnames from the .rosinstall file
This command batch-calls SCM status. The output is the concatenation status commands for all versioned entries.
Options:
| --untracked | also shows untracked files |
This command batch-calls SCM diffs. The output is the concatenation of diff commands for all versioned entries.
Such a feature was suggested on ros-developers. The reference implementation shows how this could be done. However the solution is neither very robust nor flexible for SCM diff options.
There is room for ambiguity when the version specified in the .rosinstall does not match
the version in the directory.
A warning should be given on stderr at least.
This command regenerates the setup.sh, setup.bash and setup.zsh files that setup a shell environment. The rosinstall command overwrites these files on each invocation for historical reasons.
There is as of today no need to regenerate these files even if the .rosinstall file changed in acceptable use-cases. The only cases where the files would be changed is when configuring a workspace against a changed ROS distribution, or after an update of the rosws tool.
Options
--catkin --cmake-prefix-path
Options retain semantics as in rosinstall
The following are a list of use cases for rosinstall. U1-U6 are based on use cases described in the rosinstall documentation [1].
The new command syntax examples usually have several alternatives to use context or options to specify the workspace to operate on..
E.g. Developing on top of boxturtle shared install
rosinstall ~/workspace /opt/ros/boxturtle http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall
New command example:
rosws ~/workspace /opt/ros/boxturtle cd ~/workspace rosws merge http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall rosws update
E.g. Full source checkout
rosinstall ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall
New command example:
rosws ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall
E.g. Developing a stack against a full tree
rosinstall ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall my_stack.rosinstall
New command example:
rosws ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall cd ~/workspace rosws merge my_stack.rosinstall rosws update
E.g. Adding a rosinstall layout to an existing workspace
rosinstall ~/workspace http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall
New command example:
cd ~/workspace rosws merge http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall rosws update
rosinstall ~/workspace
New command example:
cd ~/workspace rosws update sometree
You can manually edit the file .rosinstall in a rosinstall managed workspace and then call rosinstall ~/workspace.
New command example:
cd ~/workspace rosws set stackname --version=newversion
e.g. taking the sources from trunk, using roslocate
roslocate info common_msgs | rosinstall add .
New command example:
cd ~/workspace roslocate info common_msgs | rosws merge -
You can manually edit the file .rosinstall in a rosinstall managed workspace and then call rosinstall ~/workspace
You can manually edit the file .rosinstall in a rosinstall managed workspace and then call rosinstall ~/workspace
New command example:
cd ~/workspace rosws remove foo
using diverse file exploration tools
rosinstall ~/workspace --generate-versioned-rosinstall=GENERATE_VERSIONED
New command example:
cd ~/workspace rosws info --yaml > GENERATE_VERSIONED
same as U4
Users should generall avoid doing this, as it can lead to failure in many ways.
Therefore not implementing the usecase in this REP, meaning user has
to manually modify .rosinstall and call rosinstall
rosinstall ~/workspace /path/to/other/workspace
New command example:
rosws init ~/workspace /path/to/other/workspace
The user has some ROS stack in directory foo, and want to move that
stack to directory bar. This means moving the files as well as
updating the .rosinstall and the setup.sh.
Will not be supported for now.
An expert user can change path, version, scm, entry order, etc. in his
rosinstall. In some cases he then just wants his setup.sh file to be
regenerated, in other cases he may want to update directories to
specific revisions.
rosinstall ~/workspace
New command example:
cd ~/workspace rosws update foostack rosws regenerate
When the user works on several interdependent stacks, he wants to create a unified diff of those stacks against their checked-out revision.
rosinstall ~/workspace --diff
New command example:
cd ~/workspace rosws diff
When the user works on several interdependent stacks, he wants to see what files are currently in a changed status in their respective SCM.
rosinstall ~/workspace --status
New command example:
cd ~/workspace rosws status
All rosinstall use-cases deal with a user trying to maintain a ROS environment on a local machine.
A ROS environment means the assignment of values to ROS_PACKAGE_PATH, PATH, PYTHONPATH
and ROS_MASTER_URI, as well as further changes to support ROS toolchains.
Given the description, a user of rosinstall needs to be aware of the following artifacts:
- any
.rosinstallfile in the current directory - the setup.* files in the current directory
- the state of local directories
- the information contained in other rosinstall files
As a user is free to manipulate entry in the configuration files, a major cause for errors are mistakes during such modifications.
As an automated tool, rosinstall currently only knows of this:
The user may have changed the rosinstall file, the user may have provided additional locations,
now a new .rosinstall file may have to be merged with additional locations, and new setup.* files
have to be generated.
An early attempt for this REP tried to provide different commands for
different use-cases, such as init, remove, add and
update. However the code for all those commands had in all cases
to remain largely similar to cope with the situation where a user had
manually changed his or her .rosinstall, or filesystem.
The list of use-cases thus only displays the range of distinctive usages of rosinstall, but does not provide help into splitting up the main functionality into many small commands.
| UC | init | merge | set | info | update | diff | status | remove |
|---|---|---|---|---|---|---|---|---|
| U1 | x | x | x | |||||
| U2 | x | |||||||
| U3 | x | x | x | |||||
| U4 | x | x | ||||||
| U5 | x | |||||||
| U6 | x | x | ||||||
| U7 | x | x | ||||||
| U8 | ||||||||
| U9 | x | |||||||
| U10 | ||||||||
| U11 | x | |||||||
| U12 | x | x | ||||||
| U14 | x | x | ||||||
| U15 | ||||||||
| U16 | x | |||||||
| U17 | x | |||||||
| U18 | x |
The rosinstall tool, while useful, has a command-line syntax that
can be difficult for beginners of ROS to become familiar with. Users
who are intimidated by rosinstall can resort to manually creating
source trees and maintaining ROS_PACKAGE_PATH by hand. Improving
the usability of the rosinstall tool can improve the ROS experience
for beginner users as well as introduce advanced users to more
powerful features.
The extended command structure allows to perform a broader range of operations concerning multiple SCMs, and mere investigative commands.
Breaking down functionality helps self-explaining of the tool.
Also this makes it easier to extend rosinstall with other functions, and to use rosintall in scripts (e.g. provide a curses/tk-based rosinstall ui).
Another problem are the --options. Those are only required in
specific use-cases, but in the current solution the user may provide
them in any case, and thus the user will have difficulties to tell
what his choices for options are, given his intention. Having distinct
commands allows a help function to quickly tell the user what options
are available for his specific intention.
The SCM like structure also allows code completion to provide command choices, which is a significant help for users when learning a new tool.
The vision of the source code is that it will one day in large parts be moved out of rosinstall to provide ROS-independent functionality for managing directories of mutliple SCMs. However the rosinstall functionality of generating setup files for ROS environments will remain important. Therefore a name change to rosws is suggested rather than a ROS-independent name.
The current rosinstall command structure allows a command such as:
rosinstall ~/workspace /opt/ros/diamondback ~/workspace2 ~/workspace/foo ~/download/temp.rosinstall
The intention of the user typing this cannot be inferred from the command, nor can what the tool will do be inferred from the syntax. The key idea is to have a syntax that allows the user to express a specific intention and to understand what the tool will do.
The commands info, diff, and status are self-explanatory.
The word install was dropped completely with the broad
functionality it offers in rosinstall. install makes users shy
away from usage fearing they might corrupt their debian installation
with a tool.
The set and merge commands are somewhat ambiguous, as the user
would need to use them for multiple purposes of adding entries or
modifying versions of entries. However the different use-cases are
algorithmically so similar that own commands for each seem superflous.
Rejected names include:
add, edit, modify, read, load, import, change
The rosinstall setup files remain unchanged.
Current rosinstall syntax is:
Usage: rosinstall PATH [<options> ...] [URI]...
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-n, --nobuild skip the build step for the ROS stack
--rosdep-yes Pass through --rosdep-yes to rosmake
--continue-on-error Continue despite checkout errors
--delete-changed-uris
Delete the local copy of a directory before changing
uri.
--abort-changed-uris Abort if changed uri detected
--backup-changed-uris=BACKUP_CHANGED
backup the local copy of a directory before changing
uri to this directory.
--generate-versioned-rosinstall=GENERATE_VERSIONED
generate a versioned rosintall file
--generate-versioned-rosinstall is a special case and is replaced
by rosinstall snapshot.
This will be fulfilled by rosws install.
The change in command syntax impacts rosinstall based tools such as the new proposal for a rosworkspace.
As originally developed by Tully Foote.
Commands are:
rosinstall PATH [options] [URIs]
with a 4 Phase model, Merging, Checking out, Generating setup,
bootstrapping ROS serving most use-cases and
--generate-versioned-rosinstall being an exceptional command
Similar to the commands provided in the REP, some of the functionality
could be provided by additional --options.
e.g.:
rosinstall --info rosinstall --check rosinstall --diff rosinstall --status
This is less useful for bash completion and focused help.
As initially suggested for this REP, all commands require the path on which to work as the first argument.
Commands are:
rosinstall <command> PATH [options] rosinstall <command> PATH [options] [URIs]
The SCM-like Designs are in line with the other ROS tools. However,
SCM tools usually drop the PATH argument and work in the current
directory tree.
Inferring the path from context can be done to varying degrees, like ignoring the ROS_WORKSPACE variable or only considering the current folder, not its ancestors as candidates.
Another alternative is to try first interpreting any first argument
as a workspace folder, however this may conflict with the syntax of
command set.
The install command suggested provides support for several use cases
which could be further split up into different commands, e.g. add
and apply. See Rejects.
Reference implementation is provided with the latest rosinstall release.
Ideas for rosinstall commands that were dropped for this REP, but may become interesting later:
In 2012 there was a second surge to realize the REP. In this it was decided to make a more significant step away from the rosinstall design. As a consequence, certain ideas were brought up and rejected:
- rosws install: replicates rosinstall
A command doing many things at once confuses and scares the occasional user Similarly commands set, merge nad update only do one thing, and do not allow doing more than one thing.
- update after set/merge
It could be convenient if the tool did or offered to checkout/update tree entries after a set/merge operation. This can be added as an option in the future if users want it.
- many uris for init and merge
These commands could technically take many uri arguments instead of maximally one. The decision to allow only one is to help users understand the syntax. The problem with many arguments we saw with rosinstall is that users could not read from the syntax whether the second argument had a special semantic or not, or whether the order of the arguments mattered.
- Maintain rosinstall assumptions about ROS
rosinstall was provided in support of early ROS releases, this REP is provided in the wake of ROS fuerte, which makes significant changes to how a ROS environment is set up. rosws can have a fresh start with fuerte only providing minimal support for ealier distros. In particular rosws will not insist on having a ROS stack, and will not build ros or ros_comm even after source checkouts, nor have an option to do so. A future development could be to even drop generation of the setup.* files.
- optional localname when adding using
set
With SCM providers, it is possible to checkout a new local copy from a url without giving a target folder name. The same could be achieved with the rosws set syntax. However this makes the syntax more confusing, and in many cases we want the user to be aware of names because the repository names often have dashes where the localname should have underscores.
- rosws snapthos
The command is just a variant of rosws info and can be a --option
- making
setandmergea single command
Both commands add/modify entries, so the case was made to unify them.
It was rejected after lengthy discussions for these reasons:
There is the fact that the 1st argument is mandatory for set, but not useful as localname for merge.
There are the different -- options for the different cases.
For the user, it should be clear whether his focus is some "remote" resource containing a set of many yaml entries, or on
a single entry to add/change. That split is also the split of commands.
The user won't be surprised by tab completion options --hg (or just hg as positional argument) or --version when
merging with rosinstall files. They would make no sense, but code completion cannot know that.
The localnames wont get in the way of tab completion user when wanting to point to a local xyz.rosinstall file.
It is true that merge is semantically like n calls to set, except it is an atomic operation (all succeed, or all fail).
In the far future, we might think of nested workspaces having state and version of their own and defining commit() and add()and update() actions.
Then we could have a different type for the "set" command, BUT this would be very different semantics from the merge command, and the merge command would not become obsolete.
This possibility should also indicate why merging the set and merge command above would hinder things in the future.
- making all entry arguments in
setmandatory
The set semantics could be cleaner if it followed the
principle that always a full entry (scmtype, uri, version)
must be provided. This is however unnecessarily tedious when
the user just wants to change one of the details of an
existing entry
- splitting
setintoaddandedit
The split has the advantage that more warnings can be provided in case the user tries to add an entry which overlaps another. However a confirmation step is sufficient. Also this would allow having mandatory positional parameters for scmtype and uri in the add case, and optional ones for the edit case, but two different syntax structures for very similar cases would be confusing as well.
- making the scmtype in
seta positional argument
positional arguments like hg, git and svn do not work well with an optional third positional argument URI.
- detach as a command of its own
set has option --detached to remove all scm information.
The case was made to make this a command of its own, in particular
for a fuse set/merge command. But it fits better with the set - merge
split.
removeas asetoption
removal is an instance of modifying, so it could be
unified with set as a --option. However the end effect
of a set operation is always that something exists
afterwards, so these semantics get muddied by remove
as an option.
- default merge option killAppend
When using merge or set, a decision is when an overlapping
entry exist, whether to replace it in place, or whether to remove
the existing and append the new one at the end. rosinstall has the
latter semantic because it conveniently implies that it is possible
to ensure an ordering of entries in the config file by ordering them
in the argument list.
For novice users however this behavior is confusing and also hard to describe. The default semantics for merge is therefore to replace the entry in place. An option exists for merge to change that behavior.
The following attempts were rejected in the first REP draft in 2011
- rosws regenerate: Generates setup files but does not attempt to update SCM based entries
- rosws init: first time command demands a ROS root to be specified. too similar to install
- rosws update: merges new entries and runs update on all scm based entries. too similar to install
- rosws remove: removes an entry from .rosinstall. Better be done by hand
In the second draft, those command were resurrected in particular in the absence of an all encompassing rosws install command.
Those commands were too similar to rosws install and their behavior to bulk update and regenerate all trees conflicted with the semantics sugested by their name.
- rosws overlay: looks up repository location based on roslocate and rosinstalls it
- rosws change-ros: allows to move from e.g. /opt/ros/diamondback to ~/ros or /opt/ros/cturtle
- rosws mv: Moves local checked out stack to different location on filesystem
- rosws reorder: changes the order of stacks in
ROS_PACKAGE_PATH - rosws change-version: changes the desired version of an SCM based local stack
- rosws check --repair: Attempts to (interactively) fix inconsistencies such as duplicate paths
Those commands make the initial version of the REP more complex to discuss and harder to implement, therefore they were left out.
| [1] | (1, 2) rosinstall (http://www.ros.org/wiki/rosinstall) |
Copyright (c) 2011 by Thibault Kruse. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v0.1 or later (the latest version is presently available at http://www.opencontent.org/openpub/).