When running a process it is sometimes necessary to pass environment
variables and to run the process in a particular directory. Add support
for these features.

Signed-off-by: Simon Glass <[email protected]>

It isn't generally useful to print out every command before executing
it. Now that the client shows failed commands fully, it seems better to
change this to show only when debugging.

Signed-off-by: Simon Glass <[email protected]>

The current subprocess helper assumes it is the only thing running. In
some cases (such as the Dediprog em100 tool) we want to start a process
and have it run for a while in the background.

Rather than invent another helper, modify the existing helper to support
this.

For existing use cases, no functional change is intended.

Signed-off-by: Simon Glass <[email protected]>

Some drivers need to be told what to do from the command line. This is
particularly true of complex strategy drivers, such as the one to write
U-Boot to board media, or send it over USB.

Provide a way to specify driver-specific variables with a string value,
so that they can control the strategy. Examples include whether to
bootstrap the device or just boot the existing image, whether to build a
new image or use an existing one and whether to write the image to boot
media or send it over USB.

With these variables a useful selection of shell scripts can provide a
highly productive development and test environment, without a huge
prolification of very similar drivers. This approach also avoids the
complex interactions that would be necessary between different drivers
aimed towards the same goal.

Signed-off-by: Simon Glass <[email protected]>

The EM100-pro is a SPI-flash emulator which supports a variety of chips.
It can be used to provide an image to a booting system.

Add a driver which allows images to be loaded ready for use. Provide a
trace facility for use when the board does not boot.

Signed-off-by: Simon Glass <[email protected]>

Some boards have a way to put them into a 'recovery' mode provided by
the SoC. Recovery is a signal asserted at power-on (or when the board is
reset). It causes the SoC to listen on USB.

Add a protocol for this feature.

Signed-off-by: Simon Glass <[email protected]>

The reset protocol only supports doing a reset. Sometimes we want to
assert reset while something is done (e.g. a power cycle or a recovery
signal), then de-assert it afterwards.

Add a new set_reset_enable() method for this.

The name is a little strange: one might expect set_enable() instead. But
with things like the PowerResetMixin we want to avoid ambiguity over
whether this refers to powering the device or resetting it.

Signed-off-by: Simon Glass <[email protected]>

When integrating Labgrid into other projects it is useful to be able to
query information known by a particular driver. For example, U-Boot
wants to know the board name of a particular target, so that it can
pass this to pytest.

Add a new driver method to provide this feature.

Series-changes: 6
- Show a more friendly error when the wrong query arguments are given

Signed-off-by: Simon Glass <[email protected]>

The role (or is it target?) is used in the environment and allows
multiple roles to use the same place. It is therefore often better to
specify the role to use rather than the place, since the place can be
determined from the role.

Add a new -r option to support this.

Signed-off-by: Simon Glass <[email protected]>

At least for x86 devices it is not necessary to provide these arguments.
Make them optional.

Signed-off-by: Simon Glass <[email protected]>

These arguments are currently set on activation. Where the driver is
being used by a strategy, it may need to set the arguments after that.
There is no need to have everything set in stone until the on() method
is called, so move it there.

Signed-off-by: Simon Glass <[email protected]>

The U-Boot strategy needs to provide the filename of the BIOS directly.
Add a method to support this.

Signed-off-by: Simon Glass <[email protected]>

Rather than provide a strange exception, complain when a strategy does
not use the QEMU driver properly, perhaps due to user settings.

Series-changes: 6
- Add new patch to report an error if QEMU is not turned on

Signed-off-by: Simon Glass <[email protected]>

It is common to use a digital output (such as a relay) to assert the
recovery signal. Add a driver to handle this.

Signed-off-by: Simon Glass <[email protected]>

Servo is a range of boards used by ChromiumOS devices to provide reset,
recovery and power control, as well as multiple serial consoles.

Add a resource and driver which provides these basic facilities.

Signed-off-by: Simon Glass <[email protected]>

Building firmware images is fairly complex these days, due to the wide
variety of binaries that need to be assembled together.

U-Boot provides two useful tools here:
- buildman can build U-Boot itself, selecting a suitable toolchain
- binman can assemble / sign images, pulling in required binaries

It is convenient to build these images within Labgrid, since each board
has a selection of necessary binaries, which can be provided as part
of the lab configuration.

This allows a fully functional firmware image to be produced for any
board as part of setting up a Labgrid configuration for that board.

A particularly useful case is for interactive use, where the current
source tree is built and booted on a selected board, resulting in a
console for that board (if everything goes well). The driver includes
some features which make this easy to implement.

Note that Labgrid itself does not build U-Boot. It just calls buildman
to make it happen.

Series-changes: 6
- Improve handling of dual-build targets so pytest works correctly

Signed-off-by: Simon Glass <[email protected]>

Each SoC has its own way of booting and this means that the firmware
layout for each SoC is typically different. On top of that, some SoCs
allow writing an image over USB. which uses a different format, e.g.
separate binaries for each stage.

Trying to make all this work is extremely challenging. Labgrid provides
a nice unification point, where the method appropriate for each SoC can
be included, with a small amount of configuration to control things.

Binary blobs can be provided to make the writer work correctly.

This driver is capable of writing SD-card images for several popular
devices as well as handling USB booting in some cases. It also supports
QEMU. It can be easily extended as necessary.

Series-changes: 6
- Support QEMU on ARM
- Support writing a VBE (Verified Boot for Embedded) TPL file

Signed-off-by: Simon Glass <[email protected]>

Some devices do not have power control. Add a driver which models this
situation, so that strategies don't need to have it as a special case.

Signed-off-by: Simon Glass <[email protected]>

Add quotes around the driver name, as expected.

Signed-off-by: Simon Glass <[email protected]>

Some SoCs need multiple USB-loading steps to function. For example,
Samsung devices require BL1, then SPL then U-Boot.

Add a new parameter to the protocol to support this.

Signed-off-by: Simon Glass <[email protected]>

Add a USB loader for sunxi, supporting SPL and U-Boot.

Signed-off-by: Simon Glass <[email protected]>

Add a USB loader for tegra, supporting BCT and U-Boot.

Signed-off-by: Simon Glass <[email protected]>

Add a USB loader for samsung, supporting BL1, SPL and U-Boot.

Signed-off-by: Simon Glass <[email protected]>

Sometimes it is useful to specify the block size when writing to media,
since some SoCs require different block sizes.

Add a new block_size argument which allows the value to be set. Support
the existing defaults based on other arguments.

IMO the behaviour of the automatic block-size-guessing logic is a
little odd. Perhaps we should just have a default of 512 in all cases?

Signed-off-by: Simon Glass <[email protected]>

Sometimes it is useful to specify the block count when writing to media,
e.g. when a limited area is available or an area must be filled.

Add a new count argument which allows the value to be set.

Signed-off-by: Simon Glass <[email protected]>

The current approach is slow, since each block must be individually
written to the disk before the next can start.

The more common approach is to do a manual 'sync' after writing. Adjust
write_image() to use this approach. It is several times faster on
non-trivial files, even with the need for the sync.

Signed-off-by: Simon Glass <[email protected]>

It isn't very useful to show the dd stats on every transfer. Make this
silent. It may be useful to tie this to the -v option but I am not sure
how best to do that.

Signed-off-by: Simon Glass <[email protected]>

Provide an overall time for the dd operation.

Signed-off-by: Simon Glass <[email protected]>

For generic labs with lots of boards it is helpful to use the same test
for all boards. Add a command-line option to specify which target to
use, so that we don't need to add a separate test class for every
single target.

Signed-off-by: Simon Glass <[email protected]>

Allow setting variables when running a test so that it is possible to
control complex strategy-drivers.

Signed-off-by: Simon Glass <[email protected]>

On some boards a relay is used to assert the recovery signal, which can
be used to force the DUT into SoC-recovery mode (boot ROM). Add support
for this.

Signed-off-by: Simon Glass <[email protected]>

This tool prints lots of information to the terminal so it is not
easy to see if anything went wrong. Use the logging subsystem to handle
most of this output.

Signed-off-by: Simon Glass <[email protected]>

This tool prints out the configuration information that it reads. This
is not generally useful, so put it behind a -v flag.

Signed-off-by: Simon Glass <[email protected]>

It isn't obvious when the exporter has finished setting things up and
the labe is ready for use. This seems to take quite a while - e.g. 15
seconds for 125 resources.

Add a message to provide this signal, including how many resources are
exported.

Signed-off-by: Simon Glass <[email protected]>

It is common to acquire a place and then use it to access the console.
Add a -a option to support this, to avoid running labgrid-client twice,
which takes longer and makes error handling harder.

If the place is acquired, it is released on exit.

Signed-off-by: Simon Glass <[email protected]>

Some messages are printed regardless of the logging level. Convert them
to logging statements so they can be suppressed.

Signed-off-by: Simon Glass <[email protected]>

When a command fails in labgrid-client it just shows the return code.
This makes it very hard to see what actually went wrong. Handle this
exception explicitly, showing the output of the tool which failed. This
allows easy debugging of the problem.

This is particularly important with things like USB loaders.

Signed-off-by: Simon Glass <[email protected]>

When a target may or may not be acquired, it is useful to be able to
release it without checking first. Add an option for this, so that it
is not necessary to check first.

Signed-off-by: Simon Glass <[email protected]>

Fix a small typo in a message.

Signed-off-by: Simon Glass <[email protected]>

Use logging rather than print so that this output can be controlled.

Signed-off-by: Simon Glass <[email protected]>

At present if a console device is used by a strategy driver it is then
removed by the client immediately afterwards. If the 'console' command
is used, this means that any console data is lost.

This is done so that the console device is release for microcom, so
move the logic in with the microcom logic. With the forthcoming internal
terminal, it will not be needed.

Signed-off-by: Simon Glass <[email protected]>

Put this code into a function to reduce the size of the _get_target()
function.

Signed-off-by: Simon Glass <[email protected]>

It is possible to set the initial state of a strategy, but not the final
state. In many cases we want to power the board off and otherwise clean
things up at the end. Provide a -e option to accomplish this.

Signed-off-by: Simon Glass <[email protected]>

The current _check_allowed() function is not very friendly in that it
produces an exception if something is wrong. Refactor it to add a new
function which just returns an error message in that case. This will
allow other modules to call it without creating an exception.

Signed-off-by: Simon Glass <[email protected]>

There is quite a lot of code here, so move the terminal function into
its own file. This will make it easier to extend it later.

Signed-off-by: Simon Glass <[email protected]>

At present Labgrid uses microcom as its console. This has some
limitations:

- console output is lost between when the board is reset and microcom
  connects
- txdelay cannot be handled in microcom, meaning that boards may fail
  to receive expected output
- the console may echo a few characters back to the caller in the time
  between when 'labgrid-client console' is executed and when microcom
  starts (which causes failures with U-Boot test system)

For many use cases, microcom is more than is needed, so provide a simple
internal terminal which resolved the above problems.

It is enabled by a '-i' option to the 'console' command, as well as an
environment variable, so that it can be adjustly without updating a lot
of scripts.

To exit, press Ctrl-] twice, quickly.

Series-changes: 4
- Get internal console working with qemu
- Show a prompt when starting, to indicate it is waiting for the board

Signed-off-by: Simon Glass <[email protected]>

Transferring a file around is an operation which happens internally
within Labgrid. It is confusing to show these transfers as they happen.
Use quiet mode with rsync to avoid unwanted output.

Signed-off-by: Simon Glass <[email protected]>

This class has a lot of members whose type and purpose is hard to figure
out. Add some documentation at the top.

Signed-off-by: Simon Glass <[email protected]>

This function lacks documentation. Tidy it up.

While we are here, reverse the order of the 'active' and 'activate'
arguments, to separate the arguments into those at the start which
provide the filter and the one at the end which requests an action.

Signed-off-by: Simon Glass <[email protected]>

Some strategies may want to use a driver if it exists, without failing
if it does not. Add an argument to get_driver() to support this.

Signed-off-by: Simon Glass <[email protected]>

U-Boot prints a banner when it starts up, which contains the version
string. Collect this so it can be used by the strategy driver as needed.

Signed-off-by: Simon Glass <[email protected]>

When something goes wrong it is useful to see what output was processed
as a clue to what happened. Collect everything up to the autoboot
prompt, since once it hits that things are normally working OK.

Signed-off-by: Simon Glass <[email protected]>

At present the 'start' state of U-Boot consumes all the initial U-Boot
output, including the SPL and proper headers and any version
information. This makes it impossible to run the U-Boot pytests, since
this output is expected from the board.

There are two ways to fix this:
- Show the output
- Print a message indicating that U-Boot has booted, so tests don't
  wait for the headers

Do both, the first to make tests work existing U-Boot work with tests,
the second to resolve the problem for future U-Boot versions, once the
U-Boot pytest patch is accepted:

   test: Allow signaling that U-Boot is ready

Signed-off-by: Simon Glass <[email protected]>

The U-Boot strategy only allows bringing up a U-Boot prompt. If for some
reason this fails (e.g. due to a bug), nothing is shown on the console.

For interactive use, it is helpful to see any output, even if it does
not ultimately result in reach the U-Boot prompt.

Add a new 'start' state, to start U-Boot. Use this from the 'uboot'
state.

Signed-off-by: Simon Glass <[email protected]>

It is useful to write a new U-Boot version to the board before starting
it up. This is particularly valuable when using labgrid interactively,
to test out / debug U-Boot on various boards.

We end up with several different paths through the strategy:

- just power on/reset the board
- bootstrap: write an existing U-Boot to the board, then power it on
- build and bootstrap: build a new U-Boot, write it to the board, then
  power it on

All of these are useful in an interactive context.

Implement this by calling the U-Boot provider to build U-Boot, then the
U-Boot writer to write it to the board. Finally, power on/reset the
board as normal.

This is controlled by two variables:

   do-bootstrap: 1 to bootstrap the board, 0 to leave it along
   do-build:     1 to build U-Boot, 0 to use the existing build

Signed-off-by: Simon Glass <[email protected]>

Some boards do not have boot media, or it cannot be accessed by the test
system.

In this case it is useful to send U-Boot to the board using the
USB-recovery mechanism. Add the logic for this, controlled by a new
'do-send' variable.

Even for boards which do have boot media, it can still be useful to use
the USB mechanism.

Add the logic for this, using a new 'do-send' variable to control it. If
'do-send' is 1, U-Boot is written over USB. If it is 0, then U-Boot is
written to the boot media.

Signed-off-by: Simon Glass <[email protected]>

When Labgrid fails to get to the U-Boot command line, show the output
that was received, so that the user can see what is going on.

Signed-off-by: Simon Glass <[email protected]>

Some boards boot from internal flash unless a 'recovery' button is
pressed. Add support for this, so that these boards can be used in a
lab.

When Labgrid fails to get to the U-Boot command line, show the output
that was received, so that the user can see what is going on.

Signed-off-by: Simon Glass <[email protected]>

U-Boot's pytest framework has lots of logic to detect the U-Boot banner
and deal with failures to do so.

Labgrid supports this in its U-Boot driver.

We don't need both and Labgrid's implementation is better. It has more
knowledge of what is going on, since it is in charge of bringing the
board to life. It can be configured to know how long the board takes to
start up.

Add a 'lab mode' where Labgrid does all the work and just announces when
things are ready. This is enabled by presence of the U_BOOT_SOURCE_DIR
environment variable, which is set by U-Boot's pytest system.

This greatly simplifies operation of lab testing and ensures that pytest
is not trying to guess what happened.

Signed-off-by: Simon Glass <[email protected]>

Provide some scripts which facilitate U-Boot testing and development
using Labgrid.

Series-changes: 6
- Fix reference to crossbar in _ub-bisect-try and ub-smoke
- Add -x option to specify the extra board directory
- Add comments to the example lg-env
- Define USE_LABGRID to tell u-boot-hooks to use labgrid hooks
- Add a few more comments to ub-pyt
- Show the build path with ub-pyt -v
- Add --allow-unmatched to ub-smoke to avoid QEMU error

Signed-off-by: Simon Glass <[email protected]>

Add an example lab, based on my one, so people can see how to configure
things for this integration.

Series-changes: 6
- Add a patch with an example lab

Signed-off-by: Simon Glass <[email protected]>

Cover the drivers, scripts, pytest and the Gitlab information.

Note: This implementation is the result of working through what is
needed in Labgrid to support U-Boot labs. I did initially file quite
a few issues[1] but there has not been a lot of traction and I got
feedback that I have overwhelmed people with too many. So I have
stopped filing issues on the things I hit along the way. I have carried
on with the implementation in the hope that this can be a better basis
for discussion.

Note that all of these patches are work-in-progress, but it is becoming
more stable.

Feedback may still change the implementation and documentation but the
time for wholesale changes of approach has passed.

Cover-letter:
U-Boot integration
END

Changes in v6:
- Fix reference to crossbar in _ub-bisect-try and ub-smoke
- Add -x option to specify the extra board directory
- Add comments to the example lg-env
- Define USE_LABGRID to tell u-boot-hooks to use labgrid hooks
- Add a few more comments to ub-pyt
- Show the build path with ub-pyt -v
- Add --allow-unmatched to ub-smoke to avoid QEMU error
- Add a new section on compatibility
- Add a new section on script usage

Changes in v5:
- Rebase on latest grpc branch
- Fix pylint errors and warnings

Changes in v4:
- Support for Beagleplay, which needs files from two separate U-Boot
  builds
- Support for a 'recovery' button needed to boot the image
- Tidy up the internal-console support
- Fix pytest behaviour with an unpatched U-Boot (that doesn't have lab
  mode)

Changes in v3:
- Rebase on top of grpc branch
- Don't mess with terminal setting unless stdin is a terminal
- Don't show an error if there are no resources when auto-acquiring
- Support QEMU in UBootWriter

Some changes in v2:
- Rationalise the flags for the U-Boot scripts
- Support tracing with em100
- Support an internal terminal instead of microcom
- Add a -D flag for debugging
- Support send-only boards
- Add a way to build the U-Boot config
- Add a control for buildman's process-limit
- allow the build-dir to be specified in a variable
- add documentation about U-Boot-pytest integration
- add source_dir and config_file to UBootProviderDriver
- add an internal terminal
- expand the U-Boot scripts
- significantly improve the U-Boot-pytest integration

The approximate diffstat is:
 contrib/sync-places.py                |  23 +-
 contrib/u-boot/.gitignore             |   1 +
 contrib/u-boot/_ub-bisect-try         |  47 ++
 contrib/u-boot/conftest.py            |  21 +
 contrib/u-boot/get_args.sh            | 128 +++++
 contrib/u-boot/index.rst              | 232 +++++++++
 contrib/u-boot/lg-client              |  11 +
 contrib/u-boot/lg-env                 |  10 +
 contrib/u-boot/test_smoke.py          |   3 +
 contrib/u-boot/ub-bisect              |  44 ++
 contrib/u-boot/ub-cli                 |  39 ++
 contrib/u-boot/ub-int                 |  41 ++
 contrib/u-boot/ub-pyt                 |  69 +++
 contrib/u-boot/ub-smoke               |  48 ++
 doc/configuration.rst                 | 611 ++++++++++++++++++++++-
 doc/usage.rst                         | 288 +++++++++++
 labgrid/driver/__init__.py            |  10 +-
 labgrid/driver/common.py              |  11 +
 labgrid/driver/consoleexpectmixin.py  |   7 +
 labgrid/driver/powerdriver.py         |  29 ++
 labgrid/driver/qemudriver.py          |  84 ++--
 labgrid/driver/recoverydriver.py      |  25 +
 labgrid/driver/resetdriver.py         |   7 +
 labgrid/driver/servodriver.py         | 157 ++++++
 labgrid/driver/sfemulatordriver.py    | 102 ++++
 labgrid/driver/ubootdriver.py         |  27 +-
 labgrid/driver/ubootproviderdriver.py | 323 ++++++++++++
 labgrid/driver/ubootwriterdriver.py   | 160 ++++++
 labgrid/driver/usbhidrelay.py         |   7 +-
 labgrid/driver/usbloader.py           | 170 ++++++-
 labgrid/driver/usbstoragedriver.py    |  18 +-
 labgrid/factory.py                    |   4 +-
 labgrid/protocol/__init__.py          |   1 +
 labgrid/protocol/bootstrapprotocol.py |   8 +-
 labgrid/protocol/recoveryprotocol.py  |  14 +
 labgrid/protocol/resetprotocol.py     |  12 +
 labgrid/pytestplugin/fixtures.py      |  21 +-
 labgrid/pytestplugin/hooks.py         |   8 +
 labgrid/remote/client.py              | 351 ++++++++-----
 labgrid/remote/config.py              |   7 +-
 labgrid/remote/exporter.py            | 122 ++++-
 labgrid/resource/__init__.py          |   6 +
 labgrid/resource/remote.py            |  24 +
 labgrid/resource/servo.py             | 485 ++++++++++++++++++
 labgrid/resource/sfemulator.py        |  33 ++
 labgrid/resource/suggest.py           |   6 +
 labgrid/resource/udev.py              |  33 ++
 labgrid/strategy/ubootstrategy.py     | 151 +++++-
 labgrid/target.py                     | 120 ++++-
 labgrid/util/helper.py                | 220 ++++----
 labgrid/util/ssh.py                   |   3 +-
 labgrid/util/term.py                  | 184 +++++++
 labgrid/var_dict.py                   |   8 +
 man/labgrid-client.1                  |   6 +
 man/labgrid-client.rst                |   4 +
 man/labgrid-device-config.5           |   4 +
 56 files changed, 4267 insertions(+), 321 deletions(-)

[1] https://github.com/labgrid-project/labgrid/issues/created_by/sjg20

Signed-off-by: Simon Glass <[email protected]>