Add firmware_update phase #91

nancyc12 · 2023-10-13T12:44:17Z

Description

Base on CR033 - Firmware updates for regression testing, introduce a new optional phase firmware_update after provision phase to handle request for firmware update on the device. As in the spec, this new phase is implemented in DefaultDevice, so it won't need any change on existing DeviceConnector codes.

To trigger firmware_update phase, the following section needs to be provided in job.yaml. If the firmware_update_data section is missing from the job, this phase will not run.

firmware_update_data:
  version: < latest >
  ignore_failure: < false | true >

Variables in firmware_update_data:

version: The desired firmware level on the device. Currently it only supports latest (upgrading all components in the device with the latest firmware release). More options are planned to be implemented in the future once we gathering data for working firmware baseline on devices.
ignore_failure: When set to false, Testflinger Agent will suspend the job if firmware_update phase return a status other than 0, which implies there's a failure during firmware_update phase. If set to true, the job will continue regardless the status of firmware_update phase.

Code changes

Add firmware_update phase in DefaultDevice to handle the firmware update for all child DeviceConnector class. It will gather DMI chassis_vendor and chassis_type from the DUT to decide whether firmware update is applicable. The current support scope is limited the devices as below.

chassis_vendor: HP, Lenovo, Dell
chassis_type: All In One, Desktop, Mini PC, Rack Mount Chassis

In order to make it work as a whole, there're also PRs raised in testflinger and testflinger-agent.

Resolved issues

N/A

Documentation

Covered in testflinger monorepo.

Tests

It's tested by building a dev testflinger server with docker along with modified testflinger-agent and snappy-device-agents.

Unit tests

(env) ubuntu@juju-ee0bf9-0:/srv/testflinger-agent/dell-optiplex3070-c26629/snappy-device-agents$  sudo  /srv/testflinger-agent/dell-optiplex3070-c26629/env/bin/python3 -m coverage run -m pytest .
============================================== test session starts ==============================================
platform linux -- Python 3.8.10, pytest-7.4.2, pluggy-1.3.0
rootdir: /srv/testflinger-agent/dell-optiplex3070-c26629/snappy-device-agents
plugins: requests-mock-1.11.0
collecting ... 
collected 36 items                                                                                              

src/testflinger_device_connectors/devices/maas2/tests/test_maas_storage.py ..................             [ 50%]
src/testflinger_device_connectors/devices/multi/tests/test_multi.py ....                                  [ 61%]
src/testflinger_device_connectors/devices/muxpi/tests/test_muxpi.py E                                     [ 63%]
src/testflinger_device_connectors/fw_devices/LVFS/tests/test_LVFS.py ......                               [ 80%]
src/testflinger_device_connectors/fw_devices/tests/test_firmware_update.py ....                           [ 91%]
src/tests/test_snappy_device_agents.py ...                                                                [100%]

==================================================== ERRORS =====================================================
_________________________________ ERROR at setup of test_check_ce_oem_iot_image _________________________________
file /srv/testflinger-agent/dell-optiplex3070-c26629/snappy-device-agents/src/testflinger_device_connectors/devices/muxpi/tests/test_muxpi.py, line 20
  def test_check_ce_oem_iot_image(mocker):
E       fixture 'mocker' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, requests_mock, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/srv/testflinger-agent/dell-optiplex3070-c26629/snappy-device-agents/src/testflinger_device_connectors/devices/muxpi/tests/test_muxpi.py:20
=============================================== warnings summary ================================================
../env/lib/python3.8/site-packages/testflinger_device_connectors/devices/__init__.py:15
  /srv/testflinger-agent/dell-optiplex3070-c26629/env/lib/python3.8/site-packages/testflinger_device_connectors/devices/__init__.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================ short test summary info ============================================
ERROR src/testflinger_device_connectors/devices/muxpi/tests/test_muxpi.py::test_check_ce_oem_iot_image
==================================== 35 passed, 1 warning, 1 error in 0.25s =====================================

Testflinger Agent is able to work with Testflinger Device Connector (tested with noprovision) to run firmware_update phase

**************************************************************************

* Starting testflinger firmware_update phase on dell-optiplex3070-c26629 *

**************************************************************************
2023-10-13 12:06:46,578 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: BEGIN firmware_update
2023-10-13 12:06:47,130 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: 10.102.163.206 is a Dell Inc. LVFSDevice
2023-10-13 12:06:57,154 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: collect firmware info
2023-10-13 12:06:58,600 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: [ST500LM021-1KJ152] current version: 0005SDM1 - no available firmware on LVFS
2023-10-13 12:06:58,601 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: [System Firmware] current version: 1.21.0 - LVFS upgradable version(s): 1.22.0 - LVFS downgradable version(s): 1.20.0
2023-10-13 12:06:58,601 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: [UEFI dbx] current version: 77 - no available firmware on LVFS
2023-10-13 12:06:58,601 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: start upgrading
2023-10-13 12:06:58,601 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: [System Firmware] try upgrading to 1.22.0
2023-10-13 12:07:05,600 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: reboot DUT
2023-10-13 12:07:16,680 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: check and wait for 900s until 10.102.163.206 is SSHable
2023-10-13 12:08:53,927 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: 10.102.163.206 is SSHable after 97s
2023-10-13 12:09:01,255 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: [System Firmware] FWUPD_UPDATE_STATE_FAILED: failed to run update on reboot
2023-10-13 12:09:01,255 dell-optiplex3070-c26629 INFO: DEVICE CONNECTOR: END firmware_update

…cal/snappy-device-agents into add_firmware_update_phase

plars

Great progress, and I like what I see so far! This is obviously the trickiest part of the implementation. You have a few formatting errors that the auto-checker caught, but I think those should be easy to fix. I have a few additional comments below.

plars · 2023-10-16T19:13:52Z

src/testflinger_device_connectors/fw_devices/tests/test_firmware_update.py

+
+    def mock_run_cmd_fail(*args, **kwargs):
+        """
+        Mock run_cmd for a Rigado Cascade 500 device (201810-26506), which doesn't


There's no need for specific device data in places like this, please remove things like this.

plars · 2023-10-16T19:13:57Z

src/testflinger_device_connectors/fw_devices/tests/test_firmware_update.py

+
+    def mock_run_cmd_unsupported(*args, **kwargs):
+        """
+        Mock run_cmd for an Intel Desktop device (201808-26453), which doesn't


There's no need for specific device data in places like this, please remove things like this.

plars · 2023-10-16T19:14:31Z

src/testflinger_device_connectors/fw_devices/tests/test_firmware_update.py

+            return (
+                255,
+                "",
+                "ssh: connect to host 10.102.161.93 port 22: No route to host",


replace with some simpler fake IP that's easier to read for things like this - it's all mock data anyway

plars · 2023-10-16T19:18:19Z

src/testflinger_device_connectors/fw_devices/LVFS/tests/fwupd_data.py

@@ -0,0 +1,387 @@
+""" fwupdmgr json outputs """


The data here seems very specific, and I'm guessing we grabbed it from a fwupdmgr run to use as test data. Can you add a few details in the comments here about how it was extracted, just in case the format ever changes and we need to update the unit test data?

plars · 2023-10-20T17:03:39Z

@nancyc12 also here - we'll need to redo this patch against https://github.com/canonical/testflinger, let me know if there's anything I can do to help.

nancyc12 · 2023-10-23T07:31:58Z

@plars copy that.

nancyc12 · 2023-10-25T09:52:46Z

Move the changes to canonical/testflinger#141

nancyc12 added 4 commits October 11, 2023 07:44

add firmware update phase

2df1a56

add firmware update phase

f1a5d71

Merge branch 'add_firmware_update_phase' of https://github.com/canoni…

cd878f7

…cal/snappy-device-agents into add_firmware_update_phase

remove unsupported devices

463b4d8

This was referenced Oct 13, 2023

Add firmware_update phase canonical/testflinger-agent#40

Closed

Add firmware_update phase canonical/testflinger#114

Closed

plars suggested changes Oct 16, 2023

View reviewed changes

fix formatting errors

d71c025

nancyc12 mentioned this pull request Oct 25, 2023

Add firmware_update phase canonical/testflinger#141

Merged

8 tasks

nancyc12 closed this Oct 25, 2023

nancyc12 mentioned this pull request Nov 28, 2023

Update unit test docstring and data canonical/testflinger#164

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add firmware_update phase #91

Add firmware_update phase #91

nancyc12 commented Oct 13, 2023 •

edited

Loading

plars left a comment

plars Oct 16, 2023

plars Oct 16, 2023

plars Oct 16, 2023

plars Oct 16, 2023

plars commented Oct 20, 2023

nancyc12 commented Oct 23, 2023 •

edited

Loading

nancyc12 commented Oct 25, 2023

Add firmware_update phase #91

Add firmware_update phase #91

Conversation

nancyc12 commented Oct 13, 2023 • edited Loading

Description

Code changes

Resolved issues

Documentation

Tests

plars left a comment

Choose a reason for hiding this comment

plars Oct 16, 2023

Choose a reason for hiding this comment

plars Oct 16, 2023

Choose a reason for hiding this comment

plars Oct 16, 2023

Choose a reason for hiding this comment

plars Oct 16, 2023

Choose a reason for hiding this comment

plars commented Oct 20, 2023

nancyc12 commented Oct 23, 2023 • edited Loading

nancyc12 commented Oct 25, 2023

nancyc12 commented Oct 13, 2023 •

edited

Loading

nancyc12 commented Oct 23, 2023 •

edited

Loading