[sonic-package-manager] first phase implementation of sonic-package-manager

doc/Command-Reference.md

@@ -143,6 +143,7 @@
   * [Watermark Show commands](#watermark-show-commands)
   * [Watermark Config commands](#watermark-config-commands)
 * [Software Installation and Management](#software-installation-and-management)
+  * [SONiC Package Manager](#sonic-package-manager)
   * [SONiC Installer](#sonic-installer)
 * [Troubleshooting Commands](#troubleshooting-commands)
 * [Routing Stack](#routing-stack)
@@ -7961,8 +7962,294 @@ Go Back To [Beginning of the document](#) or [Beginning of this section](#waterm
 
 ## Software Installation and Management
 
-SONiC software can be installed in two methods, viz, "using sonic-installer tool", "ONIE Installer".
+SONiC image can be installed in one of two methods:
+1. From within a running SONiC image using the `sonic-installer` utility
+2. From the vendor's bootloader (E.g., ONIE,  Aboot, etc.)
 
+SONiC feature Docker images (aka "SONiC packages") available to be installed with *sonic-package-manager* utility.
+
+### SONiC Package Manager
+
+This is a command line tool that provides functionality to manage SONiC Packages on SONiC device.
+
+**sonic-package-manager list**
+
+This command lists all available SONiC packages, their desription, installed version and installation status.
+SONiC package status can be *Installed*, *Not installed* or *Built-In*. "Built-In" status means that a feature is built-in to SONiC image and can't be upgraded or uninstalled.
+
+- Usage:
+  ```
+  sonic-package-manager list
+  ```
+
+- Example:
+  ```
+  admin@sonic:~$ sonic-package-manager list
+  Name            Repository                   Description                   Version    Status
+  --------------  ---------------------------  ----------------------------  ---------  ---------
+  database        docker-database              SONiC database package        1.0.0      Built-In
+  dhcp-relay      docker-dhcp-relay            SONiC dhcp-relay package      1.0.0      Installed
+  fpm-frr         docker-fpm-frr               SONiC fpm-frr package         1.0.0      Built-In
+  lldp            docker-lldp                  SONiC lldp package            1.0.0      Built-In
+  macsec          docker-macsec                SONiC macsec package          1.0.0      Built-In
+  mgmt-framework  docker-sonic-mgmt-framework  SONiC mgmt-framework package  1.0.0      Built-In
+  nat             docker-nat                   SONiC nat package             1.0.0      Built-In
+  pmon            docker-platform-monitor      SONiC pmon package            1.0.0      Built-In
+  radv            docker-router-advertiser     SONiC radv package            1.0.0      Built-In
+  sflow           docker-sflow                 SONiC sflow package           1.0.0      Built-In
+  snmp            docker-snmp                  SONiC snmp package            1.0.0      Built-In
+  swss            docker-orchagent             SONiC swss package            1.0.0      Built-In
+  syncd           docker-syncd-mlnx            SONiC syncd package           1.0.0      Built-In
+  teamd           docker-teamd                 SONiC teamd package           1.0.0      Built-In
+  telemetry       docker-sonic-telemetry       SONiC telemetry package       1.0.0      Built-In
+  ```
+
+**sonic-package-manager repository add**
+
+This command will add a new entry in the package database. The package has to be *Not Installed* in order to be removed from package database. *NOTE*: this command requires elevated (root) privileges to run.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager repository add [OPTIONS] NAME REPOSITORY
+
+    Add a new repository to database.
+
+  Options:
+    --default-reference TEXT  Default installation reference.
+    --description TEXT        Optional package entry description.
+    --help                    Show this message and exit.
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sudo sonic-package-manager repository add \
+    cpu-report azure/sonic-cpu-report --default-reference 1.0.0
+  ```
+
+**sonic-package-manager repository remove**
+
+This command will remove an entry from the package database. *NOTE*: this command requires elevated (root) privileges to run.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager repository remove [OPTIONS] NAME
+
+    Remove package from database.
+
+  Options:
+    --help  Show this message and exit.
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sudo sonic-package-manager repository remove cpu-report
+  ```
+
+**sonic-package-manager install**
+
+This command pulls and installs package on SONiC host. *NOTE*: this command requires elevated (root) privileges to run.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager install [OPTIONS] [PACKAGE_EXPR]
+
+    Install package
+
+  Options:
+    --enable                        Set the default state of the feature to
+                                    enabled and enable feature right after
+                                    installation. NOTE: user needs to execute
+                                    "config save -y" to make this setting
+                                    persistent
+
+    --default-owner [local|kube]    Default owner configuration setting for a
+                                    feature
+
+    --from-repository TEXT          Fetch package directly from image registry
+                                    repository NOTE: This argument is mutually
+                                    exclusive with arguments: [from_tarball,
+                                    package_expr].
+
+    --from-tarball FILE             Fetch package from saved image tarball
+                                    NOTE: This argument is mutually exclusive
+                                    with arguments: [from_repository,
+                                    package_expr].
+
+    -f, --force                     Force operation by ignoring failures
+    -y, --yes                       Automatically answer yes on prompts
+    -v, --verbosity LVL             Either CRITICAL, ERROR, WARNING, INFO or
+                                    DEBUG
+
+    --skip-cli-plugin-installation  Do not install CLI plugins provided by the
+                                    package on the host OS. NOTE: In case when
+                                    package /cli/mandatory field is set to True
+                                    this option will fail the installation.
+
+    --help                          Show this message and exit.
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sudo sonic-package-manager install dhcp-relay==1.0.2
+  ```
+  ```
+  admin@sonic:~$ sudo sonic-package-manager install --from-repository azure/sonic-cpu-report:latest
+  ```
+  ```
+  admin@sonic:~$ sudo sonic-package-manager install --from-tarball sonic-docker-image.gz
+  ```
+
+**sonic-package-manager uninstall**
+
+This command uninstalls package from SONiC host. *NOTE*: this command requires elevated (root) privileges to run.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager uninstall [OPTIONS] NAME
+
+    Uninstall package
+
+  Options:
+    -f, --force          Force operation by ignoring failures
+    -y, --yes            Automatically answer yes on prompts
+    -v, --verbosity LVL  Either CRITICAL, ERROR, WARNING, INFO or DEBUG
+    --help               Show this message and exit.
+
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sudo sonic-package-manager uninstall dhcp-relay
+  ```
+
+**sonic-package-manager upgrade**
+
+This command upgrades package on SONiC host to a newer version. The procedure of upgrading a package will restart the corresponding service. *NOTE*: this command requires elevated (root) privileges to run.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager upgrade [OPTIONS] [PACKAGE_EXPR]
+
+    Upgrade package
+
+  Options:
+    --from-repository TEXT          Fetch package directly from image registry
+                                    repository NOTE: This argument is mutually
+                                    exclusive with arguments: [package_expr,
+                                    from_tarball].
+
+    --from-tarball FILE             Fetch package from saved image tarball
+                                    NOTE: This argument is mutually exclusive
+                                    with arguments: [package_expr,
+                                    from_repository].
+
+    -f, --force                     Force operation by ignoring failures
+    -y, --yes                       Automatically answer yes on prompts
+    -v, --verbosity LVL             Either CRITICAL, ERROR, WARNING, INFO or
+                                    DEBUG
+
+    --skip-cli-plugin-installation  Do not install CLI plugins provided by the
+                                    package on the host OS. NOTE: In case when
+                                    package /cli/mandatory field is set to True
+                                    this option will fail the installation.
+
+    --help                          Show this message and exit.
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sudo sonic-package-manager upgrade dhcp-relay==2.0.0
+  ```
+  ```
+  admin@sonic:~$ sudo sonic-package-manager upgrade --from-repository azure/sonic-cpu-report:latest
+  ```
+  ```
+  admin@sonic:~$ sudo sonic-package-manager upgrade --from-tarball sonic-docker-image.gz
+  ```
+
+**sonic-package-manager show package versions**
+
+This command will access repository for corresponding package and retrieve a list of available versions.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager show package versions [OPTIONS] NAME
+
+    Print available versions
+
+  Options:
+    --all    Show all available tags in repository
+    --plain  Plain output
+    --help   Show this message and exit.
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sonic-package-manager show package versions dhcp-relay
+  • 1.0.0
+  • 1.0.2
+  • 2.0.0
+  ```
+
+**sonic-package-manager show package changelog**
+
+This command fetches the changelog from package manifest and displays it. *NOTE*: package changelog can be retrieved from registry or read from image tarball without installing it.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager show package changelog [OPTIONS] [PACKAGE_EXPR]
+
+    Print package changelog
+
+  Options:
+    --from-repository TEXT  Fetch package directly from image registry
+                            repository NOTE: This argument is mutually exclusive
+                            with arguments: [from_tarball, package_expr].
+    --from-tarball FILE     Fetch package from saved image tarball NOTE: This
+                            argument is mutually exclusive with arguments:
+                            [package_expr, from_repository].
+    --help                  Show this message and exit.
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sonic-package-manager show package changelog dhcp-relay
+  1.0.0:
+
+    • Initial release
+
+        Author ([email protected]) Mon, 25 May 2020 12:25:00 +0300
+  ```
+
+**sonic-package-manager show package manifest**
+
+This command fetches the package manifest and displays it. *NOTE*: package manifest can be retrieved from registry or read from image tarball without installing it.
+
+- Usage:
+  ```
+  Usage: sonic-package-manager show package manifest [OPTIONS] [PACKAGE_EXPR]
+
+    Print package manifest content
+
+  Options:
+    --from-repository TEXT  Fetch package directly from image registry
+                            repository NOTE: This argument is mutually exclusive
+                            with arguments: [package_expr, from_tarball].
+    --from-tarball FILE     Fetch package from saved image tarball NOTE: This
+                            argument is mutually exclusive with arguments:
+                            [from_repository, package_expr].
+    -v, --verbosity LVL     Either CRITICAL, ERROR, WARNING, INFO or DEBUG
+    --help                  Show this message and exit.
+  ```
+- Example:
+  ```
+  admin@sonic:~$ sonic-package-manager show package manifest dhcp-relay==2.0.0
+  {
+    "version": "1.0.0",
+    "package": {
+      "version": "2.0.0",
+      "depends": [
+        "database>=1.0.0,<2.0.0"
+      ]
+    },
+    "service": {
+      "name": "dhcp_relay"
+    }
+  }
+  ```
 
 ### SONiC Installer
 This is a command line tool available as part of the SONiC software; If the device is already running the SONiC software, this tool can be used to install an alternate image in the partition.
@@ -8033,6 +8320,13 @@ This command is used to install a new image on the alternate image partition.  T
   Done
   ```
 
+SONiC image installation will install SONiC packages that are installed in currently running SONiC image. In order to perform clean SONiC installation use *--skip-package-migration* option when installing SONiC image:
+
+- Example:
+  ```
+  admin@sonic:~$ sudo sonic-installer install https://sonic-jenkins.westus.cloudapp.azure.com/job/xxxx/job/buildimage-xxxx-all/xxx/artifact/target/sonic-xxxx.bin --skip-package-migration
+  ```
+
 **sonic-installer set_default**
 
 This command is be used to change the image which can be loaded by default in all the subsequent reboots.

setup.py

@@ -47,6 +47,8 @@
         'show.plugins',
         'sonic_installer',
         'sonic_installer.bootloader',
+        'sonic_package_manager',
+        'sonic_package_manager.service_creator',
         'tests',
         'undebug',
         'utilities_common',
@@ -149,34 +151,47 @@
             'sonic-clear = clear.main:cli',
             'sonic-installer = sonic_installer.main:sonic_installer',
             'sonic_installer = sonic_installer.main:sonic_installer',  # Deprecated
+            'sonic-package-manager = sonic_package_manager.main:cli',
+            'spm = sonic_package_manager.main:cli',
             'undebug = undebug.main:cli',
             'watchdogutil = watchdogutil.main:watchdogutil',
         ]
     },
     install_requires=[
         'click==7.0',
+        'click-log==0.3.2',
+        'docker==4.4.4',
+        'docker-image-py==0.1.10',
+        'filelock==3.0.12',
+        'enlighten==1.8.0',
         'ipaddress==1.0.23',
+        'jinja2==2.11.3',
         'jsondiff==1.2.0',
         'm2crypto==0.31.0',
         'natsort==6.2.1',  # 6.2.1 is the last version which supports Python 2. Can update once we no longer support Python 2
         'netaddr==0.8.0',
         'netifaces==0.10.7',
         'pexpect==4.8.0',
+        'poetry-semver==0.1.0',
+        'prettyprinter==0.18.0',
         'pyroute2==0.5.14',
         'requests==2.25.0',
         'sonic-platform-common',
         'sonic-py-common',
         'sonic-yang-mgmt',
         'swsssdk>=2.0.1',
         'tabulate==0.8.2',
+        'www-authenticate==0.9.2',
         'xmltodict==0.12.0'
     ],
     setup_requires= [
         'pytest-runner',
         'wheel'
     ],
     tests_require = [
+        'pyfakefs',
         'pytest',
+        'mock>=2.0.0',
         'mockredispy>=2.9.3',
         'sonic-config-engine',
         'deepdiff==5.2.3'

sonic-utilities-data/bash_completion.d/sonic-package-manager

@@ -0,0 +1,8 @@
+_sonic_package_manager_completion() {
+    COMPREPLY=( $( env COMP_WORDS="${COMP_WORDS[*]}" \
+                   COMP_CWORD=$COMP_CWORD \
+                   _SONIC_PACKAGE_MANAGER_COMPLETE=complete $1 ) )
+    return 0
+}
+
+complete -F _sonic_package_manager_completion -o default sonic-package-manager;

sonic-utilities-data/bash_completion.d/spm

@@ -0,0 +1 @@
+sonic-package-manager