Project hawthorn -Binary size reduction (ARMmbed#76)

* Remove pal_crypto calls

and call mbedtls api directly

* Add the mbed-printf library

By using this library, the code size of the bootloader will be smaller.
For instructions on how to use it, check the README file of the library.

* Use arm update client insecure rot

* Use sd blockdevice as firmware storage (ARMmbed#60)

Thus eliminating the need for a filesystem,
reducing code size.

Fix bug where we are hashing past the end
of firmware

* Trim buffer before read

So that we don't attempt an read
beyond the end of file.

Remove no longer needed include.

* Use mbed configuration system to configure blockdevice offset

MBED_CONF_UPDATE_CLIENT_STORAGE_ADDRESS and
MBED_CONF_UPDATE_CLIENT_STORAGE_SIZE

Are used directly by the implementation pal-blockdevice
This removes the need of slicing block device layer

* Update mbed-printf to solve a compile error

In c99 <stdbool.h> needs to be included
explicitly to use the bool keyword
on some platform which does not include
this automatically, compilation fails.

* In release script use custom compiler profile

To use custom version of printf from mbed-printf

* Remove PAL (ARMmbed#63)

* Remove mbed client PAL dependency from bootloader
* import update-client-common changes to remove pal refs

* Remove mbed client PAL from bootloader (ARMmbed#65)

* remove PAL from bootloader test

* Fix/power cut tests (ARMmbed#66)

* Trim buffer so we don't read beyond end of storage

* Get around limitation of pal-blockdevice

Where it cannot handle write buffer size
which is not page size aligned correctly

* Fix an bug when trimming buffers

* Add blockdevice configuration to test mbed_app.json

* Pull in fix for setting buffer size

* Remove RTOS and other mbed-os components

This PR removes the RTOS and other mbed-os components from the
bootloader code:

1. .mbedignore contains a list of mbed-os directories that are now
ignored. There are other components ignored besides the RTOS (for
example mbed-printf).
2. "Mutex=PlatformMutex" in mbed_app.json is needed to fix an issue in
the sd-driver library (see PelionIoT/sd-driver#68
for details).
3. the .lib updates contain fixes related to compiling the code in debug
mode without mbed-printf.

With these codes and using arm-none-eabi-gcc 4.9.3 (GCC ARM embedded)
with the release profile in `mbed-printf/profiles/release.json`, the
size of the bootloader binary image is 32328 bytes.

* Allow greentea compilation (needed by tests)

* greentea compilation fix for tests

* Use the ARMCC fix for update-client-pal-flashiap

* Enable nano.specs for GCC

This will result in additional size savings.
Also enable this new profile in both CircleCI and Jenkins.

* Disable fflush

Remove the fflush function from the image:

- disable the "stdio flush at exit" option in mbed-os.
- disable `tr_flush` if mbed-printf is used, since mbed-printf is not
  buffered.

* Feature/depend on hub (ARMmbed#75)

* Depend on update-client-hub

which includes all relavent modules in
this unified repo

* Modify release script to take a compiler profile argument

* Use buffer size member as indication of how many bytes need to be read.

Updage firmware even if the version is the same, if the active firmware
is corrupt.

* Fix compilation for ODIN

Clean up mbedtls config file and
remove unnecessary algorithms

* Set MBEDTLS_SHA256_SMALLER to reduce binary size

saves 1.3k

* Update hub

to pull in:
Conditional compilation of blockdevice
IAR compilation fixes

* Remove dependency on cloud client

depend on update client hub directly

* Use tiny.json as default build profile for release

* Turn on blockdevice for power cut tests

* Improve tests

Build tests only for ARMCC profile
Use ci raas user name and pass word
Use domain name istead of ip for raas server

* remove instructions on mbed-printf build profile

replaced by tiny.json

.mbedignore

@@ -1,18 +1,25 @@
 */test/*
-mbed-cloud-client-internal/factory-configurator-client/*
-mbed-cloud-client-internal/mbed-client/*
-mbed-cloud-client-internal/mbed-cloud-client/*
-mbed-cloud-client-internal/source/*
-mbed-cloud-client-internal/update-client-hub/source/*
-mbed-cloud-client-internal/update-client-hub/modules/atomic-queue/*
-mbed-cloud-client-internal/update-client-hub/modules/control-center/*
-mbed-cloud-client-internal/update-client-hub/modules/device-identity/*
-mbed-cloud-client-internal/update-client-hub/modules/firmware-manager/*
-mbed-cloud-client-internal/update-client-hub/modules/lwm2m-mbed/*
-mbed-cloud-client-internal/update-client-hub/modules/manifest-manager/*
-mbed-cloud-client-internal/update-client-hub/modules/monitor/*
-mbed-cloud-client-internal/update-client-hub/modules/source/*
-mbed-cloud-client-internal/update-client-hub/modules/source-http/*
-mbed-cloud-client-internal/update-client-hub/modules/source-http-socket/*
-mbed-cloud-client-internal/update-client-hub/modules/source-manager/*
-mbed-cloud-client-internal/update-client-hub/modules/common/source/arm_uc_scheduler.c
+mbed-os/rtos/*
+mbed-os/features/FEATURE_CLIENT/*
+mbed-os/features/FEATURE_COMMON_PAL/*
+mbed-os/features/FEATURE_UVISOR/*
+mbed-os/features/net/*
+mbed-os/features/netsocket/*
+mbed-os/features/storage/*
+mbed-os/events/*
+update-client-hub/source/*
+update-client-hub/modules/atomic-queue/*
+update-client-hub/modules/control-center/*
+update-client-hub/modules/firmware-manager/*
+update-client-hub/modules/manifest-manager/*
+update-client-hub/modules/pal-linux/*
+update-client-hub/modules/source/*
+update-client-hub/modules/source-http-socket/*
+update-client-hub/modules/device-identity/*
+update-client-hub/modules/lwm2m-mbed/*
+update-client-hub/modules/monitor/*
+update-client-hub/modules/pal-filesystem/*
+update-client-hub/modules/pal-target-specific/*
+update-client-hub/modules/source-http/*
+update-client-hub/modules/source-manager/*
+update-client-hub/modules/common/source/arm_uc_scheduler.c

Jenkinsfile

@@ -118,29 +118,25 @@ def morpheusBuildStep(target,
 
           // build with default setup
           execute("mbed ls")
-          execute("mbed compile -m ${target} -t ${toolchain} -c -v")
+          execute("mbed compile -m ${target} -t ${toolchain} -c -v --profile=tiny.json")
 
-          // setup dependencies
-          execute("power_cut_test/setup_dependencies.sh")
+          if (toolchain == "ARM") {
+            // setup dependencies
+            execute("power_cut_test/setup_dependencies.sh")
 
-          // build with test setup
-          execute("mbed ls")
-          execute("mbed compile -m ${target} -t ${toolchain} -c -v")
+            // build with test setup
+            execute("mbed ls")
+            execute("mbed compile -m ${target} -t ${toolchain} -c -v --profile=tiny.json")
 
-          // Test with power cut hardware
-          env.RAAS_USERNAME = "user"
-          env.RAAS_PASSWORD = "user"
-          env.RAAS_PYCLIENT_FORCE_REMOTE_ALLOCATION = 1
+            // Test with power cut hardware
+            env.RAAS_USERNAME = "ci"
+            env.RAAS_PASSWORD = "ci"
+            env.RAAS_PYCLIENT_FORCE_REMOTE_ALLOCATION = 1
 
-          if (toolchain == "ARM") {
             test_spec = "power_cut_test/test_spec_armcc.json"
 
-            execute("mbedgt --grm ${target}:raas_client:62.44.193.186:8055 -V -v --test-spec ${test_spec} -e power_cut_test/host_tests/")
-
-          } else if (toolchain == "GCC_ARM") {
-            test_spec = "power_cut_test/test_spec_gcc.json"
+            execute("mbedgt --grm ${target}:raas_client:oulab.mbedcloudtesting.com:8055 -V -v --test-spec ${test_spec} -e power_cut_test/host_tests/")
           }
-
         }
       }
     }

README.md

@@ -13,7 +13,7 @@ The interface between the bootloader and an 'update-client' have been described
 
 1. Install `mbed-cli` https://github.com/ARMmbed/mbed-cli
 1. Run `mbed deploy` to pull in dependencies
-1. Compile by running `mbed compile -t GCC_ARM -m (K64F|NUCLEO_F429ZI|UBLOX_EVK_ODIN_W2)`
+1. Compile by running `mbed compile -t GCC_ARM -m (K64F|NUCLEO_F429ZI|UBLOX_EVK_ODIN_W2) --profile=tiny.json`
 1. Use this [script](https://github.com/ARMmbed/mbed-cloud-client-example/blob/master/tools/combine_bootloader_with_app.py) to combine the bootloader with application `python tools/combine_bootloader_with_app.py -a {application.bin} -b {bootloader.bin} --app-offset {firmware_metadata_header_address+firmware_metadata_header_size} --header-offset {firmware_metadata_header_address} -o {combined.bin}`.
 1. Flash `{combined.bin}` to device by drag and drop.
 

circle.yml

@@ -14,7 +14,7 @@ dependencies:
 test:
   override:
     - mbed deploy --protocol ssh
-    - mbed compile -m K64F -t GCC_ARM --profile=../mbed-os-linker-report/compiler_profiles/release.json
+    - mbed compile -m K64F -t GCC_ARM --profile=tiny.json
     - echo $(stat --printf="%s" BUILD/K64F/GCC_ARM/mbed-bootloader-internal.bin) | tee BINSIZE # current build size 
   post:
     - python ../mbed-os-linker-report/elfsize.py -i BUILD/K64F/GCC_ARM/mbed-bootloader-internal.elf

firmware_update_test/firmware_update_test.cpp

@@ -22,7 +22,6 @@
 #include "update-client-paal/arm_uc_paal_update.h"
 #include "bootloader_common.h"
 #include "active_application.h"
-#include "pal.h"
 #include "mbed.h"
 #include "mbedtls/sha256.h"
 

fix-mbed-os-compile/README.md

@@ -0,0 +1 @@
+This directory contains two empty header files, meant as a temporary fix for greentea compilation when the RTOS is not enabled. See https://github.com/ARMmbed/mbed-bootloader-internal/pull/6 for more details.

mbed-printf.lib

@@ -0,0 +1 @@
+https://github.com/ARMmbed/mbed-printf/#613c02ed3eeb09b86c7d0c6c9ae0059bd0c31ae1

mbed_app.json

@@ -2,11 +2,13 @@
     "macros": [
         "MBEDTLS_USER_CONFIG_FILE=\"mbedtls_mbed_client_config.h\"",
         "PAL_USE_FATFS_SD=1",
-        "MAX_FIRMWARE_LOCATIONS=2",
+        "MAX_FIRMWARE_LOCATIONS=1",
         "MAX_COPY_RETRIES=1",
         "SHOW_PROGRESS_BAR=1",
         "MAX_BOOT_RETRIES=3",
-        "PAL_CMAC_SUPPORT=0"
+        "ARM_UC_USE_PAL_CRYPTO=0",
+        "Mutex=PlatformMutex",
+        "ARM_UC_USE_PAL_BLOCKDEVICE"
     ],
     "config": {
         "firmware_metadata_header_address": {
@@ -26,7 +28,11 @@
         "*": {
             "target.features_add": ["COMMON_PAL"],
             "target.features_remove": ["LWIP"],
-            "platform.stdio-baud-rate": 115200
+            "platform.stdio-baud-rate": 115200,
+            "update-client.storage-address": "1024*1024*64",
+            "update-client.storage-size": "1024*1024",
+            "update-client.storage-locations": "MAX_FIRMWARE_LOCATIONS",
+            "platform.stdio-flush-at-exit": false
         },
         "K64F": {
             "firmware_metadata_header_address": "0x20000",
@@ -49,6 +55,7 @@
             "firmware_metadata_header_size": "0x800"
         },
         "UBLOX_EVK_ODIN_W2": {
+            "target.device_has_remove": ["EMAC"],
             "firmware_metadata_header_address": "0x08020000",
             "update-client.application-details": "0x08020000",
             "firmware_metadata_header_size": "0x400"

mbedtls_mbed_client_config.h

@@ -22,55 +22,13 @@
 /* System support */
 #define MBEDTLS_HAVE_ASM
 
-/* Crypto flags */
+#define MBEDTLS_SHA256_SMALLER
 #define MBEDTLS_CIPHER_MODE_CTR
-#define MBEDTLS_CMAC_C
-#define MBEDTLS_PEM_WRITE_C
-#define MBEDTLS_X509_CSR_WRITE_C
-#define MBEDTLS_X509_CREATE_C
-//#define MBEDTLS_PLATFORM_TIME_ALT
-
-/* mbed TLS feature support */
-#define MBEDTLS_ECP_DP_SECP256R1_ENABLED
-#define MBEDTLS_ECP_NIST_OPTIM
-#define MBEDTLS_SSL_MAX_FRAGMENT_LENGTH
-#define MBEDTLS_SSL_PROTO_TLS1_2
-#define MBEDTLS_SSL_PROTO_DTLS
-#define MBEDTLS_SSL_DTLS_ANTI_REPLAY
-#define MBEDTLS_SSL_DTLS_HELLO_VERIFY
-#define MBEDTLS_SSL_EXPORT_KEYS
-
-/* mbed TLS modules */
-#define MBEDTLS_AES_C
-#define MBEDTLS_ASN1_PARSE_C
-#define MBEDTLS_ASN1_WRITE_C
-#define MBEDTLS_BIGNUM_C
-#define MBEDTLS_CIPHER_C
-#define MBEDTLS_CTR_DRBG_C
-#define MBEDTLS_ECP_C
-#define MBEDTLS_ENTROPY_C
-#define MBEDTLS_MD_C
-#define MBEDTLS_OID_C
-#define MBEDTLS_PK_C
-#define MBEDTLS_PK_PARSE_C
-#define MBEDTLS_SHA256_C
-#define MBEDTLS_SSL_COOKIE_C
-#define MBEDTLS_SSL_CLI_C
-#define MBEDTLS_SSL_SRV_C
-#define MBEDTLS_SSL_TLS_C
-
-// XXX mbedclient needs these: mbedtls_x509_crt_free, mbedtls_x509_crt_init, mbedtls_x509_crt_parse
-#define MBEDTLS_X509_USE_C
-#define MBEDTLS_X509_CRT_PARSE_C
-// a bit wrong way to get mbedtls_ssl_conf_psk:
 
 #undef MBEDTLS_SHA512_C
-#define MBEDTLS_ECDH_C
-#define MBEDTLS_GCM_C
-
-#define MBEDTLS_ECDH_C
-#define MBEDTLS_ECDSA_C
-#define MBEDTLS_X509_CRT_PARSE_C
+#undef MBEDTLS_MD5_C
+#undef MBEDTLS_MD4_C
+#undef MBEDTLS_SHA1_C
 
 // Remove RSA, save 20KB at total
 #undef MBEDTLS_RSA_C
@@ -87,24 +45,6 @@
 #undef MBEDTLS_VERSION_FEATURES
 #undef MBEDTLS_DEBUG_C
 
-// needed for parsing the certificates
-#define MBEDTLS_PEM_PARSE_C
-// dep of the previous
-#define MBEDTLS_BASE64_C
-
-// Needed by provisioning
-#define MBEDTLS_PEM_WRITE_C
-#define MBEDTLS_CTR_DRBG_MAX_REQUEST 2048
-
-// Reduce IO buffer to save RAM, default is 16KB
-#define MBEDTLS_SSL_MAX_CONTENT_LEN 8096
-
-// define to save 8KB RAM at the expense of ROM
-#define MBEDTLS_AES_ROM_TABLES
-
-// Save ROM and a few bytes of RAM by specifying our own ciphersuite list
-#define MBEDTLS_SSL_CIPHERSUITES MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,MBEDTLS_TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
-
 #include "mbedtls/check_config.h"
 
 #endif /* MBEDTLS_CUSTOM_CONFIG_H */

power_cut_test/test_mbed_app.json

@@ -2,10 +2,14 @@
     "macros": [
         "MBEDTLS_USER_CONFIG_FILE=\"mbedtls_mbed_client_config.h\"",
         "PAL_USE_FATFS_SD=1",
+        "MAX_FIRMWARE_LOCATIONS=1",
         "MAX_COPY_RETRIES=1",
         "SHOW_PROGRESS_BAR=1",
         "MAX_BOOT_RETRIES=3",
-        "BOOTLOADER_POWER_CUT_TEST=1"
+        "ARM_UC_USE_PAL_CRYPTO=0",
+        "BOOTLOADER_POWER_CUT_TEST=1",
+        "Mutex=PlatformMutex",
+        "ARM_UC_USE_PAL_BLOCKDEVICE"
     ],
     "config": {
         "firmware_metadata_header_address": {
@@ -24,12 +28,16 @@
     "target_overrides": {
         "*": {
             "target.features_add": ["COMMON_PAL"],
-            "platform.stdio-baud-rate": 9600
+            "platform.stdio-baud-rate": 9600,
+            "target.features_remove": ["LWIP"],
+            "update-client.storage-address": "1024*1024*64",
+            "update-client.storage-size": "1024*1024",
+            "update-client.storage-locations": "MAX_FIRMWARE_LOCATIONS"
         },
         "K64F": {
             "firmware_metadata_header_address": "0x20000",
             "update-client.application-details": "0x20000",
             "firmware_metadata_header_size": "0x400"
         }
     }
-}
+}

scripts/make_release.py

@@ -21,7 +21,7 @@
 
 targets = ["K64F", "NUCLEO_F429ZI", "UBLOX_EVK_ODIN_W2"]
 toolchain = "GCC_ARM"
-profile = "release"
+profile = "tiny.json" # default value, changed via command line argment --profile
 bootloader_repo_name = os.path.basename(os.getcwd())
 
 def check_clean():
@@ -86,6 +86,9 @@ def find_offset(map_file, symbol):
     parser.add_argument('-o', '--output', required=True,
                         help='Path to top level example application')
 
+    parser.add_argument('-p', '--profile', required=False,
+                        help='Compiler profile for mbed os')
+
     # workaround for http://bugs.python.org/issue9694
     parser._optionals.title = "arguments"
 
@@ -98,6 +101,8 @@ def find_offset(map_file, symbol):
         print args.output, "is not a readable directory"
         sys.exit(1)
 
+    profile = args.profile
+
     # write SHA1 into mbed_bootloader_info.h
     patch_version()
 

source/active_application.cpp

@@ -23,9 +23,10 @@
 #include "active_application.h"
 #include "bootloader_common.h"
 
-#include "update-client-common/arm_uc_utilities.h"
 #include "update-client-common/arm_uc_metadata_header_v2.h"
+#include "update-client-common/arm_uc_utilities.h"
 #include "update-client-paal/arm_uc_paal_update.h"
+#include "mbedtls/sha256.h"
 #include "mbed.h"
 
 #include <inttypes.h>
@@ -123,10 +124,11 @@ int checkActiveApplication(arm_uc_firmware_details_t* details)
             tr_debug("app size: %" PRIu64, details->size);
 
             /* initialize hashing facility */
-            palMDHandle_t md = { 0 };
-            pal_mdInit(&md, PAL_SHA256);
+            mbedtls_sha256_context mbedtls_ctx;
+            mbedtls_sha256_init(&mbedtls_ctx);
+            mbedtls_sha256_starts(&mbedtls_ctx, 0);
 
-            uint8_t SHA[PAL_SHA256_SIZE] = { 0 };
+            uint8_t SHA[SIZEOF_SHA256] = { 0 };
             uint32_t remaining = details->size;
             int32_t status = 0;
 
@@ -143,7 +145,7 @@ int checkActiveApplication(arm_uc_firmware_details_t* details)
                                     readSize);
 
                 /* update hash */
-                pal_mdUpdate(md, buffer_array, readSize);
+                mbedtls_sha256_update(&mbedtls_ctx, buffer_array, readSize);
 
                 /* update remaining bytes */
                 remaining -= readSize;
@@ -155,11 +157,11 @@ int checkActiveApplication(arm_uc_firmware_details_t* details)
             }
 
             /* finalize hash */
-            pal_mdFinal(md, SHA);
-            pal_mdFree(&md);
+            mbedtls_sha256_finish(&mbedtls_ctx, SHA);
+            mbedtls_sha256_free(&mbedtls_ctx);
 
             /* compare calculated hash with hash from header */
-            int diff = memcmp(details->hash, SHA, PAL_SHA256_SIZE);
+            int diff = memcmp(details->hash, SHA, SIZEOF_SHA256);
 
             if (diff == 0)
             {
@@ -306,6 +308,10 @@ bool writeActiveFirmware(uint32_t index, arm_uc_firmware_details_t* details)
             /* clear most recent UCP event */
             event_callback = CLEAR_EVENT;
 
+            /* set the number of bytes expected */
+            buffer.size = (details->size - offset) > buffer.size_max ?
+                            buffer.size_max : (details->size - offset);
+
             /* fill buffer using UCP */
             arm_uc_error_t ucp_status = ARM_UCP_Read(index, offset, &buffer);
 

source/active_application.h

@@ -18,8 +18,6 @@
 
 #include "update-client-paal/arm_uc_paal_update_api.h"
 
-#include "pal.h"
-
 #include <stdint.h>
 
 bool activeStorageInit(void);

source/bootloader_common.c

@@ -44,12 +44,12 @@ void arm_ucp_event_handler(uint32_t event)
  * Helper function to print a SHA-256 in a nice format.
  * @param [in]  SHA  The array of PAL_SHA256_SIZE containing the SHA256
  */
-void printSHA256(const uint8_t SHA[PAL_SHA256_SIZE])
+void printSHA256(const uint8_t SHA[SIZEOF_SHA256])
 {
     /* allocate space for string */
-    char buffer[2 * PAL_SHA256_SIZE + 1] = { 0 };
+    char buffer[2 * SIZEOF_SHA256 + 1] = { 0 };
 
-    for (uint_least8_t index = 0; index < PAL_SHA256_SIZE; index++)
+    for (uint_least8_t index = 0; index < SIZEOF_SHA256; index++)
     {
         uint8_t value = SHA[index];