Improvement to build, install, and distribution mechanisms

[sdavtaker]

Describe the issue

Feedback from our customers show that AWS SDK for C++ complexity to build is high. Our customers come from very diverse disciplines and backgrounds, that requires AWS SDK for C++ to build under a vast set of constraints. These constraints usually come from using different operating systems, toolchains, memory management processes, and domain specific practices to mention a few.

Since AWS SDK for C++ was first released, we extended our cmake scripts to fulfill needs from new platforms, new uses cases we identified, and new tooling mostly as a collection of small individual changes with no significant impact on the build as a whole. However, as the collection of them grew, the build as a whole became a very complex piece of code to work with. This building complexity many times translates in time spent for developers who want to install the AWS SDK for C++ to work in their projects, for contributors and reviewers wanting to implement changes to fix issues with it, for package managers to release distributions, for release managers to distribute it as a dependency of their projects.

We are working in a redesign of our cmake scripts to be released at v1.10. We expect to provide a simpler and idiomatic modern cmake experience based in standard industry patterns.
Here is a list of some of the changes we will introduce:

• Implement superbuild pattern.
• Remove the need to pull dependencies via git submodules.
• Restructure project directories to easily distinct between generated-src and src.
• Provide CPack targets.
• Declaring dependencies closer to their targets so they are not required when not used. For example, no test tools will be required when tests are not expected to be run.
• Make configuration parameters names reflect better cmake community practices.
• A legacy mode that will mantain backward compatibility with current build scripts for allowing an easy transition to customers.
• Simplify toolchain customizations.
• Set cmake_minimum_required to 3.12 version.

We will develop these changes in branch build-dev-1.10 until ready for release. We are going to publish PRs in each step of the way and link them to this Issue. We encourage everyone interested to participate in this Issue discussion thread, and the PRs.

Steps to Reproduce

No response

Current behavior

No response

AWS CPP SDK version used

1.9

compiler and version used

any

Operating System and version

any

[sdavtaker]

First preparation PR: #1886

[kaihirota]

I'm trying to use DynamoDB Streams and Kinesis within Unreal Engine, and the process is extremely complex and painful. Of course, part of it is also due to UE's fault, but that's another story. Anyway, to use a third party library within Unreal Engine, I have to package the library as a Unreal Engine "plugin" that contains a "third party library" folder which includes compiled and source code for Windows, Mac, etc. Like many people, I could not figure out how to make this work. Fortunately, I was able to purchase a plugin for a fee from the UE marketplace, and the plugin's sole value proposition is enabling users to use the AWS SDK (but only dynamodb submodule) within UE.

While I'm sure this represents a niche and rare use case of AWS SDK, this is one area which would benefit largely from easier integration of AWS.

[Mike4Online]

I run into two difficulties with building and linking to the AWS SDK for C++:

Background

I need to build past and current versions of my software product, as well as current and past versions of the AWS SDK for C++ on which it depends. My build PC runs on older version of Ubuntu (18.04 Bionic) intentionally to ensure my customers can run my program on their older Linux systems.

Issue 1: CMake builds don't let you specify paths for all dependencies

So I cannot ever rely on system LIB and INCLUDE paths for transitive dependencies such as OpenSSL, zlib, libCURL, liblzma, libuuid, etc. These would all be too old in the Ubuntu 18 system directories.

Version 1.11 of the AWS SDK for C++ allows you to explicitly specify the path to the CURL library, i.e. by defining CURL_LIBRARY='/path/to/curl/libcurl.a'. But have not not found similar support for explicitly defining paths to libraries or headers for other dependencies (OPENSSL, ZLIB, LIBUUID, LIBLZMA, etc.). Adding such support would facilitate CMake builds of the AWS SDK for C++, when they need to depend on specific versions of dependencies. Until then, I have to rely on building with vcpkg, which somehow manages to keep the dependencies straight and contained within the vcpkg build tree.

Issue 2: AWS SDK for C++ header include files are spread across multiple subdirectories

I have built the AWS SDK for C++ libraries successfully multiple times, only to find that when I run the S3-Crt demo project I get SEGFAULTS. Turns out I had not copied all of the header include files from the CMake build tree to my own header include directory. So there was a mix of AWS SDK headers from a previous build as well as updated headers from the latest build.

While this is a novice mistake to make, the AWS SDK for C++ makes it easier to mess this up, specifically:

The AWS SDK for C++ often includes API changes between two tags, even when the major and minor version number have not changed. This should be highlighted in build documentation, as some libraries follow semantic versioning practices and will bump the minor version if there is an API change.

The CMake and vcpkg builds of the AWS SDK for C++ place header include files in several directories. vcpkg does a decent job of this:

cp -r ${VCPKG_ROOT}/installed/x64-linux/include/aws/*             /my_include_path/aws/aws/
cp -r ${VCPKG_ROOT}/installed/x64-linux/include/s2n/*             /my_include_path/aws/s2n/
cp -r ${VCPKG_ROOT}/installed/x64-linux/include/smithy/*          /my_include_path/aws/smithy/
cp    ${VCPKG_ROOT}/installed/x64-linux/include/z*                /my_include_path/aws/
cp    ${VCPKG_ROOT}/installed/x64-linux/include/s2n.h             /my_include_path/aws/

But the CMake build places headers is many far-flung directories. And these directories can change from tag to tag:

cp -R /CodeRepos/aws-sdk-cpp/aws-cpp-sdk-core/include/*                                  /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/src/aws-cpp-sdk-core/include/aws/*                          /my_include_path/aws/aws/
cp -R /CodeRepos/aws-sdk-cpp/src/aws-cpp-sdk-core/include/smithy/*                       /my_include_path/aws/smithy/
cp -R /CodeRepos/aws-sdk-cpp/aws-cpp-sdk-s3/include/*                                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/aws-cpp-sdk-s3-crt/include/*                                /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/include/*                                   /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-common/include/*                  /my_include_path/aws/
cp -R /CodeRepos/sdk_build_debug/crt/aws-crt-cpp/crt/aws-c-common/generated/include/*    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-auth/include/*                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-cal/include/*                     /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-compression/include/*             /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-event-stream/include/*            /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-checksums/include/*                 /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-http/include/*                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-io/include/*                      /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-mqtt/include/*                    /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-s3/include/*                      /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-sdkutils/include/*                /my_include_path/aws/
cp -R /CodeRepos/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-lc/include/*                        /my_include_path/aws/

Enhancing the CMake build so it prepares a consolidated set of header directories (similar to what vcpkg does) would be a handy way to ensure the entire set of headers could be easily found and copied elsewhere.

The S3 Crt demo project will log an API version mismatch error if it detects that the version in <aws/core/VersionConfig.h> does not correspond with the version within the linked or loaded AWS SDK libraries. If a similar check were to be added early in Aws::InitAPI() and made a runtime error, it might help catch this common mistake, rather allowing execution to continue to a SEGFAULT.

[Mike4Online]

The AWS SDK online documentation still shows that building the AWS SDK for C++ on Linux requires CMake - minimum version 3.13, maximum version 3.21.

Is this maximum CMake version still valid? Ideally the SDK could be built with the latest CMake.

[sbiscigl]

Is this maximum CMake version still valid? Ideally the SDK could be built with the latest CMake.

there is no maximum version and it will buid on latest, i routinely use latest. let me see if i can track down the documentation to update that wording to remove maximum.

[Mike4Online]

There is another documentation page stating that the CMake maximum version is 3.21. This one is for Windows:

Build the AWS SDK for C++ on Windows

[ahmedyarub]

Any updates?