-
Notifications
You must be signed in to change notification settings - Fork 273
Building an OpenSearch and OpenSearch Dashboards Distribution
The distribution workflow builds a complete OpenSearch and OpenSearch Dashboards distribution from source. This system performs a top-down build of all components required for a specific OpenSearch and OpenSearch Dashboards release, then assembles a distribution. The input to the system is a manifest that defines the order in which components should be built. All manifests for our current releases are placed in manifests folder. For older versions, please see legacy-manifests folder.
Before we start to build, let's understand what is manifest and the types of manifest that you will come across. Manifests are the single source of truth for building and testing the distributions. There are 3 types of manifest, each with a difference schema.
-
Input Manifest : As the name suggests, input manifest is the starting point and source of building OpenSearch and OpenSearch Dashboard distributions. See the schema of the input manifest. This manifest consist of CI docker image that contains all the dependencies to build an OpenSearch and OpenSearch Dashboard distribution. It also contains the components' GitHub repository URLs, reference to build, basic CI checks as well as platforms the components supports.
-
Build Manifest : Build Manifest is the interim manifest that is generated after components are built. See the schema for build manifest. You can find those manifests in
builds
folder locally. This manifest contains interim artifacts such as zips, libraries, maven artifacts, etc. See sample build manifest. -
Distribution Manifest : Distribution manifest, as the name suggests is the final manifest bundles with the distribution. See the schema for the distribution manifest. This manifest contains all the information about a distribution and the components bundled in that distribution.
Input manifests are the source of building the distributions. Each input manifest also contains the docker image that can be used to build these distributions and avoid installing bunch of packages. See CI image details in the manifest.
To run locally:
docker run -it -d --entrypoint bash opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v2 -e JAVA_HOME=/opt/java/openjdk-20
Then build from source:
./build.sh manifests/3.0.0/opensearch-3.0.0.yml
Additional arguments:
optional arguments:
-h, --help show this help message and exit
-l, --lock Generate a stable reference manifest.
-s, --snapshot Build snapshot.
-c [COMPONENTS ...], --component [COMPONENTS ...]
Rebuild one or more components.
--keep Do not delete the working temporary directory.
-p {linux,darwin,windows}, --platform {linux,darwin,windows}
Platform to build.
-a {x64,arm64}, --architecture {x64,arm64}
Architecture to build.
-v, --verbose Show more verbose output.
-d {tar,zip,rpm,deb}, --distribution {tar,zip,rpm,deb}
Distribution to build.
This builds OpenSearch 3.0.0 from source, placing the output into ./builds/opensearch
.
See build workflow for more information.
Assembling a distribution needs build manifest as an input.
./assemble.sh builds/opensearch/manifest.yml
The assembling step takes output from the build step, installs plugins, and assembles a full distribition into the dist
folder.
See assemble workflow for more information.
A patch release contains output from previous versions mixed with new source code. Manifests can mix such references. See opensearch-1.3.1.yml for an example.
OpenSearch is often released with changes in opensearch-min
, and no changes to plugins other than a version bump. This can be performed by a solo Engineer following a cookbook. See also opensearch-build#1375 which aims to automate incrementing versions for the next development iteration.
We use Jenkins as our CI/CD infrastructure to build, test and release OpenSearch and OpenSearch Dashboards. Access it here
We build, assemble, and test our artifacts on docker containers. We provide docker files in docker/ci folder, and images on staging docker hub repositories. All Jenkins pipelines can be found in jenkins. The Jenkins is deployed using infrastructure as a Code and can be found in opensearch-ci repository.
See jenkins and docker for more information.
The distribution url and the build output manifest include a Jenkins auto-incremented build number. For example, the manifest from OpenSearch build 5905 contains the following.
build:
name: OpenSearch
version: 2.2.0
platform: linux
architecture: x64
distribution: rpm
id: '5905'
Below are the details where you can find the nighly artifacts build for OpenSearch and OpenSearch-Dashboards distribution. Please note that since these nightly artifacts, they are not recommended for production use.
Snapshots for OpenSearch core/min can be downloaded and used in CI's, local development, etc using below links:
Linux:
https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-linux-x64-latest.tar.gz
Macos:
https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-darwin-x64-latest.tar.gz
Windows:
https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-windows-x64-latest.zip
Use the latest
keyword in the URL to obtain the latest nightly build for a given version. For example https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.12.0/latest/linux/x64/rpm/dist/opensearch/manifest.yml redirects to build 9234 at the time of writing this.
The latest
keyword is resolved to a specific build number by checking an index.json
file. This file is maintained individually for each platform, architecture and distribution. The index.json file has contents similar to
{"latest":"9445"}
The resolution logic is implemented in the CloudFront URL rewriter.
The TTL (time to live) is set to 5 mins
which means that the latest
url may need up to 5 mins to get new contents after index.json
is updated.
All the artifacts accessible through the regular distribution URL can be accessed by the latest
URL. This includes both OpenSearch Core, OpenSearch Dashboards Core and their plugins.
For example, you can download the latest .tar.gz distribution build of OpenSearch 2.12.0 directly at https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.12.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.12.0-linux-x64.tar.gz, without having to first download and parse the complete build manifest.
For plugin artifacts, you can also use the latest
keyword to get the latest plugin artifacts with a known version. E.g. in order to get performance-analyzer x64 tarball artifacts for 2.1.0, you can obtain it with link https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip, which will redirect you to https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/5757/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip.
For bundled artifacts, here are some examples for LINUX and Windows:
- Linux Tar: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.4.0-linux-x64.tar.gz
- Windows Zip: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/windows/x64/zip/dist/opensearch/opensearch-2.4.0-windows-x64.zip
The nightly build docker images are published to opensearchstaging account with version tag.
See OpenSearch and OpenSearch-Dashboards images.
Anything unclear or inaccurate in our Wiki? You can contribute a change here.