The current system using Jenkins and AWS S3 helped bootstrap our move for our 3rd Party libraries to open source, but much of the current workflow depends on:
Internally, we did this to thoroughly verify legacy packages for NDA'ed code. However, these components (specifically 2, 3, 5) are not visible to the public and cannot be directly contributed or packaged without having access to the internal systems.
We aim to make this public and the intake process transparent, as the community currently has no direct means to make contributions. In order to do this, we will need an end to end workflow that accomplishes the following:
$~$
We propose using a combination of Github Actions and Github Packages to achieve our goals.
First, an overview of each of these services:
What is it: A build pipeline service offered by Github, similar to Jenkins. They offer managed Windows, Ubuntu Linux, and Mac instances that come pre-loaded with build dependencies (such as msbuild/vc142, cmake, android SDK) for a variety of use cases, and can be loaded with other dependencies. It can also run builds on hardware hosted by the user through an agent.
How do you use it: Build scripts are stored in .github/workflows
and can be forked along with the repo to provide the same automation to the fork. It can pull scripts within its own repo and execute them as "stages". These can be triggered on various actions, such as merge or PR creation.
Cost: Github actions is free for public repos and any forks made from it. Usage limits are not explicitly defined, but it is currently in use by large projects without known issues. For private repos, cost is broken down by a "bucket" of minutes.
What does this replace: Jenkins and local build infrastructure
$~$
What is it: A package distribution service offered by Github. They offer storage for a variety of package systems, like nuget, npm, and docker (OCI) containers. This is backed by a managed CDN. Note: Github does not offer a service like S3, it must be a managed package format.
How do you use it: For this use case, we're looking to use a specification for OCI that allow arbitrary artifacts to be stored as containers, which come with hash generation, binary layering, and signing. These artifacts can be created as a tar.xz file with metadata (using www RFC 1341 content type headers), converted to container form and uploaded to Github Packages using pre-existing open source libraries/clients (specifically, "cosign")
Cost: Github Packages is free for public repos and any forks made from it. Usage limits are not explicitly defined, but it is currently in use by large projects without known issues. For private repos, cost is broken down by storage and aggregate transfer rates
What does this replace: AWS S3 buckets and Cloudfront CDN
With this in mind, some examples of the two main workflow components, packaging and building:
$~$
We will be using a client application called "cosign" a Linux Foundation project that signs and packages arbitrary artifacts in OCI artifact format. This will be used to login to Github Packages, and publish a container to the root of the Github user or organization profile (it's not explicitly tied to a repo until assigned)
We can do the following to login to Github Packages. This creates a authentication token, stored in the home drive
echo $GITHUB_ACCESS_TOKEN | cosign login https:http://ghcr.io/ -u $GITHUB_USERNAME --password-stdin
Then we upload a pre-built sample package openexr-3.1.3-rev4-windows, including the associated SHA256 hash and json file:
cosign upload http://ghcr.io/o3de/openexr-3.1.3-rev4-windows:1.0 ./openexr-3.1.3-rev4-windows.tar.xz
If we take a look at the uploaded package in Github Packages, it comes with all the associated metadata. The tar.xz
file is in the layers under the "application/octet-stream"
media type.
{
"digest": "sha256:03c26a2e95393ec57484c1b2beae6eb0ae569ed3b4c977ad2f59475315f91f56",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 402,
"config": {
"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
"mediaType": "application/vnd.unknown.config.v1+json",
"size": 2
},
"layers": [
{
"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
"mediaType": "application/octet-stream",
"size": 2
},
{
"digest": "sha256:c850268e849171751cdaefdab1952333ac38afbb771b999e99d67f9761706d98",
"mediaType": "application/octet-stream",
"size": 4838876
}
]
}
If we want to pull the package, we can take the root digest hash of the generated container and curl the package (the bearer token is a generic credential used for all clients)
$~$
## Github Actions building example
Using the cosign application demonstrated, we will use an Ubuntu environment with some dependencies, run "[pull\_and\_build\_from\_git.py](https://github.com/amzn-changml/3p-package-source/blob/main/Scripts/extras/pull_and_build_from_git.py)" for a package, then upload the resulting .tar.xz file to Github packages
name: Package-CI-Build
on: push:
branches:
- main
- stabilization-*
permissions: read-all
jobs:
build:
name: build
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b # v2.4.0
- uses: sigstore/cosign-installer@7e0881f8fe90b25e305bbf0309761e9314607e25 # v2.3.0
- uses: actions/setup-go@b22fbbc2921299758641fab08929b4ac52b32923 # v2.2.0 with: go-version: '1.17' check-latest: true
- run: |
python3 ${GITHUB_WORKSPACE}/Scripts/extras/pull_and_build_from_git.py $GITHUB_WORKSPACE/package-system/$PACKAGE_NAME
echo $GITHUB_ACCESS_TOKEN | cosign login https://ghcr.io -u $GITHUB_USERNAME --password-stdin
cosign upload ghcr.io/o3de/$PACKAGE_NAME:$VERSION ./$PACKAGE_NAME.tar.xz ```
$~$
[build_packages.py](https://github.com/o3de/3p-package-scripts/blob/main/o3de_package_scripts/build_package.py)
script but can be agnostic), which will consume a primary metadata file for each platformpackage_build_host_list_<platform>.json
file), which will define custom steps specific to each packagePackageInfo.json
, a metadata file containing package version and licenses, but one can be generated as part of the buildLY_PACKAGE_SERVER_URLS
environment variable$~$
PackageInfo.json
file for completeness and the license indicated in the file is "blessed" by the project.$~$
**cmake/3rdParty/Platform/<platform>/BuiltInPackages_<platform>.cmake**
file for the staged package**BuiltInPackages_<platform>.cmake**
in the development branch and "promoting" the 3p package to production$~$
$~$
$~$
$~$
The advantages to this approach:
$~$
There's some cons however:
How do we prevent supply chain attacks?
tar.xz
file's hash is part of the metadata list file (example: https://github.com/o3de/o3de/blob/development/cmake/3rdParty/Platform/Windows/BuiltInPackages_windows.cmake), which will not match if modified. The 3P downloader will not extract and utilize a package if there is a hash mismatch.How do we prevent package contributions with licenses incompatible with the O3DE project?
build_config.json
) should be included in each package to be built, which will conatin the SPDX license shortname. An additonal automated scan will be performed during the build process to look for license files and will alert the maintainers if a mismatch is found.