The purpose of this guide is to provide steps to automate and troubleshoot Jenkins pipeline deployments. The steps here will focus on O3DE's use cases including CDK Pipeline deployments and local testing.
To get started, clone the o3de-jenkins-pipeline repo and review the setup instructions provided in the README.
Testing can be completed on a docker container running on your machine in addition to the sandbox instance.
The main workflow is to update the files in your workspace, create an image, and run the container.
Setup workspace:
Setup configs:
clientID
and clientSecret
. Remember: Do not commit secrets to the repo.Run container:
docker build -t jenkins .
Run container:
# Simple run command using named volume:
docker run -d -p 8080:8080 --mount type=volume,target=/var/jenkins_home,source=<volume-name> jenkins
# Example with AWS credentials
docker run -e "AWS_REGION=us-west-2" -e <AWS_ACCESS_KEY_ID> -e <AWS_SECRET_ACCESS_KEY> -d -p 8080:8080 --mount type=volume,target=/var/jenkins_home,source=<volume-name> jenkins
# Example with updated Java heap size. Useful when testing locally. Default is 8g.
docker run -e JAVA_OPTS=' -Xms1g -Xmx1g' -d -p 8080:8080 jenkins
The version numbers for plugin updates can be found on the update center page: $JENKINS_URL/pluginManager/
If a version shows up as Unavailable, this means it requires a newer version of Jenkins. This may appear even if the instance is running on the latest LTS version.
The latest available LTS version can be found here (https://www.jenkins.io/changelog-stable/). Also the associated docker tag can be found here (https://hub.docker.com/r/jenkins/jenkins/tag).
Note: Specify the version number when updating the tag. For example use 2.277.4-lts-jdk11
instead of lts-jdk11
)
This section contains details on how the Jenkins service starts up using the scripts defined in the docker container. This startup process is different from how it runs on a typical server install.
This base image is maintained by the Jenkins community and published to Docker hub. This is the image we use when creating our own custom docker container.
One important note about JENKINS_HOME is that it is captured as a Volume in the base image. This prevents us from editing the contents of the directory when creating downstream images. In order to add contents to JENKINS_HOME, files must be copied into the $REF directory (set to /usr/share/jenkins/ref). The Jenkins start up scripts are setup to copy the contents of this directory into JENKINS_HOME when the container starts.
To run Jenkins, the container is configured to using the following entrypoint. This is the command that the container will execute on start up.
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/jenkins.sh"]
Tini is a init utility for containers: https://github.com/krallin/tini
The jenkins.sh script performs these main actions:
Using the base image above, we create a dockerfile to generate our own Jenkins container image. This allows us to install our required plugins and load other custom options.
We utilize a custom entrypoint to override the original one mainly to clear out the contents of the plugins directory prior to startup. The default location of this directory is in JENKINS_HOME which is stored on a shared filesystem and there currently isn't an option to change this location. This results in stale plugin data that needs to be deleted or manually uninstalled from the Jenkins UI.
The custom entrypoint is configured to perform our required tasks then call the original entrypoint listed above.
After the start up scripts are executed and Jenkins starts, it also performs the following actions:
Configs and templates are stored in the /configs directory of the repo. Any .yaml or .yml files stored in this directory will be picked up by the JCasC plugin.
The configs of an existing Jenkins server can be exported to a YAML file that can be used to generate new config files.
If there are credentials in the configs, they are encrypted by the Jenkins server. To decrypt, go to the script console on the Jenkins server used to export the file and run the following command:
println(hudson.util.Secret.decrypt("{ABC}"))
Replace 'ABC' with the string listed in the exported config.
There are 3 main sections to the pipeline:
Click on the Details link on each section to view the related logs.
If a new task is failing to start, you can view the logs in the ECS service page. Click on the task ID then click on the Logs tab. The older task will not be stopped until the new one passes the health checks.
Error:
Some plugins failed to download! Not downloaded: <plugin_name>
Causes:
$JENKINS_URL/pluginManager/
otherwise retry the buildThe Jenkins home directory is stored in EFS. There are times when you may need to access the home directory to update configs in the event the server fails to start up. More info on the home directory can be found here: https://www.jenkins.io/doc/book/managing/system-configuration/
To access the home directory for each instance, spin up an EC2 instance in that AWS account and perform the following steps:
Remember to terminate the instance when completed.
When starting up Jenkins, the login page fails to load and the following error is displayed:
com.thoughtworks.xstream.mapper.CannotResolveClassException: org.jenkinsci.plugins.GithubSecurityRealm
[...]
Caused: java.io.IOException: Unable to read ${JENKINS_HOME}/config.xml
Cause: Plugins were updated as part of the latest deployment and there is an incompatibility with the current version of the Github Auth plugin and Matrix Auth plugin. This can happen when only one of the plugins are updated.
Fix:
<securityRealm class="hudson.security.HudsonPrivateSecurityRealm">