[ci skip] Update the continuous benchmarking scripts (#3597)

This is basically what we use on Citrine right now.  We've gotten through
two complete runs in a row over the past week or so without any big
changes to these scripts, so I consider them stable/working.

We now use systemd instead of upstart.  The systemd script can be
installed at /lib/systemd/system/tfb.service.  The bash scripts can be
anywhere as long as tfb.service is updated accordingly.

toolset/continuous/README.md

@@ -1,12 +0,0 @@
-# TFB Continuous Benchmarking
-
-Performs continuous benchmarking of all frameworks and uploads the results to `TFB_UPLOADURI` found in `tfb.conf`.
-
-### Setup
-
-* Move the `tfb.conf` file to `/etc/init`
-* `sudo service tfb start`
-
-### Notes
-
-If `run-continuously.sh` is modified the tfb service will need to be restarted. Alternatively, any new or modified tasks in `run-tasks.sh` will now be updated automatically for each continuous run.

toolset/continuous/run-continuously.sh

@@ -1,45 +0,0 @@
-#!/bin/bash
-#
-# Sets up and runs the benchmarking suite in an infinite loop.  The intent is 
-# this script would be run by a service such as upstart, restarting if the 
-# scripts fail.
-#
-# The following environment variables must
-# be set:
-# $TFB_REPOPARENT   parent folder of the repository
-# $TFB_REPONAME     name of the repository folder
-# $TFB_REPOURI      URI for the git repository
-# $TFB_MAILINGLIST  email address for the mailing list
-# $TFB_LOGSFOLDER   location to transfer logged output
-#
-# @author A. Shawn Bandy
-#
-
-while true
-do
-  # This file is copied outside the repo and run by tfb service
-  # so that it may tear down the environment and run without interruption.
-  # If this file is changed, the tfb service will need to be restarted.
-
-  # Now if tasks are updated in the continuous toolset, they will execute
-  # properly on the next full run.
-
-  # Start by tearing down the environment.
-  # Hold on to the benchmark.cfg it it exists
-  echo Tearing down and rebuilding the environment
-  cp $TFB_REPOPARENT/$TFB_REPONAME/benchmark.cfg $TFB_REPOPARENT/benchmark.cfg 2>/dev/null
-  sudo rm -rf $TFB_REPOPARENT/$TFB_REPONAME
-  cd $TFB_REPOPARENT
-  git clone -b $TFB_REPOBRANCH $TFB_REPOURI $TFB_REPOPARENT/$TFB_REPONAME
-  # Replace the benchmark.cfg
-  cp $TFB_REPOPARENT/benchmark.cfg $TFB_REPOPARENT/$TFB_REPONAME/benchmark.cfg 2>/dev/null
-  # Set the hosts correctly in the benchmark.cfg
-  sed -i 's|server_host=.*|server_host='"$TFB_SERVER"'|g' $TFB_REPOPARENT/$TFB_REPONAME/benchmark.cfg 2>/dev/null
-  sed -i 's|database_host=.*|database_host='"$TFB_DATABASE"'|g' $TFB_REPOPARENT/$TFB_REPONAME/benchmark.cfg 2>/dev/null
-  sed -i 's|client_host=.*|client_host='"$TFB_CLIENT"'|g' $TFB_REPOPARENT/$TFB_REPONAME/benchmark.cfg 2>/dev/null
-
-  echo Running continuous tasks
-  $TFB_REPOPARENT/$TFB_REPONAME/toolset/continuous/tasks/run-tasks.sh
-
-  sleep 5
-done

toolset/continuous/tasks/keep-logs.py

@@ -1,44 +0,0 @@
-#
-# Archives, to the specified folder, the logged output generated by a benchmark
-# run.
-#
-# @author A. Shawn Bandy
-import os
-import zipfile
-import datetime
-import requests
-# Follows closely from:
-# http://stackoverflow.com/a/34153816
-#
-# Paths to the log folders are generated by TFB and where those files
-# should be archived.
-#
-path_in = os.path.abspath(os.path.normpath(os.path.expanduser(os.path.join( \
-    os.environ['TFB_REPOPARENT'], os.environ['TFB_REPONAME'], \
-    'results'))))
-date_time = datetime.datetime.now()
-dt_folder = date_time.strftime('%Y%m%d%H%M%S')
-path_out = os.path.abspath(os.path.join(os.environ['TFB_LOGSFOLDER'], \
-    dt_folder))
-
-if not os.path.exists(path_out):
-    os.makedirs(path_out)
-
-zip_path = path_out + '/results.zip'
-
-zip_file = zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED)
-
-for root, dirs, files in os.walk(path_in):
-    for file in files:
-        zip_file.write(os.path.join(root, file))
-
-zip_file.close()
-
-results_upload_uri = os.environ['TFB_UPLOADURI']
-
-if results_upload_uri != None:
-    with open(zip_path, 'rb') as file_to_upload:
-        requests.post(
-            results_upload_uri,
-            headers={'Content-Type': 'application/zip'},
-            data=file_to_upload)

toolset/continuous/tasks/record-git-commit.sh

@@ -1,10 +0,0 @@
-#!/bin/bash
-#
-# save the current git commit to results/.commit
-#
-echo "Saving current git commit to results directory..."
-GIT_COMMIT=$(git -C $TFB_REPOPARENT/$TFB_REPONAME rev-parse HEAD)
-mkdir -p $TFB_REPOPARENT/$TFB_REPONAME/results
-echo $GIT_COMMIT > $TFB_REPOPARENT/$TFB_REPONAME/results/commit_id.txt
-echo "Using commit: " $GIT_COMMIT
-

toolset/continuous/tasks/run-benchmarks.sh

@@ -1,10 +0,0 @@
-#!/bin/bash
-#
-# Start the benchmarks
-
-echo Change to benchmark root
-cd $TFB_REPOPARENT/$TFB_REPONAME
-echo Cleaning
-FWROOT=$TFB_REPOPARENT/$TFB_REPONAME PYTHONPATH=$TFB_REPOPARENT/$TFB_REPONAME python toolset/run-tests.py --clean
-echo Running tests
-FWROOT=$TFB_REPOPARENT/$TFB_REPONAME PYTHONPATH=$TFB_REPOPARENT/$TFB_REPONAME python toolset/run-tests.py

toolset/continuous/tasks/run-tasks.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-#
-# A list of tasks to be run for continuous benchmarking
-#
-
-##########################
-#  Tasks before the run  #
-##########################
-
-$TFB_REPOPARENT/$TFB_REPONAME/toolset/continuous/tasks/record-git-commit.sh
-
-
-##########################
-#   Run the benchmarks   #
-##########################
-
-$TFB_REPOPARENT/$TFB_REPONAME/toolset/continuous/tasks/run-benchmarks.sh
-
-##########################
-#  Tasks after the run   #
-##########################
-
-FWROOT=$TFB_REPOPARENT/$TFB_REPONAME PYTHONPATH=$TFB_REPOPARENT/$TFB_REPONAME python $TFB_REPOPARENT/$TFB_REPONAME/toolset/continuous/tasks/keep-logs.py

toolset/continuous/tfb-shutdown.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+set -e
+
+sudo service docker restart
+docker ps -aq | xargs --no-run-if-empty docker stop
+docker ps -aq | xargs --no-run-if-empty docker rm -f
+ssh techempower@$TFB_DATABASE_HOST << \EOF
+  sudo service docker restart
+  docker ps -aq | xargs --no-run-if-empty docker stop
+  docker ps -aq | xargs --no-run-if-empty docker rm -f
+EOF
+ssh techempower@$TFB_CLIENT_HOST << \EOF
+  sudo service docker restart
+  docker ps -aq | xargs --no-run-if-empty docker stop
+  docker ps -aq | xargs --no-run-if-empty docker rm -f
+EOF

toolset/continuous/tfb-startup.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+set -e
+
+while true
+do
+  cd $TFB_REPORTPARENT
+  ./tfb-shutdown.sh
+
+  if [ -d "$TFB_REPOPARENT/$TFB_REPONAME" ]; then
+    sudo rm -rf $TFB_REPOPARENT/$TFB_REPONAME
+  fi
+
+  git clone \
+    -b $TFB_REPOBRANCH \
+    $TFB_REPOURI \
+    $TFB_REPOPARENT/$TFB_REPONAME \
+    --depth 1
+
+  cd $TFB_REPOPARENT/$TFB_REPONAME
+
+  docker run \
+    --network=host \
+    --mount type=bind,source=$TFB_REPOPARENT/$TFB_REPONAME,target=/FrameworkBenchmarks \
+    techempower/tfb \
+    --server-host $TFB_SERVER_HOST \
+    --client-host $TFB_CLIENT_HOST \
+    --database-host $TFB_DATABASE_HOST \
+    --network-mode host \
+    --results-name "$TFB_RUN_NAME" \
+    --results-environment "$TFB_ENVIRONMENT" \
+    --results-upload-uri "$TFB_UPLOAD_URI" \
+    --quiet
+
+  zip -r results.zip results
+
+  curl \
+    -i -v \
+    -X POST \
+    --header "Content-Type: application/zip" \
+    --data-binary @results.zip \
+    $TFB_UPLOAD_URI
+
+  sleep 5
+done

toolset/continuous/tfb.conf

@@ -1,29 +0,0 @@
-# /etc/init/tfb.conf
-env TFB_REPOPARENT="/home/techempower"
-env TFB_CONTINUOUSFOLDER="tfb-continuous"
-env TFB_REPONAME="FrameworkBenchmarks"
-env TFB_REPOURI="https://github.com/TechEmpower/FrameworkBenchmarks.git"
-env TFB_LOGSFOLDER="/home/techempower/logs"
-env TFB_REPOBRANCH="master"
-env TFB_UPLOADURI="http://user:pass@example/upload"
-env HOME="/home/techempower"
-env LANG="en_US.UTF-8"
-setuid techempower
-setgid techempower
-umask 0002
-respawn
-respawn limit 5 2
-#console none
-
-script
-  cd $TFB_REPOPARENT
-  # Clone the repo if it doesn't exist
-  if [ ! -d "$TFB_REPOPARENT/$TFB_REPONAME" ]; then
-    git clone -b $TFB_REPOBRANCH $TFB_REPOURI $TFB_REPOPARENT/$TFB_REPONAME
-  fi
-
-  # Separate the entry point for the continuous run from the repo
-  mkdir -p $TFB_CONTINUOUSFOLDER
-  cp $TFB_REPOPARENT/$TFB_REPONAME/toolset/continuous/run-continuously.sh $TFB_REPOPARENT/$TFB_CONTINUOUSFOLDER
-  exec /$TFB_REPOPARENT/$TFB_CONTINUOUSFOLDER/run-continuously.sh
-end script

toolset/continuous/tfb.service

@@ -0,0 +1,25 @@
+[Service]
+Environment=TFB_SERVER_HOST=127.0.0.1
+Environment=TFB_CLIENT_HOST=127.0.0.1
+Environment=TFB_DATABASE_HOST=127.0.0.1
+Environment=TFB_REPOPARENT=/home/techempower
+Environment=TFB_REPONAME=FrameworkBenchmarks
+Environment=TFB_REPOURI=https://github.com/TechEmpower/FrameworkBenchmarks.git
+Environment=TFB_REPOBRANCH=master
+Environment=TFB_UPLOAD_URI=http://localhost/changeme
+Environment=TFB_ENVIRONMENT=ChangeMe
+Environment="TFB_RUN_NAME=Continuous Benchmarking Run %%Y-%%m-%%d %%H:%%M:%%S"
+Environment=HOME=/home/techempower
+Environment=LANG=en_US.UTF-8
+User=techempower
+Group=techempower
+WorkingDirectory=/home/techempower
+LimitNOFILE=1048576
+LimitNPROC=1048576
+UMask=0002
+ExecStart=/home/techempower/tfb-startup.sh
+ExecStop=/home/techempower/tfb-shutdown.sh
+ExecStopPost=/home/techempower/tfb-shutdown.sh
+
+[Install]
+WantedBy=multi-user.target