Merge pull request #1051 from hamiltont/vagrant-setup

Toolset: Setup Vagrant

README.md

@@ -19,25 +19,15 @@ or chat with us on [Freenode](https://freenode.net/faq.shtml#whatwhy) at `#teche
 
 # How do I contribute updates or new frameworks?
 
-If you plan to add a new framework or update an existing framework, you only 
-need to run in *verify* mode. This will launch the framework, query all URLs, and verify that 
-returned data matches the [benchmark requirements](http://www.techempower.com/benchmarks/#section=code). 
-
-If *verify* mode is all you need, you can potentially develop without ever running our 
-codebase locally - our [Travis-CI.org](https://travis-ci.org/TechEmpower/FrameworkBenchmarks) setup 
-can run the verification for you. This is as simple as going to travis-ci.org, using the 
-`Log in with Github` button, and enabling Travis-CI for your fork. All commit pushes 
-will be automatically verified by Travis-CI, and you will get full log output. 
-You can submit a pull request when your code passes the Travis-CI verification and have 
-confidence it will be merged quickly. While this development route is slightly slower than 
-standard local development, it does not require any local setup process.
-
-You can also run *verify* mode on a single computer, although you should be comfortable with us 
-installing multiple large software suites. 
-You will need to enable passwordless SSH access to localhost ([search Google for help](https://www.google.com/#hl=en&q=passwordless%20SSH%20access), yielding references such as these: [article 1](http://hortonworks.com/kb/generating-ssh-keys-for-passwordless-login/) [article 2](http://superuser.com/questions/336226/how-to-ssh-to-localhost-without-password) [article 3](https://help.ubuntu.com/community/SSH/OpenSSH/Keys)), and you will also 
-need to enable passwordless sudo access ([Google for help](https://www.google.com/#hl=en&q=passwordless%20sudo)).
-Once you have cloned our repository, run `toolset/run-tests.py --help` for detailed 
-help on running in *verify* mode and see the sections below for more guidance. 
+[Setup a development environment](deployment), typically by using a 
+remote server dedicated to this project or by using a virtual machine. 
+Then enable Travis-CI on your project fork, so that any commits you send
+to Github are automatically verified for correctness (e.g. meeting all 
+[benchmark requirements](http://www.techempower.com/benchmarks/#section=code)). 
+This is as simple as going to travis-ci.org, using the 
+`Log in with Github` button, and enabling Travis-CI for your fork.
+When your development is done and your changes pass the Travis-CI verification,
+submit a pull request with confidence that it can be merged quickly. 
 
 # How do I run the benchmark myself? 
 
@@ -47,15 +37,18 @@ connection in between, all of which you should be comfortable installing a large
 of additional software on. One of these computers (the `app server`) must have passwordless
 SSH access to the other two ([search Google for help](https://www.google.com/#hl=en&q=passwordless%20SSH%20access), yielding references such as these: [article 1](http://hortonworks.com/kb/generating-ssh-keys-for-passwordless-login/) [article 2](http://superuser.com/questions/336226/how-to-ssh-to-localhost-without-password) [article 3](https://help.ubuntu.com/community/SSH/OpenSSH/Keys)), and on every computer 
 you will need to have passwordless sudo access ([Google for help](https://www.google.com/#hl=en&q=passwordless%20sudo)).
-Once you have cloned our repository, run `toolset/run-tests.py --help` for detailedhelp
-on running in *benchmark* mode and see the sections below for more guidance. 
+Once you have cloned our repository, run `toolset/run-tests.py --help` for detailed help
+on running in *benchmark* mode. 
+
+If you wish to benchmark on Amazon EC2, see [our scripts](deployment/vagrant-production) for 
+launching a benchmark-ready Amazon environment. 
 
 If you are not an expert, please ensure your setup can run in *verify* mode before 
 attempting to run in *benchmark* mode. 
 
 # Project Overview 
 
-Running the full benchmark requires at least three computers:
+The full benchmark requires at least three computers:
 
 * `app server`: The computer that your framework will be launched on.
 * `load server`: The computer that will generate client load. Also known as the `client machine`.
@@ -76,19 +69,6 @@ When run, `TFB` will:
 * gather the results
 * halt the framework
 
-# Installation and Usage Details
-
-If you choose to run TFB on your own computer, you will need to install 
-passwordless SSH to your `load server` and your `database server` from 
-your `app server`. You will also need to enable passwordless sudo access
-on all three servers. If you are only planning to use *verify* mode, then
-all three servers can be the same computer, and you will need to ensure
-you have passwordless sudo access to `localhost`. 
-
-For all Linux framework tests, we use [Ubuntu 14.04](http://www.ubuntu.com/download/server), so 
-it is recommended you use this for development or use.  Furthermore, the load server is Linux-only,
-even when testing Windows frameworks.
-
 ## Configuration File Usage
 
 TFB takes a large number of command-line arguments, and it can become tedious to 
@@ -107,6 +87,8 @@ After you have a configuration file, run the following to setup your
 various servers. We use the `--install-only` flag in our examples to 
 prevent launching tests at this stage. 
 
+See [here](deployment) for additional details. 
+
 **Setting up the `load server`**
 
 ```
@@ -354,7 +336,9 @@ The steps involved are:
 permutation to your `benchmark_config` file for the Windows test.  When 
 the benchmark script runs on Linux, it skips tests where `os` in `Windows`
 and vice versa. 
-* Add the necessary tweaks to your [setup file](#setup-files) to start and stop on the new operating system.  See, for example, [the script for Go](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/go/setup.py).
+* Add the necessary tweaks to your [setup file](#setup-files) to start and 
+stop on the new operating system.  See, for example, 
+[the script for Go](frameworks/Go/go/setup.py).
 * Test on Windows and Linux to make sure everything works as expected.
 
 ### Install File
@@ -451,17 +435,25 @@ Do not cause any output, such as using `echo`, inside of `bash_profile.sh`
 The setup file is responsible for starting and stopping the test. This script is responsible for (among other things):
 
 * Modifying the framework's configuration to point to the correct database host
-* Compiling and/or packaging the code
+* Compiling and/or packaging the code (if impossible to do in `install.sh`)
 * Starting the server
 * Stopping the server
 
-The setup file is a python script that contains a start() and a stop() function.  The start function should build the source, make any necessary changes to the framework's configuration, and then start the server.  The stop function should shutdown the server, including all sub-processes as applicable.
+The setup file is a python script that contains a start() and a stop() function.  
+The start function should build the source, make any necessary changes to the framework's 
+configuration, and then start the server. The stop function should shutdown the server, 
+including all sub-processes as applicable.
 
 #### Configuring database connectivity in start()
 
-By convention, the configuration files used by a framework should specify the database server as `localhost` so that developing tests in a single-machine environment can be done in an ad hoc fashion, without using the benchmark scripts.
+By convention, the configuration files used by a framework should specify the database 
+server as `localhost` so that developing tests in a single-machine environment can be 
+done in an ad hoc fashion, without using the benchmark scripts.
 
-When running a benchmark script, the script needs to modify each framework's configuration so that the framework connects to a database host provided as a command line argument.  In order to do this, use setup_util.replace_text() to make necessary modifications prior to starting the server.
+When running a benchmark script, the script needs to modify each framework's configuration
+so that the framework connects to a database host provided as a command line argument. 
+In order to do this, use `setup_util.replace_text()` to make modifications prior to 
+starting the server.
 
 For example:
 
@@ -469,7 +461,12 @@ For example:
 setup_util.replace_text("wicket/src/main/webapp/WEB-INF/resin-web.xml", "mysql:\/\/.*:3306", "mysql://" + args.database_host + ":3306")
 ```
 
-Using `localhost` in the raw configuration file is not a requirement as long as the `replace_text` call properly injects the database host provided to the benchmarker toolset as a command line argument.
+Note: `args` contains a number of useful items, such as `troot`, `iroot`, `fwroot` (comparable
+to their bash counterparts in `install.sh`, `database_host`, `client_host`, and many others)
+
+Note: Using `localhost` in the raw configuration file is not a requirement as long as the
+`replace_text` call properly injects the database host provided to the benchmark 
+toolset as a command line argument.
 
 #### A full example
 
@@ -489,7 +486,7 @@ import setup_util
 def start(args, logfile, errfile):
 
 # setting the database url
-setup_util.replace_text("wicket/src/main/webapp/WEB-INF/resin-web.xml", "mysql:\/\/.*:3306", "mysql://" + args.database_host + ":3306")
+setup_util.replace_text(args.troot + "/src/main/webapp/WEB-INF/resin-web.xml", "mysql:\/\/.*:3306", "mysql://" + args.database_host + ":3306")
 
 # 1. Compile and package
 # 2. Clean out possible old tests
@@ -498,7 +495,7 @@ setup_util.replace_text("wicket/src/main/webapp/WEB-INF/resin-web.xml", "mysql:\
 try:
   subprocess.check_call("mvn clean compile war:war", shell=True, cwd="wicket", stderr=errfile, stdout=logfile)
   subprocess.check_call("rm -rf $RESIN_HOME/webapps/*", shell=True, stderr=errfile, stdout=logfile)
-  subprocess.check_call("cp wicket/target/hellowicket-1.0-SNAPSHOT.war $RESIN_HOME/webapps/wicket.war", shell=True, stderr=errfile, stdout=logfile)
+  subprocess.check_call("cp $TROOT/target/hellowicket-1.0-SNAPSHOT.war $RESIN_HOME/webapps/wicket.war", shell=True, stderr=errfile, stdout=logfile)
   subprocess.check_call("$RESIN_HOME/bin/resinctl start", shell=True, stderr=errfile, stdout=logfile)
   return 0
 except subprocess.CalledProcessError:
@@ -521,40 +518,3 @@ except subprocess.CalledProcessError:
 #### A tool to generate your setup file
  
 A contributor named [@kpacha](https://github.com/kpacha) has built a pure JavaScript tool for generating the `setup.py` file for a new framework via an in-browser form.  Check out his [FrameworkBenchmarks Setup Builder](http://kpacha.github.io/FrameworkBenchmarks-setup-builder/).
-
----
-
-## Windows server setup
-
-* Connect to the Windows server via Remote Desktop.
-* Copy `installer-bootstrap.ps1` from "toolset/setup/windows" to the server (use CTRL-C and CTRL-V).
-* Copy your Linux client private key too.
-* Right click on the installer script and select `Run with PowerShell`.
-* Press Enter to confirm.
-* It will install git and then launch `installer.ps1` from the repository, which will install everything else.
-* The installation takes about 20 minutes.
-* Then you have a working console: try `python`, `git`, `ssh`, `curl`, `node` etc. and verify that everything works + PowerShell goodies.
-
-The client/database machine is still assumed to be a Linux box. You can install just the client software via
-
-    python toolset\run-tests.py -s server-private-ip -c client-private-ip -i "C:\Users\Administrator\Desktop\client.key" --install-software --install client --list-tests
-
-but this step is not required if you already installed the Linux server and client as described above.
-
-Now you can run tests:
-
-    python toolset\run-tests.py -s server-private-ip -c client-private-ip -i "C:\Users\Administrator\Desktop\client.key" --max-threads 2 --duration 30 --sleep 5 --name win --test aspnet --type all
-
----
-
-## SQL Server setup
-
-* Connect to the SQL Server host via Remote Desktop.
-* Run a `Command Prompt` as Administrator.
-* Enter this command:
-
-        powershell -ExecutionPolicy Bypass -Command "iex (New-Object Net.WebClient).DownloadString('https://raw.github.com/TechEmpower/FrameworkBenchmarks/master/toolset/setup/sqlserver/setup-sqlserver-bootstrap.ps1')"
-
-* This will configure SQL Server, the Windows Firewall, and populate the database.
-
-Now, when running `run-tests.py`, just add `-d <ip of SQL Server instance>`. This works for the (Windows Server-based) `aspnet-sqlserver-raw` and `aspnet-sqlserver-entityframework` tests.

config/10gen.list

@@ -1 +0,0 @@
-deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

config/database_sftp_batch

@@ -3,7 +3,6 @@ put config/my.cnf
 put config/mongodb.conf
 put config/create.sql
 put config/create.js
-put config/10gen.list
 put config/create-postgres-database.sql
 put config/create-postgres.sql
 put config/postgresql.conf

config/python_requirements.txt

@@ -1 +1,4 @@
-colorama
+colorama==0.3.1
+# Version 2.3 has a nice Counter() and other features
+# but it requires —-allow-external and -—allow-unverified
+progressbar==2.2

deployment/README.md

@@ -0,0 +1,140 @@
+# Benchmark Suite Deployment
+
+This directory contains information for deploying the benchmark 
+on a range of cloud or self-hosted environments. While you can always use 
+*manual deployment*, automated scripts exist for many scenarios. 
+
+Before forging ahead, you should determine if your primary interest is
+**development** or **benchmarking**. Once you know this, look at the 
+'Summary of Script Directories' section to figure out which directory
+has scripts relevant to your use case. 
+
+If your primary interest is **development**, e.g. adding or updating frameworks,
+you can use an environment with a single computer running the database, 
+the load generation system, and the web framework. We provide scripts for 
+configuring a Linux development environment using either Virtualbox or Amazon EC2,
+and are actively searching for help adding Windows.
+
+If your primary interest is **benchmarking**, then you will need three computers
+(or virtual machines) networked together to fill the three benchmark roles of 
+database server, load generation server, and web framework server. We provide 
+a Vagrant script to setup a full benchmark-ready Linux environment in Amazon EC2.
+
+If you wish to benchmark a framework on a non-Amazon cloud, e.g. Azure or 
+Rackspace, or a framework that requires Windows OS or SQL server, there are 
+some limited helper scripts available at the moment. We welcome any pull requests 
+along these lines. 
+
+**Please note**: Running this software will make many modifications to software
+and settings, so it's recommended you either use a VM or hardware dedicated to this project. 
+
+## Required Host Roles
+
+**Client (aka load generation) Server**
+
+* Runs the benchmark client
+* Operating system: Ubuntu Server 14.04 LTS 64-bit.
+
+**Database server**
+
+* Hosts databases (e.g. `MySQL`, `PostgreSQL`, `MongoDB`, `SQL Server`).
+* Operating system: `Ubuntu Server 14.04 LTS 64-bit` or `Windows Server 2012 Datacenter 64-bit`
+
+**App (aka web framework) Server**
+
+* Hosts the web frameworks under test
+* Operating system: `Windows Server 2012 Datacenter 64-bit` or `Ubuntu Server 14.04 LTS 64-bit`
+
+## Summary of Script Directories
+
+
+| Directory Name | Summary |
+| -------------- | :-------- |
+| [azure](azure)  | Scripts to help deploy onto Microsoft Azure
+| [common](common) | Common-use scripts that could be used in many environments
+| [vagrant-common](vagrant-common) | The core directory for vagrant-based setups. Currently only supports deplconfiguring Linux-based environments
+| [vagrant-aws](vagrant-aws) | Setup scripts for configuring Amazon such that you can use `vagrant-development` and `vagrant-production`. Also outlines Amazon-specific options, such as instance type, you can use with `vagrant-development` and `vagrant-production`
+| [vagrant-virtualbox](vagrant-virtualbox) | A readme defining all the Virtualbox-specific options you can use with `vagrant-development` and `vagrant-production`, such as amount of RAM per virtual machine
+| [vagrant-development](vagrant-development) | Sets up a development environment using a single Virtual Machine. Can use either Virtualbox (for a free, locally run virtual machine) or Amazon EC2 (for a `$1/day` remote virtual machine)
+| [vagrant-production](vagrant-production) | Sets up a 3-virtual machine environment in either Amazon or Virtualbox. If Amazon is used, this is intended to be an environment you could run an official `TechEmpower` round with. A 3-VM VirtualBox setup is mainly used to test-run in a free environment before launching into a paid environment
+
+
+## Manual Deployment
+
+
+**Evaluation downloads**
+
+If you want to run the benchmark on Windows and/or SQL Server, you can download evaluation versions from Microsoft:
+* [Download Windows Server 2012](http://technet.microsoft.com/en-us/evalcenter/hh670538.aspx)
+* [Download SQL Server 2012 SP1](http://www.microsoft.com/betaexperience/pd/SQL2012EvalCTA/enus/default.aspx)
+
+**Prerequisites**
+
+Before you get started, there are a couple of steps you can take to make running the tests easier on yourself.
+
+Since the tests can run for several hours, it helps to set everything up so that once the tests are running, you can leave the machines unattended and don't need to be around to enter ssh or sudo passwords.
+
+1. Enable passwordless SSH access to localhost ([search Google for help](https://www.google.com/#hl=en&q=passwordless%20SSH%20access), yielding references such as these: [article 1](http://hortonworks.com/kb/generating-ssh-keys-for-passwordless-login/) [article 2](http://superuser.com/questions/336226/how-to-ssh-to-localhost-without-password) [article 3](https://help.ubuntu.com/community/SSH/OpenSSH/Keys))
+2. Enable passwordless sudo access ([Google for help](https://www.google.com/#hl=en&q=passwordless%20sudo)).
+
+After this, clone our repository and run `toolset/run-tests.py --help` for 
+detailed guidance. You can also refer to the [Benchmark Suite Setup README file](../setup/README.md) for more information. 
+
+**Installation and Usage Details**
+
+If you choose to run TFB on your own computer, you will need to install 
+passwordless SSH to your `load server` and your `database server` from 
+your `app server`. You will also need to enable passwordless sudo access
+on all three servers. If you are only planning to use *verify* mode, then
+all three servers can be the same computer, and you will need to ensure
+you have passwordless sudo access to `localhost`. 
+
+For all Linux framework tests, we use [Ubuntu 14.04](http://www.ubuntu.com/download/server), so 
+it is recommended you use this for development or use. Furthermore, the load server is Linux-only,
+even when testing Windows frameworks.
+
+## Windows server setup
+
+* Connect to the Windows server via Remote Desktop.
+* Copy `installer-bootstrap.ps1` from "toolset/setup/windows" to the server (use CTRL-C and CTRL-V).
+* Copy your Linux client private key too.
+* Right click on the installer script and select `Run with PowerShell`.
+* Press Enter to confirm.
+* It will install git and then launch `installer.ps1` from the repository, which will install everything else.
+* The installation takes about 20 minutes.
+* Then you have a working console: try `python`, `git`, `ssh`, `curl`, `node` etc. and verify that everything works + PowerShell goodies.
+
+The client/database machine is still assumed to be a Linux box. You can install just the client software via
+
+    python toolset\run-tests.py -s server-private-ip -c client-private-ip -i "C:\Users\Administrator\Desktop\client.key" --install-software --install client --list-tests
+
+but this step is not required if you already installed the Linux server and client as described above.
+
+Now you can run tests:
+
+    python toolset\run-tests.py -s server-private-ip -c client-private-ip -i "C:\Users\Administrator\Desktop\client.key" --max-threads 2 --duration 30 --sleep 5 --name win --test aspnet --type all
+
+## SQL Server setup
+
+* Connect to the SQL Server host via Remote Desktop.
+* Run a `Command Prompt` as Administrator.
+* Enter this command:
+
+        powershell -ExecutionPolicy Bypass -Command "iex (New-Object Net.WebClient).DownloadString('https://raw.github.com/TechEmpower/FrameworkBenchmarks/master/toolset/setup/sqlserver/setup-sqlserver-bootstrap.ps1')"
+
+* This will configure SQL Server, the Windows Firewall, and populate the database.
+
+Now, when running `run-tests.py`, just add `-d <ip of SQL Server instance>`. This works for the (Windows Server-based) `aspnet-sqlserver-raw` and `aspnet-sqlserver-entityframework` tests.
+
+## Other hosting
+
+The benchmark suite will work on physical or virtual servers that have local network connectivity between them and Internet access for software download.
+
+In general, any private or public hosting can be used, following this procedure:
+
+* Provision the instances (servers) according to the roles, operating system and additional software described in the deploment overview.
+* Configure networking as needed.
+* Set the Linux hosts for passwordless sudo as described in the instructions for dedicated hardware.
+
+Specific provisioning instructions and/or scripts for particular hosting or cloud services providers are welcome!
+

deployment/vagrant-aws/README.md

@@ -0,0 +1,195 @@
+# Automated Deployment Onto Amazon Web Services
+
+Scripts exist to setup either a development environment, or a benchmark-ready
+production environment. The development environment uses one small instance to 
+run all three TFB roles (app server, database server, client load generation). 
+The production environment uses three different instances networked together. 
+There are a number of environmental variables listed below that allow you to 
+change features such as instance type, availability region, etc. 
+
+We use [Vagrant](https://www.vagrantup.com) to launch/terminate EC2 instances, 
+and we provide a python script to configure the network as needed. 
+
+All amazon resources are tagged with `Project=FrameworkBenchmarks` and 
+`TFB_Role=[database|network|client|app]` so you can easily get a breakdown on 
+your bill. A default development setup costs about `$1/day`, while a default
+production setup costs about TODO. 
+
+## Prerequisites
+
+* **A recent version of Vagrant**, like 1.6.3 (NOTE: `apt-get` is 
+too old, download the newest `deb` directly). See 
+[here](https://www.vagrantup.com/downloads.html) for downloads
+
+* **Vagrant AWS Plugin** from [here](https://github.com/mitchellh/vagrant-aws)
+
+* **Python 2.7** to run the amazon setup script
+
+* **Amazon Command Line Interface**: From [here](https://github.com/aws/aws-cli)
+
+## Using Vagrant to Run Amazon-powered Virtual Machine
+
+The high level steps are 
+1) clone this project 
+2) set environment variables allowing us to log into your amazon account
+3) Run amazon setup script to create network
+4) Run `vagrant up --provider=aws` to launch into amazon
+5) Run `vagrant ssh` to log into the application server
+
+By default, your local git clone of this project will not by synced with the 
+remote amazon machines.
+
+## Required Environment Variables
+
+These environment variables are always required: 
+
+    TFB_AWS_ACCESS_KEY : Your Amazon Web Services Access Key
+    TFB_AWS_SECRET_KEY : Your Amazon Web Services Secret Access Key
+    TFB_AWS_KEY_NAME   : The name of the keypair you are using
+    TFB_AWS_KEY_PATH   : Path to the *.pem file for the keypair you are using
+
+You can declare these in your `~/.bash_profile`, using lines like:
+
+    export TFB_AWS_ACCESS_KEY="<your access key>"
+
+Or you can specify them on the command line as so: 
+
+    $ TFB_AWS_ACCESS_KEY="<your key>" TFB_AWS_SECRET_KEY="<your secret>" TFB_AWS_KEY_NAME="<your keypair name>" TFB_AWS_KEY_PATH="/home/you/.ssh/your_key.pem" vagrant up --provider aws
+
+## Directory Syncing
+
+By default, our Vagrant setup will not syncronize your local git working copy 
+with the remote filesystem on Amazon. Instead, the `~/FrameworkBenchmarks` 
+folder will be a fresh clone of `github.com/TechEmpower/FrameworkBenchmarks`
+
+This is due to the project currently being multiple GB in size (see #1050). 
+Synchronizing the local filesystem with the remote filesystem would require an
+initial upload to the remote system, and most people do not want to wait for 
+a multi-GB upload. 
+
+If you are interested in having this synchronization, you can use set an 
+environment variable as so: 
+
+    $ TFB_FORCE_SYNC=true vagrant up --provider aws
+
+This will tell vagrant to keep your git working copy in sync with the remote
+filesystem. While the initial upload is quite arduous, any later changes are
+synchronized quite quickly (the underlying system is rsync). This would 
+allow you to continue editing files on your local machine and then using the 
+remote virtual machine to test your edits. As long as you don't run `vagrant 
+destroy`, any later synchronization shouldn't take more than a few seconds. 
+
+## Environmental Variables 
+
+These only have effect if the vagrant aws provider is used. See 
+[here](../vagrant-virtualbox) for variables that take effect if the 
+virtualbox provider is used. 
+
+| Name                             | Values              | Purpose                  |
+| :------------------------------- | :------------------ | :----------------------- | 
+| <sup>1</sup>`TFB_AWS_ACCESS_KEY` | `<your_key>`        |  Your Amazon Web Services Access Key 
+| <sup>1</sup>`TFB_AWS_SECRET_KEY` | `<your_key>`        |  Your Amazon Web Services Secret Access Key 
+| <sup>1</sup>`TFB_AWS_KEY_NAME`   | `<key_name>`        |  The name of the key pair you are using 
+| <sup>1</sup>`TFB_AWS_KEY_PATH`   | `<path_to_keyfile>` |  Path to the *.pem file for the keypair you are using 
+| `TFB_FORCE_SYNC`                 | `true,false`        | Tells vagrant to rsync your local git clone with the remote amazon instance. Note: This requires an initial multi-GB upload 
+| `TFB_AWS_REPO_SLUG`              | default is `TechEmpower/FrameworkBenchmarks`  | If `TFB_FORCE_SYNC` is false, this defines the repository that will be checked out from Github on the Amazon instances
+| `TFB_AWS_REPO_BRANCH`            | default is `master`  | If `TFB_FORCE_SYNC` is false, this defines the repository branch that will be checked out from Github on the Amazon instances
+| `TFB_AWS_EC2_TYPE`               | `m1.small,c1.xlarge,etc` | The instance type used. Requires [previous generation instance](http://aws.amazon.com/ec2/previous-generation/) names because vagrant-aws does not yet support the new names
+| <sup>2</sup>`TFB_AWS_SUBNET`     | `<subnet_id>` (e.g. `subnet-2737230f`) | Which subnet instances should be launched into. The python setup script can generate this value for you
+| <sup>2</sup>`TFB_AWS_SEC_GROUP`  | `<security_group_id>` (e.g. `sg-871240e2`) | The default security group instances use
+| `TFB_AWS_EBS_TYPE`               | `gp2,standard,io1`  | The EBS [type](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#block-device-mapping-def). `gp2` is a general-purpose SSD, `standard` is a magnetic drive, `io1` is a guaranteed I/O SSD drive. Our benchmark mode defaults to `gp2`
+| `TFB_AWS_EBS_IO`                 | `<number>`          | Only used if `TFB_AWS_EBS_TYPE=io1`. The number of IO operations per second
+| `TFB_AWS_EBS_DELETE`             | `true,false`        | Should the EBS volume be deleted when `vagrant destroy` happens? Note: We cannot currently tag EBS with `Project=FrameworkBenchmarks`, so you'll need to remember which volumes were from this project, and which of those was the application server volume
+| `TFB_AWS_APP_IP`                 | `<ip address>`      | IP address used inside Amazon for the application server. Default is `172.16.0.16`
+| `TFB_AWS_LOAD_IP`                | `<ip address>`      | IP address used inside Amazon for the application server. Default is `172.16.0.17`
+| `TFB_AWS_DB_IP`                  | `<ip address>`      | IP address used inside Amazon for the database server. Default is `172.16.0.18`
+| `TFB_AWS_REGION`                 | `<aws region>`      | Region to launch the instances in. Default is `us-east-1`
+
+<sup>1</sup> Variable is required
+
+<sup>2</sup> Variable can be auto-generated using `setup_aws.py`
+
+**Note:** We do not currently support setting instance-specific variables, like instance type 
+or EBS variables, on a per-instance basis. The only exception is the ip address used for the 
+instances. 
+
+### Amazon Tips and Tricks
+
+**Halt Amazon instances to save money**
+
+You can use `vagrant stop` with the AWS provider, and using 
+`vagrant up` at a later date to restart the virtual machine. You will 
+continue to be charged for the block storage (about $0.05 per month per 
+GB), but you will not be charged for the CPU required to run the VM (about
+$5 per month). With our normal setup, this block storage cost is about 
+$0.40 per month (we currently use an 8GB volume)
+
+**Use Disk Snapshots For Rapid Launching**
+
+If you hate waiting 20 minutes for the virtual machine to initially
+be ready, just snapshot it after the first `vagrant up`. You'll have to 
+pay around $0.10 per month per GB to maintain this snapshot, but you can 
+use the snapshot as the disk image when starting a new virtual machine and 
+have a AWS development environment ready in seconds. 
+
+This trick is of limited utility if you are just doing the initial setup, 
+but it can become very valuable if you have run a **full** installation of 
+server software (e.g. `toolset/run-tests.py --install server --install-only`), 
+as this takes multiple hours to complete. 
+
+### FAQs
+
+**How much does running on Amazon cost?**: 
+
+Roughly `$1/day` with default settings.  
+
+Running `vagrant up --provider aws` takes about 20 minutes to complete. Running 
+`toolset/run-tests.py --install server --mode verify --test <foo>` takes between
+5 and 60 minutes, depending on `<foo>`. [Prices as of writing](http://aws.amazon.com/ec2/previous-generation/) show the default `m1.small` instance type is 
+`$0.044` per Hour, or roughly `$1/day`. You are charged for a full hour even
+if you only use a few minutes, so if you contantly cycle between `vagrant up`
+and `vagrant destroy` you will be charged `$0.044` per cycle. 
+
+Running `toolset/run-tests.py --mode benchmark --test <foo>` takes between 15-60
+minutes per test in `<foo>`, and running the full benchmark would likely take 
+between 1 and 5 days of continuous execution. Note that this single VM runs the 
+framework, the load generation, and the database, and is therefore not a stable 
+benchmarking setup. 
+
+**Are spot instances supported?**:
+
+There is an open issue at [mitchellh/vagrant-aws#32](https://github.com/mitchellh/vagrant-aws/issues/32) that appears to have 
+working code for spot instances. This could reduce the amazon cost up to 100x!
+Once this is supported in vagrant-aws it will be supported here. 
+
+**I disagree with the Amazon Setup!**:
+
+You're free to submit a PR to improve the automated deployment scripts. Please
+justify why you feel that your changes provide an improvement, and explain the
+total cost of running a benchmark with your update setup. Be aware that 
+you can use `vagrant provision` to selectively re-run your provision scripts,
+instead of having to constantly terminate and launch instances. 
+
+**Can I use VirtualBox and AWS Simultaneously?**:
+
+Not out of the box. Vagrant does not currently support using two 
+providers simultaneously for the same Vagrantfile. The quick fix is to 
+just copy the folder, and run one provider in one folder and one provider
+in another. 
+
+**Can I run multiple benchmarks simultaneously?**:
+
+Sure, but you will need to duplicate the directory containing the
+`Vagrantfile` and you will also need to use `TFB_AWS_APP_IP` (and similar 
+variables) to avoid IP address conflicts. You could re-run the `setup_aws.py`
+script and avoid changing the IP addresses, but this creates a new 
+Virtual Private Cloud (VPC), and you are limited to 5 VPC's per region. If 
+you are reading this FAQ, you may be interested in running more than 5 
+simultaneous benchmarks, so the better approach is to just increase all
+the IP addresses by 3 and run the additional benchmarks in the same VPC. 
+
+
+**I'm getting an AuthFailure but my Credientials are Correct!**:
+
+This normally means the AMI has been moved from public to private. Ubuntu's 
+Cloud image team occasionally does this. Navigate [here](http://cloud-images.ubuntu.com/trusty/current/) and find a more current AMI. 

deployment/vagrant-aws/setup_aws.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python
+#
+# Prepares Amazon network to run either vagrant-development or
+# vagrant-production. Configures subnets, virtual private clouds, 
+# security groups, etc
+#
+# Currently is a one-way operation, you have to delete these 
+# manually. Visit https://console.aws.amazon.com/vpc and remove the 
+# VPC tagged TFB_Network and that should undo all changes made my 
+# this script (for production mode). For development mode, the only 
+# things created are a security group and a subnet, so find those in
+# your standard EC2 console https://console.aws.amazon.com/ec2 and 
+# delete them manually
+
+import subprocess
+import json
+import logging
+import sys
+
+log = logging.getLogger('aws')
+
+nwtags = "Key=Project,Value=FrameworkBenchmarks Key=TFB_Role,Value=network"
+
+def setup_vpc():
+  '''Sets up a Virtual Private Cloud to allow hosts to communicate'''
+  
+  # Setup VPC
+  log.info("Creating a new Virtual Private Cloud...")
+  log.info(" See details at http://console.aws.amazon.com/vpc")
+  vpc = run_aws("create-vpc --cidr-block 172.16.0.0/16 --instance-tenancy default")
+  vpcid = vpc["Vpc"]["VpcId"]
+
+  run_aws("modify-vpc-attribute --vpc-id %s --enable-dns-support" % vpcid)
+  run_aws("modify-vpc-attribute --vpc-id %s --no-enable-dns-hostnames" % vpcid)
+  run_aws("create-tags --resources %s --tags %s Key=Name,Value=TFB_Network" % (vpcid, nwtags))
+  log.debug(run_aws("describe-vpcs --vpc-id %s" % vpcid, load=False))
+
+  # Setup internet gateway
+  log.info("Creating InternetGateway for the VPC...")
+  igw = run_aws("create-internet-gateway")
+  igwid = igw["InternetGateway"]["InternetGatewayId"]
+  run_aws("create-tags --resources %s --tags %s Key=Name,Value=TFB_Gateway" % (igwid, nwtags))
+  run_aws("attach-internet-gateway --internet-gateway-id %s --vpc-id %s" % (igwid, vpcid))
+  log.debug(run_aws("describe-internet-gateways --internet-gateway-ids %s" % igwid, load=False))
+
+  # Setup public subnet 
+  # NOTE: We considered using a public and private subnet, but 
+  # this requires us to launch an extra EC2 instance for the duration of the 
+  # benchmark to handle the NAT between the public subnet and the private subnet,
+  # so the cost is quite high. Also, Internet traffic is only generated during 
+  # framework setup stages (e.g. during software installation), not during the 
+  # running of the benchmark. 
+  # We chose to use a single public subnet and filter inbound traffic to prevent 
+  # interference during the test
+  log.info("Creating subnet inside the VPC...")
+  pubsub = run_aws("create-subnet --vpc-id %s --cidr-block 172.16.0.0/24" % vpcid)
+  pubid = pubsub["Subnet"]["SubnetId"]
+  log.debug("Found subnet id: %s", pubid)
+
+  #run_aws("modify-subnet-attribute --subnet-id %s --map-public-ip-on-launch" % pubid)
+  run_aws("create-tags --resources %s --tags %s Key=Name,Value=TFB_Public" % (pubid, nwtags))
+  log.debug(run_aws("describe-subnets --subnet-ids %s" % pubid, load=False))
+  
+  # Setup routing
+  log.info("Creating routing table for VPC...")
+  route = run_aws("describe-route-tables --filters Name=vpc-id,Values=%s" % vpcid)
+  routeid = route["RouteTables"][0]["RouteTableId"]
+  run_aws("create-tags --resources %s --tags %s Key=Name,Value=TFB_Routing" % (routeid, nwtags))
+  log.info("  Creating route to internet...")
+  run_aws("create-route --route-table-id %s --destination-cidr-block 0.0.0.0/0 --gateway-id %s" % (routeid, igwid)) 
+  log.info("  Associating routing table and subnet...")
+  run_aws("associate-route-table --route-table-id %s --subnet-id %s" % (routeid, pubid))
+
+  # Setup default security group for instances launched in the VPC
+  log.info("Creating default security group for VPC")
+  group = run_aws("create-security-group --group-name TFB_Security --vpc-id %s --description 'FrameworkBenchmarks security group'" % vpcid)
+  groupid = group["GroupId"]
+  run_aws("create-tags --resources %s --tags %s Key=Name,Value=TFB_Security" % (groupid, nwtags))
+  run_aws("authorize-security-group-ingress --group-id %s --protocol tcp --port 22 --cidr 0.0.0.0/0" % groupid)
+  # run_aws("authorize-security-group-egress --group-id %s --protocol -1 --cidr 0.0.0.0/0 --port all" % groupid)
+  run_aws("authorize-security-group-ingress --group-id %s --source-group %s --protocol -1 --port -1" % (groupid, groupid))
+
+  log.info("Complete."
+  log.info(" Here are the environment variables you should use:")
+  print "export TFB_AWS_SUBNET=%s" % pubid
+  print "export TFB_AWS_SEC_GROUP=%s" % groupid
+
+  return vpcid
+
+def unset_vpc(vpcid):
+  '''Doesn't work at the moment, we need to delete all the other items first'''
+  run_aws("delete-vpc --vpc-id %s" % vpcid)
+
+def run_aws(command, prefix=True, load=True):
+  '''Runs an AWS command and returns the JSON
+  prefix: Should we prefix "aws ec2 " to your command
+  load: Should we auto-load the response JSON into a python object? 
+  '''
+  if prefix:
+    command = "aws ec2 %s" % command
+  log.debug("Request : %s", command)
+  result = subprocess.check_output(command, shell=True)
+  log.debug("Response: %s", result)
+  if load:
+    return json.loads(result)
+  else:
+    return result
+
+if __name__ == "__main__":
+  args = sys.argv[1:]
+
+  logging.basicConfig(level=logging.INFO)
+  usage = '''Usage: setup_aws.py
+
+Prepares Amazon network to run either vagrant-development 
+or vagrant-production. Configures subnets, virtual private
+clouds, security groups, etc. 
+
+Outputs TFB_AWS_SEC_GROUP and TFB_AWS_SUBNET
+
+Currently is a one-way operation, you have to delete these 
+manually. This script expects standard AWS environment 
+variables to exist e.g. AWS_ACCESS_KEY_ID, 
+AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION
+'''
+
+  if len(args) != 0:
+    print usage
+    sys.exit(1)
+
+  setup_vpc()

deployment/vagrant-common/bootstrap.sh

@@ -0,0 +1,147 @@
+#!/usr/bin/env bash
+#
+# Prepares a virtual machine for running TFB
+#
+# Intentionally uses ~, $HOME, and $USER so that the 
+# same script can work for VirtualBox (username vagrant)
+# and Amazon (username ubuntu)
+
+# Add everything passed in the first argument to our 
+# local environment. This is a hack to let us use 
+# environment variables defined on the host inside the 
+# guest machine
+while read -r line; do  
+  export $line; 
+done <<< "$1"
+
+# Store any custom variables used at launch, in case someone forgets
+# what this instance is (e.g. SSD or HDD, etc)
+echo "$1" > ~/.tfb_launch_options
+
+# Are we installing the server machine, the client machine, 
+# the database machine, or all machines? 
+# Valid values: 
+#    - all      (we are setting up a development environment)
+#    - database (we are setting up the database machine)
+#    - client   (we are setting up the client machine for load generation)
+#    - server   (we are setting up the machine that will host frameworks)
+ROLE=${2:-all}
+
+# Set a number of variables by either pulling them from 
+# the existing environment or using the default values
+# I'm renaming them to indicate that (in this script only)
+# the values are provisioner agnostic
+SERVER_IP=${TFB_AWS_APP_IP:-172.16.0.16}
+CLIENT_IP=${TFB_AWS_LOAD_IP:-172.16.0.17}
+DATABA_IP=${TFB_AWS_DB_IP:-172.16.0.18}
+if [ "$ROLE" == "all" ]; then
+  SERVER_IP=127.0.0.1
+  CLIENT_IP=127.0.0.1
+  DATABA_IP=127.0.0.1
+fi
+
+GH_REPO=${TFB_AWS_REPO_SLUG:-TechEmpower/FrameworkBenchmarks}
+GH_BRANCH=${TFB_AWS_REPO_BRANCH:-master}
+
+# A shell provisioner is called multiple times
+if [ ! -e "~/.firstboot" ]; then
+
+  # Setup some nice TFB defaults
+  if [ "$ROLE" == "all" ]; then
+    echo "export TFB_CLIENT_IDENTITY_FILE=$HOME/.ssh/id_rsa" >> ~/.bash_profile
+    echo "export TFB_DATABASE_IDENTITY_FILE=$HOME/.ssh/id_rsa" >> ~/.bash_profile
+  else
+    echo "export TFB_CLIENT_IDENTITY_FILE=$HOME/.ssh/client" >> ~/.bash_profile
+    echo "export TFB_DATABASE_IDENTITY_FILE=$HOME/.ssh/database" >> ~/.bash_profile
+  fi
+  echo "export TFB_SERVER_HOST=$SERVER_IP" >> ~/.bash_profile
+  echo "export TFB_CLIENT_HOST=$CLIENT_IP" >> ~/.bash_profile
+  echo "export TFB_DATABASE_HOST=$DATABA_IP" >> ~/.bash_profile
+  echo "export TFB_CLIENT_USER=$USER" >> ~/.bash_profile
+  echo "export TFB_DATABASE_USER=$USER" >> ~/.bash_profile
+  echo "export FWROOT=$HOME/FrameworkBenchmarks" >> ~/.bash_profile 
+  source ~/.bash_profile
+
+  # Ensure their host-local benchmark.cfg is not picked up on the remote host
+  if [ -e "~/FrameworkBenchmarks/benchmark.cfg" ]; then
+    echo "You have a benchmark.cfg file that will interfere with Vagrant, moving to benchmark.cfg.bak"
+    mv ~/FrameworkBenchmarks/benchmark.cfg ~/FrameworkBenchmarks/benchmark.cfg.bak
+  fi
+
+  # Setup hosts 
+  echo "Setting up convenience hosts entries"
+  echo $DATABA_IP TFB-database | sudo tee --append /etc/hosts
+  echo $CLIENT_IP TFB-client   | sudo tee --append /etc/hosts
+  echo $SERVER_IP TFB-server   | sudo tee --append /etc/hosts
+
+  # Update hostname to reflect our current role
+  if [ "$ROLE" != "all" ]; then
+    echo "Updating hostname"
+    echo 127.0.0.1 `hostname` | sudo tee --append /etc/hosts
+    myhost=TFB-${ROLE}
+    echo $myhost | sudo tee --append /etc/hostname
+    sudo hostname $myhost
+    echo Updated /etc/hosts file to be: 
+    cat /etc/hosts
+  fi
+
+  # Workaround mitchellh/vagrant#289
+  echo "grub-pc grub-pc/install_devices multiselect     /dev/sda" | sudo debconf-set-selections
+
+  # Install prerequisite tools
+  echo "Installing prerequisites"
+  sudo apt-get install -y git
+  sudo apt-get install -y python-pip
+
+  # Make project available
+  # If they synced it to /FwBm, just expose it at ~/FwBm
+  # If they didn't sync, we need to clone it
+  if [ -d "/FrameworkBenchmarks" ]; then
+    ln -s /FrameworkBenchmarks $FWROOT
+    echo "Removing installs/ and results/ folders so they do not interfere"
+    rm -rf $FWROOT/installs $FWROOT/results
+  else
+    # If there is no synced folder, clone the project
+    echo "Cloning project from $GH_REPO $GH_BRANCH"
+    git clone -b ${GH_BRANCH} https://github.com/${GH_REPO}.git $FWROOT
+  fi
+  sudo pip install -r $FWROOT/config/python_requirements.txt
+
+  # Everyone gets SSH access to localhost
+  echo "Setting up SSH access to localhost"
+  ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
+  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+  chmod 600 ~/.ssh/authorized_keys
+
+  # Enable remote SSH access if we are running production environment
+  # Note : this are always copied from the local working copy using a
+  #        file provisioner. While they exist in the git clone we just 
+  #        created (so we could use those), we want to let the user
+  #        have the option of replacing the keys in their working copy
+  #        and ensuring that only they can ssh into the machines
+  if [ "$ROLE" == "server" ]; then
+    # Ensure keys have proper permissions
+    chmod 600 ~/.ssh/client ~/.ssh/database
+  elif [ "$ROLE" != "all" ]; then
+    # Ensure keys can be used to ssh in
+    echo "Setting up SSH access for the TFB-server"
+    mykey=~/.ssh/$ROLE.pub
+    echo "Using key: "
+    ssh-keygen -lv -f $mykey
+    cat $mykey >> ~/.ssh/authorized_keys
+
+    # Ensure keys have proper permissions
+    chmod 600 ~/.ssh/client ~/.ssh/database
+  fi
+
+  # Setup 
+  echo "Installing $ROLE software"
+  cd $FWROOT
+  toolset/run-tests.py --verbose --install $ROLE --install-only --test ''
+
+  # Setup a nice welcome message for our guest
+  echo "Setting up welcome message"
+  sudo rm -f /etc/update-motd.d/51-cloudguest
+  sudo rm -f /etc/update-motd.d/98-cloudguest
+  sudo mv ~/.custom_motd.sh /etc/update-motd.d/55-tfbwelcome
+fi

deployment/vagrant-common/checks.rb

@@ -0,0 +1,104 @@
+# Helps with finding current OS
+module OS
+  def OS.windows?
+    (/cygwin|mswin|mingw|bccwin|wince|emx/ =~ RUBY_PLATFORM) != nil
+  end
+
+  def OS.mac?
+   (/darwin/ =~ RUBY_PLATFORM) != nil
+  end
+
+  def OS.unix?
+    !OS.windows?
+  end
+
+  def OS.linux?
+    OS.unix? and not OS.mac?
+  end
+end
+
+# Helps with finding current OS
+module ARCH
+  def ARCH.is64?
+    (/x86_64/ =~ RUBY_PLATFORM) != nil
+  end
+
+  def ARCH.is32?
+    !ARCH.is64?
+  end
+end
+
+def get_provider()
+  # Workaround for mitchellh/vagrant#1867
+  if (ARGV[1] and ARGV[1].split('=')[0] == "--provider" and ARGV[1].split('=')[1])
+    provider = ARGV[1].split('=')[1].to_sym
+  else
+    provider = (ARGV[2] || ENV['VAGRANT_DEFAULT_PROVIDER'] || :virtualbox).to_sym
+  end
+  provider
+end
+
+def check_provider_needs(provider)
+  if provider == :"aws"
+    check_aws_needs
+  elsif provider == :"virtualbox"
+    check_vb_needs
+  end
+end
+
+def check_aws_needs()
+  # Check required variables
+  if !(ENV['TFB_AWS_ACCESS_KEY'] and ENV['TFB_AWS_SECRET_KEY'] \
+        and ENV['TFB_AWS_KEY_NAME'] and ENV['TFB_AWS_KEY_PATH'])
+    abort 'If you want to use the AWS provider, you must provide these four variables: 
+    TFB_AWS_ACCESS_KEY : Your Amazon Web Services Access Key
+    TFB_AWS_SECRET_KEY : Your Amazon Web Services Secret Access Key
+    TFB_AWS_KEY_NAME   : The name of the keypair you are using
+    TFB_AWS_KEY_PATH   : Path to the *.pem file for the keypair you are using'
+  end
+
+  # Print warning
+    warning = "\033[33m\
+WARNING: FrameworkBenchmarks is disabling folder sync between your
+local working copy and Amazon Web Services - the ~/FrameworkBenchmarks
+directory in your VM will be a git clone of TechEmpower/FrameworkBenchmarks. 
+You can re-enable folder sync using 
+    $ TFB_FORCE_SYNC=true vagrant up --provider=aws
+but be aware that you will need to upload ~2GB to Amazon before you can use
+the VM as normal.\033[0m"
+  puts warning
+end
+
+def check_vb_needs()
+  # Check if this computer can run a 64-bit OS
+  warning = "\033[31m\
+WARNING: FrameworkBenchmarks only officially supports a 64-bit 
+virtual machine, which your current system may not be able to 
+support. Use `TFB_SHOW_VM=true vagrant up` to watch the VM launch -
+if you just see a black window you likely cannot run a 64-bit VM. 
+
+To workaround, consider using the Amazon (e.g. AWS) provider
+   $ vagrant up --provider=aws
+
+Or forcing FrameworkBenchmarks to attempt a 32-bit VM
+   $ TFB_VM_ARCH=32 vagrant up
+  
+  See http://askubuntu.com/questions/41550 for more info\033[0m"
+
+  # AMD-based needs svm feature, Intel-based needs vmx feature
+  if OS.linux? and %x(egrep '(vmx|svm)' /proc/cpuinfo).empty?
+    puts warning
+  end
+
+  # Ignore PowerPC, check for intel features
+  if OS.mac? and %x(sysctl -n machdep.cpu.features | grep -i vmx).empty?
+    puts warning
+  end
+
+  # Don't really know how to check CPU features, so I'll just check 
+  # the arch
+  if OS.windows? and ARCH.is32?
+    puts warning
+  end
+end
+

deployment/vagrant-common/core.rb

@@ -0,0 +1,103 @@
+
+def provision_bootstrap(config, role)
+  # Find all environment variables that begin with 
+  # TFB_* and pass them as an argument. This is a hack to 
+  # let our bootstrap script use environment variables that 
+  # were originally defined on the host 
+  # Skip any potentially sensitive variables
+  env_arg = ""
+  skip= ['TFB_AWS_ACCESS_KEY', 'TFB_AWS_SECRET_KEY', 'TFB_AWS_KEY_NAME', 'TFB_AWS_KEY_PATH']
+  ENV.each do |key, array|
+    if (key.start_with?("TFB") && !skip.include?(key))
+      env_arg.concat key
+      env_arg.concat "="
+      env_arg.concat array
+      env_arg.concat "\n"
+    end
+  end
+  env_arg = env_arg.strip
+
+  # TODO this will break if the environment contains the ' delimiter, 
+  # so at some point we need to escape the ' character here and unescape
+  # it in bootstrap.sh
+  config.vm.provision "shell" do |sh|
+    sh.path = "../vagrant-common/bootstrap.sh"
+    sh.privileged = false
+    sh.args = "'#{env_arg}' '#{role}'"
+  end
+end
+
+def provider_aws(config, role, ip_address='172.16.0.16')
+  config.vm.provider :aws do |aws, override|
+    aws.access_key_id = ENV['TFB_AWS_ACCESS_KEY'] 
+    aws.secret_access_key = ENV['TFB_AWS_SECRET_KEY']
+    aws.keypair_name = ENV['TFB_AWS_KEY_NAME']
+    override.ssh.private_key_path = ENV['TFB_AWS_KEY_PATH']
+
+    # Use a boilerplate box - this will be replaced by the AMI
+    override.vm.box = "dummy"
+    override.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
+    
+    # From http://cloud-images.ubuntu.com/trusty/current/
+    # This is 64-bit Ubuntu 14.04 US east EBS
+    # See http://cloud-images.ubuntu.com/vagrant/trusty/current/ 
+    # for comparison to the Ubuntu Vagrant VirtualBox boxes 
+    aws.ami = "ami-f6bf659e"
+    override.ssh.username = "ubuntu"
+    
+    aws.private_ip_address = ip_address
+    aws.associate_public_ip = true
+    aws.region = ENV.fetch('TFB_AWS_REGION', 'us-east-1')
+    aws.subnet_id = ENV['TFB_AWS_SUBNET']  # subnet-2737230f for me
+    aws.security_groups = [ENV['TFB_AWS_SEC_GROUP']] # sg-871240e2 
+    aws.instance_type = ENV.fetch('TFB_AWS_EC2_TYPE', 'm1.large')
+
+    aws.tags = {
+      'Project' => 'FrameworkBenchmarks',
+      'TFB_role' => role
+     }
+    
+    # Setup disk. Defauly is 15GB General Purpose SSD
+    # Double the default volume size, as we download and 
+    # install a *lot* of stuff
+    # Documentation is at http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-blockdev-template.html
+    aws.block_device_mapping = [{ 'DeviceName' => '/dev/sda1', 
+      'Ebs.VolumeSize' => 15 ,
+      'Ebs.DeleteOnTermination' => ENV.fetch('TFB_AWS_EBS_DELETE', true),
+      'Ebs.VolumeType' => ENV.fetch('TFB_AWS_EBS_TYPE', 'gp2')
+      }]
+    if ENV.fetch('TFB_AWS_EBS_TYPE', 'standard') == 'io1'
+      aws.block_device_mapping[0]['Ebs.Iops'] = ENV.fetch('TFB_AWS_EBS_IO', '1000')
+    end
+
+    if ENV.fetch('TFB_FORCE_SYNC', "false") == "true"
+      override.vm.synced_folder "../..", "/FrameworkBenchmarks"
+    end
+
+  end
+end
+
+def provider_virtualbox(config, role)
+  config.vm.provider :virtualbox do |vb, override|
+    override.vm.hostname = "TFB-#{role}"
+
+    override.vm.box = "ubuntu/trusty64"
+    if ENV.fetch('TFB_VM_ARCH','64') == "32"
+      override.vm.box = "ubuntu/trusty32"
+    end
+    
+    if ENV.fetch('TFB_SHOW_VM', false)
+      vb.gui = true
+    end
+
+    if ENV.fetch('TFB_VB_MEM', 2048)
+      vb.customize ["modifyvm", :id, "--memory", "2048"]
+    end
+
+    override.vm.synced_folder "../..", "/FrameworkBenchmarks"
+
+    if role.eql? "all" or role.eql? "app"
+      override.vm.network :forwarded_port, guest: 8080, host: 28080
+    end
+  end
+end

deployment/vagrant-common/custom_motd.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+
+echo "$(tput setaf 4)"
+echo "Welcome to the FrameworkBenchmarks project!"
+echo ""
+echo "  To get started, perhaps try this:"
+echo "    \$ cd FrameworkBenchmarks"
+echo "    \$ toolset/run-tests.py --install server --test go"
+echo "    \$ cat results/ec2/latest/logs/go/out.txt"
+echo ""
+echo "  You can get lots of help:"
+echo "    \$ toolset/run-tests.py --help"
+echo ""
+echo "  This Vagrant environment is already setup and ready to go, so you"
+printf "  can safely ignore any flags about users, hosts, or identity files $(tput sgr 0)"

deployment/vagrant-development/.gitignore

@@ -0,0 +1 @@
+.vagrant/

deployment/vagrant-development/README.md

@@ -0,0 +1,108 @@
+# Setting up a Development Environment Using Vagrant
+
+The standard development environment is a single computer that fills the 
+role of database server, load generation server, and framework server. 
+While this is not useful for serious benchmarking, it does allow 
+painless development and testing. Using a virtual machine as a 
+development environment prevents having to install hundreds of 
+framework dependencies directly onto your computer. 
+
+We provide a [Vagrant](https://www.vagrantup.com) script to automatically 
+setup a virtual machine as a development server. Currently, you can use 
+either [VirtualBox](https://www.virtualbox.org/) or [Amazon EC2](http://aws.amazon.com/ec2/)
+to run this development environment. This page details how to use 
+VirtualBox, please read [here](blah) for information on using Amazon.
+
+Feel free to submit a PR to help us support more providers, such as 
+[Rackspace](http://www.rackspace.com/) or [VMware](http://www.vmware.com/). 
+
+## Prerequisites
+
+* **A recent version of Vagrant**, like 1.6.3 (NOTE: `apt-get` is 
+too old, download the newest `deb` directly). See 
+[here](https://www.vagrantup.com/downloads.html) for downloads
+
+* **A CPU that can virtualize a 64-bit virtual OS**, because TFB
+downloads a number of static binaries that require 64-bit. See
+the FAQs section below for more on this. If you cannot meet this 
+requirement, consider using the Amazon provider (about `$1/day`)
+
+* **VirtualBox** to run the virtual machines
+
+## Launching VirtualBox Development Environment
+
+In short, you need to clone the project and then run `vagrant up` 
+followed by `vagrant ssh`. Your cloned working directory will be 
+exposed inside the virtual machine at `~/FrameworkBenchmarks`, so you can 
+continue to edit files using your favorite IDE and the changes will
+show up inside the virtual machine. 
+
+Details: 
+
+```bash
+# Go into the right directory
+$ cd FrameworkBenchmarks/toolset/deployment/vagrant
+# Setup the VM with all initial software (takes 15-20 minutes)
+$ vagrant up
+# SSH into the machine (you get a nice welcome message)
+$ vagrant ssh
+Welcome to the FrameworkBenchmarks project!
+   
+To get started, perhaps try this:
+   $ cd FrameworkBenchmarks
+   $ toolset/run-tests.py --install server --test go
+   $ cat results/ec2/latest/logs/go/out.txt
+
+You can get lots of help:
+   $ toolset/run-tests.py --help
+
+This Vagrant environment is already setup and ready to go, so you
+can ignore any flags about users, hosts, or identity files
+```
+
+## Using VirtualBox Development Environment
+
+After using `vagrant ssh`, you will find the project at `~/FrameworkBenchmarks`. 
+You can use `run-tests.py` without specifying any information about hosts, 
+usernames, or private key files, as so: 
+
+    $ cd ~/FrameworkBenchmarks
+    $ toolset/run-tests.py --mode verify --install server --test go
+    <log for installing software needed for go test>
+    <log for launching go framework>
+    <log for verifying go framework meets requirements>
+    $ cat results/ec2/latest/logs/go/out.txt
+    <log with stdout from framework go>
+
+# Using Amazon-powered Virtual Machine
+
+Please read the document [here]
+
+The high level steps are similar, run `vagrant up --provider=aws` followed 
+by `vagrant ssh`. The main differences are 1) you need to provide a 
+number of environment variables to let vagrant log into your AWS account, 
+and 2) your git working copy will not be synced with the remote virtual 
+machine (by default). 
+
+### FAQs
+
+**I'm using a 32-bit computer, can I run your Virtual Machine?**: 
+
+If 1) your CPU provides the [vmx or smv](http://en.wikipedia.org/wiki/X86_virtualization) features, and 2) they are enabled
+in your BIOS and 3) they are enabled in Virtualbox, then yes. 
+The easiest way to check this all is to attempt to launch the VM 
+without hiding the window:
+
+    $ TFB_SHOW_VM=true vagrant up
+
+If you see a boot sequence and a login, you're good to go. If you 
+only see a black screen, examine if you meet all three requirements. 
+If you do not, then you can either 1) run on Amazon or 2) try your
+luck with a 32-bit virtual machine. A number of the downloaded 
+framework binaries are 64-bit only, so those will be unable to run. 
+
+To force FrameworkBenchmarks to use a 32-bit VM, do this: 
+     
+    $ TFB_ARCH=32 vagrant up
+
+See [here](http://askubuntu.com/questions/41550) for some helpful information.  

deployment/vagrant-development/Vagrantfile

@@ -0,0 +1,27 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+require_relative '../vagrant-common/checks'
+require_relative '../vagrant-common/core'
+
+provider = get_provider
+check_provider_needs(provider)
+
+Vagrant.configure("2") do |config|
+
+  config.vm.provision :file do |file|
+      file.source = "../vagrant-common/custom_motd.sh"
+      file.destination = "~/.custom_motd.sh"
+  end
+  
+  # Set AWS variables to be more appropriate for a development machine
+  ENV['TFB_AWS_EBS_TYPE'] = ENV.fetch('TFB_AWS_EBS_TYPE', 'standard')
+  ENV['TFB_AWS_EC2_TYPE'] = ENV.fetch('TFB_AWS_EC2_TYPE', 'm1.small')
+
+  provision_bootstrap(config, 'all')
+
+  provider_virtualbox(config, 'all')
+
+  provider_aws(config, 'integrated')
+  
+end

deployment/vagrant-production/.gitignore

@@ -0,0 +1 @@
+.vagrant/

deployment/vagrant-production/README.md

@@ -0,0 +1,143 @@
+# Vagrant Automated Deployment
+
+Currently, our [Vagrant](https://www.vagrantup.com) setup can automatically
+setup a development environment running in a single virtual machine. 
+This virtual machine can run inside either [VirtualBox](https://www.virtualbox.org/) or [Amazon EC2](http://aws.amazon.com/ec2/). Feel free to submit a PR 
+to help us support more providers, such as [rackspace](http://www.rackspace.com/)
+or [VMware](http://www.vmware.com/). 
+
+This single VM runs the `app server`, `client server`, and `database server`, 
+and allows you to rapidly test if your framework/framework changes integrate
+properly with TFB. While you *can* run a full benchmark using 
+this virtual machine, your results would be quite useless. 
+
+## Prerequisites
+
+* **A recent version of Vagrant**, like 1.6.3 (NOTE: `apt-get` is 
+too old, download the newest `deb` directly). See 
+[here](https://www.vagrantup.com/downloads.html) for downloads.
+
+* **A CPU that can virtualize a 64-bit virtual OS**, because TFB
+downloads a number of static binaries that require 64-bit. See
+the FAQs section below for more on this
+
+* **VirtualBox**, if you plan to run the Virtual Machine locally. If 
+you plan to solely use Amazon-based Virtual Machines you can skip this. 
+
+## Using Vagrant to Run VirtualBox-powered Virtual Machine
+
+In short, you need to clone the project and then run `vagrant up` 
+followed by `vagrant ssh`. Your cloned working directory will be 
+exposed inside the VM at `~/FrameworkBenchmarks`, so you can 
+continue to edit files using your favorite IDE and the changes will
+show up inside the virtual machine. 
+
+Details: 
+
+```bash
+# Go into the right directory
+$ cd FrameworkBenchmarks/toolset/deployment/vagrant
+# Setup the VM with all initial software (takes 15-20 minutes)
+$ vagrant up
+# SSH into the machine (you get a nice welcome message)
+$ vagrant ssh
+Welcome to the FrameworkBenchmarks project!
+   
+To get started, perhaps try this:
+   $ cd FrameworkBenchmarks
+   $ toolset/run-tests.py --install server --test go
+   $ cat results/ec2/latest/logs/go/out.txt
+
+You can get lots of help:
+   $ toolset/run-tests.py --help
+
+This Vagrant environment is already setup and ready to go, so you
+can ignore any flags about users, hosts, or identity files
+```
+
+## Using Vagrant to Run Amazon-powered Virtual Machine
+
+The high level steps are similar, run `vagrant up --provider=aws` followed 
+by `vagrant ssh`. The main differences are 1) you need to provide a 
+number of environment variables to let vagrant log into your AWS account, 
+and 2) your git working copy will not be synced with the remote virtual 
+machine (by default). 
+
+#### Required Environment Variables for Using Amazon Web Services
+
+These are all required: 
+
+    TFB_AWS_ACCESS_KEY : Your Amazon Web Services Access Key
+    TFB_AWS_SECRET_KEY : Your Amazon Web Services Secret Access Key
+    TFB_AWS_KEY_NAME   : The name of the keypair you are using
+    TFB_AWS_KEY_PATH   : Path to the *.pem file for the keypair you are using
+
+You can declare these in your `~/.bash_profile`, using lines like:
+
+    export TFB_AWS_ACCESS_KEY="<your access key>"
+
+Or you can specify them on the command line as so: 
+
+    $ TFB_AWS_ACCESS_KEY="<your key>" TFB_AWS_SECRET_KEY="<your secret>" TFB_AWS_KEY_NAME="<your keypair name>" TFB_AWS_KEY_PATH="/home/you/.ssh/your_key.pem" vagrant up --provider aws
+
+#### Directoy Syncing and Amazon Web Services
+
+By default, our Vagrant setup will not syncronize your local git working copy 
+with the remote filesystem on Amazon. Instead, the `~/FrameworkBenchmarks` 
+folder will be a fresh clone of github.com/TechEmpower/FrameworkBenchmarks. 
+
+This is due to the project currently being multiple GB in size (see #1050). 
+Synchronizing the local filesystem with the remote filesystem would require an
+initial upload to the remote system, and most people do not want to wait for 
+a multi-GB upload. 
+
+If you are interested in having this synchronization, you can use set an 
+environment variable as so: 
+
+    $ TFB_FORCE_SYNC=true vagrant up --provider aws
+
+This will tell vagrant to keep your git working copy in sync with the remote
+filesystem. While the initial upload is quite arduous, any later changes are
+synchronized quite quickly (the underlying system is rsync). This would 
+allow you to continue editing files on your local machine and then using the 
+remote virtual machine to test your edits. As long as you don't run `vagrant 
+destroy`, any later synchronization shouldn't take more than a few seconds. 
+
+
+### FAQs
+
+**I'm using a 32-bit computer, can I run your Virtual Machine?**: 
+
+If 1) your CPU provides the [vmx or smv](http://en.wikipedia.org/wiki/X86_virtualization) features, and 2) they are enabled
+in your BIOS and 3) they are enabled in Virtualbox, then yes. 
+The easiest way to check this all is to attempt to launch the VM 
+without hiding the window:
+
+    $ TFB_SHOW_VM=true vagrant up
+
+If you see a boot sequence and a login, you're good to go. If you 
+only see a black screen, examine if you meet all three requirements. 
+If you do not, then you can either 1) run on Amazon or 2) try your
+luck with a 32-bit virtual machine. A number of the downloaded 
+framework binaries are 64-bit only, so those will be unable to run. 
+
+To force FrameworkBenchmarks to use a 32-bit VM, do this: 
+     
+    $ TFB_ARCH=32 vagrant up
+
+See [here](http://askubuntu.com/questions/41550) for some helpful information.  
+
+**How much does running on Amazon cost?**: 
+
+Roughly `$1/day` with default settings. 
+
+Running `vagrant up --provider aws` takes about 20 minutes to complete. Running 
+`toolset/run-tests.py --install server --mode verify --test <foo>` takes between
+5 and 60 minutes, depending on `<foo>`. [Prices as of writing](http://aws.amazon.com/ec2/previous-generation/) show the default `m1.small` instance type is 
+`$0.044` per Hour, or roughly `$1/day`
+
+Running `toolset/run-tests.py --mode benchmark --test <foo>` takes between 15-60
+minutes per test in `<foo>`, and running the full benchmark would likely take 
+between 1 and 5 days of continuous execution. Note that this single VM runs the 
+framework, the load generation, and the database, and is therefore not a stable 
+benchmarking setup. 

deployment/vagrant-production/Vagrantfile

@@ -0,0 +1,47 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+require_relative '../vagrant-common/checks'
+require_relative '../vagrant-common/core'
+provider = get_provider
+check_provider_needs(provider)
+
+Vagrant.configure("2") do |config|
+
+  server_ip = ENV.fetch('TFB_AWS_LOAD_IP', '172.16.0.16')
+  client_ip = ENV.fetch('TFB_AWS_DB_IP', '172.16.0.17')
+  databa_ip = ENV.fetch('TFB_AWS_APP_IP', '172.16.0.18')
+  
+  # Put the keys inside each box
+  Dir['keys/*'].each do |fname|
+    config.vm.provision :file do |file|
+      file.source = fname
+      file.destination = "~/.ssh/" + File.basename(fname)
+    end
+  end
+
+  config.vm.provision :file do |file|
+      file.source = "../vagrant-common/custom_motd.sh"
+      file.destination = "~/.custom_motd.sh"
+  end
+
+  config.vm.define "client" do |client|
+    provision_bootstrap(client, "client")
+    provider_aws(client, "loadgen", client_ip)
+    provider_virtualbox(client, "client")
+  end
+
+  config.vm.define "db" do |db|
+    provision_bootstrap(db, "database")
+    provider_aws(db, "database", databa_ip)
+    provider_virtualbox(db, "database")
+  end
+
+  # Define the app server as the primary VM
+  config.vm.define "app", primary: true do |app|
+    provision_bootstrap(app, "server")
+    provider_aws(app, "appserver", server_ip)
+    provider_virtualbox(app, "server")
+  end
+
+end

deployment/vagrant-production/keys/client

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpAIBAAKCAQEAqbwcjGjUhqqlTKt8CPiS7IyZr3AHV/ZY/lU5hqixDtX0rNUU
+LY0gSNNgDOLsk4refgZbqxUlUykcjhYv4Lbmkav7ru2cqkQcyGcXMvCluh4rZSCe
+1omCsu8RHCTVY0bR/A2/tZifmIEPMbqVxJYyYxUOeyTJfvO/SRGJDuoKNAEtpuYV
+JtvwOm6G6qHEAscrHD9aLrOVfkhUfhZ5Ikf5pehnxKI5J+i2uMGOsT4XS5BCVcMG
+ZDpy0EjNQCQN738fjbCuSWT6N+bURAKZpAayLOaJJzA0ckJwuBLN6+URnYeyT+JL
+/hAXSWC6YSdfGf7ZRzKlQ+P1keG+mnbU2rR8ZQIDAQABAoIBAB3AEgTuTun3uRdy
+K0BrSV1a24kriq35SOnzSCz5HmeqSsOMaYf+Z308R2aa07qcqZjPRnv7Ldqgpxek
+fenWRR2dVnT3wvD5FU5u+1r7YSD5LsP/7rdNzPHOA2K6dgh7nkyeoE6VHMwfOumN
+ebjGVriVNT+SWBr/YDTPdQV/MiChFyR2HgfT9wkDS4rtML1WpaQeNVHSyHNhjCdQ
+CHV8yLJGrPopXs2+DlD/QY3y9i6S6nFFLQCDgZYMW21LgtWIHdQ53xYnfR3qXrB6
+zqq5Qys0y283acRG/jwNlyu4o2K/Z95fqH9+WIn99j89UH89WUy8Ps4EM1yiik1g
+o61e+WECgYEA1jp1qJmwIz0jbV04Sf+IYspsGfdTBYxcQ3KcKfkUMOZ8t8pO7O96
+vY8JozZIyzWrh5yFZGqNBPKMAeBvFbaU3py+vBEHF7fpNsvo7TKXqx539wbThUPO
+uKrtw6+KQOa5vJfi8ASCvbzqD+hDQBseOEqiAGKJSZQy0WHMuImBMzkCgYEAytSw
+BHoMvK0wnJE045HpeM7l30skhHx/NaREUK/k0g/dc74C9Q6fwlDXCorhl+m+G3aX
+djZflOvYh60yajyjTyE5MBA9L2C8GldfjzbYYE02av3pQ+GGoN9wm2Z1FpKBU51m
+AwFme3AxkuObRhhl/7kGrmmzQCFMcijXup1ndo0CgYEAj+teOBJn3oSoVESYGD5v
+Zc8PqLIbOFL/eOB6QUhwZBm0F9T2CP8pukdp9hf1927q+YQRqVBbxMMdg/U/K5sp
+kBRFnxrhZPHM319F8HRYqEEj3vTbDWBsdGuAqEI54zgd97EiaJTfMJDFvkIPS/Xr
+621YQGHyUVgae8f0PNVD0yECgYAampMLw0ZwaPb7UkBxqO3Knrif9ScxCXNWAdnE
+C44jz1jomX+hzO0/e4L5KwYz+sCosYFkS/V1fwnDof/+1z1EX8M0fqzHzhReEB/E
+lRkEaJw+pFc4RBh4eVY3y9SHxWeZJAgBVOmwPyMBdUEkjIa0Z/xsXAja3IM+ZgUF
+wSeU8QKBgQDRv/nbrPc06JFYhGI8U+fE6Y2Qy1k5XQzZ+lAi7zHVmlS3qHpaltgM
+eU6JI34RByAofTppC4AmFI5y8Y/XYiD+yweDQzeI6jgnVyzBM/lNMdPkCtg/MoWt
+lqiAWjrXBHfTUNqtXyf8eQG8YB5Llv0Di3K8JbIS6LA2VCIje+aI0g==
+-----END RSA PRIVATE KEY-----

deployment/vagrant-production/keys/client.pub

@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpvByMaNSGqqVMq3wI+JLsjJmvcAdX9lj+VTmGqLEO1fSs1RQtjSBI02AM4uyTit5+BlurFSVTKRyOFi/gtuaRq/uu7ZyqRBzIZxcy8KW6HitlIJ7WiYKy7xEcJNVjRtH8Db+1mJ+YgQ8xupXEljJjFQ57JMl+879JEYkO6go0AS2m5hUm2/A6bobqocQCxyscP1ous5V+SFR+FnkiR/ml6GfEojkn6La4wY6xPhdLkEJVwwZkOnLQSM1AJA3vfx+NsK5JZPo35tREApmkBrIs5oknMDRyQnC4Es3r5RGdh7JP4kv+EBdJYLphJ18Z/tlHMqVD4/WR4b6adtTatHxl [email protected]

deployment/vagrant-production/keys/database

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpAIBAAKCAQEAsA0K9e7N+A1CvdxVUfM1GWR9rM/Kyo703MRw9TZHdI15QgUU
+VwIGLoNyaGeQJMfbDY6k3QzC1y2DmnVkRgy55Q8LLXN7bfHcgme8Yie1fsK9YNrN
+nzfEbzwsO+8t9tR7IJufWQ36TDl0pskBOuArABFIh22ZMD4iKc1SWPqlZYHWdA4u
+LWlQ5ZmqOhUCCdtYyQoKq/IF2G0ahiYhpEmdf+zmTvHc7BZxz0wdHCPbAc3JMMBQ
+XcjgHEPb2qVqMzx5V0X5WH7w42Q0rYkr69efJJmjj86kL5ZIMrHhNPUjGdRyw6IO
+4WK8osBoOZnCfqr668F8Nso7heOM/f8EeCkJLwIDAQABAoIBAFDurgtBLQ7LidMe
+Z1q1I8P4no3DOMbF0WtsI0GXImJCZ2wiSad2FsFrTD0S3wEJn3osCT5Vj3y8P7PS
+I7bQQuU6dTj/zc5NUcHhTokRGC6d99F+xOhdCsDYOL2Uxtyh8bib0jVuZ5KDF01V
+OmuGgj4Vs/IOQE4dQjdVXHXrQ/3VXx0xsKgVmepNUBeTd1Vvxu6LPxPLcjj3ViC/
+fADuv23sL6kqSCSICDcCYxM2jmEMN3XTHVqBubm/V7x9QaS13EyoehzqzM0R4z7f
+hFtsgrtZJ+NautzgvVq4wcBFATzQXVl08vVN/gJmf4gq92GGGzo+HU3q0C7A+bkC
+0jdJ3YECgYEA3GVcGpmzbXEpVGgkaTcx3DwSGoiJ7tbuTS84F4I5p9uOILX2JDyu
+tIicfVb0YFtnZ7wJ7ggZZUdz9gWV4TIpmZsjDDre/7TJq53z2dQpvA8DLfw1G+KN
+WjNH/o5f00hKgLgNKaApkSiENvvmXlKqLSUOigk84uOXLKXp6MdwD8UCgYEAzH3D
+OY+UJ7RGgl8HckXBCYxn+9ajuOOBHvqR5BAGcJJ2LSYErugYpGeh8W46v+vq/7Ar
+jIAcV6hqDmG62ibOErIr13QMn0V0dz6Zgf20bbCcAYCLRcFP8kA2HfxP9ntUs48q
+M+J0sG+6efzDB54W2xi8Qz1jwrtwmpVKcUFGMGMCgYEAgnwcQL95JLWXw5p43DZK
+ddq22TC3gUmNAvOO4RQ/sGrjxegVV7T5tncfN3IQ8y+KlDcXJe9zoavhMupxp0d8
+q63W3TnJp1jRAaNdqTzcjLPSrRoiE2VZTOsJl+ORRlUJjxXo7RkfIwr65FbVOEj2
+6g2DgyQfATG/4jZEu4XAigUCgYEAnb82ZqT36VkMO24LVH+y1ibV4FRHoE0AQmGR
+QZmLHjQdrd5yJmss90ZqitA5yeu4MN45+fyp4IUuq0GA3uCof4uqfl4rjoIJwiMs
+UDbihOuErTP4PCtP/NTD6oXFOaMSSCHcsIG0+pZ2B7yQxsORZhrCwCwPVR07VERu
+VMfB2XkCgYA1Fp/pThnPQAOjYUftj9LSTj6Zm+IYZKyv3oXiO6IGjc4MWvpRmyj+
+G/gXbCZhL40z0hGIHMdyeu3ddQ+pBSncI0ycccbRGvPbITw9EXs+uzFflaYR/ocf
+v0hOmP/FvhuKH6shk8QjMPYENt5M5dRDZMujqtIgD+p40z98YqIxow==
+-----END RSA PRIVATE KEY-----

deployment/vagrant-production/keys/database.pub

@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwDQr17s34DUK93FVR8zUZZH2sz8rKjvTcxHD1Nkd0jXlCBRRXAgYug3JoZ5Akx9sNjqTdDMLXLYOadWRGDLnlDwstc3tt8dyCZ7xiJ7V+wr1g2s2fN8RvPCw77y321Hsgm59ZDfpMOXSmyQE64CsAEUiHbZkwPiIpzVJY+qVlgdZ0Di4taVDlmao6FQIJ21jJCgqr8gXYbRqGJiGkSZ1/7OZO8dzsFnHPTB0cI9sBzckwwFBdyOAcQ9vapWozPHlXRflYfvDjZDStiSvr158kmaOPzqQvlkgyseE09SMZ1HLDog7hYryiwGg5mcJ+qvrrwXw2yjuF44z9/wR4KQkv [email protected]

deployment/vagrant-virtualbox/README.md

@@ -0,0 +1,12 @@
+# Using Vagrant and Virtualbox 
+
+README's for development and production environments exist. This
+page is solely for building a list of the environment variables 
+that modify how TFB launches your Virtuabox virtual machines. 
+
+| Name                             | Values              | Purpose                  |
+| :------------------------------- | :------------------ | :----------------------- | 
+| `TFB_VB_SHOW`                    | `true,false`        | Show the VM in a window when running? Default is false
+| `TFB_VB_ARCH`                    | `64,32`             | Used to force TFB to run a 32-bit virtual machine. This is unsupported, as many of the framework binaries we download are 64-bit only and will not launch. If you cannot run a 64-bit VM, then we recommend using the Amazon AWS provider instead of using this variable
+| `TFB_VB_MEM`                     | `<number>` e.g. `2048` | Size of VM's RAM in MB. Default is `2048`
+

toolset/benchmark/benchmarker.py

@@ -21,6 +21,9 @@ from datetime import datetime
 # Cross-platform colored text
 from colorama import Fore, Back, Style
 
+# Text-based progress indicators
+import progressbar
+
 class Benchmarker:
 
   ##########################################################################################
@@ -446,6 +449,9 @@ class Benchmarker:
   ############################################################
 
   def __run_tests(self, tests):
+    if len(tests) == 0:
+      return 0
+
     logging.debug("Start __run_tests.")
     logging.debug("__name__ = %s",__name__)
 
@@ -459,8 +465,19 @@ class Benchmarker:
           error_happened = True
     else:
       logging.debug("Executing __run_tests on Linux")
+
+      logging.info("Running %s" % tests)
+      # Setup a nice progressbar and ETA indicator
+      widgets = [self.mode, ': ',  progressbar.Percentage(), 
+                 ' ', progressbar.Bar(),
+                 ' Full ', progressbar.ETA()]
+      pbar = progressbar.ProgressBar(widgets=widgets, maxval=len(tests)).start()
+      pbar_test = 0
+
       # These features do not work on Windows
       for test in tests:
+        pbar.update(pbar_test)
+        pbar_test = pbar_test + 1
         if __name__ == 'benchmark.benchmarker':
           print header("Running Test: %s" % test.name)
           with open('current_benchmark.txt', 'w') as benchmark_resume_file:
@@ -476,6 +493,7 @@ class Benchmarker:
             test_process.join()
           if test_process.exitcode != 0:
             error_happened = True
+      pbar.finish()
     if os.path.isfile('current_benchmark.txt'):
       os.remove('current_benchmark.txt')
     logging.debug("End __run_tests.")
@@ -794,28 +812,27 @@ class Benchmarker:
   # __finish
   ############################################################
   def __finish(self):
-    if self.mode == "verify":
-      tests = self.__gather_tests
-      # Normally you don't have to use Fore.BLUE before each line, but 
-      # Travis-CI seems to reset color codes on newline (see travis-ci/travis-ci#2692)
-      # or stream flush, so we have to ensure that the color code is printed repeatedly
-      prefix = Fore.CYAN
-      for line in header("Verification Summary", top='=', bottom='').split('\n'):
-        print prefix + line
-      for test in tests:
-        print prefix + "| Test: %s" % test.name
-        if test.name in self.results['verify'].keys():
-          for test_type, result in self.results['verify'][test.name].iteritems():
-            if result.upper() == "PASS":
-              color = Fore.GREEN
-            elif result.upper() == "WARN":
-              color = Fore.YELLOW
-            else:
-              color = Fore.RED
-            print prefix + "|       " + test_type.ljust(11) + ' : ' + color + result.upper()
-        else:
-          print prefix + "|      " + Fore.RED + "NO RESULTS (Did framework launch?)"
-      print prefix + header('', top='', bottom='=') + Style.RESET_ALL
+    tests = self.__gather_tests
+    # Normally you don't have to use Fore.BLUE before each line, but 
+    # Travis-CI seems to reset color codes on newline (see travis-ci/travis-ci#2692)
+    # or stream flush, so we have to ensure that the color code is printed repeatedly
+    prefix = Fore.CYAN
+    for line in header("Verification Summary", top='=', bottom='').split('\n'):
+      print prefix + line
+    for test in tests:
+      print prefix + "| Test: %s" % test.name
+      if test.name in self.results['verify'].keys():
+        for test_type, result in self.results['verify'][test.name].iteritems():
+          if result.upper() == "PASS":
+            color = Fore.GREEN
+          elif result.upper() == "WARN":
+            color = Fore.YELLOW
+          else:
+            color = Fore.RED
+          print prefix + "|       " + test_type.ljust(11) + ' : ' + color + result.upper()
+      else:
+        print prefix + "|      " + Fore.RED + "NO RESULTS (Did framework launch?)"
+    print prefix + header('', top='', bottom='=') + Style.RESET_ALL
 
     print "Time to complete: " + str(int(time.time() - self.start_time)) + " seconds"
     print "Results are saved in " + os.path.join(self.result_directory, self.timestamp)

toolset/deployment/README.md

@@ -1,111 +0,0 @@
-# Benchmark Suite Deployment
-
-This directory contains tools and documentation to deploy the benchmark suite on cloud or self-hosted environments.
-
-## Deploment overview
-
-The deployment consists of two phases:
-
-1. Provisioning phase
-2. Setup phase
-
-The provisioning phase varies according to the hosting environment: EC2, Windows Azure, dedicated hardware or other hosting. 
-
-The setup phase is, for the most part, the same for all environments, because the benchmark suite specifies particular versions of operating systems and tools that will be used.
-
-For any hosting option, these are the operating systems and tools that should be provisioned for each role:
-
-**Linux client**
-
-* Runs the benchmark client and the Linux databases (e.g. MySQL, PostgreSQL, MongoDB).
-* Operating system: Ubuntu Server 12.04 LTS 64-bit.
-
-**Linux server**
-
-* Hosts the web frameworks under test on a Linux environment.
-* Operating system: Ubuntu Server 12.04 LTS 64-bit.
-
-**Windows server**
-
-* Hosts the web frameworks under test on a Windows environment.
-* Operating system: Windows Server 2012 Datacenter 64-bit.
-
-**SQL Server**
-
-* Hosts the Microsoft SQL Server database server used by some tests.
-* Operating system: Windows Server 2012 Datacenter 64-bit.
-* Software: SQL Server 2012 SP1 Standard Edition 64-bit.
-
-## EC2 Deployment
-
-To deploy the benchmark suite on Amazon Web Services (AWS) Elastic Compute Cloud (EC2):
-
-**Create the EC2 instances**
-
-Create one instance for each role described in the deploment overview above.
-* For the Windows server, use the "Microsoft Windows Server 2012 Base" image on Amazon EC2.
-* For the SQL Server database server, use the [Windows Server 2012 RTM English 64-bit SQL 2012 Standard](https://aws.amazon.com/amis/amazon-ebs-backed-windows-server-2012-rtm-english-64-bit-sql-2012-standard) image on Amazon EC2.
-
-We tested on m1.large instances, but feel free to experiment with different configurations.
-
-Give the instance that will act as the application server more then the default 8GB of disk capacity (we used 20GB).
-
-**Security Group**
-
-When prompted to create a security group for the instances, here are the ports that you'll need to open:
-
-* 22 (SSH)
-* 80 (php-senthot)
-* 3000 (yesod)
-* 3306 (MySQL)
-* 5432 (PostgreSQL)
-* 8000 (snap, racket-ws, wai, yesod)
-* 8080 (Most of the tests run on 8080)
-* 9000 (Play Framework, Dropwizard, Unfiltered)
-* 9080 (Plain)
-* 16969 (cpoll)
-* 27017 (MongoDB)
-
-## Windows Azure
-
-For Windows Azure deployment instructions see the [Windows Azure Deployment README file](azure/README.md).
-
-## Dedicated hardware
-
-If you have two servers or workstations lying around, then you can install and run the tests on physical hardware.
-
-Please be aware that these setup instructions **can overwrite software and settings**, so it's best to follow these instructions on clean hardware.
-
-**Evaluation downloads**
-
-If you want to run the benchmark on Windows and/or SQL Server, you can download evaluation versions from Microsoft:
-* [Download Windows Server 2012](http://technet.microsoft.com/en-us/evalcenter/hh670538.aspx)
-* [Download SQL Server 2012 SP1](http://www.microsoft.com/betaexperience/pd/SQL2012EvalCTA/enus/default.aspx)
-
-**Prerequisites**
-
-Before you get started, there are a couple of steps you can take to make running the tests easier on yourself.
-
-Since the tests can run for several hours, it helps to set everything up so that once the tests are running, you can leave the machines unattended and don't need to be around to enter ssh or sudo passwords.
-
-1. Setup an ssh key for the client machine.
-2. Edit your sudoers file so that you do not need to enter your password for sudo access.
-
-## Other hosting
-
-The benchmark suite will work on physical or virtual servers that have local network connectivity between them and Internet access for software download.
-
-In general, any private or public hosting can be used, following this procedure:
-
-* Provision the instances (servers) according to the roles, operating system and additional software described in the deploment overview.
-* Configure networking as needed.
-* Set the Linux hosts for passwordless sudo as described in the instructions for dedicated hardware.
-
-Specific provisioning instructions and/or scripts for particular hosting or cloud services providers are welcome!
-
-## Setup
-
-After the servers are provisioned as described above, it's time to deploy the benchmark suite and its dependencies.
-
-* **Automated deployment:** To use the automated deployment script, refer to the [Benchmark Suite Automated Deployment README file](common/README.md).
-* **Manual setup:** To manually execute the deployment steps, refer to the [Benchmark Suite Setup README file](../setup/README.md).

toolset/setup/linux/installer.py

@@ -137,8 +137,13 @@ class Installer:
     sudo apt-get -y install build-essential git libev-dev libpq-dev libreadline6-dev postgresql redis-server
     sudo sh -c "echo '*               -    nofile          65535' >> /etc/security/limits.conf"
 
+    # Create a user-owned directory for our databases
     sudo mkdir -p /ssd
     sudo mkdir -p /ssd/log
+    sudo chown -R $USER:$USER /ssd
+
+    # Additional user account (only use if required)
+    sudo useradd benchmarkdbuser -p benchmarkdbpass
 
     ##############################
     # MySQL
@@ -150,9 +155,9 @@ class Installer:
 
     sudo stop mysql
     # disable checking of disk size
-    sudo cp mysql /etc/init.d/mysql
+    sudo mv mysql /etc/init.d/mysql
     sudo chmod +x /etc/init.d/mysql
-    sudo cp mysql.conf /etc/init/mysql.conf
+    sudo mv mysql.conf /etc/init/mysql.conf
     # use the my.cnf file to overwrite /etc/mysql/my.cnf
     sudo mv /etc/mysql/my.cnf /etc/mysql/my.cnf.orig
     sudo mv my.cnf /etc/mysql/my.cnf
@@ -165,15 +170,18 @@ class Installer:
 
     # Insert data
     mysql -uroot -psecret < create.sql
+    rm create.sql
 
     ##############################
     # Postgres
     ##############################
-    sudo useradd benchmarkdbuser -p benchmarkdbpass
     sudo -u postgres psql template1 < create-postgres-database.sql
     sudo -u benchmarkdbuser psql hello_world < create-postgres.sql
+    rm create-postgres-database.sql create-postgres.sql
 
     sudo -u postgres -H /etc/init.d/postgresql stop
+    # NOTE: This will cause errors on Ubuntu 12.04, as apt installs 
+    # an older version (9.1 instead of 9.3)
     sudo mv postgresql.conf /etc/postgresql/9.3/main/postgresql.conf
     sudo mv pg_hba.conf /etc/postgresql/9.3/main/pg_hba.conf
 
@@ -185,51 +193,56 @@ class Installer:
     # MongoDB
     ##############################
     sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
-    sudo cp 10gen.list /etc/apt/sources.list.d/10gen.list
+    echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
     sudo apt-get -y update
     sudo apt-get -y remove mongodb-clients
     sudo apt-get -y install mongodb-org
 
-    sudo stop mongodb
+    sudo service mongod stop
     sudo mv /etc/mongodb.conf /etc/mongodb.conf.orig
     sudo mv mongodb.conf /etc/mongodb.conf
     sudo cp -R -p /var/lib/mongodb /ssd/
     sudo cp -R -p /var/log/mongodb /ssd/log/
-    sudo start mongodb
+    sudo service mongod start
+
+    until nc -z localhost 27017 ; do echo Waiting for MongoDB; sleep 1; done
+    mongo < create.js
+    rm create.js
 
     ##############################
     # Apache Cassandra
     ##############################
     sudo apt-get install -qqy openjdk-7-jdk
     export CASS_V=2.0.7
-    wget http://archive.apache.org/dist/cassandra/$CASS_V/apache-cassandra-$CASS_V-bin.tar.gz
+    wget -nv http://archive.apache.org/dist/cassandra/$CASS_V/apache-cassandra-$CASS_V-bin.tar.gz
     tar xzf apache-cassandra-$CASS_V-bin.tar.gz
-    rm apache-cassandra-*-bin.tar.gz
-    fuser -k -TERM /ssd/log/cassandra/system.log
-    sleep 5
-    sudo rm -rf /ssd/cassandra /ssd/log/cassandra
-    sudo mkdir -p /ssd/cassandra /ssd/log/cassandra
-    sudo chown tfb:tfb /ssd/cassandra /ssd/log/cassandra
+    
+    rm -rf /ssd/cassandra /ssd/log/cassandra
+    mkdir -p /ssd/cassandra /ssd/log/cassandra
+    
     sed -i "s/^.*seeds:.*/          - seeds: \"%s\"/" cassandra/cassandra.yaml
     sed -i "s/^listen_address:.*/listen_address: %s/" cassandra/cassandra.yaml
     sed -i "s/^rpc_address:.*/rpc_address: %s/" cassandra/cassandra.yaml
-    cp cassandra/cassandra.yaml apache-cassandra-$CASS_V/conf
-    cp cassandra/log4j-server.properties apache-cassandra-$CASS_V/conf
-    pushd apache-cassandra-$CASS_V
-    nohup ./bin/cassandra
+    
+    mv cassandra/cassandra.yaml apache-cassandra-$CASS_V/conf
+    mv cassandra/log4j-server.properties apache-cassandra-$CASS_V/conf
+    nohup apache-cassandra-$CASS_V/bin/cassandra > cassandra.log
+
     sleep 10
-    cat ../cassandra/create-keyspace.cql | ./bin/cqlsh $TFB_DATABASE_HOST
-    python ../cassandra/db-data-gen.py | ./bin/cqlsh $TFB_DATABASE_HOST
-    popd
+    cat cassandra/create-keyspace.cql | apache-cassandra-$CASS_V/bin/cqlsh $TFB_DATABASE_HOST
+    python cassandra/db-data-gen.py | apache-cassandra-$CASS_V/bin/cqlsh $TFB_DATABASE_HOST
+    rm -rf apache-cassandra-*-bin.tar.gz cassandra
 
     ##############################
     # Redis
     ##############################
     sudo service redis-server stop
-    sudo cp redis.conf /etc/redis/redis.conf
+    # NOTE: This will cause errors on Ubuntu 12.04, as apt installs 
+    # an older version of redis
+    sudo mv redis.conf /etc/redis/redis.conf
     sudo service redis-server start
     bash create-redis.sh
-
+    rm create-redis.sh
     """ % (self.benchmarker.database_host, self.benchmarker.database_host, self.benchmarker.database_host)
     
     print("\nINSTALL: %s" % self.benchmarker.database_ssh_string)

toolset/setup/linux/setup_util.py

@@ -44,8 +44,16 @@ def replace_environ(config=None, root=None, print_result=False, command='true'):
 
         # Run command, source config file, and store resulting environment
         setup_env = "%s && . %s && env" % (command, config)
-        env = subprocess.check_output(setup_env, shell=True, env=mini_environ,
-          executable='/bin/bash')
+        env = ""
+        try:
+            env = subprocess.check_output(setup_env, shell=True, env=mini_environ,
+           executable='/bin/bash')
+        except subprocess.CalledProcessError:
+            # Ensure that an error here does not crash the toolset
+            print "CRITICAL: Loading %s returned non-zero exit" % config
+            for key,value in mini_environ.iteritems():
+                os.environ[key]=value
+            return
         for line in env.split('\n'):
             try:
                 key, value = line.split('=', 1)