Tons of README updates

toolset/deployment/amazon/README.md

@@ -0,0 +1,178 @@
+# Automated Deployment Onto Amazon Web Services
+
+Scripts exist to setup either a development environment, or a benchmark-ready
+production environment. The development environment uses one small instance to 
+run all three TFB roles (app server, database server, client load generation). 
+The production environment uses three different instances networked together. 
+There are a number of environmental variables listed below that allow you to 
+change features such as instance type, availability region, etc. 
+
+We use [Vagrant](https://www.vagrantup.com) to launch/terminate EC2 instances, 
+and we provide a python script to configure the network as needed. 
+
+All amazon resources are tagged with `Project=FrameworkBenchmarks` and 
+`TFB_Role=[database|network|client|app]` so you can easily get a breakdown on 
+your bill. A default development setup costs about `$1/day`, while a default
+production setup costs about TODO. 
+
+## Prerequisites
+
+* **A recent version of Vagrant**, like 1.6.3 (NOTE: `apt-get` is 
+too old, download the newest `deb` directly). See 
+[here](https://www.vagrantup.com/downloads.html) for downloads
+
+* **Vagrant AWS Plugin** from [here](https://github.com/mitchellh/vagrant-aws)
+
+* **Python 2.7** to run the amazon setup script
+
+* **Amazon Command Line Interface**: From [here](https://github.com/aws/aws-cli)
+
+## Using Vagrant to Run Amazon-powered Virtual Machine
+
+The high level steps are 
+1) clone this project 
+2) set environment variables allowing us to log into your amazon account
+3) Run amazon setup script to create network
+4) Run `vagrant up --provider=aws` to launch into amazon
+5) Run `vagrant ssh` to log into the application server
+
+By default, your local git clone of this project will not by synced with the 
+remote amazon machines.
+
+## Required Environment Variables
+
+These environment variables are always required: 
+
+    TFB_AWS_ACCESS_KEY : Your Amazon Web Services Access Key
+    TFB_AWS_SECRET_KEY : Your Amazon Web Services Secret Access Key
+    TFB_AWS_KEY_NAME   : The name of the keypair you are using
+    TFB_AWS_KEY_PATH   : Path to the *.pem file for the keypair you are using
+
+You can declare these in your `~/.bash_profile`, using lines like:
+
+    export TFB_AWS_ACCESS_KEY="<your access key>"
+
+Or you can specify them on the command line as so: 
+
+    $ TFB_AWS_ACCESS_KEY="<your key>" TFB_AWS_SECRET_KEY="<your secret>" TFB_AWS_KEY_NAME="<your keypair name>" TFB_AWS_KEY_PATH="/home/you/.ssh/your_key.pem" vagrant up --provider aws
+
+## Directory Syncing
+
+By default, our Vagrant setup will not syncronize your local git working copy 
+with the remote filesystem on Amazon. Instead, the `~/FrameworkBenchmarks` 
+folder will be a fresh clone of `github.com/TechEmpower/FrameworkBenchmarks`
+
+This is due to the project currently being multiple GB in size (see #1050). 
+Synchronizing the local filesystem with the remote filesystem would require an
+initial upload to the remote system, and most people do not want to wait for 
+a multi-GB upload. 
+
+If you are interested in having this synchronization, you can use set an 
+environment variable as so: 
+
+    $ TFB_FORCE_SYNC=true vagrant up --provider aws
+
+This will tell vagrant to keep your git working copy in sync with the remote
+filesystem. While the initial upload is quite arduous, any later changes are
+synchronized quite quickly (the underlying system is rsync). This would 
+allow you to continue editing files on your local machine and then using the 
+remote virtual machine to test your edits. As long as you don't run `vagrant 
+destroy`, any later synchronization shouldn't take more than a few seconds. 
+
+## Environmental Variables 
+
+
+| Name                             | Values              | Purpose                  |
+| :------------------------------- | :------------------ | :----------------------- | 
+| <sup>1</sup>`TFB_AWS_ACCESS_KEY` | `<your_key>`        |  Your Amazon Web Services Access Key 
+| <sup>1</sup>`TFB_AWS_SECRET_KEY` | `<your_key>`        |  Your Amazon Web Services Secret Access Key 
+| <sup>1</sup>`TFB_AWS_KEY_NAME`   | `<key_name>`        |  The name of the key pair you are using 
+| <sup>1</sup>`TFB_AWS_KEY_PATH`   | `<path_to_keyfile>` |  Path to the *.pem file for the keypair you are using 
+| `TFB_FORCE_SYNC`                 | `true,false`        | Tells vagrant to rsync your local git clone with the remote amazon instance. Note: This requires an initial multi-GB upload 
+| `TFB_AWS_REPO_SLUG`              | default is `TechEmpower/FrameworkBenchmarks`  | If `TFB_FORCE_SYNC` is false, this defines the repository that will be checked out from Github on the Amazon instances
+| `TFB_AWS_REPO_BRANCH`            | default is `master`  | If `TFB_FORCE_SYNC` is false, this defines the repository branch that will be checked out from Github on the Amazon instances
+| `TFB_AWS_EC2_TYPE`               | `m1.small,c1.xlarge,etc` | The instance type used. Requires [previous generation instance](http://aws.amazon.com/ec2/previous-generation/) names because vagrant-aws does not yet support the new names
+| <sup>2</sup>`TFB_AWS_SUBNET`     | `<subnet_id>` (e.g. `subnet-2737230f`) | Which subnet instances should be launched into. The python setup script can generate this value for you
+| <sup>2</sup>`TFB_AWS_SEC_GROUP`  | `<security_group_id>` (e.g. `sg-871240e2`) | The default security group instances use
+| `TFB_AWS_EBS_TYPE`               | `gp2,standard,io1`  | The EBS [type](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#block-device-mapping-def). `gp2` is a general-purpose SSD, `standard` is a magnetic drive, `io1` is a guaranteed I/O SSD drive. Our benchmark mode defaults to `gp2`
+| `TFB_AWS_EBS_IO`                 | `<number>`          | Only used if `TFB_AWS_EBS_TYPE=io1`. The number of IO operations per second
+| `TFB_AWS_EBS_DELETE`             | `true,false`        | Should the EBS volume be deleted when `vagrant destroy` happens? 
+
+<sup>1</sup> Variable is required
+
+<sup>2</sup> Variable can be auto-generated using `setup_aws.py`
+
+**Note:** We do not currently support setting instance-specific variables, like instance type 
+and EBS type, on a per-instance basis. 
+
+### Amazon Tips and Tricks
+
+**Halt Amazon instances to save money**
+
+You can use `vagrant stop` with the AWS provider, and using 
+`vagrant up` at a later date to restart the virtual machine. You will 
+continue to be charged for the block storage (about $0.05 per month per 
+GB), but you will not be charged for the CPU required to run the VM (about
+$5 per month). With our normal setup, this block storage cost is about 
+$0.40 per month (we currently use an 8GB volume)
+
+**Use Disk Snapshots For Rapid Launching**
+
+If you hate waiting 20 minutes for the virtual machine to initially
+be ready, just snapshot it after the first `vagrant up`. You'll have to 
+pay around $0.10 per month per GB to maintain this snapshot, but you can 
+use the snapshot as the disk image when starting a new virtual machine and 
+have a AWS development environment ready in seconds. 
+
+This trick is of limited utility if you are just doing the initial setup, 
+but it can become very valuable if you have run a **full** installation of 
+server software (e.g. `toolset/run-tests.py --install server --install-only`), 
+as this takes multiple hours to complete. 
+
+### FAQs
+
+**How much does running on Amazon cost?**: 
+
+Roughly `$1/day` with default settings.  
+
+Running `vagrant up --provider aws` takes about 20 minutes to complete. Running 
+`toolset/run-tests.py --install server --mode verify --test <foo>` takes between
+5 and 60 minutes, depending on `<foo>`. [Prices as of writing](http://aws.amazon.com/ec2/previous-generation/) show the default `m1.small` instance type is 
+`$0.044` per Hour, or roughly `$1/day`. You are charged for a full hour even
+if you only use a few minutes, so if you contantly cycle between `vagrant up`
+and `vagrant destroy` you will be charged `$0.044` per cycle. 
+
+Running `toolset/run-tests.py --mode benchmark --test <foo>` takes between 15-60
+minutes per test in `<foo>`, and running the full benchmark would likely take 
+between 1 and 5 days of continuous execution. Note that this single VM runs the 
+framework, the load generation, and the database, and is therefore not a stable 
+benchmarking setup. 
+
+**Are spot instances supported?**:
+
+There is an open issue at [mitchellh/vagrant-aws#32](https://github.com/mitchellh/vagrant-aws/issues/32) that appears to have 
+working code for spot instances. This could reduce the amazon cost up to 100x!
+Once this is supported in vagrant-aws it will be supported here. 
+
+**I disagree with the Amazon Setup**:
+
+You're free to submit a PR to improve the automated deployment scripts. Please
+justify why you feel that your changes provide an improvement, and explain the
+total cost of running a benchmark with your update setup. Also, be aware that 
+you can use `vagrant provision` to selectively re-run your provision scripts,
+instead of having to constantly terminate and launch instances. 
+
+**Can I use VirtualBox and AWS Simultaneously?**:
+
+Not out of the box. Vagrant does not currently support using two 
+providers simultaneously for the same Vagrantfile. The quick fix is to 
+just copy the folder, and run one provider in one folder and one provider
+in another. 
+
+
+
+
+
+
+
+Use SSD on benchmark server

toolset/deployment/vagrant/README.md

@@ -1,38 +1,39 @@
-# Vagrant Automated Deployment
+# Setting up a Development Environment Using Vagrant
 
-Currently, our [Vagrant](https://www.vagrantup.com) setup can automatically
-setup a development environment running in a single virtual machine. 
-This virtual machine can run inside either [VirtualBox](https://www.virtualbox.org/) or [Amazon EC2](http://aws.amazon.com/ec2/). Feel free to submit a PR 
-to help us support more providers, such as [rackspace](http://www.rackspace.com/)
-or [VMware](http://www.vmware.com/). 
+The standard development environment is a single computer that fills the 
+role of database server, load generation server, and framework server. 
+While this is not useful for serious benchmarking, it does allow 
+painless development and testing. Using a virtual machine as a 
+development environment prevents having to install hundreds of 
+framework dependencies directly onto your computer. 
 
-This single VM runs the `app server`, `client server`, and `database server`, 
-and allows you to rapidly test if your framework/framework changes integrate
-properly with TFB. While you *can* run a full benchmark using 
-this virtual machine, your results would be quite useless. 
+We provide a [Vagrant](https://www.vagrantup.com) script to automatically 
+setup a virtual machine as a development server. Currently, you can use 
+either [VirtualBox](https://www.virtualbox.org/) or [Amazon EC2](http://aws.amazon.com/ec2/)
+to run this development environment. This page details how to use 
+VirtualBox, please read [here](blah) for information on using Amazon.
+
+Feel free to submit a PR to help us support more providers, such as 
+[Rackspace](http://www.rackspace.com/) or [VMware](http://www.vmware.com/). 
 
 ## Prerequisites
 
 * **A recent version of Vagrant**, like 1.6.3 (NOTE: `apt-get` is 
 too old, download the newest `deb` directly). See 
-[here](https://www.vagrantup.com/downloads.html) for downloads.
+[here](https://www.vagrantup.com/downloads.html) for downloads
 
 * **A CPU that can virtualize a 64-bit virtual OS**, because TFB
 downloads a number of static binaries that require 64-bit. See
-the FAQs section below for more on this
-
-* **VirtualBox**, if you plan to run the Virtual Machine locally. If 
-you plan to solely use Amazon-based Virtual Machines you can skip this. 
+the FAQs section below for more on this. If you cannot meet this 
+requirement, consider using the Amazon provider (about `$1/day`)
 
-* **Vagrant AWS Plugin** from [here](https://github.com/mitchellh/vagrant-aws),
-if you plan to run the Virtual Machine on Amazon's Web Services (e.g. use 
-EC2). If you plan to solely use VirtualBox you can skip this. 
+* **VirtualBox** to run the virtual machines
 
-## Using Vagrant to Run VirtualBox-powered Virtual Machine
+## Launching VirtualBox Development Environment
 
 In short, you need to clone the project and then run `vagrant up` 
 followed by `vagrant ssh`. Your cloned working directory will be 
-exposed inside the VM at `~/FrameworkBenchmarks`, so you can 
+exposed inside the virtual machine at `~/FrameworkBenchmarks`, so you can 
 continue to edit files using your favorite IDE and the changes will
 show up inside the virtual machine. 
 
@@ -59,7 +60,23 @@ This Vagrant environment is already setup and ready to go, so you
 can ignore any flags about users, hosts, or identity files
 ```
 
-## Using Vagrant to Run Amazon-powered Virtual Machine
+## Using VirtualBox Development Environment
+
+After using `vagrant ssh`, you will find the project at `~/FrameworkBenchmarks`. 
+You can use `run-tests.py` without specifying any information about hosts, 
+usernames, or private key files, as so: 
+
+    $ cd ~/FrameworkBenchmarks
+    $ toolset/run-tests.py --mode verify --install server --test go
+    <log for installing software needed for go test>
+    <log for launching go framework>
+    <log for verifying go framework meets requirements>
+    $ cat results/ec2/latest/logs/go/out.txt
+    <log with stdout from framework go>
+
+# Using Amazon-powered Virtual Machine
+
+Please read the document [here]
 
 The high level steps are similar, run `vagrant up --provider=aws` followed 
 by `vagrant ssh`. The main differences are 1) you need to provide a 
@@ -67,47 +84,6 @@ number of environment variables to let vagrant log into your AWS account,
 and 2) your git working copy will not be synced with the remote virtual 
 machine (by default). 
 
-#### Required Environment Variables for Using Amazon Web Services
-
-These are all required: 
-
-    TFB_AWS_ACCESS_KEY : Your Amazon Web Services Access Key
-    TFB_AWS_SECRET_KEY : Your Amazon Web Services Secret Access Key
-    TFB_AWS_KEY_NAME   : The name of the keypair you are using
-    TFB_AWS_KEY_PATH   : Path to the *.pem file for the keypair you are using
-
-You can declare these in your `~/.bash_profile`, using lines like:
-
-    export TFB_AWS_ACCESS_KEY="<your access key>"
-
-Or you can specify them on the command line as so: 
-
-    $ TFB_AWS_ACCESS_KEY="<your key>" TFB_AWS_SECRET_KEY="<your secret>" TFB_AWS_KEY_NAME="<your keypair name>" TFB_AWS_KEY_PATH="/home/you/.ssh/your_key.pem" vagrant up --provider aws
-
-#### Directoy Syncing and Amazon Web Services
-
-By default, our Vagrant setup will not syncronize your local git working copy 
-with the remote filesystem on Amazon. Instead, the `~/FrameworkBenchmarks` 
-folder will be a fresh clone of github.com/TechEmpower/FrameworkBenchmarks. 
-
-This is due to the project currently being multiple GB in size (see #1050). 
-Synchronizing the local filesystem with the remote filesystem would require an
-initial upload to the remote system, and most people do not want to wait for 
-a multi-GB upload. 
-
-If you are interested in having this synchronization, you can use set an 
-environment variable as so: 
-
-    $ TFB_FORCE_SYNC=true vagrant up --provider aws
-
-This will tell vagrant to keep your git working copy in sync with the remote
-filesystem. While the initial upload is quite arduous, any later changes are
-synchronized quite quickly (the underlying system is rsync). This would 
-allow you to continue editing files on your local machine and then using the 
-remote virtual machine to test your edits. As long as you don't run `vagrant 
-destroy`, any later synchronization shouldn't take more than a few seconds. 
-
-
 ### FAQs
 
 **I'm using a 32-bit computer, can I run your Virtual Machine?**: 
@@ -130,24 +106,3 @@ To force FrameworkBenchmarks to use a 32-bit VM, do this:
     $ TFB_ARCH=32 vagrant up
 
 See [here](http://askubuntu.com/questions/41550) for some helpful information.  
-
-**How much does running on Amazon cost?**: 
-
-Roughly `$1/day` with default settings. 
-
-Running `vagrant up --provider aws` takes about 20 minutes to complete. Running 
-`toolset/run-tests.py --install server --mode verify --test <foo>` takes between
-5 and 60 minutes, depending on `<foo>`. [Prices as of writing](http://aws.amazon.com/ec2/previous-generation/) show the default `m1.small` instance type is 
-`$0.044` per Hour, or roughly `$1/day`
-
-Running `toolset/run-tests.py --mode benchmark --test <foo>` takes between 15-60
-minutes per test in `<foo>`, and running the full benchmark would likely take 
-between 1 and 5 days of continuous execution. Note that this single VM runs the 
-framework, the load generation, and the database, and is therefore not a stable 
-benchmarking setup. 
-
-**Are spot instances supported?**: 
-
-There is an open issue at [mitchellh/vagrant-aws#32](https://github.com/mitchellh/vagrant-aws/issues/32) that appears to have 
-working code for spot instances. This could reduce the amazon cost up to 100x!
-Once this is supported in vagrant-aws it will be supported here. 

toolset/deployment/virtualbox/README.md

@@ -0,0 +1,11 @@
+# Using Vagrant and Virtualbox 
+
+README's for development and production environments exist. This
+page is solely for building a list of the environment variables 
+that modify how TFB launches your Virtuabox virtual machines. 
+
+| Name                             | Values              | Purpose                  |
+| :------------------------------- | :------------------ | :----------------------- | 
+| `TFB_VB_SHOW`                    | `true,false`        | Show the VM in a window when running? Default is false
+| `TFB_VB_ARCH`                    | `64,32`             | Used to force TFB to run a 32-bit virtual machine. This is unsupported, as many of the framework binaries we download are 64-bit only and will not launch. If you cannot run a 64-bit VM, then we recommend using the Amazon AWS provider instead of using this variable
+