Commit everything so I can move folders easily

toolset/deployment/amazon/README.md

@@ -81,6 +81,9 @@ destroy`, any later synchronization shouldn't take more than a few seconds.
 
 ## Environmental Variables 
 
+These only have effect if the vagrant aws provider is used. See [here](fii) for variables
+that always take effect, and [here] for variables that take effect if the vagrant virtualbox 
+provider is used. 
 
 | Name                             | Values              | Purpose                  |
 | :------------------------------- | :------------------ | :----------------------- | 
@@ -97,13 +100,18 @@ destroy`, any later synchronization shouldn't take more than a few seconds.
 | `TFB_AWS_EBS_TYPE`               | `gp2,standard,io1`  | The EBS [type](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#block-device-mapping-def). `gp2` is a general-purpose SSD, `standard` is a magnetic drive, `io1` is a guaranteed I/O SSD drive. Our benchmark mode defaults to `gp2`
 | `TFB_AWS_EBS_IO`                 | `<number>`          | Only used if `TFB_AWS_EBS_TYPE=io1`. The number of IO operations per second
 | `TFB_AWS_EBS_DELETE`             | `true,false`        | Should the EBS volume be deleted when `vagrant destroy` happens? 
+| `TFB_AWS_APP_IP`                 | `<ip address>`      | IP address used inside Amazon for the application server. Default is `172.16.0.16`
+| `TFB_AWS_LOAD_IP`                | `<ip address>`      | IP address used inside Amazon for the application server. Default is `172.16.0.17`
+| `TFB_AWS_DB_IP`                  | `<ip address>`      | IP address used inside Amazon for the database server. Default is `172.16.0.18`
+| `TFB_AWS_REGION`                 | `<aws region>`      | Region to launch the instances in. Default is `us-east-1`
 
 <sup>1</sup> Variable is required
 
 <sup>2</sup> Variable can be auto-generated using `setup_aws.py`
 
 **Note:** We do not currently support setting instance-specific variables, like instance type 
-and EBS type, on a per-instance basis. 
+or EBS variables, on a per-instance basis. The only exception is the ip address used for the 
+instances. 
 
 ### Amazon Tips and Tricks
 
@@ -154,11 +162,11 @@ There is an open issue at [mitchellh/vagrant-aws#32](https://github.com/mitchell
 working code for spot instances. This could reduce the amazon cost up to 100x!
 Once this is supported in vagrant-aws it will be supported here. 
 
-**I disagree with the Amazon Setup**:
+**I disagree with the Amazon Setup!**:
 
 You're free to submit a PR to improve the automated deployment scripts. Please
 justify why you feel that your changes provide an improvement, and explain the
-total cost of running a benchmark with your update setup. Also, be aware that 
+total cost of running a benchmark with your update setup. Be aware that 
 you can use `vagrant provision` to selectively re-run your provision scripts,
 instead of having to constantly terminate and launch instances. 
 
@@ -169,8 +177,16 @@ providers simultaneously for the same Vagrantfile. The quick fix is to
 just copy the folder, and run one provider in one folder and one provider
 in another. 
 
+**Can I run multiple benchmarks simultaneously?**:
 
-
+Sure, but you will need to duplicate the directory containing the
+`Vagrantfile` and you will also need to use `TFB_AWS_APP_IP` (and similar 
+variables) to avoid IP address conflicts. You could re-run the `setup_aws.py`
+script and avoid changing the IP addresses, but this creates a new 
+Virtual Private Cloud (VPC), and you are limited to 5 VPC's per region. If 
+you are reading this FAQ, you may be interested in running more than 5 
+simultaneous benchmarks, so the better approach is to just increase all
+the IP addresses by 3 and run the additional benchmarks in the same VPC. 
 
 
 

toolset/deployment/vagrant-multi/Vagrantfile

@@ -8,6 +8,10 @@ check_provider_needs(provider)
 require_relative 'production-env'
 
 Vagrant.configure("2") do |config|
+
+  server_ip = ENV.fetch('TFB_LOAD_IP', '172.16.0.16')
+  client_ip = ENV.fetch('TFB_DB_IP', '172.16.0.17')
+  databa_ip = ENV.fetch('TFB_APP_IP', '172.16.0.18')
   
   # Put the keys inside each box
   Dir['keys/*'].each do |fname|
@@ -19,21 +23,22 @@ Vagrant.configure("2") do |config|
 
   config.vm.define "client" do |client|
     provision_bootstrap(client, "client")
-    provider_aws(client, "loadgen", "172.16.0.17")
-    provider_virtualbox(client, "client", "172.16.0.17")
+    provider_aws(client, "loadgen", client_ip)
+    provider_virtualbox(client, "client", client_ip)
   end
 
   config.vm.define "db" do |db|
     provision_bootstrap(db, "database")
-    provider_aws(db, "database", "172.16.0.18")
-    provider_virtualbox(db, "database", "172.16.0.18")
+    provider_aws(db, "database", databa_ip)
+    provider_virtualbox(db, "database", databa_ip)
   end
 
   # Define the app server as the primary VM
   config.vm.define "app", primary: true do |app|
     provision_bootstrap(app, "server")
-    provider_aws(app, "appserver", "172.16.0.16")
-    provider_virtualbox(app, "server", "172.16.0.16")
+    ENV.fetch
+    provider_aws(app, "appserver", server_ip)
+    provider_virtualbox(app, "server", server_ip)
   end
 
 end

toolset/deployment/vagrant-multi/bootstrap.sh

@@ -6,90 +6,123 @@
 # same script can work for VirtualBox (username vagrant)
 # and Amazon (username ubuntu)
 
+# Set a number of variables by either pulling them from 
+# the existing environment or using the default values
+# I'm renaming them to indicate that (in this script only)
+# the values are provisioner agnostic
+SERVER_IP=${TFB_AWS_APP_IP:=172.16.0.16}
+CLIENT_IP=${TFB_AWS_LOAD_IP:=172.16.0.17}
+DATABA_IP=${TFB_AWS_DB_IP:=172.16.0.18}
 
+GH_REPO=${TFB_AWS_REPO_SLUG:=TechEmpower/FrameworkBenchmarks}
+GH_BRANCH=${TFB_AWS_REPO_BRANCH:=master}
 
-# Setup some nice TFB defaults
-echo "export TFB_SERVER_HOST=172.16.0.16" >> ~/.bash_profile
-echo "export TFB_CLIENT_HOST=172.16.0.17" >> ~/.bash_profile
-echo "export TFB_DATABASE_HOST=172.16.0.18" >> ~/.bash_profile
-echo "export TFB_CLIENT_USER=$USER" >> ~/.bash_profile
-echo "export TFB_DATABASE_USER=$USER" >> ~/.bash_profile
-echo "export TFB_CLIENT_IDENTITY_FILE=$HOME/.ssh/client" >> ~/.bash_profile
-echo "export TFB_DATABASE_IDENTITY_FILE=$HOME/.ssh/database" >> ~/.bash_profile
-echo "export FWROOT=$HOME/FrameworkBenchmarks" >> ~/.bash_profile 
-if [ -e "~/FrameworkBenchmarks/benchmark.cfg" ]; then
-  echo "You have a benchmark.cfg file that will interfere with Vagrant, moving to benchmark.cfg.bak"
-  mv ~/FrameworkBenchmarks/benchmark.cfg ~/FrameworkBenchmarks/benchmark.cfg.bak
-fi
-source ~/.bash_profile
-
-# Setup hosts 
-echo Setting up convenience hosts entries
-echo 172.16.0.18 TFB-database | sudo tee --append /etc/hosts
-echo 172.16.0.17 TFB-client | sudo tee --append /etc/hosts
-echo 172.16.0.16 TFB-server | sudo tee --append /etc/hosts
-
-# Update hostname to reflect our current role
-echo Updating hostname
-echo 127.0.0.1 `hostname` | sudo tee --append /etc/hosts
-myhost=TFB-${1}
-echo $myhost | sudo tee --append /etc/hostname
-sudo hostname $myhost
-echo Updated /etc/hosts file to be: 
-cat /etc/hosts
-
-# Workaround mitchellh/vagrant#289
-echo "grub-pc grub-pc/install_devices multiselect     /dev/sda" | sudo debconf-set-selections
-
-# Install prerequisite tools
-echo "Installing prerequisites"
-sudo apt-get install -y git
-sudo apt-get install -y python-pip
-
-# Initial project setup
-if [ -d "/FrameworkBenchmarks" ]; then
-  ln -s /FrameworkBenchmarks $FWROOT
-  echo "Removing any install or results files so they do not interfere"
-  rm -rf $FWROOT/installs $FWROOT/results
-else
-  # If there is no synced folder, clone the project
-  echo "Cloning project from TechEmpower/FrameworkBenchmarks master"
-  git clone https://github.com/TechEmpower/FrameworkBenchmarks.git $FWROOT
-fi
-sudo pip install -r $FWROOT/config/python_requirements.txt
-
-# Everyone gets SSH access to localhost
-echo "Setting up SSH access to localhost"
-ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
-cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-chmod 600 ~/.ssh/authorized_keys
-
-# Ensure keys have proper permissions
-chmod 600 ~/.ssh/client ~/.ssh/database
-
-# Database and client need to add their specific keys
-# Note : this are always copied from the local working copy using a
-#        file provisioner. While they exist in the git clone we just 
-#        created (so we could use those), we want to let the user
-#        have the option of replacing the keys in their working copy
-#        and ensuring that only they can ssh into the machines
-mykey=~/.ssh/${1}.pub
-if [ -e $mykey ]; then
-  echo "Setting up SSH access for the TFB-server"
-  echo "Using key: "
-  ssh-keygen -lv -f $mykey
-  cat $mykey >> ~/.ssh/authorized_keys
-fi
+# Are we installing the server machine, the client machine, 
+# the database machine, or all machines? 
+ROLE=${1:=all}
+
+# Are we installing in production mode, or development mode?
+# It's odd to have ROLE=all and MODE=prod, but I suppose it 
+# could happen so I'm using two variables instead of one
+MODE=${2:=dev}
+
+# A shell provisioner is called multiple times
+if [ ! -e "~/.firstboot" ]; then
+
+  # Setup some nice TFB defaults
+  echo "export TFB_SERVER_HOST=$SERVER_IP" >> ~/.bash_profile
+  echo "export TFB_CLIENT_HOST=$CLIENT_IP" >> ~/.bash_profile
+  echo "export TFB_DATABASE_HOST=$DATABA_IP" >> ~/.bash_profile
+  echo "export TFB_CLIENT_USER=$USER" >> ~/.bash_profile
+  echo "export TFB_DATABASE_USER=$USER" >> ~/.bash_profile
+  echo "export TFB_CLIENT_IDENTITY_FILE=$HOME/.ssh/client" >> ~/.bash_profile
+  echo "export TFB_DATABASE_IDENTITY_FILE=$HOME/.ssh/database" >> ~/.bash_profile
+  echo "export FWROOT=$HOME/FrameworkBenchmarks" >> ~/.bash_profile 
+  if [ -e "~/FrameworkBenchmarks/benchmark.cfg" ]; then
+    echo "You have a benchmark.cfg file that will interfere with Vagrant, moving to benchmark.cfg.bak"
+    mv ~/FrameworkBenchmarks/benchmark.cfg ~/FrameworkBenchmarks/benchmark.cfg.bak
+  fi
+  source ~/.bash_profile
+
+  # Setup hosts 
+  echo "Setting up convenience hosts entries"
+  echo $DATABA_IP TFB-database | sudo tee --append /etc/hosts
+  echo $CLIENT_IP TFB-client   | sudo tee --append /etc/hosts
+  echo $SERVER_IP TFB-server   | sudo tee --append /etc/hosts
 
-# Setup 
-echo "Running software installation for TFB-${1}"
-cd $FWROOT
-toolset/run-tests.py --verbose --install $1 --install-only --test ''
+  # If we are using development mode, route all remote
+  # queries to the localhost interface
+  if [ "$MODE" == "dev" ]; then
+    echo "Routing database and client to localhost"
+    sudo route add -host $DATABA_IP lo
+    sudo route add -host $CLIENT_IP lo
+    sudo route add -host $SERVER_IP lo
+  fi
 
-# Setup a nice welcome message for our guest
-echo "Setting up welcome message"
-sudo rm -f /etc/update-motd.d/51-cloudguest
-sudo rm -f /etc/update-motd.d/98-cloudguest
-sudo cp /vagrant/custom_motd.sh /etc/update-motd.d/55-tfbwelcome
+  # Update hostname to reflect our current role
+  if [ "$ROLE" != "all" ]; then
+    echo "Updating hostname"
+    echo 127.0.0.1 `hostname` | sudo tee --append /etc/hosts
+    myhost=TFB-${1}
+    echo $myhost | sudo tee --append /etc/hostname
+    sudo hostname $myhost
+    echo Updated /etc/hosts file to be: 
+    cat /etc/hosts
+  fi
 
+  # Workaround mitchellh/vagrant#289
+  echo "grub-pc grub-pc/install_devices multiselect     /dev/sda" | sudo debconf-set-selections
 
+  # Install prerequisite tools
+  echo "Installing prerequisites"
+  sudo apt-get install -y git
+  sudo apt-get install -y python-pip
+
+  # Make project available
+  # If they synced it to /FwBm, just expose it at ~/FwBm
+  # If they didn't sync, we need to clone it
+  if [ -d "/FrameworkBenchmarks" ]; then
+    ln -s /FrameworkBenchmarks $FWROOT
+    echo "Removing installs/ and results/ folders so they do not interfere"
+    rm -rf $FWROOT/installs $FWROOT/results
+  else
+    # If there is no synced folder, clone the project
+    echo "Cloning project from TechEmpower/FrameworkBenchmarks master"
+    git clone -b ${GH_BRANCH} https://github.com/${GH_REPO}.git $FWROOT
+  fi
+  sudo pip install -r $FWROOT/config/python_requirements.txt
+
+  # Everyone gets SSH access to localhost
+  echo "Setting up SSH access to localhost"
+  ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
+  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+  chmod 600 ~/.ssh/authorized_keys
+
+  # Ensure keys have proper permissions
+  chmod 600 ~/.ssh/client ~/.ssh/database
+
+  # Database and client need to add their specific keys
+  # Note : this are always copied from the local working copy using a
+  #        file provisioner. While they exist in the git clone we just 
+  #        created (so we could use those), we want to let the user
+  #        have the option of replacing the keys in their working copy
+  #        and ensuring that only they can ssh into the machines
+  mykey=~/.ssh/$ROLE.pub
+  if [ -e $mykey ]; then
+    echo "Setting up SSH access for the TFB-server"
+    echo "Using key: "
+    ssh-keygen -lv -f $mykey
+    cat $mykey >> ~/.ssh/authorized_keys
+  fi
+
+  # Setup 
+  echo "Installing $ROLE software"
+  cd $FWROOT
+  toolset/run-tests.py --verbose --install $ROLE --install-only --test ''
+
+  # Setup a nice welcome message for our guest
+  echo "Setting up welcome message"
+  sudo rm -f /etc/update-motd.d/51-cloudguest
+  sudo rm -f /etc/update-motd.d/98-cloudguest
+  sudo cp /vagrant/custom_motd.sh /etc/update-motd.d/55-tfbwelcome
+fi

toolset/deployment/vagrant-multi/production-env.rb

@@ -1,5 +1,4 @@
 
-
 def provision_bootstrap(config, role)
   config.vm.provision "shell" do |sh|
     sh.path = "bootstrap.sh"

toolset/deployment/vagrant/Vagrantfile

@@ -7,14 +7,22 @@ provider = get_provider
 check_provider_needs(provider)
 
 Vagrant.configure("2") do |config|
-  config.vm.hostname = "TFB"
-  config.vm.box = "ubuntu/trusty64"
-
-  # Add settings for advanced users doing development
-  config.vm.synced_folder "../../..", "/FrameworkBenchmarks"
-  config.vm.network :forwarded_port, guest: 8080, host: 28080
+  
+  config.vm.provision :shell do |sh|
+    sh.path = "bootstrap.sh"
+    
+    # We run the bootstrap as the normal vagrant user, not as root
+    sh.privileged = false
+    
+    # If you need arguments, you must pass them in, becuase Vagrant 
+    # cleans the environment before running the provision script
+    # sh.args = ""
+  end
 
   config.vm.provider :virtualbox do |vb, override|
+    override.vm.hostname = "TFB"
+    override.vm.box = "ubuntu/trusty64"
+
     vb.customize ["modifyvm", :id, "--memory", "2048"]
     if ENV['TFB_SHOW_VM'] and ENV['TFB_SHOW_VM'] == "true"
       vb.gui = true
@@ -28,9 +36,12 @@ Vagrant.configure("2") do |config|
     # 192.168.X.X or 10.X.X.X and we cannot have a collision 
     # with the host network
     override.vm.network "private_network", ip: "172.16.16.16"
+
+    # Add settings for development
+    override.vm.synced_folder "../../..", "/FrameworkBenchmarks"
+    override.vm.network :forwarded_port, guest: 8080, host: 28080
   end
 
-  # Requires the vagrant-aws plugin from github.com/mitchellh/vagrant-aws
   config.vm.provider :aws do |aws, override|
     aws.access_key_id = ENV['TFB_AWS_ACCESS_KEY'] 
     aws.secret_access_key = ENV['TFB_AWS_SECRET_KEY']
@@ -79,14 +90,4 @@ Vagrant.configure("2") do |config|
     end
   end
   
-  #config.vm.provision :shell do |sh|
-  #  sh.path = "bootstrap.sh"
-  #  
-  #  # We run the bootstrap as the normal vagrant user, not as root
-  #  sh.privileged = false
-  ##  
-  #  # If you need arguments, you must pass them in, becuase Vagrant 
-  #  # cleans the environment before running the provision script
-  #  # sh.args = ""
-  #end
 end

toolset/deployment/virtualbox/README.md

@@ -8,4 +8,5 @@ that modify how TFB launches your Virtuabox virtual machines.
 | :------------------------------- | :------------------ | :----------------------- | 
 | `TFB_VB_SHOW`                    | `true,false`        | Show the VM in a window when running? Default is false
 | `TFB_VB_ARCH`                    | `64,32`             | Used to force TFB to run a 32-bit virtual machine. This is unsupported, as many of the framework binaries we download are 64-bit only and will not launch. If you cannot run a 64-bit VM, then we recommend using the Amazon AWS provider instead of using this variable
+| `TFB_VB_MEM`                     | `<number>` e.g. `2048` | Size of VM's RAM in MB. Default is `2048`