Toolset directory reorganization

README.md

@@ -8,170 +8,9 @@ Join in the conversation at our Google Group: https://groups.google.com/forum/?f
 
 ## Running the test suite
 
-We ran our tests using two dedicated i7 2600k machines as well as two EC2 m1.large instances. Below you will find instructions on how to replicate our tests using either EC2 or your own dedicated machines.
+We ran our tests using two dedicated i7 2600k machines as well as two EC2 m1.large instances.
 
-###EC2 Instructions
-
-#### 1. Create EC2 Instances
-
-Create two EC2 instances running Ubuntu Server 12.04.1 LTS 64-bit. We tested on m1.large instances, but feel free to experiment with different configurations. Give the instance that will act as the application server more then the default 8GB of disk capacity (we used 20GB).
-
-##### Security Group
-
-When propmted to create a security group for the instances, here are the ports that you'll need to open.
-
-* 22 (SSH)
-* 8080 (Most of the tests run on 8080)
-* 3306 (MySQL)
-* 5432 (PostgreSQL)
-* 9000 (Play Framework)
-* 27017 (MongoDB)
-* 3000 (yesod)
-* 8000 (snap)
-* 16969 (cpoll)
-
-
-#### 2. Setting up the servers
-
-To coordinate the tests via scripting, the servers need to be able to work together. So once the instances are running, the first thing you'll want to do is copy your ssh key to the application server instance so that you can ssh between the two machines:
-
-	sftp -i path-to-pem-file ubuntu@server-instance-ip
-	put path-to-pem-file .ssh/
-	exit
-
-Now ssh into the server instance and clone the latest from this repository (the scripts we use to run the tests expect that you'll clone the repository into your home directory):
-
-	ssh -i path-to-pem-file ubuntu@server-instance-ip
-	yes | sudo apt-get install git-core
-	git clone https://github.com/TechEmpower/FrameworkBenchmarks.git
-	cd FrameworkBenchmarks
-
-Next, we're going to setup the servers with all the necessary software:
-
-	./run-tests.py -s server-private-ip -c client-private-ip -i path-to-pem --install-software --list-tests
-    source ~/.bash_profile
-    # For your first time through the tests, set the ulimit for open files
-    ulimit -n 8192
-    # Most software is installed automatically by the script, but running the mongo command below from 
-    # the install script was causing some errors. For now this needs to be run manually.
-    cd installs/jruby-rack && rvm jruby-1.7.4 do jruby -S bundle exec rake clean gem SKIP_SPECS=true
-    cd target && rvm jruby-1.7.4 do gem install jruby-rack-1.2.0.SNAPSHOT.gem
-    cd ../../..
-    cd installs && curl -sS https://getcomposer.org/installer | php -- --install-dir=bin
-    cd ..
-    sudo apt-get remove --purge openjdk-6-jre openjdk-6-jre-headless
-    mongo --host client-private-ip < config/create.js
-
-Assuming the above finished without error, we're ready to start the test suite:
-
-	nohup ./run-tests.py -s server-private-ip -c client-private-ip -i path-to-pem --max-threads number-of-cores &
-
-For the number-of-cores parameter, you will need to know your application server's core count. For example, Amazon EC2 large instances have 2 cores.
-
-This script will run the full set of tests. Results of all the tests will output to ~/FrameworkBenchmarks/results/ec2/*timestamp*. If you use a different configuration than two m1.large instances, please use the --name option to name the results appropriately.
-
-	nohup ./run-tests.py -s server-private-ip -c client-private-ip -i path-to-pem --max-threads cores --name ec2-servertype-clienttype &
-
-So if you were running an m1.large and an m1.medium, it would look like this:
-
-	nohup ./run-tests.py -s server-private-ip -c client-private-ip -i path-to-pem --max-threads cores --name ec2-m1.large-m1.medium &
-
-This will allow us to differentiate results.
-
-Be aware that on Large instances, if you include the slower frameworks (and they are included by default), the total runtime of a full suite of tests can be measured in days, not just hours. The EC2 bill isn't going to break the bank, but it's also not going to be chump change.
-
-### Dedicated Hardware Instructions
-
-If you have two servers or workstations lying around, then you can install and run the tests on physical hardware. Please be aware that these setup instructions can overwrite software and settings, It's best to follow these instructions on clean hardware. We assume that both machines are running Ubuntu Server 12.04 64-bit.
-
-#### 1. Prerequisites
-
-Before you get started, there are a couple of steps you can take to make running the tests easier on yourself. Since the tests can run for several hours, it helps to set everything up so that once the tests are running, you can leave the machines unattended and don't need to be around to enter ssh or sudo passwords.
-
-1. Setup an ssh key for the client machine
-2. Edit your sudoers file so that you do not need to enter your password for sudo access
-
-#### 2. Setting up the servers
-
-As it currently stands, the script that runs the tests makes some assumptions about where the code is placed, we assume that the FrameworkBenchmarks repository will be located in your home directory.
-
-Check out the latest from github:
-
-	cd ~
-	git clone https://github.com/TechEmpower/FrameworkBenchmarks.git
-	cd FrameworkBenchmarks
-
-Next, we're going to setup the servers with all the necessary software:
-
-	./run-tests.py -s server-ip -c client-ip -i path-to-ssh-key --install-software --list-tests
-    source ~/.bash_profile
-    # For your first time through the tests, set the ulimit for open files
-    # Most software is installed automatically by the script, but running the mongo command below from
-    # the install script was causing some errors. For now this needs to be run manually.
-    cd installs/jruby-rack && rvm jruby-1.7.4 do jruby -S bundle exec rake clean gem SKIP_SPECS=true
-    cd target && rvm jruby-1.7.4 do gem install jruby-rack-1.2.0.SNAPSHOT.gem
-    cd ../../..
-    cd installs && curl -sS https://getcomposer.org/installer | php -- --install-dir=bin
-    cd ..
-    sudo apt-get remove --purge openjdk-6-jre openjdk-6-jre-headless
-    mongo --host client-ip < config/create.js
-
-Assuming this finished without error, we're ready to start the test suite:
-
-	nohup ./run-tests.py -s server-ip -c client-ip -i path-to-ssh-key --max-threads cores --name unique-machine-name &
-
-This will run the full set of tests. Results of all the tests will output to ~/FrameworkBenchmarks/results/unique-machine-name/*timestamp*.
-
-### Windows Instructions
-Generously provided by [@pdonald](https://github.com/pdonald)
-
-Server installation scripts for Windows Server 2012 R2 on Amazon EC2.
-
-Instructions:
-
-* Create an instance from the `Microsoft Windows Server 2012 Base` image on Amazon EC2
-* Connect to it via Remote Desktop
-* Copy `installer-bootstrap.ps1` from this repo to the server (for files CTRL-C + CTRL-V works alright)
-* Copy your client private key too while you're at it
-* Right click on the installer script and select `Run with PowerShell`
-* It will ask something, just hit enter
-* It will install git and then launch `installer.ps1` from the repo which will install everything else
-* Installation shouldn't take more than 5 to 10 minutes
-* Then you have a working console: try `python`, `git`, `ssh`, `curl`, `node` etc. everything works + PowerShell goodies
-
-The client/database machine is still assumed to be a Linux box, you can install just the client software via
-
-    python run-tests.py -s server-ip -c client-ip -i "C:\Users\Administrator\Desktop\client.key" --install-software --install client --list-tests
-
-Now you can run tests:
-
-    python run-tests.py -s server-ip -c client-ip -i "C:\Users\Administrator\Desktop\client.key" --max-threads 2 --duration 30 --sleep 5 --name win --test aspnet --type all
-
-
-### SQL Server on Windows Instructions
-
-Server installation scripts for Windows Server 2012 with SQL Server 2012 Standard on Amazon EC2.
-
-Instructions:
-
-* Create an instance from the [Windows Server 2012 RTM English 64-bit SQL 2012 Standard](https://aws.amazon.com/amis/amazon-ebs-backed-windows-server-2012-rtm-english-64-bit-sql-2012-standard) image on Amazon EC2
-* Connect to it via Remote Desktop
-* Run a `Command Prompt` as Administrator
-* Enter `powershell -ExecutionPolicy Bypass -Command "iex (New-Object Net.WebClient).DownloadString('https://raw.github.com/TechEmpower/FrameworkBenchmarks/master/setup-sqlserver-bootstrap.ps1')"`
-* This will configure SQL Server, the Windows Firewall, and populate the database.
-
-Now, when running `python run-tests.py`, just add `-d <ip of SQL Server instance>`. This works for the (Windows Server-based) `aspnet-sqlserver-raw` and `aspnet-sqlserver-entityframework` tests.
-
-
-## Result Files
-
-After a test run, the directory ~/FrameworkBenchmarks/results/machine-name/timestamp will contains all the result files. In this folder are four files: three CSV files, one for each of the test types (json, db, query), and a single results.json file that contains all the results as well as some additional information. The results.json file is what we use to drive our blog post, and may or may not be useful to you. There are three subdirectories: one for each of the test types (json, db, query), each of these directories contain the raw weighttp results for each framework.
-
-## Benchmarking a Single Test
-
-If you are making changes to any of the tests, or you simply want to verify a single test, you can run the script with the --test flag. For example, if you only wanted to run the JRuby tests:
-
-	nohup ./run-tests.py -s server-ip -c client-ip -i path-to-ssh-key --max-threads cores --name unique-machine-name --test rack-jruby sinatra-jruby rails-jruby
+On the [Toolset README file](toolset/README.md) you will find tools and instructions to replicate our tests using either EC2 or your own dedicated machines.
 
 ## Updating Tests
 

toolset/README.md

@@ -0,0 +1,61 @@
+# Benchmark Tools
+
+This directory contains tools and instructions that you can use to deploy the benchmark suite on an environment under your control.
+
+## Deployment
+
+For deployment instructions, refer to the [Toolset Deployment README file](deployment/README.md).
+
+## Running the test suite on Linux
+
+Assuming the deployment finished without error, we're ready to start the test suite. On the Linux server host, type:
+
+	nohup toolset/run-tests.py -s server-private-ip -c client-private-ip -i path-to-ssh-key --max-threads number-of-cores &
+
+For the number-of-cores parameter, you will need to know your application server's core count. For example, Amazon EC2 large instances have 2 cores.
+
+`path-to-ssh-key` is the pathname of the private SSH key file of the Linux client's user. This file is set up during deployment.
+
+This script will run the full set of tests. Be aware that on Large instances, if you include the slower frameworks (and they are included by default), **the total runtime of a full suite of tests can be measured in days**, not just hours. The EC2 bill isn't going to break the bank, but it's also not going to be chump change.
+
+The test results are saved in the `results` directory, as described below.
+
+## Running the test suite on Windows
+
+To run the test suite on the Windows server, log on the server, open the PowerShell command line environment and type a command like:
+
+    python toolset\run-tests.py -s server-ip -c client-ip -i path-to-ssh-key --max-threads cores --duration 30 --sleep 5 --name win --test aspnet --type all
+    
+## Specifying the machine name
+
+If you use a different configuration than two m1.large instances, please use the --name option to name the results appropriately.
+
+	nohup toolset/run-tests.py -s server-private-ip -c client-private-ip -i path-to-ssh-key --max-threads cores --name ec2-servertype-clienttype &
+
+So if you were running an m1.large and an m1.medium, it would look like this:
+
+	nohup toolset/run-tests.py -s server-private-ip -c client-private-ip -i path-to-ssh-key --max-threads cores --name ec2-m1.large-m1.medium &
+
+This will allow us to differentiate results.
+
+## Benchmarking a Single Test
+
+If you are making changes to any of the tests, or you simply want to verify a single test, you can run the script with the --test flag. For example, if you only wanted to run the JRuby tests:
+
+	nohup toolset/run-tests.py -s server-ip -c client-ip -i path-to-ssh-key --max-threads cores --name unique-machine-name --test rack-jruby sinatra-jruby rails-jruby
+
+## Result Files
+
+After a test run, the directory `~/FrameworkBenchmarks/results/machine-name/timestamp` will contain all the result files.
+
+In this folder there is a directory for each of the test types, each containing the raw results for each framework.
+
+There is also a single `results.json` file that contains all the results as well as some additional information. This file is what we use to drive our blog post, and may or may not be useful to you.
+
+## Additional information
+
+Use `toolset/run-tests.py --help` to see other options that `run-tests.py` recognizes.
+
+## Support
+
+For assistance deploying and running the benchmarks, use our [Google Group](https://groups.google.com/forum/?fromgroups=#!forum/framework-benchmarks).

toolset/benchmark/README.md

@@ -0,0 +1,8 @@
+# Toolset Benchmark Files
+
+This directory contains files that implement or support the benchmark feature:
+
+* `benchmarker.py`: Benchmark execution methods. Called by `run-tests.py`.
+* `framework_test.py`: Methods to run a framework's tests. Called by `benchmarker.py`.
+
+These files are not meant for you to run directly. For instructions on running the benchmark, please refer to the [parent directory's README file](../README.md).

toolset/benchmark/__init__.py

@@ -0,0 +1 @@
+# Benchmark

benchmarker.py → toolset/benchmark/benchmarker.py

@@ -1,6 +1,5 @@
-from installer import Installer
-from framework_test import FrameworkTest
-import framework_test
+from setup.linux.installer import Installer
+from benchmark import framework_test
 
 import os
 import json
@@ -197,8 +196,8 @@ class Benchmarker:
     # Loop through each directory (we assume we're being run from the benchmarking root)
     # and look for the files that signify a benchmark test
     for dirname, dirnames, filenames in os.walk('.'):
-      # Look for the benchmark_config file, this will set up our tests
-      # It's format looks like this:
+      # Look for the benchmark_config file, this will set up our tests.
+      # Its format looks like this:
       #
       # {
       #   "framework": "nodejs",
@@ -490,7 +489,7 @@ class Benchmarker:
     # Load the latest data
     self.latest = None
     try:
-      with open('latest.json', 'r') as f:
+      with open('toolset/benchmark/latest.json', 'r') as f:
         # Load json file into config object
         self.latest = json.load(f)
     except IOError:
@@ -537,5 +536,3 @@ class Benchmarker:
   ############################################################
   # End __init__
   ############################################################
-  
-  

framework_test.py → toolset/benchmark/framework_test.py

@@ -501,7 +501,7 @@ class FrameworkTest:
     self.benchmarker = benchmarker
     self.__dict__.update(args)
 
-    # ensure diretory has __init__.py file so that we can use it as a pythong package
+    # ensure diretory has __init__.py file so that we can use it as a Python package
     if not os.path.exists(os.path.join(directory, "__init__.py")):
       open(os.path.join(directory, "__init__.py"), 'w').close()
 

toolset/deployment/README.md

@@ -0,0 +1,99 @@
+# Toolset Deployment
+
+This directory contains tools and documentation to deploy the benchmark suite on cloud or self-hosted environments.
+
+## Deploment overview
+
+The deployment consists of two phases: provisioning and setup. The provisioning phase varies according to the hosting environment: EC2, dedicated hardware or other hosting. The setup phase is, for the most part, the same for all environments, because the benchmark suite specifies particular versions of operating systems and tools that will be used.
+
+For any hosting option, these are the operating systems and tools that should be provisioned for each role:
+
+#### Linux client
+
+* Runs the benchmark client and the Linux databases (e.g. MySQL, PostgreSQL, MongoDB).
+* Operating system: Ubuntu Server 12.04 LTS 64-bit.
+
+#### Linux server
+
+* Hosts the web frameworks under test on a Linux environment.
+* Operating system: Ubuntu Server 12.04 LTS 64-bit.
+
+#### Windows server
+
+* Hosts the web frameworks under test on a Windows environment.
+* Operating system: Windows Server 2012 Datacenter 64-bit.
+
+#### SQL Server
+
+* Hosts the Microsoft SQL Server database server used by some tests.
+* Operating system: Windows Server 2012 Datacenter 64-bit.
+* Software: SQL Server 2012 SP1 Standard Edition 64-bit.
+
+## Provisioning instructions
+
+These are the instructions for provisioning the benchmark environment on several hosting alternatives:
+
+### EC2
+
+To deploy the benchmark suite on Amazon Web Services (AWS) Elastic Compute Cloud (EC2):
+
+#### Create the EC2 instances
+
+Create one instance for each role described in the deploment overview above.
+* For the Windows server, use the "Microsoft Windows Server 2012 Base" image on Amazon EC2.
+* For the SQL Server database server, use the [Windows Server 2012 RTM English 64-bit SQL 2012 Standard](https://aws.amazon.com/amis/amazon-ebs-backed-windows-server-2012-rtm-english-64-bit-sql-2012-standard) image on Amazon EC2.
+
+We tested on m1.large instances, but feel free to experiment with different configurations.
+
+Give the instance that will act as the application server more then the default 8GB of disk capacity (we used 20GB).
+
+#### Security Group
+
+When prompted to create a security group for the instances, here are the ports that you'll need to open:
+
+* 22 (SSH)
+* 8080 (Most of the tests run on 8080)
+* 3306 (MySQL)
+* 5432 (PostgreSQL)
+* 9000 (Play Framework)
+* 27017 (MongoDB)
+* 3000 (yesod)
+* 8000 (snap)
+* 16969 (cpoll)
+
+### Dedicated hardware
+
+If you have two servers or workstations lying around, then you can install and run the tests on physical hardware.
+
+Please be aware that these setup instructions **can overwrite software and settings**, so it's best to follow these instructions on clean hardware.
+
+#### Evaluation downloads
+
+If you want to run the benchmark on Windows and/or SQL Server, you can download evaluation versions from Microsoft:
+* [Download Windows Server 2012](http://technet.microsoft.com/en-us/evalcenter/hh670538.aspx)
+* [Download SQL Server 2012 SP1](http://www.microsoft.com/betaexperience/pd/SQL2012EvalCTA/enus/default.aspx)
+
+#### Prerequisites
+
+Before you get started, there are a couple of steps you can take to make running the tests easier on yourself.
+
+Since the tests can run for several hours, it helps to set everything up so that once the tests are running, you can leave the machines unattended and don't need to be around to enter ssh or sudo passwords.
+
+1. Setup an ssh key for the client machine.
+2. Edit your sudoers file so that you do not need to enter your password for sudo access.
+
+### Other hosting
+
+The benchmark suite will work on physical or virtual servers that have local network connectivity between them and Internet access for software download.
+
+In general, any private or public hosting can be used, following this procedure:
+
+* Provision the instances (servers) according to the roles, operating system and additional software described in the deploment overview.
+* Configure networking as needed.
+* Set the Linux hosts for passwordless sudo as described in the instructions for dedicated hardware.
+
+Specific provisioning instructions and/or scripts for particular hosting or cloud services providers are welcome!
+
+## Setup instructions
+
+After the servers are provisioned as described above, execute the setup instructions documented at the [Toolset Setup README file](../setup/README.md).

run-tests.py → toolset/run-tests.py

@@ -1,15 +1,22 @@
 #!/usr/bin/env python
 import argparse
 import sys
-from benchmarker import Benchmarker
-from unbuffered import Unbuffered
+from benchmark.benchmarker import Benchmarker
+from setup.linux.unbuffered import Unbuffered
 
 ###################################################################################################
 # Main
 ###################################################################################################
 
+# Enable unbuffered output so messages will appear in the proper order with subprocess output.
 sys.stdout=Unbuffered(sys.stdout)
 
+# Ensure the current directory (which should be the benchmark home directory) is in the path so that the tests can be imported.
+sys.path.append('.')
+
+# Ensure toolset/setup/linux is in the path so that the tests can "import setup_util".
+sys.path.append('toolset/setup/linux')
+
 ##########################################################
 # Set up argument parser
 ##########################################################

toolset/setup/README.md

@@ -0,0 +1,102 @@
+# Toolset Setup
+
+This directory contains scripts to install the benchmark suite on adequately provisioned hosts.
+
+It is composed of these directories:
+
+* `linux`: Scripts to setup the Linux server and the Linux client and database server.
+* `sqlserver`: Scripts to setup the SQL Server database server.
+* `windows`: Scripts to setup the Windows server.
+
+## Prerequisites
+
+Before starting setup, all the required hosts must be provisioned, with the respective operating system and required software installed, and with connectivity for remote management (SSH on Linux, RDP and WinRM on Windows).
+
+Refer to [Toolset Deployment README file](../deployment/README.md) for the provisioning procedures documentation.
+
+## Linux server and client setup
+
+To coordinate the tests via scripting, the servers need to be able to work together. So once the instances are running, the first thing you'll want to do is copy your ssh key to the application server instance so that you can ssh between the two machines:
+
+	sftp -i path-to-ssh-key ubuntu@server-instance-ip
+	put path-to-ssh-key .ssh/
+	exit
+
+This will copy your private ssh key file to ~/.ssh on the Linux server, where it will be used to connect to the benchmark client.
+
+Now, connect via ssh to the Linux server:
+
+    ssh -i path-to-ssh-key ubuntu@server-instance-ip
+
+On the Linux server, check out the latest version from GitHub:
+
+	cd ~
+    sudo apt-get install git -qq
+	git clone https://github.com/TechEmpower/FrameworkBenchmarks.git
+	cd FrameworkBenchmarks
+
+As it currently stands, the script that runs the tests makes some assumptions about where the code is placed. We assume that the FrameworkBenchmarks repository will be located in your home directory.
+
+Next, we're going to setup the Linux server and the client with the required software:
+
+	toolset/run-tests.py -s server-private-ip -c client-private-ip -i path-to-ssh-key --install-software --list-tests &> install.log
+    
+This step can take a couple hours. If you like, you can open another shell and watch the progress:
+
+    tail -f ~/FrameworkBenchmarks/install.log | grep '^INSTALL'
+
+After the installation is done, reboot both the Linux server and the Linux client.
+
+When they are online again, connect via ssh to the Linux server and type these commands for additional setup:
+
+    source ~/.bash_profile
+    # For your first time through the tests, set the ulimit for open files
+    ulimit -n 8192
+    # Most software is installed automatically by the script, but running the mongo command below from
+    # the install script was causing some errors. For now this needs to be run manually.
+    cd installs/jruby-rack && rvm jruby-1.7.4 do jruby -S bundle exec rake clean gem SKIP_SPECS=true
+    cd target && rvm jruby-1.7.4 do gem install jruby-rack-1.2.0.SNAPSHOT.gem
+    cd ../../..
+    cd installs && curl -sS https://getcomposer.org/installer | php -- --install-dir=bin
+    cd ..
+    sudo apt-get remove --purge openjdk-6-jre openjdk-6-jre-headless
+    mongo --host client-private-ip < config/create.js
+
+You can validate that the setup worked by running a smoke test like this:
+
+    toolset/run-tests.py -s server-private-ip -c client-private-ip -i path-to-ssh-key --max-threads 1 --name smoketest --test servlet-raw --type all -m verify
+
+This should run the verification step for a single framework.
+
+## Windows server setup
+
+* Connect to the Windows server via Remote Desktop.
+* Copy `installer-bootstrap.ps1` from this repository to the server (use CTRL-C and CTRL-V).
+* Copy your Linux client private key too.
+* Right click on the installer script and select `Run with PowerShell`.
+* Press Enter to confirm.
+* It will install git and then launch `installer.ps1` from the repository, which will install everything else.
+* The installation takes about 20 minutes.
+* Then you have a working console: try `python`, `git`, `ssh`, `curl`, `node` etc. and verify that everything works + PowerShell goodies.
+
+The client/database machine is still assumed to be a Linux box. You can install just the client software via
+
+    python toolset\run-tests.py -s server-private-ip -c client-private-ip -i "C:\Users\Administrator\Desktop\client.key" --install-software --install client --list-tests
+
+but this step probably is not required if you installed the Linux server and client as described above.
+
+Now you can run tests:
+
+    python toolset\run-tests.py -s server-private-ip -c client-private-ip -i "C:\Users\Administrator\Desktop\client.key" --max-threads 2 --duration 30 --sleep 5 --name win --test aspnet --type all
+
+## SQL Server setup
+
+* Connect to the SQL Server host via Remote Desktop.
+* Run a `Command Prompt` as Administrator.
+* Enter this command:
+
+        powershell -ExecutionPolicy Bypass -Command "iex (New-Object Net.WebClient).DownloadString('https://raw.github.com/TechEmpower/FrameworkBenchmarks/master/setup-sqlserver-bootstrap.ps1')"
+
+* This will configure SQL Server, the Windows Firewall, and populate the database.
+
+Now, when running `run-tests.py`, just add `-d <ip of SQL Server instance>`. This works for the (Windows Server-based) `aspnet-sqlserver-raw` and `aspnet-sqlserver-entityframework` tests.

toolset/setup/__init__.py

@@ -0,0 +1 @@
+# Setup

toolset/setup/linux/__init__.py

@@ -0,0 +1 @@
+# Linux server and client setup