Merge pull request #1115 from hamiltont/bash-1112

Toolset: Fix a myriad of small issues

README.md

@@ -96,11 +96,19 @@ Note: environment variables can also be used for a number of the arguments.
 
 ## Installation Basics
 
-After you have a configuration file, run the following to setup your 
-various servers. We use the `--install-only` flag in our examples to 
-prevent launching tests at this stage. 
-
-See [here](deployment) for additional details. 
+To install TFB components onto the various servers, you must provide
+basic login details for each server (e.g. usernames, IP addresses, 
+private key files). While these details can all be passed 
+using command line flags, it's easier to use a configuration 
+file and avoid having huge flag lists in your commands. 
+The examples in this section assume you have created a configuration 
+file containing login details for the various servers. 
+
+We use the `--install-only` flag in our examples to 
+prevent launching tests at this stage. All of these commands 
+should be run from the `app server` - it will SSH into other
+hosts as needed. For more information on how TFB installation 
+works, see [here](deployment). 
 
 **Setting up the `load server`**
 

config/create-redis.sh

@@ -7,15 +7,19 @@ do
 done
 
 echo "DEL fortunes" | redis-cli
-echo "RPUSH fortunes 'fortune: No such file or directory' \
-\"A computer scientist is someone who fixes things that aren't broken.\" \
-'After enough decimal places, nobody gives a damn.' \
-'A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1' \
-'A computer program does what you tell it to do, not what you want it to do.' \
-'Emacs is a nice operating system, but I prefer UNIX. — Tom Christaensen' \
-'Any program that runs right is obsolete.' \
-'A list is only as strong as its weakest link. — Donald Knuth' \
-'Feature: A bug with seniority.' \
-'Computers make very fast, very accurate mistakes.' \
-'<script>alert(\"This should not be displayed in a browser alert box.\");</script>' \
-'フレームワークのベンチマーク'" | redis-cli
+
+# Please don't compress these into one statement, TFB is not always
+# run using Redis 2.4+ and we don't need optimization when 
+# adding ~10 entries
+echo "RPUSH fortunes \"fortune: No such file or directory\"" | redis-cli
+echo "RPUSH fortunes \"A computer scientist is someone who fixes things that aren't broken.\"" | redis-cli
+echo "RPUSH fortunes \"After enough decimal places, nobody gives a damn.\"" | redis-cli
+echo "RPUSH fortunes \"A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1\"" | redis-cli
+echo "RPUSH fortunes \"A computer program does what you tell it to do, not what you want it to do.\"" | redis-cli
+echo "RPUSH fortunes \"Emacs is a nice operating system, but I prefer UNIX. — Tom Christaensen\"" | redis-cli
+echo "RPUSH fortunes \"Any program that runs right is obsolete.\"" | redis-cli
+echo "RPUSH fortunes \"A list is only as strong as its weakest link. — Donald Knuth\"" | redis-cli
+echo "RPUSH fortunes \"Feature: A bug with seniority.\"" | redis-cli
+echo "RPUSH fortunes \"Computers make very fast, very accurate mistakes.\"" | redis-cli
+echo "RPUSH fortunes \"<script>alert(\\"This should not be displayed in a browser alert box.\\");</script>\"" | redis-cli
+echo "RPUSH fortunes \"フレームワークのベンチマーク\"" | redis-cli

deployment/vagrant-common/core.rb

@@ -81,10 +81,13 @@ def provider_virtualbox(config, role)
   config.vm.provider :virtualbox do |vb, override|
     override.vm.hostname = "TFB-#{role}"
 
-    override.vm.box = "ubuntu/trusty64"
-    if ENV.fetch('TFB_VM_ARCH','64') == "32"
-      override.vm.box = "ubuntu/trusty32"
-    end
+    # Valid values are 32 and 64
+    arch = ENV.fetch('TFB_VB_ARCH','64')
+
+    # Value values are precise, trusty, etc
+    code = ENV.fetch('TFB_VB_CODE','trusty')
+    
+    override.vm.box = "ubuntu/" + code + arch
     
     if ENV.fetch('TFB_SHOW_VM', false)
       vb.gui = true

deployment/vagrant-virtualbox/README.md

@@ -8,6 +8,7 @@ that modify how TFB launches your Virtuabox virtual machines.
 | :------------------------------- | :------------------ | :----------------------- | 
 | `TFB_VB_SHOW`                    | `true,false`        | Show the VM in a window when running? Default is false
 | `TFB_VB_ARCH`                    | `64,32`             | Used to force TFB to run a 32-bit virtual machine. This is unsupported, as many of the framework binaries we download are 64-bit only and will not launch. If you cannot run a 64-bit VM, then we recommend using the Amazon AWS provider instead of using this variable
+| `TFB_VB_CODE`                    | `trusty,precise`    | Force TFB to run a specific Ubuntu codename. Only trusty (14.04) is officially supported, but it's occasionally useful to see how the framework runs on precise (12.04)
 | `TFB_VB_MEM`                     | `<number>` e.g. `2048` | Size of VM's RAM in MB. Default is `2048`
 | `TFB_VB_CPU`                     | `<number>` e.g. `2` | Number of host CPUs that the VM can access
 
@@ -22,3 +23,10 @@ My standard workflow is to do `vagrant up` and immediately
 do a `vagrant snap` to preserve the initial state. Then I can
 install things, work on pull requests, etc, and roll back to the 
 initial state each time to avoid interference. 
+
+**Use Guest Additions Plugin**
+
+[This](https://github.com/dotless-de/vagrant-vbguest) Vagrant plugin will
+automatically build and inject the correct version of Oracle's Guest 
+Additions for the VM you are running. It's helpful for avoiding annoying 
+errors like "host additions are 4.3.10 but guest is 4.3.18"

frameworks/C++/cpoll_cppsp/Makefile

@@ -3,7 +3,7 @@ all: cppsp_0.2.3
 clean:
 	rm -f www/*.so www/*.txt
 	rm -f www/forcedynamic.cppsm.*
-	cd $(IROOT)/cppsp_0.2.3 all && $(MAKE) clean
+	$(MAKE) -C $(IROOT)/cppsp_0.2.3 clean
 
 cppsp_0.2.3:
 	$(MAKE) CXX=g++-4.8 -C $(IROOT)/cppsp_0.2.3 all

frameworks/C++/cpoll_cppsp/install.sh

@@ -1,8 +1,18 @@
 #!/bin/bash
 
+RETCODE=$(fw_exists cppsp.installed)
+[ ! "$RETCODE" == 0 ] || { return 0; }
+
 sudo apt-get install -y postgresql-server-dev-9.3 libpq-dev
 
 fw_get -O cppsp_0.2.3.tar.xz http://downloads.sourceforge.net/project/cpollcppsp/CPPSP%200.2%20%28testing%29/cppsp_0.2.3.tar.xz
 fw_untar cppsp_0.2.3.tar.xz
 
-mv cppsp_rel0.2.3/ cppsp_0.2.3
+# Using cp+rm over mv intentionally, because apparently this download
+# causes oddball issues when mv'ed around inside a folder mounted 
+# inside of VirtualBox (may have something to do with case-sensitive 
+# filesystems)
+cp -R cppsp_rel0.2.3/ $IROOT/cppsp_0.2.3
+rm -rf cppsp_rel0.2.3/
+
+touch cppsp.installed

frameworks/Clojure/compojure/bash_profile.sh

@@ -1,2 +1,7 @@
+#!/bin/bash
+
 export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64
 export RESIN_HOME=${IROOT}/resin-4.0.36
+export LEIN_HOME=$IROOT/bin
+
+export PATH="$JAVA_HOME/bin:$LEIN_HOME:$PATH"

toolset/benchmark/benchmarker.py

@@ -4,6 +4,7 @@ from setup.linux import setup_util
 from benchmark import framework_test
 from utils import header
 from utils import gather_tests
+from utils import gather_frameworks
 
 import os
 import json
@@ -15,7 +16,10 @@ import csv
 import sys
 import logging
 import socket
+import threading
+
 from multiprocessing import Process
+
 from datetime import datetime
 
 # Cross-platform colored text
@@ -331,44 +335,6 @@ class Benchmarker:
   # End __gather_tests
   ############################################################
 
-  ############################################################
-  # Gathers all the frameworks
-  ############################################################
-  def __gather_frameworks(self):
-    frameworks = []
-    # Loop through each directory (we assume we're being run from the benchmarking root)
-    for dirname, dirnames, filenames in os.walk('.'):
-      # Look for the benchmark_config file, this will contain our framework name
-      # It's format looks like this:
-      #
-      # {
-      #   "framework": "nodejs",
-      #   "tests": [{
-      #     "default": {
-      #       "setup_file": "setup",
-      #       "json_url": "/json"
-      #     },
-      #     "mysql": {
-      #       "setup_file": "setup",
-      #       "db_url": "/mysql",
-      #       "query_url": "/mysql?queries="
-      #     },
-      #     ...
-      #   }]
-      # }
-      if 'benchmark_config' in filenames:
-        config = None
-        with open(os.path.join(dirname, 'benchmark_config'), 'r') as config_file:
-          # Load json file into config object
-          config = json.load(config_file)
-        if config == None:
-          continue
-        frameworks.append(str(config['framework']))
-    return frameworks
-  ############################################################
-  # End __gather_frameworks
-  ############################################################
-
   ############################################################
   # Makes any necessary changes to the server that should be 
   # made before running the tests. This involves setting kernal
@@ -730,7 +696,7 @@ class Benchmarker:
     # Time to create parsed files
     # Aggregate JSON file
     with open(os.path.join(self.full_results_directory(), "results.json"), "w") as f:
-      f.write(json.dumps(self.results))
+      f.write(json.dumps(self.results, indent=2))
 
   ############################################################
   # End __parse_results
@@ -739,24 +705,28 @@ class Benchmarker:
 
   #############################################################
   # __count_sloc
-  # This is assumed to be run from the benchmark root directory
   #############################################################
   def __count_sloc(self):
-    all_frameworks = self.__gather_frameworks()
+    frameworks = gather_frameworks(include=self.test,
+      exclude=self.exclude, benchmarker=self)
+    
     jsonResult = {}
+    for framework, testlist in frameworks.iteritems():
+      # Unfortunately the source_code files use lines like
+      # ./cpoll_cppsp/www/fortune_old instead of 
+      # ./www/fortune_old
+      # so we have to back our working dir up one level
+      wd = os.path.dirname(testlist[0].directory)
 
-    for framework in all_frameworks:
       try:
-        command = "cloc --list-file=" + framework['directory'] + "/source_code --yaml"
-        lineCount = subprocess.check_output(command, shell=True)
+        command = "cloc --list-file=%s/source_code --yaml" % testlist[0].directory
         # Find the last instance of the word 'code' in the yaml output. This should
         # be the line count for the sum of all listed files or just the line count
         # for the last file in the case where there's only one file listed.
-        lineCount = lineCount[lineCount.rfind('code'):len(lineCount)]
-        lineCount = lineCount.strip('code: ')
-        lineCount = lineCount[0:lineCount.rfind('comment')]
-        jsonResult[framework['name']] = int(lineCount)
-      except:
+        command = command + "| grep code | tail -1 | cut -d: -f 2"
+        lineCount = subprocess.check_output(command, cwd=wd, shell=True)
+        jsonResult[framework] = int(lineCount)
+      except subprocess.CalledProcessError:
         continue
     self.results['rawData']['slocCounts'] = jsonResult
   ############################################################
@@ -765,19 +735,44 @@ class Benchmarker:
 
   ############################################################
   # __count_commits
+  #
   ############################################################
   def __count_commits(self):
-    all_frameworks = self.__gather_frameworks()
+    frameworks = gather_frameworks(include=self.test,
+      exclude=self.exclude, benchmarker=self)
 
-    jsonResult = {}
-
-    for framework in all_frameworks:
+    def count_commit(directory, jsonResult):
+      command = "git rev-list HEAD -- " + directory + " | sort -u | wc -l"
       try:
-        command = "git rev-list HEAD -- " + framework + " | sort -u | wc -l"
         commitCount = subprocess.check_output(command, shell=True)
         jsonResult[framework] = int(commitCount)
-      except:
-        continue
+      except subprocess.CalledProcessError:
+        pass
+
+    # Because git can be slow when run in large batches, this 
+    # calls git up to 4 times in parallel. Normal improvement is ~3-4x
+    # in my trials, or ~100 seconds down to ~25
+    # This is safe to parallelize as long as each thread only 
+    # accesses one key in the dictionary
+    threads = []
+    jsonResult = {}
+    t1 = datetime.now()
+    for framework, testlist in frameworks.iteritems():
+      directory = testlist[0].directory
+      t = threading.Thread(target=count_commit, args=(directory,jsonResult))
+      t.start()
+      threads.append(t)
+      # Git has internal locks, full parallel will just cause contention
+      # and slowness, so we rate-limit a bit
+      if len(threads) >= 4:
+        threads[0].join()
+        threads.remove(threads[0])
+
+    # Wait for remaining threads
+    for t in threads:
+      t.join()
+    t2 = datetime.now()
+    # print "Took %s seconds " % (t2 - t1).seconds
 
     self.results['rawData']['commitCounts'] = jsonResult
     self.commits = jsonResult
@@ -792,7 +787,7 @@ class Benchmarker:
     try:
       self.results["completed"][test_name] = status_message
       with open(os.path.join(self.latest_results_directory, 'results.json'), 'w') as f:
-        f.write(json.dumps(self.results))
+        f.write(json.dumps(self.results, indent=2))
     except (IOError):
       logging.error("Error writing results.json")
 

toolset/benchmark/framework_test.py

@@ -1290,8 +1290,9 @@ class FrameworkTest:
   ##########################################################################################
   # Constructor
   ##########################################################################################  
-  def __init__(self, name, directory, benchmarker, runTests, args):
+  def __init__(self, name, framework, directory, benchmarker, runTests, args):
     self.name = name
+    self.framework = framework
     self.directory = directory
     self.benchmarker = benchmarker
     self.runTests = runTests
@@ -1351,6 +1352,7 @@ def parse_config(config, directory, benchmarker):
   for test in config['tests']:
     for key, value in test.iteritems():
       test_name = config['framework']
+      test_framework = config['framework']
       
       runTests = dict()
 
@@ -1367,7 +1369,7 @@ def parse_config(config, directory, benchmarker):
         # we need to use the key in the test_name
         test_name = test_name + "-" + key
 
-      tests.append(FrameworkTest(test_name, directory, benchmarker, runTests, value))
+      tests.append(FrameworkTest(test_name, test_framework, directory, benchmarker, runTests, value))
 
   return tests
 ##############################################################

toolset/benchmark/utils.py

@@ -62,7 +62,7 @@ def gather_tests(include = [], exclude=[], benchmarker=None):
     with open(config_file_name, 'r') as config_file:
       try:
         config = json.load(config_file)
-      except:
+      except ValueError:
         # User-friendly errors
         print("Error loading '%s'." % config_file_name)
         raise
@@ -81,6 +81,20 @@ def gather_tests(include = [], exclude=[], benchmarker=None):
   tests.sort(key=lambda x: x.name)
   return tests
 
+def gather_frameworks(include = [], exclude=[], benchmarker=None):
+  '''Return a dictionary mapping frameworks->[test1,test2,test3]
+  for quickly grabbing all tests in a grouped manner. 
+  Args have the same meaning as gather_tests'''
+
+  tests = gather_tests(include, exclude, benchmarker)
+  frameworks = dict()
+
+  for test in tests:
+    if test.framework not in frameworks: 
+      frameworks[test.framework] = []
+    frameworks[test.framework].append(test)
+  return frameworks
+
 def header(message, top='-', bottom='-'):
     '''
     Generates a clean header

toolset/run-ci.py

@@ -309,24 +309,27 @@ class CIRunnner:
     log.info("Setting up Travis-CI")
     
     script = '''
-    # Needed to download latest MongoDB
+    export DEBIAN_FRONTEND=noninteractive
+
+    # Turn on command tracing
+    set -x 
+
+    # Setup Apt For MongoDB
     #   Due to TechEmpower/FrameworkBenchmarks#989 and travis-ci/travis-ci#2655, 
     #   we put this into a loop
     until timeout 15s sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10; do echo 'Waiting for apt-key' ; done
     echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
 
-    # Add Apache Cassandra repository
+    # Setup apt for Apache Cassandra
     until timeout 15s sudo apt-key adv --keyserver pgp.mit.edu --recv 4BD736A82B5C1B00; do echo 'Waiting for apt-key' ; done
     sudo apt-add-repository  'deb http://www.apache.org/dist/cassandra/debian 20x main'
 
+    # Run installation
     sudo apt-get -q update
-    
-    # MongoDB takes a good 30-45 seconds to turn on, so install it first
-    sudo apt-get -q install mongodb-org
-
-    sudo apt-get install -o Dpkg::Options::="--force-confnew" cassandra
-
-    sudo apt-get -q install openssh-server
+    sudo apt-get -q -y install -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" \
+      mongodb-org \
+      cassandra \
+      openssh-server
 
     # Run as travis user (who already has passwordless sudo)
     ssh-keygen -f /home/travis/.ssh/id_rsa -N '' -t rsa
@@ -338,29 +341,29 @@ class CIRunnner:
     #       It changes DB configuration files and will break everything
     # =======================================================
 
-    # Add data to mysql
-    echo "Populating MySQL DB data"
+    # Setup MySQL
+    echo "Populating MySQL database"
     mysql -uroot < config/create.sql
 
     # Setup Postgres
-    echo "Setting up Postgres database"
+    echo "Populating Postgres database"
     psql --version
     sudo useradd benchmarkdbuser -p benchmarkdbpass
     sudo -u postgres psql template1 < config/create-postgres-database.sql
     sudo -u benchmarkdbuser psql hello_world < config/create-postgres.sql
 
     # Setup Apache Cassandra
-    echo "Setting up Apache Cassandra database"
+    echo "Populating Apache Cassandra database"
     until nc -z localhost 9160 ; do echo Waiting for Cassandra; sleep 1; done
     cat config/cassandra/cleanup-keyspace.cql | sudo cqlsh
     python config/cassandra/db-data-gen.py > config/cassandra/tfb-data.cql
     sudo cqlsh -f config/cassandra/create-keyspace.cql
     sudo cqlsh -f config/cassandra/tfb-data.cql
 
-    echo "Setting up MongDB database"
-    # Setup MongoDB (see install above)
-    mongod --version
+    # Setup MongoDB
+    echo "Populating MongoDB database"
     until nc -z localhost 27017 ; do echo Waiting for MongoDB; sleep 1; done
+    mongod --version
     mongo < config/create.js
     '''
 

toolset/setup/linux/bash_functions.sh

@@ -73,22 +73,33 @@ fw_depends() {
     wd=$(pwd)
     relative_wd=\$FWROOT${wd#$FWROOT}
 
+    # Find and run the installer.sh file for this dependency
+    # Turn on some bash options before sourcing: 
+    #   - (x) errtrace : Print commands before they are run
+    # Note: A shebang is just a comment when you source a script, 
+    #       so if you need to modify the default options use  
+    #       `set -e` instead of `#!/bin/bash -e`
     if [ -f $FWROOT/toolset/setup/linux/systools/${depend}.sh ]; then
       echo Installing system tool: $depend in $relative_wd
+      set -x
       . $FWROOT/toolset/setup/linux/systools/${depend}.sh
     elif [ -f $FWROOT/toolset/setup/linux/languages/${depend}.sh ]; then
       echo Installing language: $depend in $relative_wd
+      set -x
       . $FWROOT/toolset/setup/linux/languages/${depend}.sh
     elif [ -f $FWROOT/toolset/setup/linux/webservers/${depend}.sh ]; then
       echo Installing webserver: $depend in $relative_wd
+      set -x
       . $FWROOT/toolset/setup/linux/webservers/${depend}.sh
     elif [ -f $FWROOT/toolset/setup/linux/frameworks/${depend}.sh ]; then
       echo Installing framework: $depend in $relative_wd
+      set -x
       . $FWROOT/toolset/setup/linux/frameworks/${depend}.sh
     else
       echo WARN: No installer found for $depend
       continue
     fi
+    set +x
 
     # For a sourced script to pass, all internal commands must return
     # non-zero. If you want to intentionally cause a failed install

toolset/setup/linux/client.sh

@@ -0,0 +1,46 @@
+#!/bin/bash
+
+export DB_HOST={database_host}
+
+set -x
+export DEBIAN_FRONTEND=noninteractive
+
+##############################
+# Prerequisites
+##############################
+# WARNING: DONT PUT A SPACE AFTER ANY BACKSLASH OR APT WILL BREAK
+# Dpkg::Options avoid hangs on Travis-CI, don't affect clean systems
+sudo apt-get -y install -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" \
+    build-essential git libev-dev libpq-dev libreadline6-dev
+
+sudo sh -c "echo '*               -    nofile          65535' >> /etc/security/limits.conf"
+
+##############################
+# wrk
+##############################
+
+git clone https://github.com/wg/wrk.git
+cd wrk
+git checkout 205a1960c8b8de5f500bb143863ae293456b7add
+make
+sudo cp wrk /usr/local/bin
+cd ~
+
+#############################
+# pipeline.lua
+#############################
+cat << EOF | tee pipeline.lua
+init = function(args)
+  wrk.init(args)
+  local r = {}
+  local depth = tonumber(args[1]) or 1
+  for i=1,depth do
+    r[i] = wrk.format()
+  end
+  req = table.concat(r)
+end
+
+request = function()
+  return req
+end
+EOF

toolset/setup/linux/database.sh

@@ -0,0 +1,199 @@
+#!/bin/bash
+#
+# Configures the database server for TFB
+#
+# Note: This is not used for Travis-CI. See run-ci.py to see
+# how databases are configured for Travis.
+#
+# Note on Compatibility: TFB *only* supports Ubuntu 14.04 64bit
+# (e.g. trusty64). However, it's nice to retain 12.04 support 
+# where possible, as it's still heavily used. 
+#
+# Database setup is one core area where we can help ensure TFB 
+# works on 12.04 with minimal frustration. In some cases we  
+# manually install the DB version that's expected, instead of the 
+# 12.04 default. In other cases we can use a 12.04 specific 
+# configuration file. These patches are not intended to enable 
+# benchmarking (e.g. there are no guarantees that 
+# the databases will be tuned for performance correctly), but 
+# they do allow users on 12.04 to install and run most TFB tests. 
+# Some tests internally have 12.04 incompatibilities, we make no 
+# concentrated effort to address these cases, but PR's for specific 
+# problems are welcome
+
+export DB_HOST={database_host}
+
+set -x
+export DEBIAN_FRONTEND=noninteractive
+
+##############################
+# Prerequisites
+##############################
+sudo apt-get -y update
+# WARNING: DONT PUT A SPACE AFTER ANY BACKSLASH OR APT WILL BREAK
+# Dpkg::Options avoid hangs on Travis-CI, don't affect clean systems
+sudo apt-get -y install -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" \
+    build-essential \
+    git \
+    libev-dev \
+    libpq-dev \
+    libreadline6-dev \
+    postgresql        `# Installs 9.1 or 9.3, based on Ubuntu version` \
+    redis-server      `# Installs 2.4 or 2.6, based on Ubuntu version` \
+    lsb-core          `# Ensure that lsb_release can be used`
+
+CODENAME=$(lsb_release -sc)
+
+sudo sh -c "echo '*               -    nofile          65535' >> /etc/security/limits.conf"
+
+# Create a user-owned directory for our databases
+sudo mkdir -p /ssd
+sudo mkdir -p /ssd/log
+sudo chown -R $USER:$USER /ssd
+
+# Additional user account (only use if required)
+sudo useradd benchmarkdbuser -p benchmarkdbpass
+
+##############################
+# MySQL
+##############################
+echo "Setting up MySQL database"
+sudo sh -c "echo mysql-server mysql-server/root_password_again select secret | debconf-set-selections"
+sudo sh -c "echo mysql-server mysql-server/root_password select secret | debconf-set-selections"
+
+sudo apt-get -y install mysql-server
+
+sudo stop mysql
+# disable checking of disk size
+sudo mv mysql /etc/init.d/mysql
+sudo chmod +x /etc/init.d/mysql
+sudo mv mysql.conf /etc/init/mysql.conf
+# use the my.cnf file to overwrite /etc/mysql/my.cnf
+sudo mv /etc/mysql/my.cnf /etc/mysql/my.cnf.orig
+sudo mv my.cnf /etc/mysql/my.cnf
+
+sudo cp -R -p /var/lib/mysql /ssd/
+sudo cp -R -p /var/log/mysql /ssd/log
+sudo cp usr.sbin.mysqld /etc/apparmor.d/
+sudo /etc/init.d/apparmor reload
+sudo start mysql
+
+# Insert data
+mysql -uroot -psecret < create.sql
+rm create.sql
+
+##############################
+# Postgres
+##############################
+echo "Setting up Postgres database"
+if [ "$CODENAME" == "precise" ]; then
+  echo "WARNING: Force upgrading Postgres for Ubuntu 12.04"
+  sudo apt-get remove -y postgresql postgresql-9.1 postgresql-client-9.1
+
+  echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list
+  wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
+  sudo apt-get update
+  sudo apt-get install -y postgresql-9.3 postgresql-client-9.3
+  sudo -u postgres -H /etc/init.d/postgresql start
+fi
+
+sudo -u postgres psql template1 < create-postgres-database.sql
+sudo -u benchmarkdbuser psql hello_world < create-postgres.sql
+rm create-postgres-database.sql create-postgres.sql
+
+sudo -u postgres -H /etc/init.d/postgresql stop
+sudo mv postgresql.conf /etc/postgresql/9.3/main/postgresql.conf
+sudo mv pg_hba.conf /etc/postgresql/9.3/main/pg_hba.conf
+
+sudo cp -R -p /var/lib/postgresql/9.3/main /ssd/postgresql
+sudo -u postgres -H /etc/init.d/postgresql start
+sudo mv 60-postgresql-shm.conf /etc/sysctl.d/60-postgresql-shm.conf
+
+##############################
+# MongoDB
+#
+# Note for 12.04: Using mongodb.org ensures 2.6 is installed
+##############################
+echo "Setting up MongoDB database"
+sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
+echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
+sudo apt-get -y update
+sudo apt-get -y remove mongodb-clients
+sudo apt-get -y install mongodb-org
+
+sudo service mongod stop
+sudo mv /etc/mongodb.conf /etc/mongodb.conf.orig
+sudo mv mongodb.conf /etc/mongodb.conf
+sudo mv mongodb.conf /etc/mongod.conf
+sudo cp -R -p /var/lib/mongodb /ssd/
+sudo cp -R -p /var/log/mongodb /ssd/log/
+sudo service mongod start
+
+until nc -z localhost 27017 ; do echo Waiting for MongoDB; sleep 1; done
+mongo < create.js
+rm create.js
+mongod --version
+
+##############################
+# Apache Cassandra
+##############################
+echo "Setting up Apache Cassandra database"
+sudo apt-get install -qqy openjdk-7-jdk
+export CASS_V=2.0.7
+wget -nv http://archive.apache.org/dist/cassandra/$CASS_V/apache-cassandra-$CASS_V-bin.tar.gz
+tar xzf apache-cassandra-$CASS_V-bin.tar.gz
+
+rm -rf /ssd/cassandra /ssd/log/cassandra
+mkdir -p /ssd/cassandra /ssd/log/cassandra
+
+sed -i "s/^.*seeds:.*/          - seeds: \"$DB_HOST\"/" cassandra/cassandra.yaml
+sed -i "s/^listen_address:.*/listen_address: $DB_HOST/" cassandra/cassandra.yaml
+sed -i "s/^rpc_address:.*/rpc_address: $DB_HOST/" cassandra/cassandra.yaml
+
+mv cassandra/cassandra.yaml apache-cassandra-$CASS_V/conf
+mv cassandra/log4j-server.properties apache-cassandra-$CASS_V/conf
+nohup apache-cassandra-$CASS_V/bin/cassandra -p c.pid > cassandra.log
+
+until nc -z $DB_HOST 9160 ; do echo Waiting for Cassandra; sleep 1; done
+cat cassandra/cleanup-keyspace.cql | apache-cassandra-$CASS_V/bin/cqlsh $DB_HOST
+python cassandra/db-data-gen.py > cassandra/tfb-data.cql
+apache-cassandra-$CASS_V/bin/cqlsh -f cassandra/create-keyspace.cql $DB_HOST
+apache-cassandra-$CASS_V/bin/cqlsh -f cassandra/tfb-data.cql $DB_HOST
+rm -rf apache-cassandra-*-bin.tar.gz cassandra
+
+##############################
+# Redis
+##############################
+echo "Setting up Redis database"
+if [ "$CODENAME" == "precise" ]; then
+  echo "WARNING: Downgrading Redis configuration for Ubuntu 12.04"
+
+  # On 12.04, Redis 2.4 is installed. It doesn't support 
+  # some of the 2.6 options, so we have to remove or comment
+  # those
+  sed -i 's/tcp-keepalive/# tcp-keepalive/' redis.conf
+  sed -i 's/stop-writes-on-bgsave-error/# stop-writes-on-bgsave-error/' redis.conf
+  sed -i 's/rdbchecksum/# rdbchecksum/' redis.conf
+  sed -i 's/slave-read-only/# slave-read-only/' redis.conf
+  sed -i 's/repl-disable-tcp-nodelay/# repl-disable-tcp-nodelay/' redis.conf
+  sed -i 's/slave-priority/# slave-priority/' redis.conf
+  sed -i 's/auto-aof-rewrite-percentage/# auto-aof-rewrite-percentage/' redis.conf
+  sed -i 's/auto-aof-rewrite-min-size/# auto-aof-rewrite-min-size/' redis.conf
+  
+  sed -i 's/lua-time-limit/# lua-time-limit/' redis.conf
+  sed -i 's/notify-keyspace-events/# notify-keyspace-events/' redis.conf
+  sed -i 's/hash-max-ziplist-entries/# hash-max-ziplist-entries/' redis.conf
+  sed -i 's/hash-max-ziplist-value/# hash-max-ziplist-value/' redis.conf
+  sed -i 's/zset-max-ziplist-entries/# zset-max-ziplist-entries/' redis.conf
+  sed -i 's/zset-max-ziplist-value/# zset-max-ziplist-value/' redis.conf
+  sed -i 's/client-output-buffer-limit/# client-output-buffer-limit/' redis.conf
+ 
+  sed -i 's/hz 10/# hz 10/' redis.conf
+  sed -i 's/aof-rewrite-incremental-fsync/# aof-rewrite-incremental-fsync/' redis.conf
+fi
+
+sudo service redis-server stop
+sudo mv redis.conf /etc/redis/redis.conf
+sudo service redis-server start
+bash create-redis.sh
+rm create-redis.sh

toolset/setup/linux/installer.py

@@ -16,15 +16,36 @@ class Installer:
   # install_software
   ############################################################
   def install_software(self):
-    if self.benchmarker.install == 'all' or self.benchmarker.install == 'server':
-        self.__install_server_software()
-
-    if self.benchmarker.install == 'all' or self.benchmarker.install == 'database':
-        self.__install_database_software()
-
-    if self.benchmarker.install == 'all' or self.benchmarker.install == 'client':
-        self.__install_client_software()
-
+    linux_install_root = self.fwroot + "/toolset/setup/linux"
+    imode = self.benchmarker.install
+
+    if imode == 'all' or imode == 'server':
+      self.__install_server_software()
+
+    if imode == 'all' or imode == 'database':
+      print("\nINSTALL: Installing database software\n")   
+      self.__run_command("cd .. && " + self.benchmarker.database_sftp_string(batch_file="../config/database_sftp_batch"), True)
+      with open (linux_install_root + "/database.sh", "r") as myfile:
+        remote_script=myfile.read().format(database_host=self.benchmarker.database_host)
+        print("\nINSTALL: %s" % self.benchmarker.database_ssh_string)
+        p = subprocess.Popen(self.benchmarker.database_ssh_string.split(" ") + ["bash"], stdin=subprocess.PIPE)
+        p.communicate(remote_script)
+        returncode = p.returncode
+        if returncode != 0:
+          self.__install_error("status code %s running subprocess '%s'." % (returncode, self.benchmarker.database_ssh_string))
+      print("\nINSTALL: Finished installing database software\n")
+
+    if imode == 'all' or imode == 'client':
+      print("\nINSTALL: Installing client software\n")    
+      with open (linux_install_root + "/client.sh", "r") as myfile:
+        remote_script=myfile.read()
+        print("\nINSTALL: %s" % self.benchmarker.client_ssh_string)
+        p = subprocess.Popen(self.benchmarker.client_ssh_string.split(" ") + ["bash"], stdin=subprocess.PIPE)
+        p.communicate(remote_script)
+        returncode = p.returncode
+        if returncode != 0:
+          self.__install_error("status code %s running subprocess '%s'." % (returncode, self.benchmarker.client_ssh_string))
+      print("\nINSTALL: Finished installing client software\n")
   ############################################################
   # End install_software
   ############################################################
@@ -120,226 +141,6 @@ class Installer:
   # End __install_error
   ############################################################
 
-  ############################################################
-  # __install_database_software
-  ############################################################
-  def __install_database_software(self):
-    print("\nINSTALL: Installing database software\n")
- 
-    self.__run_command("cd .. && " + self.benchmarker.database_sftp_string(batch_file="../config/database_sftp_batch"), True)
-
-    remote_script = """
-
-    ##############################
-    # Prerequisites
-    ##############################
-    sudo apt-get -y update
-    sudo apt-get -y install build-essential git libev-dev libpq-dev libreadline6-dev postgresql redis-server
-    sudo sh -c "echo '*               -    nofile          65535' >> /etc/security/limits.conf"
-
-    # Create a user-owned directory for our databases
-    sudo mkdir -p /ssd
-    sudo mkdir -p /ssd/log
-    sudo chown -R $USER:$USER /ssd
-
-    # Additional user account (only use if required)
-    sudo useradd benchmarkdbuser -p benchmarkdbpass
-
-    ##############################
-    # MySQL
-    ##############################
-    echo "Setting up MySQL database"
-    sudo sh -c "echo mysql-server mysql-server/root_password_again select secret | debconf-set-selections"
-    sudo sh -c "echo mysql-server mysql-server/root_password select secret | debconf-set-selections"
-
-    sudo apt-get -y install mysql-server
-
-    sudo stop mysql
-    # disable checking of disk size
-    sudo mv mysql /etc/init.d/mysql
-    sudo chmod +x /etc/init.d/mysql
-    sudo mv mysql.conf /etc/init/mysql.conf
-    # use the my.cnf file to overwrite /etc/mysql/my.cnf
-    sudo mv /etc/mysql/my.cnf /etc/mysql/my.cnf.orig
-    sudo mv my.cnf /etc/mysql/my.cnf
-
-    sudo cp -R -p /var/lib/mysql /ssd/
-    sudo cp -R -p /var/log/mysql /ssd/log
-    sudo cp usr.sbin.mysqld /etc/apparmor.d/
-    sudo /etc/init.d/apparmor reload
-    sudo start mysql
-
-    # Insert data
-    mysql -uroot -psecret < create.sql
-    rm create.sql
-
-    ##############################
-    # Postgres
-    ##############################
-    echo "Setting up Postgres database"
-    sudo -u postgres psql template1 < create-postgres-database.sql
-    sudo -u benchmarkdbuser psql hello_world < create-postgres.sql
-    rm create-postgres-database.sql create-postgres.sql
-
-    sudo -u postgres -H /etc/init.d/postgresql stop
-    # NOTE: This will cause errors on Ubuntu 12.04, as apt installs 
-    # an older version (9.1 instead of 9.3)
-    sudo mv postgresql.conf /etc/postgresql/9.3/main/postgresql.conf
-    sudo mv pg_hba.conf /etc/postgresql/9.3/main/pg_hba.conf
-
-    sudo cp -R -p /var/lib/postgresql/9.3/main /ssd/postgresql
-    sudo -u postgres -H /etc/init.d/postgresql start
-    sudo mv 60-postgresql-shm.conf /etc/sysctl.d/60-postgresql-shm.conf
-
-    ##############################
-    # MongoDB
-    ##############################
-    echo "Setting up MongoDB database"
-    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
-    echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
-    sudo apt-get -y update
-    sudo apt-get -y remove mongodb-clients
-    sudo apt-get -y install mongodb-org
-
-    sudo service mongod stop
-    sudo mv /etc/mongodb.conf /etc/mongodb.conf.orig
-    sudo mv mongodb.conf /etc/mongodb.conf
-    sudo mv mongodb.conf /etc/mongod.conf
-    sudo cp -R -p /var/lib/mongodb /ssd/
-    sudo cp -R -p /var/log/mongodb /ssd/log/
-    sudo service mongod start
-
-    until nc -z localhost 27017 ; do echo Waiting for MongoDB; sleep 1; done
-    mongo < create.js
-    rm create.js
-
-    ##############################
-    # Apache Cassandra
-    ##############################
-    echo "Setting up Apache Cassandra database"
-    sudo apt-get install -qqy openjdk-7-jdk
-    export CASS_V=2.0.7
-    wget -nv http://archive.apache.org/dist/cassandra/$CASS_V/apache-cassandra-$CASS_V-bin.tar.gz
-    tar xzf apache-cassandra-$CASS_V-bin.tar.gz
-    
-    rm -rf /ssd/cassandra /ssd/log/cassandra
-    mkdir -p /ssd/cassandra /ssd/log/cassandra
-    
-    sed -i "s/^.*seeds:.*/          - seeds: \"{database_host}\"/" cassandra/cassandra.yaml
-    sed -i "s/^listen_address:.*/listen_address: {database_host}/" cassandra/cassandra.yaml
-    sed -i "s/^rpc_address:.*/rpc_address: {database_host}/" cassandra/cassandra.yaml
-    
-    mv cassandra/cassandra.yaml apache-cassandra-$CASS_V/conf
-    mv cassandra/log4j-server.properties apache-cassandra-$CASS_V/conf
-    nohup apache-cassandra-$CASS_V/bin/cassandra -p c.pid > cassandra.log
-
-    until nc -z {database_host} 9160 ; do echo Waiting for Cassandra; sleep 1; done
-    cat cassandra/cleanup-keyspace.cql | apache-cassandra-$CASS_V/bin/cqlsh {database_host}
-    python cassandra/db-data-gen.py > cassandra/tfb-data.cql
-    apache-cassandra-$CASS_V/bin/cqlsh -f cassandra/create-keyspace.cql {database_host}
-    apache-cassandra-$CASS_V/bin/cqlsh -f cassandra/tfb-data.cql {database_host}
-    rm -rf apache-cassandra-*-bin.tar.gz cassandra
-
-    ##############################
-    # Redis
-    ##############################
-    echo "Setting up Redis database"
-    sudo service redis-server stop
-    # NOTE: This will cause errors on Ubuntu 12.04, as apt installs 
-    # an older version of redis
-    sudo mv redis.conf /etc/redis/redis.conf
-    sudo service redis-server start
-    bash create-redis.sh
-    rm create-redis.sh
-    """.format(database_host=self.benchmarker.database_host)
-    
-    print("\nINSTALL: %s" % self.benchmarker.database_ssh_string)
-    p = subprocess.Popen(self.benchmarker.database_ssh_string.split(" ") + ["bash"], stdin=subprocess.PIPE)
-    p.communicate(remote_script)
-    returncode = p.returncode
-    if returncode != 0:
-      self.__install_error("status code %s running subprocess '%s'." % (returncode, self.benchmarker.database_ssh_string))
-
-    print("\nINSTALL: Finished installing database software\n")
-  ############################################################
-  # End __install_database_software
-  ############################################################
-
-  ############################################################
-  # __install_client_software
-  ############################################################
-  def __install_client_software(self):
-    print("\nINSTALL: Installing client software\n")
-
-    remote_script = """
-
-    ##############################
-    # Prerequisites
-    ##############################
-    sudo apt-get -y update
-    sudo apt-get -y install build-essential git libev-dev libpq-dev libreadline6-dev 
-    sudo sh -c "echo '*               -    nofile          65535' >> /etc/security/limits.conf"
-
-
-    ##############################
-    # wrk
-    ##############################
-
-    git clone https://github.com/wg/wrk.git
-    cd wrk
-    git checkout 205a1960c8b8de5f500bb143863ae293456b7add
-    make
-    sudo cp wrk /usr/local/bin
-    cd ~
-    
-    #############################
-    # pipeline.lua
-    #############################
-cat << EOF | tee pipeline.lua
-init = function(args)
-  wrk.init(args)
-  local r = {}
-  local depth = tonumber(args[1]) or 1
-  for i=1,depth do
-    r[i] = wrk.format()
-  end
-  req = table.concat(r)
-end
-
-request = function()
-  return req
-end
-EOF
-    """
-    
-    print("\nINSTALL: %s" % self.benchmarker.client_ssh_string)
-    p = subprocess.Popen(self.benchmarker.client_ssh_string.split(" "), stdin=subprocess.PIPE)
-    p.communicate(remote_script)
-    returncode = p.returncode
-    if returncode != 0:
-      self.__install_error("status code %s running subprocess '%s'." % (returncode, self.benchmarker.client_ssh_string))
-
-    print("\nINSTALL: Finished installing client software\n")
-  ############################################################
-  # End __install_client_software
-  ############################################################
-
-  ############################################################
-  # __path_exists
-  ############################################################
-  def __path_exists(self, path, cwd=None):
-    full_path = os.path.join(cwd or self.install_dir, path)
-
-    if os.path.exists(full_path):
-        print("\nEXISTS: %s " % full_path)
-        return True
-
-    print("\nNOT_EXISTS: %s" % full_path)
-    return False
-  ############################################################
-  # End __path_exists
-  ############################################################
-
   ############################################################
   # __run_command
   ############################################################
@@ -389,33 +190,6 @@ EOF
   # End __run_command
   ############################################################
 
-  ############################################################
-  # __bash_from_string
-  # Runs bash -c "command" in install_dir.
-  ############################################################
-  def __bash_from_string(self, command):
-    self.__run_command('bash -c "%s"' % command)
-  ############################################################
-  # End __bash_from_string
-  ############################################################
-
-  ############################################################
-  # __download
-  # Downloads a file from a URI.
-  ############################################################
-  def __download(self, uri, filename=""):
-    if filename:
-      if os.path.exists(filename):
-        return
-      filename_option = "-O %s " % filename
-    else:
-      filename_option = ""
-    command = "wget -nv --no-check-certificate --trust-server-names %s%s" % (filename_option, uri)
-    self.__run_command(command, retry=True)
-  ############################################################
-  # End __download
-  ############################################################
-
   ############################################################
   # __init__(benchmarker)
   ############################################################

toolset/setup/linux/languages/haskell78.sh

@@ -1,11 +1,8 @@
-#!/bin/bash -ex
+#!/bin/bash 
 
 RETCODE=$(fw_exists /opt/ghc/7.8.3/bin/ghc)
 [ ! "$RETCODE" == 0 ] || { return 0; }
 
-lsb_release -a
-env
-
 export LANG=en_US.UTF-8
 
 sudo add-apt-repository -y ppa:hvr/ghc