Upgrade to swift 4.1 and mongodb 3.6, fix vapor tests (#3242)

* Upgrade to swift 4, repair vapor build

This fixes issue #3181.

I think we were running into this issue in the stable vapor release:
https://bugs.swift.org/browse/SR-5800

...which the comments claim is really about this issue:
https://bugs.swift.org/browse/SR-5802

...which was fixed in the main swift source repository:
https://github.com/apple/swift-package-manager/pull/1419

That fix happened pretty recently and doesn't seem to be included in the
latest stable releases (I don't see the PackageManager.swiftdoc file in
those releases), but it does seem to be fixed in the development snapshots
(I do see PackageManager.swiftdoc there).

With the development snapshot, the vapor build progresses much farther.
It eventually runs into an error compiling the vapor framework.  That is
an error in the outdated version of vapor we were using, and it was fixed
later:
https://github.com/vapor/vapor/commit/5a31fc8c7c6a54a8a98494795e0668666c341a33#diff-4a85e78227146848570c0c42d00d2f17

When I specify 2.4.4 as our version of vapor (the most recent release),
the build is successful and all the tests pass.

I renamed "swift3.sh" to "swift.sh" instead of "swift4.sh" because we
prefer to keep the language version out of these filenames unless we
can't avoid having multiple versions of the same language in the toolset.

It looked like vapor's setup-*.sh files were declaring unnecessary
database dependencies, so I cleaned those up.

* Upgrade mongdb from 3.2 to 3.6

The vapor tests rely on a feature that was added in mongodb 3.4, so we
might as well take this opportunity to upgrade mongodb to the latest
release.

* Disable vapor+mongodb multi-query and updates endpoints

They fail verification locally because they take too long to produce a
response.

* Remove vapor's Package.pins file, which is out of date and maybe unused?

It's showing the wrong version of Vapor, at least.  Also, it's not clear
to me if "Package.pins" is still used in this version of swift or if it
was replaced by "Package.resolved".

* Add index for world.id in mongodb

Some frameworks query for worlds by _id, and others by id.  It looks like
the ones that queried by id were being penalized heavily because we had
no index on that field.  In local testing, the RPS of nodejs-mongodb in
the db test went up by 8x (!) after I added the index.

Use createIndex instead of ensureIndex because, according to mongodb's
documentation, ensureIndex is:

  Deprecated since version 3.0.0: db.collection.ensureIndex() is now an
  alias for db.collection.createIndex().

Remove comment with a link to mongodb optimization because it doesn't
seem to say anything related to what we're doing in this file.

* Try to prevent restexpress from creating its own index in mongodb

The restexpress build on travis is failing with an exception like this:

  Exception in thread "main" com.mongodb.MongoException: Index with name: id_1 already exists with different options
    at com.mongodb.CommandResult.getException(CommandResult.java:100)
    at com.mongodb.CommandResult.throwOnError(CommandResult.java:134)
    at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:142)
    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:183)
    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)
    at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:270)
    at com.mongodb.DBApiLayer$MyCollection.createIndex(DBApiLayer.java:365)
    at com.mongodb.DBCollection.createIndex(DBCollection.java:484)
    at com.mongodb.DBCollection.ensureIndex(DBCollection.java:560)
    at com.github.jmkgreen.morphia.DatastoreImpl.ensureIndex(DatastoreImpl.java:431)
    at com.github.jmkgreen.morphia.DatastoreImpl.ensureIndexes(DatastoreImpl.java:536)
    at com.github.jmkgreen.morphia.DatastoreImpl.ensureIndexes(DatastoreImpl.java:495)
    at com.github.jmkgreen.morphia.DatastoreImpl.ensureIndexes(DatastoreImpl.java:584)
    at com.github.jmkgreen.morphia.DatastoreImpl.ensureIndexes(DatastoreImpl.java:573)
    at com.strategicgains.repoexpress.mongodb.MongodbRepository.initialize(MongodbRepository.java:106)
    at com.strategicgains.repoexpress.mongodb.MongodbRepository.<init>(MongodbRepository.java:67)
    at hello.controller.persistence.WorldsMongodbRepository.<init>(WorldsMongodbRepository.java:14)
    at hello.config.Configuration.initialize(Configuration.java:53)
    at hello.config.Configuration.fillValues(Configuration.java:45)
    at com.strategicgains.restexpress.util.Environment.load(Environment.java:61)
    at com.strategicgains.restexpress.util.Environment.from(Environment.java:50)
    at com.strategicgains.restexpress.util.Environment.fromDefault(Environment.java:40)
    at hello.Main.loadEnvironment(Main.java:49)
    at hello.Main.main(Main.java:18)

Individual frameworks should not be attempting to modify the indexes in
the database.  Hopefully that's all this @Indexed annotation was doing.

frameworks/Java/restexpress/src/main/java/hello/domain/World.java

@@ -4,7 +4,6 @@ import org.bson.types.ObjectId;
 
 import com.github.jmkgreen.morphia.annotations.Entity;
 import com.github.jmkgreen.morphia.annotations.Id;
-import com.github.jmkgreen.morphia.annotations.Indexed;
 import com.strategicgains.repoexpress.domain.Identifiable;
 
 @Entity(value="world", noClassnameStored=true)
@@ -13,8 +12,7 @@ implements Identifiable
 {
 	@Id
 	private ObjectId oid;
-	
-	@Indexed(unique=true)
+
 	private Long id;
 	private int randomNumber;
 

frameworks/Swift/README.md

@@ -4,19 +4,19 @@ The information below contains information specific to Python. For further guida
 
 ## Infrastructure Software Versions
 
-[Swift 3](https://swift.org)
+[Swift](https://swift.org)
 
 ## Adding a New Framework
 
-In order to declare that your framework requires Swift 3, you should have an `install.sh` that contains at least
+In order to declare that your framework requires Swift, you should have an `install.sh` that contains at least
 ```
 #!/bin/bash
 
-fw_depends swift3
+fw_depends swift
 
 ```
 
-This installs the Swift 3.
+This installs Swift.
 
 ## Get Help
 

frameworks/Swift/vapor/Package.pins

@@ -1,228 +0,0 @@
-{
-  "autoPin": true,
-  "pins": [
-    {
-      "package": "BCrypt",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/bcrypt.git",
-      "version": "1.0.0"
-    },
-    {
-      "package": "BSON",
-      "reason": null,
-      "repositoryURL": "https://github.com/OpenKitten/BSON.git",
-      "version": "5.0.5"
-    },
-    {
-      "package": "Bits",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/bits.git",
-      "version": "1.0.0"
-    },
-    {
-      "package": "CMySQL",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/cmysql.git",
-      "version": "2.0.2"
-    },
-    {
-      "package": "CPostgreSQL",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor-community/cpostgresql.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "CTLS",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/ctls.git",
-      "version": "1.0.0"
-    },
-    {
-      "package": "Cheetah",
-      "reason": null,
-      "repositoryURL": "https://github.com/OpenKitten/Cheetah.git",
-      "version": "1.0.1"
-    },
-    {
-      "package": "Console",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/console.git",
-      "version": "2.1.0"
-    },
-    {
-      "package": "Core",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/core.git",
-      "version": "2.0.2"
-    },
-    {
-      "package": "Crypto",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/crypto.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "CryptoSwift",
-      "reason": null,
-      "repositoryURL": "https://github.com/krzyzanowskim/CryptoSwift.git",
-      "version": "0.6.9"
-    },
-    {
-      "package": "Debugging",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/debugging.git",
-      "version": "1.0.0"
-    },
-    {
-      "package": "Engine",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/engine.git",
-      "version": "2.0.4"
-    },
-    {
-      "package": "Fluent",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/fluent.git",
-      "version": "2.0.4"
-    },
-    {
-      "package": "FluentProvider",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/fluent-provider.git",
-      "version": "1.1.0"
-    },
-    {
-      "package": "JSON",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/json.git",
-      "version": "2.0.2"
-    },
-    {
-      "package": "KittenCore",
-      "reason": null,
-      "repositoryURL": "https://github.com/OpenKitten/KittenCore.git",
-      "version": "0.2.3"
-    },
-    {
-      "package": "Leaf",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/leaf.git",
-      "version": "2.0.1"
-    },
-    {
-      "package": "LeafProvider",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/leaf-provider.git",
-      "version": "1.0.0"
-    },
-    {
-      "package": "MongoDriver",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/mongo-driver.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "MongoKitten",
-      "reason": null,
-      "repositoryURL": "https://github.com/OpenKitten/MongoKitten.git",
-      "version": "4.0.9"
-    },
-    {
-      "package": "MongoProvider",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/mongo-provider.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "Multipart",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/multipart.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "MySQL",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/mysql.git",
-      "version": "2.0.1"
-    },
-    {
-      "package": "MySQLDriver",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/mysql-driver.git",
-      "version": "2.0.1"
-    },
-    {
-      "package": "MySQLProvider",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/mysql-provider.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "Node",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/node.git",
-      "version": "2.0.3"
-    },
-    {
-      "package": "PostgreSQL",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor-community/postgresql.git",
-      "version": "2.0.1"
-    },
-    {
-      "package": "PostgreSQLDriver",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/postgresql-driver.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "PostgreSQLProvider",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/postgresql-provider.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "Random",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/random.git",
-      "version": "1.0.0"
-    },
-    {
-      "package": "Routing",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/routing.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "SQLite",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/sqlite.git",
-      "version": "2.0.0"
-    },
-    {
-      "package": "Schrodinger",
-      "reason": null,
-      "repositoryURL": "https://github.com/OpenKitten/Schrodinger.git",
-      "version": "1.0.1"
-    },
-    {
-      "package": "Sockets",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/sockets.git",
-      "version": "2.0.1"
-    },
-    {
-      "package": "TLS",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/tls.git",
-      "version": "2.0.4"
-    },
-    {
-      "package": "Vapor",
-      "reason": null,
-      "repositoryURL": "https://github.com/vapor/vapor.git",
-      "version": "2.0.8"
-    }
-  ],
-  "version": 1
-}

frameworks/Swift/vapor/Package.swift

@@ -8,7 +8,7 @@ let package = Package(
     Target(name: "vapor-tfb-mongodb", dependencies: ["TfbCommon"])
   ],
   dependencies: [
-    .Package(url: "https://github.com/vapor/vapor.git", majorVersion: 2, minor: 0),
+    .Package(url: "https://github.com/vapor/vapor.git", "2.4.4"),
     .Package(url: "https://github.com/vapor/postgresql-provider.git", majorVersion:2),
     .Package(url: "https://github.com/vapor/mysql-provider.git", majorVersion: 2),
     .Package(url: "https://github.com/vapor/mongo-provider.git", majorVersion: 2),
@@ -22,4 +22,3 @@ let package = Package(
     "Resources",
   ]
 )
-

frameworks/Swift/vapor/Sources/vapor-tfb-mongodb/Routes.swift

@@ -26,6 +26,12 @@ final class Routes: RouteCollection {
             return try World.find(worldId)?.makeJSON() ?? JSON(node: .null)
         }
 
+        // FIXME: The multi-query and updates endpoints are currently omitted
+        //        from benchmark_config.json because they take too long and
+        //        timeout with queries=500, which is one of the verification
+        //        steps.  Something here is taking far too long... probably
+        //        World.find(id).
+
         // Test type 3: Multiple database queries
         builder.get("queries") { req in
             let queries = queriesParam(for: req)

frameworks/Swift/vapor/benchmark_config.json

@@ -53,9 +53,7 @@
       "setup_file": "setup-mongodb",
       "json_url": "/json",
       "db_url": "/db",
-      "query_url": "/queries?queries=",
       "fortune_url": "/fortunes",
-      "update_url": "/updates?queries=",
       "plaintext_url": "/plaintext",
       "port": 8080,
       "approach": "Realistic",

frameworks/Swift/vapor/setup-mongodb.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-fw_depends swift3 ctls cmysql mongodb mysql postgresql
+fw_depends swift ctls cmysql mongodb
 
 swift build -Xswiftc -DNOJSON -c release
 

frameworks/Swift/vapor/setup-mysql.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-fw_depends swift3 ctls cmysql mongodb mysql postgresql
+fw_depends swift ctls cmysql mysql
 
 swift build -Xswiftc -DNOJSON -c release
 

frameworks/Swift/vapor/setup-postgresql.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-fw_depends swift3 ctls cmysql mongodb mysql postgresql
+fw_depends swift ctls cmysql postgresql
 
 swift build -Xswiftc -DNOJSON -c release
 

toolset/setup/linux/databases/mongodb/create.js

@@ -4,8 +4,8 @@ for (var i = 1; i <= 10000; i++) {
   db.world.save( { _id: i, id: i, randomNumber: (Math.floor(Math.random() * 10000) + 1) })
 }
 
-// http://docs.mongodb.org/manual/applications/optimization/
-db.world.ensureIndex({_id: 1})
+db.world.createIndex({_id: 1})
+db.world.createIndex({id: 1})
 
 db.fortune.drop()
 
@@ -22,4 +22,4 @@ db.fortune.save( {_id: 10, id: 10, message: 'Computers make very fast, very accu
 db.fortune.save( {_id: 11, id: 11, message: '<script>alert("This should not be displayed in a browser alert box.");</script>'} );
 db.fortune.save( {_id: 12, id: 12, message: 'フレームワークのベンチマーク'} );
 
-db.fortune.ensureIndex({_id: 1})
+db.fortune.createIndex({_id: 1})

toolset/setup/linux/databases/mongodb/mongodb.conf

@@ -21,3 +21,4 @@ systemLog:
 # network interfaces
 net:
   port: 27017
+  bindIpAll: true

toolset/setup/linux/databases/mongodb/mongodb.sh

@@ -11,20 +11,30 @@ scp $FWROOT/toolset/setup/linux/databases/mongodb/create.js $DBHOST:~/
 # install mongo on database machine
 ssh $DBHOST 'bash' <<"EOF"
 echo "Setting up MongoDB database"
-sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
-echo 'deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse' | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
+
+if service --status-all | grep -Fq 'mongod'; then
+  sudo service mongod stop
+fi
+sudo apt-get -y remove --purge mongodb-org
+sudo apt-get -y autoremove
+sudo find /etc/apt/sources.list.d -type f -name 'mongodb-org-*.list' -delete -maxdepth 1
+sudo find /etc -type f -name 'mongo*.conf*' -delete -maxdepth 1
+sudo rm -rf /var/lib/mongodb
+sudo rm -rf /var/log/mongodb
+
+sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
+echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
 sudo apt-get -y update
-sudo apt-get -y remove mongodb-clients
 sudo apt-get -y install mongodb-org
 
 sudo service mongod stop
-sudo mv /etc/mongodb.conf /etc/mongodb.conf.orig
-sudo cp mongodb.conf /etc/mongodb.conf
 sudo mv mongodb.conf /etc/mongod.conf
 sudo rm -rf /ssd/mongodb
 sudo rm -rf /ssd/log/mongodb
-sudo cp -R -p /var/lib/mongodb /ssd/
-sudo cp -R -p /var/log/mongodb /ssd/log/
+sudo mkdir -p /ssd/mongodb
+sudo mkdir -p /ssd/log/mongodb
+sudo chown -R mongodb:mongodb /ssd/mongodb
+sudo chown -R mongodb:mongodb /ssd/log/mongodb
 sudo service mongod start
 
 for i in {1..15}; do

toolset/setup/linux/languages/swift.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+fw_installed swift && return 0
+
+fw_depends clang-3.9
+
+sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.9 100
+sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.9 100
+
+sudo apt-add-repository --yes ppa:george-edison55/cmake-3.x
+sudo apt-get update -qq
+sudo apt-get install -qqy cmake
+
+# TODO: Use Swift 4.1 stable once it's released.
+fw_get -O https://swift.org/builds/development/ubuntu1404/swift-DEVELOPMENT-SNAPSHOT-2018-01-30-a/swift-DEVELOPMENT-SNAPSHOT-2018-01-30-a-ubuntu14.04.tar.gz
+fw_untar swift-DEVELOPMENT-SNAPSHOT-2018-01-30-a-ubuntu14.04.tar.gz
+mv swift-DEVELOPMENT-SNAPSHOT-2018-01-30-a-ubuntu14.04 swift
+
+echo -e "export PATH=${IROOT}/swift/usr/bin:\$PATH" >> $IROOT/swift.installed
+
+source $IROOT/swift.installed

toolset/setup/linux/languages/swift3.sh

@@ -1,25 +0,0 @@
-#!/bin/bash
-
-fw_installed swift3 && return 0
-
-fw_depends clang-3.9
-
-sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-3.9 100
-sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-3.9 100
-
-sudo apt-add-repository --yes ppa:george-edison55/cmake-3.x
-sudo apt-get update -qq
-sudo apt-get install -qqy cmake
-
-SWIFT_SNAPSHOT="swift-3.1.1-RELEASE"
-SWIFT_SNAPSHOT_LOWERCASE="swift-3.1.1-release"
-UBUNTU_VERSION="ubuntu14.04"
-UBUNTU_VERSION_NO_DOTS="ubuntu1404"
-
-fw_get -O https://swift.org/builds/$SWIFT_SNAPSHOT_LOWERCASE/$UBUNTU_VERSION_NO_DOTS/$SWIFT_SNAPSHOT/$SWIFT_SNAPSHOT-$UBUNTU_VERSION.tar.gz
-fw_untar $SWIFT_SNAPSHOT-$UBUNTU_VERSION.tar.gz
-mv $SWIFT_SNAPSHOT-$UBUNTU_VERSION swift
-
-echo -e "export PATH=${IROOT}/swift/usr/bin:\$PATH" >> $IROOT/swift3.installed
-
-source $IROOT/swift3.installed

toolset/setup/linux/systools/cmysql.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-fw_depends swift3
+fw_depends swift
 
 fw_installed cmysql && return 0
 

toolset/setup/linux/systools/ctls.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-fw_depends swift3
+fw_depends swift
 
 fw_installed ctls && return 0