Documentation updates, add rules engine revision to network config request meta-data.

controller/README.md

@@ -1,15 +1,33 @@
 Network Controller Microservice
 ======
 
-ZeroTier's 16-digit network IDs are really just a concatenation of the 10-digit ZeroTier address of a network controller followed by a 6-digit (24-bit) network number on that controller. Fans of software defined networking will recognize this as a spin on the familiar [separation of data plane and control plane](http://sdntutorials.com/difference-between-control-plane-and-data-plane/) SDN design pattern.
+Every ZeroTier virtual network has a *network controller*. This is our reference implementation and is the same one we use to power our own hosted services at [my.zerotier.com](https://my.zerotier.com/). Network controllers act as configuration servers and certificate authorities for the members of networks. Controllers are located on the network by simply parsing out the first 10 digits of a network's 16-digit network ID: these are the address of the controller.
 
-This code implements the *node/NetworkController.hpp* interface to provide an embedded microservice configurable via the same local HTTP control plane as ZeroTier One iteself. It is built by default in ZeroTier One in desktop and server builds. This is the same code we use to run [my.zerotier.com](https://my.zerotier.com/), which is a web UI and API that runs in front of a pool of controllers.
+As of ZeroTier One version 1.2.0 this code is included in normal builds for desktop, laptop, and server (Linux, etc.) targets, allowing any device to create virtual networks without having to be rebuilt from source with special flags to enable this feature. While this does offer a convenient way to create ad-hoc networks or experiment, we recommend running a dedicated controller somewhere secure and stable for any "serious" use case.
 
-Data is stored in JSON format under `controller.d` in the ZeroTier working directory. It can be copied, tar'd, placed in `git`, etc. The files under `controller.d` should not be modified in place while the controller is running or data loss may result, but they can be safely backed up or copied.
+Controller data is stored in JSON format under `controller.d` in the ZeroTier working directory. It can be copied, rsync'd, placed in `git`, etc. The files under `controller.d` should not be modified in place while the controller is running or data loss may result, and if they are edited directly take care not to save corrupt JSON since that can also lead to data loss when the controller is restarted. Going through the API is strongly preferred to directly modifying these files.
+
+### Upgrading from Older Versions
+
+Older versions of this code used a SQLite database instead of in-filesystem JSON. A migration utility called `migrate-sqlite` is included here and *must* be used to migrate this data to the new format. If the controller is started with an old `controller.db` in its working directory it will terminate after printing an error to *stderr*. This is done to prevent "surprises" for those running DIY controllers using the old code.
+
+The migration tool is written in nodeJS and can be used like this:
+
+    cd migrate-sqlite
+    npm install
+    node migrate-sqlite.js <path to ZeroTier working directory>
+
+You may need to `sudo node ...` if the ZeroTier working directory is owned by root.
+
+This code will dump the contents of any `controller.db` in the ZeroTier working directory and recreate its data in the form of JSON objects under `controller.d`. The old `controller.db` will then be renamed to `controller.db.migrated` and the controller will start normally.
+
+After migrating make sure that the contents of `controller.d` are owned and writable by the user that will be running the ZeroTier controller process! (Usually this is root but controllers that don't also join networks are sometimes run as unprivileged users.)
+
+If you don't have nodeJS on the machine running ZeroTier it is perfectly fine to just copy `controller.db` to a directory on another machine and run the migration tool there. After running your migration the contents of `controller.d` can be tar'd up and copied back over to the controller. Just remember to rename or remove `controller.db` on the controller machine afterwords so the controller will start.
 
 ### Scalability and Reliability
 
-Controllers can in theory host up to 2^24 networks and serve many millions of devices (or more), but we recommend spreading large numbers of networks across many controllers for load balancing and fault tolerance reasons.
+Controllers can in theory host up to 2^24 networks and serve many millions of devices (or more), but we recommend spreading large numbers of networks across many controllers for load balancing and fault tolerance reasons. Since the controller uses the filesystem as its data store we recommend fast filesystems and fast SSD drives for heavily loaded controllers.
 
 Since ZeroTier nodes are mobile and do not need static IPs, implementing high availability fail-over for controllers is easy. Just replicate their working directories from master to backup and have something automatically fire up the backup if the master goes down. Many modern orchestration tools have built-in support for this. It would also be possible in theory to run controllers on a replicated or distributed filesystem, but we haven't tested this yet.
 
@@ -154,6 +172,7 @@ The entry types and their additional fields are:
 | `ACTION_ACCEPT`                 | Accept any packets matching this rule                             | (none)         |
 | `ACTION_TEE`                    | Send a copy of this packet to a node (rule parsing continues)     | `zt`           |
 | `ACTION_REDIRECT`               | Redirect this packet to another node                              | `zt`           |
+| `ACTION_DEBUG_LOG`              | Output debug info on match (if built with rules engine debug)     | (none)         |
 | `MATCH_SOURCE_ZEROTIER_ADDRESS` | Match VL1 ZeroTier address of packet sender.                      | `zt`           |
 | `MATCH_DEST_ZEROTIER_ADDRESS`   | Match VL1 ZeroTier address of recipient                           | `zt`           |
 | `MATCH_ETHERTYPE`               | Match Ethernet frame type                                         | `etherType`    |
@@ -174,10 +193,12 @@ The entry types and their additional fields are:
 | `MATCH_TAGS_BITWISE_OR`         | Match if both sides' tags OR to value                             | `id`,`value`   |
 | `MATCH_TAGS_BITWISE_XOR`        | Match if both sides` tags XOR to value                            | `id`,`value`   |
 
-Notes:
+Important notes about rules engine behavior:
 
- * `MATCH_DEST_ZEROTIER_ADDRESS` does not work for multicast packets since these have many and varying destinations and can take circuitous routes. Use MAC rules instead.
- * IPv4 and IPv6 rules do not match for frames that are not IPv4 or IPv6 respectively.
+ * IPv4 and IPv6 IP address rules do not match for frames that are not IPv4 or IPv6 respectively.
+ * `ACTION_DEBUG_LOG` is a no-op on nodes not built with `ZT_RULES_ENGINE_DEBUGGING` enabled (see Network.cpp). If that is enabled nodes will dump a trace of rule evaluation results to *stdout* when this action is encountered but will otherwise keep evaluating rules. This is used for basic "smoke testing" of the rules engine.
+ * Multicast packets and packets destined for bridged devices treated a little differently. They are matched more than once. They are matched at the point of send with a NULL ZeroTier destination address, meaning that `MATCH_DEST_ZEROTIER_ADDRESS` is useless. That's because the true VL1 destination is not yet known. Then they are matched again for each true VL1 destination. On these later subsequent matches TEE actions are ignored and REDIRECT rules are interpreted as DROPs. This prevents multiple TEE or REDIRECT packets from being sent to third party devices.
+ * Rules in capabilities are always matched as if the current device is the sender (inbound == false). A capability specifies sender side rules that can be enforced on both sides.
 
 #### `/controller/network/<network ID>/member`
 
@@ -228,6 +249,7 @@ Note that managed IP assignments are only used if they fall within a managed rou
 | clientMajorVersion    | integer       | Client major version or -1 if unknown             |
 | clientMinorVersion    | integer       | Client minor version or -1 if unknown             |
 | clientRevision        | integer       | Client revision or -1 if unknown                  |
+| clientProtocolVersion | integer       | ZeroTier protocol version reported by client      |
 | fromAddr              | string        | Physical address if known                         |
 
 The controller can only know a member's `fromAddr` if it's able to establish a direct path to it. Members behind very restrictive firewalls may not have this information since the controller will be receiving the member's requests by way of a relay. ZeroTier does not back-trace IP paths as packets are relayed since this would add a lot of protocol overhead.

include/ZeroTierOne.h

@@ -101,6 +101,11 @@ extern "C" {
  */
 #define ZT_MAX_NETWORK_MULTICAST_SUBSCRIPTIONS 4096
 
+/**
+ * Rules engine revision ID, which specifies rules engine capabilities
+ */
+#define ZT_RULES_ENGINE_REVISION 1
+
 /**
  * Maximum number of base (non-capability) network rules
  */
@@ -114,7 +119,7 @@ extern "C" {
 /**
  * Maximum number of per-member tags per network
  */
-#define ZT_MAX_NETWORK_TAGS 16
+#define ZT_MAX_NETWORK_TAGS 128
 
 /**
  * Maximum number of direct network paths to a given peer

node/Network.cpp

@@ -38,36 +38,6 @@
 // Uncomment to enable ZT_NETWORK_RULE_ACTION_DEBUG_LOG rule output to STDOUT
 #define ZT_RULES_ENGINE_DEBUGGING 1
 
-/*
-{
-	"name": "filter_log_test",
-	"private": true,
-	"v4AssignMode": {
-		"zt": true
-	},
-	"v6AssignMode": {
-		"rfc4193": true,
-		"zt": false,
-		"6plane": false
-	},
-	"routes": [
-		{ "target": "10.140.140.0/24", "via": null }
-	],
-	"ipAssignmentPools": [
-		{ "ipRangeStart": "10.140.140.2", "ipRangeEnd": "10.140.140.254" }
-	],
-	"rules": [
-		{ "type": "MATCH_ETHERTYPE", "etherType": 0x0800 },
-		{ "type": "ACTION_DEBUG_LOG" },
-
-		{ "type": "MATCH_ETHERTYPE", "etherType": 0x0800, "not": true },
-		{ "type": "ACTION_DEBUG_LOG" },
-
-		{ "type": "ACTION_ACCEPT" }
-	]
-}
-*/
-
 namespace ZeroTier {
 
 #ifdef ZT_RULES_ENGINE_DEBUGGING
@@ -162,7 +132,7 @@ static int _doZtFilter(
 #ifdef ZT_RULES_ENGINE_DEBUGGING
 	std::vector<std::string> dlog;
 	char dpbuf[1024];
-#endif
+#endif // ZT_RULES_ENGINE_DEBUGGING
 
 	for(unsigned int rn=0;rn<ruleCount;++rn) {
 		const ZT_VirtualNetworkRuleType rt = (ZT_VirtualNetworkRuleType)(rules[rn].t & 0x7f);
@@ -172,6 +142,9 @@ static int _doZtFilter(
 				if (thisSetMatches) {
 					return -1; // match, drop packet
 				} else {
+#ifdef ZT_RULES_ENGINE_DEBUGGING
+					dlog.clear();
+#endif // ZT_RULES_ENGINE_DEBUGGING
 					thisSetMatches = 1; // no match, evaluate next set
 				}
 				continue;
@@ -179,6 +152,9 @@ static int _doZtFilter(
 				if (thisSetMatches) {
 					return 1; // match, accept packet
 				} else {
+#ifdef ZT_RULES_ENGINE_DEBUGGING
+					dlog.clear();
+#endif // ZT_RULES_ENGINE_DEBUGGING
 					thisSetMatches = 1; // no match, evaluate next set
 				}
 				continue;
@@ -199,13 +175,19 @@ static int _doZtFilter(
 				if (rt == ZT_NETWORK_RULE_ACTION_REDIRECT) {
 					return -1; // match, drop packet (we redirected it)
 				} else {
+#ifdef ZT_RULES_ENGINE_DEBUGGING
+					dlog.clear();
+#endif // ZT_RULES_ENGINE_DEBUGGING
 					thisSetMatches = 1; // TEE does not terminate evaluation
 				}
 			}	continue;
 			case ZT_NETWORK_RULE_ACTION_DEBUG_LOG:
 #ifdef ZT_RULES_ENGINE_DEBUGGING
 				if (thisSetMatches) {
-					printf("[FILTER] MATCH %s->%s %.2x:%.2x:%.2x:%.2x:%.2x:%.2x->%.2x:%.2x:%.2x:%.2x:%.2x:%.2x inbound=%d noRedirect=%d frameLen=%u etherType=%u" ZT_EOL_S,
+					printf(" _ " ZT_EOL_S);
+					for(std::vector<std::string>::iterator m(dlog.begin());m!=dlog.end();++m)
+						printf(" | %s" ZT_EOL_S,m->c_str());
+					printf(" + MATCH %s->%s %.2x:%.2x:%.2x:%.2x:%.2x:%.2x->%.2x:%.2x:%.2x:%.2x:%.2x:%.2x inbound=%d noRedirect=%d frameLen=%u etherType=%u" ZT_EOL_S,
 						ztSource.toString().c_str(),
 						ztDest.toString().c_str(),
 						(unsigned int)macSource[0],
@@ -225,10 +207,8 @@ static int _doZtFilter(
 						frameLen,
 						etherType
 					);
-					for(std::vector<std::string>::iterator m(dlog.begin());m!=dlog.end();++m)
-						printf("         %s" ZT_EOL_S,m->c_str());
-					dlog.clear();
 				}
+				dlog.clear();
 #endif // ZT_RULES_ENGINE_DEBUGGING
 				thisSetMatches = 1; // DEBUG_LOG does not terminate evaluation
 				continue;
@@ -793,6 +773,7 @@ void Network::requestConfiguration()
 	rmd.add(ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_MAX_CAPABILITY_RULES,(uint64_t)ZT_MAX_CAPABILITY_RULES);
 	rmd.add(ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_MAX_NETWORK_TAGS,(uint64_t)ZT_MAX_NETWORK_TAGS);
 	rmd.add(ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_FLAGS,(uint64_t)0);
+	rmd.add(ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_RULES_ENGINE_REV,(uint64_t)ZT_RULES_ENGINE_REVISION);
 
 	if (ctrl == RR->identity.address()) {
 		if (RR->localNetworkController) {

node/NetworkConfig.hpp

@@ -107,6 +107,8 @@ namespace ZeroTier {
 #define ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_NODE_MAJOR_VERSION "majv"
 #define ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_NODE_MINOR_VERSION "minv"
 #define ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_NODE_REVISION "revv"
+// Rules engine revision
+#define ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_RULES_ENGINE_REV "revr"
 // Maximum number of rules per network this node can accept
 #define ZT_NETWORKCONFIG_REQUEST_METADATA_KEY_MAX_NETWORK_RULES "mr"
 // Maximum number of capabilities this node can accept

one.cpp

@@ -1086,6 +1086,12 @@ int main(int argc,char **argv)
 		}
 	}
 
+	// This can be removed once the new controller code has been around for many versions
+	if (OSUtils::fileExists((homeDir + ZT_PATH_SEPARATOR_S + "controller.db").c_str(),true)) {
+		fprintf(stderr,"%s: FATAL: an old controller.db exists in %s -- see instructions in controller/README.md for how to migrate!" ZT_EOL_S,argv[0],homeDir.c_str());
+		return 1;
+	}
+
 #ifdef __UNIX_LIKE__
 #ifndef ZT_ONE_NO_ROOT_CHECK
 	if ((!skipRootCheck)&&(getuid() != 0)) {