Update release notes, rip out trusted paths which were clunky and rarely used and of dubious value, and some other cleanup.

RELEASE-NOTES.md

@@ -1,25 +1,38 @@
 ZeroTier Release Notes
 ======
 
-# Version 2.0.0
+# Version 1.9.0 (2.0 beta)
 
-### This is a major milestone release with numerous changes. It's technically backward compatibile with older nodes but we highly recommend upgrading as soon as possible. We will not force upgrade however as this is a major release and contains changes that may require updates to some configurations, such as CLI changes that may required updates to scripts.
+Version 2.0 is a very significant release with many changes. It remains backward compatibility to version 1.4.0 (and possibly earlier versions but this is not guaranteed) but makes numerous local and behavioral changes that should be reviewed before upgrading production systems.
+
+After this release we're going to be working to get to a more frequent, less extreme, more "agile" release cadence.
+
+Protocol changes:
+
+ * Trusted paths have been completely removed. The new AES mode is so fast on CPUs with AES acceleration that much of the rationale for this is gone, and this feature was never used much to begin with due to inconvenience and obvious security concerns. Environments using trusted paths will need to upgrade all nodes at once.
+ * The symmetric encryption algorithm and mode is now AES-GMAC-SIV, a variation of AES-GCM using the same primitives but offering superior security bounds and behavior under non-ideal conditions. It's also a lot faster than Salsa20/12 and Poly1305 on CPUs with AES acceleration (almost all desktops, laptops, and newer routers and phones). Salsa20/12 with Poly1305 is still supported for communication with older versions and small devices that lack AES acceleration.
+ * A new identity type (1) has been introduced that contains both Curve25519 and NIST P-384 public key types, but classic type 0 remains the default for new identities for now. ECDH key agreement between V1 identities uses both keys and hashes the resulting secrets to yield security equal to the best of the two, but V1 identities can also agree with V0 identities using only their Curve25519 component.
+ * Roots can now be joined and left like networks in a much more convenient way, and the old "moon" and "planet" terminology is deprecated.
+ * A new peer to peer multicast algorithm has been introduced that offers much better scalability and better performance, especially when the physical network itself is hub-and-spoke with many low latency peers connected by higher latency WAN links.
+ * Forward secrecy is finally supported via periodic re-keying using ephemeral asymmetric keys. Both Curve25519 and NIST P-384 keys are used with secrets being hashed to provide security equal to the stronger of the two curves.
+ * As part of forward secrecy implementation peers now always exchange HELLO messages even if they don't have a direct path.
+ * Compression is only enabled for control packets as almost all data packets are largely un-compressable.
+ * New NAT traversal tricks have been added, such as (ab)use of port 500.
+
+Code changes:
 
  * Migrated from GNU make to cmake for easier cross platform builds and simplified build files.
- * Service management code for desktop, laptop, and server ZeroTier service is now written in Go. Core and packet handling (performance critical) code remains in C++.
- * Reworked CLI for improved ease of use, more readable and detailed display output, and added new root management commands.
- * "Moon" and "planet" terminology and associated commands are now gone in favor of fully decentralized roots.
- * Service now has a fully multithreaded UDP I/O path written in C++ for superior scaling on large systems.
- * Entirely new P2P based multicast propagation algorithm for improved multi-data-center and global scale multicast propagation and improved multicast discovery. Some backward compatibility is included with pre-2.0 but upgrading of all nodes is recommended if you depend on multicast for anything but ARP, NDP, and occasional service advertisements. 
- * ZeroTier now implements ephemeral keys with continuos re-keying for forward secrecy! (FINALLY!)
- * Separated root into its own highly optimized code base.
- * Changed default primary ZeroTier port to 893 (for new nodes) to exploit friendlier NAT behavior on ports numbered under 1024. Added some more aggressive NAT-t techniques that work with this and can often traverse symmetric NATs.
- * Added stubs for future support for alternative transports including HTTP/HTTPS, WebRTC, Web Sockets, and "naked" Ethernet (on LAN).
- * Improved packet assemble/decode performance by moving to a lock-free bounds-checking scheme for buffers and a shared memory buffer pool.
- * AES encryption is now the default for communicating with 2.0+ nodes.
- * Added support for a new identity type with NIST P-384 curves for future FIPS compliance. Curve25519 is still the default.
- * Compression is now only enabled for control packets as most data these days is encrypted or already compressed. This improves performance in almost all cases.
- * Minor API changes (for those who use the core directly) to support faster buffer handling, reduced memory copying, exposure of identity functions and full node identity, and improved state object load/store semantics.
+ * The core network hypervisor has been significantly refactored, almost amounting to a partial rewrite.
+ * Critical packet handling paths have been streamlined with unnecessary memcpy() steps removed.
+ * Host service code has been completely rewritten in Go. Packet handling code remains in C++, but Go offers superior developer productivity when it comes to implementing more complex local service and local API features. Go imposes a little bit more memory overhead but not much and has been tuned to minimize memory use.
+
+User interface:
+
+ * Command line interface has been redesigned and rewritten. Old commands names are supported but their output will be different.
+
+Other things:
+
+ * The V2 design, protocol, and cryptographic primitives (AES-GMAC-SIV) have been security audited by [Trail of Bits](https://www.trailofbits.com), and the code is being audited as well prior to full 2.0 release.
 
 ---
 

go/cmd/zerotier/cli/help.go

@@ -15,18 +15,15 @@ package cli
 
 import (
 	"fmt"
-
 	"zerotier/pkg/zerotier"
 )
 
-var copyrightText = fmt.Sprintf(`ZeroTier Network Virtualization Service Version %d.%d.%d
-(c)2013-2020 ZeroTier, Inc.
-Licensed under the ZeroTier BSL (see LICENSE.txt)`, zerotier.CoreVersionMajor, zerotier.CoreVersionMinor, zerotier.CoreVersionRevision)
-
 // Help dumps help to stdout
 func Help() {
-	fmt.Println(copyrightText)
-	fmt.Println(`
+	fmt.Printf(`ZeroTier Network Hypervisor Service Version %d.%d.%d
+(c)2013-2020 ZeroTier, Inc.
+Licensed under the ZeroTier BSL (see LICENSE.txt)
+
 Usage: zerotier [-options] <command> [command args]
 
 Global Options:
@@ -38,14 +35,14 @@ Commands:
   help                                 Show this help
   version                              Print version
   service                              Start as service
-  status                               Show ZeroTier status and config
-  peers                                Show VL1 peers and link information
-  roots                                Show only root peers
-  addroot <identity> [IP/port]         Add root with optional bootstrap IP
-  removeroot <address|identity>        Remove root
+  status                               Show node status, identity, and config
+  peers                                List all VL1 peers
+  roots                                List root peers
+  addroot <path/URL to spec>           Add root
+  removeroot <address>                 Remove a peer from the root list
   join <network ID> [fingerprint]      Join a virtual network
   leave <network ID>                   Leave a virtual network
-  networks                             List joined VL2 virtual networks
+  networks                             List VL2 virtual networks
   network <network ID>                 Show verbose network info
   set <network ID> [option] [value]    Get or set a network config option
     manageips <boolean>                Is IP management allowed?
@@ -54,32 +51,29 @@ Commands:
     globalroutes <boolean>             Can global IP space routes be set?
     defaultroute <boolean>             Can default route be overridden?
   set [option] [value]                 Get or set a service config option
-    phy <IP/bits> blacklist <boolean>  Set or clear blacklist for CIDR
-    phy <IP/bits> trust <path ID/0>    Set or clear trusted path ID for CIDR
-    port <port>                        Set primary port for P2P links
-    secondaryport <port/0>             Set secondary P2P port (0 disables)
-    portsearch <boolean>               Enable/disable port search on startup
-    portmapping <boolean>              Enable/disable use of uPnP/NAT-PMP
+    port <port>                        Primary P2P port
+    secondaryport <port/0>             Secondary P2P port (0 to disable)
+    blacklist cidr <IP/bits> <boolean> Toggle physical path blacklisting
+    blacklist if <prefix> <boolean>    Toggle interface prefix blacklisting
+    portmap <boolean>                  Toggle use of uPnP or NAT-PMP
   identity <command> [args]            Identity management commands
     new [c25519|p384]                  Create identity pair (default: c25519)
     getpublic <identity>               Extract only public part of identity
     validate <identity>                Locally validate an identity
     sign <identity> <file>             Sign a file with an identity's key
     verify <identity> <file> <sig>     Verify a signature
+    makeroot <identity> <address> ...  Make a root spec (see docs)
 
 The 'service' command does not exit until the service receives a signal.
-This is typically run from launchd (Mac), systemd or init (Linux), etc.
+This is typically run from launchd (Mac), systemd or init (Linux), a Windows
+service harness (Windows), etc.
 
 If 'set' is followed by a 16-digit hex number it will get/set network config
-options. Otherwise it will get/set service options. Run with no arguments to
-see all options.
+options. Otherwise it will get/set local options that pertain to the entire
+node.
 
-An identity can be specified as a file or directly. This is auto-detected.
+Identities can be specified verbatim on the command line or as a path to
+a file. This is detected automatically.
 
-Most commands require a secret token to permit control of a running
-service. The CLI will automatically try to read this token from the
-authtoken.secret file in the service's working directory and then from a
-file called .zerotierauth in the user's home directory. The -t option can
-be used to explicitly specify a location.`)
-	fmt.Println()
+`,zerotier.CoreVersionMajor, zerotier.CoreVersionMinor, zerotier.CoreVersionRevision)
 }

include/ZeroTierCore.h

@@ -380,12 +380,12 @@ enum ZT_TracePacketDropReason
 	ZT_TRACE_PACKET_DROP_REASON_PEER_TOO_OLD =            1,
 	ZT_TRACE_PACKET_DROP_REASON_MALFORMED_PACKET =        2,
 	ZT_TRACE_PACKET_DROP_REASON_MAC_FAILED =              3,
-	ZT_TRACE_PACKET_DROP_REASON_NOT_TRUSTED_PATH =        4,
-	ZT_TRACE_PACKET_DROP_REASON_RATE_LIMIT_EXCEEDED =     5,
-	ZT_TRACE_PACKET_DROP_REASON_INVALID_OBJECT =          6,
-	ZT_TRACE_PACKET_DROP_REASON_INVALID_COMPRESSED_DATA = 7,
-	ZT_TRACE_PACKET_DROP_REASON_UNRECOGNIZED_VERB =       8,
-	ZT_TRACE_PACKET_DROP_REASON_REPLY_NOT_EXPECTED =      9
+	ZT_TRACE_PACKET_DROP_REASON_RATE_LIMIT_EXCEEDED =     4,
+	ZT_TRACE_PACKET_DROP_REASON_INVALID_OBJECT =          5,
+	ZT_TRACE_PACKET_DROP_REASON_INVALID_COMPRESSED_DATA = 6,
+	ZT_TRACE_PACKET_DROP_REASON_UNRECOGNIZED_VERB =       7,
+	ZT_TRACE_PACKET_DROP_REASON_REPLY_NOT_EXPECTED =      8
+
 };
 
 /**
@@ -1118,22 +1118,6 @@ typedef struct
 	int permanent;
 } ZT_InterfaceAddress;
 
-/**
- * Physical path configuration
- */
-typedef struct
-{
-	/**
-	 * If non-zero set this physical network path to be trusted to disable encryption and authentication
-	 */
-	uint64_t trustedPathId;
-
-	/**
-	 * Physical path MTU from ZT_MIN_PHYSMTU and ZT_MAX_PHYSMTU or <= 0 to use default
-	 */
-	int mtu;
-} ZT_PhysicalPathConfiguration;
-
 /**
  * Physical network path to a peer
  */
@@ -1154,11 +1138,6 @@ typedef struct
 	 */
 	int64_t lastReceive;
 
-	/**
-	 * Is this a trusted path? If so this will be its nonzero ID.
-	 */
-	uint64_t trustedPathId;
-
 	/**
 	 * Is path alive?
 	 */
@@ -1915,16 +1894,6 @@ ZT_SDK_API int ZT_Node_sendUserMessage(ZT_Node *node,void *tptr,uint64_t dest,ui
  */
 ZT_SDK_API void ZT_Node_setController(ZT_Node *node,void *networkConfigMasterInstance);
 
-/**
- * Set configuration for a given physical path
- *
- * @param node Node instance
- * @param pathNetwork Network/CIDR of path or NULL to clear the cache and reset all paths to default
- * @param pathConfig Path configuration or NULL to erase this entry and therefore reset it to NULL
- * @return OK or error code
- */
-ZT_SDK_API enum ZT_ResultCode ZT_Node_setPhysicalPathConfiguration(ZT_Node *node,const struct sockaddr_storage *pathNetwork,const ZT_PhysicalPathConfiguration *pathConfig);
-
 /* ---------------------------------------------------------------------------------------------------------------- */
 
 /**

node/Node.cpp

@@ -582,12 +582,6 @@ bool Node::externalPathLookup(void *tPtr,const Identity &id,int family,InetAddre
 	return false;
 }
 
-ZT_ResultCode Node::setPhysicalPathConfiguration(const struct sockaddr_storage *pathNetwork, const ZT_PhysicalPathConfiguration *pathConfig)
-{
-	RR->topology->setPhysicalPathConfiguration(pathNetwork,pathConfig);
-	return ZT_RESULT_OK;
-}
-
 bool Node::localControllerHasAuthorized(const int64_t now,const uint64_t nwid,const Address &addr) const
 {
 	m_localControllerAuthorizations_l.lock();
@@ -973,15 +967,6 @@ void ZT_Node_setController(ZT_Node *node,void *networkControllerInstance)
 	} catch ( ... ) {}
 }
 
-enum ZT_ResultCode ZT_Node_setPhysicalPathConfiguration(ZT_Node *node,const struct sockaddr_storage *pathNetwork,const ZT_PhysicalPathConfiguration *pathConfig)
-{
-	try {
-		return reinterpret_cast<ZeroTier::Node *>(node)->setPhysicalPathConfiguration(pathNetwork,pathConfig);
-	} catch ( ... ) {
-		return ZT_RESULT_FATAL_ERROR_INTERNAL;
-	}
-}
-
 void ZT_version(int *major,int *minor,int *revision,int *build)
 {
 	if (major)

node/Node.hpp

@@ -280,15 +280,6 @@ public:
 	 */
 	bool externalPathLookup(void *tPtr,const Identity &id,int family,InetAddress &addr);
 
-	/**
-	 * Set physical path configuration
-	 *
-	 * @param pathNetwork Physical path network/netmask bits (CIDR notation)
-	 * @param pathConfig Path configuration
-	 * @return Return to pass through to external API
-	 */
-	ZT_ResultCode setPhysicalPathConfiguration(const struct sockaddr_storage *pathNetwork,const ZT_PhysicalPathConfiguration *pathConfig);
-
 	/**
 	 * @return This node's identity
 	 */

node/Protocol.hpp

@@ -249,12 +249,13 @@
 #define ZT_PROTO_PACKET_MAC_INDEX         19
 #define ZT_PROTO_PACKET_VERB_INDEX        27
 
-#define ZT_PROTO_HELLO_NODE_META_INSTANCE_ID      "i"
-#define ZT_PROTO_HELLO_NODE_META_LOCATOR          "l"
-#define ZT_PROTO_HELLO_NODE_META_SOFTWARE_VENDOR  "s"
-#define ZT_PROTO_HELLO_NODE_META_COMPLIANCE       "c"
-#define ZT_PROTO_HELLO_NODE_META_EPHEMERAL_PUBLIC "e"
-#define ZT_PROTO_HELLO_NODE_META_EPHEMERAL_ACK    "E"
+#define ZT_PROTO_HELLO_NODE_META_INSTANCE_ID                "i"
+#define ZT_PROTO_HELLO_NODE_META_PREFERRED_SYMMETRIC        "a"
+#define ZT_PROTO_HELLO_NODE_META_LOCATOR                    "l"
+#define ZT_PROTO_HELLO_NODE_META_SOFTWARE_VENDOR            "s"
+#define ZT_PROTO_HELLO_NODE_META_COMPLIANCE                 "c"
+#define ZT_PROTO_HELLO_NODE_META_EPHEMERAL_PUBLIC           "e"
+#define ZT_PROTO_HELLO_NODE_META_EPHEMERAL_ACK              "E"
 
 static_assert(ZT_PROTO_MAX_PACKET_LENGTH < ZT_BUF_MEM_SIZE,"maximum packet length won't fit in Buf");
 static_assert(ZT_PROTO_PACKET_ENCRYPTED_SECTION_START == (ZT_PROTO_MIN_PACKET_LENGTH-1),"encrypted packet section must start right before protocol verb at one less than minimum packet size");
@@ -339,6 +340,7 @@ enum Verb
 	 * Dictionary fields (defines start with ZT_PROTO_HELLO_NODE_META_):
 	 * 
 	 *   INSTANCE_ID - a 64-bit unique value generated on each node start
+	 *   PREFERRED_SYMMETRIC - preferred symmetric encryption mode
 	 *   LOCATOR - signed record enumerating this node's trusted contact points
 	 *   EPHEMERAL_PUBLIC - Ephemeral public key(s)
 	 * 

node/Topology.cpp

@@ -59,18 +59,6 @@ SharedPtr<Peer> Topology::add(void *tPtr, const SharedPtr<Peer> &peer)
 	return peer;
 }
 
-void Topology::setPhysicalPathConfiguration(const struct sockaddr_storage *pathNetwork, const ZT_PhysicalPathConfiguration *pathConfig)
-{
-	RWMutex::Lock l(m_paths_l);
-	if (!pathNetwork) {
-		m_physicalPathConfig.clear();
-	} else if (!pathConfig) {
-		m_physicalPathConfig.erase(asInetAddress(*pathNetwork));
-	} else {
-		m_physicalPathConfig[asInetAddress(*pathNetwork)] = *pathConfig;
-	}
-}
-
 struct p_RootSortComparisonOperator
 {
 	ZT_INLINE bool operator()(const SharedPtr<Peer> &a, const SharedPtr<Peer> &b) const noexcept

node/Topology.hpp

@@ -182,11 +182,6 @@ public:
 			allPeers.push_back(i->second);
 	}
 
-	/**
-	 * Set or clear physical path configuration (called via Node::setPhysicalPathConfiguration)
-	 */
-	void setPhysicalPathConfiguration(const struct sockaddr_storage *pathNetwork,const ZT_PhysicalPathConfiguration *pathConfig);
-
 	/**
 	 * Add or update a root server and its locator
 	 *
@@ -230,33 +225,26 @@ private:
 	// This gets an integer key from an InetAddress for looking up paths.
 	static ZT_INLINE uint64_t s_getPathKey(const int64_t l,const InetAddress &r) noexcept
 	{
+		// SECURITY: these will be used as keys in a Map<> which uses its own hasher that
+		// mixes in a per-invocation secret to work against hash collision attacks. See the
+		// map hasher in Containers.hpp. Otherwise the point here is really really fast
+		// path lookup by address. The number of paths is never likely to be high enough
+		// for a collision to be something we worry about. That would require a minimum of
+		// millions and millions of paths on a single node.
 		if (r.family() == AF_INET) {
-			return ((uint64_t)(reinterpret_cast<const sockaddr_in *>(&r)->sin_addr.s_addr) << 24U) +
-			       ((uint64_t)reinterpret_cast<const sockaddr_in *>(&r)->sin_port << 8U) +
-						 (uint64_t)l;
+			return ((uint64_t)(r.as.sa_in.sin_addr.s_addr) << 32U) ^ ((uint64_t)r.as.sa_in.sin_port << 16U) ^ (uint64_t)l;
 		} else if (r.family() == AF_INET6) {
-#ifdef ZT_NO_UNALIGNED_ACCESS
-			uint64_t htmp[2];
-			Utils::copy<16>(htmp,reinterpret_cast<const sockaddr_in6 *>(&r)->sin6_addr.s6_addr);
-			const uint64_t h = htmp[0] ^ htmp[1];
-#else
-			const uint64_t h = reinterpret_cast<const uint64_t *>(reinterpret_cast<const sockaddr_in6 *>(&r)->sin6_addr.s6_addr)[0] ^
-			                   reinterpret_cast<const uint64_t *>(reinterpret_cast<const sockaddr_in6 *>(&r)->sin6_addr.s6_addr)[1];
-#endif
-			return (h + (uint64_t)Utils::ntoh(reinterpret_cast<const struct sockaddr_in6 *>(&r)->sin6_port)) ^ (uint64_t)l;
+			return Utils::loadAsIsEndian<uint64_t>(r.as.sa_in6.sin6_addr.s6_addr) + Utils::loadAsIsEndian<uint64_t>(r.as.sa_in6.sin6_addr.s6_addr + 8) + (uint64_t)r.as.sa_in6.sin6_port + (uint64_t)l;
 		} else {
+			// This should never really be used but it's here just in case.
 			return (uint64_t)Utils::fnv1a32(reinterpret_cast<const void *>(&r),sizeof(InetAddress)) + (uint64_t)l;
 		}
 	}
 
 	const RuntimeEnvironment *const RR;
-
-	RWMutex m_paths_l; // locks m_physicalPathConfig and m_paths
+	RWMutex m_paths_l; // locks m_paths
 	RWMutex m_peers_l; // locks m_peers, m_roots, and m_rootPeers
-
-	Map< InetAddress,ZT_PhysicalPathConfiguration > m_physicalPathConfig;
 	Map< uint64_t,SharedPtr<Path> > m_paths;
-
 	Map< Address,SharedPtr<Peer> > m_peers;
 	Map< Identity,Locator > m_roots;
 	Vector< SharedPtr<Peer> > m_rootPeers;