Merge branch 'master' into github-action

.github/workflows/buildandrelease.yml

@@ -45,7 +45,7 @@ jobs:
         uses: svenstaro/upload-release-action@v2
         with:
           repo_token: ${{ secrets.GITHUB_TOKEN }}
-          file: build/netclient
+          file: netclient/build/netclient
           tag: ${{ env.NETMAKER_VERSION }}
           overwrite: true
           prerelease: true
@@ -89,7 +89,7 @@ jobs:
         uses: svenstaro/upload-release-action@v2
         with:
           repo_token: ${{ secrets.GITHUB_TOKEN }}
-          file: build/netclient-arm64/netclient
+          file: netclient/build/netclient-arm64/netclient
           tag: ${{ env.NETMAKER_VERSION }}
           overwrite: true
           prerelease: true

Dockerfile

@@ -27,5 +27,5 @@ COPY --from=builder /app/config config
 EXPOSE 8081
 EXPOSE 50051
 
-CMD ["./app"]
+CMD ["./app", "--clientmode=off"]
 

README.md

@@ -1,5 +1,3 @@
-
-
 <p align="center">
   <img src="netmaker.png"><break/>
 </p>
@@ -8,7 +6,7 @@
 </p>
 
 ## What is Netmaker?
-Netmaker is a tool for creating and managing virtual networks. The goal is to make virtual/overlay/mesh networking easy for non-networking people. It should be like clicking a button. Netmaker consists of a server, an agent, and a UI. You spin up the Netmaker server and then install netclient (the agent) on your computers. Netmaker will do the rest. It will tell all of your computers how to reach each other and will keep them informed of any changes to the network.
+Netmaker is a tool for creating and managing virtual networks. The goal is to provide functionality similar to Tailscale, ZeroTier, and Nebula, but faster, easier, and more dynamic. It should be like clicking a button. Netmaker consists of a server, an agent, and a UI. You spin up the Netmaker server and then install netclient (the agent) on your computers. Netmaker will do the rest. It will tell all of your computers how to reach each other and will keep them informed of any changes to the network.
 
 Netmaker's handy dandy UI can be found [here](https://github.com/gravitl/netmaker-ui).
 
@@ -47,25 +45,34 @@ In future releases, we have plans to support other platforms such as Windows and
 
 ## Quick Start
 
-[Video Tutorial](https://youtu.be/PWLPT320Ybo)
-
+[Intro/Overview Video Tutorial](https://youtu.be/PWLPT320Ybo)  
+[Site-to-Site Video Tutorial](https://youtu.be/krCKBJhwwDk)  
+   
 #### Prereqs:
 1. A server with an IP reachable by your computers (a small ec2 instance or droplet would do just fine).
 2. Linux installed on the above server (we use Ubuntu, but anything that runs Docker should work).
-3. Install Docker (can run without Docker as well, but is not preferred. If this is a requirement, view the Advanced Usage docs).
+3. Install Docker if running in Docker Mode (see below).
 
 
 #### Launch Netmaker:
+
+Netmaker v0.2 introduces the server as a 'client'. This means the server can add itself into networks if you would like. To do so, Netmaker requires privileged access where it is running, and needs to modify the host filesystem. To run in this mode, we are not currently using Docker.
+
+**If you would like to run with "client mode", you can use the following script to deploy:**
+`sudo curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/v0.2/netmaker-install-clientmode.sh | sudo SERVER_DOMAIN=< your server IP > sh -`
+
+**If you would like to run without "client mode", and manually add/remove the server from networks:**
+
 1. Clone this repo or just copy contents of "docker-compose.yml" to your Netmaker server (from prereqs).
-2. In docker-compose.yml, change BACKEND_URL to the public IP ofthat machine.
+2. In docker-compose.yml, change BACKEND_URL to the public IP of your server.
 3. Run `sudo docker-compose up`
 4. Navigate to your server's IP in the browser and you should see the Netmaker UI asking to create a new admin user.
 5. Create a new admin user
-6. . Click "Create Network" and fill out the details
-7. You are now ready to begin using Netmaker. Create a key or enable manual node sign up so that your nodes can connect.
+6. You are now ready to begin using Netmaker. There should be a default network you can use or you can create your own. Then, Create a key or enable manual node sign up so that your nodes can connect.
 
 #### On your machines :
-Run the following: `curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/v0.1/netclient-install.sh | SERVER_URL=<your server ip>:50051 NET_NAME=<your network name> KEY=<your access key> sh -`  
+A command will be displayed when you generate a token for signing up nodes. Run it on each machine you would like to connect.
+`curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/v0.2/netclient-install.sh | KEY=<your access key> sh -`  
 (Note: Key can be left out if manual node signup is enabled)
 
 #### LICENSE
@@ -76,4 +83,3 @@ Netmaker's source code and all artifacts in this repository are freely available
 
 Email: [email protected]  
 Discord: https://discord.gg/zRb9Vfhk8A
-

docker-compose.yml

@@ -16,7 +16,7 @@ services:
     container_name: netmaker
     depends_on:
       - mongodb
-    image: gravitl/netmaker:v0.1-hotfix
+    image: gravitl/netmaker:v0.2
     ports:
       - "8081:8081"
       - "50051:50051"
@@ -27,7 +27,7 @@ services:
     container_name: netmaker-ui
     depends_on:
       - netmaker
-    image: gravitl/netmaker-ui:v0.1
+    image: gravitl/netmaker-ui:v0.2
     ports:
       - "80:80"
     environment:

docs/ROADMAP.md

@@ -22,34 +22,34 @@
  - [x] Config file for modifying node 
 
 ### 0.2
-- [ ] Separate out README into DOCS folder with the following:
-	- [ ] API Docs
-	- [ ] Getting Started
+- [x] Separate out README into DOCS folder with the following:
+	- [x] API Docs
+	- [x] Usage
 	- [ ] Advanced Usage
-	- [ ] Contributing
+	- [x] Contributing
 	- [ ] Roadmap
 	- [ ] Troubleshooting
 
 **Server:**
- - [ ] Allow tracking multiple networks per node
+ - [x] Allow tracking multiple networks per node
  - [ ] Configure Check-in thresholds
  - [ ] Separate sign-up endpoint to allow VPN-only comms after joining network
  - [ ] Swagger Docs
- - [ ] Build Out README
- - [ ] Encode Server, Port, and Network into Keys
+ - [x] Build Out README
+ - [x] Encode Server, Port, and Network into Keys
  - [ ] Switch to Unique ID for nodes instead of MacAddress
- - [ ] Public Key refresh
+ - [x] Public Key refresh
  - [ ] Enable  ipv6 addresses
- - [ ] Have a "default" network created at startup
+ - [x] Have a "default" network created at startup
  
 **Agent:**
- - [ ] Test / get working on multiple linux platforms
+ - [x] Test / get working on multiple linux platforms
  - [ ] Set private DNS via etc hosts (node name + ip). Make it optional flag on agent.
- - [ ] Decode Server, Port, and Network from Key
+ - [x] Decode Server, Port, and Network from Key
  - [ ] Service ID / unit file for SystemD Service
- - [ ] Allow multiple interfaces
+ - [x] Allow multiple interfaces
  - [ ] Use "Check in interval" from server
- - [ ] Pre-req check on machine (wg, port forwarding)
+ - [x] Pre-req check on machine (wg, port forwarding)
  - [ ]  Enable  ipv6 addresses
 
 ### 0.3

docs/USAGE.md

@@ -2,39 +2,23 @@
 
 This guide covers advanced usage of Netmaker. If you are just looking to get started quickly, check out the Quick Start in the [README](../README.md).
 
-## Index
-
- - Config
-	 - Server Config
-	 - Agent Config
-	 - UI Config
- - Creating Your  Network
-	 - Creating Networks
-	 - Creating Keys
-	 - Creating Nodes
-	 - Managing Your Network
- - Cleaning up
- - Non-Docker Installation
- - Building
- - Testing 
-
 ## Server Config
 Netmaker settings can be set via Environment Variables or Config file. There are also a couple of runtime arguments that can optionally be set.
 
 ### Environment Variables
-**APP_ENV**: default=dev. Determines which environment file to use. Will look under config/environments/APP_ENV.yaml. For instance, you can  have different environments  for dev,  test, and prod,  and store different settinggs  accordingly.
-**GRPC_PORT**: default=50051. The port for GRPC (node/client) communications
-**API_PORT**: default=8081. The port for API and UI communications
-**MASTER_KEY**: default=secretkey. The skeleton key used for authenticating with server as administrator.
-
-MongoDB Connection Env Vars:
-**MONGO_USER**:default=admin
-**MONGO_HOST**:default=password
-**MONGO_PASS**:default=localhost
-**MONGO_PORTS**:default=27017
-**MONGO_OPTS**:default=/?authSource=admin
-
-**BACKEND_URL**: default=nil. The address of the server. Used for setting token values  for client/nodes. If not set, will run a command to retrieve the server URL.
+**APP_ENV**: default=dev. Determines which environment file to use. Will look under config/environments/APP_ENV.yaml. For instance, you can  have different environments  for dev,  test, and prod,  and store different settinggs  accordingly.  
+**GRPC_PORT**: default=50051. The port for GRPC (node/client) communications  
+**API_PORT**: default=8081. The port for API and UI communications  
+**MASTER_KEY**: default=secretkey. The skeleton key used for authenticating with server as administrator.  
+  
+MongoDB Connection Env Vars:  
+**MONGO_USER**:default=admin   
+**MONGO_HOST**:default=password  
+**MONGO_PASS**:default=localhost   
+**MONGO_PORTS**:default=27017  
+**MONGO_OPTS**:default=/?authSource=admin  
+  
+**BACKEND_URL**: default=nil. The address of the server. Used for setting token values  for client/nodes. If not set, will run a command to retrieve the server URL.  
 
 ###   Config File
 Stored as config/environments/*.yaml. Default used is dev.yaml
@@ -63,26 +47,61 @@ Stored as config/environments/*.yaml. Default used is dev.yaml
 **clientmode**: (default=on) E.x.: `sudo netmaker --clientmode=off` Run the Server as a client (node) as well.
 **defaultnet**:  (default=on) E.x.: `sudo netmaker --defaultnet=off` Create a default network on startup.
 
-### Running the Backend Components on Different Machines
-HTTP, GRPC, MongoDB
+## Client  Config
 
-### Non-Docker Installation
+Client config files are stored under /etc/netclient  per network as /etc/netclient/netconfig-< network name >  
+**server:**  
+    address: The address:port of the server  
+    accesskey: The acceess key used to sign up with the server  
+  
+**node:**  
+    name: a displayname for the node, e.g. "mycomputer" 
+    interface:  the network interface name, by default something like "nm-"  
+    network: the netmaker network being attached to  
+    password: the node's hashed password. Can be changed by putting a value in here and setting "postchanges" to "true"   
+    macaddress: the mac address of the node  
+    localaddress: the local network address   
+    wgaddress: the wireguard private address  
+    roamingoff: flag to update the IP address automatically based on network changes  
+    islocal: whether or not this is a local or public network   
+    allowedips: the allowedips addresses that other nodes will recieve  
+    localrange: the local address range if it's a local network  
+    postup: post up rules for gateway nodes  
+    postdown: post down rules for gateway nodes  
+    port: the wiregard port   
+    keepalive: the default keepalive value between this and all other nodes  
+    publickey: the public key other nodes will use to access this node   
+    privatekey: the private key of the nodes (this field does nothing)  
+    endpoint: the reachable endpoint of the node for routing, either local or public.  
+    postchanges: either "true" or "false" (with quotes). If true, will post any changes you make to the remote server. 
+
+
+## Non-Docker Installation
+
+### MongoDB Setup
+1.  Install MongoDB on your server. For Ubuntu: `sudo apt install -y mongodb`. For more advanced installation or other operating systems, see  the [MongoDB documentation](https://docs.mongodb.com/manual/administration/install-community/).
+
+2. Create a user:
+`mongo admin`
+`db.createUser({ user: "mongoadmin" , pwd: "mongopass", roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase"]})`
 
 ### Server Setup
- 1. Get yourself a linux server and make sure it has a public IP.
- 2. Deploy MongoDB `docker volume create mongovol && docker run -d --name mongodb -v mongovol:/data/db --network host -e MONGO_INITDB_ROOT_USERNAME=mongoadmin -e MONGO_INITDB_ROOT_PASSWORD=mongopass mongo --bind_ip 0.0.0.0 `
- 3. Pull this repo: `git clone https://github.com/gravitl/netmaker.git`
- 4. Switch to the directory and source the default env vars `cd netmaker && source defaultvars.sh`
- 5. Run the server: `go run ./`
-### Optional (For  Testing):  Create Networks and Nodes
- 
- 1. Create Network: `./test/networkcreate.sh`
- 2. Create Key: `./test/keycreate.sh` (save the response for step 3)
- 3. Open ./test/nodescreate.sh and replace ACCESSKEY with value from #2
- 4. Create Nodes: `./test/nodescreate.sh`
- 5. Check to see if nodes were created: `curl -H "authorization: Bearer secretkey" localhost:8081/api/skynet/nodes | jq`
+ 1. **Run the install script:** sudo curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/v0.2/netmaker-server.sh | sh -
+ 2. Check status:  `sudo journalctl -u netmaker`
+2. If any settings are incorrect such as host or mongo credentials, change them under /etc/netmaker/config/environments/ENV.yaml and then run `sudo systemctl restart netmaker`
+
 ### UI Setup
-Please see [this repo](https://github.com/gravitl/netmaker-ui)  for instructions on setting up your UI.
+1. **Download UI asset files:** `sudo wget -O /usr/share/nginx/html/netmaker-ui.zip https://github.com/gravitl/netmaker-ui/releases/download/latest/netmaker-ui.zip`
+
+2. **Unzip:** `sudo unzip /usr/share/nginx/html/netmaker-ui.zip -d /usr/share/nginx/html`
+
+3. **Copy Config to Nginx:** `sudo cp /usr/share/nginx/html/nginx.conf /etc/nginx/conf.d/default.conf`
+
+4. **Modify Default Config Path:** `sudo sed -i 's/root \/var\/www\/html/root \/usr\/share\/nginx\/html/g' /etc/nginx/sites-available/default`
+
+5. **Change Backend URL:** `sudo sh -c 'BACKEND_URL=http://<YOUR BACKEND API URL>:PORT /usr/share/nginx/html/generate_config_js.sh >/usr/share/nginx/html/config.js'`
+
+6. **Start Nginx:** `sudo systemctl start nginx`
 
 ### Agent  Setup
 
@@ -91,11 +110,11 @@ On each machine you would like to add to the network, do the following:
 1. Confirm wireguard is installed: `sudo apt install wireguard-tools`
 2. Confirm ipv4 forwarding is enabled: `sysctl -w net.ipv4.ip_forward=1`
 3. Create a key or enable manual node signup at the network level
-4. Get the binary: `sudo wget 52.55.6.84:8081/meshclient/files/meshclient`
-5. Make it executable: `sudo chmod +x meshclient`
-6. Run the install command: `sudo ./meshclient -c install -g <network name> -s <server:port> -k <key value>`
+4. Run the install command generated by key create: `sudo curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/v0.2/netclient-install.sh | KEY=YOUR_TOKEN sh -`  
+4.a. For additional networks on the same machine, you can just run `sudo netclient -c install -t YOUR_TOKEN`   
+4.b. For offline installs (no access to github), you can self-host a netclient file server on netmaker  
 
-This will install netclient.service and netclient.timer in systemd, which will run periodically to call the netclient binary, which will check to see if there are any updates that it needs and update WireGuard appropriately.
+This will install netclient@.service and netclient-YOUR_NET.timer in systemd, which will run periodically to call the netclient binary, which will check to see if there are any updates that it needs and update WireGuard appropriately.
 
 ## BUILDING
 **Back End Compilation** 
@@ -119,4 +138,3 @@ When making changes to Netmaker, you may wish to create nodes, networks, or keys
 
 **Integration Testing**
 Similarly, several go  scripts have been created under the test directory (*.go) to test out changes to the code base.  These will be run automatically when PR's are submitted but can also be run manually using "go test."
-

netmaker-install-clientmode.sh

@@ -0,0 +1,53 @@
+#!/bin/sh
+set -e
+
+[ -z "$SERVER_DOMAIN" ] && echo "Need to set SERVER_DOMAIN (format: 1.2.3.4 or mybackend.com)" && exit 1;
+
+
+docker volume create mongovol && docker run -d --name mongodb -v mongovol:/data/db --network host -e MONGO_INITDB_ROOT_USERNAME=mongoadmin -e MONGO_INITDB_ROOT_PASSWORD=mongopass mongo --bind_ip 0.0.0.0 
+
+mkdir -p /etc/netmaker/config/environments
+wget -O /etc/netmaker/netmaker https://github.com/gravitl/netmaker/releases/download/latest/netmaker
+chmod +x /etc/netmaker/netmaker
+
+
+cat >/etc/netmaker/config/environments/dev.yaml<<EOL
+server:
+  host: "$SERVER_DOMAIN"
+  apiport: "8081"
+  grpcport: "50051"
+  masterkey: "secretkey"
+  allowedorigin: "*"
+  restbackend: true            
+  agentbackend: true
+  defaultnetname: "default"
+  defaultnetrange: "10.10.10.0/24"
+  createdefault: true
+mongoconn:
+  user: "mongoadmin"
+  pass: "mongopass"
+  host: "localhost"
+  port: "27017"
+  opts: '/?authSource=admin'
+EOL
+
+cat >/etc/systemd/system/netmaker.service<<EOL
+[Unit]
+Description=Netmaker Server
+After=network.target
+
+[Service]
+Type=simple
+Restart=on-failure
+
+WorkingDirectory=/etc/netmaker
+ExecStart=/etc/netmaker/netmaker
+
+[Install]
+WantedBy=multi-user.target
+EOL
+systemctl daemon-reload
+systemctl start netmaker.service
+
+
+docker run -d --name netmaker-ui -p 80:80 -e BACKEND_URL="http://$SERVER_DOMAIN:8081" gravitl/netmaker-ui:v0.2

netmaker-server.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+set -e
+
+mkdir -p /etc/netmaker/config/environments
+wget -O /etc/netmaker/netmaker https://github.com/gravitl/netmaker/releases/download/latest/netmaker
+chmod +x /etc/netmaker/netmaker
+
+
+cat >/etc/netmaker/config/environments/dev.yaml<<EOL
+server:
+  host:
+  apiport: "8081"
+  grpcport: "50051"
+  masterkey: "secretkey"
+  allowedorigin: "*"
+  restbackend: true            
+  agentbackend: true
+  defaultnetname: "default"
+  defaultnetrange: "10.10.10.0/24"
+  createdefault: true
+mongoconn:
+  user: "mongoadmin"
+  pass: "mongopass"
+  host: "localhost"
+  port: "27017"
+  opts: '/?authSource=admin'
+EOL
+
+cat >/etc/systemd/system/netmaker.service<<EOL
+[Unit]
+Description=Netmaker Server
+After=network.target
+
+[Service]
+Type=simple
+Restart=on-failure
+
+WorkingDirectory=/etc/netmaker
+ExecStart=/etc/netmaker/netmaker
+
+[Install]
+WantedBy=multi-user.target
+EOL
+systemctl daemon-reload
+systemctl start netmaker.service