GL, more pipelines and update version (#4531)

* Adding GreenLightning

* fix missing quote

create object for json production

* turn off telemetry for default test

* allow any external domain or ip

* Update to next version and template construction

* Revert "Update to next version and template construction"

This reverts commit da4fbc3b085f6e3b3bb50283d4d6e79efffdd3bb.

* Revert "Revert "Update to next version and template construction""

This reverts commit eb05517a192554caec7c5f690eeaaab171a549b6.

* Revert "Update to next version and template construction"

This reverts commit da4fbc3b085f6e3b3bb50283d4d6e79efffdd3bb.

* fix host ip to 0.0.0.0 to take load

added database read tests

* response large enough for multi db

remove epoll, it was not helping

added required headers

* update list of which tests we have implmented

* removed unused dependency

* remove unrequired 16G memory grab to get past travis check.

* Update to next GL version to fix overload issue past 1.5M rps

Refine template method to remove dead argument

* simplify arg parse

fixed issue with multi db under heavy load

* updat to atomic int

* Removed support for MultiTest, not stable at this time.

* added clean for safety

* remove dead code

* remove old comments

* [ci fw-only Java/greenlightning]

added comment

* [ci fw-only Java/greenlightning]

Added tests for remaining multi, update and fortunes

* [ci fw-only Java/greenlightning]

disable update test, seems to be missing some writes.

* [ci fw-only Java/greenlightning]

re-test of DBUpdate

* [ci fw-only Java/greenlightning]

disabled multi and update while tracking issue

* fixed muti paylod response JSON dups.

* [ci fw-only Java/greenlightning]

narrow building

* [ci skip] update readme

* [ci fw-only Java/greenlightning]

Update to version supporting 0.0.0.0 wildcard host

* [ci fw-only Java/greenlightning]

explicit memory min/max setting instead of default

update to new maven for build

* lower required low end to 8G

* [ci fw-only Java/greenlightning]

dropped min to 6G

* [ci fw-only Java/greenlightning]

memory reduction, was using as much as 20G, paging may have slowed test

reduce pipe memory allocations

combined behaviors to reduce memory

reduce db inflight collection to reduce memory

* Revert "[ci fw-only Java/greenlightning]"

This reverts commit 47c351db1ddeb64e65801f15e3c4398d5eca84ad.

* [ci fw-only Java/greenlightning]

Was using 16-20G which may cause paging and slow results

Reduced Pipe lenghts

Reduced concurrent connections

Combined behaviors to reduce Pipe counts for more memory.

Lowered limit to 16G to know it is enforced.

* removed minimum required memory in docker file

* [ci fw-only Java/greenlightning]  Ready for merge

* [ci fw-only Java/greenlightning]

Lowering memory usage to 7G from 14G

Clean up design to be easier to read

* [ci fw-only Java/greenlightning]

Investigating pef issue, returned threads to normal priority

Reduce the new and ongoing network kernel calls

Tread Executor using hardcoded count instead of asking docker

* [ci fw-only Java/greenlightning]

remove dead code

* [ci fw-only Java/greenlightning]

Test CentOS runtime in container

Add minimal logging to track runtime issues

* Revert "[ci fw-only Java/greenlightning]"

This reverts commit 7119eaa0df0f10ac5d5d6aca4bd15e04bc89de32.

* [ci fw-only Java/greenlightning]

added centOS in container as test

added minimal logging to debug deployment

* [ci fw-only Java/greenlightning]

1G less, narrow test scope

* [ci fw-only Java/greenlightning]

streamline trie parser for headers etc

load balance based on primes to ensure even distribution

clean up ServerSocketWriter for larger blocks

Doubled ServerSocketWriters for greater volume.

* [ci fw-only Java/greenlightning]

Upgrade to Java 11 build and run pipeline

Update socket reader to optimize for greater volume

Upate pipe lengths to support greater volume

* [ci fw-only Java/greenlightning]

Double allowed limit on database connections

Bumped up to new version of GreenLightning

Doubled clock rate for reading new requests off socket

* missing version number restored

* [ci fw-only Java/greenlightning]

bump up in flight db count

* [ci fw-only Java/greenlightning]

Update to next version for larger read socket blocks

* [ci fw-only Java/greenlightning]

Bump up reactive pg client version to 11.1

Simplify build process

* [ci fw-only Java/greenlightning]

Bump up to next version

* [ci fw-only Java/greenlightning]

comments

* [ci fw-only Java/greenlightning]

update to new version of GL

update memory usage for volume

* [ci fw-only Java/greenlightning]

reduce DB calls in flight

* [ci fw-only Java/greenlightning]

bump up memory limit by 30%

* [ci fw-only Java/greenlightning]

outgoing buffers do not need to be that large reduced by 75%

* [ci fw-only Java/greenlightning]

Minimize open connections to save memory

* [ci fw-only Java/greenlightning]

next version of GL

bump up connections

* [ci fw-only Java/greenlightning]

update version

reduce memory per connection

reduce deep sleep

* Revert "[ci fw-only Java/greenlightning]"

This reverts commit f6f398f84f787b83c71d69d1087498d30b7e1bec.

* [ci fw-only Java/greenlightning]

next version

* [ci fw-only Java/greenlightning]

change memory usage to favor more connections

* update docker file for more memory

* update version

* [ci fw-only Java/greenlightning]

update memory limit

* Revert "[ci fw-only Java/greenlightning]"

This reverts commit 3a840590bd391a488244610dcbe9d61b1972577b.

* [ci fw-only Java/greenlightning]

limit selectors and increase tracks

* [ci fw-only Java/greenlightning]

update to new version

* [ci fw-only Java/greenlightning]

update version

* [ci fw-only Java/greenlightning]

update port

* [ci fw-only Java/greenlightning]

removed epoll

corrected pipe count

update version

* [ci fw-only Java/greenlightning]

update to next version fixing lock issue

testing direct memory pipe

* [ci fw-only Java/greenlightning]

record the network config data to the log

add more parallel pipelines for processing

fixed bug in db processing

udpate version

frameworks/Java/greenlightning/greenlightning.dockerfile

@@ -12,4 +12,6 @@ RUN mvn clean install -q
 FROM azul/zulu-openjdk-alpine:11.0.1
 WORKDIR /greenlightning
 COPY --from=maven /greenlightning/target/greenlightning-test.jar app.jar
-CMD ["java", "-server", "-Xmx26g", "-XX:+UseNUMA", "-jar", "app.jar"]
+
+#records to our log all the known network settings on the host connection 
+CMD sysctl -a && java -server -Xmx26g -XX:+UseNUMA -jar app.jar

frameworks/Java/greenlightning/pom.xml

@@ -4,7 +4,7 @@
 
 	<groupId>com.ociweb.gl.benchmark</groupId>
 	<artifactId>benchmark-test</artifactId>
-	<version>1.0.32</version>
+	<version>1.0.33</version> 
 
 	<properties>
 		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
@@ -14,7 +14,7 @@
 		<dependency>
 			<groupId>com.ociweb</groupId>
 			<artifactId>greenlightning</artifactId>
-			<version>1.0.32</version> <!-- must be a fixed version for techempower releases -->
+			<version>${project.version}</version> <!-- must be a fixed version for techempower releases -->
 		</dependency>
 		<dependency>
 			<groupId>org.slf4j</groupId>
@@ -37,7 +37,7 @@
 		<dependency>
 		    <groupId>io.reactiverse</groupId>
 		    <artifactId>reactive-pg-client</artifactId>
-		    <version>0.11.1</version>
+		    <version>0.11.2</version>
 		</dependency>
 
 	</dependencies>

frameworks/Java/greenlightning/src/main/java/com/ociweb/gl/benchmark/DBRest.java

@@ -20,8 +20,12 @@ public class DBRest implements RestMethodListener, PubSubMethodListener, TickLis
 		
 		pm = new PoolManager(options);
 		
-		HTTPResponseService service = runtime.newCommandChannel().newHTTPResponseService(maxResponseCount, maxResponseSize);
-
+		maxResponseCount = Math.max(maxResponseCount, ((1<<pipelineBits)/20));//match response count to expected db calls
+		
+		HTTPResponseService service = runtime.newCommandChannel().newHTTPResponseService(
+				                maxResponseCount, 
+				                maxResponseSize);
+		
 		processUpdate = new ProcessUpdate(pipelineBits, service, pm);
 		processFortune = new ProcessFortune(pipelineBits, service, pm);
 		processQuery = new ProcessQuery(pipelineBits, service, pm);

frameworks/Java/greenlightning/src/main/java/com/ociweb/gl/benchmark/FrameworkTest.java

@@ -12,11 +12,10 @@ import com.ociweb.gl.api.GreenCommandChannel;
  */
 import com.ociweb.gl.api.GreenFramework;
 import com.ociweb.gl.api.GreenRuntime;
-import com.ociweb.pronghorn.network.ServerSocketWriterStage;
 
 import io.reactiverse.pgclient.PgClient;
+import io.reactiverse.pgclient.PgPool;
 import io.reactiverse.pgclient.PgPoolOptions;
-import io.vertx.core.VertxOptions;
 
 public class FrameworkTest implements GreenApp {
 
@@ -53,7 +52,8 @@ public class FrameworkTest implements GreenApp {
 	public static String connectionUser =     "postgres";
 	public static String connectionPassword = "postgres";
 	
-	static final int c = 1120;// to reach 16K simultainious calls
+	//TODO: add utility to compute this based on need.
+	static final int c = 592; // to reach 16K simultainious calls
 
     public FrameworkTest() {
     	    	
@@ -64,8 +64,8 @@ public class FrameworkTest implements GreenApp {
     	//this server works best with  -XX:+UseNUMA    	
     	this(System.getProperty("host","0.0.0.0"), 
     		 8080,    	//default port for test 
-    		 c,//250 goal,       //needed to reach 16K simultainious calls
-    		 c*4,// c*2,     //1<<14 (router to module)
+    		 c,         //pipes per track
+    		 c*4,       //(router to module)
     		 1<<11,     //default total size of network buffer used by blocks 
     		 Integer.parseInt(System.getProperty("telemetry.port", "-1")),
     		 "tfb-database", // jdbc:postgresql://tfb-database:5432/hello_world
@@ -85,6 +85,7 @@ public class FrameworkTest implements GreenApp {
     		             String dbUser,
     		             String dbPass) {
     	
+    	
     	this.connectionsPerTrack = 2;
     	this.connectionPort = 5432;
     	this.bindPort = port;
@@ -93,7 +94,7 @@ public class FrameworkTest implements GreenApp {
     	this.queueLengthOfPendingRequests = queueLengthOfPendingRequests;
     	this.minMemoryOfInputPipes = minMemoryOfInputPipes;
     	this.telemetryPort = telemetryPort;
-    	this.pipelineBits = 14;//max concurrent in flight database requests 1<<pipelineBits
+    	this.pipelineBits = 15;//max concurrent in flight database requests 1<<pipelineBits
     	            
     	this.dbCallMaxResponseCount = c*2;//1<<6;
     	this.jsonMaxResponseCount = c*2;//1<<14;
@@ -101,7 +102,7 @@ public class FrameworkTest implements GreenApp {
     	this.dbCallMaxResponseSize = 20_000; //for 500 mult db call in JSON format
     	this.jsonMaxResponseSize = 1<<8;
 
-    	this.maxQueueOut = 4;    	
+    	this.maxQueueOut = 4;   	
     	this.maxConnectionBits = 14; //16K connections, for test plus overhead
     	
     	this.maxRequestSize = 1<<9;
@@ -138,12 +139,30 @@ public class FrameworkTest implements GreenApp {
 
     		///early check to know if we have a database or not,
 	    	///this helps testing to know which tests should be run on different boxes.
-	    	PgClient.pool(options).getConnection(a->{
+	    	PgPool pool = PgClient.pool(options);
+			pool.getConnection(a->{
 	    		foundDB.set(a.succeeded());
 	    		if (null!=a.result()) {
 	    			a.result().close();
 	    		}
 	    	});
+			pool.close();
+	    	
+//	    	pool.preparedQuery("SELECT * FROM world WHERE id=$1", Tuple.of(1), r -> {
+//				if (r.succeeded()) {
+//
+//					PgIterator resultSet = r.result().iterator();
+//					Tuple row = resultSet.next();			        
+//					System.out.println("successfull query");
+//
+//				} else {
+//					System.out.println("fail: "+r.cause().getLocalizedMessage());
+//					 
+//				}		
+//				
+//			});	
+	    	
+	    	
     	} catch (Throwable t) {
     		//t.printStackTrace();
     		System.out.println("No database in use");
@@ -155,14 +174,17 @@ public class FrameworkTest implements GreenApp {
 	@Override
     public void declareConfiguration(GreenFramework framework) {
 		
-		framework.setDefaultRate(50_000L);			
+		framework.setDefaultRate(60_000L);			
 	
 		//for 14 cores this is expected to use less than 16G, must use next largest prime to ensure smaller groups are not multiples.
 		framework.useHTTP1xServer(bindPort, this::parallelBehavior) //standard auto-scale
     			 .setHost(host)
     			 .setMaxConnectionBits(maxConnectionBits)
-    			 .setConcurrentChannelsPerDecryptUnit(concurrentWritesPerChannel)
-    			 .setConcurrentChannelsPerEncryptUnit(concurrentWritesPerChannel)
+    			 .setConcurrentChannelsPerDecryptUnit(concurrentWritesPerChannel)                //16K   14 bits
+    	
+    			 //TODO: not sure this is optimal..
+    			 .setConcurrentChannelsPerEncryptUnit(Math.max(1,concurrentWritesPerChannel/4))  //4K    
+    		//	 .setConcurrentChannelsPerEncryptUnit(concurrentWritesPerChannel)
     			 .disableEPoll()
  						 
     			 .setMaxQueueIn(queueLengthOfPendingRequests)

frameworks/Java/greenlightning/src/main/java/com/ociweb/gl/benchmark/ProcessFortune.java

@@ -30,8 +30,8 @@ public class ProcessFortune {
 		
 		//for fortune
 		{
-			FortunesObject temp = fortuneInFlight.tailObject();
-			while (isReadyFortune(temp)) {			
+			FortunesObject temp;
+			while (isReadyFortune(temp = fortuneInFlight.tailObject())) {			
 				if (consumeResultObjectFortune(temp)) {
 					temp = fortuneInFlight.tailObject();
 				} else {

frameworks/Java/greenlightning/src/main/java/com/ociweb/gl/benchmark/ProcessQuery.java

@@ -36,8 +36,8 @@ public class ProcessQuery {
 	public void tickEvent() { 
 		//for DBRest
 		{
-			ResultObject temp = DBRestInFlight.tailObject();
-			while (isReadyDBRest(temp)) {
+			ResultObject temp;
+			while (isReadyDBRest(temp = DBRestInFlight.tailObject())) {
 				if (consumeResultObjectDBRest(temp)) {
 					temp = DBRestInFlight.tailObject();
 				} else {

frameworks/Java/greenlightning/src/main/java/com/ociweb/gl/benchmark/ProcessUpdate.java

@@ -3,6 +3,7 @@ package com.ociweb.gl.benchmark;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicInteger;
 
 import com.ociweb.gl.api.HTTPRequestReader;
 import com.ociweb.gl.api.HTTPResponseService;
@@ -15,11 +16,11 @@ import io.reactiverse.pgclient.Tuple;
 public class ProcessUpdate {
 	
 	private transient ObjectPipe<ResultObject> DBUpdateInFlight;	
-	private boolean collectionPendingDBUpdate = false;	
 	private final transient List<ResultObject> collectorDBUpdate = new ArrayList<ResultObject>();
 	private final transient ThreadLocalRandom localRandom = ThreadLocalRandom.current();
 	private final HTTPResponseService service;
 	private final transient PoolManager pm;
+	private final AtomicInteger requestsInFlight = new AtomicInteger();
 	
 	public ProcessUpdate(int pipelineBits, HTTPResponseService service, PoolManager pm) {
 		this.DBUpdateInFlight = new ObjectPipe<ResultObject>(pipelineBits, ResultObject.class,	ResultObject::new);
@@ -30,17 +31,12 @@ public class ProcessUpdate {
 	
 	public void tickEvent() { 
 
-		{
 			ResultObject temp = DBUpdateInFlight.tailObject();
-			while (isReadyDBUpdate(temp)) {			
-				if (consumeResultObjectDBUpdate(temp)) {
-					temp = DBUpdateInFlight.tailObject();
-				} else {
-					break;
-				}
+			while (null!=temp && temp.getStatus()>=0) {			
+				consumeResultObjectDBUpdate(temp);
+				temp = DBUpdateInFlight.tailObject();				
 			}	   
-		}
-			
+		
 	}
 	
 
@@ -57,9 +53,10 @@ public class ProcessUpdate {
 		}
 		long conId = request.getConnectionId();
 		long seqCode = request.getSequenceCode();
-
-		if (DBUpdateInFlight.hasRoomFor(queries)) {		
-				    	
+		int temp = requestsInFlight.incrementAndGet();
+		
+		if (DBUpdateInFlight.hasRoomFor(queries) || service.hasRoomFor(temp)) {		
+			    
 				int q = queries;
 				while (--q >= 0) {
 				
@@ -83,17 +80,16 @@ public class ProcessUpdate {
 							        
 							        //read the existing random value and store it in the world object
 							        worldObject.setResult(row.getInteger(1));
-							        
+							        ///////////////////////////////////
+							        //the object can be used here with the old value
 							        ///////////////////////////////////
 							        //set the new random value in this object
-							        worldObject.setResult(randomValue());			       
+							        worldObject.setResult(randomValue());							        
 							        
 							        pm.pool().preparedQuery("UPDATE world SET randomnumber=$1 WHERE id=$2", 							        		
 							        			Tuple.of(worldObject.getResult(), worldObject.getId()), ar -> {							        	
 										if (ar.succeeded()) {
-											
-								        	worldObject.setStatus(200);							
-								        	
+								        	worldObject.setStatus(200);	
 										} else {	
 											System.out.println("unable to update");
 											if (ar.cause()!=null) {
@@ -101,9 +97,9 @@ public class ProcessUpdate {
 											}
 											
 											worldObject.setStatus(500);
-										}	
-																													
+										}																			
 							        });
+							        
 								} else {	
 									System.out.println("unable to query");
 									if (r.cause()!=null) {
@@ -113,6 +109,12 @@ public class ProcessUpdate {
 									worldObject.setStatus(500);
 								}		
 								
+								//on all N responses.....
+								
+								
+								
+								
+								
 								
 							});	
 									
@@ -122,38 +124,23 @@ public class ProcessUpdate {
 				
 			return true;
 		} else {
+			requestsInFlight.decrementAndGet();
 			return false;
 		}
 	}
 
-	private boolean isReadyDBUpdate(ResultObject temp) {
-
-		if (collectionPendingDBUpdate) {
-			//now ready to send, we have all the data	
-			if (!publishMultiResponseDBUpdate(collectorDBUpdate.get(0).getConnectionId(), collectorDBUpdate.get(0).getSequenceId() )) {
-				return false;
-			}
-		}
-		
-		return null!=temp && temp.getStatus()>=0;
-	}
+	private void consumeResultObjectDBUpdate(final ResultObject t) {
 
-	private boolean consumeResultObjectDBUpdate(final ResultObject t) {
-		boolean ok;
 		//collect all the objects
 		collectorDBUpdate.add(t);
 		DBUpdateInFlight.moveTailForward();//only move forward when it is consumed.
 		if (collectorDBUpdate.size() == t.getGroupSize()) {
 			//now ready to send, we have all the data						
-			ok =publishMultiResponseDBUpdate(t.getConnectionId(), t.getSequenceId());
-		} else {
-			ok = true;//added to list
-		}				
-		
-		return ok;
+			publishMultiResponseDBUpdate(t.getConnectionId(), t.getSequenceId());
+		}
 	}
 
-	private boolean publishMultiResponseDBUpdate(long conId, long seqCode) {
+	private void publishMultiResponseDBUpdate(long conId, long seqCode) {
 		boolean result =  service.publishHTTPResponse(conId, seqCode, 200,
 					    				   HTTPContentTypeDefaults.JSON,
 					    				   w-> {
@@ -167,8 +154,8 @@ public class ProcessUpdate {
 					    					   collectorDBUpdate.clear();
 					    					   DBUpdateInFlight.publishTailPosition();
 					    				   });
-		collectionPendingDBUpdate = !result;
-		return result;
+		assert(result) : "internal error, we should not pick up more work than we can send";
+		requestsInFlight.decrementAndGet();
 	}