Green Luo 6259cac572 Java/act fix mongo update test (#2553)		8 years ago
..
src	6259cac572 Java/act fix mongo update test (#2553)	8 years ago
.gitignore	6db40e537e add ActFramework benchmark set	8 years ago
README.md	341ca13af7 update README.md file with latest locale benchmark result	8 years ago
benchmark_config.json	52d7be095c fix ci issues	8 years ago
pipeline.lua	6db40e537e add ActFramework benchmark set	8 years ago
pom.xml	341ca13af7 update README.md file with latest locale benchmark result	8 years ago
run.sh	6259cac572 Java/act fix mongo update test (#2553)	8 years ago
setup.sh	94efb7d09c ensure update actframework snapshots on build	8 years ago
source_code	6db40e537e add ActFramework benchmark set	8 years ago

ActFramework Benchmarking Test

This is the ActFramework portion of a benchmarking test suite comparing a variety of web development platforms.

This is using ActFramework to build an application.

How to run this app locally

You can easily run this sample application with mongodb and mysql, postgresql setup correctly. For that, type:

mvn clean package
cd target/dist
unzip *.zip
./start

Or simply run run.sh

Application Endpoints

JSON serialization

http://localhost:8080/json

Single database query

Source code

App entry and non-database request handlers

Database relevant source files

This application is organized to support access three database sources: mongo, mysql, pgsql. However most of the logic is the same when handling request sent to different data sources. We have organized the project in a way to maximize the reuse of coding for the same logic. Each piece of source unit are put into four groups:

common: encapsulate the common logic or data models
mongo : the mongodb adaptor
mysql : the mysql adaptor
pgsql : the postgresql adaptor

Domain Models

Common

Mongo

SQL Common

The common base classes for mysql and pgsql classes

MySQL

Postgresql

Controllers

Common

Fortune controller - for fortunes endpoint
World controller - for db, queries and updates endpoint

Mongo

Fortune controller - for /mongo/fortunes endpoint
World controller - for /mongo/db, /mongo/queries and /mongo/updates endpoint

SQL Common

The common base classes for mysql and pgsql classes

MySQL

Fortune controller - for /mysql/fortunes endpoint
World controller - for /mysql/db, /mysql/queries and /mysql/updates endpoint

Postgresql

Fortune controller - for /pgsql/fortunes endpoint
World controller - for /pgsql/db, /pgsql/queries and /pgsql/updates endpoint

Infrastructure Software Versions

The tests were run with:

Local Test Result

Test	Throughput	Latency	Timeout
json	828527.50	4.43ms
single query - mongo	44575.49	15.12ms
single query - mysql	25300.02	55.57ms
single query - pgsql	36930.28	48.51ms
20 queries - mongo	2655.75	107.18ms
20 queries - mysql	1225.00	n/a	768
20 queries - pgsql	2010.60	386.54ms	22
fortunes - mongo	22417.17	48.37ms
fortunes - mysql	22326.85	47.14ms
fortunes - pgsql	24216.44	85.19ms
20 updates - mongo	2250.82	562.06ms
20 updates - mysql	408.33	n/a	256
20 updates - pgsql	707.77	n/a	444
plaintext	1361223.78	2.36ms

Local Test Logs

JSON Serialization

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /json 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.43ms    4.34ms 202.38ms   90.32%
    Req/Sec   209.33k    59.03k  389.44k    62.25%
  Latency Distribution
     50%    3.70ms
     75%    6.09ms
     90%    8.69ms
     99%   15.91ms
  8359840 requests in 10.09s, 1.34GB read
Requests/sec: 828527.50
Transfer/sec:    135.91MB

Plain text

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /plaintext 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.36ms    2.08ms  39.37ms   85.72%
    Req/Sec   344.48k    82.82k  691.36k    66.25%
  Latency Distribution
     50%    1.89ms
     75%    3.16ms
     90%    4.58ms
     99%   10.10ms
  13751216 requests in 10.10s, 1.95GB read
Requests/sec: 1361223.78
Transfer/sec:    197.32MB

Single Query - MongoDB

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mongo/db 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    15.12ms   30.09ms 126.68ms    0.00%
    Req/Sec    11.23k   657.00    13.74k    67.75%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  447408 requests in 10.04s, 75.00MB read
Requests/sec:  44575.49
Transfer/sec:      7.47MB

Single query - MySQL

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mysql/db 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    55.57ms   50.98ms 326.24ms    5.78%
    Req/Sec     6.39k     1.07k   10.34k    78.09%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  253872 requests in 10.03s, 42.56MB read
Requests/sec:  25300.02
Transfer/sec:      4.24MB

Single query - Postgres

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /pgsql/db 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.51ms   32.12ms 335.31ms   28.60%
    Req/Sec     9.33k     0.93k   12.61k    89.67%
  Latency Distribution
     50%  109.70ms
     75%    0.00us
     90%    0.00us
     99%    0.00us
  370144 requests in 10.02s, 62.05MB read
Requests/sec:  36930.28
Transfer/sec:      6.19MB

20 queries - Mongo

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mongo/queries?queries=20 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   107.18ms  361.87ms   1.52s     0.00%
    Req/Sec     2.07k     2.81k    8.96k    82.14%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  26720 requests in 10.06s, 19.94MB read
Requests/sec:   2655.75
Transfer/sec:      1.98MB

20 queries - MySQL

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mysql/queries?queries=20 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     1.76k     1.97k    5.82k    78.69%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  12288 requests in 10.03s, 9.17MB read
  Socket errors: connect 0, read 0, write 0, timeout 768
Requests/sec:   1225.00
Transfer/sec:      0.91MB

20 queries - Postgres

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /pgsql/queries?queries=20 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   386.54ms  624.21ms   2.00s     0.00%
    Req/Sec     1.10k     1.14k    4.36k    82.18%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  20208 requests in 10.05s, 15.08MB read
  Socket errors: connect 0, read 0, write 0, timeout 22
Requests/sec:   2010.60
Transfer/sec:      1.50MB

20 updates - Mongo

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mongo/updates?queries=20 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   562.06ms  567.30ms   1.98s     1.11%
    Req/Sec     1.57k     1.62k    7.43k    80.83%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  22624 requests in 10.05s, 16.89MB read
Requests/sec:   2250.82
Transfer/sec:      1.68MB

20 updates - MySQL

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mysql/updates?queries=20 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     1.06k     1.02k    3.39k    64.52%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  4096 requests in 10.03s, 3.06MB read
  Socket errors: connect 0, read 0, write 0, timeout 256
Requests/sec:    408.33
Transfer/sec:    312.06KB

20 updates - Postgresql

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /pgsql/updates?queries=20 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     1.21k     1.28k    4.80k    84.78%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  7104 requests in 10.04s, 5.30MB read
  Socket errors: connect 0, read 0, write 0, timeout 444
Requests/sec:    707.77
Transfer/sec:    540.90KB

Fortunes - Mongo

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mongo/fortunes 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.37ms   62.79ms 283.94ms    0.00%
    Req/Sec     5.66k     1.15k   10.08k    77.02%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  224908 requests in 10.03s, 313.15MB read
Requests/sec:  22417.17
Transfer/sec:     31.21MB

Fortunes - MySQL

luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /mysql/fortunes 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    47.14ms   62.73ms 291.87ms    0.00%
    Req/Sec     5.64k     1.34k    9.76k    73.43%
  Latency Distribution
     50%    0.00us
     75%    0.00us
     90%    0.00us
     99%    0.00us
  224776 requests in 10.07s, 312.97MB read
Requests/sec:  22326.85
Transfer/sec:     31.09MB

Fortunes - Postgresql

luog@luog-Satellite-P50-A:~/p/greenlaw110/FrameworkBenchmarks/frameworks/Java/act$ wrk -t4 -c256 -d10s http://localhost:8080 -s pipeline.lua --latency -- /pgsql/fortunes 16
Running 10s test @ http://localhost:8080
  4 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    85.19ms   40.72ms 378.41ms   50.28%
    Req/Sec     6.26k     1.75k   13.66k    86.67%
  Latency Distribution
     50%  123.14ms
     75%  169.66ms
     90%    0.00us
     99%    0.00us
  242972 requests in 10.03s, 338.31MB read
Requests/sec:  24216.44
Transfer/sec:     33.72MB

README.md

ActFramework Benchmarking Test

How to run this app locally

Application Endpoints

JSON serialization

Single database query

Multiple queries

Fortunes

DB updates

Plain text

Source code

App entry and non-database request handlers

Database relevant source files

Domain Models

Common

Mongo

SQL Common

MySQL

Postgresql

Controllers

Common

Mongo

SQL Common

MySQL

Postgresql

Infrastructure Software Versions

Local Test Result

Local Test Logs

JSON Serialization

Plain text

Single Query - MongoDB

Single query - MySQL

Single query - Postgres

20 queries - Mongo

20 queries - MySQL

20 queries - Postgres

20 updates - Mongo

20 updates - MySQL

20 updates - Postgresql

Fortunes - Mongo

Fortunes - MySQL

Fortunes - Postgresql