JDBCBench

In looking to categorize the performance of different databases and JDBC drivers, I found that there were no benchmarks available. I decided to create my own simple benchmark, based on standards from other benchmarks. I call it JDBCBench, and the source is available here on the main page.

JDBCBench uses a simple update-intensive transaction to load the system to be benchmarked. This workload reflects the database aspects of applications such as servlets and intranet applications, but does not reflect the entire range of transaction processing requirements characterized by multiple transaction types of varying complexities encountered in OLTP and data-mining applications. It is derived from the TPC-B benchmark available from the TPC (www.tpc.org).

The single transaction type provides a simple, repeatable unit of work, and is designed to exercise the basic components of a database system.

Preliminary Results

In my first round of testing I created two workloads, one heavy in transactions, and one heavy in the number of simultaneous clients (threads) that were accessing the database.

The results can be seen below in both graphical and tabular formats.

It is interesting to note that only one product (mm.mysql/mysql) completed the benchmark without failing any transactions. Both mysql drivers scaled better than the others during heavy loads, but mm scaled the best, providing anywhere from 6 to 12 times the transaction processing rate under heavy client load, and 2 to 4 times the transaction processing rate under heavy transaction load.

Graphical Depictions

Click to see larger version of same chart

Click to see larger version of same chart

Raw Data for Heavy Transaction Load

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 1 100 35.36 835808 0
twz/mysql 1 100 62.15 901120 0
jdbc-odbc:jet 1 100 13.73 884800 0

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 5 100 87.9 933816 0
twz/mysql 5 100 102.98 987952 14
jdbc-odbc:jet 5 100 7.54 968912 0

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 10 100 75.12 945464 0
twz/mysql 10 100 96.94 994240 73
jdbc-odbc:jet 10 100 10.69 894800 1000

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 50 100 79.4 925896 0
twz/mysql 50 100 52.3 985904 1901
jdbc-odbc:jet 50 100 DNF DNF 5000

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 100 100 79.53 949120 0
twz/mysql 100 100 30.19 1008672 1868
jdbc-odbc:jet 100 100     10000

Raw Data for Heavy Client Load

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 1 4 9.48 167192 0
twz/mysql 1 4 8.26 218608 0
jdbc-odbc:jet 1 4 4.66 192296 0

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 10 4 36.56 699792 0
twz/mysql 10 4 12.17 484624 9
jdbc-odbc:jet 10 4 12.92 461614 0

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 20 4 58.19 961912 0
twz/mysql 20 4 38.76 633304 40
jdbc-odbc:jet 20 4 12.22 753264 0

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 50 4 70.32 919856 0
twz/mysql 50 4 57.72 947000 16
jdbc-odbc:jet 50 4 10.22 847784 0

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 100 4 36.99 901816 0
twz/mysql 100 4 33.16 1005816 85
jdbc-odbc:jet 100 4 7.42 973768 0

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 200 4 58.45 932080 0
twz/mysql 200 4 14.02 1020168 180
jdbc-odbc:jet 200 4 3.74 1002280 391

Driver/DB Clients TPC Transactions/Sec Max Mem Failed Transactions
mm/mysql 300 4 63 932080 0
twz/mysql 300 4 10.12 1025028 261
jdbc-odbc:jet 300 4 2.55 986696 904