1 Introduction

Use YCSB to test the performance of rocksdb (the code you modified yourself), the operating environment:

• Operating system: ubuntu 14.04
• Rockdb version: 5.18.3
• YCSB version: 0.15.0

2 rocksb jni package generation

2.1 rocksdb version code

Get the code for rocksdb version 5.18.3 via github as the basis:

> git clone https://github.com/facebook/rocksdb.git
> cd rocksdb 
 > git tag # View the version number of the release, the label of the corresponding branch
 > git checkout v5.18.3 #Switch branches, view current branch git branch View all branches git branch -a
 > git checkout -b doycsb v5.18.3 # Create replication branch v5.18.3 for the new branch doycsb, easy to modify the code submission of v5.18.3, of course, the best way is to fork first

2.2 rocksdb java compilation

1. The java folder in the rocksdb folder is the jni package. Since I am running on a linux64-bit system, there is no need to complete cross-platform compilation. There are no need for 5 compression methods (too many download processes may go wrong), only Snappy compression, modify the rocksdb/Makefile:

ifneq ($(ROCKSDB_JAVA_NO_COMPRESSION), 1)
# JAVA_COMPRESSIONS = libz.a libbz2.a libsnappy.a liblz4.a libzstd.a
JAVA_COMPRESSIONS = libsnappy.a 
endif

#JAVA_STATIC_FLAGS = -DZLIB -DBZIP2 -DSNAPPY -DLZ4 -DZSTD
JAVA_STATIC_FLAGS = -DSNAPPY 
#JAVA_STATIC_INCLUDES = -I./zlib-$(ZLIB_VER) -I./bzip2-$(BZIP2_VER) -I./snappy-$(SNAPPY_VER) -I./lz4-$(LZ4_VER)/lib -I./zstd-$(ZSTD_VER)/lib/include
JAVA_STATIC_INCLUDES =  -I./snappy-$(SNAPPY_VER)

2. Use make rocksdbjavastaticrelease to compile the rules, find the rule, cancel the vagrantd cross-platform, do not need to install Vagrant and Virtualbox, that is, under the rocksdbjavastaticrelease rule, comment out:

#cd java/crossbuild && vagrant destroy -f && vagrant up linux32 && vagrant halt linux32 && vagrant up linux64 && vagrant halt linux64

3. In the rocksdb folder, executemake rocksdbjavastaticrelease -j16, the following error will occur:

cd java/target;jar -uf rocksdbjni-5.18.3.jar librocksdbjni-*.so librocksdbjni-*.jnilib
librocksdbjni-*.jnilib : no such file or directory
make: *** [rocksdbjavastaticrelease] Error 1

This is because we canceled the cross-platform java package, and .jnilib is under the mac os platform, so just remove librocksdbjni-*.jnilib:

cd java/target;jar -uf $(ROCKSDB_JAR_ALL) librocksdbjni-*.so

Continue to compilemake rocksdbjavastaticrelease -j16。

4. Jni package will be generated inrocksdb/java/target/rocksdbjni-5.18.3.jarWill be used later.

3 YCSB compilation and operation

3.1 YCSB version code acquisition

> git clone https://github.com/brianfrankcooper/YCSB.git
> git checkout 0.15.0

3.2 YCSB compilation

> mvn -pl com.yahoo.ycsb:rocksdb-binding -am clean package

This is a dependency package for downloading the rocksdb-binding module via mvn. Don't directlymvn clean packageThis will download the dependencies of all database modules and takes a very long time.

Can be seen in the YCSB/pom.xml file<rocksdb.version>5.11.3</rocksdb.version>This will be through mvnhttps://repo.maven.apache.org/maven2/org/rocksdb/rocksdbjni/Download the corresponding 5.11.3 version of the jni package. If you don't want to use your own rocksdbjni package, you can directly modify the corresponding version to get a new version of the jni package.

3.2 Operation of YCSB

1. There are three implementations of YCSB:

• Bin/ycsb # python script, corresponding to cross-platform
• Bin/ycsb.sh # linux script, suitable for linux system
• Bin/ycsb.bat # windows script, suitable for windows system

I chose the bin/ycsb.sh script to run and execute:

> bin/ycsb.sh load rocksdb -s -P workloads/workloada -p rocksdb.dir=/home/lzw/ceshi

Where rocksdb.dir is the database path, run load to load the data, the configuration is the workloads/workloada file.
Execute the output of the previous section of the load command:

/usr/lib/jvm/jdk1.8.0_181/bin/java  -classpath /home/lzw/ce_ycsb/YCSB/conf:/home/lzw/ce_ycsb/YCSB/core/target/core-0.15.0.jar:/home/lzw/ce_ycsb/YCSB/rocksdb/target/rocksdb-binding-0.15.0.jar:/home/lzw/ce_ycsb/YCSB/rocksdb/target/dependency/jcip-annotations-1.0.jar:/home/lzw/ce_ycsb/YCSB/rocksdb/target/dependency/rocksdbjni-5.11.3.jar:/home/lzw/ce_ycsb/YCSB/rocksdb/target/dependency/slf4j-api-1.7.25.jar:/home/lzw/ce_ycsb/YCSB/rocksdb/target/dependency/slf4j-simple-1.7.25.jar com.yahoo.ycsb.Client -load -db com.yahoo.ycsb.db.rocksdb.RocksDBClient -s -P workloads/workloada -p rocksdb.dir=/home/lzw/ceshi

This run command is very important, this is the classpath, the jni package, you can also view the contents of the bin/ycsb.sh script, these jni packages determine the results of the run.

The above load run results will have the following problems:

Loading workload...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
	at com.yahoo.ycsb.Client.getTracer(Client.java:903)
	at com.yahoo.ycsb.Client.main(Client.java:752)
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 2 more

This is due to the lack of htrace-core4 in the core module of YCSB, which can be viewed.YCSB/core/pom.xmlFile, found that the core module's dependencies are not downloaded, willYCSB/core/pom.xmlContent:

<dependency>
      <groupId>org.apache.htrace</groupId>
      <artifactId>htrace-core4</artifactId>
      <version>4.1.0-incubating</version>
</dependency>

add toYCSB/rocksdb/pom.xmlDocumentary<dependencies>Under the tab, then run:

> mvn -pl com.yahoo.ycsb:rocksdb-binding -am clean package

Can be found inYCSB/rocksdb/target/dependency/Added htrace-core4-4.1.0-incubating.jar to the directory.

Continue to run the load command, the following problems will occur:

Exception in thread "Thread-3" java.lang.NoClassDefFoundError: org/HdrHistogram/EncodableHistogram
	at com.yahoo.ycsb.measurements.Measurements.constructOneMeasurement(Measurements.java:129)
	at com.yahoo.ycsb.measurements.Measurements.getOpMeasurement(Measurements.java:220)
	at com.yahoo.ycsb.measurements.Measurements.measure(Measurements.java:188)
	at com.yahoo.ycsb.DBWrapper.measure(DBWrapper.java:178)
	at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:223)
	at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:588)
	at com.yahoo.ycsb.ClientThread.run(Client.java:468)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.HdrHistogram.EncodableHistogram
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 8 more

This problem is the same as the above problem. The lack of the core module depends on the HdrHistogram.YCSB/core/pom.xmlContent:

<dependency>
      <groupId>org.hdrhistogram</groupId>
      <artifactId>HdrHistogram</artifactId>
      <version>2.1.4</version>
</dependency>

add toYCSB/rocksdb/pom.xmlDocumentary<dependencies>Under the tab, then run:

> mvn -pl com.yahoo.ycsb:rocksdb-binding -am clean package

Can be found inYCSB/rocksdb/target/dependency/HdrHistogram-2.1.4.jar has been added to the directory.

Run the load command:

> bin/ycsb.sh load rocksdb -s -P workloads/workloada -p rocksdb.dir=/home/lzw/ceshi

Finally, it can run normally, you can see at the beginning of the LOG file in the database path.RocksDB version: 5.11.3On behalf of the running is rocksdb v5.11.3.

2. Generated in the previous steprocksdb/java/target/rocksdbjni-5.18.3.jarcopy toYCSB/rocksdb/target/dependency/Directory, then putrocksdbjni-5.11.3.jardelete.

Then delete the file under the database path, that is, delete the original database.

Run the load command:

> bin/ycsb.sh load rocksdb -s -P workloads/workloada -p rocksdb.dir=/home/lzw/ceshi

Can be seen at the beginning of the LOG file in the database pathRocksDB version: 5.18.3On behalf of the running is rocksdb v5.18.3.

3. Run the run command:

> bin/ycsb.sh run rocksdb -s -P workloads/workloada -p rocksdb.dir=/home/lzw/ceshi

4 rocksdb parameter settings under YCSB

1. The simplest configuration isworkloads/workloadaSimilar to the configuration of the file, the command to run load is simplyrecordcount=1000The number of entries is inserted into the database, and the data size of the inserted database is:

Fieldcount=10 (default)
 Fieldlength=100 (default)

Fieldcount indicates how many segments are in a record. Fieldlength represents the number of bytes in a segment. The value of a data inserted is approximately equal to fieldcount*fieldlength.

2. The file connecting the rocksdb client in YCSB:

YCSB\rocksdb\src\main\java\com\yahoo\ycsb\db\rocksdb\RocksDBClient.java

Test the rocksdb file in YCSB:

YCSB\rocksdb\src\test\java\com\yahoo\ycsb\db\rocksdb\RocksDBClientTest.java

inRocksDBClient.javaIn the file:

Located in the initRocksDB() function:
final Options options = new Options()
    .optimizeLevelStyleCompaction()
    .setCreateIfMissing(true)
    .setCreateMissingColumnFamilies(true)
    .setIncreaseParallelism(rocksThreads)
    .setMaxBackgroundCompactions(rocksThreads)
    .setInfoLogLevel(InfoLogLevel.INFO_LEVEL);

final DBOptions options = new DBOptions()
    .setCreateIfMissing(true)
    .setCreateMissingColumnFamilies(true)
    .setIncreaseParallelism(rocksThreads)
    .setMaxBackgroundCompactions(rocksThreads)
    .setInfoLogLevel(InfoLogLevel.INFO_LEVEL);

 Located in the createColumnFamily() function:
final ColumnFamilyOptions cfOptions = new ColumnFamilyOptions().optimizeLevelStyleCompaction();

Both of these will have an impact on the options of rocksdb, especially the optimizeLevelStyleCompaction() function in rocksdb, which optimizes multiple parameters. The result of the parameters in rocksdb will be saved under the OPTIONS file in the database path.

If modifiedRocksDBClient.javaThe contents of the file, you can compile YCSB again:

> mvn -pl com.yahoo.ycsb:rocksdb-binding -am package

Remove the clean, it will not be the originalrocksdb/java/target/rocksdbjni-5.18.3.jardelete. Will only re-download the package, will be downloadedrocksdb/java/target/rocksdbjni-5.11.3.jarDelete it.