Fork Me On GitLab

The providence project contains integration tests. The integration tests can be run separately, and is not a dependency for running the main test suites.

There are currently two IT test suites.

  • [it-serialization]: Thorough testing of serialization focusing on serialization speed. The thrift compatibility is for the most part already tested in providence-thrift.
  • [it-hazelcast]: Proper testing of generated hazelcast generated code by using actual generated code against a local hazelcast instance including how thrift IDL updates are handled both forward and backward.

Serialization Speed

To run the serialization integration test suite, after installing the library, run:

mvn package -Pit-serialization
java -jar it-serialization/target/it-serialization.jar --help
java -jar it-serialization/target/it-serialization.jar

It will take a while, and print quite a lot to std out.

This is a comparative serialization speed test. It is used to see the progress of serialization optimization, and to compare with the native thrift libraries.

  • The pvd columns are the native providence serialization or in the case of the *_protocol the thrift protocol wrapper performance.
  • The thr columns is for the native thrift protocol read / write. The “content” is exactly same as the providence version. But compatibility tests are not done here, but in the providence-testing module.

Latest output from the serialization speed IT, sorted by the SUM of the providence serialization time. Lower is better. There are a number tests, each with a structure focusing on some aspect or style of structures to test. Exceptions are not tested as part of this, as the way the stack traces are handled quickly deteriorates the results for reading providence, though not affecting the others much.

java -jar it-serialization/target/it-serialization.jar --runs 50 --generate 1000

Last updated using 2.0.0-SNAPSHOT release and compared against apache thrift 0.12.0.

Many containers:

                       READ           WRITE          SUM        SIZE
    name        :   pvd   thr  --  pvd   thr  =   pvd   thr  -- (ratio / size)
          binary:   1,84  1,00 --  1,06  1,00 =   1,45  1,00 -- ( 1,00 / 8,7 MiB)
     fast_binary:   2,11       --  2,03       =   2,07       -- ( 0,85 / 7,4 MiB)
  tuple_protocol:   3,17  1,10 --  2,53  0,67 =   2,85  0,89 -- ( 0,81 / 7,0 MiB)
 binary_protocol:   3,28       --  3,16       =   3,22       -- ( 1,00 / 8,7 MiB)
compact_protocol:   3,77  1,41 --  3,03  0,91 =   3,40  1,16 -- ( 0,85 / 7,3 MiB)
            json:   9,01       --  6,64       =   7,83       -- ( 1,18 / 10,2 MiB)
      json_named:  10,19       --  7,18       =   8,69       -- ( 1,43 / 12,4 MiB)
     json_pretty:  14,24       --  9,84       =  12,04       -- ( 2,31 / 20,0 MiB)
          pretty:  18,94       --  8,79       =  13,86       -- ( 1,46 / 12,6 MiB)
          config:  22,23       -- 10,04       =  16,13       -- ( 1,94 / 16,8 MiB)

Many Optional Fields:

                       READ           WRITE          SUM        SIZE
    name        :   pvd   thr  --  pvd   thr  =   pvd   thr  -- (ratio / size)
          binary:   1,14  1,00 --  0,76  1,00 =   0,95  1,00 -- ( 1,00 / 2,4 MiB)
     fast_binary:   1,49       --  1,53       =   1,51       -- ( 0,89 / 2,2 MiB)
  tuple_protocol:   2,14  1,08 --  1,89  0,76 =   2,01  0,92 -- ( 0,83 / 2,0 MiB)
 binary_protocol:   2,21       --  2,41       =   2,31       -- ( 1,00 / 2,4 MiB)
compact_protocol:   2,60  1,42 --  2,30  0,91 =   2,45  1,16 -- ( 0,87 / 2,1 MiB)
            json:   6,60       --  5,29       =   5,94       -- ( 1,27 / 3,1 MiB)
      json_named:   7,62       --  5,76       =   6,69       -- ( 1,53 / 3,7 MiB)
     json_pretty:   9,16       --  6,95       =   8,06       -- ( 2,02 / 4,9 MiB)
          pretty:  13,12       --  6,73       =   9,93       -- ( 1,49 / 3,6 MiB)
          config:  14,48       --  7,14       =  10,81       -- ( 1,78 / 4,3 MiB)

Many Required Fields:

                       READ           WRITE          SUM        SIZE
    name        :   pvd   thr  --  pvd   thr  =   pvd   thr  -- (ratio / size)
          binary:   1,05  1,00 --  0,80  1,00 =   0,93  1,00 -- ( 1,00 / 2,4 MiB)
     fast_binary:   1,42       --  1,60       =   1,51       -- ( 0,89 / 2,2 MiB)
  tuple_protocol:   1,88  0,99 --  1,85  0,69 =   1,86  0,84 -- ( 0,83 / 2,0 MiB)
 binary_protocol:   2,11       --  2,47       =   2,29       -- ( 1,00 / 2,4 MiB)
compact_protocol:   2,46  1,40 --  2,36  0,90 =   2,41  1,15 -- ( 0,87 / 2,1 MiB)
            json:   6,33       --  5,45       =   5,89       -- ( 1,27 / 3,1 MiB)
      json_named:   7,34       --  5,98       =   6,66       -- ( 1,54 / 3,7 MiB)
     json_pretty:   8,80       --  7,19       =   7,99       -- ( 2,03 / 4,9 MiB)
          pretty:  12,45       --  6,91       =   9,68       -- ( 1,49 / 3,6 MiB)
          config:  13,95       --  7,47       =  10,71       -- ( 1,79 / 4,3 MiB)

Deep Structure:

                       READ           WRITE          SUM        SIZE
    name        :   pvd   thr  --  pvd   thr  =   pvd   thr  -- (ratio / size)
          binary:   1,11  1,00 --  0,65  1,00 =   0,88  1,00 -- ( 1,00 / 7,4 MiB)
     fast_binary:   1,64       --  1,82       =   1,73       -- ( 0,82 / 6,0 MiB)
  tuple_protocol:   2,56  0,97 --  2,43  0,58 =   2,50  0,78 -- ( 0,75 / 5,6 MiB)
 binary_protocol:   2,77       --  3,16       =   2,96       -- ( 1,00 / 7,4 MiB)
compact_protocol:   3,13  1,39 --  2,99  0,87 =   3,06  1,13 -- ( 0,81 / 6,0 MiB)
            json:   6,88       --  5,41       =   6,15       -- ( 1,22 / 9,0 MiB)
      json_named:   8,47       --  6,17       =   7,32       -- ( 1,73 / 12,8 MiB)
     json_pretty:  13,55       --  9,22       =  11,38       -- ( 3,40 / 25,1 MiB)
          pretty:  17,37       --  7,29       =  12,33       -- ( 1,71 / 12,6 MiB)
          config:  20,87       --  8,86       =  14,87       -- ( 2,64 / 19,5 MiB)

NOTE: The thrift JSON protocol was removed because some weird bug keeps messing up deserialization and stopping the test. I will put it back once the bug is fixed or can be bypassed.

NOTE: Since the test is for the speed of the serialization, we are only interested in the comparison between the serializers, not the absolute values. All numbers have been normalised to be relative to the native thrift binary protocol implementation (that’s why it’s 1.00 for thrift binary).

NOTE It seems to me like providence fares better the larger the test is and the more contentious the threads and memory are. E.g. this is a comparison over 100 runs of 5000 random generated structures each for the binary protocol only:

                           READ           WRITE           SUM         SIZE
        name        :   pvd   thr  --  pvd   thr   =   pvd   thr   -- (ratio / size)
          containers:   1,45  1,00 --  1,43  1,00  =   1,44  1,00  -- ( 1,00 / 43,2 MiB)
         many_fields:   0,91  1,00 --  0,97  1,00  =   0,94  1,00  -- ( 1,00 / 12,2 MiB)
many_required_fields:   0,86  1,00 --  0,97  1,00  =   0,92  1,00  -- ( 1,00 / 12,2 MiB)
      deep_structure:   0,82  1,00 --  0,85  1,00  =   0,84  1,00  -- ( 1,00 / 36,8 MiB)

But I have not been able to verify the cause. Sometimes I get the same change in timing with the same tests as above, still without me knowing why.