Android programming has a small but very peculiar problem. And
when you want to introduce auto-generated code like thrift models into
it you tend to see it clearly. It has a 64k method limit per
file. And thrift generates a lot of methods. Larger RPC systems can
easily generate 20000 methods or more. And it was such a hassle that
even the folks at
decided they wanted to solve the problem, which they called
Well, when I saw that announcement I had already made
and it was only a few days ago.
But (and probably for this reason alone) the protobuf serializer model system has a separate nano format, which is designed to have as few methods as possible. And it goes to extremes to address the limitation. It has (per default):
- All fields are public, writable, no getter or setter methods.
- No descriptors for reflective uses.
- Repeated fields (similar to thrift list<*> types) are placed in arrays instead of list objects (no dependent classes).
bytesfield type (similar to binary) is placed in byte arrays, not ByteBuffers (ditto).
- Enums are integer constants only. Validity is only checked during (de)serialization (no enum, and can be optimized away by ProGuard).
- Only the read (mergeFrom) and write (writeTo) methods for serialization remain.
And everything else is optional. All of these are ways to avoid generating classes (all of which has at least some methods), enums (which on older android systems could slow down the app considerably), dependencies (which may require more classes to be kept by ProGuard), and otherwise avoid generating methods in general.
The Intermediate Solution
But removing all of that will still put some limitations of what the system can do. E.g. protobuf nano-java can only serialize to and from it’s main binary format and has no RPC service support. And thrifty is still limited by the TProtocol design flaw.
I have decided to have a look at this too, and made the
format for providence, which does
quite a bit of what thrifty did, but using the main structure from
providence, though no longer generating the reflection helpers
and thus rendering the default serializers useless, and cutting out most
The goal is to:
- Generate few methods (but not to avoid generating methods to the extreme).
- Requiring no java-reflection, which has a tendency of confusing ProGuard.
- Keep dependencies to the minimum (requiring few referenced classes).
And everything else that may drag in more dependencies or generate extra methods should be optional.
The generated classes
Not everything is supported yet, and some ways of using the classes are altogether removed. So the remaining part of the generated java is for enums:
- The value and name is wrapped in
asStringmethods, which are inherited from io-util interfaces, mainly used for serialization.
- The static
forNamemethods. These are needed for de-serialization.
And for messages:
- A single getter method for each field. Required and default primitive
type fields are always present. Otherwise only
nullis used as a presence outside the builder.
_Builderclass with a setter method per field. The builder keeps using a union field reference for presence checking during message building (unions only).
- The standard Object, Comparable and Stringable overridden methods:
- The standard builder-related methods:
message.mutate()are still there.
- Unions still has the
unionFieldmethod, but the fields are declared as int constants in place of the
The messages are still true immutable objects, so having messages in constants still make true constants. The whole has two dependencies:
net.morimekta.utils:io-util:0.2.3+for binary field support and some interfaces and helpers used in toString / asString methods, and for the
LinkedHash*Builderhelper classes for
com.google.guava:guava:*for the other immutable collections support.
Serialization with tiny-java
Since the new
tiny java classes no longer have the reflection tools
in place, serialization needs to be handled in a different way. I have
not settled on a method yet, but will probably try out something like
The interface would be two methods on each message class, one to write and one to read the desired binary format, probably decided at compile time:
MyMessage myMessage = MyMessage.read(istream); myMessage.write(ostream);
Then the methods themselves would use the
net.morimekta.utils:io-util to do the actual serialization
and deserialization, similar to how the default serializers work. The
dependency is already there for binary field support (and more).
Json can be handled in an entirely different manner, by using
I prefer Jackson 2 as it supports both using builders and specifying
custom serializers and deserializers. Many android apps these days already
rely on jackson to do some JSON, as it is a pretty fast, well tested
and not too include-intensive JSON library.
By adding a custom deserializer class for each message type, we can
ensure that we can deserialize all the various nits and tricks that comes
from the default format’s
JsonSerializer. For unions, a deserializer class
is also added, to avoid serializing all the fields. The net result is
as if we were using the
JsonSerializer(fields = NAME, enums = ID) variant
that also has disabled serializing to
And if JSON serialization is no longer supported, it can be removed by simply
not compiling with the
jackson option. Everything else should still work
- Remove jackson integration from the main java generator. The JsonSerializer format generates json that is compatible with the tiny jackson integration, so it should not be needed. And since the code has diverged so much it needs a serious update anyway.
- Generate read and write methods (as described above). Can add
helper classes in a new
providence-tinymodule, but mainly try to keep with the no extra deps policy. And rename the jackson module to
providence-tiny-jackson. It is no longer ‘core’.
- Generate pretty printing toString / asString variants on the tiny
messages, since the
PrettyPrinteris also useless.
- Make a release with all these updates. The current 0.1.0 version does
not have much of the updates I did during the long weekend for the
- Figure out how services would be done with the new format. It took me quite a while to figure out services for the main providence java format, so this is probably not trivial either.