Skip to main content

FlatBuffers instead of JSON (or Protocol Buffers)

FlatBuffers is an efficient cross platform serialization library for C++, Java, C#, Go, Python and JavaScript (C, PHP & Ruby in progress). It was originally created at Google for game development and other performance-critical applications.

Why use FlatBuffers?

  • Access to serialized data without parsing/unpacking - What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility).
  • Memory efficiency and speed - The only memory needed to access your data is that of the buffer. It requires 0 additional allocations (in C++, other languages may vary). FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details.
  • Flexible - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don't have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don't, and how you design data structures.
  • Tiny code footprint - Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details.
  • Strongly typed - Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you.
  • Convenient to use - Generated C++ code allows for terse access & construction code. Then there's optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers). Java and Go code supports object-reuse. C# has efficient struct based accessors.
  • Cross platform code with no dependencies - C++ code will work with any recent gcc/clang and VS2010. Comes with build files for the tests & samples (Android .mk files, and cmake for all other platforms).
Why not use Protocol Buffers, or .. ?

Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocation. The code is an order of magnitude bigger, too. Protocol Buffers has neither optional text import/export nor schema language features like unions.


FlatBuffers on GitHub

Improving Facebook's performance on Android with FlatBuffers
FlatBuffers is a data format that removes the need for data transformation between storage and the UI. In adopting it, we have also driven additional architectural improvements in our app like Flat Models. The mutation extensions that we built on top of FlatBuffers allow us to track server data, mutations, and local state all in a single structure, which has allowed us to simplify our data model and expose a unified API to our UI components.

In last six months, we have transitioned most of Facebook on Android to use FlatBuffers as the storage format. Some performance improvement numbers include:


  • Story load time from disk cache is reduced from 35 ms to 4 ms per story.
  • Transient memory allocations are reduced by 75 percent.
  • Cold start time is improved by 10-15 percent.
  • We have reduced storage size by 15 percent.

FlatBuffersSwift on GitHub
Infrastructure for FlatBuffers, contains a reader and a builder

Comments

Most Favorite Posts

How to link to TestFlight App in iOS

There are two things you need to do. First, check to see if TestFlight is installed. Then create a new link to your app.

NSURL *customAppURL = [NSURL URLWithString:@"itms-beta://"];
if ([[UIApplication sharedApplication] canOpenURL:customAppURL]) {

    // TestFlight is installed

    // Special link that includes the app's Apple ID
    customAppURL = [NSURL URLWithString:@"https://beta.itunes.apple.com/v1/app/978489855"]; 
    [[UIApplication sharedApplication] openURL:customAppURL];
}

This special https://beta.itunes.apple.com URL will be opened directly in TestFlight.

Finally, if you are using iOS 9 (or later), you need to make an addition to your Info.plist to get the canOpenURL: method to work.

If your app is linked on or after iOS 9.0, you must declare the URL schemes you want to pass to this method. Do this by using the LSApplicationQueriesSchemes array in your Xcode project’s Info.plist file. For each URL scheme you want your app to use with this method, add it …

Running Espresso Tests in parallel with Spoon and Android Test Sharding

Introduction to Android Espresso Testing and Spoon

Espresso UI test automation framework is Google’s de-facto testing platform for Android app developers.

No test engineer or developer will be quite unless it validates the functionality of his app on multiple devices and emulators. For that, there is another widely used tool called Spoon (there are also cloud-based solutions as mentioned above that support parallel execution on real devices). This tool, will collect all the target devices (that are visible via adb devices) test results and aggregate them into one HTML view that can be easily investigated.

Mobile Testing Blog


Android Test Sharding

The test runner supports splitting a single test suite into multiple shards, so you can easily run tests belonging to the same shard together as a group, under the same Instrumentation instance. Each shard is identified by an index number. When running tests, use the -e numShards option to specify the number of separate shards to create and the…

Using Speech with iOS and Android: SiriKit, Voice Capabilities, Google Assistant

SiriKit
SiriKit enables your iOS apps and watchOS apps to work with Siri, so users can get things done using just their voice. Your content and services can be used in new scenarios including access from the lock screen and hands-free use.

Apps adopt SiriKit by building an extension that communicates with Siri, even when your app isn’t running. The extension registers with specific domains and intents that it can handle. For example, a messaging app would likely register to support the Messages domain, and the intent to send a message. Siri handles all of the user interaction, including the voice and natural language recognition, and works with your extension to get information and handle user requests.

Apple Developer


Adding Voice Capabilites

Voice actions are an important part of the wearable experience. They let users carry out actions hands-free and quickly. Wear provides two types of voice actions:

System-provided
These voice actions are task-based and are built into the Wear platfo…

NSURLConnection with Accept-Encoding: gzip

For quite some time I ranted about not being able to use compressed network communcation out-of-the-box on the iPhone.

Despite being undocumented (or I just overlooked the hint), NSURLConnection does gzip decompression transparently!

That’s how to use it:

NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url
cachePolicy:NSURLRequestReloadIgnoringLocalCacheData
timeoutInterval:60.0];
// set explicitly:
[request setValue:@"gzip" forHTTPHeaderField:@"Accept-Encoding"];

MRo BLOG

iOS: LLVM based mutation testing to check your unit test reliability, Android and Kotlin: JVM based mutation testing

LLVM-BASED MUTATION TESTING SYSTEM

Several years ago I discovered very powerful technique - Mutation Testing. Since then I was (and still am) dreaming to have a tool which will do the job for languages like C, Objective-C and C++.

Now I have confidence in that it can be implemented using LLVM.

Outline for this article:

overview of mutation testingmutation testing system - proof of conceptthoughts on potential implementation of MT system using LLVM
LowLevelBits
Mutation testing system built on top of LLVM on GitHub.
mull-project / mull


JVM-BASED MUTATION TESTING

PIT is a state of the art mutation testing system, providing gold standard test coverage for Java and the jvm. It's fast, scalable and integrates with modern test and build tooling.

PIT Mutation Testing

App Store Optimization (ASO) Factors and Trends 2018

The ASO Factors & Trends for 2018 study is based on survey data from over 60 ASO and app marketing professionals who were asked to evaluate the effect of different on-metadata and off-metadata factors on Search Rankings and Conversion Rate of mobile apps and games.

The TOP 5 factors responsible for ASO on Google Play Store and Apple App Store (on both Search and Conversion Rate level) are nearly identical:

App Name / TitleLocalized product pageUser RatingsSubtitle / Short DescriptionUser reviewsTheTool

Another nice blog about ASO:

Important Tips on Writing An App DescriptionThe Apple App Store Optimization Guide5 Tips on How to Increase App DownloadsKeywords Optimization...MobileAction

Validity Time Auto-Renewables in Sandbox

The subscription durations, sandbox durations and incentive durations of auto-renewables.

Hint: In Sandbox the validity time differs from live environment!!!


DurationsSandbox DurationIncentive Durations (optional)7 days3 minutes7 days1 month5 minutes7 days, 1 month2 months10 minutes7 days, 1 month3 months