Skip to main content

FlatBuffers instead of JSON (or Protocol Buffers)

FlatBuffers is an efficient cross platform serialization library for C++, Java, C#, Go, Python and JavaScript (C, PHP & Ruby in progress). It was originally created at Google for game development and other performance-critical applications.

Why use FlatBuffers?

  • Access to serialized data without parsing/unpacking - What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility).
  • Memory efficiency and speed - The only memory needed to access your data is that of the buffer. It requires 0 additional allocations (in C++, other languages may vary). FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details.
  • Flexible - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don't have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don't, and how you design data structures.
  • Tiny code footprint - Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details.
  • Strongly typed - Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you.
  • Convenient to use - Generated C++ code allows for terse access & construction code. Then there's optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers). Java and Go code supports object-reuse. C# has efficient struct based accessors.
  • Cross platform code with no dependencies - C++ code will work with any recent gcc/clang and VS2010. Comes with build files for the tests & samples (Android .mk files, and cmake for all other platforms).
Why not use Protocol Buffers, or .. ?

Protocol Buffers is indeed relatively similar to FlatBuffers, with the primary difference being that FlatBuffers does not need a parsing/ unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocation. The code is an order of magnitude bigger, too. Protocol Buffers has neither optional text import/export nor schema language features like unions.


FlatBuffers on GitHub

Improving Facebook's performance on Android with FlatBuffers
FlatBuffers is a data format that removes the need for data transformation between storage and the UI. In adopting it, we have also driven additional architectural improvements in our app like Flat Models. The mutation extensions that we built on top of FlatBuffers allow us to track server data, mutations, and local state all in a single structure, which has allowed us to simplify our data model and expose a unified API to our UI components.

In last six months, we have transitioned most of Facebook on Android to use FlatBuffers as the storage format. Some performance improvement numbers include:


  • Story load time from disk cache is reduced from 35 ms to 4 ms per story.
  • Transient memory allocations are reduced by 75 percent.
  • Cold start time is improved by 10-15 percent.
  • We have reduced storage size by 15 percent.

FlatBuffersSwift on GitHub
Infrastructure for FlatBuffers, contains a reader and a builder

Comments

Most Favorite Posts

Server-driven UI (SDUI): Meet Zalandos AppCraft and AirBnB Lona

A short WTF: Joe Birch:  SERVER DRIVEN UI, PART 1: THE CONCEPT Zalando seems to follow the SDUI principle as well - defining a common design language and construct the screens on the backend while displaying them natively on the clients. They even go one step further; they implemented a mighty toolset to enable non-technical stakeholders to define their own native app screens Compass: Web tooling to create screens and bind data Beetroot: Backend service that combines the screen layout definition with the data Lapis/Golem: iOS/Android UI render engines Crazy cool! Good job, guys (when you do an open-source release?) To even move faster a Flutter based UI render engine implementation was great! See also AirBnB Lona SDUI approach Building a Visual Language Why Dropbox sunsetted its universal C++ mobile project and AirBnB its React Native implementation

FF Chartwell: Defining pie chart, bar charts and many others via vector font!

  With FF Chartwell you can define Charts via OpenType font, and transform a string, e.g. " 60+ 30 + 10 " into a Pie Chart 60%-30%-10% including color support (not only b/w!) without rendering anything: directly via font without using any images! Check this out: FontBlog FF Chartwell So great!

Apple Sneaks A Big Change Into iOS 5: Phasing Out Developer Access To The UDID

Apple is making a lot of big changes to its mobile operating system with iOS 5, which is dribbling out in betas for developers ahead of a general release later this year. But there is one big change some developers are just starting to take notice of that Apple isn’t talking about that much. In a recent update to the documentation for iOS 5 (which is only available to registered Apple developers, but a copy was forwarded to me), Apple notes that it will be phasing out access to the unique device identifier, or UDID, on iOS devices such as iPhones and iPads. TechCrunch Solution A: UIDevice-with-UniqueIdentifier-for-iOS-5 Brings back the unique identifier support under iOS 5, it uses the device's mac address in combination with the bundle identifier to generate a new hashed unique identifier. gekitz GitHub Alternate Solution B: Appsfire Announces Open Source UDID Replacement For iOS: OpenUDID It is based on the NSProcessInfo (see globallyUniqueString), but may change ...

Writing High-Performance Swift Code

Enabling Optimizations Whole Module Optimizations (WMO) Reducing Dynamic Dispatch Dynamic Dispatch Advice: Use 'final' when you know the declaration does not need to be overridden Advice: Use 'private' and 'fileprivate' when declaration does not need to be accessed outside of file Advice: If WMO is enabled, use 'internal' when a declaration does not need to be accessed outside of module Using Container Types Efficiently Advice: Use value types in Array Advice: Use ContiguousArray with reference types when NSArray bridging is unnecessary Advice: Use inplace mutation instead of object-reassignment Wrapping operations Advice: Use wrapping integer arithmetic when you can prove that overflow cannot occur Generics Advice: Put generic declarations in the same module where they are used The cost of large Swift values Advice: Use copy-on-write semantics for large values Unsafe code Advice: Use unmanaged references to avoid reference counting overhead Protocols Ad...

iOS In-App Browser JavaScript injections to spy on user behaviour on 3rd party websites

 iOS Privacy: Instagram and Facebook can track anything you do on any website in their in-app browser The iOS Instagram and Facebook app render all third party links and ads within their app using a custom in-app browser. This causes various risks for the user, with the host app being able to track every single interaction with external websites, from all form inputs like passwords and addresses, to every single tap. KrauseFX

Starting UIAutomation via command line

You can do it now, starting with XCode 4.2 for iOS5 beta 4 From command line, you can run instruments pointing to the automation template and specify as environment variables the test script you want to execute and destination path for results: instruments -w device_id -t /Developer/Platforms/iPhoneOS.platform/Developer/Library/Instruments/PlugIns/AutomationInstrument.bundle/Contents/Resources/Automation.tracetemplate application -e UIASCRIPT script -e UIARESULTSPATH results path DevForums Apple instruments Mac OS X Developer Tools Manual Page

10 Coolest Keyboard Shortcuts You Never Knew About

Keyboard shortcuts are the lifeblood of many productive Mac users. Not only can keyboard shortcuts make for a nicer user experience and cut down on the time needed to do a task, they can also just be plain cool. Mac OS X has many shortcuts ready and willing to do your bidding, but also has many hidden and unknown shortcuts. That's why we've compiled a list of 10 of the coolest keyboard shortcuts you never knew about. MacLife