A quick trick to make binary data in Flutter a breeze
16 Feb 2022
Introduction
Not all data comes in a nice, easy to deal with format.
On a audio-MIDI device Flutter app project I've been working on recently, I needed to deal with binary data in an arbitrary format. Now the data format is luckily well documented by the device manufacturer Korg (in nice ascii-art tables of all things!) but it was still proving a real pain to deal with in Dart.
The problem
Why was dealing with the binary data in Dart such an issue? Well because its made up of many well defined fields, the data was essentially just a blob of bytes in a custom format, so I didn't have any existing packages that support commonly used data exchange formats (JSON, XML, ProtoBufs, etc) to help me deal with parsing from it and encoding to it.
Having to deal with the binary data in Dart as just a list of ints or Uint8List's, I found I was having to reams of boiler plate code to parse it out into instances of Dart classes.
To give you a better idea of the kind of data I needed to deal with, here is a (very!) truncated part of the data's documentation:
TABLE 1 : Pattern Parameter ( 1 Pattern, Current Pattern )
No. : No. in the Pattern dump data.
+-------------+-------------------+--------------------------------------+
| No. | PARAMETER | VALUE/DESCRIPTION |
+-------------+-------------------+--------------------------------------+
| 0~3 | Header | 'PTST' = 54535450[HEX] |
+-------------+-------------------+--------------------------------------+
| 4~7 | Size | |
+-------------+-------------------+--------------------------------------+
| 8~11 | (reserved) | |
+-------------+-------------------+--------------------------------------+
| Pattern Version (TABLE 2) |
+-------------+-------------------+--------------------------------------+
| 16~33 | Pattern Name | null terminated |
+-------------+-------------------+--------------------------------------+
| 34~35 | Tempo | 200~3000 = 20.0 ~ 300.0 |
+-------------+-------------------+--------------------------------------+
| 36 | Swing | -48 ~ 48 |
+-------------+-------------------+--------------------------------------+
| 37 | Length | 0~3 = 1~4bar(s) |
+-------------+-------------------+--------------------------------------+
...
+-------------+-------------------+--------------------------------------+
| 1840~2047 | (reserved) |
+-------------+-------------------+--------------------------------------+
| 2048~2863 | Part 1 Parameter (TABLE 6) |
+-------------+-------------------+--------------------------------------+
| . | |
| . | |
| . | |
+-------------+-------------------+--------------------------------------+
| 14288~15103 | Part 16 Parameter (TABLE 6) |
+-------------+-------------------+--------------------------------------+
...
As you can see from the table, the data is not only separated into arbitrary length fields, but is also nested, as the "Part X Parameter" field is itself it's own data structure of fields, which in turn has other nested data structures! If you are curious, feel free to grab the docs and see just how many and varied fields there are.
So as I was writing out one .sublist(x, y) piece of Dart code after another, I quickly began trying to think of a better way of dealing with this...
The neat solution
And sure enough, there was! What came to my rescue were actually 2 things.
There was and always will be C...
My first stroke of luck was that a kind soul had already written code to parse and encode the data and had published their work on Github. Of course they had chosen to do this in C, which actually makes perfect sense, because this kind of work in C is its bread and butter and easy to deal with given the use of C structs. With C structs, the above table becomes:
struct PatternType{
unsigned char header[4];
unsigned char size[4];
unsigned char fill1[4];
unsigned char fill2[4];
unsigned char name[18];
unsigned char tempo1;
unsigned char tempo2;
unsigned char swing;
...
struct PartType part[16];
...
which may not seem that much of a big deal until you realise that "parsing" the data is as simple as just pointing to a bit of memory with the data and saying that bit of memory is the above struct. And then accessing each field is as trivial as:
if (data->header[0] != 0x50) {
return -1;
}
And now for that neat trick...
When I thought about using that C code via FFI, at first I thought it would not actually be that useful for me, as I'd still be left with needing to write lots of Dart FFI boilerplate code, essentially lots of getters/setters calling lots more C code I would have to write, wrapping up doing each field access like above.
But then I remembered that when in the past I had used the very handy Dart ffigen package, it had actually generated the Dart boilerplate to access structs that were needed to call the C functions in the libraries I was using.
It was at this point that I had my eureka moment and realised: what if I just ffigen generate just the code for the structs. Infact the header file for the elecmidi tool actually only defined the structs, without any functions, which makes sense given it was written to be a cli executable and not a library.
And sure enough it worked! Given the elecmidi header file of structs, ffigen generated Dart code like this:
class PatternType extends ffi.Struct {
@ffi.Array.multi([4])
external ffi.Array<ffi.Uint8> header;
@ffi.Array.multi([4])
external ffi.Array<ffi.Uint8> size;
@ffi.Array.multi([4])
external ffi.Array<ffi.Uint8> fill1;
@ffi.Array.multi([4])
external ffi.Array<ffi.Uint8> fill2;
@ffi.Array.multi([18])
external ffi.Array<ffi.Uint8> name;
Which meant I could write my code like this:
if (patternData.ref.header[0] != 0x50) {
throw Exception('invalid header: [${patternData.ref.header[0]}]');
}
All the "parsing" done for me and very nice, named properties for each field!
Easy, peasy, lemon squeezy!
But wait there's more
What I haven't covered above is how to get a raw list of bytes in Dart into something that Dart is happy treating as a instance of the struct. If the data was coming from a FFI function call return value, this would "just work" but because I was actually dealing with data only in Dart I ended up having to do a few little hoop-jumps like so:
final decoded = getTheListofBytesFromSomewhere();
final ffi.Pointer<ffi.Uint8> p = calloc(decoded.length);
final buf = p.asTypedList(decoded.length);
for (var i = 0; i < decoded.length; i++) {
buf[i] = decoded[i];
}
final Pointer<PatternType> patternData = Pointer.fromAddress(p.address);
There may well be a better/simpler way to achieve the above, so do please let me know via twitter or a comment here if you know of it.
And there was one last little thing to be aware of. The careful reader would have noticed that in the above example, many of the fields are arrays in C, eg:
unsigned char name[18];
and in Dart become:
@ffi.Array.multi([18])
external ffi.Array<ffi.Uint8> name;
which are not the same as pointers to arrays and this stumped me for a while as to how to access them as the documentation on ffi Arrays is a bit sparse at the moment. However I did figure it out, it turns out that you can access elements of the array using the normal Dart [] operator, though not via an iterator, though an issue for that has been filed.
Conclusion
And that really is all there is to it. While dealing with binary data may not be a common requirement in Flutter (or plain Dart) development, I hope the above "trick" helps someone in the future avoid the need to write alot of error-prone boilerplate and I would even suggest it is worth writing C struct definitions for this use even if you are not as lucky as I was to have had someone already write them for you!
I hope this has been of help to you and if it has or you have any other handy tips to share, please let me know in the comments below or via Twitter.
