Exchange JSON between Flutter App and JavaScript via URL
Being forced to cut down sports for a while… I started playing around with Flutter, checking out the flutter about Flutter.
Example Training App
The example project was a training app, to allow trainers to select predefined exercises for the next training session— and create new, custom exercises. To make it more interesting I wanted to work without backend, totally decentralized, exchanging data via URLs between users only.
Mermaids URL state
The basic idea of transporting data decentralized via URL was inspired by Mermaid.js, the great JS based UML tool.
If you didn’t check out Mermaid, yet: Do it!
It’s great for sharing ideas and specifications — and adopting them quickly!
The Mermaid online editor does an interesting thing, to persist the state of the diagram: meanwhile the Mermaid description language is text-based, the state of the diagram is updated with any change in the URL:
https://mermaid.live/edit#pako:eNpV…QG-cVWh
If you want to send you diagram to someone else, the only thing you have to share was the URL, and the receiver can directly continue your work in the Mermaid online editor.
Take a look at the following Mermaid example diagram, play around with it and checkout the changes in the URL:
Even though the URL size isn’t limited by RFC, different browsers have a de facto limit of around 2k chars, which makes is necessary to compress the JSON to stay in this limit.
Internally Mermaid is using the JavaScript pako zlib library to gzip the text and store it in the URL:
Caveat
One caveat of the decentralized transportation of data though is being out-of-sync quickly: if the data is changed by another person, you will not get the changes unless the other persons sends you the updated URL containing the updated data.
The advantage is you don’t have to deal with any backend infrastructure beyond serving a static HTML file. It’s fast & cheap.
So, let’s bring it on!
Back to the mission
The mission was to design a training app, making it possible to not only select exercises for the next training, but to create new trainings as well!
The exercise structure is defined in a JSON format. Next question arises is of course: How to write an exercise editor?
Yes, you could implement a full-fledged, gesture based editor with Flutter, but let’s go MVP and try something else…
JSON-Editor
There is an existing JavaScript lib solution, that automatically creates a visual editor, based on a JSON schema:
Ok, we have some JSON examples, and I know the structure, but how to formalize this in the JSON schema syntax? Yes, I could do this by hand, but…
ChatGPT can help!
So throwing the examples in ChatGPT, plus specifying the possible enum values, that are not seen in the examples, did the job very well!
Feeding the JSON-Editor with the ChatGPT-generated schema now results in a usable JavaScript editor.
Feels like magic!
Even though it still has the haptics of a SAP input mask…
Joining Flutter app with the JavaScript JSON editor
Now we have a Flutter training app that can visualize JSON-based exercises, plus we have a separate JavaScript-based exercise editor.
How do we get this together?
Flutter can not only run as app, but in browser as well. So the basic idea was to trigger the exercise editor via the Flutter app, by constructing a URL with the exercise as payload, then edit or create a new exercise in the JSON editor and finally construct a URL with payload to send the edited exercise back to the Flutter app.
Libraries
For Flutter there is the great archive package, providing different compression options.
For the JavaScript editor we can follow the example of Mermaid and use pako as well! Let’s take a look at sample code.
Additionally to the pure compression, we’ll add base64 encoding, and make sure not to break the UTF-8 encoding.
Un-/compress with Flutter and archive
String compressJson(String input
String) {
final utf8String = utf8.encode(inputString);
final encodedData = GZipEncoder().encode(utf8String);
final encodedString = base64.encode(encodedData!);
return encodedString;
}
static Map<String, dynamic> uncompressJson(String compressedJson) {
final decodedString = base64.decode(compressedJson);
final decodedData = GZipDecoder().decodeBytes(decodedString);
final utf8String = utf8.decode(decodedData);
return jsonDecode(utf8String);
}Un-/compress with JavaScript and pako
// Function to uncompress JSON data that was previously compressed and base64-encoded
function uncompressJson(compressedBase64Data) {
// Step 1: Decode the base64-encoded data to binary
var binData = atob(compressedBase64Data);
// Step 2: Convert the binary data to an array of character codes
var charData = binData.split('').map(function (e) {
return e.charCodeAt(0);
});
// Step 3: Create a Uint8Array from the character data
var binDataUint8 = new Uint8Array(charData);
// Step 4: Inflate (decompress) the binary data using the pako library
var data = pako.inflate(binDataUint8);
// Step 5: Convert the uncompressed binary data to a string
var uncompressedString = String.fromCharCode.apply(null, new Uint16Array(data));
// Step 6: Parse the uncompressed string as JSON and return the JavaScript object
return JSON.parse(uncompressedString);
}
// Function to compress JSON data and base64-encode result
function compressJson(jsonData) {
// Step 1: Convert JSON to a string
var jsonString = JSON.stringify(jsonData);
// Step 2: Compress the string using zlib
var compressedData = pako.gzip(jsonString);
// Step 3: Convert the compressed data to base64
var compressedBase64Data = btoa(String.fromCharCode.apply(null, compressedData));
// Step 4: Return the base64-encoded compressed data
return compressedBase64Data;
}Was a little hassle at the beginning to get the right encoding and decodings in the right order, but now it works as a charm!
The compressed and encoded data will then be sent as HTTP GET param, making sure to URLDe-/Encode it before.
Works as a charm?!
So… in general everything worked out as a charm, but the devil lies in the details (German proverb):
I had a bad time finding out, that the GoRouter, that I am using on the Flutter side for navigation, seems to break the URL encoding from GET params, replacing plus-signs with empty spaces...
WoMan that’s a bad thing!
I am pretty sure the GoRouter messes up the decoding, but as said, I didn’t dig into it, and am not sure that I have pointed to the real culprit!
Didn’t had the time to go into the details, and voted for a simple hack on Flutter side, to repair the encoding:
final queryParams = state.queryParams;
var jsonQueryParam = queryParams['json'];
// TODO ok, now the following line is an evil hack to fix encoding/decoding bullshit!
// index.html seems to encode the '+' sign correctly with '%2B' while on flutter this is decoded as ' '
// Didn't find the reason...
jsonQueryParam = jsonQueryParam?.replaceAll(' ', '+');Summary
During my Odyssee I learned a lot:
- How to work with Flutter, GoRouter, BLoC and Cubit
- How to use compression, both on Flutter and JavaScript side
- The URL length limit of browsers
- How to use decentralized data exchange
- How to auto-generate a JSON schema by JSON examples using ChatGPT
- How to generate an visual editor by a JSON schema
Leaving out the details, this article focusses on the possibility of exchanging data via URL, and how to compress the data not to exceed the URL length limit.
Of course there are different ways of solving communcation between Flutter and JavaScript; you find additional hints in the appendix.
Appendix
How to use JS with Flutter Web
I wanted to keep the apps separated, otherwise I could have used this approach:
How to use JS with Flutter Web
How to interop between JavaScript and Dart with Flutter Web
fireship.io
Flutter: Two-way communication between a WebView and a web page
Best JSON Compression
gzip is fine for the use case, but if you are heading for more optimal solutions you might want to checkout:
