Realm JavaScript v12: complete rewrite without surprises
The first commit in the Realm JavaScript repository is from May 2015. With 100+ contributors and 5.3k Github stars, the team, product priorities, JavaScript engines, and frameworks like React Native have evolved in those last eight years. The architecture worked great in the early years, while it has become increasingly difficult keeping up with the thriving React Native community, addingnew features, and supporting the increasing number of JavaScript engines and runtime environments.
About one year ago, the team began a complete rewrite to leverage the things we have learned and to accommodate the changes to the supported platforms.
The goal of the project has been to move to a totally new architecture without bringing any major surprises (aka breaking changes) for our users.
In the last months, we have released a couple of prereleases of the rewrite in order to learn from our users. Luckily, many users have tried these prereleases and reported back issues. We appreciate the time you have spent trying it out, and it has been motivating to engage with dedicated users.
Changes to the existing API
The goal of the rewrite is to keep the public API as it is and change the internal implementation. To ensure that we are keeping the API mostly untouched, we are either reusing or rewriting the tests we have written over the years.
We are changing our collection classes a bit. Today, they derive from a common Collection class that is modelled over ReadonlyArray. It is problematic for Realm. Dictionary as there is no natural ordering. Furthermore, we’ve contemplated deprecating our namespaced API since we find it out of touch with modern TypeScript and JavaScript development. However, to keep the breakage to a minimum, we’ll leave the namespaced API as-is for the 12.0.0 release. We are deprecating support for Altas push notifications (they have been deprecated some time ago).
How to define schemas has been cleaned up. In the last four years, we have been adding new data and collection types, and it has increased the complexity of our schema parser. You might experience a breaking change as our new schema parser is providing a much better and consistent schema syntax. Mixing object style and short-hand style is not possible. In the code snippet “Bad” is what is no longer possible, and “Good” is what you should write instead.
const TaskSchema = {
name: "Task",
properties: {
description: /* property schema (shorthand or object form) */,
},
};
// Explicitness
"[]" // Bad (previously parsed as implicit "mixed")
"mixed[]" // Good
{ type: "list" } // Bad
{ type: "list", objectType: "mixed" } // Good
// Mixing shorthand and object form
{ type: "int[]" } // Bad
"int[]" // Good
{ type: "list", objectType: "int" } // Good
{ type: "int?" } // Bad
"int?" // Good
{ type: "int", optional: true } // Good
// Specifying object types
{ type: "SomeType" } // Bad
"SomeType" // Good
{ type: "object", objectType: "SomeType" } // Good
{ type: "object[]", objectType: "SomeType" } // Bad
"SomeType[]" // Good
{ type: "list", objectType: "SomeType" } // Good
{ type: "linkingObjects", objectType: "SomeType", property: "someProperty" } // GoodMoreover, we discourage mixing class-based models and schemas defined by JavaScript objects when developing in TypeScript. You will likely get an error if you do realm.objects<Person>(“person”). If you use class-based models, consider using realm.objects<Person>(Person) instead.
Our new architecture
Realm JavaScript builds on Realm Core, which is composed of a storage engine, query engine, and sync client connecting your client device with MongoDB Atlas. Realm Core is a C++ library, and the vast majority of Realm JavaScript’s C++ code in our old architecture calls into Realm Core. Another large portion of our old C++ code is interfacing with the different JavaScript engines we are supporting (currently using NAPI (Node.js and Electron) and JSI (JavaScriptCore and Hermes on React Native)).
Our rewrite creates two separated layers: i) a handcrafted SDK layer and ii) a generated binding layer. The binding layer is interfacing with the JavaScript engines and Realm Core. It is generated code, and our code generator (aka binding generator) will read a specification of the Realm Core API and generate C++ code and TypeScript definitions. The generated C++ code can be called from our SDK, which is written in TypeScript.
On top of the binding layer, we implement a hand-crafted SDK layer. It is an implementation of the Realm JavaScript API as you know it. It is implemented using classes and methods in the binding layer as building blocks.
We see a number of benefits from this rewrite:
Deliver new features faster
First, our hypothesis is that we are able to deliver new functionality faster. We don’t have to write so much C++ boilerplate code as we have done in the past.
Provide a TypeScript-first experience
Second, we are implementing the SDK in TypeScript, which guarantees that the TypeScript definitions will be accurate and consistent with the implementation. If you are a TypeScript developer, this is for you. Likely, your editor will guide you through integrating with Realm, and it will be possible to do static type checking and analysis before deploying your app in production. We are also moving from JSDoc to TSDoc so the API documentation will coexist with the SDK implementation. Again, it will help you and your editor in your day-to-day work, as well as eliminate the previously seen inconsistencies between the API documentation and TypeScripts definitions.
Facilitate community contributions
Third, we are lowering the bar for you to contribute. In the past, you likely had to have a good understanding of C++ to open a pull request with either a bug fix or a new feature. Many features can now be implemented in TypeScript alone by using the building blocks found in the binding layer. We are looking forward to seeing contributions from you.
Generate more optimal code
Last but not least, we hope to be able to generate more optimal code for the supported JavaScript engines. In the past, we had to write C++ code which was working across multiple JavaScript engines. Our early measurements indicate that many parts of the API will be a little faster, and in a few places, it will be much faster.
New features
As the number of our tests passing increased during the rewrite, our confidence that the rewrite is the right approach, we stopped developing new features for series 11 of Realm JavaScript. A hard decision was made: all new features will only be introduced in version 12, and let us briefly introduce some highlights.
First, a new unified logging mechanism has been introduced. It means that you can get more insights into what the storage engine, query engine, and sync client are doing. The goal is to make it easier for you to debug. You provide a callback function to the global logger, and log messages will be captured by calling your function.
type Log = {
message: string;
level: string;
};
let logs: Log[] = [];
Realm.setLogger((level, message) => {
logs.push({ level, message });
});
Realm.setLogLevel("all");Second, the full-text search will be supported. You can mark a string property to be indexed for full-text search, and Realm Query Language allows you to query your Realm. Currently, the feature is limited to Latin alphabets. Advanced functionality like stemming and spanning across properties will be added later.
interface IStory {
title?: string;
content?: string;
}
class Story extends Realm.Object<Story> implements IStory {
title?: string;
content?: string;
static schema: ObjectSchema = {
name: "Story",
properties: {
title: { type: "string" },
content: { type: "string", indexed: "full-text" },
},
primaryKey: "title",
};
}
// ... initialize your app and open your Realm
let amazingStories = realm.objects(Story).filtered("content TEXT 'amazing'");We are introducing our first version of geospatial queries. The Realm Query Language has been expanded with a new operator geoWithin. In order for your geodata to work with Realm you must transform the data into certain shapes (it will be carefully documented how to do it). In the future, we might add more operators and special types for geodata.
// Example of a user-defined point class that can be queried using geospatial queries
class MyGeoPoint extends Realm.Object implements CanonicalGeoPoint {
coordinates!: GeoPosition;
type = "Point" as const;
static schema: ObjectSchema = {
name: "MyGeoPoint",
embedded: true,
properties: {
type: "string",
coordinates: "double[]",
},
};
}
class PointOfInterest extends Realm.Object {
name!: string;
location!: MyGeoPoint;
static schema: ObjectSchema = {
name: "PointOfInterest",
properties: {
name: "string",
location: "MyGeoPoint",
},
};
}
realm.write(() => {
realm.create(PointOfInterest, {
name: "Copenhagen",
location: {
coordinates: [12.558892784045568, 55.66717839648401],
type: "Point",
} as MyGeoPoint
});
realm.create(PointOfInterest, {
name: "New York",
location: {
coordinates: [-73.92474936213434, 40.700090994927415],
type: "Point",
} as MyGeoPoint
});
});
const berlinCoordinates: GeoPoint = [13.397255909303222, 52.51174463251085];
const radius = kmToRadians(500); // 500 km = 0.0783932519 rad
// Circle with a radius of 500 kms centered in Berlin
const circleShape: GeoCircle = {
center: berlinCoordinates,
distance: radius,
};
// All points of interest in a 500 kms radius from Berlin
let result = realm.objects(PointOfInterest)
.filtered("location geoWithin $0", circleShape);Last, a new experimental subscription API for flexible sync has been added. The aim is to make it easier to subscribe and unsubscribe by providing subscribe() and unsubscribe() methods directly on the query result.
const peopleOver20 = await realm
.objects("Person")
.filtered("age > 20")
.subscribe({
name: "peopleOver20",
behavior: WaitForSync.FirstTime, // Default
timeout: 2000,
});
// …
peopleOver20.unsubscribe();A better place
While Realm JavaScript version 12 does not bring major changes for you as a developer, we believe that the code base is now at a better place. The code base is easier to work with, and it is an open invitation for you to contribute.
If you are using React Native, we encourage you to try out @realm/react together with Realm JavaScript. It will save you from writing a lot of boilerplate code, and you will be able to get the most out of your Realm database as possible.
