Building Languages on ADAM Layer
ADAM serves as an interoperability layer, seamlessly merging various decentralized, centralized, and federated networks and applications into a unified, distributed social environment.
Currently, its core backend infrastructure is built on top of Holochain. With our recent Rust refactoring, we’ve included Holochain directly into our binary as a lib and upgraded to v0.2. While this version of Holochain is not considered stable yet, it showed better performance in most cases — so we wanted to go with it. However, during the testing phase of our flagship application, Flux, we noticed certain inconsistencies in gossip between peers within a Holochain DHT. These inconsistencies were seen with our LinkLanguage; that is our implementation for graph syncing between peers. Other Languages which handle object, and file storage (Expression Languages) were working with reasonable success. These inconsistencies caused delays in message delivery or in sync time when accessing a space.
To ensure a reliable user experience with ADAM & Flux, instead of rolling back to the stable Holochain version 0.1.6, we decided to start building a new Language implementation on a centralized server, providing a stable foundation for ADAM applications.
A key advantage of ADAM is its flexibility; it allows a complete infrastructure switch without needing to alter the app code. This empowers users to choose their preferred level of decentralization and grants developers the freedom to select the most suitable tech stack. This also marks the start of our integration process for systems beyond Holochain, with Gun.db and Hypercore being our next targets.
This post delves into the construction of this centralized LinkLanguage, its essence, and its integration into the ADAM framework.
ADAM Languages
The ADAM Layer brings an innovative meta ontology that replaces traditional applications with a cohesive and interconnected ecosystem. A “Language” is a Deno bundle that provides specific functions within the ADAM Layer. These Languages provide the core functioning of social spaces in ADAM. Allowing developers to easily build applications across ecosystems and leverage the work of developers that came before them.
A Language must make available the following interface for ADAM:
export interface Language {
readonly name: string;
// Adapter implementations:
/** ExpressionAdapter implements means of getting an Expression (object)
* by address and putting an expression */
readonly expressionAdapter?: ExpressionAdapter;
/** Interface for getting UI/web components for rendering Expressions of this
Language */
readonly expressionUI?: ExpressionUI;
/** Interface of LinkLanguages for the core implementation of Neighbourhoods */
readonly linksAdapter?: LinkSyncAdapter;
/** Additional Interface of LinkLanguages that support telepresence features,
* that is:
* - seeing who is online and getting a status
* - sending/receiveing p2p signals to other online agents without affecting
* the shared Perspective (Graph) of the Neighbourhood
* (see TelepresenceAdapter for more details) */
readonly telepresenceAdapter?: TelepresenceAdapter;
/** Implementation of a Language that defines and stores Languages*/
readonly languageAdapter?: LanguageAdapter;
/** Optional adapter for getting Expressions by author */
readonly getByAuthorAdapter?: GetByAuthorAdapter;
/** Optional adapter for getting all Expressions */
readonly getAllAdapter?: GetAllAdapter;
/** Optional adapter for direct messaging between agents */
readonly directMessageAdapter?: DirectMessageAdapter;
/** Interface for providing UI components for the settings of this Language */
readonly settingsUI?: SettingsUI;
/** Optional function to make any cleanup/teardown if your language gets deleting
in the ad4m-executor */
readonly teardown?: () => void;
/** All available interactions this agent could execute on given expression */
interactions(expression: Address): Interaction[];
}
LinkLanguage
Based on its purpose, a Language may offer a range of Adapters. A complete list of interfaces is available here.
One of the fundamental adapters is the LinkSyncAdapter, implementing this on a Language will enable it to be used a “LinkLanguage”. A closer look at its interface reveals:
export interface LinkSyncAdapter {
//Is writeable by current agent
writable(): boolean;
//Is a public space
public(): boolean;
//Other people using this language
others(): Promise<DID[]>;
//My current latest state
currentRevision(): Promise<string>;
//Fetch Links that I have not already seen
sync(): Promise<PerspectiveDiff>;
//Fetch all the Links in this space
render(): Promise<Perspective>;
//Commit a link into this space
commit(diff: PerspectiveDiff): Promise<string>;
//Register a callback that can be triggered on Link mutations (links added or removed)
addCallback(callback: PerspectiveDiffObserver): number;
…
}In ADAM, LinkLanguage drives an essential component named the ‘Neighbourhood’.
Imagine the Neighbourhood as a collaborative hub where agents gather and exchange information. Incorporating Links into a Neighbourhood enables shared application data dissemination between agents. Interestingly, the mechanism your application employs to interact with a Neighbourhood remains constant, regardless of the underlying LinkLanguage in play!
There are a lot of competing database structures out there, like document, tree, tabular, relational…it just seems to go on. But there is one structure that rules them all — the graph. With graphs, you can make any other data structure.
A Link in a Neighbourhood consists of “triples” (subject-predicate-target). These triples serve as the foundational pillars in LinkLanguage, forming an intricate web of interconnected knowledge. The potential applications are immense; from LinkLanguages that operate over existing platforms like Wikipedia or Discord to ones that birth entirely unique spaces.
Here’s a basic illustration of how one might incorporate a chat message into a space using the JS ad4m-client:
let addLink = perspective.addLink("<neighbourhood-uuid>", {
source: "ad4m://self",
predicate: "ad4m://has_message",
target: "literal://HelloWorld!"
});
let getLinks = perspective.queryLinks("<neighbourhood-uuid>", {
source: "ad4m://self",
predicate: "ad4m://has_message"
});
console.log(getLinks);
Output:
[
{
author: "<my-did>",
timestamp: "2023–10–19T16:44:29+0000",
data: {
source: "ad4m://self",
predicate: "ad4m://has_message",
target: "literal://HelloWorld!"
},
proof: {
key: "<did-signing-key>",
signature: "<signature-hash>",
valid: true
}
}
]Note the responses here being complete with signature information. ADAM handles link signing for you, giving automatic provenance.
You can find more about that API here and class interfaces for defining and interacting with a specific graph structure in a neighbourhood here.
Lastly, to see more information about how to associate a Language with a given Neighbourhood take a look at the docs here.
Centralizing LinkLanguage
For our centralized link language we are going to use socket.io as a way to share links with other agents.
This setup is based on a straightforward `express.js` server showcasing our socket.io handlers. The complete code can be found here, and it’s hosted at [https://socket.ad4m.dev]. Note that while this is our custom implementation for link synchronization, you can implement a similar logic to launch your server. This could be a server initiating fresh spaces or a wrapper for an existing service or database.
Let’s briefly touch upon the primary functions our Language should possess:
1. Committing Data: Agents should be able to add/remove links from the network.
2. Data Synchronization: Agents must have a mechanism to fetch all data updates and avoid duplicate entries.
3. Live Updates: Agents should receive real-time updates as links are added or removed, essential for real-time apps!
For a deeper dive into the logic and structure, you can access the full Language implementation here.
0. Language Creation and Initialization
Create
At the heart of language instantiation in the ADAM Layer lies the `create` function. This function is crucial for the ADAM runtime, allowing access to a Language’s underlying Adapters during installation. When someone creates or joins a Neighbourhood with a Language that implements a `LinkSyncAdapter` implementation, this function is executed.
//!@ad4m-template-variable
const name = "centralized-perspective-diff-sync";
//!@ad4m-template-variable
const uid = "centralized-perspective-diff-sync-uuid";
export default async function create(context: LanguageContext): Promise<Language> {
let socketClient = io("https://socket.ad4m.dev", {
transports: ['websocket', 'polling'],
autoConnect: true,
query: { did: context.agent.did, linkLanguageUUID: uid }
});
console.log("Created socket connection");
const linksAdapter = new LinkAdapter(context, uid, socketClient);
return {
name,
linksAdapter as LinkSyncAdapter,
interactions
} as Language;
}Upon connection, we use our `did` to represent our identity and our requests. We also use the `linkLanguageUUID`, which is auto-templated upon Neighbourhood generation to represent the Neighbourhood’s unique space. After establishing the socket connection, we initialize the `LinkSyncAdapter` interface, the primary medium for data communication, handling tasks like link syncing and commits.
Lets take a look at that constructor:
constructor(
context: LanguageContext,
uid: String,
socketClient: Socket<ServerToClientEvents, ClientToServerEvents>
) {
this.me = context.agent.did;
this.languageUid = uid;
this.socketClient = socketClient;
this.socketClient.on('connect', async () => {
console.log('Connected to the server');
try {
console.log("Trying to join room", this.languageUid);
this.socketClient.emit("join-room", this.languageUid);
console.log("Sent the join-room signal");
} catch (e) {
console.error("Error in socket connection: ", e);
}
});
}As depicted above, once connected, our Language requests the socket.io server to `join-room`, using its unique `languageUid`. This action signals the server that we’re online and ready to receive any events from other peers within the Neighbourhood.
The corresponding server code for `join-room`:
// Join a specific room (Subscribe to a unique ID)
socket.on("join-room", function (roomId) {
socket.join(roomId);
console.log(`Socket ${socket.id} joined room ${roomId}`);
});1. Committing Data: `commit`
Great! So we now have constructed a Language inside of ADAM and attached it to a given Neighbourhood. Now when an app calls `perspective.addLink()` we want to make a commit on behalf of the agent in the space! Lets take a look at how that functions.
The LinkAdapter exposes the `commit()` function, this is responsible for sending a set of Link mutations to the network in the shape of `PerspectiveDiff.` This is analogous to what we saw previously with `perspective.addLink()`, but ADAM’s internal mapping for that function.
Client-side:
async commit(diff: PerspectiveDiff): Promise<string> {
try {
const preppedDiff = {
additions: diff.additions,
removals: diff.removals,
linkLanguageUUID: this.languageUid,
did: this.me,
};
const signal = await this.emitCommit(preppedDiff);
if (signal.status === "Ok") {
// Update our local timestamp to match the server
this.myCurrentTime = signal.serverRecordTimestamp;
this.updateServerSyncState();
// Resolve the function with an empty string
return "";
} else {
throw new Error("Commit failed with non-Ok status");
}
} catch (e) {
console.error("PerspectiveDiffSync.commit(); got error", e);
throw e; // Propagate the error up
}
}
// Utility method to wrap the socketClient.emit in a Promise
private emitCommit(preppedDiff: any): Promise<any> {
return new Promise((resolve, reject) => {
this.socketClient.emit("commit", preppedDiff, (err, signal) => {
if (err) {
console.error("Error in commit call", err);
reject(err);
} else {
resolve(signal);
}
});
});
}
// Tell the server that we have updated our current timestamp so that the server can keep in sync with what we have seen
updateServerSyncState() {
if (this.myCurrentTime) {
this.socketClient.emit(
"update-sync-state",
{
did: this.me,
date: this.myCurrentTime,
linkLanguageUUID: this.languageUid
},
(err, signal) => {
if (err) {
console.error("Error in update-sync-state call", err);
}
}
);
}
}In addition to the basic commit logic, we’ve introduced the `updateServerSyncState()` & `signal.serverRecordTimestamp`. The purpose and functionality for this be discussed shortly.
Server-side:
The server, upon receiving the `commit` signal, stores these LinkMutation in its database and broadcasts to all agents in the neighbourhood. Finally it hits the callback provided by the client with a success or failure.
// Allows the client to save a commit to the server;
// and have that commit be signaled to all agents in the room
socket.on("commit", async ({ additions, removals, linkLanguageUUID, did }, cb) => {
let serverRecordTimestamp = new Date();
try {
//Save the LinkMutations in the database
const results = await Diff.create({
LinkLanguageUUID: linkLanguageUUID,
DID: did,
Diff: {
additions: JSON.stringify(additions),
removals: JSON.stringify(removals),
},
ServerRecordTimestamp: serverRecordTimestamp,
});
// Send a signal to all agents online in the link language with the commit data
io.to(linkLanguageUUID).emit("signal-emit", {
payload: {
additions,
removals,
},
serverRecordTimestamp,
});
// Notify the client of the successful update using the callback
cb(null, {
status: "Ok",
payload: {
additions,
removals,
},
serverRecordTimestamp,
});
} catch (error) {
console.error("Error updating diff records:", error);
// Notify the client of the error using the callback
cb(error, null);
}
});It’s worth highlighting the use of `io.to(linkLanguageUUID).emit()`. This command sends live signals to other agents using this linkLanguage, leveraging the connection established during the `join-room` event. This mechanism is crucial for real-time applications like chat apps where immediate message delivery is vital.
The Language’s handler for this event:
// Response from a given call to commit by us or any other agent
// contains all the data we need to update our local state and our recordTimestamp as held by the server
this.socketClient.on("signal-emit", async (signal) => {
try {
let serverRecordTimestamp = signal.serverRecordTimestamp;
//Check that the timestamp of this commit is not before our timestamp as understood by the server, this ensures we do not accidentaly emit signals that arrive during our sync process which would appear as a duplicate
if (!this.myCurrentTime || this.myCurrentTime < serverRecordTimestamp) {
// console.log("Returning that live signal to executor");
this.myCurrentTime = serverRecordTimestamp;
this.updateServerSyncState();
this.handleSignal(signal.payload);
}
} catch (e) {
console.error("PerspectiveDiffSync.signal-emit(); got error", e);
}
});2. Data Synchronization: `sync`
A key function in our LinkAdapter is `sync()`, designed to retrieve link mutations the current agent hasn’t seen — akin to executing a `git pull`. This function is executed by every agent within the Neighbourhood. As the ADAM runtime receives links via `sync()`, it updates the local data state for that agent’s Neighbourhood. However, this is an internal operation, distinct from the direct call to `perspective.queryLinks()` that we previously discussed.
Client-side:
/**
* Call sync on the server, which should fetch all the links we missed
* since the last start of the link language.
*/
async sync(): Promise<PerspectiveDiff> {
// Only allow sync to be called once since once we have sync'd
// once we will get future links via signal
if (!this.hasCalledSync) {
try {
this.socketClient.emit("sync", {
linkLanguageUUID: this.languageUid,
did: this.me,
}, (err, signal) => {
if (err) {
console.error("Error in sync call", err);
throw Error(err);
}
this.myCurrentTime = signal.serverRecordTimestamp;
this.updateServerSyncState();
this.hasCalledSync = true;
//Check that the live signal did not come before when we sync'd
if (signal.payload.additions.length > 0 || signal.payload.removals.length > 0) {
this.handleSignal(signal.payload);
}
// Emit an event saying that we are synced
this.syncStateChangeCallback(PerspectiveState.Synced);
});
} catch (e) {
console.error("PerspectiveDiffSync.sync(); got error", e);
}
}
//We return and empty mutation set here because we already emitted the data with this.handleSignal()
return new PerspectiveDiff();
}In the code above, we dispatch a `sync` event to the server, passing our `languageUid` & `did`. The server responds with all the Link additions & removals new to us, which are then returned to ADAM (and any connected apps) via `this.handleSignal()`.
Server-side:
The server listens for a `sync` signal from any given agent. Upon receipt, it retrieves all relevant links since the agent’s last known timestamp (or all links if no timestamp is provided) and communicates them back through the provided callback.
// Allows an agent to sync the links since the last timestamp
// where they received links from.
socket.on("sync", async ({ linkLanguageUUID, did, timestamp }, cb) => {
try {
// If timestamp is not provided, retrieve it from AgentSyncState.
if (!timestamp) {
const agentSyncStateResult = await AgentSyncState.findAll({
where: {
DID: did,
LinkLanguageUUID: linkLanguageUUID,
},
});
timestamp = agentSyncStateResult[0]?.Timestamp;
}
timestamp = 0;
// Retrieve records from Diff .
const results = await Diff.findAll({
where: {
LinkLanguageUUID: linkLanguageUUID,
ServerRecordTimestamp: {
[Sequelize.Op.gt]: timestamp,
},
},
order: [["ServerRecordTimestamp", "DESC"]],
});
const value = {
additions: [],
removals: [],
};
for (const result of results) {
value.additions.push(…JSON.parse(result.Diff.additions));
value.removals.push(…JSON.parse(result.Diff.removals));
}
//Set the server timestamp we will return to the latest entry
let serverRecordTimestamp;
if (results.length > 0) {
serverRecordTimestamp = results[0]?.ServerRecordTimestamp;
} else {
serverRecordTimestamp = new Date();
}
cb(null, {
status: "Ok",
payload: value,
serverRecordTimestamp,
});
} catch (error) {
console.error("Error on sync:", error);
cb(error, null);
}
});The `commit` & `sync` operations communicate a `serverRecordTimestamp` back and forth, updating the server’s `AgentSyncState` with `this.updateServerSyncState()` within the Language. This mechanism is vital for synchronization in a centralized environment, enabling us to track the data an agent has viewed.
Sync & Ordering Strategy Across Network Topologies
The server, acting as a “super agent,” maintains all data and is seen as a data integrity source. It possesses the complete network state and is aware of the data each agent has seen based on their `AgentSyncState` timestamp. This mechanism ensures the server dispatches only updates since the last agent sync or addresses any potential conflicts. Moreover, this centralized approach can be employed to validate incoming data, ensuring events are executed in the correct order.
Contrastingly, in P2P systems, there’s no objective event ordering. Data is simultaneously authored and stored in various places. In our Holochain-based language, we’ve architected a solution where the network consistently converges programmatically on an event order. An upcoming blog post will delve into this.
Blockchains solve the event ordering challenge using a hard hashing puzzle (SHA). Since it’s improbable for multiple peers to solve this puzzle simultaneously, the first peer to solve it effectively determines the order of events. This method ensures consistent event ordering across a distributed system. This hash puzzle in blockchains, is the reason why blockchains are expensive to maintain and run. They are not suited for all use cases which is why we opt to allow developers and users to choose their own systems based on their needs. ADAM sits above any network architecture and gives you directly the data you care about.
Closing
We’ve presented some of ADAM’s core structures through its Language interface. These structures empower users to integrate diverse technologies into a unified layer. If you’re keen on developing your own Language or overlaying it on an existing or envisioned service, join us on Discord. We’re eager to collaborate with partners to develop integrations with various services. If you’re interested in Hypercore, GunDB, or related systems, don’t hesitate to get in touch!
