Hands on with a Solid Web App
Background information
In 1989, Sir Tim Berners Lee invented the World Wide Web. A year later, he released the first browser. At the time, it was a means for researchers at CERN and other research institutions to share information. It rapidly exploded in the following years and now dominates the world economy. But the web we use now is not what Sir Berners Lee had envisioned. While there is widespread sharing of information, as was the original intention, it is controlled by a few big tech companies acting as gatekeepers. They make profit by selling the user’s data. The users do not have control of their data, having limited ability to retrieve, delete or control who can access it. Legislation, such as GDPR in the EU, has been slow to adapt and often falls short of resolving the fundamental issues at hand.
Today, Sir Berners Lee is leading the charge to push the web back towards the vision he had had at the beginning. Giving the users the control without losing the power of the apps which have been developed in the past decades to improve our lives.
How to fix it
The key step is to invert the idea of where data is kept. Currently, most web services have their own databases which store all the relevant information about you. The key idea of Solid is to put all that data into a single “Pod” which all apps can access (given appropriate permission). This pod would be under the user’s direct control so that they can control the permissions as they wish. If they come to disagree with a company’s policies, they can immediately cut off access, without that company’s involvement. If the user moves to a new house, or changes phone number, they have only a single location to change it, rather than having to remember which of the dozens of services need to be updated.
With application data being stored in the user’s control, rather than the apps’, this would also allow alternative apps to work with the same dataset — preventing vendor lock-in.
An added benefit is that since most app backends are there to provide an interface to the backend and control access rights, which many apps can do entirely without one, as these two functions are replaced by the pod provider.
How to implement this theory
If this vision comes to fruition, it could completely change the web. It would significantly undermine some of the largest, most influential companies in the world and give users back the ability to control their data. As one of the key issues of our time, this was a prime candidate for a project in Motius’ Discovery Conference 2021. The platform provided at the Discovery Conference allows Motius to experiment with emerging technologies while staying at the far front of new tech developments. For our latest Discovery Camp, I and a few of my colleagues undertook the challenge to explore this technology hands-on.
To execute this, we at Motius decided to create a suite of apps, which would share data. The app itself was not really too important, as its purpose was simply to provide context. My app was a homebrew beer recipe manager. For four days, we worked on our apps, comparing notes daily.
Our experience with developing Solid
Everyone in our Discovery group was quite technical. Our specializations are mixed, but we all know our way around web development quite well. But working with linked data — as is used in the Solid protocol — was a hurdle. We’re very familiar with relational databases (e.g. SQL), or document databases (e.g. mongoDB), and all the layers of abstraction built over the top of them to make working with them easier. Linked data is not yet as mature as these other technologies, and the tools are not yet as sophisticated as what we are accustomed to. So, our initial progress was slow.
To demonstrate, here is the same data, shown in both JSON and turtle (a linked data format used in Solid). First, the old school JSON:
{
“name”: “Stuart Heap”,
“email”: {
“work”: “fake.email@mail.com”
},
“address”: {
“country”: “Germany”,
“locality”: “Munich”,
“postal-code”: “xxxxx”,
“region”: “Bavaria”,
“street-address”: “123 Fake Street”
},
“organization-name”: “Motius GmbH”,
“role”: “Fullstack Developer”
}
And now, the turtle (some fields have been removed for simpler presentation):
@prefix : <#>.
@prefix pro: <./>.
@prefix n: <http://www.w3.org/2006/vcard/ns#>.
@prefix n0: <http://xmlns.com/foaf/0.1/>.
@prefix schem: <http://schema.org/>.pro:card a n0:PersonalProfileDocument; n0:maker :me; n0:primaryTopic :me.:id1614167342660 a n:Work; n:value <fake.email@mail.com>.:id1614174549144
n:country-name “Germany”;
n:locality “Munich”;
n:postal-code “xxxxx”;
n:region “Bavaria”;
n:street-address “123 Fake Street”.:me
a schem:Person, n0:Person;
n:fn “Stuart Heap”;
n:hasEmail :id1614167342660;
n:organization-name “Motius GmbH”;
n:role “Fullstack Developer”.
Now let’s demonstrate how one would get the user’s name in each example.
Traditional:
const getName = async () => {
const user = await getUser()
return user.name
}
Solid:
const getName = async () => {
const dataset = await getSolidDataset(webId)
if (dataset) {
const profile = getThing(dataset, webId)
if (profile) {
return getStringNoLocale(profile, VCARD.fn) || ‘Unknown’
}
}
return ‘Unknown’
}
The increased line count is one thing, but not really the key point. It is pretty clear what user.name means. The developer knows immediately what is going on and can move on without thinking. However, getThing says basically nothing. Indeed, the Thing it returns could represent any form of data. In our case, where we wrote the data ourselves, there will be a Person stored, and that that person has a VCARD.fn (formatted name) value stored as a string. That however is not necessarily true. With a traditional architecture, with a specific database, I can ensure that every user will have a name, and that name will be a string. The frontend code does not need to be careful (that’s not to say exception handling isn’t needed, but let’s not get off topic here). In Solid, the user owns the data. And if there’s one thing which is true in the life of a web developer, it’s that users will do weird things. You can’t assume anything about the data, and so a Solid app will need much more care.
Given well written vocabularies (essentially schemas for linked data), it is imaginable that we could have tools which take a dataset and return an easily usable JSON object, or at least a similarly easy to use format. But during our short time with this technology, we found no such tools.
This is perhaps not a truly fair example. As it’s ignoring the fact that in most traditional apps, there would be a backend involved which would have to interface with the database to produce the JSON. One might argue that all we’ve done is move some of the backend complexity to the frontend, which would be a lateral move at worst. However, many modern databases simply store data in JSON (or something very much like it) already (e.g. MongoDB), and there are many tools which abstract this conversion away, and given a schema, will produce an interface to retrieve JSON (e.g. Prisma).
When the data gets more complicated, the added complexity here scales too. And it gets even worse when writing data to a pod. Here is the function from my beer recipe app for saving a new beer with a given name:
const addBeer = async (name: string) => {
if (beerDataset.value) {
const id = `#${uniqid()}`
const newBeer = createThing({
url: storageLocation.value + BEER_STORAGE + id,
})
const typedBeer = setUrl(
newBeer,
RDF.type,
BEER_RECIPE.BeerRecipe.value,
)
const namedBeer = setStringNoLocale(
typedBeer,
BEER_RECIPE.name.value,
name,
)
const ownedBeer = setUrl(
namedBeer,
BEER_RECIPE.writtenBy.value,
webId.value,
)
beers.value.push(ownedBeer)
beerDataset.value = setThing(beerDataset.value, ownedBeer)
saveSolidDatasetAt(
storageLocation.value + BEER_STORAGE,
beerDataset.value,
{ fetch: session.value.fetch },
)
}
}
For some context — it’s a Vue app, meaning there is data stored on the component in refs, hence all the .values you see littered around.
First we have to create a Thing. At this point it’s a completely abstract thing, the only distinguishing feature is where it will be located. We then start assigning properties to it one at a time. We say that it is of type BeerRecipe (a custom type published in my pod), and give it a name, id, and an author (each property needing its own function call). We then set that Thing in the dataset, and save it back to the pod.
For context, if we were using a traditional backend which accepted HTTP requests, the equivalent might look something like this:
fetch(‘/beers’, {
method: ‘POST’,
headers: {
‘Content-Type’: ‘application/json’,
},
body: {
name,
author: user.id,
},
})
Again, the argument could be made that the traditional example is hiding backend complexity. This is true, but we’re also skipping over writing the vocabulary for the beer recipe data. Which with my current understanding is a much more difficult task than handling the request on the backend and storing it in SQL, even if I were to forgo all the tools to make that process easier.
The bottom line here is that the fundamental CRUD operations are significantly more cumbersome to write when using the current Solid libraries, when compared to what we’re used to. If we want to produce features as quickly as we’re used to, with a nice developer experience, we’re going to need to improve the tooling. But that’s really no surprise: people have been iteratively developing the tooling to work with traditional technologies for decades. Solid has a lot of catching up to do.
The deeper challenge
The above problems are simply down to immature tooling. That situation improving is simply a function of time and developer interest. But there are some other technical issues which are inherent in the design.
For better demonstration, let us imagine that my beer brewing app was further developed. Users will want to be able to select a given hops variety to include in their recipe. There are potentially hundreds, and the user probably has an idea what they’re looking for. A natural interface element to achieve this would be an autocomplete field. They would start typing the name, and see the choice they want, which they would select. With a traditional architecture, I would query the backend with the typed input, and retrieve any matching hops varieties to narrow down their search. At no point do I need to retrieve the full list of hops.
With Solid however, we could find no method for retrieving part of a dataset. It’s all or nothing. You could achieve the same UI easily enough, but it cannot be asynchronous. The full dataset must be retrieved. This slows down the app, and increases data traffic significantly. In our use-case of a list of hops, this would only be hundreds of entries long at most, so it would be acceptable. If you have a bigger dataset, this problem could become a deal breaker.
So far, we’ve found no discussion of this issue, although one would hope that it’s being addressed by someone. It’s not an unsolvable problem — we’ve discussed some possible solutions internally, although they all complicate retrieval or interoperability of data — but whatever solution is reached needs to become a general pattern used by all apps. The lack of visible discussion is a concern.
Other possible issues to solve
Here are some more issues we encountered in this project:
- The frontend of the official pod server (the default way to manage your data and permissions) has pretty poor UI/UX. For example, if you try to add a custom permission for a given file/folder, it immediately applies the default permission rule, of “nobody has access, including me”, locking it as unreadable and unwritable by anyone forever.
- Changes are not logged anywhere, meaning it is not possible to account for which apps are changing which data.
- The official pod server’s auth portal defaults to giving global read/write access, with vague instructions on how to give limited access should you want to. However, given the above UI issues, that’s easier said than done. The end result will be users will just give up and give full access to everything, undermining the key point of Solid.
- Sharing vocabularies is not all that is necessary to achieve interoperability. We were unable to achieve real interoperability of data between our separate apps, beyond basic user card information.
But I don’t want to come across as too negative. These complaints feel a bit like suggesting jet engines to the Wright Brothers. I’m sure many of these problems are just down to this being in early development, and not yet ready for mass consumption. A developer making an app doesn’t need to care that they can’t lock down the permissions of the app they themselves are making.
Also, we were only working with this technology for a few days. We’ve been working with traditional technologies for years. We’re bound to have misunderstood some things and gotten bits wrong.
What could the future look like?
Solid is clearly designed with the end user in mind. And the benefits for the end user are necessary and valuable. From the developer’s side, there are apparent tradeoffs, but it is a community task to build the tools to make working with this technology easier and more accessible. With these tools in place, I think working with Solid will end up being easier than with a traditional backend. But for the time being, it makes things harder.
However, for the companies building apps, and hoping to make money out of them (ultimately, where most apps come from), I think there are more downsides than upsides. I think this is going to make widespread adoption of this technology an uphill battle. The entrenched position of established companies, especially with them already owning much of their users’ data, and holding it hostage, makes undercutting them a difficult proposition — even if the deal for the user is better. There is little to no incentive for companies to give up the huge economic benefit granted to them from having unrestricted access to user data.
If you found this in depth article about Solid web apps interesting, then go and check out the recorded clip of the live presentation from our Discovery Conference!
This post was originally published on our website www.motius.de
Do you want to build the products of the future? Come talk to us!
Thanks to Stuart Heap for his input on this project. Check out more of his articles!
