The submarine/tank/helicopter from Three Stooges in Orbit. Seven-year-old me thought this was a great idea.

XBlock Field Data Complexity

After years of working together, a fellow architect at edX once told me that he had figured out our core philosophical disagreement on API design. We actually agree more often than not, but when we disagree it’s usually because he’s willing to tolerate more implementation complexity if it yields a simpler interface for the API consumer. My bias is to keep the overall system simpler, even if it means placing a higher burden on the consumer (i.e. I’m in the New Jersey camp of Worse Is Better).

Given that outlook, it’s probably not surprising that I see XBlock Fields as a case where we have unnecessarily complicated the design with over-abstraction. Worse than that though, I think that it’s actually tainted our mental model of courseware over the years and pushed us further away from where we need to go as a platform. But let’s start at the beginning.

Field Basics

XBlock is the plugin mechanism and base class used to represent every type of content that students interact with in the Courseware tab of the Open edX LMS. Videos are XBlocks, multiple choice questions are XBlocks, the containing Sequential is an XBlock, etc. For more details, see Part 1 of the XBlock Lessons series.

Fields are how state is loaded and persisted for XBlocks. You declare them in a way that’s similar to how you’d declare the fields of a Django model:

They are also like Django models in that you can access the fields of any particular XBlock instance as if they were simple attributes on the object (e.g. my_block.question). The fields also have type information, making it easier to generate UIs for editing them.

Unlike the Django ORM, XBlock Fields have a concept of Scopes. Where a Django model typically maps to a table in a single database, a single XBlock can have fields of multiple Scopes, with each Scope mapping to a different table or a different database entirely.

The Many Kinds of Scoped Fields

The Scopes abstraction starts with the question of, “What is the matrix of all the different kinds of state associated with an XBlock?” It categorizes state with enumerations for users and enumerations for content, and then generates Scopes as a product of those enumerations. Those enumerations are as follows:

UserScope (User)

• NONE: State that is completely independent of any user. All LMS users can read from this scope but no user can write to this scope.
• ONE: State specific to a single user. Each LMS user sees their own state here and can read and write that state independently of other users.
• ALL: State shared across all users. All LMS users can read and write to this shared scope.

BlockScope (Content)

• USAGE: State for a piece of content in the context of a course.
• DEFINITION: State for a piece of content independent of a course.
• TYPE: State associated with the particular content type (i.e. XBlock class), not an individual piece of content.
• ALL: State common to all content.

Named Scopes for Fields (UserScope, BlockScope)

The named products of these User and Content enumerations are:

• content = (UserScope.NONE, BlockScope.DEFINITION)
  This would hold something like the text of an assessment problem or a snippet of HTML. It is not specific to any user, or usage in a particular course.
• settings = (UserScope.NONE, BlockScope.USAGE)
  This scope was intended for state specific to a piece of content in the context of a given course. An example of this would be the due date for an assignment. In practice, this scope has often been abused over the years, with many XModules and XBlocks putting things here that should be content scoped.
• user_state = (UserScope.ONE, BlockScope.USAGE)
  This is the scope to use when you want to store state for a single user against a single piece of content in a course, such as their answer for a particular question.
• preferences = (UserScope.ONE, BlockScope.TYPE)
  This is state that the XBlock keeps about a single user across all instances of content. For example, the default playback speed for the VideoModule is stored using a field with this scope and would apply to all videos being watched by a given user.
• user_info = (UserScope.ONE, BlockScope.ALL)
  This is a bit of an oddball. The intention is that it represent a field that applies to the user across the entire platform, not even constrained to a single XBlock class. Plausible use cases would be to hold data like the user’s timezone. This scope is rarely used. The VideoModule uses it to track whether a user has access to YouTube, which could have been done using the preferences scope.
• user_state_summary = (UserScope.ALL, BlockScope.USAGE)
  This scope is shared across all users for a single piece of content. The canonical example of this is allowing users to respond to a poll. Concurrency support is poor though, so this scope needs to be used with caution — users can easily write over each other, making it impossible to guarantee accuracy when you have many users hitting it at once.

Of these scopes, the first three are by far the most common: content, settings, and user_state.

Forced Unity

There’s a sort of mathematical appeal to having this state matrix, but it feels forced. We only use six of the twelve possible permutations of UserScope and BlockScope, since some combinations would be nonsensical. UserScope.ALL and BlockScope.ALL are each only ever used once. The user_info scope’s value is debatable, and user_state_summary is not an interface we can make any guarantees about at scale.

It gets worse when we examine the operational aspects of scopes in more detail. We quickly see that these scopes represent concepts with dramatically different lifecycles and data access patterns:

• Content is created by an author in Studio. It is persisted in MongoDB through the Modulestore interface, and treated as read-only by the LMS.
• Settings represent policy that is set by the people administering the course in the LMS. Examples of this are due dates and grading policy. In the case of Custom Courses for edX (CCX), these people are not the same as the people authoring content in Studio.
• User State is heavily read-write, and is persisted through the Django ORM to a SQL database. Rendering a piece of courseware typically requires fetching multiple rows (sometimes dozens of rows). XBlock handler calls will typically update a single row in response to user action. This generates by far the most state data and highest concurrency in the system.
• Preferences is a mostly read operation with small bits of data loaded on a per-user basis.
• User Info is meant to encapsulate small bits of data about a user that any XBlock might want access to, such as a user’s name or email address.
• User State Summary is unique among the scopes in that it has a high degree of write contention across users. It is the only named scope with UserScope.ALL in it. This scope has been used to implement voting, messaging, and collaboration (the latter two via a shared file system field). The only reason it isn’t a major performance issue in practice is that it is so infrequently used.

Why Does it Matter?

So we united a number of different concepts into an unnecessary abstraction. How does it actually hurt us?

Lifecycle Complexity Leaks out Eventually

Encapsulation is a good thing, and it’s natural to use objects to hide implementation details from API consumers. But there’s a difference between hiding how two things work and hiding the fact that two things are actually completely different from each other in terms of ownership, usage, and lifecycle. Those kinds of differences inevitably leak out of our “they’re just attributes” facade.

The LMS generally cannot write to the content or settings scopes. Any attempt to do so will just throw an error. Studio can make use of the user_state scope in a limited manner for previews, but it’s not tied to the actual user_state storage of the LMS. The Ironwood release of Open edX introduced a new public_view handler that allows anonymous access to XBlocks in the LMS, but developers have to remember that user_state fields will only ever return their default values and any attempts to write to them will simply get lost.

So even if Fields allow us to make an XBlock superficially resemble a simple Python object with attributes, XBlock developers still need to deal with the details of where the data lives and what can be done with it in any given context. We’ve made the code look simpler in many common cases, but we’ve also made it much harder to reason about. Many XBlock developers have stumbled over this issue as they’re starting out.

Fields are Difficult to Query

Something that I didn’t appreciate in those early days was that settings scoped fields aren’t just a slightly different flavor of content fields — they actually represent windows into entirely different systems. A piece of content will generally be accessed or written to either on its own or in the context of the pieces of content that are immediately adjacent. The title may be extracted for some summary view, but it’s still conceptually a key-value store where the value is something like the text of a homework problem.

With settings scoped fields, the data access patterns are potentially much more sophisticated. Take start and due dates for example. Ideally, we would be able to support:

• Queries that cross course boundaries so you can ask, “What assignments are due for this user this week?”
• Dynamically setting deadlines to set goals and pacing for an individual user or cohort of users who are starting a self-paced course.
• Individual due date extensions so that the team running the course can accommodate students who were out sick or had other extenuating circumstances.

Those kinds of operations imply a rich, relational data model behind an actual Scheduling API, and not just a simple key/value in a generic field storage system. The individual XBlock doesn’t need to make all these queries itself, but the fact that Scheduling needs to support these operations implies that the XBlock doesn’t own this data, but instead just queries it in a limited way.

That sort of separation does not exist today. For most practical purposes, there’s not really a difference between how XBlock authors treat the content and settings scopes. There are minor performance-related representational differences, but they’re persisted together in the same database in a key-value manner and have the same lifecycle.

Early Abstractions Guide Later Decisions

CCX is a less well known Open edX feature that allows course instructors to appoint coaches that can then teach their private versions of a course. These courses are limited in that additions and edits to course content are not permitted (there is no access to Studio). However, CCX Coaches can make changes that affect scheduling and grading policy in order to adapt the content to their students’ needs. A common use case is a small in-person class that covers mostly the same material, but possibly at a slower pace and on a different holiday schedule.

In a world where Grading and Scheduling are separate systems with their own data model, implementing CCX should be a relatively straightforward extension of those services. But instead, the people designing CCX were given a starting point in which all the relevant information for both grading and scheduling sat in simple fields on individual XBlocks. This eventually led us to create the Field Override mechanism, which allows plugins to intercept and alter arbitrary Field data, either on a per-course or per-user basis.

Let’s take the due date as an example. Due dates in Open edX are tracked by a settings scoped field named due. When the ModuleStore detects that you’re asking for field data in a CCX course, it pulls the root course’s content and settings scoped data from MongoDB and applies any CCX-specific overrides from CustomCoursesForEdxOverrideProvider (which stores the override data in a Django model). Your XBlock doesn’t know that any of this is happening under the covers — it just sees the overridden value for due date.

Field overrides would later be used to implement Individual Due Date Extensions (IDDE), allowing due date overrides on a per-student basis. That certainly sounds reasonable. But remember that bit where it would be nice to see your schedule across courses?

The Field Override mechanism takes us further away from cross-course scheduling queries because it stores schedule-related information below the XBlock field data interface. If we were building a Scheduling system today, what information is safe to cache across students and what information is user-specific? How do we know when or where a plugin like IDDE has munged with the data on a per-user basis? Furthermore, field overrides are stored in a simple key-value fashion — there’s no way to ask “what are all the upcoming due date extensions for this student?”. Instead, you can only query “Does this student have a due date extension for this one particular XBlock in this one course?”. We have taken complex data that belongs in a separate service, broken it into tiny bits, mixed it with data that belongs in still other services, and injected it all behind the opaque interface that is XBlock Fields.

Now we could retrofit a new Scheduling service into XBlocks and make the due property do an implicit call into this service rather than a simple property access. That’s the direction that edx-when is taking, and is probably the only practical way to evolve the system while preserving backwards compatibility. But that introduces even more magic and complexity behind our “it’s just an object with attributes” facade.

To be clear, I believe that Field Overrides were the right call for CCX and IDDE given the state of the platform at the time and the requirements. It was a pragmatic extension to the existing field-centric design of Open edX, it helped to encapsulate the complexity of CCX, and it provided a lot of functionality for relatively little code. But I also believe that it took us a step further away from where we eventually want to go as a platform.

Field Inheritance Becomes Even More Confusing

In the early days, all course content was written in XML files. Due date information is necessary for every leaf-node problem XBlock to see, since they have to disable certain features when the due date passes (though really, problems don’t need the exact dates so much as the states those dates imply). But it’s annoying to have to type in things like the due date separately for every problem when what you really want to do is set the due date for the entire assignment in one place. The right thing to do would probably have been to make Assignment a first class entity and made an explicit API boundary around what kind of Scheduling policy data the problems were fed from that entity. The quick thing to do in a field-centric, turtles-all-the-way-down view of the world is to just let the problems inherit field values from their containing Sequence.

Frankly, I think that allowing field inheritance at all was another mistake we made — one that actually predates scopes. But the intersection of field inheritance and scoped fields added even more confusion to the mix. For instance:

• The due field is an inherited, settings scoped field that is used by the grading system. What happens if my XBlock defines a due field with a user_state scope instead?
• Almost all inherited fields are settings scoped, but one is user_info scoped. But shouldn’t user_info scoped fields be available anywhere, without inheritance?
• Will a content or user_state scoped inherited field even work?
• Now that we have Field Overrides, what happens if we override an inherited field? Does the value get overridden for all child nodes of the container, or does inheritance only happen before the overrides are applied?

Not only is the actual behavior unclear, even the desired behavior is debatable in most of these cases.

Another consequence of relying on field inheritance is that each leaf node XBlock now has to worry about how various due dates and policies interact in order to translate it into how it should present itself. Should the input be disabled because the due date has passed? Is it okay to show answers after something is due? These are interactions of due date and policy that should be computed elsewhere, allowing leaf node XBlocks to get a simple enumeration of possible behavior states. Instead, every XBlock has to compute this for itself, leading to inconsistent behavior between XBlock types.

How Did We Get Here?

The universal answer for “why did we build something we regret?” is inevitably some combination of “we didn’t have time” and “we didn’t know better”. But what led us down this specific path? I have a few guesses.

Object Oriented Design Encourages Physical Mappings

I’m not qualified to write one of those trendy articles about how object oriented design is horrible and [insert paradigm] is the future, but one thing I have observed is that it tends to encourage unhelpful analogs to the physical world. An object in programming can be any sort of data and logic bundled together, but we instinctively map them to physical objects that we’re used to manipulating, without adequate regard for how we’re going to query them.

One of the classic intro examples in object oriented programming texts is a Car. Maybe they’ll talk about subclasses like Truck and Van. Add a few attributes and a drive() method, and the example is complete. The resulting Car class might actually be fine for many different kinds of programs, but it might also be rubbish for a city traffic simulator with a million vehicles running through various intersections because there’s no efficient way to manipulate the positions of all those cars concurrently.

In the same way, we can imagine an individual XBlock instance as that one problem we’re looking at on the page. It has a due date, it has a grading policy, and a host of other attributes. Instead decomposing all that into different systems in our heads, the real-world interface seems reasonable enough at first glance, and we draw our object boundaries accordingly.

Open edX Started With One Course

A major reason why we didn’t run into the cross-course use cases early on was that Open edX started as a prototype for only one course. Much of that first summer was spent to enable support for multiple courses and to untangle code and dependencies that were specific to that first course. If you’ve ever wondered why MathJax is loaded just about everywhere, it’s because that first course was an introduction to circuits.

Even when the next batch of courses went online, they were large courses and it wasn’t realistic for most people to be taking more than one or two of them at a time. It wasn’t that we didn’t see a need for cross-course schedules, but it was a low priority when compared to the frantic rush to get Studio built. By the time better scheduling functionality bubbled up the list of priorities, so much had been built around fields that the development cost was prohibitively high relative to other features being considered.

We Thought About Interfaces Instead of Data Lifecycles

There’s a common sentiment in object oriented programming that interfaces are the paramount thing, and that data is a pesky detail that can be fixed up later. That it’s the responsibility of the object to actually hide the lifecycle of the data entirely, so that the program can pretend that it’s “just there”. If you have this mindset and stare at a set of requirements long enough, you inevitably hear this siren voice in your head that whispers, “You know, these are all just special cases of this more abstract concept…” It feels elegant to find and factor out all the commonalities until you’re left with some tiny base primitives. We started to ignore the reality of the data in order to focus on the power and simplicity of the interface, which eventually led us to create an interface that was simpler than the data actually was — by essentially ignoring all the critical differences in lifecycles, query patterns, and performance.

We Wanted the Universal, Pluggable Open edX Framework

There was a lot of disagreement on this internally, but at least one early view of XBlock was that it should be the cornerstone for pretty much everything you interact with in Open edX — not just courseware, but pretty much everything a user sees in the LMS or Studio. The end goal of that vision was to have as dynamic and pluggable a system as possible, but that view pushes a design to be very generic and hinders optimization for concrete use cases.

What’s the Alternative?

I actually believe the Field abstraction is helpful for what is today the content scope, since it provides serialization and help text that makes building a Studio-like UI simpler. All other scoped data belongs in other systems (and the concept of “scopes” therefore goes away altogether). If I had it to do all over again:

• content remains the only thing using the Field abstraction, and there is no concept of scopes. Content is accessed via an ORM-like interface that is separate from the XBlock views/controller object.
• settings becomes a set of services for things like Scheduling and Grading, each controlling their own data. XBlocks get a read-only interface to this data via explicit methods (e.g. get_due_date()). We are already moving in this direction for scheduling.
• user_state becomes an explicit method call and returns JSON data that the XBlock author needs to handle. User submitted answers should also be tracked via a separate service, as that has special auditing requirements (e.g. tracking which submissions got which scores and when).
• preferences likewise becomes a method call to a service.
• user_info is replaced by a more robust User runtime service (one exists today, but could use work).
• user_state_summary is interesting because we’ve never made it particularly robust. A version of this that actually had to deal with concurrency might see a severe limiting of the interface — e.g. instead of being able to arbitrarily read/write, you might only have an increment/decrement operation or a set-add operation (it’s often used for voting). Yet given the kind of specialized cross-user information sharing that needs to happen in this scope, it’s quite possible that the best answer here is to simply provide nothing and leave it to each XBlock to implement in their own database models. Complex XBlocks like Open Response Assessment take this approach.

That being said, we can’t responsibly just start all over again. We have thousands of courses of data on edx.org alone, with many more in the Open edX community. But the split between Studio and the LMS gives us a viable migration path. For the moment, we can leave Studio and the course run OLX it imports and exports alone, and focus on translating that OLX into more performant data models when publishes are pushed to the LMS. It’s not quite as clean, but it gives us the power we need and allows for a gradual transition.