A Brief Tour of Single-Level Memory (SLM)

Functional Programming without Borders

Please look here for background on CodeBuilder technology as a programming model for persistent Single Level Memory (also related to Single-Level Store or SLS)

Stop the Programming Language Madness!

The fundamental motivation behind using a CodeBuilder VM is to make programming more productive by moving developer activity from outside to inside the machine. This allows us to delegate control-flow and other related artifacts to the underlying system and hopefully free us from forever chasing the TIOBE index for the latest language fad. Hence, JavaScript should be sufficient for demonstration purposes here and I am going to base this discussion on our Inception reference implementation.

Since we move beyond the half-century old concept of “source files”, the reader probably wonders (1) how we organize program elements and (2) how we navigate inside persistent program memory.

Back to the future?

Single-Level Store

A key part of understanding how to leverage Persistent Memory is understanding Single-Level Store (or Single-Level Memory) — a Multics concept that has since been adopted by IBM and is considered by some to be their “crown jewel”.

Essentially, memory is virtualized and extended across systems, hardware etc. to provide a uniform programming surface. The trick is arranging things in a topology (usually a graph) so the operating system can understand where to find things. They can also serve as threads, vectors or table rows, holders to expose monad “plumbing voodoo”, or “rule boards” in the case of AI. Unlike the usual concept of “scope”, contexts can persist independently of control flow or function execution and are essentially virtual objects (like Go routines without having to use Go).

Once the developer has “logged” into one of these contexts, the REPL experience is similar to a UNIX shell (as we would expect given the UNIX/Multics lineage). The directory listing command “ls” gives us a list of contents:

> ls
No children found

This shows we are in an “empty” running context. Now, let’s create some stuff:

> x = 6
> test = {}
> foo = function() {}

Using ordinary JS syntax, we just established a variable x, an object test and a function foo. Let’s verify:

> ls
3 children found:
test {0}
foo()
x: 6

This is essentially what we would expect if we were running inside a Node REPL or debugger session but the “ls” command illustrates how Multics/UNIX was originally supposed to work.

Going Behind the Matrix

A single-level memory (SLM) system can map to a virtual filesystem. As we see below, the CodeBuilder automatically maps our memory to a context (see tioga in this case) and renders the following: (1) a folder called test and (2) a file called foo.js that contains the JS implementation of function foo and (3) a translation of variable x to some sort of JSON.

JS memory as seen through a filesystem surface

So we are free to edit foo.js using an ordinary text editor (e.g. Atom) as indicated above and the definition of foo is automatically updated (hot-loaded) in memory whenever we save. Note this is where late-binding references play a key role, as a program would normally need to restart to see the newer version of foo.

A new way to view “Functional Programming”

A key role of the CodeBuilder is to insert a level of indirection in function invocation, which avoids hard-wiring control flow paths. Note the implementation of foo takes a parameter and doubles it. Hence, we get the following expected results:

> foo(23)
46
> foo(x)
12

Not terribly interesting, but let’s go back to the empty object test we created earlier and put some data inside it:

> test.x = 5
> test.y = 9
> ls test
2 children found:
x: 5
y: 9

Now treat test as a vector and apply it to foo:

> test | foo | test.z
> cd test
> ls
3 children found:
x: 5
y: 9
z: 10

Note the pipe symbol (“|”) above is reserved CodeBuilder syntax (not JS or any other DSL) but you can immediately see the UNIX/Multics influence here. This tells the CodeBuilder to set up whatever messaging/ streams/ concurrency necessary to bind test.x to the function foo parameter named x and put the results in test.z. We then go into test (which is not really a folder but rather a form of network path navigation) and verify that z has been created and is x*2=5*2=10.

For years, others have noted how UNIX-style pipes are a much simpler way to express flow-based programming (used heavily in AI). However they also imply a certain amount of system infrastructure (such as async coordination) and thus help illustrate the weakness of language-oriented approaches. In particular, the context can determine how the pipe symbol is to be interpreted by the system (for example sync versus async) without having the programmer having to hardcode it. The codebuilder then has the freedom to select the appropriate underlying queuing or messaging mechanism based on performance etc. This is a good example of declarative versus imperative design — even with a normally imperative language like JavaScript that does not support native FBP.

Set Handling

Many CodeBuilders automatically support external database access (albeit some better than others). Using Inception, suppose we connect to a SQL database and query a table tab1 (assuming it contains a column named x):

> select x from tab1
34
12
3

Note the CodeBuilder automatically infers SQL syntax over JS (where practical) because we try to avoid embedded/quoted strings and treat all DSLs as peers. Avoiding subjugation of one DSL over another is a hallmark of SLM and why trying to pick one programming language over another might be the wrong approach.

We apply JS function foo to each row (for column x) and place the result in a new array:

> select * from tab1 | foo | result1
> ls result1
3 children found:
[0] 68
[1] 24
[3] 6

Again the “functional programming” sits above the individual DSLs. Implied set handling is another example of clarity of intent you would expect in an SLM system and also shows why a CodeBuilder is an interesting fresh approach to FP.

Finally, the SLM mapping means that result1 is automatically rendered in the filesystem (in this case as a CSV file):

JS array automatically rendered as CSV

How’s that for improved coding productivity?

The reader probably wonders why SAP, Oracle or Postgres never went down this road given previous efforts to embed Java or JS into the database (perhaps they will with in-memory databases) but SQL should not be the primary DSL any more than JS or Java.

Perhaps Node itself will someday evolve into a CodeBuilder. Elixir seems to be on the right track (particularly how its code can be manipulated as AST tuples) depending on where BEAM goes versus V8 etc. But ideally CodeBuilders should be language-agnostic so programmers don’t have to worry about betting on the right language. This is why CodeBuilders also make for a compelling integration mechanism. Basically, we are seeing the evolution of another layer atop the operating system. The most interesting work I see is in the decentralization space.

Conclusion

This only scratches the surface but I hope this little walkthrough helps get your mind thinking about SLM as an interesting programming paradigm. It is certainly time to revisit prior work on UNIX/Multics and create more programmer-friendly operating systems.