TL;DR: An edition release doesn’t mean that the work on that edition is over. In 2019, we should continue to pursue and drive home the goals set in the spirit of productivity. That being said, I’m going to argue that for the upcoming years and the next edition, we should pursue software composability as a new overarching theme and go over some thoughts on what that means in a concrete way.

On editions and themes and the following year

First of all, let me start with an observation about the themes of the past and the current editions. The overarching theme of Rust 2015 was stability. The work towards stabilising the language started in 2014 and was publicly announced in the blog post Road to Rust 1.0. A huge amount of work was coordinated and performed for the effort that culminated in releasing Rust 1.0 in June, 2015.

However, the release, although a culmination point, didn’t mean that the job was done. For months and even years after the big release, the community continued the effort of polishing and stabilising yet unstable parts of the standard library and the community-maintained libraries release by release. One could say that the Libz Blitz initiative in 2017 was still made in the spirit of stability!

Now, in 2017 roadmap it was announced that productivity was going to be the theme of that year, and in 2018 it was solidified as a the overarching theme of the Rust 2018 edition. The effort finally culminated in a big release a few days ago.

Does that mean that the work on productivity is done? I hope the reader is going to agree with me that the answer is clearly no! I think we are seeing a recurring pattern what the edition releases are about: a flurry of preparation, design and implementation work intensifying and finally building up to a release[1], followed by a phase of consolidating work and driving smaller, supporting goals into completion.

Moreover, although there are clear-cut releases, there is no hard cutover for overarching themes. They just express the spirit of the community and give general direction as we are moving forward with the Rust project and the ecosystem — and I think we can be sure that the theme of productivity continues to define the direction we are moving to during the following year.

This was a long-winded way of stating the first point I want to make about Rust in 2019: I think we should continue to pursue the goals we set in the spirit of productivity. To mention some specifics: I think that finishing the the support for asynchronous Rust is incredibly important. Another very important goal we should focus on is improving the compiler; the query-based compilation model and refined incremental compilation are going to be a huge boon for productivity, as they enable faster builds and better support for IDE-like use cases. Completing the refactorings of trait matching and borrow checking should also allow doing some experimental work more easily. Finally, I’d also like to see the 1.0 releases of some important community-maintained libraries such as log, rand, tokio and hyper during the year 2019. All in all, I very much agree with Jonathan Turner in that I’d like the year 2019 to be a “fallow year” of maturing the ecosystem.

Now, that being said, let us talk about the future of Rust on a longer timeline.

Composability as a core value

I think Rust should consider setting composability as the overarching theme for the Rust 2021 edition and possibly upgrading it to a core value of the project. Rust already has almost all the qualities to make it the most composable language in existence — all that is missing is a concentrated community effort to realise those qualities into a polished vision. We’ll see next what I mean by composability and why it is desirable by going through some examples.

Examples from the composability zoo

Software reusability has always been a dream of many programmers. It’s not an easy thing to achieve; I’ve even heard some people calling it a pipe dream (usually in combination of trying to achieve it through some failed promises of object-oriented programming). However one can’t help but accept that it for sure is a desirable thing; reinventing the wheel every time certainly isn’t. I’m going to argue that composability — the ease of composing software from smaller building blocks — is a key property that makes code reuse possible. However, I don’t think there is a single feature that makes code composable. I think there is multiple excellent examples of some programming languages that are very composable in different ways; here I’m going to concentrate on C, Lua, Haskell and JavaScript.

C can be seen as the one of the most composable programming languages in existence, if you consider the vast amount of libraries written in C and the huge software stacks that consist of software written in C. There are two key features in C that make this possible: the lack of a runtime and ABI stability. The lack of a runtime means it doesn’t have much requirements for its runtime environment. This makes it possible for C to run almost anywhere — be it an embedded microcontroller, a mainframe or your smartphone. It also makes it possible to run C code in presence or in spite of a runtime, which makes C code easy to call from Python, Java, Ruby etc. The ABI stability makes also libraries written in C easy to package, distribute, wrap and reuse.

Lua is another example of being able to run almost anywhere. However, as a high-level language it has a totally different flavor; it’s an extremely simple language, and can be implemented as a C library. In this sense, it leapfrogs C’s ability to be run almost everywhere. However, there is another sense in which Lua is very composable: it is self-contained. Its runtime is a sandboxed context with a very clear, non-global state. If you want, you can easily run multiple different Lua instances and they don’t need to care about each others. Lua even makes possible running code in self-contained sandboxes inside its runtime by allowing creating “environments” to run code in. The code running inside an empty environment can’t access global state as it doesn’t have any reference to it. Lua makes it possible to run untrusted code and get away with it. This also enables the main usecase Lua is geared for: describing behaviour as scripts, and making it possible to create flexibly-behaving applications by composing the desired behaviour from those scripts.

There is a lot to love about Haskell. It has multiple features that make it extremely composable. I think the most notable one is the strict control of side-effects. It’s not a security feature and you can’t run untrusted code relying on just to the lack of side effects. However, it makes one confident that library code is self-contained and doesn’t do anything funny. All the side-effects are indicated and documented by the types in the function signatures. Another feature that makes Haskell composable is its generic polymorphism. It makes code composition flexible by not setting in stone the exact types that are needed for interacting with libraries, allowing very general and multi-purpose code to be written.

Finally, there is JavaScript. Most people wouldn’t think of JavaScript as an especially composable language, but I’d like to highlight a single feature that makes it easy to compose code in real life: a good package and dependency manager. Because of NPM, JavaScript library ecosystem is thriving.

Let’s return to our examples from a reverse perspective; there is some recent counterexamples that highlight the uncomposability of JavaScript. There was the eslint-scope hijacking incident and there was the left-pad incident. These were both cases that erode the concept of composing software from libraries. There are some problems with other languages too: Haskell has a runtime and garbage collection which makes it not always straightforward to call from other languages. This, combined with it’s lazyness make it also hard to build embedded software in Haskell. Lua, while having very strong story in embeddability, is as a scripting language not suitable for base systems implementation. It also does not have a strong “batteries included” library story as it is aimed for very specialised, application-specific use cases where a “catch-it-all” default set of libraries wouldn’t make sense. Finally, C, while being able to run almost everywhere, is ironically not easy to get to run almost anywere. This is because of the lack of polished dependency management and complicated, non-portable build systems. C also does nothing to ensure some properties of the code. It’s up to you to verify that it does what it claims to do, and if it segfaults, good luck with debugging nightmares. That’s not encouraging for code reuse!

As we can see, composability is not a single thing, and certainly it’s not just a technical language feature. Composability just means the ease of composing software from building blocks, and everything that makes the process of code composition easier and more robust, increases composability. Everything that makes the process harder or more unreliable decreases it. We can, for example, see that having strong standards for documentation increases composability, because it makes it easier and more accessible to understand each of the building blocks. Here, I find Rust’s story already quite good with the venerable websites like https://docs.rs/ and https://crates.io/.

Enter Rust

I’d like to argue that Rust is one of the most composable language there exists, and with a concentrated community effort it’s possible to polish it to be the composable language that is second to none in almost every aspect. We can already see that Rust shares similar qualities with many of our examples.

The lack of runtime is similar to C, and that makes it possible to run Rust code on embeddable devices and develop libraries that are easy to re-package and wrap for other languages to call.

It doesn’t have a sandboxed model like Lua, but it supports compiling to WebAssembly; I think that in the future we’ll see software even outside web browsers that implement plugins functionality by running sandboxed WASM.

Like Haskell, Rust also has a strong culture of not relying on global state and being explicit (in documentation) about side-effects which helps to assure that libraries you use are not going to interfere with each others (or with themselves, or with your code). This is further improved by recently stabilised support for deterministic const fns that hopefully continue to improve.

Also like Haskell, Rust has very flexible generics that makes writing generic libraries possible.

I’d love to see Rust further polish these aspects. I want Rust to be a language that runs anywhere, interoperates with anything, and gives you the peace of mind that it does what you expect it to. Here’s some ideas of features and tools we could spearhead to make Rust the most composable language in existence:

• More embedded targets
• Streamlined build process for embedded
• Easier and more complete story for no_std
• Complete and stabilise per-object allocators
• Make the WASM story even better
• Finish the work on generic associated types to allow more generic libraries and interfaces to be built
• Support projects that aim to make calling Rust code from other languages easy, such as Helix
• Create a strong culture for community code reviews to improve trust in the ecosystem libraries and support the development of tooling for that. Crev is an extremely promising project.
• Make builds generally more robust by sandboxing the build scripts and procedural macros (how awesome an use case for WASM would that be!)
• Make it easier to standardise build scripts by allowing them to depend on some community-maintained well-behaved libraries (RFC)
• Make separation between private and public dependencies clearer in Cargo, and make tooling for checking it automatically. (Edit: added this item on 2018–12–10)
• Support tooling for automatically detecting semver bumps; make Semverver a more integrated first-class tool. (Edit: added this item on 2018–12–10)
• Assess the problems with the orphan rules / coherence restrictions and try to solve the cental problems of “glue crates”. (Edit: added this item on 2018–12–10)
• Provide a versioned, stable ABI. It doesn’t have to include everything in Rust, but it should allow using some common types such as slices and trait objects (with possibly only some subset of the all features of trait objects) for stable FFI calls.
• Make lifetimes more composable through new typesystem features. I’ve highlighted some problems in my earlier post Things Rust doesn’t let you do. Especially the last three items 12–14 apply here.
• Make it possible to reason about side effects in a generic way, introducing a polymorphic effect-handler system.

I think that the most of the items are relatively uncontroversial. The last ones might raise some objections; there are some opinions about Rust being already having too much features or having already spent its complexity budget. However, I think that with good design and discretion, features such as these are only going to make code more comprehensible and easier to control.

In particular, I’d like to point out that at the moment, it’s hard to control the side effects of code that you call. In Rust the problem is generally not as bad as in some languages where there is a culture of writing very side-effect happy code. However, as const fns have been stabilised, we have already a way to restrict side-effects. While that is great, we now find ourselves on a head-on collision course with the problem of red and blue functions. What if we want to debug log from our const function? That’s a side effect. Should we make logging and log-less versions? At some point, we want to be polymorphic over side effects, and we want our users to decide how to handle them to improve composability. I don’t think it’s realistic nor desirable to do huge changes to the type system at short time scales, but the capability of controlling side effects would make sense in the long run.

All in all, I think we should seriously consider composability as a core value and overarching theme for the upcoming years and start sketching, planning and designing accordingly — as we have seen with the async support, these things take time.

Happy Rusting, and let’s make 2019 a great year for Rust!

Footnotes

[1] To be sure, the release model of Rust is to release often and do small, non-feature-based releases, but when it comes to edition releases, I think we can all accept that they feel “big”.

Contents:

About references and lifetimes

The design decisions to live by

The shortcomings with answers
1. Doing control flow aware stuff
2. Postponing mutability of a lifetime
3. Skipping trivial bounds in data types
4. Splitting up mutable references
5. Having multiple aliasing mutable references
6. Being able to point inside Cell types
7. Having self-referencing structs
8. Capturing only disjoint fields in closures
9. Having associated types that are generic over lifetimes
Addendum: Getting ownership over a mutable reference

Open problems: from here on there be dragons
10. Downgrading a mutable lifetime to a shared one
11. Calling mutable methods that don’t access overlapping fields
12. Hiding mutable lifetimes in data types
13. Using “ambient” lifetimes
14. Moving the owner of a heap-allocated object that has an inbound reference

Closing words

The borrow checker is undisputedly the weirdest and most novel feature of Rust the programming language. It’s what makes Rust the what it is — a memory safe language without a garbage collector that strives for zero-overhead abstractions. Rust manages its memory using compile-time static analysis: checking for dangling pointers, ensuring mutability constraints and inserting calls for freeing memory as a part of type-checking. This analysis is not always perfect and sometimes it requires jumping through the hoops to get a Rust program to compile. Understanding the lifetime and borrow system thoroughly gets you quite far, but there are still some cases you can’t convince the compiler to accept the code, even if you can convince yourself that it is indeed safe. In this article I’ll list such cases: limitations of the borrow checker, the reasons why it doesn’t let the code pass and how the situation could possibly improve in the future. I hope that the borrow checker keeps evolving and some day this list becomes redundant.

About references and lifetimes

This is not meant to be a tutorial for Rust but I’ll briefly introduce the reference system a bit for starters. You can skip this part if you are already familiar with the concepts. Rust has the concept of “owned” values. Owning a value means that you have the single right and responsibility to dispose of it once you are done with it. Because there is no shared ownership built into the language, the compiler always knows when you are done with a value, so it inserts the call to the destructor automatically.

Other than using a value directly, you can take a reference to it — this is called borrowing in Rust parlance. References are like pointers in C, but they are checked for correctness. The lifetime system in Rust ensures that a reference can’t outlive the value it points to. You can’t also construct bogus references such as null references; you can only take a reference of an existing object. References are thus, always valid.

There are two kinds of references: shared references that look like this: &MyType and mutable references that look like this &mut MyType. You can have many shared references to a value, but you can’t mutate the value they point to. (There are some exceptions though, elaborated later.) If you want to mutate values through references, you can do that using a mutable reference, but you can have only one of those at a time. As long as a mutable reference to a value exists, that value can only be accessed (read or written) through that reference. No one else— including the owner — are allowed to access the value while the mutable reference exists.

The design decisions to live by

There are some good reasons why the Rust reference system is so restrictive. First of all, having single ownership ensures that it’s not easy to leak values and it’s impossible to “double-free” them — calling the destructor twice. Rust ensures the single ownership principle by being move-by-default. That means that if you pass a value somewhere, you lose the ownership over it and can’t use it anymore. This is also called affine typing. (Often confused with linear typing; see here for further discussion: https://gankro.github.io/blah/linear-rust/) Rust also supports types that are copy-by-default; many fundamental types such as integers are defined such and you can define your own but move-by-default makes sense as a conservative default.

As for the mutable references, the principle of a single mutable reference may feel overly restrictive from the perspective of a C programmer who can have multiple mutable pointers to a value. However, there are valid reasons for that. For a convincing practical reason from software development perspective, see this blog post by Manish Goregaokar: https://manishearth.github.io/blog/2015/05/17/the-problem-with-shared-mutability/ .

Other than that, there is some additional reasons: one of them has to do with compiler optimizations around aliasing and another has to do with thread safety. To quickly explain the gist of these two: single mutability ensures that you can safely keep the value in processor register without fear that some other piece of code invalidates the version that resides in RAM — this can enable nice optimisation speedups. It also ensures that if you send a mutable reference to another thread, there aren’t any other mutable references left that could cause race conditions when writing through them in an unsynchronized manner from multiple threads. Mutable references in Rust are guaranteed to be unique, and it would cause undefined behaviour to somehow being able to clone one. Fortunately the type checker protects you from that.

There is still one general principle that affects the design decisions of Rust: all analysis should be local. No whole-program stuff. Rust programs must be type checkable function-by-function. The function signatures have to contain enough of lifetime and type information that the body can be checked. This also means that some situations where the borrow checker might seem stupid (“This function only mutates only field A, so why can’t I call also that function that mutates field B, they are different fields! It should allow that much!”), but the point is that it isn’t allowed to “peek into” functions other than the one it is currently checking. All it gets to know are the function signatures.

The shortcomings with answers

So, here is the meat of this article. I’d like to review some cases that are certainly safe, but for one reason or another, the borrow checker isn’t sophisticated enough to see that. I’ve ordered the cases by whether there exists an upcoming solution for the problem or if the problem is still unsolved. The first part consists of problems that are about to get fixed. That’s exciting, so let’s get started!

1. Doing control flow aware stuff

At the moment, the borrow checker thinks of the code as a bunch of hierarchically nested scopes, or blocks. The outer scopes outlive the inner ones and the borrow lifetimes behave accordingly. This is a very simple way to think of the borrows — but a rather unsophisticated one. It breaks down when the control flow doesn’t match the block structure. Here’s an example borrowed (ha!) from Niko Matsaki’s excellent introduction to the problem. (http://smallcultfollowing.com/babysteps/blog/2016/04/27/non-lexical-lifetimes-introduction/) As you can see, the borrow checker is being overly conservative; even in the branch None where value, derived from map, doesn’t exist, it considers map as borrowed:

fn process_or_default<K,V:Default>(map: &mut HashMap<K,V>,
                                   key: K) {
    match map.get_mut(&key) { // -------------+ 'lifetime
        Some(value) => process(value),     // |
        None => {                          // |
            map.insert(key, V::default()); // |
            //  ^~~~~~ ERROR.              // |
        }                                  // |
    } // <------------------------------------+
}

There’s some other juicy examples too in the linked blog post; highly recommended reading!

Remedy: Non-lexical lifetimes

There are ongoing efforts to land improvements to the borrow checker that allow it to reason about the borrows with finer granularity. The RFC describing the proposal in detail can be found here: https://github.com/rust-lang/rfcs/blob/master/text/2094-nll.md. The improved borrow checker is currently available on the beta version of the compiler and will be available stabilised on release 1.31, as a part of the new 2018 edition. Only some weeks to go! It’s not panacea, though; as mentioned before, it can only reason about things local to the current function, so however simple, no “intra-procedural analysis” is done.

2. Postponing mutability of a lifetime

An oft-recurring pattern:

items.mutate_n(items.len());

At glance, this looks fine — first the length of the container items is measured by the method len which takes a shared, immutable reference to items. After the value has returned, it is passed to the method mutate_n along with a mutable reference to items. No mutable and shared lifetimes are supposed to overlap. However, there’s a complication due to Rust’s evaluation order. Here’s a desugared version of the method calls:

let receiver_of_mutate_n = &mut items; // A mutable (unique) borrow!
let receiver_of_len = &items; // A shared borrow!
let result_of_len = Collection::len(receiver_of_len);
let result_of_mutate_n = Collection::mutate_n(
    receiver_of_mutate_n,
    result_of_len
);

As you can see, the receiver is resolved before the expressions inside the parentheses! This means that there exists shared references at the same time there exists a mutable reference, which isn’t allowed! Admittedly, if the nested call would mutate items there would be some fertile ground for nasty and hard to notice bugs, but in this case we’d want to allow this, as len is only a read-only method that can’t cause any harm.

Remedy: Enabling nested method calls

Granted, “postponing” the mutability of a lifetime in presence of nested method calls feels kind of a special case but it’s nice for ergonomics since it’s an often recurring pattern. There has been an approved RFC around this case (https://github.com/rust-lang/rfcs/pull/2025), and it too is going to land on 1.31, edition 2018!

3. Skipping trivial bounds in data types

When defining data types with generics and lifetimes, the compiler can be a bit pedantic:

struct MyGenericDataType<'a, T: 'a> {
    foo: &'a T,
}

See the <'a, T: 'a> part there? It’s needed. What we have here is a generic struct that holds a reference to any type T. First of all, Rust requires you to spell out the lifetime of the reference. Since it’s a data type that can be instantiated at any point of our program, there is no one and true lifetime that our struct will have — after all, it depends on the reference we store in there! That’s why the type is generic over lifetimes. < > is the Rust syntax for declaring generic types, and by specifying a lifetime annotation 'a there, we declare that the struct is valid for any lifetime the reference it contains is valid for. (Note that the type of field foo contains the lifetime 'a.) However, our struct is also generic over the actual type the reference points to! That’s what T refers to. It stands for “any type”.

Enter the pedantic part: we need one more annotation, T: 'a, which means that the type T outlives lifetime 'a. What does that mean? It means that the value the reference points to must live longer than the reference itself. Makes sense! If it wouldn’t, we would have a dangling pointer!

Except that this is totally trivial. Of course it has to live longer! Why do we have to spell that out? There’s no way that the pointer could soundly live longer than the pointee! It’s a model example of boilerplate code — it’s the only sensible choice and yet we have to spell it out.

Remedy: Inferring outlives requirements on structs

There is an accepted RFC (https://github.com/rust-lang/rfcs/pull/2093) that says that the trivial bounds such as introduced above can be elided. That would allow us to write just <'a, T> instead of the verbose <'a, T: 'a>. This feature, too, will land on 1.31, as a part of edition 2018.

4. Splitting up mutable references

A commonly expressed concern about Rust is that it’s hard to get from one mutable reference to many. If you have a HashMap of items you can get a mutable reference to a single item inside it, but you can’t get many! Why is that? The get_mut method of HashMap receives a mutable reference to the hash map itself: &mut self. The method then returns a reference to an item contained in the hash map: &mut Item. Because the item reference points inside the hash map, it must have the same or shorter lifetime as the &mut self reference — it can’t outlive that, because that would allow us to dispose of the hash map, and the item reference would become a dangling pointer. Since we are talking about mutable references here, that also means that we can’t have many of them! We can only call get_mut again after the lifetime of the last borrow has ended!

But, you say, obviously calling get_mut multiple times should be allowed here, since the call doesn’t actually do anything bad! And being able to get multiple mutable references out of a hash map sounds so elementary that of course it should be allowed. However, imagine what we could do without that limitation: we’d call get_mut to get a mutable reference to item A. Then we’d call get_mut to get another mutable reference to item A and find ourselves from the world of undefined behaviour.

There’s a similar, but even more obviously “stupid” limitation:

let mut array = [1, 2, 3, 4];

let ref_a = &mut array[1];
let ref_b = &mut array[2]; // This isn't allowed!
*ref_a = 9;
*ref_b = 9;

I’ve seen people new to Rust complain about this many times. Obviously the indexes 1 and 2 do not overlap, so having a mutable references to them should be allowed. However, the compiler doesn’t have any specialised knowledge about array indexing to understand this! It just plays its game with lifetimes, borrows and mutability. The end result of having two non-overlapping mutable references is fine from this perspective, but if the means to achieve that are against the rules, the compiler is not going to give in.

Remedy: Helper APIs

In the Rust standard library, there is the method split_mut on slices that is the model example of helper API that improves the situation. Using split_mut we can split a mutable slice into two non-overlapping subslices. For example, we can split &mut [1, 2, 3, 4] to &mut [1, 2] and &mut [3, 4] or to &mut [1] and &mut [2, 3, 4]. These subslices can be accessed separately and of course, split even further. Another example is iterators: you can mutably iterate over a vector, and as a result get a mutable reference to each of the elements.

So the borrow checker doesn’t actually need to be super smart. Using a bit of unsafe code and wrapping that behind a safe interface, helper APIs can be defined to save the day. In the future I would like to see more these kind of APIs in the standard library.

For example, HashMap could have today a method get_pair_mut that returns two mutable references to two separate items. Of course it would have to perform a run-time check that the items are actually separate, but that’s the price of a safe API. It would also be possible to have facades on top of existing containers that would keep track dynamically which items are borrowed out and which aren’t. Actually, I did a bit of experimenting with such APIs a year back: https://github.com/golddranks/multi_mut

There aren’t any proposals to expand the current set of helper APIs that I know of. We need a hero that champions an RFC for that! The upcoming language features such as const generics and stack-allocated dynamically sized types are likely to help defining better helper APIs too. (I’m imagining a facade over containers that uses dynamically sized types on stack to keep track of the borrows without needing to heap allocate a buffer for that.)

5. Having multiple aliasing mutable references

From a semi-seasoned Rustacean, this sounds like an oxymoron. “Rust isn’t supposed to have these! They were supposed to be awful!” Yet it sometimes helps to be able to have multiple mutable pointers to the same location. Every C programmer knows that aliasing pointers is definitely possible technically, so why doesn’t Rust cut us some slack?

The aliasing restrictions of Rust have good reasons I already spelled out in the above chapter The design decisions to live by. On the other hand, if C gets away with mutable aliased pointers, why should we constrain ourselves to the spartan asceticism of the current borrow checker? Fortunately Rust provides us an escape hatch (other than using raw pointers and unsafe code): the Cell type. Cell is a wrapper type that can be mutated even through shared — an thus normally immutable — references. Using Cell requires the compiler to be a bit more cautious. It must be more careful with aliasing and it mustn’t allow any references to a Cell-wrapped type across threads, as that would lead to data races.

Here’s an example — from the viewpoint of a seasoned Rustacean, this might feel abhorrent; the value of fuga just changes under your feet. However, it shows that this kind of code is possible in Rust too.

use std::cell::Cell;

fn borrow_add_one(val: &Cell<u32>) -> &Cell<u32> {
    val.set(val.get() + 1);
    val
}

fn main() {
    let hoge = Cell::new(4);
    let fuga = borrow_add_one(&hoge);

    // Prints "fuga == 5"
    println!("fuga == {:?}", fuga);

    hoge.set(10); // Mutating hoge but fuga changes too!

    // Prints "hoge == 10, fuga == 10"
    println!("hoge =={:?}, fuga == {:?}", hoge, fuga);
}

Anyway, here’s the actual problem: Cell allows us to bend the curve when we need it, but it also requires us to define our types as Cell beforehands! It’s a wrapper type, after all. What if we have a huge program with established data types? If we want to use Cell, it would be awful to refactor the whole program to use this pattern if we need to.

Remedy: Conversions from &mut T to &Cell<T>

There is an accepted RFC (https://github.com/rust-lang/rfcs/pull/1789) that basically states that the actual byte representation of T and Cell<T> is the same. That means that converting between them, and even converting between references to them is a no-op procedure from runtime viewpoint. The only difference between the two is what the compile-time type system allows. It then becomes possible to convert and use &mut T to &Cell<T> even if the codebase T originates from doesn’t have any Cell types to begin with. Once you have a unique, mutable reference to some type T, you can “fan out” that reference out to multiple shared references Cell<T> do what you must, and then give the references up — everything’s back to normal. The conversion is currently implemented, but not stabilised yet.

There is also already existing pattern for the other direction: since the memory representations of T and Cell<T> were defined to be equivalent, it is possible to go from &mut Cell<T> to &mut T! The mutable reference to Cell ensures that no other references pointing to the Cell exist. That means that the compiler can safely relax a bit, and consider the inner type as a non-Cell type for the lifetime of the mutable reference. It is helpful for passing Cell-wrapped types to APIs that don’t expect Cell types. The API for that was stabilised in Rust 1.11.

6. Being able to point inside Cell types

Being able to convert &mut T to &Cell<T> allows for great flexibility, but it only goes so far. The greatest flaw in &Cell<T>, besides the caveats mentioned this far, is that you can’t have any references pointing to the insides of it. Let’s think for a moment why.

In Rust, there are basically two kinds of data types: structs and enums. Structs are familiar to many, but enums are a rarer feature; they are not like enums in C; they are basically tagged unions: a pair that consists of a union — a memory area that can represent any one of the many declared types or variants — and a tag that is used to tell which one it currently is. Let’s suppose that we have the following enum:

enum JsonValue {
    String(String),
    Number(f64),
    Boolean(bool),
    Object(Map),
}

Let’s build a JsonValue that is inside a Cell: Cell::new(JsonValue::Boolean(false))Then, suppose that we take a reference to the inner boolean. Our reference of type &bool points to the value false. Let’s suppose that we would change the value of our enum, using another&Cell<JsonValue> reference, to Number(2). That would break everything! Why? Because that would make our first reference that is supposed to be a &bool to point something that is definitely not a bool. We have just broken our type system! Beware of the nasal demons. (http://catb.org/jargon/html/N/nasal-demons.html)

So, there’s a good reason why the Cell types prevent inner references. Especially the type of mutability that can change the memory layout of the type can easily cause UB, but as mentioned before, there are subtler reasons around aliasing and threading too. When the type is wrapped in Cell, the compiler knows to be careful, but if we could get a normal reference to the inner value, we would be able to “cheat”, having just a normal &T reference that the compiler doesn’t know to be careful of. The compiler would have a false sense of security that the pointed value couldn’t possibly change, while we could actually change it through another &Cell<T> reference. But what if we really really want to access the insides of Cell<T> with a finer granularity?

Remedy: Conversion between &Cell<[T]> → &[Cell<T>], conversion between &Cell<T> → (&Cell<field of T>, ...).

Actually, there is a pattern that allows referencing the insides of a Cell<T> safely: splitting it up to non-overlapping parts and having each of the constituents live behind another Cell reference. The point is to never allow “bare” references point in. References with Cell are safe to modify, because the compiler knows to be careful.

The same RFC mentioned in the last item also allows conversions between &Cell<[T]> and &[Cell<T>]. That basically means that you can have just a normal slice of values of type T and go from &mut [T] to &Cell[T] to &[Cell<T>] and end up with a sliceful of mutably aliasable values! As said, the RFC is accepted and implemented, but not stabilised yet.

Similarly, it would be possible to convert a &Cell<MyStruct> to a tuple that contains Cell references to the fields of that struct: (&Cell<type of field 1>, &Cell<type of field 2>, …) . However, there is no plausible mechanism in the language at the moment to specify a general pattern like that. There has been some design discussion circling around the topic. (https://internals.rust-lang.org/t/idea-derefpin-derefcell/7292) Maybe we’ll see a Cell field projection in the future? That would require, again, someone to think about the design and write an RFC.

7. Having self-referencing structs

Sometimes there is a need for a struct to have a reference pointing to itself. This has become especially important recently with the work on generators. Generators need to represent their suspended stack frames as first-class values. One can take a reference to a value in the same stack frame inside a generator and then suspend. This makes the the generators self-referencing.

Why self-references are a problem? Because stuff moves and references stay valid only as long as the objects they point to stay put, so allowing self-references while not forbidding moving, we are going to have dangling pointers. Rust is actually able to do some reasoning around self-referential structs:

#[derive(Debug)]
struct Game<'p> {
    player_a: u32,
    player_b: u32,
    current_player: Option<&'p u32>,
}

fn main() {
    let mut g = Game {
        player_a: 10,
        player_b: 20,
        current_player: None
    };
    g.current_player = Some(&g.player_a);
    println!("{:?}", g.current_player); // prints Some(10)
    g.current_player = Some(&g.player_b);
    println!("{:?}", g.current_player); // prints Some(20)
}

If we try to move g, it complains that there is a reference to it. However, problems start right away with examples any more complicated than this — with mutability, to begin with. For example, if you store the struct in heap using a Box and then try to access it mutably, initialising the field current_player with a self reference, the whole lifetime of the struct gets “tainted” with the mutable borrow, because by storing a reference derived from that borrow in the struct itself, we accidentally set the lifetime of the mutable borrow equal to the lifetime of the struct itself. The struct “locks up” — we lose the ability to access it using any other reference until it’s dropped.

We would need some way to signal the borrow checker that after mutating the struct, we have “downgraded” the lifetime (see also the the item 10 about downgrading mutable lifetimes) to a shared one. But even if we would be able to do that, since the struct still contains a reference derived from a borrow of the struct itself, the struct would stay in a “borrowed mode” until it gets dropped — we could never mutate it again.

Wrapping current_player into a Cell cuts us some slack with mutability. But even then we are in troubles with lifetimes:

use std::cell::Cell;

#[derive(Debug)]
struct Game<'p> {
    player_a: u32,
    player_b: u32,
    current_player: Cell<Option<&'p u32>>,
}

fn init_game<'???>() -> Box<Game<'???>> {
    let g = Box::new(Game {
        player_a: 10,
        player_b: 20,
        current_player: Cell::new(None)
    });
    g.current_player.set(Some(&g.player_a));
    g // Doesn't work!
}

fn main() {
    let game = init_game();
    println!("{:?}", game);
}

Note the lifetime '???! There is no lifetime that we could name that fits there. The lifetime clearly isn’t something that the caller of the function can decide as a parameter, since it’s simply the lifetime of how long the heap-allocated Game happens to live — that might depend on anything, including the runtime control flow. On the other hand, the self-reference, originally borrowed from g lives only as long as the variable g does — the borrow checker doesn’t understand that the reference actually points to a heap allocation that is going to live longer than the variable g. This means that we can’t meaningfully pass the self-referential types down or up the stack, even if it would be safe in the sense that the heap allocation doesn’t move. (See the item 14 for further discussion!)

No, it seems that we have to use unsafe code and raw pointers. The borrow checker doesn’t check raw pointers, so they provide us all the flexibility we need. But then we face another problem: if nobody checks for the correctness, do we lose the ability to abstract the unsafety away, behind a safe interface? What if we release a library that uses self-referencing types, but our users shoot themselves in foot because they accidentally move the value without realising that isn’t allowed?

Remedy: Pin references

There was a recent RFC that addresses this problem: Pin references. (https://github.com/rust-lang/rfcs/blob/master/text/2349-pin.md) Having an object behind a pin reference provides an important guarantee: either the object is safe to move or it is not safe but will not move until it’s dropped. So using pin, one is able to require the users of a type not to move it. Actually getting a mutable reference to the insides of a pin that contains a self-referencing type requires unsafe code; anybody getting a mutable reference will do so knowing that they must not move the value. Pin references are going to be in the standard library soonish—the stabilisation has been a proposed but not decided upon yet.

8. Capturing only disjoint fields in closures

There is a slight ergonomics problem when using closures: they tend to capture values too eagerly:

fn update(&mut self) {
    // borrowing self.list mutably
    self.list.retain(
        |i| self.filter.allowed(i) // can't borrow self!
    );
}

The problem here is that although self.list and self.filter are separate fields and can normally be borrowed separately, the closure tries to borrow self as a whole! There is a simple workaround: manually borrow the field and then let the closure capture that:

fn update(&mut self) {
    let filter = &self.filter;

    self.list.retain(|i| filter.allowed(i));
}

Having to do this is annoying; it’s not a show stopper, but certainly it would be nice if would be a bit smarter automatically.

Remedy: Capture disjoint fields

There is a merged RFC that makes the closure capture smarter by default: https://github.com/rust-lang/rfcs/pull/2229 It’s not stabilised yet though.

9. Having associated types that are generic over lifetimes

When processing data from a stream, it’s not uncommon to have a buffer that holds a “chunkful” of the data being processed. You can reuse the buffer when you are done with processing the current data. This helps to avoid allocations during the operation. Of course this means that references to the buffer are valid only for the lifetime of the current chunk — once we start overwriting the buffer with new data, all the references pointing the previous data in the buffer must be gone.

It would be nice to express this as an iterator pattern! Think of it: looping over the contents of the stream like we usually loop over data containers. However, there’s a problem with the iterator interface that blocks us from expressing this:

trait Iterator {
    type Item;
    fn next(&mut self) -> Option<Self::Item>;
}

The main workhorse of the Iterator trait is the next method that returns the items of the iterator. Item is an associated type — every implementation of the trait can decide what that iterator spits out. As we are iterating over a stream, we would like to spit out a reference to the buffer that holds the current chunk of the data. However, if we set type Item = &'a Chunk, we soon stumble into expressiveness problems. What is the lifetime 'a? It should be the same lifetime as of the &mut self of the next method, but as Self::Item doesn’t have any lifetime parameters, we are unable to express that!

It turns out that iterators are able to return only 'static items (items that don’t contain lifetimes at all or items that contain only lifetime 'static) such as String or u32, or items with lifetimes that are nameable by the type that implements Iterator. Here’s an example of the latter:

struct StrVecIter<'a> {
    index: usize,
    inner_vec: &'a Vec<String>,
}

impl<'a> Iterator for StrVecIter<'a> {
    type Item=&'a str; // We can use StrVecIter's 'a here
    fn next(&mut self) -> Option<&'a str> {
        let s = self.inner_vec[self.index].as_str();
        self.i += 1;
        Some(s)
    }
}

Turns out that Rust’s iterators don’t support so-called streaming iterator pattern. They can only return references that live as long as the container they refer to, lives. We want to return references with a shorter lifetime — by the next call to next the previous reference should already be gone!

Remedy: generic associated types

The problem was with the associated Item type — it should be generic over lifetimes so that we could use it in the next method to equate it with the lifetime of &mut self each call. As it turns out, Rust doesn’t at the moment support generics in the associated types of traits. Fortunately that is subject to change: an RFC that enables that feature has been accepted: https://github.com/rust-lang/rfcs/blob/master/text/1598-generic_associated_types.md

The implementation is not done yet and the feature is not considered high priority before 2018 edition, but I’m hopeful that we’ll hear more about it early next year and possibly see it stabilised at some point of the year. This feature doesn’t only provide added expressiveness around streaming iterators, it should enable a whole lot of other nice patterns too.

Addendum: Getting ownership over a mutable reference

(Added 15th September 2018. In the end of this article, I asked people to point me out cases that I didn’t think of, and the people of the /r/rust subreddit did! Thank you.)

Because Rust differentiates between owned and borrowed values, sometimes it so happens that you have only a mutable reference whereas you need to pass owned value in a function to use some API. You might need to refactor and get an owned value instead of a reference, which may be cumbersome because then you just move the requirement for ownership around, or you might have to clone which can be expensive. Sometimes you can’t even do that; some values don’t even implement the Clone trait. (Often if they don’t, they have a good reason not to, but there can be omissions, of course.)

However, it gets irritating when the requirement for ownership is seemingly trivial: an API that takes in an owned object but returns an owned object of the same type. After the fact, you still have an object of the same type, so you haven’t lost anything, type theoretically speaking. If you could move an object out of a mutable borrow for just a bit, and then return it, everything would be easier. Is there a way to do it, or is it a hard limitation of the borrowing system?

Actually, Niko Matsakis started recently a blog series about the current limitations of the borrow checker, which coincides with the theme of this article almost perfectly. I’m doing to forward the further discussion about this problem and the workarounds to his blog. As basically everything Niko writes, it’s highly recommended reading! See it here: http://smallcultfollowing.com/babysteps/blog/2018/11/10/after-nll-moving-from-borrowed-data-and-the-sentinel-pattern/

Open problems: from here on there be dragons

At this point, we have exhausted stuff that are agreed upon via the RFC process. For the rest of the items in this article, there are going to problems without ready-made remedies. I think each of these problems can be solved, but doing so will require a significant amount of design and consensus work.

10. Downgrading a mutable lifetime to a shared one

It’s not uncommon to call methods that mutate some fields but return just a shared reference:

impl Request {
   fn get_header(&mut self) -> &Header {
      if let Some(cached) = self.cache.retrieve() {
         cached
      } else {
         let parsed = self.parse_header();
         let cached = self.cache.store(&parsed); // Needs &mut self
         cached
      }
   }
}
...
let header = request.get_header();

However, as long as we grab to our header, we aren’t allowed to call any other methods on request! It’s completely locked up. Why? Because we originally borrowed request with a mutable lifetime and no additional borrows are possible until that lifetime has ended.

But, you say, the returned header is only a shared reference! Surely calling other methods that take &self, a shared reference, is okay. But it’s actually not. For what the borrow checker knows, get_header could have stashed a mutable reference to request somewhere. Maybe send it to another thread? Maybe store it in the local thread storage? Maybe hide it in a Cell<Option<&mut Cache>> field in the returned header? There isn’t any ironclad guarantee that the mutability of the lifetime would be actually over until the end of the borrow. Here’s another great example of this: https://internals.rust-lang.org/t/blog-post-nested-method-calls-via-two-phase-borrowing/4886/33

Anyway, the request “locking up” is annoying. In some cases, it’s sensible to wrap the cache field in a Cell and be done it. But wouldn’t it be nice if the API itself could declare that it really is done with mutating things?

Remedy: Read-write lifetimes? Downgrade declarations?

There have been some ideas around having two lifetimes when taking a reference: a read lifetime and a write lifetime. Strawman syntax:

fn mut_then_share<'w, 'r: 'w>(&{'w -> 'r} mut self) -> &'r u32

The idea behind this is that the write lifetime is a subset of the read lifetime. The concerns are separated, so the write lifetime may be dropped earlier, leaving only the shared lifetime. Another plausible syntax would be some kind of “downgrade declaration”:

fn mut_then_share<'a, 'b>(&'a mut self) -> &'b u32
    where 'b: 'a + const

There seems to be some support behind these ideas, but they are effectively postponed after the work on non-lexical lifetimes has finished. There are also going to be some design issues about the conditions where downgrading can be regarded safe but I’m hopeful we’ll see this feature in some form in the future!

11. Calling mutable methods that don’t access overlapping fields

I remember being frustrated with this one when I started Rust:

struct Brute {
    name: String,
    cry: String,
}

impl Brute {
    fn new() -> Self {
        Self { name: "Pochi".into(), cry: "WOOF!".into() }
    }

    fn get_name(&self) -> &str { &self.name }

    fn set_cry(&mut self, cry: &str) -> bool {
        self.cry = cry.to_ascii_uppercase();
    }
}

fn main() {
    let mut pochi = Brute::new();
    
    let pochis_name = pochi.get_name();
    
    pochi.set_cry(&pochis_name); // Can't borrow!
}

This pattern often emerges with getter/setter style accessors. The problem is that the methods take references to self as a whole and — as mentioned earlier — by design, the borrow checker can’t peek into the method bodies and see which fields are actually accessed.

If a similar access pattern is done locally, there is no problem; the borrow checker can see that the name and cry fields are non-overlapping and can be borrowed separately. That means that as a workaround with structs, you can set the fields public and access them directly, but this isn’t good if your type has invariants you want to protect. For example, here we want to ensure that cry is always uppercase!

This is especially troubling when working with traits and generic code. Traits are collections of interface methods; even if the underlying type that implements a trait has non-overlapping fields, the trait hides that as an implementation detail, so you are forced to access self without finer granularity.

Remedy: Fields in traits? Read-only fields? Partial borrows?

To address the problem with traits, there has been an RFC (https://github.com/rust-lang/rfcs/pull/1546) that allows defining field in traits that each implementer maps to the corresponding fields it has. Fields are different from setter and getter methods in the sense that the compiler can verify that they actually map to non-overlapping memory, allowing safe access. The RFC was postponed for now— but there is still demand for a feature like this and I’m quite sure it will be revisited after the 2018 edition has shipped.

Part of the design space has also got to do with mutability of fields. There are quite often patterns where you can show the contents of a field, but not allow it to be modified safely. As getters have the granularity problem described above, it would be desirable to expose the field itself publicly, while restricting the access to it to immutable only.

There has been also some ideas floating around about refining the granularity of self: methods would declare the fields they access in the function signature:

fn get_name(self { &name }) -> &str { &self.name }

fn set_cry(self { &mut cry }, cry: &str) -> bool {
    self.cry = cry.to_ascii_uppercase();
}

This would allow the borrow checker to conclude that the method calls access non-overlapping fields without peeking into the method body.

All of these ideas have some drawbacks in the sense that they expose things that are currently thought of implementation details but we will see what comes out of them.

12. Hiding mutable lifetimes in data types

Some time ago there was a blog post by Aleksey Kladov (https://matklad.github.io/2018/05/04/encapsulating-lifetime-of-the-field.html) that highlighted a problem with lifetime annotations in data types. The problem raises its head when nesting types with mutable lifetimes:

struct Foo<'s> {
    string: &'s mut String,	
}

struct Bar<'f, 's: 'f> {
    foo: &'f mut Foo<'s>
}

struct Hoge<'b, 'f: 'b, 's: 'f> {
    bar: &'b mut Bar<'f, 's>
}

// As you see, the declarations are getting longer and longer!
struct Piyo<'p, 'b: 'p, 'f: 'b, 's: 'f> {
    hoge: &'p mut Hoge<'b, 'f, 's>
}

The lifetime annotations don’t compose well! This isn’t a problem with shared lifetimes, which stay clean:

struct Foo<'s> {
    string: &'s String,	
}

struct Bar<'f> {
    foo: &'f Foo<'f>
}

struct Hoge<'b> {
    bar: &'b Bar<'b>
}

struct Piyo<'p> {
    hoge: &'p Hoge<'p>
}

The difference here is that shared lifetimes — due to their restrictions with mutation — can be subtypes of other shared lifetimes. Mutable lifetimes, on the other hand don’t “mix and match”. This is a quite fundamental difference between the two kinds, and it must be respected to avoid soundness issues. However, the proliferating lifetime annotations get icky rather quick.

Remedy: Hiding mutable lifetimes from type signature?

Inspired by Aleksey’s blog post and the trick explained there that lifetimes can be hidden with trait objects, I started thinking about reifying this hiding mechanism as a language feature. I started writing an RFC, but it’s essentially just a draft at the moment: https://internals.rust-lang.org/t/pre-rfc-encapsulating-private-lifetimes/7500 Getting busy with other things in life, I haven’t been polishing it, but hopefully I’ll manage to return into it some day.

13. Using “ambient” lifetimes

One of the important concerns for library code is modularity. A big part of application programming is composing available libraries to achieve higher-level goals, but if the libraries don’t play nicely together, this gets troublesome. Libraries should be either as simple or as generic as possible; or preferably if there exists a way to be simple and generic, do that. Especially requirements of specific “ambient” features in the runtime environment limit the generality of libraries. A great example of this is that the libraries with no_std capabilities have greater composability because they don’t depend on the standard library.

This is why I often think of'static lifetime bounds as undesirable things in APIs. It limits what the user can pass in. If possible, it’s always better to be generic with regards to lifetimes to allow the user pass values with lifetimes that suit themselves.

What makes “forced” 'static even worse is the fact that statics are so hard to initialise. The crate lazy_static helps and there is also pattern where you can “forget” a heap-allocated value to protect it from deallocation for the rest of the program lifetime and turn it into a reference to 'static. But these are essentially hacks: if you are unable to clean up after finishing your business with a library, that library can’t be said to be composable. Rust doesn’t have “life before main”, which is a great design decision—besides the other problems it prevents, it also reifies the fact that one should avoid “hard-coding” lifetimes. But requiring ‘static is essentially that: hard-coding lifetimes.

However, there’s an understandable reason why one would like to use the 'static lifetime for things: it’s the only globally nameable lifetime. An ambient lifetime, so to say. Because it’s a concrete lifetime that’s available everywhere, it doesn’t proliferate in the type declarations like generic lifetimes do. It’s more ergonomic to use and easier to understand.

Is there any way we could prevent the hard-coding problem and still have the ease of using 'static?

Remedy: ambient lifetimes/module-level lifetimes

Imagine if there would be a lifetime like 'static in the sense that you don’t have to write it into the signature of your types? A lifetime that would say: I live longer than this struct could ever possibly live, so you don’t have to care what I am.

mod library<'ambient> {
    struct StringRefs {
        ref_a: &'ambient str,
        ref_b: &'ambient str,
    }

fn take_refs(refs: StringRefs) {
        println!("{}", refs.ref_a);
    }
}

Here, 'ambient would live longer than any type defined in module library. It’s like a local version of 'static! Then, repurposing the use import statement a bit:

fn main() {
    let string_a = String::from("No life");
    let string_b = String::from("Before main!");

    use library as lib { // 'ambient gets assigned to this scope
        let r = lib::StringRefs {
            ref_a: string_a.as_ref(),
            ref_b: string_b.as_ref(),
        };
        lib::take_refs(r);
    }
}

Of course, the compiler would prevent any type with 'ambient leaving the scope. This allows initializing everything in main but after that initialization is done, the lifetime of the state that lives there can now be “freely” referred by the types, without the need to carry the lifetime information around in the type signatures.

Another interesting idea would have scopes that “repurpose” what 'static means for the code inside that scope: the code thinks that it has references to static things, but the caller has actually redefined it to be a narrower lifetime. I haven’t thought much about the soundness implications though; it might prove to be an outrageously unsafe idea.

14. Moving the owner of a heap-allocated object that has an inbound reference

I think of this problem as the granddaddy of all lifetime problems and that’s why I left it in the end. Lifetimes in Rust are essentially subject to stack discipline. An “outlives” relationship between two lifetimes means that the longer-living one originates from an outer scope or an earlier (shallower) stack frame.

A reference such as &'a Foo<'b> can be thought as two values: the &'a part, which is essentially just a pointer, and the value being pointed at, here Foo<'b>. The basic rule, of course, is that the value being pointed at must live longer than the pointer—but to elaborate, it must live longer at the location being pointed at. You see, there’s another distinction to be made: we can think of the value as pure information that we can copy and move around — or we can think of the memory slot the value resides in. That can’t move around. Pointers point at memory slots, so Rust ensures that slot stays valid by freezing the value, keeping it there.

Here the stack discipline kicks in: we can call other functions and pass the pointer deeper in the stack, while the pointed value stays put. But once we return, at some point, the memory location where the value resides must be given up. That’s the maximum extent the reference can live. (The minimum extent is of course, up to us, because we can just drop the reference anytime we want.)

This principle works wonderfully with the call stack. But it’s too restrictive with the heap. The lifetimes are not aware of the heap — there can be references to the heap, but they act as if the heap would be just a nice extension to the stack that allows dynamically sized allocations. The lifetime of a reference to a heap allocation is still constrained by the stack frame where the lifetime of the reference originated from.

That’s not how heap works though: unlike with stack, it’s possible to do a heap allocation and return that allocation from a function—so it’s unordered! And here’s the problem, hinted in the item 7 about self-referential structs: the lifetime system doesn’t understand that the lifetime of a heap-pointing reference is not constrained to the extent where the stack-allocated owner of the heap allocation — such as a Box—is located at the moment of the borrow. It’s constrained to the extent the heap allocation lives, and the heap allocation in many cases lives as long as the value of its owner lives. Note that here I don’t mean the memory slot of its owner but the actual value; the piece of information that can be moved around until it’s destructed.

It would be sound in principle to have a reference to a heap-allocated value and return that reference alongside of the owner of the heap allocation from a function, down the stack. Likewise, it would be sound to store them in a struct and encapsulate all the lifetimes involved. One could pass the struct along without any lifetime restrictions, because the lifetimes would be implementation details. This would be a boon for zero-copy parsers, graphics API wrappers (https://github.com/vulkano-rs/vulkano/blob/master/TROUBLES.md) and basically everyone using the crates Rental and Owning-Ref at the moment.

Remedy: Dependent/existential lifetimes?

There is a significant challenge in designing a lifetime system that would ensure the soundness in safe code while allowing greater flexibility with references to heap allocations. Such a system must adhere to all the design principles of Rust mentioned earlier: it should be locally analysable and statically checked. It should be combatible with the current lifetime system and preferably a minimal extension. I’m not aware of any serious attempts at trying to come up with a working solution yet.

We could think of the pointer and owner of the pointed allocation as a pair that are connected by the guarantee that at no point the pointer is “lower” in the call stack or in the local scope than the pointee — either the pointer is deeper in the stack or they are being passed down together. Imagine something like this:

// Note the for syntax!
fn init() -> for<'base: 'ref> (BorrowedBox<u32, 'base>, &'ref u32) {
    let heap_box = Box::new(10);
    
    // get_existential takes the box as a value
    // and returns a pair with an existential lifetime
    let (borrowed_box, heap_ref) = heap_box.get_existential();
    
    // The borrow checker locally checks that any value
    // containing 'ref doesn't outlive a value with 'base
    
    // stored in a single data type whose soundness is ensured
    // by the for<'base: 'ref> annotation of the function signature
    (borrowed_box, heap_ref)
}

And this:

struct Encapsulated { // No lifetime here!
    for 'base: 'ref,
    heap_box: BorrowedBox<u32, 'base>,
    heap_ref: &'ref u32,
}

I’m thinking of continuing sketching around this.

Closing words

Rust is marching towards the 2018 edition release and a huge amount of work has been done to stabilise some long-awaited features and polish the ergonomics story. Most improvements mentioned in items 1–9 are landing really soon, which is incredibly exciting.

However, when considering the future of Rust in the long term, I think the story around lifetimes and mutability isn’t done yet. Lifetimes are Rust’s flagship feature and they have shown the world that memory management without garbage collector is possible in a sound and a practical way.

In the future I’d like Rust to go all the way and prove to the world that not only memory management with lifetimes is possible, it can be also ergonomic and expressive. The items 10–14 are things that I consider problems that continue hindering users of lifetimes even well after the 2018 edition has shipped. They are also problems that I think are worth solving especially because lifetimes are such a central feature of Rust and Rust has spearheaded their use in real-life code. It frustrates me to think that the problems we still have with granularity, leaking abstractions, unpolished ergonomics and lacking expressiveness might lead some people to think that lifetimes are not worth the hassle. I want the world to see the ultimate form of lifetime-based memory management!

P.S. If you think that I’ve missed some obvious problem with the current lifetime/mutability system, please let me know!

A short primer on phonemes

We, humans who wield language, use a great many parts of it unconsciously— most of the time, at least — directing our attention only on meaning we intend to communicate to our fellow language users. This is why it escapes the most of us — expect for some experts — how language actually works in detail. There’s much to be uncovered, but some basic facts are known and have been know for a long time. One such fact is that we parse the language we hear in more than one phase; when an utterance, a waveform of pressure waves in the air, reaches our ear and is processed by the auditory system, we at first process the pure sound material into something called phonemes.

There is an infinite amount of different sounds (spanning both frequency domain and time domain), but our brain chunks and classifies the continuous input of sound to something more tangible, into a finite amount of categories. I’m speaking about something like /p/ in the wort “part”, or /t/ in the word “tart”, or the rising tone in the word “麻” (/má/)or the “dipping” tone in the word “馬” (/mǎ/). There is a million ways to say “part” or “tart” (for sure, not once in the history of mankind have these words been pronounced exactly the same way, if that’s even a meaningful thing to say), but they are still only two words, depending on the consonant — the phoneme — that starts the word. To be sure, phonemes are not letters or anything graphical or anything related to writing, but abstract categories that form the low-level basic units of a language. (Especially spoken language.)

Phonemes are language-specific. The phonemes of English are different from the phonemes of Finnish and those are different from the phonemes of Japanese. Sure, there are sounds that would be categorised as a good example of /e/ of Finnish and /e/ of Japanese, but as categories, that is, some kind of delimited spaces of possible sounds, their borders are more or less different and their exemplars — the most /e/-like /e/ you could think of — are more or less different from each other. Heck, forget about borders and exemplars, they are just special cases anyway; the distribution of sounds is different.

Here’s a chart of Japanese vowels, if it helps to think about the situation. Each vowel gets its “sound” or quality, from the resonances of a specific posture the speaker’s mouth makes when pronouncing the vowel. These postures and resonances are similar enough between speakers that we can generalise our ability to recognise vowels even if we hear the voice of that particular speaker for the first time.

Now, Japanese uses clusters or categories like presented above for its vowels. But other languages might use entirely different categories. You can imagine that the categories between languages might overlap more or less, but that doesn’t make them the same categories.

That raises the question: when we learn another languages, how do we learn novel phonemes with their associated sound distributions? That is an empirical question, and it is studied in the field of second language acquisition research, SLA for short.

Parsing beyond phonemes

After parsing the limitless sea of possible sounds into a limited and categorical assortment of phonemes, we then further parse those into words. Or morphemes, linguistically speaking. To be able to do that, we have to possess some kind of a mental lexicon; a database in the brain that maps strings of phonemes (there may possibly be some intermediate representations that chunk phonemes together to help them form more organised patterns: moras, syllables, metrical foots etc., but you don’t need to care about those if phonology doesn’t rock your boat) into formal features and meanings (these may be associations to all kinds of sensations, feelings, memories or more abstract concepts like “going somewhere” or “having empathy towards others” or “doing something over the night without any sleep”). The formal features are like tags that are further used to understand how the morphemes relate to each other when parsing goes on. The morphemes are only basic units of meaning, and further parsing is still needed to form a bigger picture of how those units combine and interract to communicate some meaning that wouldn’t have been expressable using the basic units only.

It should be obvious that when we use language to understand meanings that are communicated to us using the language, we must parse the sounds we hear into phonemes and then parse those to morphemes and then go on parsing the bigger picture (morphology→syntax→discourse). If we want to get the meaning, we can’t skip stuff or stop after phoneme level — at least the two first processing steps — from sound to phonemes and from phonemes to morphemes — are needed to get even to the most basic form of meaning.

Additional remarks on the nature of phonemes

Now, I hear the voices of some sceptics. Are phonemes even real? Why do we need this kind of an “intermediate representation”? What if the words are stored directly as audio patterns in the brain? If the chart above with clear visible clusters doesn’t convince you, I have some good news: there’s been quite an amount of discussion and research about this and the evidence is in: yes, phonemes are psychologically real. Here’s some links to get you started: [StackExchange] Is the very concept of the phoneme disputed?, a classic text by one of the founding fathers of linguistics: [Google Books] Sapir: The Psychological Reality of Phonemes, and last but not least, actual empirical evidence: McQueen et al. (2006) Phonological Abstraction in the Mental Lexicon.

There’s one more thing you should know about phonemes to understand some of the empirical questions I’m going to present later: phonemes are categorical. Not only they are clustered into blocks like on the vowel chart above, but there’s something called “categorical perception”. Speakers of Japanese, for example, are lousy at distinguishing two vowels whose formant F1 differs by 50 Hz, if both of the vowels are good examples of /a/. They just don’t hear the difference. But you bet they’ll hear the difference even if F1 differs only by 30 Hz, if another vowel happens to reside in the distribution of /i/ and the other in the distribution of /e/. (Check the chart; /i/ and /e/ are actually quite close!) In other words, people are insensitive of sounds within the phoneme categories, but very sensitive at or over the borders of phoneme categories. Since the borders of the categories are different in every language and this sensitivity has been empirically shown to exist in the speakers of different languages, it’s clear that the sensitivity at specific zones is an acquired or learned phenomenon. How that learning happens? That — again—is an empirical question.

Studies on phoneme learning

As interested I am in second language acquisition — that is, how we acquire languages after our mother tongue — I must admit that there’s still quite lot of mysteries to solve about the learning process of phonemes and categorical perception. It’s a well-known fact that even though individuals learning a second language can become quite good at it, provided that there is enough time, enough input and interaction in that language and enough motivation, they tend to retain audible “foreign” accents. (Note that there is a huge amount of individual variability and a lots of factors affecting this. Nothing is simple in SLA.) It seems that acquiring novel phoneme categories is very hard for adult learners, and even if the linguistic system of the learner has reached highly advanced levels, pronunciation often lags behind or is at standstill (Tsukada & Birdsong 2005).

There is empirical evidence that pronunciation of second language hinges on the perception of the phonetic categories; unless you are able to perceive and distinguish the phonetic categories more or less like a native speaker would, you will have little hope of pronouncing sounds in those categories right, except by chance. (To be sure, I’m talking about pronouncing while paying attention only to meaning in a communicative setting. That means letting the automatised processes in the internal linguistic system to manage the details of the linguistic processing such as forming sounds. I think many people are able to learn quickly to “imitate” native-like accents for short words and phrases when they have a model to listen to and specifically pay attention to that, but that’s different from acquiring a language.)

In hindsight this seems obvious and it is analogious to the whole process how language acquisition works in general: if you don’t have sufficient exposure to input data, your internal model of the language won’t have had developed (we are talking, again, about unconscious processes here) enough to be accurate and complete. Based on inaccurate or underdeveloped internal model, it’s no wonder that the production of sound is inaccurate too. Succintly said: output depends on input.

Not surprisingly, there has been multiple attempts to teach people to perceive the phonemic categories of the target language and to try and overcome the seemingly difficult task of re-adjusting the phonetic system. After Logan and Lively (1991), there has been a flurry of studies on a training paradigm called high variability training. These studies have successfully demonstrated perception of novel perceptual categories. They have also demonstrated that this learned categorical knowledge can transfer from perception to pronunciation. So… huzzah, our foreign accent problem is solved! Just have the learners do these high variability shenanigans, and the funny accents of Arnold Schwarzenegger and Slavoj Žižek (not to even mention Finnish rally drivers) are a thing of the past.

Alas, things are never so simple. After reading a study after study, it has become painfully clear to me, that while the results they sport may be indeed true, their application is limited in the process of actual language acquisition. (Oh, and by the way, getting rid of the funny accents was a joke. We are talking about acquiring phoneme contrasts here, which should help with speech intelligibility, but doesn’t guarantee native-likeness, since one can still realise the sounds inside a category in a way that is quite far from the exemplar.)

What the studies are neglecting

Here’s a bunch of question that I think we should be asking, but that I seldom see the studies to ask. To understand some of the critique you should be aware that in the field of SLA, some scholars distinguish between learning and language acquisition. It has been empirically shown, that while people can quickly be taught to retain facts such as item-label pairs (=”learning words”), apply rules to modify symbolic structures (=”learning grammar”) and so on, this learned skill doesn’t seem to readily transfer to actual use when using language to communicate. Indeed, it appears to be that the implicit linguistic system in the brain is very resistant to external manipulation, and develops slowly in response to using language to understand/parse meaning. But when it develops, it seems to have this automatic quality that you don’t have to think about nouns and verbs, you don’t have to think about conjugation rules — the implicit system has got your back. (Or doesn’t, if it hasn’t acquired enough language yet.) Most of the studies that recognise the distinction, however, are done in the fields of morphology and syntax. I don’t remember seeing much of thought to be given to this difference in the pronunciation training studies. (Some recent evidence supporting that speech learning is optimally learned by implicit neural systems exists, however: Chandrasekaran et al. 2014)

So, here goes my list of questions we should be asking:

Is “perceptual category” different from “phonemic category?”

What I mean by these terms:

perceptual category — people are able to identify an aural stimulus as a member of some category. These categories are shown to be formed with short-term high-variability training, and they allow even naive (=zero experience of L2) trainees to identify L2 sound categories when instructed to do so.

phonemic category — a perceptual category that is part of the linguistic system and can be readily and automatically used as a part of the linguistic system to communicate meaning.

Being a “phonemic category” requires the category to exist and be retained not only in the aural speech processing subsystem but also in mental lexicon. For example: Chinese tones form a phonemic category for Chinese speakers (both L1 and sufficiently advanced L2); they are used to distinguish between meanings of words which may be otherwise similar. That means that Chinese speakers are not only able to distinguish aural stimuli based on these categories, but they are also able to retain words that are distinguished by these categories in their mental lexicon, and they are able to use these words to communicate meaning without much of a thought. But I’ve seldom seen a study on high variability training to ask, whether the perceptual categories are represented in mental lexicon. They always seem to concentrate on just “hearing” the difference. Only two studies that I know of, make a difference by focusing also on the lexical aspect: Wong & Perrachione (2007) and Chandrasekaran (2010), but even they focus only on learning pseudowords, which is hardly representative of experience of actual language acquisition.

Does peaking in discrimination sensitivity at the category border imply the category being “phonemic” and thus part of the linguistic system?

In other words: language acquisition is known to “warp” the perceptual space, but does the warping of perceptual space always mean that a phonemic category is formed? (Is the relationship → or ↔?)

As said above, native speakers, for sure, have heightened sensitivity of the perceptual space at category border. There is evidence (Heeren 2008) that although phonetic (not meaning-based) short-term high variability training helps to form categories of sound identification, the trainees fail to develop sensitivity peaks at category borders — however, advanced second language learners (three years of majoring in the second language in college) do develop sensitivity peaks at category borders. This begs the question: is a heightened sensitivity at the category border a sign of an acquired phonemic category? (Rather than learned perceptual category?) Short-time explicit learning certainly helps to form perceptual categories, but is the development of phonemic categories due to (implicit) language acquisition?

However, there’s another possibility to keep in mind: it might be just so that the trainees need longer periods of (non-meaningful or not) training to develop sensitivity at phonemic borders. Maybe the development of sensitivity doesn’t have anything to do with language acquisition in general?

Does the learning of “perceptual categories” help people to succeed in communicative tasks?

If they do, that means that category learning in phonetic domain might have actual value in communicative language teaching. Maybe percieving phoneme categories better can have a facilitative effect to advance acquistion in other domains? Wong and Perrachione (2007) claim so, but their data is based only in a short-term training of an artificial language.

There is also evidence that forming of perceptual categories transfers from non-communicative input to output. By non-communicative I mean perceiving category differences and producing them in non-communicative situations, such as reading aloud a list of words. It’s relevant to ask, then: does the transfer occur also in communicative situations, when attention is directed to communicating meaning; in other words, in “spontaneous speech”? Using a category in spontaneous speech is usually considered a telltale sign of language acquisition.

Do explicitly learned perceptual categories transfer to phonemic categories? Or do phonemic categories develop independently?

This question resembles the question about the “interface position” in SLA.

Can meaning-based structured input activities (as pioneered by VanPatten & Cadierno 1993) help forming “phonemic” categories?

Most (if not all) of the research on processing instruction (=a treatment based on meaning-based structured input activities) has happened in the domain of morphosyntax. It is originally based on VanPatten’s model of input processing, which is in the domain of syntax, and considers only linguistic universals (instead of cross-linguistic influence). Applying similar ideas to a totally different domain such as phonology may seem odd, but the idea behind the training paradigm remain sound: focus on processing from form to meaning to overcome processing problems on the way. (Not only phonology is a different domain, but for example, L1 transfer/inference is much more apparent in phonology and phonetics than in morphology and syntax — the inference manifests as the “foreign accent”.)

It has been shown that perceptual categories can be formed with phonetic (not meaning-based) training, but none of the studies have shown that such training could help the trainees to form phonemic categories. However, it seems plausible to me that meaning-based structured input activities could help forming actual phonemic categories. Of course, this needs to be empirically tested, but if that’s true, it would give some tools to teaching pronunciation efficiently.

As stated above, there is evidence that language acquisition over long periods may lead to formation of phonemic categories (defined at least by usage in spontaneous speech and possibly also by the sensitivity peak at category border). There is also evidence that adult learners of SLA often fail to form these categories (Tsukada & Birdsong 2005). However, when trainees are forced to parse from form to meaning, as they are in structured input activities, it seems that the category and the processing needed would need to develop not only in phonetic domain but also in the mental lexicon and other relevant linguistic domains.

There are two studies that have attempted this: Gonzales-Bueno & Quintana-Lara (2011) (unfortunately with little conclusive evidence), and Hirano-Cook (2011) who reports having used processing instruction to train students to perceive Japanese pitch accents, but she seems to have misunderstood the idea of processing instruction slightly based on the description of the activities. She also tests only for pitch pattern identification, which doesn’t tell anything about language acquisition. So the field is still open for a study that probes the effects of meaning-based structured input activities for acquisition of novel phoneme contrasts.

As a summary, I think that exercising the whole chain of processing from form to meaning in a communicative setting might actually lead to acquisition. Exercising just the first part of the chain (parsing from sound to phonemes) perhaps not.

Also note that there is tangentual counter-evidence against the benefits of meaning-based activities: Guion & Pederson (2007) claim that paying attention to phonetic difference (instead of meaning) is more benefical for forming categories. However, they test only for perceptual categories, not for phonemic. If perceptual categories can develop into phonemic categories, paying attention to phonetic difference can pay off when starting the training. But if they can’t (no-interface position), it’s just waste of time, since phonemic categories develop independently.

Summary: towards better questions

All in all, I think that while much has been researched about learning pronunciation, much remains to be uncovered by future research. Already by adopting experimental design the field has moved forward. (For example: it seems that there has been quite a lot of studies on acquisition of Japanese lexical accent since the 90’s but only in the year 2011 the first study that used a control group and statistical tests emerged (Hirano-Cook 2011).)

Also, some variables have been identified that are controlled for quite universally nowadays. The main one is former exposure to languages with relevant categories. Another one is age, which contrary to expectations, hasn’t shown up as a factor in perceptual category learning. (Contrasting Tsukada & Birdsong 2005 and Heeren 2010, it might be that pronunciation acquisition might be subject to a ”sensitive period” or neurological maturing, whereas perceptual category learning isn’t. This demands for investigation!) Curiously, for studies researching tonal patterns, musical training has been shown to improve performance. Also testing for unfamiliar stimuli that aren’t part of the training regime (novel phonological context, unfamiliar voices, possibly even unfamiliar but related phonological phenomena) is — and should be — commonplace today to check whether the learning generalises.

It would be immensely helpful if the studies in the future controlled for these variables:

• Are the trainees able to use the learned category communicatively? (For example, to understand correctly the meaning of an ambiguous sentence disambiguated only the category that’s being trained?) A gold standard for this would be demonstrating the usage of the category in spontaneous, meaning-oriented speech.
• Are the learning effects shown only in identification tests or also in discrimination tests with modified stimulus? (Discrimination tests with carefully crafted stimulus tend to reveal the sensitivity peaks at category borders.)
• It should be always made clear if the study is aimed to test for development of perceptual categories and not conflating that to something else, such as “language learning” or “language acquisition” without considering if the experimental design used is adequate to catch whether language acquisition is taking place.

Some day, I wish to see that we’d have some solid scientific evidence of how acquiring second language pronunciation happens, and also better understanding of how to facilitate the development of pronunciation when teaching. I suspect that there might exist some training paradigms that actually improve one’s perception, accent and speech intelligibility, but I can’t help thinking that most of the training studies are missing the language part of the equation, focusing only in acoustics.