Closures, part 2
I’d like to spend some time studying closures. It’s a tricky topic. Before everything else, I’d like to admire their syntax: It’s way more elegant then python’s keyword-based lambda ugliness. Just compare:map(|x| x > 0, somelist) and map(lambda x: x > 0, somelist).
I like how the idea of closures conjoin with the ownership model. The ownership transfer/borrow into a closure is amazing, and it completely surpasses Python inconsistency with modification of immutable/hashable containers. Just look on the tuple of two sets, where you can change content of each set:
>>> a=(set(),set())
>>> a[0].add(1)
>>> a
(set([1]), set([]))Tuples are immutable, huh-huh… One can live with this, but if we can do better, why not?
The next thing to talk is readability of the language. I have had a lunch discussion with my colleague, a good programmer with strong interest in Computer Science. He does not know a lot about Rust, but have an extensive knowledge of other ‘noble’ languages — R, Haskell, Lisp, Schema, etc.
He said that he appreciate clarity of function signatures, but dislike ‘&’ overuse. In his opinion default mode for type declaration should be ‘reference’, and ownership transfer should be specified by special character or keyword. I couldn’t have took sides on this topic, but I felt that for me, excessive amount of ‘&’ made Rust code (especially in function signatures with generics, which are enclosed with angular brackets) looks like incorrectly rendered HTML-code. &<Example>&…, like that.
Playing with closures
Last time I hadn’t played enough with closures.
I just started when I found a wasp nest. I tried to return closure from a function. My first function was nice and worked as expected:
fn foo<T>(f: T)->T where T: Fn()->i32 { f }I just returned the same closure I got in my function. Actually, that was too strict, and the simplest form of that function looks like this: fn foo<T>(f: T) -> T {f}.
Things shifted into aforementioned nest when I decided to return another closure, different from original.
My best attempt:
fn test<T>() -> T where T: Fn()->u32 {
|| 0
}fn main() {
println!("{}", test()());
}
There are two complicated errors:
error[E0308]: mismatched types
--> src/main.rs:2:3
|
1 | fn test<T>() -> T where T: Fn()->u32 {
| - expected `T` because of return type
2 | || 0
| ^^^^ expected type parameter, found closure
|
= note: expected type `T`
found type `[closure@src/main.rs:2:3: 2:7]`error[E0619]: the type of this value must be known in this context
--> src/main.rs:7:20
|
7 | println!("{}", test()());
| ^^^^^^^^
I can’t do this. Ok, ok, I got it. But googling pointed me to this question with amazing answers. Apparently, it’s something crazy: ‘abstract return types’, which was thought to be implemented in Rust, but wasn’t.
This simple problem ‘how to return a closure from a function’ stuck me deep. I may scold Python for ‘lambda’ ugliness, but at least you are free to return lambdas from any function without any issues. Lambdas can be used in almost any context as value, hashable key, etc. Rust is more restrictive here.
Why?
After few days of thinking, I start to understand the problem. What is a closure from the memory allocator standpoint? It’s a pointer to call. We can’t have the actual code be placed on stack (the most of the modern OS’es forbid this due to security reasons — we don’t want stack overruns/underruns to produce a malicious code on stack to run).
So lambda (or any other function) may be only a pointer. At the same time, we need to know a signature for the function to call. It need something on stack before calling, it need something to be removed from the stack after calling.
From the type system point of view we need to know precisely types in and out. Borrow/ownership model wants its information too.
It still does not ring to me.
If I write this signature for the function:
fn test() -> Fn()->u32 {
|| 0
}I expect it to work. We return something which a function with no arguments which move out (returns) u32 upon call. Still, it does not compile:
error[E0308]: mismatched types
--> src/main.rs:2:5
|
1 | fn test() -> Fn()->u32 {
| --------- expected `std::ops::Fn() -> u32 + 'static` because of return type
2 | || 0
| ^^^^ expected trait std::ops::Fn, found closure
|
= note: expected type `std::ops::Fn() -> u32 + 'static`
found type `[closure@src/main.rs:2:5: 2:9]`error[E0277]: the trait bound `std::ops::Fn() -> u32 + 'static: std::marker::Sized` is not satisfied
--> src/main.rs:1:16
|
1 | fn test() -> Fn()->u32 {
| ^^^^^^^^^ `std::ops::Fn() -> u32 + 'static` does not have a constant size known at compile-time
|
= help: the trait `std::marker::Sized` is not implemented for `std::ops::Fn() -> u32 + 'static`
= note: the return type of a function must have a statically known size
This, actually, more interesting. Closures aren’t functions because they borrow/move values. Let’s try it with functions:
fn test() -> Fn()->u32 {
fn x() ->u32 {0};
x
}cargo builderror[E0308]: mismatched types
--> src/main.rs:3:5
|
1 | fn test() -> Fn()->u32 {
| --------- expected `std::ops::Fn() -> u32 + 'static` because of return type
2 | fn x() -> u32 {0}
3 | x
| ^ expected trait std::ops::Fn, found fn item
|
= note: expected type `std::ops::Fn() -> u32 + 'static`
found type `fn() -> u32 {test::x}`error[E0277]: the trait bound `std::ops::Fn() -> u32 + 'static: std::marker::Sized` is not satisfied
It sounds like ‘you fool’. I’m fool. I have no idea what’s this and what it talking about. But I see something funny: Expected Fn, found fn.
hah! Let’s try it!
fn test() -> fn()->u32 {
fn x() -> u32 {
0
};
x
}
fn main() {
println!("{}", test()());
}And, finally, it worked! 😊
I got my 0 in output. Therefore, two days of thinking was not a complete waste. My reasoning was right, information of signatures and ownership is enough to Rust to deal with ‘return a function’.
Can we do the same with a closure?
We can! This is a perfectly vaid Rust code:
fn test() -> fn()->u32 {
|| 0
}So, my struggle insofar was with this strange Fn trait. I’ll leave it aside for now, because I want to learn something, not to be puzzled to death.
Moreover, this is a valid code:
fn test(x: fn()->u32) -> fn()->u32 {
|| 0
}But if I try to use ‘x’, I’ll run into trouble:
fn test(x: fn()->u32) -> fn()->u32 {
let l = x();
|| l
}Nah-nah:
error[E0308]: mismatched types
--> src/main.rs:3:5
|
1 | fn test(x: fn()->u32) -> fn()->u32 {
| --------- expected `fn() -> u32` because of return type
2 | let l = x();
3 | || l
| ^^^^ expected fn pointer, found closure
|
= note: expected type `fn() -> u32`Even if I shrink back, the same error will persists:
fn test() -> fn()->u32 {
let l = 1;
|| l
}I’ve changed almost nothing compare to the working example, but this ‘almost nothing’ made the compiler unhappy.
The change was in the fact, that it become closure. Can I do it with no closure part? (I’ll return to the closure issue later):
fn test() -> fn(u32)->u32 {
|x| x+1
}
fn main() {
println!("{}", test()(3));
}Yes, I can. It compiles and works as expected.
So, the ‘closure’ part of the closure is the issue. I found this post which discussed in length for this specific topic. Author had come to conclusion that closures aren’t functions and we need to use heap-allocated memory to manage closure as a dynamic object with unknown (at a compile time) size.
Why Rust couldn’t handle this in a compile time?
… Moreover, what is a closure?
|| 0 is a zero (trivial) closure — it contains a single constant. The same is applicable to any number of constants. It does not capture anything, and that’s why Rust ‘converts’ it to the function.
|| x, where x is been captured from current state, is a completely different thing. Some closure magic is happens here.
A closure need a place to store a value if it’s not a trivial closure. It requires memory. Where this memory is coming from? If I use a closure inside a context of my function, it ‘knew’ what it had just captured and how large it is. This information is stored on function stack and dies as soon as function terminates. That’s why we can pass closures into other functions: Compiler knew a location of data for closure on stack and can easily pass a pointer to a closure function with a pointer to closure data… (Hm… Is a closure a single pointer or it’s a two pointers?). Anyway, at call time compiler knew all information about passed closure, and calling function is responsible for deallocating it’s memory (closure’s lifetime is the same, as the function, and the closure is automatically deallocated at the end of the calling function).
Quite opposite happens in the case of ‘returning a closure’. A closure is a code (fine) with some data on stack. How large are the data? Can a caller predict how large those data are? Probably, not. I struggle to come up with example, but I have a gut feeling, that it’s not a compile time possible task to guess memory consumption of a complicated closure returned from some function. Moreover, even a pointer to those data was returned, where whole data are stored? Should they be returned as value? Ultra-big ugly value. That may works if we can deduct a closure size in a compile time. As I said, I have a gut feeling, that it’s not possible. Therefore, we have ‘something’ with unknown size, and such thing should not be on stack flying around between functions.
A short version of that runt:
We can return functions and anonymous functions which have no data to enclosure. We couldn’t return closures with ‘enclosed’ data as a size of those closures is hard to calculate at a compile time.
I got a better feeling of what are closures in Rust. In the next chapter I’ll finish iterators and come back to IO part to learn the path of fd inside of the Rust program.
