Demystifying Memory Management

Memory, in computing, is a precious resource. There’s never enough of it to go around, especially when it comes to intensive uses like gaming or video editing software. Thus, it’s important as a software developer to use as little memory as possible in your code, and make sure to clean up what you’ve used after yourself to make it available to other programs. And as anyone who’s ever run into a StackOverflowError or OutOfMemoryError can tell you, it’s possible to mess up your memory usage badly enough that things start breaking.

Different languages have used a variety of approaches to help with managing memory and avoid some of the worst failures that can happen. This article will cover a few of those approaches, and dive a bit into the reasons that they’re necessary.

The Stack and the Heap

To understand when and why we need to manage memory, it’s important to understand what the stack and the heap are. The stack (sometimes called the execution stack or call stack) is a fixed-size region of memory using a last in, first out (LIFO) approach. When a new function runs, some memory is allocated on top of the stack, and then is deallocated when the function ends and the data it’s using goes out of scope. This type of memory management is trivial because by definition, every variable being used by the current function is all in one spot, on top of the stack. So where does the difficulty lie?

Well, because the stack is of a fixed size, we have to be careful about running out of space. If a program uses too much memory for one function, or has too many nested function calls, then the stack will fill to the brim, causing a stack overflow and program crash. To avoid that happening is why the heap exists. The heap is an amorphous zone of memory that’s of variable size, and only bounded by the physical limits of the hardware. If we have a large object that would take up too much stack space, we can allocate an area of memory in the heap to store it, and keep a reference to it on the stack with the rest of the function’s data. Generally speaking, variables that are small and of known size can be stored in the stack, such as fixed-length arrays, primitives, and other small objects. More complex objects and ones of unknown size get stored on the heap.

Now, here’s where the need for careful management arises. Let’s say that a function runs, it defines some variables which get stored on the heap (with references to them on the stack), and then the function terminates. All of its data on the stack gets removed, including those references, but the memory in the heap is still occupied, and is thus unusable by any other code. This is called a memory leak. Every time that function runs, more and more memory gets occupied, until there’s none available for any software to use.

Thus, the real problem of memory management is to make sure that when an object on the heap is no longer needed, the space it is occupying gets deallocated to free it up for other uses. It’s not a difficult task per se, but there’s inevitably tradeoffs that have to be made in terms of performance, memory safety, and work for the programmer. Each of the methods below approaches the problem in its own way.

Manual Allocation

This is the simplest method of managing memory used on the heap, and tends to be found in older, more low-level languages. We’ll use C++ as an example. In that language, any object declared inside a function is placed on the stack by default. The programmer has to explicitly state that an object will be stored on the heap, and then remove that object before the function terminates. A simple example:

void MyFunction( … ) {
MyObject obj; // this is placed on the stack by default
MyObject* obj2 = new MyObject( … ); // using the asterisk here indicates that this is a reference, and causes this object to be placed on the heap
delete obj2; // we have to perform a manual deletion before the function ends in order to free up the memory on the heap
}

The main issue here should be obvious; there’s nothing stopping someone from failing to delete an object they’ve created on the heap, which would cause a memory leak. Code with a memory leak will compile fine (although there are compiler options that will detect leaks), and will often run without issue (at least for a while). If the amount of memory in question is small, it may not even cause issues on a local computer, but running it on a server that is rarely or never restarted will eventually cause the server to fail.

In addition to the problem of forgetting to delete objects, the programmer has to understand when to declare objects on the stack versus the heap. If they decide to skip the hassle of manually allocating and deleting, and just leave everything on the stack instead, it increases the chances of a stack overflow.

And even if they are using the memory correctly, ensuring that objects are created on the stack or heap as appropriate and deleting objects before the function terminates, doing so requires a lot of mental bandwidth. Many people wanted a more user-friendly solution, which is why the next approach was created, one with a great deal more automation.

To summarize, the pros of manual allocation:

• Simple
• No system overhead
• Gives programmer the ability to control where variables are stored

The cons:

• Large room for user error
• Adds extra work for the programmer

Garbage Collection

Garbage collection was created to take the task of manual allocation and deletion away from the user. Languages that use this method will automatically determine whether an object should exist on the stack or heap, and will keep the heap as empty as possible so that there’s always open memory available. They do this by periodically checking the heap for any objects that are no longer being used, and clearing out those areas of memory. Garbage collection tends to be used by more modern, higher-level languages, like Python and Java — we’ll use Python as an example.

In Python (at least CPython, the most common implementation), all objects within a function are created on the heap, with only references being kept on the stack. When a function terminates, all those references are deleted. The garbage collector keeps a count of how many references exist to an object, so when no more references exist, it can clean out that block of memory. Thus, the developer doesn’t have the mental overhead of keeping track of memory to deallocate, and doesn’t have to worry about their own errors causing memory leaks..

However, this comes at a cost. Generally, a garbage collector has to occasionally halt the execution of the program in order to delete unused objects, reducing performance. There are also edge cases like circular references (an array that contains itself, for example) that require special handling, which further impacts performance. These impacts aren’t huge, but they can add up, especially on large scales. Furthermore, objects stored on the stack are much faster to access, because they’re readily available and don’t have to be fetched from the heap, but a garbage collected language removes the user’s ability to choose what is stored on the stack, preventing the user from benefiting from this efficiency.

Again, to summarize the pros of garbage collection:

• No additional work for the programmer
• Highly memory-safe

The cons:

• Performance is impacted
• Edge cases like circular references can cause problems
• Can’t manually control what is stored on the stack versus the heap

Reference Management

This is a relatively new approach to handling memory, used by languages like Rust. With reference management, each object on the heap can only have one reference on the stack at a time, and when that reference is deleted, the object on the heap is immediately deleted as well. Contrast this to garbage collection, where many references are allowed to a given object, but a separate service is needed to keep track of them and periodically run cleanup.

Enforcing a single reference in this way allows for memory-safe programming without performance drawbacks, but can require some creative work on the part of the programmer. Let’s look at some examples in Rust to see why.

fn my_function( … ) {
let foo: i32 = 17; // this is a primitive 32-bit integer, so it’s placed on the stack
let foobar = MyObject::new( … ); // this object is defined using the keyword ‘new’, so it goes on the heap
}

Here, we see something similar to the C++ code from before — some objects can be declared on the stack, and a special keyword ‘new’ is used to define objects on the heap. But what about when we want to use these variables elsewhere — as arguments for another function, for example?

fn my_function( … ) {
let obj: i32 = 17;
let obj2 = MyObject::new( … );
some_function(obj2);
some_other_function(obj2); // This causes an error
}

So why does this fail? Remember, we said only one reference to an object could exist. Well, the ‘obj2’ variable is the only reference to a MyObject on the heap. When that reference is passed to ‘some_function’, it is no longer usable in the current scope. There’s ways to get around this, such as “borrowing” a variable by creating a meta-reference on the stack pointing at the original variable, but these add complexity and also come with their own set of rules regarding mutable and non-mutable borrowing. There’s definitely a learning curve associated with managing memory in this way, but the tradeoffs can be worth it.

The pros of using reference management:

• Memory safety of a garbage-collected language
• Performance of a language using manual allocation

The cons:

• Enforces patterns of object usage that would be unfamiliar to most developers

Conclusion

As we said, there are three metrics that memory management has to balance- memory safety, performance, and programmer effort. Manual allocation has no guarantee of memory safety, but is highly performant, and takes a moderate amount of work on the part of the user. Garbage collection is memory safe and trivial to use, but can impact performance to some degree. And finally, reference management is memory safe and performant, but requires a significant shift in how one handles variables in a program. Each of these paradigms has its place, and hopefully this article gives you a better sense of which best fits the needs of your use case.

If you’re interested in solving interesting problems alongside a passionate group of engineers, apply to join our team and help us build the future of conversational AI!