The Stack Exhaustion Ruby Fail on Huge Complex Strings

pancake
3 min readAug 6, 2014

A month ago I spend some time tickling the limits of the RubyVM. First of all I started reading the source code which is in C and focused on all the numbering conversion code mainly because it looked complex and powerful. Ruby aims to impose very few limitations, so, when handling numbers it quickly jumps into the BigNum world when the number can’t fit in 64bits.

That’s nice, because the language hugely simplifies working with numbers, it even supports complex, rational and other kind of numbers properly. But this tends to make the implementation more complex and imposes some performance penalty. This turns to be pretty powerful and it would be great if machines could own infinite memory to handle this. But reality differs a bit, and machines own a limited memory.. and to optimize some of those cases RubyVM uses alloca() to store some of the intermediate operations.

This is bad. Very bad. Using alloca() is a very bad idea for various reasons, first of all because there’s no portable way to determine its boundaries and depending on the implementation it can handle negative allocations, never return NULL, etc.. so, it’s usually a root of stack exhaustion crashes, and under some situations it is possible to execute code.

Disclosure

I took this issue as a serious business and decided to contact the Ruby Security team in order to make a responsible disclosure.. But after waiting a month without any response.. I decided to use full disclosure because I believe this is by far more responsible than waiting more time for an answer or a new release.

The crash

The crash can be reproduced on with this simple line:

Complex(‘8===D’*9999999,0)

When Complex gets a very long string in any of the two arguments it produces an out of bounds access in the stack. Running the same line with different lengths produces crashes at different places in the code, all of them produced by invalid memory accesses.

Some of the places it may crash are in calls (where the pc is pushed in the stack), read and write memory accesses. Depending on the memory layout and the compiler used, the alloca may miss-check the parameter and result in growing the stack enough forward (or backward if the value is negative) to read/write or pivot the stack to a newly controlled area and perform code execution.

Checking out the bug tracker I found the following issue:

https://bugs.ruby-lang.org/issues/9608

This is exactly the issue I found and it doesn’t seems to have any CVE associated. In the git repository this fix has been added in May 2th:

The 2.1.2 release (the last one in the website) is from May 9th, 7 days after this commit, but the release doesn’t includes this commit. (???)

This is a security vulnerability that should be properly handled by CVE and a new release should be done in order to make all users and customers aware of the problem which not only affects 2.1.2, it crashes on all previous versions (tested on 1.9.3 and 2.0).

http://www.cvedetails.com/vulnerability-list/vendor_id-7252/product_id-12215/Ruby-lang-Ruby.html

That’s the ASAN backtrace of one of my latests tests with ruby-2.1.2 (latest release) running the following script multiple times (ASLR enabled), and it confirms the theory that this vulnerability can cause arbitrary code execution rather than just a simple DoS.

Hopefully, making this public will arise the concern for such problems and make developers be aware of the long strings in their applications/services and the Ruby team push for a new release.

—pancake

--

--

pancake

Holding stuff in my head since 0x7bf 1PANCAkeVfUxkGyfBGLPmsRGgYwtmrkfsk