On the Internet, routing is easy. You throw a packet out with an address stamped on it, and the recipient figures out how to get it closer. It mostly Just Works.
If you want anonymity, that scheme doesn’t work. Early on, Joseph Poon challenged me with the concept of a completely decentralized lightning network where (well-connected) phones could be first-class members. And you can’t work with Adam Back and Greg Maxwell for long without gaining a deep appreciation for the importance of privacy.
Thus it was important that this be implemented from the very first versions, as it affects almost everything else (especially route discovery!). The result is that Christian Decker and Olaoluwa Osuntokun adapted the previously-published Sphinx scheme pointed out by Laolu for use in lightning.
The construction ensures that no node can tell how many hops have already passed, or how many remain; only the previous and next hops. How to achieve that robustly is a little complicated, but this spec spells it out in detail.
It’s worth noting that both c-lightning and lnd already have compatible implementations, and there were minimal changes in Milan, so this spec is currently a little lazy: it’ll be cleared up into a step-by-step process like that we’ll see when we cover BOLT #8, and test vectors added. It also fails to specify the exact contents of the hop payload (fixed soon, see https://github.com/lightningnetwork/lightning-rfc/issues/54), and I think we’ll be tweaking the format of the failure messages (https://github.com/lightningnetwork/lightning-rfc/issues/53).