From TTL to hop distance

If you want to know how far your customers are…

Hop distance heatmap shows most “hot” distance for exact period.

What is TTL?

TTL stands for Time To Live and for IPv4 protocol was described in RFC791 as:

Time to Live: 8 bits

This field indicates the maximum time the datagram is allowed to remain in the internet system. If this field contains the value zero, then the datagram must be destroyed. This field is modified in internet header processing. The time is measured in units of seconds, but since every module that processes a datagram must decrease the TTL by at least one even if it process the datagram in less than a second, the TTL must be thought of only as an upper bound on the time a datagram may exist. The intention is to cause undeliverable datagrams to be discarded, and to bound the maximum datagram lifetime.

Because requests always take time lower than 1 second, TTL is lowered by 1 on each host it passes and in real life doesn’t have anything in common with time.

So for IPv6 this header was changed from TTL to Hop Limit and now we can see the right description of it’s behaviour in RFC8200:

Hop Limit 8-bit unsigned integer.

Decremented by 1 by each node that forwards the packet. When forwarding, the packet is discarded if Hop Limit was zero when received or is decremented to zero. A node that is the destination of a packet should not discard a packet with Hop Limit equal to zero; it should process the packet normally.

Hop distance

Now we know that every packet has its limit and from the header field size we know that maximum hosts that packet can pass is 255.

Different systems have different default TTL values, but luckily there isn’t much of them and we can calculate hop distance like this:

if ( TTL > 128 )
    hop = 255 - TTL
else if ( TTL > 64 )
    hop = 128 - TTL
else if ( TTL > 32 )
    hop = 64 - TTL
else
    hop = 32 - TTL

Hop distance or hop count is the number of hosts (routers) on the path to the destination of IP packet.

High number of hops isn’t bad, but it has its influence on network latency.

Latency

Network latency is a time interval between request packet send and response packet receive, so it’s round trip time of the packet that we can measure with tool like ping.

The more hops we have, the more variative latency become. Different wires have different latency, routers add latency while packet-forwarding, chances to get on congested network link arise…

Use-cases

So can we use hop distance for anything else than just theoretical thinking about its influence?

Deny mobile Internet sharing

Mobile network operators uses hop distance to deny internet sharing on plans that don’t allow it. They just drop all the packets that have hop distance more than 0.

Find out async paths

To find out hosts on the way of the packet to the destination we can use traceroute, but return path could be different.

Linux version of traceroute has ‘ — back’ flag:

Print the number of backward hops when it seems different with the forward direction. This number is guessed in assumption that remote hops send reply packets with initial ttl set to either 64, or 128 or 255 (which seems a common practice). It is printed as a negate value in a form of ‘-NUM’ .

Async paths can help to find if the destination host have anti-DDoS service connected or using internal filtering system for outgoing connections.

Anycast DNS optimization

Anycast DNS helps response to DNS request faster by deploying DNS servers in different regions that helps reduce network latency. Watching for hop distance in a region can help detect degradation of connectivity.

Service Level Objective

Case for anycast DNS could be applied for any other distributed system that tries to get closer to the customer.

SLO for distributed system can help to watch for network health in terms of hop distance.

Routing

Some routing protocols, such as Routing Information Protocol(RIP), use hop count as their sole metric. You can read RFC1058 for more information about RIP using hop.

Proof-of-Concept

As a Proof-of-Concept I wrote some scripts for tcpdump parsing, that you can find at hopmon repo.

Awk script will parse tcpdump output and show time, hop distance and remote IP of the client.

tcpdump -Qin -nlv "tcp[tcpflags] & tcp-syn != 0"|awk -f ttl.awk

If you want to collect hop distance to prometheus and create charts like in this post, you should use hopdistance.sh that will feed your prometheus node-exporter textfile-collector.

Conclusion

Hop distance isn’t essential metric to think about, but it could be interesting for some projects to watch for it.

If you know other use-cases for hop distance, feel free to comment and i’ll update the post.

Running tcpdump on production systems could add some overhead, so to capture TTL more efficiently, it’s better to take a look at eBPF tracing. I’ve created an issue in iovisor/bcc project, so feel free to join.