Introduction

MTR, short for My TraceRoute, is a powerful tool for troubleshooting network issues by providing detailed information on the path that packets take from the source to the destination. It combines the functionality of both traceroute and ping, and provides a real-time view of the route that packets take to reach their destination, as well as information on packet loss and latency at each hop.

MTR is a powerful tool for troubleshooting network issues, especially when it comes to anycast IP addresses. It provides detailed information on the path that packets take to reach the destination, including packet loss and latency at each hop. This can help identify problems and provide valuable information for resolving network related issues.

You can use MTR to gather path information to your service IP or backend communication addresses (e.g from your backend server). Filtered IP addresses are typically Anycast IPs as such the route you receive to them may differ depending on your source network (ASN / IP) and their routing preference at the time.

An anycast IP address is a type of IP address that is assigned to multiple devices in different locations. When a packet is sent to an anycast IP address, it is routed to the nearest device that has that IP address. This provides a number of benefits, such as improved performance and increased availability, but it can also make troubleshooting more complex.

To troubleshoot issues related to an IP address, one can use MTR to see the path that packets take to reach that IP address. This can provide valuable information on where packets are being dropped, or where they are experiencing high latency.

To run MTR on linux to an anycast IP address, one can use the following command:

mtr [anycast IP address]

Or on windows you can use traceroute or the WinMTR.exe application.

This will start a continuous MTR session and display the results in real-time. The output will show the hop number, the IP address of the hop, the hostname of the hop (if available), the packet loss, and the round-trip time for each hop.

The first column of the output shows the hop number, with the destination being hop number 0. The second column shows the IP address of the hop. The third column shows the hostname of the hop, if available. The fourth column shows the percentage of packets that were lost at that hop. The fifth column shows the round-trip time in milliseconds.

One can use the information provided by MTR to identify any issues that may be affecting the performance of the network. For example, if there is high packet loss at a particular hop, it may indicate that there is a problem with the device at that hop. Similarly, if the round-trip time is high at a particular hop, it may indicate that there is a problem with the network at that hop.

What does packet loss mean

Packet loss on gateways refers to the situation where packets are not successfully transmitted from one network to another through a gateway. A gateway is a device that connects two networks and is responsible for forwarding packets from one network to the other. If packets are not successfully transmitted through a gateway, it means that they are being dropped or discarded.

This will show in a MTR as packet loss % and number of packets lost. MTR may show packets being lost for many reasons including:

  • Packets being filtered by a firewall at that hop or gateway ratelimiting
  • Firewall rules deployed by us or a customer to limit traffic to a service (this is particularly common in UDP based traceroute application)
  • Overloaded network port / gateway
  • Routing issues: Packets may be taking a bad path within ours, an upstream or your upstreams network

Packet loss that occurs between the source and destination but does not affect the end service is commonly rate limiting at the gateway level. Devices responsible for routing traffic ("gateways") commonly employ rate limiting to prevent certain types of network attacks, such as ping floods and ping of death or to reduce resource expenditure. These types of attacks involve sending a large number of ICMP echo request (ping) packets to a target with the intention of overwhelming the target's network resources. By limiting the rate at which ICMP packets are processed, gateways can help prevent these types of attacks from being successful.

Gateway limiting of ICMP can be implemented in a number of ways, such as:

  • Configuring firewalls to limit the rate of incoming ICMP packets
  • Using rate-limiting tools to limit the rate of ICMP packets on a network
  • Limiting the rate of generated ICMP responses to UDP and/or TCP ping requests

A gateway limiting ICMP responses does not affect your end service and is fairly normal. We also implement ratelimits in this area and you may see us limit our responses to both UDP and ICMP pings at fairly low limits.

What is high packet loss

High packet loss is a situation where a large percentage of packets are being lost. This can be caused by a number of different issues, such as:

Typically high packet loss is anything above 5% and is a sign of a problem. If you are seeing high packet loss, you should investigate the cause of the packet loss and resolve the issue.

TCP is typically fairly resilient against low rates of packet loss, however it should be advoided for latency reasons.

What does high latency mean

Latency is the time it takes for a packet to travel from the source to the destination and back. Latency is measured in milliseconds (ms). Latency is commonly measured using the ping command, which sends a packet to a target and measures the time it takes for the packet to be received back from the target.

Latency is commonly measured using the ping command, which sends a packet to a target and measures the time it takes for the packet to be received back from the target.

You will get a reading for latency with a tool such as MTR however this may differ from the ping result depending on the protocol used and method of resolution.

Nameserver resolution, that is conversion of IP addresses in the output to DNS host names may increase reported latency in tools such as ping and MTR (this depends on implementation and OS).

What if the packet loss or bad route is between you and my backend server?

Packet loss on your backend server is not something we can typically do much about. In most cases its something your backend provider is best placed to resolve. You should contact your backend provider for assistance with this.

You should also check the health of your server and network for any active incidents.