Update: This post has been updated to account for pings being round trip times while distances are only one way. The original post failed to account for this in the last two tables, thanks to Steve for pointing this out.
Running a global network of servers for GeoIP application testing leaves you with a lot of servers and some interesting questions, occasionally an interesting combination. I found myself asking if we could compare ping times to physical distances, to see how efficient the internet was, and to confirm my suspicion that transferring data between Sydney Australia and Fremont California would be faster, mile per mile, than transferring between Boston and Fremont. My reasoning was that Australia → United States is one long cable, whereas within the US it would be switched frequently, which is slower.
First we generated a script to ping every city in our network from every other city in our network (Boston → New York and New York → Boston would both be executed). This took something like ten hours (we’ve since smartened up and now execute the test in parallel). Our results look something like this:
Ping between cities
This table shows us what we expected. Sydney is far away from most of our other servers, so the ping time is high. Our Fremont server is incredibly well connected with a top tier provider so it has good routes. The Baltimore → New York ping may raise your eyebrows, it certainly caught our attention. A quick look at the traceroute shows:
traceroute to newyork.wonderproxy.com (184.108.40.206), 30 hops max, 60 byte packets 1 220.127.116.11 (18.104.22.168) 0.551 ms 0.557 ms 0.604 ms 2 4xe-pc400.vcore1-dc1.balt.gandi.net (22.214.171.124) 0.606 ms 0.661 ms 0.693 ms 3 xe3-4-core4-d.paris.gandi.net (126.96.36.199) 97.342 ms 97.325 ms 97.272 ms 4 p251-core3-d.paris.gandi.net (188.8.131.52) 123.541 ms 123.547 ms 123.528 ms 5 linx.ge1-0.cr01.lhr01.mzima.net (184.108.40.206) 119.101 ms 119.085 ms 119.017 ms 6 te0-5.cr1.nyc2.us.packetexchange.net (220.127.116.11) 181.881 ms 176.377 ms 176.349 ms
Our provider in Baltimore seems to be routing all of their traffic through their central datacenter in Paris, rather sub-optimal (we’ve opened a ticket).
Next we needed to determine how far apart these cities were, that being the other half of the equation. We worked with the city center when more specific information on location wasn’t available, and generated the approximate latitude and longitude for every server in our network using publicly available sources. We then used the excel calculation script from Movable Type Scripts to determine distances, and came up with this:
Distance between cities
Finally, we merged the two tables, did some math and came up with this:
Miles per Milisecond
Here things start to look a bit better for several of the connections. In the first chart which looked at raw ping times Sydney faired poorly. Here, accounting for the extreme distance between Sydney and the majority of our network we see that, mile for mile, it’s actually doing quite well. Other links like Paris → Milan that were looking quite good previously are now exposed as being rather inefficient.
While we’re examining the efficiency of our network, this is the chart we’ll use. Simple ping times tell a story (we use smoke ping to monitor consistency and packet loss), but not the whole story. From this we get a more realistic view of hour our connections are performing.
One last experiment
Networks are fast, but just how fast? In a vacuum Light travels roughly 186,000 miles/second, or 186 miles/millisecond. Light travels slower through fiber optics, on average about 35% slower, which gives us ~120.9 miles/millisecond. Let’s look at the speed of pings across our network, as a percentage of the theoretical maximum:
Network speed as a percentage of the speed of light
For this comparison to be fair cables would need to be run directly between cities, which is clearly not the case. We still think it’s interesting. Hitting 68% of the theoretical best speed is quite remarkable.