WonderProxy Blog

February 9, 2011

Miles per Milisecond, a Look at the WonderProxy Network

Filed under: Uncategorized — Paul Reinheimer @ 2:33 pm

Update: This post has been updated to account for pings being round trip times while distances are only one way. The original post failed to account for this in the last two tables, thanks to Steve for pointing this out.

Update 2: You can now view updates data with ping time between cities at our new WonderNetwork site.

Running a global network of servers for GeoIP application testing leaves you with a lot of servers and some interesting questions, occasionally an interesting combination. I found myself asking if we could compare ping times to physical distances, to see how efficient the internet was, and to confirm my suspicion that transferring data between Sydney Australia and Fremont California would be faster, mile per mile, than transferring between Boston and Fremont. My reasoning was that Australia → United States is one long cable, whereas within the US it would be switched frequently, which is slower.

First we generated a script to ping every city in our network from every other city in our network (Boston → New York and New York → Boston would both be executed). This took something like ten hours (we’ve since smartened up and now execute the test in parallel). Our results look something like this:

Ping between cities

Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 239.38 35.64 79.68 104.70 141.15 174.71 97.71 251.07 114.11
Brisbane 238.32 221.60 174.38 350.12 357.21 245.89 335.79 33.45 346.92
Dallas 34.53 220.14 44.00 140.00 168.00 36.80 130.94 221.72 127.74
Fremont 79.39 176.07 44.57 167.85 214.59 78.43 176.22 184.91 167.16
Milan 104.84 339.54 139.86 170.73 67.08 112.17 21.61 344.70 11.17
Moscow 140.82 366.42 169.04 209.00 67.07 131.29 56.74 387.09 60.29
New York 174.61 246.87 39.46 77.49 111.60 131.34 78.33 319.33 101.03
Paris 97.75 337.47 131.79 175.19 21.29 56.13 77.31 348.84 108.06
Sydney 262.50 42.98 222.25 191.96 345.21 376.57 261.24 354.16 358.75
Zurich 102.18 346.71 127.59 173.61 11.25 60.32 100.24 108.25 345.05

This table shows us what we expected. Sydney is far away from most of our other servers, so the ping time is high. Our Fremont server is incredibly well connected with a top tier provider so it has good routes. The Baltimore → New York ping may raise your eyebrows, it certainly caught our attention. A quick look at the traceroute shows:

traceroute to newyork.wonderproxy.com (69.147.239.239), 30 hops max, 60 byte packets
 1  173.246.103.253 (173.246.103.253)  0.551 ms  0.557 ms  0.604 ms
 2  4xe-pc400.vcore1-dc1.balt.gandi.net (173.246.96.33)  0.606 ms  0.661 ms  0.693 ms
 3  xe3-4-core4-d.paris.gandi.net (217.70.176.233)  97.342 ms  97.325 ms  97.272 ms
 4  p251-core3-d.paris.gandi.net (217.70.176.253)  123.541 ms  123.547 ms  123.528 ms
 5  linx.ge1-0.cr01.lhr01.mzima.net (195.66.225.15)  119.101 ms  119.085 ms  119.017 ms
 6  te0-5.cr1.nyc2.us.packetexchange.net (69.174.120.89)  181.881 ms  176.377 ms  176.349 ms

Our provider in Baltimore seems to be routing all of their traffic through their central datacenter in Paris, rather sub-optimal (we’ve opened a ticket).

Next we needed to determine how far apart these cities were, that being the other half of the equation. We worked with the city center when more specific information on location wasn’t available, and generated the approximate latitude and longitude for every server in our network using publicly available sources. We then used the excel calculation script from Movable Type Scripts to determine distances, and came up with this:

Distance between cities

Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 9553 1216 2448 4211 4853 171 3819 9845 4122
Brisbane 9553 8375 7138 10162 8795 9688 10349 457 10143
Dallas 1216 8375 1466 5355 5788 1376 4956 8638 5255
Fremont 2448 7138 1466 5983 5914 2564 5596 7481 5857
Milan 4211 10162 5355 5983 1429 4040 400 10348 137
Moscow 4853 8795 5788 5914 1429 4694 1554 9060 1371
New York 171 9688 1376 2564 4040 4694 3648 9993 3952
Paris 3819 10349 4956 5596 400 1554 3648 10600 304
Sydney 9845 457 8638 7481 10348 9060 9993 10600 10355
Zurich 4122 10143 5255 5857 137 1371 3952 304 10355

Finally, we merged the two tables, did some math and came up with this:

Miles per Milisecond

  Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 79.81 68.24 61.44 80.44 68.76 1.96 78.17 78.43 72.25
Brisbane 80.17 75.59 81.87 58.05 49.24 78.80 61.64 27.33 58.48
Dallas 70.42 76.09 66.63 76.50 68.90 74.78 75.70 77.92 82.28
Fremont 61.67 81.08 65.78 71.29 55.12 65.38 63.51 80.92 70.08
Milan 80.33 59.86 76.58 70.09 42.61 72.03 37.02 60.04 24.52
Moscow 68.92 48.00 68.48 56.59 42.61 71.51 54.78 46.81 45.48
New York 1.96 78.49 69.75 66.18 72.40 71.48 93.14 62.59 78.24
Paris 78.14 61.33 75.21 63.88 37.58 55.37 94.38 60.77 5.63
Sydney 75.01 21.26 77.73 77.94 59.95 48.12 76.50 59.86 57.73
Zurich 80.68 58.51 82.37 67.47 24.36 45.46 78.85 5.62 60.02

Here things start to look a bit better for several of the connections. In the first chart which looked at raw ping times Sydney faired poorly. Here, accounting for the extreme distance between Sydney and the majority of our network we see that, mile for mile, it’s actually doing quite well. Other links like Paris → Milan that were looking quite good previously are now exposed as being rather inefficient.

While we’re examining the efficiency of our network, this is the chart we’ll use. Simple ping times tell a story (we use smoke ping to monitor consistency and packet loss), but not the whole story. From this we get a more realistic view of hour our connections are performing.

One last experiment

Networks are fast, but just how fast? In a vacuum Light travels roughly 186,000 miles/second, or 186 miles/millisecond. Light travels slower through fiber optics, on average about 35% slower, which gives us ~120.9 miles/millisecond. Let’s look at the speed of pings across our network, as a percentage of the theoretical maximum:

Network speed as a percentage of the speed of light

  Baltimore Brisbane Dallas Fremont Milan Moscow New York Paris Sydney Zurich
Baltimore 66.02 56.44 50.82 66.53 56.88 1.62 64.66 64.87 59.76
Brisbane 66.31 62.52 67.72 48.01 40.73 65.18 50.98 22.60 48.37
Dallas 58.25 62.93 55.12 63.27 56.99 61.85 62.61 64.45 68.05
Fremont 51.01 67.06 54.41 58.96 45.59 54.08 52.53 66.93 57.96
Milan 66.44 49.51 63.34 57.97 35.24 59.58 30.62 49.66 20.28
Moscow 57.01 39.71 56.64 46.81 35.25 59.15 45.31 38.72 37.62
New York 1.62 64.92 57.69 54.74 59.88 59.12 77.04 51.77 64.71
Paris 64.63 50.73 62.21 52.84 31.08 45.80 78.06 50.27 4.65
Sydney 62.04 17.59 64.29 64.47 49.59 39.80 63.28 49.51 47.75
Zurich 66.74 48.40 68.13 55.81 20.15 37.60 65.22 4.65 49.65

For this comparison to be fair cables would need to be run directly between cities, which is clearly not the case. We still think it’s interesting. Hitting 68% of the theoretical best speed is quite remarkable.

About these ads

7 Comments »

  1. Ping times are round-trip, so you must double the distances.

    This makes your miles per millisecond and percentage of the speed of light tables half their correct value.

    Comment by Steve Weller — February 9, 2011 @ 3:25 pm

    • Hi Steve,

      Well don’t we look silly. Thanks for pointing this out, I’ve corrected the later two tables.

      paul

      Comment by Paul Reinheimer — February 9, 2011 @ 4:20 pm

  2. This would seem to reflect propagation delay. Did you do anything to factor in transmission delays (e.g., the time it takes to actually put the packet on the fiber)? The turnaround time at the other end of the ping? I suspect that after you factor these in, you’ll see that things are very good indeed.

    It would also be wonderful to understand what’s going wrong that’s causing the poor times.

    Comment by Tony Li — February 9, 2011 @ 7:54 pm

    • Tony,

      These values are straight from ping without any modification, so they’ll include the time it takes to get to/from the copper as well as the time spent on the remote machine identifying the packet and responding to it. Do you have any suggestions on how to calculate that?

      Our main goal was a comparison between our servers instead of absolute times. We assumed that the turnaround time at each server would be constant so it’s easy to ignore in our comparison.

      Comment by Will Roberts — February 9, 2011 @ 8:14 pm

  3. Reminds of the story of the 500 mile email.

    http://www.ibiblio.org/harris/500milemail.html

    Comment by Andy Delcambre — February 10, 2011 @ 1:11 am

  4. The reason for the transatlantic path is because peering always takes precedence over transit. We peer with packet exchange at the LINX in London, but do not have a session with them at Ashburn. I have raised this with them to attempt to setup a session at Ashburn as well so that the north-american traffic to that destination exits via Baltimore.

    Comment by Leland — February 10, 2011 @ 6:39 am

  5. The other thing to take into account is that ICMP is typically downgraded in priority in routers. You may want to try a custom UDP packet based test or to-the-metal TCP implementation (so you can track retries).

    Comment by JD Conley — June 27, 2011 @ 5:36 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Shocking Blue Green Theme Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: