Public clouds, such as Amazon AWS and Microsoft Azure, are increasingly becoming a popular choice for hosting an organization’s IT infrastructure and services. They can often provide managed and semi-managed services more efficiently than an average in-house IT department. With more Cybera members leveraging public cloud services such as these, we are looking at ways to ensure they have the best network access possible to the public clouds’ data centres.
When leveraging a public cloud, you often have the ability to choose the geographical data centre to host your service. Years ago, it was rare — sometimes impossible — to find a public cloud with a Canadian-based data centre. (When CANARIE partnered with Compute Canada and Cybera in 2011 to create the DAIR cloud for Canadian small-to-medium sized businesses, it was one of the first Infrastructure as a Service clouds in the country). Fortunately, times have changed, and all major public cloud providers now have a presence in Canada.
As great as this is, getting the best possible network path to these Canadian data centres can be a challenge. Focusing on Microsoft Azure, all traffic, regardless of the data centre, is first routed west from Cybera to Seattle. This means that traffic to Azure’s Canadian data centres in Toronto and Quebec City first travel west before going east. This results in a round trip of approximately 59 to 68 milliseconds. While not bad, it’s not great, either.
Route Path from Cybera to Azure Canada East (Quebec City):
traceroute to 40.69.97.22 (40.69.97.22), 50 hops max, 60 byte packets 1 * * * * * 2 162.246.156.1 (162.246.156.1) [AS15296] 1.009 ms 0.978 ms 0.949 ms 0.918 ms 0.895 ms 3 host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296] 0.767 ms 0.783 ms 0.756 ms 0.727 ms 0.706 ms 4 clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509] 0.711 ms 9.107 ms 0.598 ms 9.032 ms 8.934 ms 5 * * * * * 6 sttl1rtr2.canarie.ca (206.81.80.189) [*] 15.760 ms 15.745 ms 15.463 ms 15.368 ms 15.323 ms 7 microsoft-1-lo-std-707.sttlwa.pacificwave.net (207.231.242.7) [*] 14.981 ms 15.058 ms 15.030 ms 14.995 ms 14.912 ms 8 ae28-0.ear01.pdx31.ntwk.msn.net (104.44.233.84) [AS8075] 19.164 ms * 19.190 ms * * 9 be-21-0.ibr02.pdx31.ntwk.msn.net (104.44.21.61) [AS8075] 69.633 ms 69.320 ms 69.381 ms be-20-0.ibr01.pdx31.ntwk.msn.net (104.44.21.59) [AS8075] 69.589 ms 69.866 ms 10 104.44.16.70 (104.44.16.70) [AS8075] 69.920 ms 69.884 ms 69.846 ms 70.806 ms 104.44.16.72 (104.44.16.72) [AS8075] 69.761 ms 11 be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075] 69.491 ms be-8-0.ibr01.cys04.ntwk.msn.net (104.44.18.222) [AS8075] 70.083 ms 71.049 ms be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075] 69.461 ms 69.284 ms 12 be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075] 69.375 ms be-5-0.ibr01.dsm05.ntwk.msn.net (104.44.19.87) [AS8075] 71.407 ms 69.775 ms 69.780 ms be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075] 69.295 ms 13 be-7-0.ibr01.ch2.ntwk.msn.net (104.44.19.250) [AS8075] 69.842 ms be-4-0.ibr02.ch2.ntwk.msn.net (104.44.19.252) [AS8075] 69.720 ms be-7-0.ibr01.ch2.ntwk.msn.net (104.44.19.250) [AS8075] 69.705 ms be-4-0.ibr02.ch2.ntwk.msn.net (104.44.19.252) [AS8075] 69.235 ms * 14 be-5-0.ibr02.yto20.ntwk.msn.net (104.44.17.146) [AS8075] 69.766 ms 69.830 ms 69.793 ms be-8-0.ibr01.yto20.ntwk.msn.net (104.44.17.144) [AS8075] 69.830 ms 69.933 ms 15 be-7-0.ibr02.yqb20.ntwk.msn.net (104.44.28.102) [AS8075] 69.486 ms be-4-0.ibr01.yqb20.ntwk.msn.net (104.44.28.100) [AS8075] 69.992 ms 78.365 ms 69.817 ms 69.635 ms 16 ae24-0.yqb20-96cbe-1a.ntwk.msn.net (104.44.11.198) [AS8075] 68.855 ms 69.067 ms
Route Path from Cybera to Azure Canada Central (Toronto):
traceroute to 52.237.21.246 (52.237.21.246), 50 hops max, 60 byte packets 1 * * * * * 2 162.246.156.1 (162.246.156.1) [AS15296] 1.627 ms 1.638 ms 1.629 ms 1.610 ms 1.597 ms 3 host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296] 1.599 ms 1.583 ms 1.559 ms 1.562 ms 1.544 ms 4 clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509] 0.880 ms 0.572 ms 0.815 ms 0.790 ms 0.784 ms 5 * * * * * 6 sttl1rtr2.canarie.ca (206.81.80.189) [*] 15.227 ms 15.148 ms 15.174 ms 15.132 ms 15.341 ms 7 six1.microsoft.com (206.81.80.30) [*] 14.818 ms 15.107 ms 15.022 ms 15.026 ms 15.103 ms 8 * * ae27-0.ear01.pdx31.ntwk.msn.net (104.44.236.18) [AS8075] 18.740 ms * * 9 be-20-0.ibr01.pdx31.ntwk.msn.net (104.44.21.59) [AS8075] 60.141 ms be-21-0.ibr02.pdx31.ntwk.msn.net (104.44.21.61) [AS8075] 60.472 ms 59.710 ms be-20-0.ibr01.pdx31.ntwk.msn.net (104.44.21.59) [AS8075] 59.965 ms be-21-0.ibr02.pdx31.ntwk.msn.net (104.44.21.61) [AS8075] 59.630 ms 10 104.44.16.70 (104.44.16.70) [AS8075] 60.126 ms 104.44.16.72 (104.44.16.72) [AS8075] 59.877 ms 59.650 ms 59.696 ms 59.677 ms 11 be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075] 59.730 ms be-8-0.ibr01.cys04.ntwk.msn.net (104.44.18.222) [AS8075] 60.088 ms be-7-0.ibr02.cys04.ntwk.msn.net (104.44.18.224) [AS8075] 59.528 ms 59.516 ms 59.925 ms 12 * be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075] 59.810 ms be-5-0.ibr01.dsm05.ntwk.msn.net (104.44.19.87) [AS8075] 60.049 ms * be-8-0.ibr02.dsm05.ntwk.msn.net (104.44.18.151) [AS8075] 59.385 ms 13 be-4-0.ibr02.ch2.ntwk.msn.net (104.44.19.252) [AS8075] 59.482 ms 59.443 ms 59.397 ms 59.836 ms 59.331 ms 14 be-5-0.ibr02.yto20.ntwk.msn.net (104.44.17.146) [AS8075] 59.555 ms 59.741 ms be-8-0.ibr01.yto20.ntwk.msn.net (104.44.17.144) [AS8075] 60.213 ms be-5-0.ibr02.yto20.ntwk.msn.net (104.44.17.146) [AS8075] 59.482 ms 60.872 ms 15 ae102-0.icr02.yto20.ntwk.msn.net (104.44.20.150) [AS8075] 59.790 ms be-1-0.ibr02.yto30.ntwk.msn.net (104.44.7.162) [AS8075] 59.889 ms be-1-0.ibr01.yto30.ntwk.msn.net (104.44.7.156) [AS8075] 60.158 ms 60.535 ms ae122-0.icr02.yto20.ntwk.msn.net (104.44.20.166) [AS8075] 59.074 ms 16 * ae122-0.icr02.yto30.ntwk.msn.net (104.44.20.178) [AS8075] 59.743 ms * *
In each of the above traces, you can see the traffic leaving Calgary at hop 4 and arriving in Seattle at hop 6, before entering Microsoft’s network and ultimately going back east.
The Pilot
Last month, in collaboration with CANARIE, Cybera began a pilot to more efficiently route network traffic destined to Microsoft Azure’s Canadian data centres. Instead of going west to Seattle, traffic was instead sent east to Winnipeg, then Toronto, and then optionally on to Quebec City.
Route Path from Cybera to Azure Canada East (Quebec City):
traceroute to 52.229.126.237 (52.229.126.237), 50 hops max, 60 byte packets 1 * * * * * 2 162.246.156.1 (162.246.156.1) [AS15296] 0.743 ms 0.826 ms 0.793 ms 0.770 ms 0.739 ms 3 host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296] 1.239 ms 1.189 ms 1.075 ms 1.082 ms 1.065 ms 4 clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509] 0.950 ms 0.923 ms 0.897 ms 0.864 ms 0.783 ms 5 wnpg2rtr2.canarie.ca (205.189.33.199) [AS6509] 37.139 ms 36.964 ms 37.180 ms 37.165 ms 37.148 ms 6 205.189.33.182 (205.189.33.182) [AS6509/AS53904] 37.106 ms 37.082 ms 37.231 ms 37.186 ms 37.164 ms 7 peer-as6509.pr03.yyz1.tfbnw.net (103.4.99.3) [AS32934] 37.110 ms 37.261 ms 37.224 ms 37.071 ms 37.037 ms 8 canarie.yto01-96cbe-1a.ntwk.msn.net (207.46.219.84) [AS8075] 36.860 ms 36.842 ms 36.823 ms 36.533 ms 36.523 ms 9 ae21-0.icr01.yto30.ntwk.msn.net (104.44.237.155) [AS8075] 37.364 ms ae22-0.icr01.yto20.ntwk.msn.net (104.44.237.151) [AS8075] 37.415 ms 37.440 ms ae21-0.icr01.yto30.ntwk.msn.net (104.44.237.155) [AS8075] 37.550 ms ae22-0.icr01.yto20.ntwk.msn.net (104.44.237.151) [AS8075] 37.328 ms 10 be-120-0.ibr02.yto20.ntwk.msn.net (104.44.20.165) [AS8075] 47.681 ms be-100-0.ibr01.yto30.ntwk.msn.net (104.44.20.161) [AS8075] 47.449 ms 47.424 ms be-100-0.ibr01.yto20.ntwk.msn.net (104.44.20.149) [AS8075] 47.344 ms be-100-0.ibr01.yto30.ntwk.msn.net (104.44.20.161) [AS8075] 47.432 ms 11 be-5-0.ibr01.yqb20.ntwk.msn.net (104.44.28.7) [AS8075] 47.614 ms be-3-0.ibr02.yqb20.ntwk.msn.net (104.44.28.79) [AS8075] 47.364 ms be-4-0.ibr01.yqb20.ntwk.msn.net (104.44.28.100) [AS8075] 47.256 ms be-3-0.ibr02.yqb20.ntwk.msn.net (104.44.28.79) [AS8075] 47.219 ms be-5-0.ibr01.yqb20.ntwk.msn.net (104.44.28.7) [AS8075] 47.516 ms 12 ae23-0.yqb20-96cbe-1b.ntwk.msn.net (104.44.11.202) [AS8075] 47.025 ms
Route Path from Cybera to Azure Canada Central (Toronto):
traceroute to 52.228.17.116 (52.228.17.116), 50 hops max, 60 byte packets 1 * * * * * 2 162.246.156.1 (162.246.156.1) [AS15296] 0.653 ms 0.771 ms 0.757 ms 0.772 ms 0.739 ms 3 host-199.116.233.45.cybera.ca (199.116.233.45) [AS15296] 0.844 ms 1.101 ms 1.105 ms 1.094 ms 1.044 ms 4 clgr2rtr2.canarie.ca (199.212.24.66) [AS6327/AS6509] 1.018 ms 0.731 ms 0.922 ms 0.654 ms 0.654 ms 5 wnpg2rtr2.canarie.ca (205.189.33.199) [AS6509] 37.467 ms 37.497 ms 37.203 ms 37.168 ms 37.212 ms 6 205.189.33.182 (205.189.33.182) [AS6509/AS53904] 37.771 ms 37.778 ms 37.510 ms 37.487 ms 37.450 ms 7 peer-as6509.pr03.yyz1.tfbnw.net (103.4.99.3) [AS32934] 36.876 ms 37.296 ms 37.230 ms 37.215 ms 37.237 ms 8 canarie.yto01-96cbe-1a.ntwk.msn.net (207.46.219.84) [AS8075] 36.809 ms 36.798 ms 36.538 ms 36.545 ms 36.691 ms 9 ae22-0.icr01.yto20.ntwk.msn.net (104.44.237.151) [AS8075] 36.728 ms
The above traces are quite different than before. Here, we can see traffic leaving Calgary at hop 4 and arriving in Winnipeg (instead of Seattle) at hop 5. In addition, you can see the round trip times have dropped from 59-68 milliseconds to 36-47 milliseconds — that’s quite an improvement!
Lessons Learned
However, this wouldn’t be a true pilot unless there were complications.
For members who were leveraging Microsoft Azure prior to this pilot, it made sense for them to pick the data centre that had the best network performance. And since all their traffic was travelling to the western United States, the best data centre choices were in… the western United States.
Now that we’ve redirected all Azure traffic to go east, traffic destined for the western US data centre has to travel east before going west. We’ve effectively flipped the problem, and that’s not great for some use cases. Fortunately, members who find themselves in this situation are able to easily opt out of our pilot and return to the original network path.
While it would be perfect if we could provide the best of both worlds — western US traffic go west, and eastern Canada traffic go east — it’s unfortunately not that simple. When we route this traffic, we have to think of Azure as a single entity, rather than as a distributed public cloud with several data centres. As this pilot progresses, we will continue to investigate how to best handle this situation.
This pilot is also notable because it raises the issue of accessible data sovereignty. What conditions are needed for Canadian institutions to leverage more cloud services from Canadian-based data centres? As we look to grow the value and utility of Canada’s National Research and Education Network (NREN) in the future, how can we ensure those requirements are met?
Piloting new solutions and ideas is a core value of what we do, especially when it can provide our members with better access to new technologies. Working through the difficult problems that are identified during a pilot is just another day in the office for us. We’ll keep you updated as to how this pilot progresses.