Netflix released its latest ISP Speed Index yesterday, including Canada for the first time. Given the popularity of the online video service, the Netflix report has attracted increasing attention as it offers a comparative look at the average download speeds for Netflix customers across Internet providers around the world. While the company acknowledges that there are various factors that influence speed (including device used, video quality, etc.), those issues are found across all ISPs, so the comparisons remain valid.
Canada’s performance is middling at best as the Netflix data indicates that we are a mid-tier country at best. Canadian speeds that do not compare well with most European countries (note that Asian countries such as South Korea and Japan are not included but would likely rank far ahead of Canada as well). The biggest surprise in the report is how poorly Rogers ranked, coming in last among the 14 Canadian ISPs that were measured. The ranking is particularly surprising since the other large cable companies (Shaw, Videotron, Cogeco, and Eastlink) all ranked in the upper half of Canadian ISPs.
The poor ranking, which would have placed Rogers in last place in many other countries (Denmark, Finland, Ireland, the Netherlands, Norway, Sweden, the UK, and Uruguay) raises questions about Rogers’ Internet traffic management practices. In response to the Netflix story and some tweets on the issue, Rogers responded:
Netflix test done just before we virtually doubled Netflix capacity, we’ll continue to add more capacity as required
I followed with a tweet raising questions about the meaning of doubling Netflix capacity and asking whether the company was throttling Netflix traffic. Rogers replied:
We don’t throttle Netflix. We’ve doubled capacity in the links that carry traffic from Netflix to our customers.
While these responses are meant to be reassuring, they raise troubling questions about how Rogers manages its network and whether the slow Netflix speeds could have been used to create a competitive advantage for its own online video services. While the company says that it does not throttle Netflix traffic (ie. deliberately slow it down), its response also suggests that it knew that the service was being slowed by insufficient capacity. I wrote about net neutrality in my weekly technology law column this week (Toronto Star version, homepage version) and the Rogers responses raise a host of related regulatory questions:
- How long did it know that Netflix speeds were slow?
- Why are Netflix-specific links within the network the problem?
- Does Rogers separate Netflix traffic from other traffic?
- If so, why does it not disclose the practice?
- Is the slowing of video a violation of Section 36 of the Telecommunications Act, which the CRTC has said amounts to controlling the content?
- Are other online video services affected in the same manner?
- Are Rogers online video services affected?
The Netflix rankings are presumably designed to provide greater transparency on actual ISP speeds. Now that we have Canadian data, we need some answers from one of Canada’s largest ISPs on why it ranked so badly.
A little niggle: speed isn’t capacity
As Mr. Macawber says, “Annual income twenty pounds; annual expenditure, nineteen six, result — happiness. Annual income, twenty pounds; annual expenditure, twenty pounds ought and six, result — misery.â€
The same applies to capacity: performance falls precipitously the moment you exceed capacity, even for only a fraction of a second. Doubling capacity can have no effect at all, if the demand in the evening is 2.001 times the capacity of Rogers’ original pipe.
–dave
[In this case, performance is actually response time (ie, slowness). The response time curve is a hyperbola, and increases without bound beyond the capacity. It first starts to gets bad at around 80%, because of momentary overloads.]
TekSavvy
Teksavvy is a Bell/Rogers reseller AFAIK, so it’s most interesting how a company that resells Rogers bandwidth performs better than Rogers. Nope, no shenanigans here.
would you beleive….
I’ve MET the (german) students that do this work. (throttling)
now YOU get a choice; foreign workers, throttling, buy the byte, censorship
high prices for crummy services, leaks, dirty (cancer) tech…
have a day of it, eh?
pat
(lets give ’em conniptions and say ISPs HAVE to provide TV-time (cable) for broadcasters.)
TekSavvy
@digitaldude – wrong. TekSavvy resells the physical connection. The network and bandwidth are entirely different.
The numbers don’t make sense
I have a 25 Mbps ADSL connection that costs me about $38/mo, I don’t have fibre. I do frequent speed tests on my connection and routinely see 26-27 Mbps download speeds so I know I am getting what I pay for.
If Netflix is complaining that its users only see 2.5 to 3.5 Mbps, in pretty much every country in the world, it would lead you to the conclusion that the speed bottleneck is at the Netflix end of the delivery path. Otherwise you are forced to draw the conclusion that every ISP in the world (including Google’s fibre service) is throttling Netflix. Either that, or Netflix doesn’t have a clue what they are measuring and the numbers are all bogus.
The numbers
The explanation I’ve seen is that Netflix is assessing the average dopwnload speed experience. They don’t necessarily give you the download at full throttle, because you might start watching and change your mind, and then they’ve used up more bandwidth than they needed. That was my layperson’s understanding anyway.
The numbers – explanation
Hi Cynic,
The reason you don’t see numbers much higher than 3mbps is the HD stream doesn’t require any more, so there’s no sense in Netflix transferring the movie at 25 mbps.
The data does indicate that Rogers only takes the data at half the HD rate, meaning that Netflix has to adjust the stream to fit thru their little pipe. (probably various ways they do this – shift to SD, increase compression etc)
Best course of action is to go with a re-seller, and to call Rogers and the CRTC to complain weekly.
Rogers’ excuse doesn’t fly
The fact that an ISP the size of Rogers needs to double their capacity to Netflix to resolve congestion is nuts. That means that they let the connection get incredibly oversaturated, and left it like that for an extended period of time.
This means that Rogers was either incompetent in managing their network capacity, or malicious in intentionally allowing congestion to discourage competition. I don’t know which one it is, but either is bad.
Who is responsible for Netflix QoS
@Adam
I have no love for Rogers so don’t take this the wrong way. The only party responsible for managing the quality of service to Netflix customers is Netflix. The onus should be completely on Netflix to monitor their service quality and make sure that they have sufficient capacity into Rogers network to deliver their service. Maybe they have been doing that all along and were working with Rogers to increase capacity when they saw the QoS dipping below their required thresholds and decided to build more capacity. Does anybody really know or are we jumpoing to the conclusion that Rogers is intentionally try to mess with Netflix? Why has Netflix been silent, has anybody asked them?
Let’s be honest with each other, the vast majority of streaming video on the internet is porn. Is it Roger’s job to monitor all the millions of porn sites to make sure they have sufficient IP trunk capacity into Rogers network? Or do the purveyors of porn have some responsibility to make sure they have adequate capacity from their servers to keep their porn surfers happy?
Open Connect
Netflix has a program called Open Connect which allows participating ISPs to deploy caches of Netflix data onsite. Perhaps Rogers participates in this system and just doubled the number of caches on their network.
https://www.netflix.com/openconnect
* I just noticed that Rogers isn’t listed as an active participant, but maybe the page isn’t complete or out of date. I can’t imagine why an ISP wouldn’t want to participate.
Testing my connection on wireless at home I’m just about 26 MB/S with Rogers. Over the past month or so while gaming, I’ve noticed that I’m getting lag again. I’ve been testing different internal network settings, haven’t yet come to the conclusion that DPI is the culprit again, however with this latest revelation very well could be. Will the CRTC audit Rogers or do we have to file yet another complaint to initiate the audit?
Huh?
“The only party responsible for managing the quality of service to Netflix customers is Netflix.”
I’m not sure why you’d think that.
A provider can allocate (whether legally or otherwise) whatever bandwidth it wants to a specific service, IP block, or individual site. This capability helps form the threat of the multi-tiered internet we’re all talking about, and all the anti-competitive and censor-enabling BS that goes along with that.
The results discussed what the providers were delivering, and don’t appear to have anything to do with the Netflix side of the equation.
Other services around the world are offering anywhere from 5 to 100 times the speed and capacity we’re allowed to have, and at a fraction of the price we pay, yet companies like Rogers continue to advertise their services as “blindingly fast” and “fully capable”. It’s an insult to the human intelligence.
Anyway, Netflix isn’t the only party that has been trying to illustrate the way we’re getting screwed on this side of the Atlantic.
Admission of throttling netflix
Rogers just basically admitted to what throttling connections in this statement:
http://redboard.rogers.com/2014/more-bandwidth-for-netflix-customers/
“Netflix’s test was done just before we virtually doubled Netflix capacity and we’ll continue to add more capacity as it’s needed. These results only apply to customers’ specific Netflix connection and not overall internet speeds.”
Why are they managing Netflix “capacity” at all?
That’s the way the whole world works
@Devil’s Advocate
If I buy a service from Netflix I am going to hold them accountable for delivering that service to me. That basic rule applies to every business transaction I have ever made. I don’t want to know who they use as subcontractors or partners in delivering that service, that’s their job, not mine. If one of their contractors drops the ball then it is up to Netflix to fix it.
A while ago I paid Fedex to deliver a package to a small town in Ontario. Since Fedex didn’t have an office in that small town they contracted Canada Post to deliver the package the last few miles. Canada Post lost the package so Fedex tried to get me to deal directly with Canada Post to find it. I dug in my heals and said “no way” and threatened to make a claim for the value of the lost package. Fedex accepted their responsibility and got busy and found the missing package at a Canada Post warehouse. I paid Fedex, not Canada Post so why would I want to talk to Canada Post?
If Netflix has a problem with Rogers network capacity I expect Netflix to figure it out with Rogers. I just want to watch reruns of Gilligan’s Island on Netflix, not become a lobbyist for Netflix. If Netflix wants me to be a lobbyist on their behalf they should be paying me, not the other way round.
An attempt at an explanation…
@Cynic
I’m not sure you have a good grasp of how data networks work. Comparing it to Fedex / Canada Post is not a useful analogy.
You don’t have a contract with Netflix to provide you streaming video at 1, 2 or 3 Mb/s. They provide you access to their streaming library and software to access it. They have business relationships with data centers (or run their own) and access of those data centers onto the internet. It is possible (and likely) that Netflix does not have a business relationship with Rogers at all (or Bell or the other 12 ISP’s listed).
On the other hand, Netflix customers pay some internet provider such as Rogers for a connection to the internet. At some point in the network, data that originates from a Netflix data center goes out onto the internet and eventually gets directed onto the Rogers network, then to Rogers gear in your City, then to Rogers gear in your neighborhood, and finally to your Rogers modem.
The problem isn’t insufficient bandwidth on Netflix’s end. If that was the case, all networks would have more capacity than Netflix and would have the same average speed. That speed would be less than required to transmit HD video and Netflix wouldn’t bother with this sort of ISP performance report because there would be nothing to report. That is empatically not the case. There is a range of average speeds across the 14 providers. (as a side note, this is why the best results cluster around 3Mb/s. That’s lots of data for high-def video and multi-channel audio.)
The problem is that netflix data traffic was impeded on the Rogers network. It’s not clearly explained yet what that constraint was, and Rogers has claimed to have “doubled Netflix capacity” since the tests were done. This would still only put them mid-table, but progress is progress.
You seem to be advocating for exactly what the FCC is proposing in the US. That companies like Amazon, Netflix, Google, etc. seperately negotiate and pay for enhancements to the ISP’s infrastructure so they are allocated a “fast lane” to the customer. This is antithical to how the internet is supposed to work.
A Rogers customer has an agreement with Rogers to move data from the edge of it’s network to the customer as per their plan. No further negotiations or interactions should be required between the end customer, Rogers and Netflix. Rogers should be delivering the data without constraining the pipe. Rogers (in theory) should not be treating traffic any differently if it’s Rogers on-demand movies, a game download from Steam or restoring a backup of your Photo directory from dropbox.
Question of “delivery”…
“If I buy a service from Netflix I am going to hold them accountable for delivering that service to me.”
It’s your provider that’s ultimately “delivering” to you.
Netflix could be uploading to you at warp speed, but if your provider is limiting the bandwidth allocated to Netflix traffic, you get a slower “experience”.
As Jason pointed out, Rogers’ already made the statement about “Netflix capacity”, and how they’re “doubling” it. That should tell you who’s effectively “responsible” for the traffic.
Clearly traffic segregation is at hand
What they’re saying is they don’t throttle Netflix explicitly. But they do make sure their content gets highest priority (not mentioned).
More evidence that Net Neutrality is already threatened.
“What they’re saying is they don’t throttle Netflix explicitly.”
Or, they’re saying they don’t throttle Netflix *as much* as some others.
Just a hunch, but I’m willing to bet they don’t throttle their own content offerings at all.
It’s all smoke and mirrors
If you look at the Netflix tables you will see Verizon FiOS (fibre) is 1.99 Mbps, and Google Fibre is 3.58 Mbps. Google claims to be offering 100 Mbps fibre access to their customers so does that mean Google is throttling Netflix as well – where did the other 96.5 Mbps go? I seriously doubt every ISP on the planet is throttling Netflix.
@Devil’s Advocate – I’m giving $8 a month to Netflix so I can watch their programs. My $8 is not going to my ISP. If I have a performance problem watching Netflix I am going to call Netflix to fix it since they are getting my money. Yes, I also pay my ISP $38/mo for my 25 Mbps access which I test regularly so I know I’m getting what I pay for. If my ISP is messing around with Netflix or if Netflix has overloaded servers or inadequate pipes into my ISP that is Netflix’s problem to solve, not mine.
Let’s not kid ourselves. If all the ISPs in the world have to invest in upgrades to handle all the Netflix traffic the money is going to come out of our pocketbooks. The question we should be asking is whether Netflix users should be footing the bill or should all the other ISP customers pay more to upgrade the network as part of their access fees.
@Cynic
As stated above netflix doesn’t need to send more than 3-ish MB / s for HD video. The rest of the pipe is then available for other simultaneous uses of the internet.
A speed test is not valid if you aren’t testing it with netflix explicitly. Rogers isn’t showing down the files used for a speed test. If anything, it’s in their interest to give that priority on their network.
Netflix did not control Rogers section of the internet at all. It is on Rogers and Rogers only to rectify their poor network performance. Which they have. They didn’t say “were working with netflix on this”, they said “we’ve added capacity to address this”. Rogers understand it’s their problem to fix. I hope that you are beginning to understand this.
A better analogy is that you are mad at Amazon because your city hasn’t plowed the roads after a snow storm. It’s not Amazon’s fault and they can’t fix it. Expecting them to is unfair and unlikely to bear fruit.
@Zot
I find it interesting that Mr. Geist characterizes the performance of Netflix on Canada’s ISP networks is “middling at best” and you point out that Netflix is perfectly happy around 3’ish Mbps. So are we ok here or not?
You will recall back in January that Google was praising Canadian ISP’s for excellent network performance when they launched their Video Quality Report. Here’s a quote “On some level, we think Canada could be one of the first HD verified countries out there. So we wanted to start with a market where clearly people are doing it well, and we wanted to lead with the best.†The comments from Google seem to be the opposite of what the Netflix data suggests.
Sure, Rogers has pretty poor performance on the Netflix test and most of us expect that from them on virtually everything they do. But what on Roger’s network would limit Netflix performance to only 1.67 Mbps?
Rogers got caught throttling in 2012 and claimed to have stopped that practice – if they get caught again they would be in violation of a CRTC order and they could get whacked with a massive fine or someone could go to jail.
If the problem was exclusively on Rogers network then you would be forced to conclude that all other applications on Rogers network would be similarly degraded. I suspect the trunks between Rogers network and Netflix were the real bottleneck – and fixing that problem would require joint efforts from both companies.
There are so many inconsistent data points on this issue that you need to be more cynical.
What’s good for netflix is good for…
“we virtually doubled Netflix capacity”.
So, if Rogers can do this for Netflix, you don’t suppose they are doing it for speedtest.net, do you? Naah….
…
@Cynic There are two ways for ISPs to connect to Netflix. Either via direct peering (OpenConnect), or through one of Netflix’s transit providers (like Level3). If Rogers connects to Netflix via a transit provider, then Netflix has no visibility into any congestion Rogers might face, because it’s one network removed. If Rogers is directly peering with Netflix, then Netflix would already be providing a sufficiently high bandwidth connection to the peering point (which are shared with other ISPs, not just Rogers), and the onus would be on Rogers to increase the capacity of the connection from their network to the peering point.
Netflix volume is enormous
Netflix uses their ISP rankings to name and shame ISPs that are lagging in peering / transit capacity to them. Rogers claims it’s responding to that. As most “Internet” traffic is Netflix streams, it’s no small thing to have good peering towards Netflix, and as Netflix streams adapt to congestion, it’s not critical to over-provision towards them — picture quality degrades gently as the number of streams go up once capacity towards them is reached.
As Rogers basically directly competes with Netflix, without some shaming they would have some mixed feelings about the increased infrastructure costs associated with enormous volumes of Netflix traffic, and might want to skimp a bit on their “free” support of Netflix.
It doesn’t necessarily reflect their connectivity to the rest of the Internet, though anecdotally I hear that’s pretty sketchy as well.
3 Mbps
The numbers Netflix offers as ISP speeds are intentionally pathetic. Rather than only reporting numbers for clients capable of receiving the highest speeds streams Netflix offers, their average includes all of the lower-bitrate streams going to credit-card sized cell-phone screens and other low-capability clients. You’ll note that all of the ISPs that are also sizeable mobile operators have lower numbers… that’s not likely because their wired Internet is slower than the competition.
It’s in Netflix’s interest to provide low numbers, though, as it puts public pressure on ISPs to do more for them and makes the ISPs look more like villians in public discussion.
Hi Michael, this is Chris from the Rogers social media team. We’ve addressed the questions you raised in your blog post:
How long did it know that Netflix speeds were slow?
As traffic increased, we realized we needed to add capacity to the connecting links between Rogers and Netflix. Testing took place right before we added significant capacity – we’ve now doubled Netflix capacity and we’ll continue to add more as traffic grows.
Why are Netflix-specific links within the network the problem?
Netflix prefers that internet providers use their direct connection network called Open Connect and that’s what many carriers around the world do. We connect to Open Connect at a number of their peering locations. That’s why these results only apply to customers’ specific Netflix connection and not the overall internet speeds. As traffic grows we need to increase capacity. Testing was done just before we added significant capacity to these links.
Does Rogers separate Netflix traffic from other traffic?
No, we don’t.
If so, why does it not disclose the practice?
We don’t separate traffic
Is the slowing of video a violation of Section 36 of the Telecommunications Act, which the CRTC has said amounts to controlling the content?
We do not slow down or throttle any content. For more information please visit http://www.rogers.com/web/content/network_management
Are other online video services affected in the same manner?
No. These results only apply to customers’ specific Netflix connection and not the overall internet speeds. Independent third party testing continues to show that Rogers offers top internet speeds, for example Ookla found that Rogers speeds are the fastest amongst major ISPs in Canada, SamKnows, an independent leader in internet testing, found that Rogers customers get faster speeds than advertised . Also, PCMag.com recognized Rogers as the fastest overall in its Canada’s fastest Internet Service Provider review and YouTube has ranked Rogers as a top-quality network to deliver HD videos.
Are Rogers online video services affected?
No.
@ Rogers_Chris…with regards to the above…
“How long did it know that Netflix speeds were slow?
As traffic increased, we realized we needed to add capacity to the connecting links between Rogers and Netflix. Testing took place right before we added significant capacity – we’ve now doubled Netflix capacity and we’ll continue to add more as traffic grows. ”
It is highly suspicious that this additional capacity wasn’t foreseen earlier. Even a lay person knows the capacity needed for netflix is large. For network engineers and a service provider to let this go unnoticed until after the fact simply smacks of malice or incompetence…maybe both. Rogers…still can’t be trusted.
This article outlines precisely why I switched from Rogers as an ISP and would never go back to them again.
Say what?
“If I buy a service from Netflix I am going to hold them accountable for delivering that service to me.”
So if you’re on a dial-up connection, it’s the fault of Netflix that you can’t watch HD movies?
@Rick Pali
“So if you’re on a dial-up connection, it’s the fault of Netflix that you can’t watch HD movies?”
If anyone is dumb enough to think they can watch Netflix over a dial-up connection they likely aren’t smart enough to figure out how to turn on their computer or tablet.
But if you have a 25 Mbps connection then it would seem reasonable to expect that it should work. If it doesn’t, Netflix had better be talking to the ISP about how to fix the peer to peer congestion problem they are having with the ISP otherwise I’m not going to pay them another $8 this month.
Rogers
@Cynic
I am not sure what your point is. Rogers admitted it didn’t have enough capacity for netflix traffic.Rogers knows what their customers are pulling from Netflix and that pipe was getting congested yet they sat on their hands. Note that Rogers didn’t blame Netflix for the problem, if they could they would have done it.
@thx
I think Cynic has made his point very clearly and some of the comments of others on this forum support him. I think the problem that some people are having is that they have convinced themselves that ISPs are the enemy so they are going to blame them for everything.
The concept of peering requires service providers to connect at mutually agreed locations, protocols, and speeds using compatible technology. So both parties in a peering arrangement have a shared responsibility to make it work. Obviously Netflix has a shared responsibility in the delivery of their service and collects a fee for for that service. All Cynic appears to be saying is that he chooses to work through Netflix to fix problems that might occur instead of his ISP.
Let’s say you bought antivirus software from Mcafee for your PC. If it didn’t work properly would you call Mcafee for help or would you call Microsoft or HP (your PC maker) to fix it?
Plainly obtain a assistance by Netflix My goal is to store these individuals in charge of providing in which assistance in my experience. Which standard tip applies to each and every enterprise financial transaction I have ever made. I do not want to know who many people work with since subcontractors or associates in providing in which assistance, that is certainly their career, certainly not my very own. In the event one of their contractors declines the particular soccer ball and then it can be as much as Netflix to repair the item.
Plainly obtain a assistance by Netflix My goal is to store these individuals in charge of providing in which assistance in my experience. Which standard tip applies to each and every enterprise financial transaction I have ever made.