When Netflix revealed that it had started paying Comcast for a direct connection to its network, ending a long squabble over money, Comcast subscribers almost immediately started seeing better video performance.
So when Netflix agreed to pay Verizon six weeks ago in a similar “paid peering” deal, it seemed natural to assume that Verizon subscribers would start getting better quality videos within days or weeks.
That didn't happen—in fact, Netflix performance got worse on Verizon in the month after the paid peering deal was announced. The reason, according to interviews we've conducted, is that the technical teams at Comcast and Netflix spent months working together to lay the groundwork for direct network connections even before the companies' lawyers were done arguing over money. As a result, the engineering teams were able to set the connections up almost immediately after the deal was signed.
Verizon, on the other hand, seems not to have done any major technical work until after signing its deal with Netflix.
"Once we had an agreement and knew the interconnection points from Netflix, work began," a Verizon spokesperson told Ars. Work may not be completed until the end of 2014.
Due to Netflix's selection of locations for interconnection, as well as Verizon's desire to meet demand in all of our key markets, we must jointly ready facilities to complete that interconnection. We are also ensuring that as we bring the increased traffic to our network, that we are able to meet our brand promise of an industry-leading experience not just with Netflix, but for all content and services. We are in the process of doing this, and we will be incrementally rolling it out starting next month and progressing through the fourth quarter [October to December]," a Verizon statement said.
Verizon VP David Young argued that it only seems like Comcast moved more quickly than Verizon because the Comcast/Netflix deal wasn't revealed until after the companies were already interconnecting. "The bottom line is the Comcast deal became publicly known when the work was being implemented, vs. ours became publicly known when the deal was announced, and I think that's the key difference," Young told Ars today.
But there's more to the story.
Preparation meets opportunity
It's true that Comcast and Netflix revealed their deal only after people outside the companies used traceroutes to discover that they were exchanging traffic directly. When Comcast confirmed it on February 23, the company said it had already been "working collaboratively [with Netflix] over many months" to establish "a more direct connection between Netflix and Comcast," suggesting that the work began in 2013.
But that doesn't mean the actual deal was finished much before February 23. A source close to Comcast told Ars today that the agreement was signed just days before it was revealed. While the deal was being negotiated, Comcast's and Netflix's engineering teams worked on a technical plan for at least four months, the source said.
Besides design, traffic analysis, and capacity planning, both Comcast and Netflix ordered the necessary data center equipment ahead of time, the source said. This was described to us as "router infrastructure, optical backhaul infrastructure, core router infrastructure, and long haul optical infrastructure." This is the same kind of work Comcast has done previously with other content companies that pay for direct network connections. Netflix is the first major content company to object to these arrangements, and it even asked the FCC to ban such payments.
Nonetheless, Comcast and Netflix now have multiple terabits per second worth of direct connections to support Netflix traffic. They regularly have discussions about capacity and performance, and they continue to upgrade interconnections to support additional traffic, the source close to Comcast said.
The connections are spread across the country in about 10 carrier-neutral Internet exchange points, such as ones operated by Equinix. "Once the deal was signed, it was a matter of executing on the turnups," the source said. "This is not something you can do overnight. It takes a fair amount of time to execute this level of infrastructure."
“Comcast has a total of 18 national locations, and Netflix and Comcast will initially connect in about 10 of those locations to start,” analyst Dan Rayburn told Ars this week. “Out of those 10 locations, Netflix will need 300+ 10GigE ports. It took Comcast and Netflix about two months to put it all in place. Verizon has told me it will all be in place before year's end, but won't give any other ETA.”
Verizon and Netflix did not offer an explanation for why they weren't working together before the paid peering deal was signed. They have been fighting publicly: When Netflix blamed Verizon for the poor quality, Verizon threatened to sue Netflix for harming its reputation, but it seems to have backed down from that threat. Netflix's engineers are open to working with rivals, through. “Engineering people at companies, whether large or small, operate independently of commercial interests,” David Fullagar, Netflix’s director of content delivery architecture, said recently. “In the UK, one of our biggest competitors is one of our best networking partners.”
An eight-month wait for Verizon customers?
Netflix's speed index indicates that the benefits of the Comcast upgrades came between February and April. After falling in quality each month from October through January, Netflix performance on Comcast suddenly rose in February and continued to rise in March and April before leveling off in May. A Netflix spokesperson told Ars this week that the links between Netflix and Comcast are "almost" complete.
Verizon's deal with Netflix was announced on April 28. If it takes until the end of the year to implement the peering connections, as Verizon has indicated, that would be an eight-month project.
The Wall Street Journal reported that Netflix and Verizon have set up a test connection to carry video around Dallas, and Verizon told the Journal that it’s working on setting up peering connections in 13 cities.
Young previously told CNET, "We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year."
Although Verizon wasn't able to replicate the near-immediate post-deal improvement seen on Comcast, it will be worth checking Netflix's speed index over the next few months to see whether Verizon can make steady progress. As we've reported, Netflix performance dropped for months on Comcast, as the connections between Netflix's transit providers and the ISP became more congested due to money squabbles. Then it all turned around: average stream rate on Comcast moved from 1.51Mbps in January 2014 to 1.68Mbps in February, 2.5Mbps in March, and 2.77Mbps in April, according to Netflix measurements.
Netflix performance on Verizon continued to get worse in May after the companies' paid peering deal was announced, with the average stream rate dropping from 1.99Mbps in April to 1.90Mbps in May on FiOS and from 1.08Mbps to 1.05Mbps on DSL.
Google Fiber led the country with 3.51Mbps average Netflix performance in May, while Cablevision leads big ISPs with 3.03Mbps. Google Fiber and Cablevision are both part of Netflix's Open Connect program, which involves payment-free interconnections and the placement of video caches inside ISP networks. Comcast and Verizon didn't join Open Connect because they wanted payments in exchange for accepting large quantities of Netflix traffic.
Since Netflix's speed rankings began in November 2012, Verizon FiOS has never topped 2.2Mbps and Verizon DSL has never topped 1.43Mbps.
AT&T is also trying to get money from Netflix. But as we've seen with Verizon, there's no guarantee that consumers will get better quality soon after a deal is struck.