CDN Games

The Wall Street Journal reported today that the end of the Internet is imminent as Netflix and Comcast reached a so-called “pact” to improve streaming. Or so you might imagine from the paranoid hysterical coverage.

Obvious disclaimer alert: I worked at Comcast for two years and did things with CDNs at almost every place I’ve ever been, including Comcast. I’m undoubtedly still aware of confidential internal information which I’m obligated to protect and won’t discuss here. Dark, brooding information about Comcast’s plans for the future, no doubt.

But, I’m here writing that I don’t believe one word of the fear-mongering. At least, not the way the stories are being written, which are informed by the fifty-eight word press release issued by the companies yesterday.

Working collaboratively over many months, the companies have established a more direct connection between Netflix and Comcast, similar to other networks, that’s already delivering an even better user experience to consumers, while also allowing for future growth in Netflix traffic. Netflix receives no preferential network treatment under the multi-year agreement, terms of which are not being disclosed.

Let’s take the tin foil off our heads and break this down. I’ll help to translate from Cable-company-ese as necessary:

“Working collaboratively over many months”

Netflix streaming has been growing like gangbusters. This created a problem for Netflix, because they had to pay CDN providers to deliver that traffic based on usage, and usage was going through the roof. To keep costs down, they did what most smart media companies do, and shopped around for the lowest delivery costs.

While most traditional CDN providers have paid for access to eyeball networks via programs like Akamai’s AANP program (which is over 15 years old at this point), one of the new Netflix CDN providers was Level3, which benefitted from being a major carrier and network operator, and didn’t have plans to pay Comcast (or anybody else).

This dispute made it into the media back in 2010, and ultimately was settled last year. But it forced Netflix to go another way: they started becoming their own CDN (which they call OpenConnect). In spite of the name, it isn’t open to anybody’s content but Netflix’s, which is pre-positioned on hardware they provide to ISPs who host their gear. Basically, they re-invented the Akamai AANP thing from the 90s. In sexy Netflix red.

But Comcast and Verizon have never accepted CDN vendor gear into their networks and weren’t about to start now, so this went nowhere. Traffic increased, connections degraded at peak times, customers were unhappy on both sides, and undoubtedly talks started again about Doing Something About The Problem.

the companies have established a more direct connection between Netflix and Comcast”

I’m not sure what a “more direct connection” means, but from what’s been posted it appears Netflix bought hosting, built a network, and connected it to Comcast’s network at a major peering point in San Jose. This connected Netflix’s CDN to Comcast’s backbone directly, rather than via carriers like Level3, Verizon Business, etc.

similar to other networks”

Yup. I bet if Comcast were to release a list of the networks connected at that same point, you’d see every CDN and every major network provider. And, the people not delivering roughly symmetrical amounts of traffic are probably paying for the privilege.

that’s already delivering an even better user experience to consumers”

I think that means “we know the bottleneck was at the edge of our network, and not in our backbone, regional networks, or last mile”. This makes sense — Comcast rolled out DOCSIS 3 over 2 years back and that has tons of last mile bandwidth. Regional networks probably will eventually get congested, but since there are only a handful of major peering points in the United States to exchange traffic, that’s certainly where things get congested first.

Comcast reported around 17 million Internet customers last year. The big peering points (Ashburn DC2, San Jose SV1, New York’s 111 8th Ave, Chicago’s 350 E Cermak, and a few others) number in single digits in the US.

while also allowing for future growth in Netflix traffic”

I bet this means “We’ll sell them more ports if they need them”. Which makes sense, since the alternative is messy for everybody.

Netflix receives no preferential network treatment”

There are a lot of tricks you can play with cable networks to make traffic work better. This gets done with cable voice traffic, for example, to guarantee high quality service and make sure things like 911 work. Almost all internet traffic, however, gets prioritized as “best effort” which basically means nothing special gets done to it. This only really impacts that last mile, which I speculate isn’t really the congestion point here anyway. Remember: 17 million Internet subscribers, half-dozen big peering points. So, even if Comcast or Netflix wanted to do something like this, it probably wouldn’t do any good.

Still, this feels like something a paranoid Comcast attorney inserted to make sure nobody thought Network Neutrality was at stake. Please suppress your giggles.

“under the multi-year agreement”

This isn’t a trial. This isn’t temporary. They have a deal and it’s multi-year, which suggest this will resolve this problem for a while. At least until the congestion migrates elsewhere. I’m looking at you, Verizon.

terms of which are not being disclosed”

Did you really expect otherwise?

I think the big story here is that Netflix has made an aggressive move to become its own CDN, taking traffic and revenue away from competitive players like Akamai, Level3, Limelight, Edgecast, CDNetworks, and whomever else started a CDN this month. This always made sense for the really big players, but had really challenging economics because you ended up having to pay for transit, and transit cost about the same as CDN services.

I assume that Comcast is charging Netflix something. But I wouldn’t be surprised if it was competitive .. after all, Netflix probably wouldn’t agree to pay more than the going rate in a long-term agreement. They’d scream bloody murder and probably get some sympathy. So, this means that their strategy — and the general strategy of building your own CDN for larger video streamers — makes sense again. They join YouTube in this space, and YouTube got to ride Google’s golden coattails to connectivity.

Interesting times ahead…

The Google IO Games

Last year, Google’s IO conference sold out in under an hour of painful and awkward page refreshes against a clunky Cold Fusion-based system. This year, they increased the price and brought the system in-house. The event sold out in minutes, with many insisting they never had a chance.

In the spirit of the recently released movie adaptation of the popular book The Hunger Games, I have a suggestion for how Google can structure registration next year.

There will be an open registration period (perhaps a week) where developers can put their names into a drawing. If Google wishes to provide multiple entries for those who have previously attended, have contributed to projects, or are otherwise important to them, they can do so.

At a given date and time, Google will conduct the Google IO Reaping. A certain number of names will be drawn and receive email notification that they have an opportunity to purchase a ticket–I’d suggest they have 24-48 hours, so there’s no rush. Everyone who receive an invitation from the reaping gets a ticket if they pay for one.

If seats don’t get sold as tickets, they go back into the pool and re-reaped. The goal is that only the people who signed up for the reaping can be named as Google Tributes. Of course, people sometimes will need to cancel, and we’ll put those back into the pool to be re-reaped, and if someone else takes the seat the original purchaser gets a refund.

This process avoids a huge sign-up rush. It also gives everyone in every time zone an equal chance to make it in. And, finally, it restores an opportunity for Google to give previous participants an edge while keeping things fair.

What do you think, Google? May the odds be EVER in IO’s favor?

Content Delivery Summit – CDN and Cloud Convergence

I moderated a panel at the Content Delivery Summit (affiliated with the Streaming Media East event in New York last month) and they were nice enough to record the video. Very cool stuff!

The subject was about the convergence between CDN services and Cloud Computing services. Enjoy!

AOL Cloud wins Uptime Institute Green IT Innovation Award

It isn’t every day that a project of your conception takes off and gains enough traction to change things for everyone at your company. It’s even less common that such a project gets recognized by an outside body when it’s an infrastructure effort, which companies like AOL don’t discuss that openly. The combination of these two are why I’m especially pleased to share that the AOL Cloud project, which I’d been working to make a reality since 2007, was recognized by the Uptime Institute at their 2011 Green Enterprise IT awards. We were recognized in the “IT Innovation” category on May 11th at the Uptime Symposium. Aaron Lake and I provided a presentation of our effort – Will Stevens couldn’t join us, as he had left AOL.

No effort of this nature happens as an individual effort, and this was no exception. Given the challenging circumstances at AOL in 2010, it was especially exciting to see people band together to work on such an exciting project. I’m glad we were able to embrace some of the best of current technology at AOL, and make it ours. I’m looking forward to seeing our participation in World IPv6 Day as well.

Go AOL!

MySQL DBA (Database Administrator) Opening at AOL

Full details: http://bit.ly/19Ea7I

Contact: Carl.Coppadge at corp.aol.com or AIM: carlcoppadge11

Location: Dulles, VA

MySQL Database Administrator

AOL’s People Networks Operations, which operates AIM and ICQ, has an opening for a Sr. MySQL Database Administrator. This position would be responsible for managing large, highly scaled production MySQL databases as well as pre-production (QA/Dev/Staging) environments.

Key Responsibilities:

  • All aspects of MySQL deployment, operation,and design to ensure high reliability and performance
  • Collaborate with developers and architects to create high-performance, cost-effective designs
  • Participate in an on-call rotation as part of a 24×7 operations team to resolve urgent production issues
  • Ensuring data integrity with good process and a keen eye for detecting errors and misuses of data

    Desired Skills for this Position:

  • Knowledge of database architecture concepts as well as MySQL-specific implementations
  • Significant experience with MySQL in a high-volume production environment
  • Experience with open source ETL packages and methods for large scale data migration
  • Diverse technical background with awareness of concepts in networking, Linux, and storage
  • Bachelor-level degree in Engineering, Computing, or Sciences orequivalent experience
  • Replication: managing replication delay, scaling replication throughput, and designing resilient systems
  • Configuration: tuning InnoDB, managing memory usage, and tuning file systems for maximum throughput
  • Reliability: backups under load, failover strategies, and recovery from replication issues
  • MySQL: familiarity with Percona and Google patches

    Our perfect candidate can manage many rapidly changing projects while maintaining professionalism and poise. Our environment is both highly exciting and highly demanding — those who thrive here adopt a “work smarter, not harder” attitude.

How NOT to Inform Your Customers of an Outage

There are a number of different ways to inform your customers of an outage. I’ve previously discussed how 365main and Amazon Web Services did this fairly well in the past. Unfortunately, Limelight Networks customers are hearing about issues with their CDN via GigaOM.

Continue reading

Complexity and the 4 a.m. test

 

With most technology, it’s a given that there’s almost always More Than One Way To Do It (unless you worship Python). There are always those situations where choices must be made, and different people use different yardsticks to decide. Some try to minimize “cost,” either up-front development cost or long-term engineering cost. The smarter ones have recognized the concept of “Technology Debt” as addressed by several observers. As a leader in Operations, however, I tend to subscribe to my own rule: the 4 a.m. rule.

Continue reading

The Art of the Post-Mortem

I’ve mentioned in the past that the failure of complex systems is an inevitable fact of nature. The corresponding act of human inquisition into the reasons for that failure are equally inevitable. Where I work — and almost every other large installation I’ve seen or been part of — the learnings from these inquisitions are shared for educational reasons. The name for this differs from company to company: some call it a RFO (reason for outage) or an After-Action Report, but for whatever reasons the name for this at AOL is a Post-Mortem.

Continue reading

Velocity and Structure08

A whole lot of conferences are happening this week, and I’ll be attending two of them. On Monday and Tuesday of this week I’ll be attending O’Reilly’s Velocity conference, where I’ll be moderating a panel entitled “Everything You Ever Wanted to Know about CDNs (but were afraid to ask).” I’m hoping that seems to be fun, but there ought to be a lot of other interesting people I’d like to see while there as well, including two other very smart folks from AOL (Mandi Walls and Eric Goldsmith). I’ve been thinking about this as “Web 2.0 Expo without all that boring UI and Business Stuff”. 


Velocity, the Web Performance and Operations Conference 2008

The second event I’ll be at will be GigaOM’s Structure 08. Cloud computing is really leveling the playing field, giving small start-ups access to world-class operational assets… which to me only underscore the importance of having brilliant Ops folks to run those systems. I’m eager to see what sort of discussions emerge.

If you happen to be at either, give me a buzz in the comments, and I’ll try and catch up with you.