Introduction to a double issue on

Peering, Transit, and IXs: Economics and Policy

Find out how to order single copy ($125 part 1) or group license ($250) for just the November issue.

Contents:

Economics of IXs, Peering, and Transit

Reaction and Comments by Andrew Odlyzko .....p. 4

Why Peering and Transit Is Badly Broken at the Global Tier One Level

Bill Woodcock Explains New Framework on Which to Build Peering and Transit

ICANN's Season of Delusions: Attempt to Spin Court Defeat is Rebuffed in IETF List

RIAA Runs Amok, Sues Four Backbones Over Chinese Site and the Sues Verizon

Old Time Net Architect Makes Mistake with Affidavit in Support of First RIAA Suit .....pp. 35 - 39

Economics of IXs, Peering, and Transit

Development of Bandwidth Cost Management Capability May Be Critical to ISP Survival

Coping with the Internet Core Oligopoly Demands Innovation

A Synthetic Path Analysis Examination of Netflow Data

Introduction

This issue of the COOK Report explores peering, transit and exchanges for the first time since about 1999. While a lot has changed, a lot remains the same. The Tier 1's oligopoly of peering only with themselves is still well entrenched. Farooq Hussain has written for this issue a remarkably candid summary of the evolution of the Tier 1s peering policy. They are, he says, the Internet Core Networks that announced anonymously on December 5, 2001 their decision to move their peering to Equinix Exchanges. He identifies them as UUNET, Sprint, Cable and Wireless, Genuity, Level 3, Qwest, and AT&T. He also finds their peering requirements to be arbitrary beyond reason. For example, interconnection at OC48 is one thing, but to be forced to do so at 15 locations around the United States is something else again.

Estimates of the capacity utilization of the Tier 1 backbones show them to be lightly utilized at about 15 to 20%. Given this situation Sprint, for example, is undoubtedly quite happy to have SBC buying nine OC-48s. Because the ISP and backbone industry is unregulated, what knowledge we have is sketchy and largely subject to the willingness of folk who both know, and will take the risks of speaking up. Given the state of the industry such folk are few and far between. Over the past five weeks we have had conversations with a dozen or so people who are closely involved. Some of these suggest that the oligopoly is engaging in behavior that could blow up in a manner similar to the capacity swaps that blew up earl this year. [See sidebar exchange between Miles Fidelman and Sean Donelan.] Certainly with all seven loosing money and four of the seven (UUNET, Genuity, Qwest and Level 3) either in bankruptcy or in dire financial difficulty, Internet Core behavior is not likely to become customer friendly.

Folk associated with the carriers complain that long haul transit backbones are being forced to sell transit essentially below their cost and suggest that something about the system has to change. Of course since almost everything is kept under strict non-disclosure and since these seven players are also either telephone companies, or associated with telephone companies, opportunity for subsidizing operations from the telco side are there. The United States government has defined the Internet as a critical telecommunications infrastructure. Yet, one of many ironies of the current situation is that virtually no one in the federal government nor in the financial community knows with accuracy how strained the financial position of each company is.

It's the Economics of Networks, Stupid!

Consequently, with the industry collapse, the tightly run economic aspects of one's business matter in a way that they never did before. In this context, some important new tools and concepts are being developed in the peering and transit arena. In addition to Farooq's examination of the current global climate at the Tier 1 level, this issue looks at the development of some tools and methodologies to manage, much more cost effectively, an ISP's interconnection costs.

Since about 1998 Cisco (and now Juniper) routers have had the capability of giving the users what is called Netflow data. Use of the Netflow data can give significant information about where a network's traffic is going, including what autonomous systems the ISP's traffic flows through to reach its customers. Various efforts are underway. A small handful of ISPs of differing sizes are beginning to use tools by companies like Route Science and SockEye to do load balancing of their up stream connections in real time. Some folk are also beginning to build tools using Netflow data to help them most cost effectively analyze how to do their transit and peering in the first place. One such effort is by Jeffrey Pappen, Peering Coordinator for Yahoo. It is called TUNDRA, The Ultimate Netflow Data Realtime Analysis, and was presented by Pappen at NANOG in October 2001. See www.nanog.org/mtg-0110/ppt/tundra.ppt Another is by Martin van den Nieuwelaar and is called Network Intelligence. See http://www.networkintelligence.biz Stephen Stuart at PAIX has looked at Netflow approaches and applied some of them there.

Another, and likely the most important effort, has been developed by Bill Woodcock of Zocalo.net, and Packet Clearing House and Alex Tudor at Agilent technologies. This is discussed in great detail in a 16,000 word interview with Bill Woodcock in this issue (November 2002). We note that Woodcock's approach is the subject of a patent application. In the language of the legal department of Agilent. "Some of technology described in this article [our interviews] has been claimed in one or more pending patent applications that are owned jointly by Zocalo and Agilent Technologies, Inc., each of whom is free to license." As we went to press on September 2 Bill explained it more clearly: "The deal between me and Agilent was that I would do a "technology transfer" and tell them what code to write. They wrote the code at my instruction, and we jointly filed a patent application to preclude someone else's making proprietary claims against it. The content of the patent disclosure and application are open source, and free for public use. Again, the purpose of the patent application, is to protect open-source use of the algorithms against future claims that they're someone's proprietary invention."

Finally we note that from, an economic point of view, the most important component of what Woodcock is doing is what he calls Synthetic Path Analysis. That analysis is publicly discussed below for the first time. It has not yet been tested in production use over a period of six month to a year's time. This should be done. Until it is, we can say only that from an intuitive point of view it makes strong sense. Certainly ISPs should be looking at experimenting with it.

We think that what Woodcock says is extremely significant. Among other things, he points out that the Tier Ones, by peering in their tight oligopoly, may have rendered themselves irrelevant. Why? Because the smaller networks with rich peering are beginning to build such a well connected donut around the Tier 1's that one may be on the verge of being able to deliver one's traffic to all of the destinations to which it needs to go without relying on the Tier 1's for transit.

In short, ideas and methodology are evolving. The new topology looks more like a multifaceted geodesic dome than like a dozen global backbones with networks hung off them tree-branching fashion. Critical components of this new topology, in addition to peering and transit circuits, are the hubs into which the circuits are attached. These are the more than 300 exchange points around the world that facilitate network interconnection. There it seems likely that newly cost-conscious ISPs will increasingly deploy Netflow data methodology to model their traffic and decide from the results where and with whom to interconnect. When we described to the technology director of a large CLEC what they do, he said: "that sounds pretty much like how we model and plan our long distance circuit interconnections with the PSTN. Do it right and you are profitable. Do it wrong and you are history."

Of course what is right and what is wrong is likely to be a function of where one is in the industry. Below the Internet Core oligopoly one has a group of very large players who are themselves under stress and therefore must find strategies for survival. This group includes AOL, SBC, Yahoo, Shaw Cable, France Telecom, Equant, Verizon, Bell South and many foreign carriers like Telstra and NTT. See for example http://www.sbcbackbone.net/peering/ They have their own ideas about peering and given cooperation by them we may be able to begin to explore them in the future. (No promises because such cooperation is not yet firmly in place.) Beneath this group are the several thousand smaller ISPs with which Woodcock works. We believe that both these groups could profit from Woodcock's understanding.

Future Direction

Vertically integrated local phone companies are forced to have a business model that squeezes every drop of money from every drop of traffic if they are to get enough income to pay the interest on their bonds. Forced into this procrustean bed, they will find that the economics of an industry concentrated on broadband with user control at the edges and less and less concern with extracting rent from the content are capable of benefiting from the declining costs of bandwidth and the hardware necessary to support it. The local companies will either adapt to the new model through bankruptcy or find through political intervention a way to bail out their debt ridden foolishness and go merrily on as before promoting bandwidth scarcity and ensuring that the telecom industry in the US in the 21st century shares the fate of the steel industry in the 20th century.

The questions facing us are no longer just ones of technology. They are ones of economics and policy. How we answer them will determine whether our information technology industry prospers or stagnates. One thing that we find fascinating about Woodcock's peering and transit methodology is that it may begin to provide an answer to the question of how asset based telecom and the fringes of the network can stitch themselves together to begin to replace the long haul carrier networks.

We begin this issue with Farooq Hussain's overview of global peering and transit and conclude it with the two part interview with Bill Woodcock. Bill suggests a very interesting world view where ISPs that are clueful and careful can make themselves competitive by avoiding interconnection with their ILEC and backhaul circuits to appropriate exchange points. They should peer as much as possible at an exchange where the cost of interconnection is as cheap as possible. Reliability is not critical because, should peering sessions fail, traffic can be delivered through the transit providers. Since transit is critical one should connect to at least two transit providers ideally at different exchanges. One figures out in each specific instance how to do this using the path analysis tools and then one simply does it.

REACTION AND COMMENTS BY ANDREW ODLYZKO

Editor's Note: We have once again a group of about a dozen experts in peering and transit gathered in a private mail list discussion. In the December issue to be published about October 1 we shall publish an edited version of that discussion. In the meantime we asked Andrew Odlyzko if he would comment on what we had concluded so far.

Odlyzko on September 2: [Herewith some comments on] the latest messages, in particular Farooq's historical sketch and Gordon's Introduction to the November issue. I find the prospects of smaller networks being able to bypass the Tier 1s fascinating. The development of tools, such as those of Bill Woodcock and others (listed by Gordon) is also very interesting, and for several reasons: (i) it should accelerate the evolution that Bill and others are describing, (ii) it offers opportunities for interesting technical research on improving the tools, and (iii) the fact that such tools, or even attempts to do such optimizations manually, have not been used much in the past, confirms the general conclusion I have drawn some years ago, namely that optimization of the usage of physical resources is not a major priority, that dealing with general complexity matters far more.

The last point, (iii), might be worth spending more time on. There are various arguments that support it. For example, given that data traffic is typically asymmetric, and traffic profiles are pretty stable, why doesn't the industry develop photonic and electronic systems that would allow for switching the direction of transmission on a fiber strand a few times per day? Yet I have not seen anyone talk about such systems, although my friends in the photonics area tell me it would not be too hard to to. Another point is the low utilization of backbones. Gordon in his Introduction mentions an estimate of 15 to 20%. My guess is that it is probably lower, more like 10 to 15% (and conceivably far lower, because we have all those new players that have built out their networks in the anticipation of a huge growth of traffic that has not materialized). As an example, consider AboveNet, with traffic data available at . During the week ending Aug. 24, the average weekly utilization of their long-haul links was 11.5%. I have a variety of other arguments for (iii) in my papers.

The reason for spending this length of time talking of point (iii) is that it is relevant to the main question of this list. I my previous message, I raised the question of just how important the cost reductions from peering are, given the low cost of peering. I have not heard any comments disproving my estimates that once an ISP (in the US, I am not necessarily talking of other parts of the world) aggregates enough traffic to fill an OC3, say, the costs of transit are not all that high. Now the assumption that pervades the discussion on this list is that costs of peering and transit dominate. In particular, Gordon in his Introduction writes about "the technology director of a large CLEC [who] said: "that sounds pretty much like how we model and our long distance circuit interconnections with the PSTN. Do it right and you are profitable. Do it wrong and you are history." I can well believe that this used to be the case for voice connections, but would like to see some real data for what goes on today. It was surely also correct a few years ago for ISPs, when transit was very expensive. But is it true today? I would like to see some real data. My strong impression is that while the costs of transit have plummeted, the prices paid by the end users (at the T1 and sub-T1 level, where the bulk of the revenues are) have declined much less, and of course there is a far greater density of them.

Along the same lines, transit revenues from ISPs that can get the $150/Mbps per month prices from Tier 1s simply do not come to all that much. Most of the money the Tier 1s make comes from end users. That should be kept in mind when evaluating the tactics those guys use in setting their peering policies.

There are some other basic assumptions in this discussion that I have doubts about. For example, in his first interview with Gordon, Bill said

Woodcock: It probably doesn't change things. My point is that if you are buying transit from two providers and you are adding your costs and your profit margin and reselling the result, you price to customers is going to be higher than their price to their customers. And your customer might as well bypass you and go directly to them. You have no value add.

Is that really so? If you are buying transit from two providers in the OC3 capacity, say, and your customers are buying T1s from you, you are providing real service (aggregation of traffic, as well as all the hand-holding for the customers, etc., which is where most of the costs are). An oil refinary (not part of an integrated business) sells to wholesalers, who sell to gas stations. This is very common, and all stages have value add. This is especially true on the Internet, where there has been little successful vertical integration. Bill talked about increasing difficulty in ISPs differentiating themselves on the basis of quality. However, we still see large differences in prices (about a factor of 2 in prices for T1s, I hear), which shows that Internet connectivity is not a commodity.