APCCIRN-015

APCCIRN-015

1992.6.12

Proposal for Global Internet Connectivity

IEPG Working Document

Guy Almes, Peter Ford, and Peter Lothberg

12 June 1992

I. Introduction

A significant subset of the IEPG membership has advocated a

common forum for the coordination of world-wide research and non-

research networking. At the November 1991 meeting in Santa Fe, the

idea of a global internet exchange emerged in basic form. At the

March 1992 informal meeting, the idea was discussed in a day-long

meeting at which goals and basic concepts were shared and at which

a consensus began to emerge. At the June 1992 meeting in Tokyo, a

detailed proposal was discussed and refined. This paper documents

the ideas that may allow us to move the ideas from discussion to

plan to implemented reality. Key technical goals are to maximise

connectivity, promote appropriate routing, promote effective

sharing of resources such as links, and enable a variety of transit

options. Key non-technical goals are to ensure the participation

by all segments of the Internet community -- both the traditional

CCIRN constituency and others -- in working toward a high-quality

world-wide Internet.

The reader should consider that the substantive ideas of the

paper reflect discussions with several people. The helpful

suggestions of Tony Hain, Geoff Huston, Bernhard Stockman and

Claudio Topolcic are especially appreciated.

II. Goals

The proposal addresses the following technical and non-

technical goals:

<> Maximal Connectivity: Enhance access and connectivity of the

global Internet. Interconnectivity of networks should be

enhanced, including those networks which are not directly

funded by traditional CCIRN constituencies.

<> Cost-effective Transit: Make available to all our networks the

best available possibilities for worldwide transit. To the

extent possible, promote the sharing of global transit

resources and the sharing of costs.

<> High-quality Routing: Work toward routing management that

supports the connectivity and transit goals while being

technically optimal.

Each of these goals is expanded in turn.

In addressing maximal connectivity, we debated on whether

'universal' or 'maximal' was the right term. There was no

question, however, but that, in the words of one member "Connectiv-

ity is the key goal". There are several possible limitations on

connectivity. Some, such as connectivity limits imposed on a

particular community of interest, are unavoidable and even

desirable. Thus, a term such as ~selective connectivity~ might be

appropriate. Others, such as any limits imposed by usage con-

straints on intercontinental transit networks, are undesirable in

the long run. Transit networks and interconnection structures open

to both research, education, and other traffic will aid the broad

growth of the Internet. To summarise, we want to promote, enable,

and maximise universal connectivity among transit networks, so that

the only limits on universal connectivity are imposed by user

communities.

In addressing cost-effective transit, we noted that there is

a diversity of transit needs: some networks need universal

affordable transit, for example, while others need higher-perfor-

mance transit even if costs are not minimised. Similarly, there is

a diversity of transit possibilities: there are at least five

national providers in the US, for example, and we could contemplate

new transit structures. To summarise, we should work to harness

the variety of transit options to serve the diversity of transit

needs. At the same time, we should ensure that issues surrounding

transit do not limit interconnectivity, defeat our routing goals,

or adversely effect existing networks.

In addressing high-quality routing, we emphasised routing as

a means to supporting the connectivity and transit goals mentioned

above, but also noted technical symmetry, stability, and manage-

ability as explicit goals. Though it might go without saying, we

also agreed that we should use the best of currently available

routing techniques, while not requiring what does not exist. At

the same time, it is desirable to provide a platform for the early

deployment of advanced routing tools. To summarise, we should work

together to build worldwide routing and routing management in

support of maximal connectivity and cost-effective transit, while

using the best available techniques to support technical qualities

such as stability and symmetry.

Along with these goals, we also agreed on the following

qualities that must apply to our approaches to them:

- -- Scalability: We understand that the size of the Internet,

measured either in traffic or number of connected networks, is

growing exponentially, and that our engineering and operations

must scale.

- -- Manageability: We understand that the worldwide Internet must

be as well-managed as one under a single administration, and

yet recognise the autonomy of each constituent network.

- -- Accountability: We understand that any successful worldwide

connectivity structure must be accountable.

- -- Timeliness: We understand that the cost of staying with the

status quo, or the cost of delay, is very high. Maintaining

the integrity of the Internet requires prompt action.

In approaching solutions to these goals, we often had to deal with

tensions among competing goals and the limits imposed on our

technology.

III. Our Vision -- Technical

<briefly, a set of global interconnect points at which

cooperative advanced inter-AS routing is supported and at

which transit and regional networks meet. Universal

connectivity, good routing, and several transit options

are supported.>

At several places around the world, we envision global

interconnection points. Each such point would consist of a managed

facility with 24-by-7 coverage and excellent environmental support.

At each point there would be a high-speed broadcast LAN freely

available for all kinds of traffic. Each participant would be free

to, at its expense, bring a circuit to this facility and place a

router on the LAN. It would then be free to exchange routes and

traffic with (the routers of) other participants at that LAN. Each

participant would also pay for a pro rata share of the cost of the

floor space, environmental support, and administrative support

required for the interconnection points.

At this level of detail, we are simply describing the

engineering of the current US FIX structure. From a technical

point of view, we consider that it is the best approach among

current models. There are, however, several non-technical

requirements needed:

>> An Open Forum for Coordination. This forum would provide full

participation for non-US and for non-research-and-education

networks.

>> Very Broad Usage. A global interconnection point must be able

to pass research, education, and other traffic. Some connect-

ing networks may still have research/education AUPs, but the

interconnection structure should not.

In addition, we need solutions to several technical shortcomings of

the traditional FIX engineering structure:

>> N-squared Routing. All routing is done on the basis of pair-

wise peering among pairs of routers on the LAN. We consider

that, as the use of the interconnection points grows, this may

not scale.

>> Anarchic Routing. When a router on the LAN advertises a route

to a given destination network, its peer has no basis for

knowing whether that router is authorised by the destination

network to advertise it.

>> Destination-based Routing. If we were able to route across

the interconnection points on the basis of both source and

destination, then we could better use the variety of transit

possibilities, including federally funded ones, without

limiting connectivity due to policy issues. Our inability to

do so limits these choices, and also leads to undesirable

asymmetric routes.

In order to achieve our technical vision, each of these shortcom-

ings will need to be dealt with. In many cases, these problems

will be dealt with by means of improved routers of participants and

not by means of new technology in the interconnection points

themselves.

N-squared Routing. We anticipate that the number of routers

at the interconnection points and the number of network numbers

advertised by each router will both increase dramatically. In

order to cope with this, we will provide for the deployment and

management of a set of Route Servers on each of the interconnection

points. In simple terms, Route Servers use inter-AS routing

protocols such as BGP to learn and advertise routes, but do not

themselves participate in the forwarding of packets. Networks that

attach will have the option of using these Route Servers, of using

traditional pair-wise peering, or of using some combination. There

may, in fact, be multiple different Route Servers at a given

connection point used by different sets of participants. The

intention of Route Servers is not to impose policy, but to

implement the dissemination of routes in a manner that scales and

can be well managed.

Such Route Servers might also help solve some of the so-called

ROAD problems. For example, CIDR support in a Route Server could

help even in the period before CIDR is supported in the routers

used within a participating network. Similar examples could be

given with respect to Policy Routing and Source Demand Routing.

Route Servers will be a subject of Section V (Research

Efforts) below, and its deployment and management will be a subject

of the cooperative routing efforts discussed throughout the paper.

Anarchic Routing. One of the major successes of the NSFnet

Backbone Service has been the Policy Routing Database, a database

designed and administered by MERIT which records the ASes author-

ised to advertise certain IP network numbers to the Backbone. By

correctly and faithfully administering this database, MERIT

provides a communication link, as it were, between network managers

who set and agree on policies and the routers which implement

routing exchanges. For each IP network number, for example, the

database records the primary, secondary, and (sometimes) tertiary

ASes that can advertise that network number.

We anticipate a similar need at the global interconnection

points. Rice University's network number (128.42), for example,

could be authorised for the NSFnet Backbone Service as primary and

SprintLink as secondary (and used only for backup). A global

routing Registry would be designed and administered, and this

Registry would be instantiated at each interconnection point.

Registry management would be selected to ensure the neutrality of

the Registry with respect to possibly competing networks that

connect at the Interconnection Point. Tools could be written to

produce derivatives of the Registry for particular engineering

purposes, such as Cisco access lists for the control of route

filtering. If we can judge from experience on the NSFnet Backbone

Service, such a Registry would provide a framework in which

misunderstandings and conflicts in routing could be avoided and/or

resolved before they could cause undesired asymmetric routes or

other problems.

Destination-based Routing. There are currently many circum-

stances in which several transit networks must be traversed from

source to destination, in which several alternate paths are

possible, and in which constraints (such as AUPs) or preferences

(based on cost or performance issues) will influence the selection

of the most appropriate path. Further, not all of the constraints

and preferences can be cast in terms of traditional routing which

considers only the destination IP address. Examples of needed

improvements include:

- -- TOS Support. Some traffic may be tagged to require low

latency (e.g., telnet) or high bandwidth (e.g., image trans-

fer) or low cost (e.g., email for high schools). To some

extent the TOS/QOS concept may support this, but operational

support for this on a worldwide basis is beyond current art.

- -- Source-based Routing. Some traffic may be appropriate to

carry over special-purpose networks due to identification of

both source and destination. For example, ESnet might be

willing to serve as a transit network, but only for traffic

from one energy-community site to another.

Similarly, consideration of both source and destination

might be useful in avoiding asymmetric routes. For example,

in the case of traffic exchanges between a university and a

commercial site, destination-based routing often leads to the

use of research transit networks in one direction and commer-

cial transit networks in the other, even when the research

transit network can support the full exchange.

- -- Flow Support. Some traffic may be part of a flow that

requires a form of bandwidth reservation supported by one

special transit network, and that transit might be configured

to support that flow. Support for global conferencing might

require this flow to be carried over general infrastructure

networks for part of the path and special transit networks for

other parts.

Routers in common use on the Internet do not support any of these

abilities, but experience with DARTnet and with wider experiments

with IDPR suggests that some of them could be deployed in the near

future.

An Open Forum for Coordination. We anticipate the need for

the full participation of engineers from all connecting networks in

the coordination of routing and management across the global

interconnection points. The global routing Registry would provide

one kind of communication path to allow for this coordination to be

natural and convenient in normal cases. The famous RIPE coffee

breaks offer another needed tool, but can only happen less

frequently. Advanced collaboration support tools, including

conferencing, shared blackboard, and email-based tools, could

provide some technical support tools for this effort.

The FEPG within the US agencies and RIPE within the European

IP community provide useful precedents. The key is effective and

full participation by networks in issues of concern to those

networks. This may require a formally incorporated organisation as

a structural umbrella.

Very Broad Usage. There will be no AUP constraints on the

interconnection points themselves. Further, by enabling effective

routing and by bringing several transit alternatives to the

interconnection points, we will enable participating networks to

cope responsibly with the AUP issues that remain.

IV. Our Vision -- Non-technical

<briefly, a structure to do collectively what must be

done collectively to accomplish the stated goals. This

collective action will probably require some organisa-

tional structure, and this structure should respect

autonomy and be accountable.>

Although the IEPG can manage the design of technical solutions

to our goals and although we perceive very broad agreement with the

goals and technical structures we propose, our proposal cannot

succeed without some cooperative structure for managing the global

interconnection points and setting the policies that govern it. At

a minimum, we need a small organisational structure that can do

this necessary management and governance.

The structure needs to secure the fullest possible participa-

tion of the Internet community. This must include networks from

North America, Europe, and the Pacific Rim. It must include

government agencies, university consortia, and for-profit and not-

for-profit companies. It must include networks that support

research, education, industrial, and mixed traffic.

The structure needs to be representative of and accountable to

all its participants. Thus, what policies the structure does set

must earn the kind of authority that comes from such accountabili-

ty.

The structure needs to address what must be done collectively

without itself becoming a network in competition with its partici-

pants. Thus, for example, the cooperative structure would not

provide connectivity directly to any user site. Similarly, the

cooperative structure would be careful not to favor on competing

participant over another.

The structure must avoid deciding issues best left to the

participants themselves. Thus, for example, the structure should

avoid deciding which networks should use which other networks for

transit, or how various pairs of networks should do settlements.

It is important to remember that the purpose of our efforts here is

to cooperatively harness the strengths of our various networks in

support of the Internet community -- not to replace our networks

with a homogeneous Internet.

Among the organisations that could help us find the right

organisational structure to meet these objectives are the CCIRN,

the Internet Society, EDUCOM, RARE, FARnet, and PACCOM.

Fortunately, there is near-universal recognition of the mutual

benefit involved in full connectivity.

V. Research Efforts Needed to Achieve the Vision

<briefly, we need specific research and development

efforts to improve on the routing techniques and network

management techniques available to us>

Several research and development efforts need to be conceived,

funded, executed, and deployed in order to achieve all the aspects

of our technical vision. Among them are:

>> Advanced Route Servers. Very urgent. While pair-wise routing

works today, scaling for the future will require successful

design and deployment of Route Servers. Initial experience

could take place using third-party BGP techniques with current

routers, and CIDR routers could become available in the near

future.

>> Source-based Routing. Early experience with source-based

inter-AS routing should be evolved into operational routers

that could pass packets among participant routers at the

interconnection points. These routers could allow us to avoid

asymmetric routes, but would have to be both sophisticated in

the policies supported and very fast to avoid becoming a

performance bottleneck.

>> TOS/QOS-based Routing. In order to support high-performance

applications on a worldwide basis, we will need to support

TOS/QOS at the interconnection points. This will need to be

supported in a way that interoperates with other emerging

approaches.

>> Flow-based Routing. As the concept of flows becomes better

defined, support for it at the interconnection points would

add significant value to the worldwide Internet. Routers that

recognise and expedite the forwarding of flows would be of

great utility at the interconnection points.

Some of these projects involve multi-year research and development.

Others, such as simpler versions of Route Servers and CIDR, may be

deployable within a year.

VI. Immediate Non-technical Agenda

<briefly, work with the Internet Society and the CCIRN to

set up an accountable cooperative structure to accomplish

the technical agenda. This should be in place by summer

1992 >

Under the leadership of such collective organisations as the

Internet Society and the CCIRN, we should immediately set up a

structure that can allow the first step at our technical agenda.

Efforts should be made to accomplish this by summer 1992.

We will consider several possibilities. One very specific

possibility, which we would view positively, follows. The CCIRN

and the Internet Society would charge a task force of the IEPG to

develop a more detailed design, mindful of the very similar agenda

before the US Interim Interagency NREN project. Considering the

urgency of providing an initial global interconnection point during

calendar 1992, and continuing this specific example, this task

force will attempt to liaise very closely with the NSF. Within

this strategy, we hope to leverage activities already planned at

FIX-E.

There are other possibilities that we will explore, but we

view this as the most hopeful.

VII. Immediate Technical Agenda

<briefly, set up one interconnect point using existing

routing and network management technology. It is likely

that the NSFnet Implementation Plan and the FEPG Interim

Interagency NREN Architecture, both of which are converg-

ing during summer 1992, will influence the resulting

design. In NSFnet/NREN terms, this section calls for a

NAP on the east coast of the United States.>

At one place on the east coast of the United States, we

envision a global interconnection point. This point would consist

of a managed facility with 24-by-7 coverage and excellent environ-

mental support. At this point there would be an FDDI ring freely

available for all kinds of traffic. Each participant will be free

to, at its own expense, bring a circuit to this facility and place

a router on the ring. It would then be free to exchange routes and

traffic with (the routers of) other participants at the ring.

N-squared routing would be addressed by the provisioning of

route servers. This would take some time to perfect, but it would

be taken on immediately both for experimentation with the route

server idea and to permit scaling of the rapidly growing route

tables we all keep.

Anarchic routing would be addressed by an interim Routing

Registry. One possible approach to this would be to adapt for our

use the registry the NSF will be setting up for the NSFnet Backbone

Service.

We would live with destination-based routing and forwarding,

but would immediately start some experiments with enhanced

forwarding. These experiments would not be allowed to compromise

the operational stability of the interconnection points.

The IEPG task force discussed in Section VII would meet

quarterly to work out problems and evolve the structure. Even

between meetings, this task force would serve an ombudsman role and

would oversee the interconnection point to ensure that the

interests of the international community are being well served. The

meetings should be coordinated with existing meetings such as RIPE,

IETF, and the Internet Society.

There would be no AUP constraints on the interconnection

points, themselves.

VIII. Evolution to the Vision

<briefly, the cooperative structure guides the immediate

technical structure toward the vision, making use of the

results of the stated research efforts>

Under the leadership of the CCIRN and the Internet Society,

the interconnection points and their management would evolve. This

will eventually lead to the need for us to procure the services

needed to support and manage multiple interconnection points. This

will require organisational maturity in order to manage such a

procurement, and in order to have an authority structure that

combines credibility, broad trust, and accountability to the

international community.

Similarly, we will need to ensure progress on the research

agenda required for high-quality interconnection points for the

future. This will pay off enormously when the time comes to

increase the number of interconnection points.

Similarly, after the initial single interconnection point is

established, a set of design tasks will be required prior to our

being able to field a second site. Examples of issues are the

coordination of the multiple instantiations of the Routing Registry

and the patterns by which the various options of transit among the

multiple sites are handled.

Later, we will also need to reconsider the technical basis of

the interconnection point as Internet technology evolves. During

the era of fractional T1 intercontinental links, our current choice

of an FDDI ring seems more than adequate. As more advanced

transmission media such as 150 Mb/s ATM with support for isochro-

nous traffic become deployed in the Internet, we will surely need

to reconsider this technical basis.

On all these areas of work, technical service, non-technical

management, and research management, we will need to ensure that

the broad needs of the international Internet community are being

addressed.

IX. Conclusion

<briefly, the Internet is saved and the IEPG wanders off

to the bar>

This paper has proposed a serious program for improving the

global interconnectivity of the Internet. The goals of maximal

connectivity subject to community-of-interest constraints, cost-

effective transit, and high-quality routing are addressed, and

attention to short- and long-term efforts, technical and non-

technical efforts, and research agenda is paid. There is solid

hope for success provided that these efforts are begun very soon.