The big things in the Internet

At Solder and Circuits, the biggest networking device is the $270 Cisco Layer-3 switch. It is able to route packets between subnets.

A bigger device would be a router.

It is interesting to think about the biggest things in the Internet. These would be the two-acre data centers (server farms), the fiber-optic infrastructure buried underground in orange tubing (or in the sea floor), and the refrigerator-size routers, about 100 per Internet Exchange (IX).

This web page talks about big routers and big Internet Exchanges.

This refrigerator-size router is a Brocade MLX. It accepts up to 32 plug-in router boards, at which point it is probably valued at $130,000. If an Internet Exchange has 100 of these, that is $13 million dollars.

http://www.brocade.com/products/all/routers/product-details/netiron-mlx-series/index.page

The Brocade MLX Series of advanced routers delivers unprecedented scale and performance for the world’s most demanding service provider and enterprise networks.

Brocade MLX Series routers are designed to enable cloud-optimized networks by providing industry-leading 100 Gigabit Ethernet (GbE), 10 GbE, and 1 GbE wire-speed density; rich IPv4, IPv6, Multi-VRF, MPLS, and Carrier Ethernet capabilities; and advanced Layer 2 switching.

Built with sixth-generation architecture and terabit-scale switch fabrics, the Brocade MLX Series has a proven heritage with more than 10,000 routers deployed worldwide. Internet Service Providers (ISPs), transit networks, Content Delivery Networks (CDNs), hosting providers, and Internet Exchange Points (IXPs) rely on these routers to meet skyrocketing traffic requirements and reduce the cost per bit. By leveraging the Brocade MLX Series, mission-critical data centers can support more traffic, achieve greater virtualization, and provide high-value cloud-based services using less infrastructure. The Brocade MLX Series can reduce complexity in large campus networks by collapsing core and aggregation layers, as well as providing connectivity between sites using MPLS/VPLS.

Brocade MLXe Routers have up to 15.36 Tbps (15,360 Gbps) of routing capacity. All chassis types provide an astounding 4.8 billion packets per second routing performance and feature data center-efficient rear air exhaust. Both models are available in 4- to 32-slot chassis, and deliver up to 256 10 GbE, 1536 1 GbE, 64 OC-192, or 256 OC-48 ports in a single system.

Designed for non-stop networking, the Brocade MLX Series features Multi-Chassis Trunking (MCT), which provides more than 30 Tbps of dual-chassis bandwidth, full active/active routing links, and uninterrupted traffic flow in the event of node failover. Organizations can achieve high resiliency through fully redundant switch fabrics, management modules, power supplies, and cooling systems. And to further ensure network and application availability, the Brocade IronWare operating system features hitless management failover and software upgrades.

Software-Defined Networking (SDN) is a powerful new network paradigm designed for the world's most demanding networking environments. The Brocade MLX Series enables SDN by supporting the OpenFlow protocol, which allows communication between an OpenFlow controller and an OpenFlow-enabled router. Using this approach, organizations can control their networks programmatically, transforming the network into a platform for innovation through new network applications and services.

------------------------------------

http://www.datacentermap.com/ixps.html

IXP list

IXP Area Country ▴ Locations

VNIX - Vietnam National Internet eXchange Hanoi Vietnam N/A

MaNAP - Manchester Network Access Point Manchester United Kingdom 3

LONAP - London Access Point London United Kingdom 4

LIPEX - London Internet Providers EXchange London United Kingdom 5

LINX - London Internet Exchange London United Kingdom 10

EMIX - Emirates Internet Exchange Abu Dhabi United Arab Emirates N/A

UA-IX - Ukrainian Internet Exchange Kiev Ukraine N/A

PHX-IX - The Phoenix Internet Exchange Phoenix, AZ USA N/A

TampaIX - Tampa Internet eXchange Tampa, FL USA N/A

OIX - Oregon Internet Exchange Eugene, OR USA N/A

PitX - The Pittsburgh Internet Exchange Pittsburgh USA N/A

TIE-PHX - Telx Internet Exchange Phoenix Phoenix, AZ USA 1

CIIX - CENIC International Internet eXchange Los Angeles USA 1

LAIIX - Los Angeles International Internet eXchange s USA 1

SFMIX - San Francisco Internet eXchange San Francisco USA 1

TMRK-MIA - Terremark NAP of the Americas Miami, FL USA 1

PAIX-ATL - Peering And Internet eXchange Atlanta Atlant USA 1

TIE-ATL - Telx Internet Exchange Atlanta Atlanta USA 1

ChIX - Chicago Internet eXchange Chicago, IL USA 1

BOSIX - Boston Internet Exchange Boston, MA USA 1

Big-APE - The Big Apple Peering Exchange New York, NY USA 1

SOAX - Southern Oregon Access Exchange Medford, OR USA 1

EQIX-DAL - Equinix Exchange Dallas Dallas, TX USA 1

PAIX-VIR - Peering And Internet eXchange Virginia USA 1

SIX - Seattle Internet Exchange Seattle, WA USA 1

EQIX-LA - Equinix Exchange Los Angeles Los Angeles USA 2

EQIX-SJC - Equinix Exchange San Jose San Jose, CA USA 2

EQIX-CHI - Equinix Exchange Chicago Chicago, IL USA 2

KCIX - The Kansas City Internet Exchange Kansas City KS USA 2

TIE-NYC - Telx Internet Exchange New York New York USA 2

NWAX - The Northwest Access Exchange Portland, OR USA 2

PAIX-DAL - Peering And Internet eXchange Dallas Dallas USA 2

TIE-DAL - Telx Internet Exchange Dallas Dallas, TX USA 2

PAIX-SEA - Peering And Internet eXchange Seattle USA 2

EQIX-NYC - Equinix Exchange New York New York, NY USA 3

NYIIX - New York International Internet eXchange New York USA 3

PAIX-NYC - Peering And Internet eXchange New York New York USA 3

PAIX-PAO - Peering And Internet eXchange Palo Alto Palo Alto USA 4

EQIX-ASH - Equinix Exchange Ashburn Ashburn, VA USA 4

GN-IX - Groningen Internet Exchange Groningen The Netherlands N/A

R-IX - Rotterdam Internet Exchange Rotterdam The Netherlands N/A

AMS-IX - Amsterdam Internet Exchange Amsterdam The Netherlands 6

NL-ix - netherlands internet exchange Amsterdam The Netherlands 18

PIE - ThaiSarn Public Internet Exchange Bangkok Thailand N/A

CIXP - CERN Internet eXchange Point Geneva Switzerland 1

EQIX-ZRH - Equinix Exchange Zürich Zürich Switzerland 2

SwissIX - Swiss Internet Exchange Zürich Switzerland 9

GIX - Gothenburg Internet Exchange Gothenburg Sweden N/A

Netnod-GBG - Netnod Internet Exchange Gothenburg Gothenburg Sweden N/A

Netnod-LUL - Netnod Internet Exchange Lulea Lulea Sweden N/A

MalmIX - Open local Internet eXchange in Malmö Malmö Sweden N/A

Netnod-MMO - Netnod Internet Exchange Malmö Malmö Sweden N/A

Solix - Stockholm Open Local Internet Exchange Stockholm Sweden N/A

Netnod-SVL - Netnod Internet Exchange Sundsvall Sweden N/A

STHIX - Stockholm Internet eXchange Stockholm Sweden 3

Netnod-STH - Netnod Internet Exchange Stockholm Sweden 4

CATNIX - Catalunya Neutral Internet Exchange Barcelona Spain N/A

TMRK-MAD - Terremark NAP de las Madrid Madrid Spain 1

ESpanix - Espana Internet Exchange Madrid Spain 3

KINX - Korea Internet Neutral Exchange Seoul South Korea N/A

GINX - Grahamstown Internet Exchange Grahamstown South Africa N/A

JINX - Johannesburg Internet exchange Johannesburg South Africa N/A

CINX - Cape Town Internet Exchange Cape Town South Africa 1

NAPAfrica - Network Access Point Africa Cape Town South Africa 1

SIX-SK - The Slovak Internet eXchange Slovakia 1

SOX - Singapore Open Exchange Singapore Singapore N/A

EQIX-SIN - Equinix Exchange Singapore Singapore Singapore 1

SPB-IX - St. Petersburg Internet Exchange St. Petersburg Russia N/A

MSK-IX - Moscow Internet Exchange Moscow Russia 2

Balcan-IX - Balcan Internet Exchange Bucharest Romania N/A

BUHIX - Internet Bucharest Exchange Bucharest Romania N/A

InterLAN - InterLAN Internet Exchange Bucharest Romania N/A

RoNIX - Romanian Internet Exchange Bucharest Romania N/A

TomIX - Tomis Internet Exchange Constanta Romania N/A

IX-PR - The Internet Exchange of Puerto Rico San Juan Puerto Rico N/A

GigaPix - Gigabit Portuguese Internet eXchange Lisbon Portugal 1

PLIX - Polish Internet eXchange Warsaw Poland N/A

Thinx - Thinx Poland Warsaw Poland N/A

WIX-PL - Warsaw Internet eXchang Warsaw Poland N/A

PHOpenIX - Philippine Open Internet Exchange Makati City Philippines N/A

PHNET-CORE - The PHNET Common Routing Exchange Quezon City Philippines N/A

NAP-Peru - NAP.Perú Lima Peru N/A

NIX - Norwegian Internet Exchange Oslo Norway N/A

FIXO - Free Internet eXchange Oslo Oslo Norway 2

IXPN - Internet Exchange Point of Nigeria Lagos Nigeria N/A

APE - Auckland Peering Exchange Auckland New Zealand N/A

WIX-NZ - Wellington Internet Exchange Wellington New Zealand N/A

CAR-IX - Caribbean Internet Exchange Curaçao Netherlands Antilles N/A

NPIX - Nepal Internet Exchange Kathmandu Nepal N/A

Moz-ix - Mozambique Internet Exchange Maputo Mozambique N/A

MIXP - Mauritius Internet Exchange Point Ebene City Mauritius N/A

MIX-MA - Malta Internet Exchange Msida Malta N/A

MyIX - Malaysia Internet Exchange Selangor Malaysia 1

LU-CIX - Luxembourg Commercial Internet exchange Point Luxembourg Luxembourg N/A

LIX - The Luxembourg Internet eXchange Luxembourg Luxembourg 1

KIXP - The Kenya Internet Exchange Point Nairobi Kenya N/A

JPNAP-OSA - Japan Network Access Point Osaka Osaka Japan N/A

JPNAP-TYO1 - Japan Network Access Point Tokyo I Tokyo Japan N/A

JPNAP-TYO2 - Japan Network Access Point Tokyo II Tokyo Japan N/A

EQIX-TYO - Equinix Exchange Tokyo Tokyo Japan 1

JPIX - Japan Internet Exchange Tokyo Japan 1

TIX-IT - Tuscany Internet eXchange Florence Italy N/A

NaMeX - Nautilus Mediterranean eXchange Rome Italy N/A

TOP-IX - TOrino Piemonte Internet eXchange Torino Italy N/A

MIX-IT - Milan Internet eXchange Milan Italy 1

MINAP - Milan Neutral Access Point Milan Italy 5

IIX-IL - Israeli Internet eXchange Tel Aviv Israel N/A

INEX - Internet Neutral Exchange Dublin Ireland 2

IIX-ID - Indonesia Internet Exchange Jakarta Indonesia N/A

OD-IX - Omadata Internet Exchange Surabaya Indonesia 1

RIX - Reykjavik Internet Exchange Reykjavik Iceland N/A

BIX - Budapest Internet Exchange Budapest Hungary 2

GR-IX - Greek Internet Exchange Athens Greece N/A

ECIX-DUS - European Commercial Internet Exchange Düsseldorf Germany 1

WORK-IX - Hamburg Germany 1

INXS - INXS MUC Munich Germany 1

ECIX-HAM - European Commercial Internet Exchange Hamburg Hamburg Germany 2

BCIX - Berlin Commercial Internet Exchange Berlin Germany 3

N-IX - Nuremberg Internet Exchange Nuremberg Germany 3

ECIX-BER - European Commercial Internet Exchange Berlin Germany 4

OCIX - OpenCarrier Düsseldorf Düsseldorf Germany 4

DE-CIX - Deutscher Commercial Internet Exchange Frankfurt Germany 9

KleyReX - Kleyer-Rebstöcker-EXchange Frankfurt Germany 14

Lyonix - Lyon Internet Exchange Lyon France N/A

EQIX-PAR - Equinix Exchange Paris Paris France 2

SFINX - Service for French INternet eXchange Paris France 2

POUIX - Paris Operators for Universal Internet eXchange Paris France 3

France-IX - France IX Services Paris France 7

FICIX - Finnish Communication and Internet Exchange Helsinki Finland N/A

TREX - Tampere Region Exchange Tampere Finland N/A

TIX-ES - Tallinn Internet eXchange Tallinn Estonia N/A

CAIX - Cairo Internet Exchange Cairo Egypt N/A

DIX - Danish Internet Exchange Copenhagen Denmark 1

NIX-CZ - Neutral Internet eXchange Prague Czech Republic 3

CIX - Croatian Internet eXchange Zagreb Croatia N/A

KINIX - Kinshasa Internet Exchange Point Kinshasa Congo N/A

NAP-CO - Network Access Point Colombia Bogota Colombia N/A

HKIX - Hong Kong Internet eXchange Hong Kong China N/A

EQIX-HKG - Equinix Exchange Hong Kong Hong Kong China 1

NAP-CL - Network Access Point Chile Santiago Chile N/A

OttIX - Ottawa Internet eXchange Ottawa Canada N/A

PIX-MON - PEER 1 Internet Exchange Montreal Montreal Canada 1

PIX-TOR - PEER 1 Internet Exchange Toronto Toronto Canada 1

TORIX - The Toronto Internet Exchange Toronto Canada 1

PIX-VAN - PEER 1 Internet Exchange Vancouver Vancouver Canada 2

HTN-CIX - Cambodia Internet Exchange Phnom Penh Cambodia N/A

FIX - Ponto Federal de Interconexão de Redes Brasília Brazil N/A

BNIX - Belgian National Internet eXchange Brussels Belgium 2

FreeBIX - Free Belgian Internet Exchange Brussels Belgium 3

BDIX - Bangladesh Internet Exchange Dhaka Bangladesh N/A

VIX - Vienna Internet eXchange Vienna Austria 1

ACT-IX - ACT Internet Exchange Canberra Australia N/A

AUSIX - GlobalCenter AUSIX Melbourne Australia N/A

WAIX - Western Australian Internet Exchange Perth Australia N/A

EQIX-SYD - Equinix Exchange Sydney Sydney Australia 1

NAP-Cabase - Network Access Point Cabase Buenos Aires Argentina N/A

ANG-IXP - Angola Internet Exchange Point Luanda Angola N/A

Total: 157 internet exchange points in 113 different areas.

An internet exchange, often called IX or IXP for short, is a switching platform where internet carriers can exchange peering traffic with each other. Exchanging peering traffic over an internet exchange point rather than via direct peering, enables a carrier to interconnect with multiple networks via one uplink and thereby reduce the cost and complication of peering, enabling them to take on more peering relationships and perhaps even peer with other carriers even though the traffic volume is low.

Most of the large internet exchange points are commercial exchanges, where the participating carriers pay a fee to the IXP operator - typically depending on the port speed they are subscribing to. There are also "open" internet exchange points though, that do not require any payment or only require a symbolic fee.

Small internet exchanges are typically just one switch located in one data center, where the members can then connect to directly or through a meet me room, while the larger ixps consist of multiple switches located in different data centers that are connected to each other via direct fiber routes. This also gives members the benefit of being able to peer with members in other physical locations, without having to establish their own fiber connections between each other.

A lot of the internet exchanges publish member lists, location lists and statistics of their traffic volume on their website, enabling potential members to evaluate the Return On Investment that connecting to the IXP might generate.

For more information about internet exchanges, please see Wikipedia, PeeringDB, and DrPeering.

This map shows the biggest U.S. Internet Exchanges. www.infocellar.com/networks/internet/nap-ixp.com

If your web browser is getting data from a server in Phoenix at the Kim Kommando Show, the data packets, each holding about 1000 bytes, will probably go to PHX-IX (see the brown IX table above) in Phoenix, get on the fiber-optic trunk at Los Angeles and be routed through Dallas (see the blue line above), be routed to Atlanta, and make their way to our local ISP. (You can use tracert in Windows or traceroute in Linux to see the actual routers.) But if a bulldozer has cut a trunk on that route, the packets will be routed some other way.

scinstr@gridubu:~$ traceroute www.kommando.com

traceroute to www.kommando.com (78.40.55.50), 30 hops max, 60 byte packets

1 10.104.192.1 (10.104.192.1) 13.000 ms 17.020 ms 17.766 ms

2 acr05sghlga-gbe-0-1-0-39.sghl.ga.charter.com (96.34.78.6) 18.639 ms 21.441 ms 22.289 ms

3 static-216-186-164-89.knology.net (216.186.164.89) 29.055 ms 29.836 ms 30.619 ms

4 dynamic-75-76-40-29.knology.net (75.76.40.29) 23.043 ms 23.857 ms 24.649 ms

5 user-24-96-110-161.knology.net (24.96.110.161) 91.851 ms 93.447 ms 92.609 ms

6 ge2-12.br01.atl01.pccwbtn.net (63.216.31.145) 38.887 ms 18.596 ms 19.155 ms

7 linx.nerim.net (195.66.224.216) 118.982 ms 122.427 ms 123.223 ms

8 gi1-18-nb-courbevoie-2.nerim.net (194.79.131.125) 128.350 ms 127.465 ms 131.396 ms

9 te2-1-95-nb-stdenis-1.nerim.net (194.79.131.85) 136.823 ms 135.947 ms 137.652 ms

10 vl800-swr401-hst-stdenis.nerim.net (194.79.128.245) 142.924 ms 140.947 ms 142.039 ms

11 hn5550.diablotin.net (78.40.55.50) 138.768 ms 144.190 ms 144.976 ms

We see that hops 1-5 are Knology. There is an Atlanta router at hop 6. But hops 8-12 are all French names, and the last hop, at Diablotin.net, seems to be in Paris. An Internet search for pccwbtn.net shows it to be PCCW Global. We start thinking that the Kim Komando Show web site might be in Paris. Look at the hop times. There is a big jump from hop 5, 92 milliseconds, to hop 7, 123 milliseconds. Maybe that is where the packets are going under the Atlantic!

scinstr@gridubu:~$ traceroute www.phoenixchamber.com

traceroute to www.phoenixchamber.com (74.206.125.5), 30 hops max, 60 byte packets

1 10.104.192.1 (10.104.192.1) 615.264 ms 615.711 ms 625.483 ms

2 acr05sghlga-gbe-0-1-0-39.sghl.ga.charter.com (96.34.78.6) 627.031 ms 628.022 ms 629.041 ms

3 static-216-186-164-89.knology.net (216.186.164.89) 630.028 ms 631.009 ms 667.627 ms

4 * * *

5 user-24-96-110-161.knology.net (24.96.110.161) 679.463 ms 677.761 ms *

6 dynamic-76-73-147-237.knology.net (76.73.147.237) 668.176 ms 659.676 ms 731.837 ms

7 te-9-3.car1.Atlanta4.Level3.net (4.53.233.25) 742.821 ms 744.551 ms *

8 * vlan52.ebr2.Atlanta2.Level3.net (4.69.150.126) 732.639 ms *

9 * * *

10 * * *

11 * * *

12 ae-61-61.ebr1.Dallas1.Level3.net (4.69.151.126) 540.576 ms

ae-71-71.ebr1.Dallas1.Level3.net (4.69.151.138) 526.314 ms

ae-61-61.ebr1.Dallas1.Level3.net (4.69.151.126) 519.235 ms

13 ae-1-8.bar1.Phoenix1.Level3.net (4.69.133.29) 569.317 ms 567.520 ms 556.471 ms

14 ae-5-5.car1.Phoenix1.Level3.net (4.69.148.117) 664.703 ms 651.511 ms 641.561 ms

15 IO-CAPITAL.car2.Phoenix1.Level3.net (4.53.106.90) 648.839 ms 705.259 ms 729.774 ms

16 vl-100.core1.phx1.ip.io.com (74.206.111.197) 730.624 ms 722.723 ms 737.451 ms

17 te-1-1-core1.scd1.ip.io.com (74.206.111.89) 795.210 ms 796.245 ms 793.985 ms

18 po-2-edge1.scd1.ip.io.com (74.206.111.70) 697.241 ms 642.859 ms 642.007 ms

19 74.206.110.28 (74.206.110.28) 645.663 ms 565.529 ms 565.396 ms

20 * * *

21 * * *

22 * * *

This traceroute to the Phoenix Chamber of Commerce, likely to be hosted in Phoenix (so that a Phoenix web-hosting company can't say, Why didn't you use our service, we are right in Phoenix with you, and we are a Chamber member?), goes through Atlanta and Dallas and then right to Phoenix. At hops 17 and 18, searching Yahoo for scd1 is something about data warehousing, which seems reasonable.

This diagram is from a report on the Internet, Wireless_Incident_Response_Augusta_ISSA_05032011_PDF.pdf, from Accuvant, which displayed no copyright.

The ISSA is the Information Systems Security Association and has an Augusta chapter. The diagram indicates how hackers can attack a wireless network. Meanwhile, hackers are attacking the firewalls from Ukraine and Peoples' Republic of China. July 2012 TV network news said 100,000 Chinese experts are employed by the government to get into the networks in other countries. Software and computer hardware, such as the many motherboards, made in China are suspected of having back doors. National security networks in the U.S. have to be built without made-in-China components. This is all evidence that employment opportunities exist in network security. Young people who want to work in this field need experience in networking to see if they like it, education through technical college or university, and advancement toward Network+ and Security+ certifications. Experience with Linux is mandatory. Check out Wireshark, BackTrack, and Security Onion. BackTrack and Security Onion are built into Ubuntu Linux 10.04 and are complete operating system installs for free. If you do this, you will probably be installing for dual boot with Windows; be careful to not wipe out your Windows.

Non-copyrighted stories from Computerworld about data centers

Huge customer effort keeps flooded NYC data center running

How do you get hundreds of gallons of fuel up 17 stories? Your customers carry it

By Patrick Thibodeau

November 1, 2012 06:43 AM ET

Computerworld - When seawater flooded streets in lower Manhattan Monday night, it filled the bottom floors of the basement of an office building at 75 Broad St. and its lobby up to four feet. This was not a good development for the data center operated by Peer1 Hosting.

Rooftop generator at 75 Broad St. NYC.

Customers and others helped lug fuel up 17 floors to keep the rooftop generator at 75 Broad St. in New York City running and a data center in the building humming. (Photo: Squarespace)

Peer1's data center had smoothly switched over to generator power, but when Con Edison cut electric power to lower Manhattan, the rooftop generator couldn't access the 20,000 gallon fuel tank in the flooded basement. Its pumping system was disabled by storm waters from Hurricane Sandy.

The rooftop generator was using as much as 40 gallons of fuel an hour to run the nearly 13,000-square-foot data center, and the "day tank" had limited capacity. Time was running out.

Early Tuesday, Peer1's Broad Street customers were warned that fuel supplies would soon be exhausted. Among those who got the email was Anthony Casalena, the founder and CEO of Squarespace, a Web publishing and content management firm. He was in his powerless apartment in downtown Manhattan.

Casalena started Squarespace in 2003 with $30,000 borrowed from his father. Today, his company has 102 employees and hundreds of thousands of customers. As a private company, it doesn't give out exact numbers.

Casalena has been using the Peer1 data center since his business started. He was familiar with its operation and may now be its largest customer. He left his downtown apartment to see for himself what was happening, to offer help, and hopefully, he said, to delay any shut down "until the last possible second." His customers were warned and he continued to report on the status of the data center.

Casalena said it quickly became clear that the data center operations team didn't have a good way to measure the fuel supplies in the rooftop day tank. "They don't really know how much fuel was left," he said.

Near the time the data center was supposed to shut down, Casalena said the data center team determined that there was a half tank left. That triggered a new plan.

Casalena said there were some 55-gallon oil drums onsite. The data center manager suggested trying to get fuel supplies to the roof. They made arrangements with one of the fuel trucks on the street for supply. Fuel trucks have become a common sight since the power outage.

Casalena was among those helping to carry fuel to the roof.

The generator-fueling effort got organized. More people begin showing up from Squarespace and another large user, Fog Creek Software. They included software developers, systems engineers, sales representatives, executives and support personnel.

A network of people, stationed at every staircase landing, formed a bucket brigade passing fuel from one person to the next. Several dozen may have been involved over multiple shifts, including some day laborers who were also hired to help.

"Over the next night into the morning, we were able to successfully continue to coordinate fuel trucks coming and manually move hundreds of gallons of diesel to the roof and keep the thing online," Casalena said.

There were moments when they felt they weren't moving fast enough. At one point, a generator light came on showing low fuel.

"It's kind of just very dramatic," said Casalena. But they managed thanks to a "huge group effort," he said.

"It looks like we have operationalized this to the point where we can make it work -- I can't honestly believe it," said Casalena, "It somehow worked."

They now have enough fuel on the roof to stagger deliveries.

"Morale was super high," said Joel Spolsky, the CEO and co-founder of Fog Creek Software, who runs a number of businesses out of the Peer1 data center. "People were just absolutely itching to get there and help and do something and make some kind of contribution."

"It was hard, but people were kind of excited to have a mission to work on," said Spolsky, who is also co-founder of StackExchange, a major site for developers. That operation was moved to a data center in Oregon, but Spolsky has other work in the Peer1 center that can't be easily moved.

Spolsky, who tweeted about the efforts, said he even had offers from customers and people on Twitter to help out. "I think people feel kind of helpless sitting at home watching the news, and they were so excited to be able to do something even if it was hard to make a substantial contribution," he said.

The efforts continued to restore the basement pumping system. Water kept finding its way into the basement, but Casalena said that as of early Wednesday evening, it appeared the basement flooding situation was getting under control.

Both Casalena and Spolsky praised the Peer1 New York operations staff for their efforts. Peer1, which has 19 data centers worldwide, has extra help arriving tonight, driving up from a Virginia because of problems getting flights to New York.

Along with the efforts to get the building's pumping system in operation, Robert Miggins, senior vice president of business development at Peer1, said the company is working on alternative methods to pump fuel up to the roof.

Miggins said the data center remains online, in part, because the company didn't keep the situation it faced in New York a secret. "We choose to tell customers all the details and that is what led them to show up and enable them to lend a hand," he said.

Without that notice, "we wouldn't have had the manpower there to actually bring the fuel up in time," Miggins said. "There's a lot of good will, and there's a lot of hard work and there's a few lucky bounces for good measure," he said.

No one is going to feel comfortable until power is fully restored. Spolsky said his staff has been marking the company's physical systems to speed reassembly at another data center if the need arises.

If there's any takeaway from this experience, it may be this: Casalena said he's glad he went to the data center when he did to help and to take an active role. Otherwise, "I don't know how this would have played out," he said.

Patrick Thibodeau

Computerworld

Storm forces Internet hubs to run on generator power

Manhattan power outage amid Hurricane Sandy hits critical communication centers, including Google-owned building

By Patrick Thibodeau

October 30, 2012 04:46 PM ET

Computerworld - Two monolithic buildings in lower Manhattan that serve as major network hubs for the U.S. are operating on generator power, thanks to Hurricane Sandy.

The buildings, known as carrier hotels, are a 2.9 million square foot structure at 111 8th Ave., and a 1.8 million square foot facility at 60 Hudson St.

Telecom companies use carrier hotels to interconnect networks to allow data sharing and users of one network to connect with those of another. Thus, the two buildings are critical to the nation's infrastructure.

In 2002, Richard Clarke, then special advisor to the president for cyberspace security, described their importance in a speech.

Clark said, "Transatlantic fiber lands at about 10 different places in Massachusetts, Rhode Island, Long Island and New Jersey that, after having landed, it all goes to one of two facilities -- 60 Hudson St. or 111 8th Ave in lower Manhattan. If that's true, that would seem to be a problem."

What is true today is that hundreds of domestic and international network connections are made at these two buildings. The close proximity to network resources has turned the buildings into major data center locations.

"It really is the heartbeat of the East Coast," said Ron Sterbenz, vice president of marketing at Telx, citing all of the communication activity that takes place in the two buildings.

Telx is a major data center provider that has co-location facilities at 60 Hudson as well as 111 8th Ave.

Michael Levy, an analyst at Datacenters Tier1 Research, a division of 451 Research, said that "111 8th Ave. and 60 Hudson are two of the most carrier dense buildings in the world."

"There is a high probability that your Internet traffic, every time you go on a Web site passes through 111 8th Ave. at some point," said Levy.

Google bought 111 8th Ave. two years ago, and has offices in it.

When Con Edison shut off power in lower Manhattan late Monday to protect equipment from storm flooding, it triggered generator backups at 60 Hudson and 111 8th Ave. The generator is powering the facilities right now.

Telx operates some 490,000 square-feet of data center space in the New Jersey, New York market. The company is running its systems at 111 8th Ave. building through the building's shared resource pool of 90,000 gallons of diesel.

Sterbenz said it had enough fuel to run until some time tomorrow, and that he does not expect any problems.

When asked what Google would be doing to ensure operations at 111 8th Ave., a spokeswoman said that the company "won't be providing any comment" about backup plans.

Sterbenz said there's enough fuel at 90 Hudson to maintain operations and fuel reserves were already on Manhattan island so transport through dangerous roads, bridges and tunnels isn't necessary.

There is no estimate from Con Edison on when power will be restored, said Sterbenz.

Internap, which also runs a data center at 111 8th Ave., said its operation there that is running on generator power. Company officials said Internap faces a more serious situation at its 75 Broad Street data center facility due to a flooded basement that knocked out fuel pumps.

In a blog post Tuesday afternoon, Steve Orchard, Internaps' senior vice president of development and operations, said that at the storm's peak there was three feet of water in the lobby at the Broad Street site, making the basement inaccessible.

Internap is implementing a workaround to get fuel to the generators, he added.

Regarding the data center at 111 8th Ave. Orchard said that the company "expects fuel delivery to the site will be possible prior to depleting on-site reserves."

Patrick Thibodeau

Undersea Fiber-Optic Cable...

is brought onshore at isolated beaches, away from where ships might drag their anchors. Here is hundreds or thousands of miles of cable in a storage drum on-ship.

This Alcatel sip is playing out the undersea cable to a landing boat using floats. The ship has to be offshore enough to be in deep water. This is at the Pacific island of Tonga. A crew, and some locals in shorts and T shirts, pull on a rope that will pull in the thick fiber-optic cable.

This is a fiber cable landing onto the east coast of Africa. The cable is trenched twelve feet deep to a small building that has the connection point to the landward fiber cable. The building has a DC power supply that puts 4000 V at maybe 30 amps onto the power conductors in the cable. This power flows the distance of the cable to power up the optical repeaters, spaced about twenty miles apart. Each repeater drops about 40 V.

The glass optical fiber has a core only .008 mm diameter. It is clad and buffered up to .25 mm. Several fibers are within the steel-armored, insulated, undersea cable.