Hurricane Sandy has barreled through New Jersey and into Pennsylvania, leaving an impressive wake of destruction for the press to report on – even if they keep showing the same @#%&ing clips over and over. But with two of the internet’s main hubs (NY/NJ and Ashburn) and some of its most trafficked routes in Sandy’s path, what about the communications grid? No really, I’m asking you folks.
Ramblings doesn’t have any of those crazy idiots that pose for cameras in front of the waves with microphones hoping a flying trash can lid doesn’t behead them. But we do have readers from around the industry who know things, lots of things. Some are actually there (with or without power), some are in NOCs further away watching over places at risk, etc.
So you tell me. What’s up, what’s down, and what’s running on a dwindling supply of diesel? Fiber cuts, flooded assets, or places you’d like an update on? In need of resources, or have some to share? Official, unofficial, or grapevine, or via email or in a comment below, feel free to use this site to spread the word.
Well, there have obviously been some online casualties, the earliest coming when Con Edison shut off the power in lower Manhattan in advance of the storm surge. Some websites dependent on data centers there, like Huffington Post and Gawker apparently, already took a hit. Data Center Knowledge says that those were in the Datagram facility at 33 Whitehall.
If you haven't already, please take our Reader Survey! Just 3 questions to help us better understand who is reading Telecom Ramblings so we can serve you better!
Categories: Datacenter · Fiber Networks · Weather
Join Our LinkedIn Group
Data Center Knowledge also reports that 75 Broad has water in the basement with damage to diesel backups, and some outages at 111 8th. http://www.datacenterknowledge.com/archives/2012/10/30/major-flooding-nyc-data-centers/
Atlantic Metro has tweeted widespread outages https://twitter.com/atlanticmetro/status/263237765951537153 from flooding.
Check out:
http://www.webhostingtalk.com/showthread.php?t=1205042
On behalf of Fred Cannone, Director of Marketing and Sales for TELEHOUSE America, please see the statement recently issued:
Since Friday, 10/26, TELEHOUSE New York initiated plans to sustain operational readiness should Hurricane Sandy strike the greater New York City area.
Since Sunday, 10/28, all three TELEHOUSE New York City facilities (25B, Chelsea and Teleport) have remained fully fueled and staffed, with extra Security and Engineering personnel.
Last night, 10/29, between 7PM and 8PM, all TELEHOUSE New York City facilities proactively engaged backup diesel power (off-grid) in anticipation of lower Manhattan’s utility power shutdown. Since that time, these facilities have remained operational without any critical system (power, cooling) outages. These facilities will remain fully staffed and off-grid (independent power generation mode), with periodic refueling until utility power is restored and stable.
During this period, both Manhattan sites (25B, Chelsea) were operationally unaffected by local flooding. Building engineers and equipment, at both Manhattan facilities, are engaged in mitigating this threat. The Teleport facility, on Staten Island, had no flood threat during this time.
All TELEHOUSE NYIIX extensions have remained operational without any outages. Apparently some 111 8th Ave. NYIIX Customers did experience their own power problems there.
Please email our NYIIX support team at, iieg@telehouse.com, for the latest status or questions.
TELEHOUSE has received reports of sporadic Carrier/ISP circuit outages, which have already been restored or corrective actions underway.
If you are experiencing any problems, please open a trouble ticket with TELEHOUSE America’s Customer Assistance Center at 718 355 2525, or via email, cac@telehouse.com.
Equinix NY4 is running on generators. Supposedly they have fuel for 2.5 days, but they can’t get new fuel until the flooding goes down in the surrounding area 🙁 Connectivity seems to be unaffected.
On this page is a map that overlays Hurricane Sandy with the Zayo network: http://www.zayo.com/map/hurricane-sandy-network-map-overlay
Zayo issued today a Letter to Customers, which you can find on the Zayo website here: http://www.zayo.com/map/hurricane-sandy-network-map-overlay.
The web page includes a status update of what is out, Zayo’s restoration efforts and generator run-time, as well as a toll-free number (866-236-2824) to call with customer questions or concerns.
Renesys has some great graphics on outages from Sandy: http://www.renesys.com/blog/2012/10/hurricane-sandy-initial-impact.shtml
And here’s some from the ILECs and MSOs from Reuters: http://www.reuters.com/article/2012/10/30/storm-sandy-telecommunications-idUSL1E8LU5EX20121030?feedType=RSS&feedName=cyclicalConsumerGoodsSector&rpc=43 and some for the wireless guys from the WSJ: http://online.wsj.com/article/SB10001424052970203335504578088722718998026.html?ru=yahoo&mod=yahoo_hs
As expected, Verizon has a lot of work piling up: http://finance.yahoo.com/news/verizon-teams-launch-restoral-effort-211000482.html
CoreSite update: http://newswire.telecomramblings.com/2012/10/coresite-data-centers-remain-operational-in-wake-of-hurricane-sandy/
Equinix update: http://blog.equinix.com/2012/10/superstorm-sandy-and-equinix-data-centers/
Both have NYC area facilities on generator power and expect to remain up.
OCG has had no outges or troubles on any part of its network. All generators/backup equipment is working 100%.
Our POP at 32 A of A if working fine due to the building being well prepared for the storm with fuel and personel.
Our crews were assisting other carriers with pumping and power generation. 75 Broad’s basment is a mess and it will take some time to dry the salt water out of the gen’s.
A number of IP carriers were 100% down for most of the day as OCG received calls to provide service to bridge the outage.
Still no power in over half of Manhattan. Downtown and western Manhattan is completely dark. It will take at least a week for the power company to get the city at least partially running.
As our OPS personel drive out and inspect our routes the scope of destruction to property homes and vehicles is stagering.
I was in Far Rockaway with a few of our military vehicles helping the rescue effort on the 5+ foot flooded streets.
The carriers will be able to recover quickly but many of the customers will be without power for a week or two.
While there were reports of ‘intermittent issues’ with AC-2, Level3 has now said there was merely a fiber cut on a diverse backhaul route on Long Island and that their transatlantic cables are fine.
At 60 Hudson some gen sets failed to fire despite being tested the previous day, this resulted in some outage time.
Hibernia Atlantic’s Executive Team has shared the following statement:
Hibernia Atlantic’s network has been operational throughout the duration of Hurricane Sandy, and we remain available to help assist others in this time of need. Our subsea cables’ landing stations are located in Lynn, MA and Halifax, Nova Scotia, areas that fortunately were not as impacted by Sandy’s reach. Hibernia´s network and landing stations are diverse from all the other trans Atlantic cable systems in that our cable landing stations are not based in the NYC area. As a result, our cables have sustained no impact from the storm, illustrating the value of physical diversity.
We want to acknowledge the ongoing efforts by our industry as a whole, including the many partners, vendors and networks who are working together to keep communications available. Hibernia has been able to assist a number of our customers and partners already that were impacted by the storm. For example, several routes on our partner’s network went down, but we successfully rerouted traffic before the trading floor opened this morning.
Our global Network Operating Centers (NOCs) remain on ‘high alert’, with our engineers well-staffed and monitoring all of our network Points of Presence (PoPs), particularly those in the New York City area most impacted by Sandy. These PoPs are currently operating on generators due to the power outage, but we are, and continue to be, up and running.
32AofA and 60 Hudson back on commercial power as of about 4am this morning.
Ccogent datacenter in lower manhattan was hit hard but worst of all it was an electricial problem that keeps taking entire site down most of two weeks after storm hit. They put all their customer and cooling units on one phase! Worst is their “expert” site manager said stupid stuff like “you should get your equipment out of here” while many colo customers are waiting for them to fix the problem! crazy horrible datacenter and the poorest customer service ever encountered.