Tales from the Field: The Wrong Tool – part 1

contributed by the_highwayman,

Some techs are techs, and some need to find a desk job real fast.

One night at the NOC [ed. Network Operations Center], I was working with my team on routine network maintenance.  One of the items on our list was to install two PHY [ed. Physical Layer Device] boards into a router. A basic job, although we all knew this particular card was a tricky one to install: the cards were thin and the guide levers had to be turned 180 degrees and then snapped into place before you could put the board in the slot.   If you don’t know how to do it, you realize it very quickly.   But in this case, the field tech doing the install was in another location and we had him on speakerphone, and he was new on the job and apparently had not absorbed anything about this router in training.  We told him to go ahead and install the cards one at a time and to wait before installing the second, and we waited there on the phone expecting him to return to the phone for aid or at least to verify that the first went well.

After 15 or 20 minutes of hearing the field tech curse, strain, grunt and generally get pissed off, suddenly there was just silence for 10 minutes.  Worried, we tried desperately to get the field tech back on the phone, but to no avail.   We then heard this loud, piercing, and utterly chilling noise like metal being hit with metal along with lots of other sounds you should never hear in a datacenter.  Ever.  And all the time he’s still cursing at the router, but now it is triumphant.  Back at the NOC, we were just sitting there speechless as error messages start scrolling and we lost contact with first one router and then the other.  Something had gone very wrong…

[To be continued]

Tales from the Field will hopefully be a recurring segment on this blog, offering a voice to the folks who install the gear, flip the switches, and do all the real work holding the internet together that we don’t hear about in the press. Do you have a story from the field to share? Contact me!

If you haven't already, please take our Reader Survey! Just 3 questions to help us better understand who is reading Telecom Ramblings so we can serve you better!

Categories: Datacenter · Guest Posts · Tech Stories