harleystagner.com

I am starting a new category called the Early Call. Those who are in the IT support arena know this category all too well. For the inaugural post I present a call that I received at 3:30 AM. The thing about these early morning calls is that while annoying, we always end up learning some lessons. I hope to pass these on to any readers who may benefit or be reminded of the lessons learned.

3:30 AM Phone Rings…

A rightfully disgruntled wife hands me the phone and says it is work. Damn! Wife goes back to sleep immediately. (How does she do that?) I am not so lucky.

User: “The internet is down.”

Me thinking: “Really? The whole thing? Did you reboot the internet?”

Me really saying: “Okay, please go to start -> run -> and type ‘cmd’ Now type ‘ping www.google.com’.”

Nothing…

Me really saying: “Okay, let me try a couple of things from here (home), then I’ll be right there.”

So, I logged into Citrix and tried to nslookup (Nada), ping (Nada). I could ping the internal interface of our firewall. Hmmmm… What about the external interface. Didn’t know that IP. The previous network admin did very little documenting.

Lesson 1: When inheriting a poorly documented network, sort out your public facing (internet or otherwise) information first. Device (routers, firewalls) logins and ip addresses would be handy in this situation. They tend to be harder to just “figure out.”

I was gonna head in early today. Besides, I wouldn’t be able to ping the external interface anyway as ICMP was blocked and I didn’t know the login to the firewall. See lesson above for how to resolve that .

4:45 AM Arrive at work…

Thankfully, I have the contact info for our browsing circuit provider in my file cabinet.

Lesson 2: Unless you are fond of unnecessary trips into work, keep vendor contact information handy wherever you are. Outlook contacts, a file at home, cell phone, etc. This may seem simple, but might often be overlooked.

As it turns out, just having the contact info this time would not have helped me. As soon as I arrived I called our browsing circuit provider and had them trace the external interface (T1 DSU) of our router. To my surprise, they were successful. Next, I checked DNS from our other public circuit (DMZ). Nslookup returned successful lookups for “www.google.com”. Since, the same DNS servers are used no matter which circuit is leaving the building, I could rule out DNS as the problem.

Okay, check list so far.

• I can ping the internal interface of our firewall
• The ISP can ping the external interface of our router
• The network topology goes Internal->Internal Firewall Interface->External Firewall Interface->Internal Router Interface->External Router Interface (T1 DSU).
• I know IP’s for Internal Firewall and External Router. I am missing some crucial information.
• DNS is working from our other connection (DMZ) to the outside world and our browsing circuit

I would normally hook a laptop up directly to the internal interface of the router, bypassing the firewall to see if the the firewall was the problem child. However, without IP’s or logins to the devices (and nobody to call at this hour who might know the logins), I wouldn’t be able to perform this crucial troubleshooting step.

At this point, I had to resort to that IT support mainstay. Power-cycling. I mean, to the users, the entire Internet was down. So, not only did I have to get the users back in business and browsing happily, all those poor “.com” sites like Google were counting on me . So, I power-cycled the firewall first. That did not bring the Internet back up.

Now the grand finale!!!

I power-cycled the router to our browsing circuit. One minute later and Voila! It must have gone stupid. I have no idea why because I can’t log in to the damn thing. However…

I have rebooted the Internet and made it work again!!! One IT Professional single handedly restored browsing goodness to the office and saved Google’s stock price. Well, at least according to the users (remember, the entire Internet was down) that’s what happened.

So in review…

Early Call Lessons Learned:

• If you have little or no documentation on a newly inherited network, get your hardest to find logins, IP’s, etc. out of the way first. This is usually the public facing equipment, since internal devices can easily be audited. I could have gotten all the information on our internal IP’s etc. by logging into Citrix had I not known them.
• Keep documentation and vendor information handy wherever you are to make remote support a more viable option. Outlook contacts or cell phone contacts usually work best for this.
• In hindsight I should have checked DNS from the DMZ remotely. That would have been a viable option.

So another Early Call taken care of… Time to catch up on my rss feeds and play some solitaire…

Technorati Tags: IT Support