Andy ITGuy – Information Security Blog

Yesterday I attended the Fishnet Enterprise Security Solutions Seminar here in Atlanta. As the name sounds it’s a dog and pony show for Fishnet and some of the companies that they work with to show their wares. The difference here is that it wasn’t all vendor pitches. Actually the vendor presentations were informative as well as sales/marketing oriented. What I really liked is that in addition to the vendor talks there was also some “real” talks. Talks that were not about a vendors product or service.

One of the talks, and the one that was the most informative, was given by fellow Security Catalyst Member and friend, Martin Fisher. Martin manages the CSIRT team for Delta Airlines and he spoke to us about Incident Response programs. Martin is someone that you need to get to know. He brings a Military and Law Enforcement background into his position and know how to lead and get others to respond. He did tell me that he has been asked to deliver his talk to a couple of other groups in the Atlanta area so if you hear that he is speaking I’d plan on attending.

I grew up in the country for the most part. I did live in a neighborhood for a while but we soon bought a small farm and moved “off the beaten path”. I spent my days exploring the woods, hunting, fishing, sport shooting, trapping, frog gigging, building underground forts and tree houses. My parents both worked full time and sometimes when both of them had to work late there wouldn’t be anything to eat. It wasn’t unusual for me to come home from school, grab the shotgun and head off into the woods looking for some squirrel to shoot for dinner.

I learned a lot about how to get along by myself during this time. I became a crack shot with a rifle and could make a meal out of most anything that I could find in the woods. One thing that baffled me though was hitting a moving target with a shotgun. I could do it easily with a rifle but for some reason I couldn’t hit the side of a moving barn with a shotgun. So needless to say the few times I did go skeet shooting I might as well have picked up each of the disc and thrown them as far as I could and watched them break. Then I could have picked up my shotgun and fired off 100 round or so into the air just for fun.

The other day IBM invited me to a event that they were hosting at Barnsley Gardens just north of Atlanta. It included breakfast, a few talks on some of their security and systems managements offerings, lunch and an afternoon of skeet shooting with the guys from Orvis. The afternoon started off with me being quiet surprised. The guys from Orvis gave us a few pointers and I grabbed a shotgun and walked up to the shooting cage. When I was ready I yelled “pull” and out launched a clay disc. It went up and I followed it with my barrel. It peaked and started it’s downward descent and I pulled the trigger and watched the disc shatter into lots of little pieces. I hit 3 of the 4 practice discs. Then the real fun began. They broke us up into groups of 4 and we split up and went to different shooting stations. Since I was an experienced shooter they made me “team captain” and I was supposed to assist any of my team mates that needed help with getting their guns loaded or anything else. Since I was team captain and had done pretty good in practice the guys told me to go first and “show them how it was done”.

Our guide had moved to the “Trappers Nest” to load and launch the skeet. He first launched a couple so we could see how they would fly at this particular station. Instead of going mostly up and down it flew across the pond in front of us much like a bird would fly. Now that I had the flight path in mind I loaded my gun, got into firing position and yelled “pull”. The disc floated across the pond, I took aim, fired………….. and watched the disc continue on it’s path unscathed. Then came number 2………… same thing. Out of 10 disc I managed to hit 2. Each of the 10 stations presented us with a different shooting challenge. Some I did pretty well at and some I was lucky to hit one or two. After all 100 rounds had been fired I  had hit about 40 targets. Not what I would have wanted but I was satisfied, plus I had a blast doing it.

After it was over and my mind started to settle back into “security” mode it occurred to me that the different challenges and scenarios that I faced shooting skeet were similar to challenges and scenarios that we face in information security (OK, I’m stretching it a bit but stick with me and hopefully I’ll be able to pull this off). What’s more is that it hit me that the way some of the other shooters handled each station was similar to home some information security professionals handle the challenges that they face.

Some guys were quick to fire and others were patient, some were too patient and waited too long to act. Some were out in front of the skeet, some were behind it, other were either above or below it. Some guys emptied both barrels at one target. Some fired both rounds quickly and others were slow and steady. There were those who seemed to be shooting from the hip and those who took careful aim.

When we are doing our job we are faced with all manner of situations that require us to act, react, make decisions quickly and with little information or sometimes lots of information. How we handle these situations will determine how successful we are at our task. There are times when we have to fire quickly or we will miss our opportunity to stop an attack before it gets out of hand. There are times when it is prudent to be patient and wait and see what it happening before we pull the trigger.

The key to being successful in these scenarios requires at least one of four qualities. Skill, knowledge, instinct, luck. The “rock star” has all four of these on his/her side and the rest of us make the best our of the ones that we have. One of the stations in the skeet shoot was supposed to mimic rabbits running across a field. The Trapper would launch 1 or 2 disc in such a way that they rolled across the ground and bounced as they quickly rolled in front of you. This was the station where I saw the most frustration from some of the other shooters. Since the pattern was more erratic than most of the other stations due to the bouncing and such you really had to just shoot blindly. This is where people would unload both barrels so quickly that it often sounded like one shot. Many were extremely frustrated because it looked like it would be so easy. This is also the type of scenario that can get us in trouble at work. We often see things happening so quickly that we don’t have time to really map out a plan. We just pull both triggers and hope we hit something. This is why it’s so important to have a well designed incident response policy that you have and continue to practice regularly.

Skill, Knowledge, Instinct and Luck all play a part in  skeet shooting as well as in Information Security and we would do well to develop those that we are weakest in when it comes to information security. When you miss a target in skeet shooting the chances of it doing anything more than landing harmlessly in a pile is very low. When you miss a target in information security you risk much, much more.

I’ve mentioned before that I’m not a forensics guy by any means. I’ve never done any "real" forensics, at least not anything beyond simple looking for fairly obvious evidence of a breach or problem. I enjoy reading about digital forensics because it fascinates me. The way that data can be extracted from media after it has been deleted, hidden, and even when the disk has been formatted. Not to mention how someone who is trained can look at the system and determine what happened, how it happened, who did it, how they gained access to the system, etc….

Last week I read this post by Harlan Carvey here. This quote that he made got me to thinking:

My personal thought on this is that ideally what an organization would want to do is develop an in-house capability for tier 1 response…trained folks whose job it is to respond to, triage, and diagnose a technical IT incident. By "trained", I mean in the basics, such as NSM, incident response, troubleshooting, etc…enough to be able to triage and accurately diagnose level 1 and 2 incidents, as well as preserve data until outside professionals can respond to level 3 or 4 incidents.

What is it that companies really need? What are the basics to ensure that triage is done in a manner that doesn’t compromise "the crime scene". I decided to post that question to my friends in the Security Catalysts Community here. As I expected I have gotten some good responses.

On Thursday of this week I attended a one day event put on by ISC2 called SecureAtlanta 2008. I had forgotten what the topic was and it turned out to be Digital Forensics. It was a high level discussion that covers a lot of the basics of what DF is and why companies need to be informed and concerned about it. Not much of the content was technical but it was informative. One of the things that grabbed my attention was the topic of DF and the law. We need to keep in mind that what we are doing in incident response and forensics needs to keep in mind the possibility of going to court. Our findings may need to be presented in court to convict or defend. Therefore we need ensure that our teams are trained in the basics but also trained in how to not contaminate the crime scene.

One last thing to consider is that just as all things related to security there has to be a balance. We have to balance IR and DF with ensuring that we get (or keep) the company running. We can’t forget that our company probably relies on these systems running in order for them to make money. So if your company doesn’t have proper policies and procedures in place for this that you start the conversation with your boss. Then work with management to put in place the proper program and training get put in place.

Yesterday I drove to work which isn’t something that I typically do. I like my sanity too much (what little is left) to fight Atlanta traffic on a regular basis. I woke up late and missed the one bus that will get me to the office in a decent amount of time so I decided to work from home for a couple of hours and then drive in after rush hour was over. I had the same thought process for my commute home. Leave before rush hour and work remotely for a couple of hours. So I left early and went to my favorite coffee house and set up office for a while. I let my wife know that I was close by in case something happened and she needed me in an emergency.

Some would say that I was setting myself up for this but about an hour later my cell phone rang and it was her. “You’ve got to come home right now! Bella drank about 1/4 cup of Hydrogen Peroxide!” CLICK My phone went dead just as I was about to tell her to call Poison Control. So, I packed up quickly and hit the road. I called back to calm my wife down and to have her call Poison Control. When I arrived home my wife informed me that our youngest daughter may have also drank some of the peroxide also.

My wife was rushing around getting ready to take the girls to the doctor and getting upset with me because I wasn’t panicking. I knew that peroxide could be dangerous to a child if enough was ingested but I also knew that it would cause them to throw up soon. So I convinced her to wait a while and see what happens. I also asked my daughters about how much they had actually drunk and called Poison Control myself to talk to them. It turns out that the oldest only had a “good swallow” and that the youngest just tasted it. The oldest did throw up and Poison Control told me not to worry.

That got me to thinking about how IS/IT teams often react to emergencies at work. Do they panic and rush into a plan that hasn’t been thought out or do they take a deep breath and look at what is going on and try to learn the facts of what has happened and what their options are? If you don’t have an incident response plan I can tell you that more than likely people are reacting instead of thinking. Even if you have an IR Plan if it hasn’t been tested and the team isn’t familiar with the plan and their role in the incident they will usually just do whatever comes to mind first. Sometimes that works well and sometimes not so much. You can’t take that chance.

The Incident Response Poll closed last week and I was out of town over the weekend so I didn’t get a chance to write up the summary. Here are the results:

When it comes to Incident Response does Your Company

Have a formal and tested plan	8 (25%)
Have a plan that hasn’t been tested	2 (6%)
Has a general idea what they will do	9 (29%)
Not have a plan	12 (38%)

67% of you answered either “Has a general idea what they will do” or “Not have a plan”. That’s not very encouraging. It shows that we have not done a good job in conveying the need to management. Perhaps you don’t think that the need is that great. I live in a world filled with compliance and most regulations out there require an IR plan. That alone should be enough for you to take to management. Not to mention the sheer lack of understanding of what needs to happen to respond to a breach. If you don’t have a plan then how will you know what to do? Do you disconnect the system from the network or leave it connected? Do you power it off or leave it on? Do you have to notify the police? The FBI? A financial institution? Your Customers? Your employees? The media? If you don’t know now how do you think you will know when the time comes and you are in the heat of the moment?

A IR Plan details all of this. It tells you what to do and what not to do. It tells you who you need to notify and how to do so. It tells you how to stop breach from continuing and how to clean it up. All of these things and much more are included. Things that can make the difference in a successful incident response and one that is a dismal failure. A successful one is one that your company survives and continues on with little impact. A failure may mean that the company has to shut their doors and go out of business. It may mean that the company survives you you don’t. It may drastically alter the way your company does business. That may be good or it may be bad.

If you yourself don’t understand the need in a IR plan PLEASE, PLEASE, PLEASE!!!!!! do some research and discover the need. If you do understand the need but haven’t been able to communicate it effectively to management PLEASE, PLEASE, PLEASE!!!!!! do some research and find someone who will help you be able to do that. The Security Catalyst Community is a great place to start with that. There are people there who will be able to help you understand the need and be able to communicate it effectively.

For the rest of you that have a plan I only have a little to say. First, congrats to you and your companies for seeing the need and doing something about it. Second, please ensure that it is kept up to date. An outdated plan is almost as bad as not having a plan. Third, if it hasn’t been tested please talk to management about testing it. Even if it’s just having several people review it and ensure that it makes sense that is better than nothing.

I’ve just posted a new poll about company Incident Response Plans. This is an area that is often over looked and under planned. Many companies don’t even realize that there is a need for an IR plan and have no real idea what they would do if an incident occurred. In this day of legal and compliance issues having a plan is no longer just a good idea. The lack of one could cost your company lots more than the cost of clean up. You need to have a plan of attack for a variety of different incidents. The way you would handle a virus outbreak is different than how you would have a server compromise that exposed financial or customer data.

If you don’t know where your company stands in regards to an IR Plan don’t just take it for granted that they have one. Ask your boss and if there isn’t one inform them of the necessity and importance of one. Be prepared to either volunteer to help or be volunteered. Do your homework and you may come out smelling like a rose.

Here is the question and the possible answers to choose from. You can find the poll itself here.

When it comes to Incident Response does Your Company

It’s always good to be prepared. I try to be prepared for situations that may arise both in my professional and personal life. I look at potential threats and issues that may arise and see what I need to do to be prepared. That’s part of any good security program. Having controls in place to prevent a breach or reduce it’s impact are important in securing a network, web site or application. Doing preventive maintenance on your systems (patching, monitoring, etc) will help ensure that they are in shape to prevent holes that can be compromised.

Similarly at home I check for leaks around windows and doors. Change my HVAC filters, maintain my vehicles to ensure that they run properly. There are many, many things that need to be taken into consideration to ensure that you prevent problems and are prepared in case they occur.

On Friday I had an incident happen that I wasn’t prepared for. I had left work about 2:45 in the afternoon to ensure that I missed the bulk of the terrible Atlanta Friday afternoon traffic. I’m traveling up the interstate when I hear a loud roar coming from my Jeep. I immediately turn of my radio and start checking for smoke or flying parts in my rear view mirror. I quickly pull to the side of the road and discover that I have a flat tire. So I get out my jack and lug wrench and start removing the flat tire. I get it off and grab the spare (one of those crappy temporary tires) and the first thing I notice is that it is also flat. Oh yeah, I forgot to mention that by this time it is now raining VERY hard. So I’m getting soaked while changing one flat for another.

I go ahead and put the spare on and then walk back to the drivers side of the car to get in. I had forgotten that my windows was down so now my drivers seat is also soaked. I then call the Georgia Department of Transportation to inform them of my situation. One of the good things about living in Atlanta is that they have incident response trucks to assist stranded motorist.
After about a 45 minute wait my HERO (Highway Emergency Response Operations) arrives and puts air in my tire and I’m on my way home. Of course now I have to sit in traffic because by this time it’s about 4:30.

So, what did I learn? It’s the little things that can bite you. Not keeping an eye on the air pressure in my spare cost me time and a headache. What is there at work that is possible being overlooked that may come back to bite me in the butt? Unfortunately I don’t know what it is right off but I’m going to start looking at those little things a little closer.

Harlan asks “who decides what best practices are” in regards to Incident Response. Harlan is a forensics guy and has written an excellent book (I’ve only read 1 chapter but many others have told me how good it is) on Windows Forensics Analysis. Obviously forensics plays a part in many Incident Response scenarios. His answer to the question of who decided best practices is “It depends”. And I agree.

Dr. Anton asked how PCI can be both complex and basic security. He asked that based on the fact that my PCI poll (as scientific as it was) said that 40% of you said that PCI is complex and 40% said that it was basic security. My take on that was “It depends”.

On the Security Catalysts forums someone asked if NAC had any real value due to the fact that there are ways around it. My response again was “It depends”.

Security depends on your company, your environment, your level of risk and risk acceptance. It depends on the level of competency of your IT and security staff. The level of competency of your end user employees. What partners, contractors, visitors, etc that are allowed to connect to your network. What controls you are willing and able to put in place. What policies you have and enforce. What level of buy-in you have from management. What does your IT environment look like. Is it new, old or a mix? It is small, medium or large? Is it complex or simple? Do you have lots of different apps or only the core ones required to do business. How big is your Internet facing presence.

This list could go on and on and on and on. There are too many variables to give a concrete answer to these and other similar questions. So the real answer is that it doesn’t matter what your idea of the answer is. Your job, as a Information Security Professional is to do the best you can with what you have and plan for the worst. That is where the concept that IR belongs to everyone comes in.

Many companies have IR teams that jump into action at a moments notice. But what happens between the time a incident is discovered and the team is able to take action can make all the difference in the outcome of the teams work. The rest of the company, from end user to IT/IS needs to know what to do in the event of a incident. If they don’t then they will invariably do something wrong that will hinder the investigation and fixing of the issue.

I’ve written too much so I don’t have time to go into details here but suffice it to say that IR goes way beyond the team. It has to be dealt with at ALL levels if success in dealing with an incident is your goal.

I was talking to someone briefly the other day about the CIA triad and it got me to thinking. Most security books teach it and many security professionals will agree that it is foundational to Information Security. As you all know the 3 legs are Confidentiality, Integrity and Availability. We all work hard to ensure that our data stays confidential, that it’s integrity is maintained and that it is available to authorized users when it is needed.

What I want to talk about is Availability. What does it involve and what are we doing to ensure that data truly is available. Availability can be affected by the following (and more that I’m sure I will miss).

• Denial of Service Attacks
• Hardware failure
• Improper device configuration
• Man-in-the-middle attacks
• Corruption of data
• Removal/deletion of data (intentional and unintentional)
• Route poisoning (ARP,DNS, etc)
• Software bugs

These things affect Information Security yet are often looked at as either belonging to another group (Network, Servers, Firewall, etc) or not being a big deal. When this happens you are setting yourself up for failure.

The best way to assure the availability of information is to have a plan and to test it.

• What is your plan to prevent MitM attacks, Route poisoning, DoS attacks? Do you test your systems to ensure that these types of attacks can be fended off? Do you have a plan to mitigate them? What about an incident response plan? Has it been tested and carefully thought through?
• What about data corruption or deletion? You have backups but are they any good? When was the last time you did a test restore? What happens if your tape drive goes bad? Can you restore on a different model if necessary?
• What steps are in place to ensure that devices are configured properly? Do you have procedures to ensure that they are configured and tested? Is the configuration backed up and documented in case of hardware failure? How quickly can you get the device back up and running or replaced? Say you lose a server with all your user files. You have a spare that you can restore to quickly, but what about ensuring that the users can connect to the new device. It likely has a different IP address and name than the original box. What are you procedures for uninstalling applications and patches that cause problems?

These are the types of things that can easily be over looked if you have not done your homework. You need to do a Risk Assessment and ensure that the basics are covered. You need to then put a plan in place and test, test, test. It’s not always the most fun thing to do, but in the process you will learn a lot about yourself, your network, your coworker and your company. It might even keep you out of the unemployment line.

_uacct = “UA-1509762-1″;
urchinTracker();