| Q&A |
|
|
In the Wake of KatrinaBrian D. Voss didn’t have time to pull out the disaster plan following Hurricane Katrina, but it didn’t matter, because it wouldn’t have prepared him for what was to come.Lee Copeland and Tom Halligan Brian D. Voss is CIO at Louisiana State University, Baton Rouge. He was appointed in 2005 and serves as a member of the Chancellor’s Executive Cabinet. Before that, Voss was associate vice president for IT at Indiana University, where he also served as COO of the Pervasive Technology Lab. With more than 20 years of leadership experience in information technology, Voss is a frequent speaker and writer on IT issues. EdTech: Brian, at Louisiana State University in the aftermath of Hurricane Katrina, you said you didn’t follow the [disaster recovery] book — you rewrote it. EdTech: From what I read, yours was almost a unique case. Voss: I’ve always thought about disaster recovery and business - continuity planning, it all had to do with what if your data center were struck down. So that’s sort of a traditional view, where an isolated incident takes out your facility in the middle of the campus, and the rest of the institution is standing by waiting to go forward. And what we saw with Katrina was what happens when the entire area — your entire city — is taken out. So we were able to observe that from the New Orleans schools. The third aspect is what happens when you're on the edge of someone's disaster and suddenly instead of doing what you were doing, you have to be able to cope and respond to circumstances that are beyond what you ever thought you would have to do. EdTech: After Katrina hit and the levees broke, you and the administrators started to realize the extent of what was happening. When the refugees started coming your way, did you take on the role of a designated relief center, or was that decision made by the state or FEMA saying, "OK, we’re going to use your facilities because you have the Maravich Center and others to house people"? How did that pan out? So, through the early storms that arose in the summer of 2005, we got wind that this might be coming. Again, our biggest concern in Baton Rouge was dealing with the evacuation of New Orleans and what that means to us. And what that physically meant was preparing our campus for being in a community that was going to swell in size. There were certainly some security aspects, as well as preparing this special needs facility to handle an influx of patients that might come to it. And the thought there was, normally this is a short-term thing, so when people evacuate New Orleans, there needs to be a place that's going to operate for 24, 48 or 72 hours, and then people would go back. Not unlike, I think, how the mayor of New Orleans and others viewed the Superdome. It was a place where you went to ride out the storm and then, when the storm was over, you left. In much the same sort of way, as the scope of the disaster dawned upon us, we realized we'd become a place that wasn't just using a small facility for some special needs, but that this is where we can help set up very expanded sorts of services. So, what happened was that medical units wanted to have a mobile hospital or a field hospital. And then you needed to have places to house disaster response workers. So there really wasn't this articulated plan to deal with something like Katrina. It sort of came to us gradually. It wasn't like someone showed up here Tuesday morning from FEMA or even the state and said, "We're in charge; this is what we're going to do; help us get this going." Separate agencies and separate response units started showing up to do things, and there was no coordination. Ultimately, LSU's administration and organization took charge of that. That happened because after a couple of days of wondering when someone's going to show up to be in charge, you've finally got to decide: OK, I guess we're going to be in charge. EdTech: That must have been a real juggling act for you and the IT staff and, obviously, for the administrators. You're used to dealing with multiple constituencies: faculty, staff, students, administrators, community, parents. And you already had everyone back and ready for school in late August, right? EdTech: So you had a full campus? EdTech: Since you had already had the students evacuate, the campus wasn't full when the storm hit? EdTech: And what percentage decided that they were going to leave campus as opposed to staying on campus The campus cleared out, and then we had to shift from our normal role of supporting IT in higher education to supporting this ever-growing disaster response effort. We spent lots of time installing telephone sets, data connections, providing network services, wireless, support for that. These agencies and people would show up, and of course they wanted IT access, and they didn't come in hauling their own computers. So we had to provide laptop computers or desktop computers to some of these people. As folks from parts of the LSU system that were affected in New Orleans started showing up in the LSU system office, there was the demand for them to have access to IT. Again we had to provide more equipment and network connectivity. Really, in the first day or two after the storm, we were scrambling to build our network out by a fairly significant amount to handle the response effort. Then, as we saw more of these evacuees and hospitals, patients, agencies and then volunteers, it became a chore to figure out how to track them all. These people show up to run these volunteer efforts, and they don't come with an information system, so they then need to know which doctors are here, which nurses are here, who's in, who's left, what patients have come in, what patients have gone out, a tracking sort of thing. We were asked to start developing some of those information tools as well. EdTech: Just so I can understand this a little bit better, tell me the size of your student population. EdTech: And the percentage that are local as opposed to out of state? EdTech: I'm assuming that the plan was that the campus would be shut down until the Thursday after the storm. You guys extended that considerably further. The following Monday after that was Labor Day, which would have been a day off, so the decision was made to restart classes on the following Tuesday. That decision was made later in the week. I think on Wednesday we told the students you're not going to come back until next Tuesday. But even that decision was jeopardized because this was just not a short-duration thing. And by that weekend, LSU made the decision to postpone its first home football game of the season, because this just wasn't a place to bring people to. There were no hotel rooms for people who were fans. There was no place to put the teams. And a lot of the facilities were in use for the response efforts. What we started to see unfold as that week went by was a real concern on the part of our chancellor that if we didn't start to make some moves back to being a university, we could find ourselves closed almost indefinitely. We had a lot of requests for, "Can we put responders up in dormitory rooms, or can we put evacuees up in dormitory rooms?" The students had already moved into them, so, no, we couldn't allow that. I think that the longer we had stayed closed, the more pressure we would have had to stay closed and be part of a longer term response effort. The other pressure was that as the week passed it became very apparent that the New Orleans-based institutions were not going to reopen in the fall, and the decision was made to offer displaced students the ability to enroll here. So there was also a pressure to extend the closing date so we could get more and more of these students registered. Ultimately, I think our chancellor made a very bold decision when he said, I think it was on Friday of that week, "Are we going to be a university this fall or not?" Of course, it was a rhetorical question, and the answer was, "Yes, we are going to be a university." So he stuck to Tuesday as the reopening date and said, "This is what we're going to do. It's going to be rough. There's going to be still an awful lot of external personnel in our campus environment. We're going to have all these new students who aren't going to know their way around. We're going to still have students who are migrating back. And," he said, "this is going to be sort of fits and starts and maybe not perhaps very elegant, but we're going to reopen classes on that Tuesday, Sept. 6, and we're going to work our way through it." And that turned out to be exactly the right decision. EdTech: I'm curious, did you have a means for keeping in touch with your students as the communications changed to tell them what the next steps were going to be? Did you have an effective means to get the information to your students? EdTech: Right. It's also useful as a way for parents to get the latest information if the phone lines are down or the shell systems are down, or some other problem exists. I know it's being used by a lot of schools now as the primary interactive way for people to communicate. EdTech: What taxed your IT systems most? I know at one point you mandated that your staff take 12 hours off; no ifs, ands or buts. What day was that, how stressed were they, and how taxed were your systems that you basically had to make some decisions to keep people working at a competent level? Also, were there any systems that were taxed to the point that your disaster recovery plan came into play or you had to rewrite the book on some things? As far as being taxed, the folks in our area who were responsible for networking were very taxed because this is a 24 x 7 operation. Not only during the first few days did you have, "We need more; we need more; we need more," but then as the agencies started trying to work with each other and trying to take on different functions concerning the overall picture, it became, "We need this moved here; we need this moved there; can you pick this up and move it over here?" Our networking staff had to do the vast majority of this. But we also pressed people into service who weren't really busy because they were in charge of supporting students on campus. We asked them to come down and help with some of these efforts as well. There were a lot of requests like these: "We need this information up or we need a little system that will help us track this." So even our systems development folks, who normally would have been standing idle at this time because they really couldn't make progress, were in working on these applications. Our high-performance computing people — the researchers — were running lots of models and wanted to look at what had happened to try to glean some of the science out of this disaster, so those resources were also taxed fairly heavily. I think that as we got to Labor Day and started to see a bit of a time shift here, I just decided that people had to go home. People who had been working 20-hour days, sometimes 24-hour days, just were not going to be able to exist in that mode for long. Plus, I knew that on Sept. 7 we were going to have our normal jobs back. Once the classes started, we were going to be spending time supporting the things we do at any other time the students are here. EdTech: In terms of the personnel resources that you had on hand, did you augment that with student resources? EdTech: Did you look for any other computer talent? EdTech: When you added new equipment, was it hardwired or was it Wi-Fi? EdTech: Did you have to put in new nodes for the wiring or were you just kind of jockeying the notebook PCs around? EdTech: That's a good thing. Looking back on the existing playbook for disaster recovery, since you've had some time to think about it, I'm curious how the playbook was before and how you see it now, in terms of preparing for any disaster you might face. Pre-Katrina versus now. Since then, I know that I've certainly taken a harder look at my own disaster plan. Recently, the chancellor asked for everyone's disaster plan because a new hurricane season is approaching. He specifically wanted to have a section in it that dealt with the lessons staffers had learned and what changes they had made since Katrina. I think that you're seeing those plans start to roll up and take advantage of this. The point is that when disaster strikes, do the rules and the plan go out the window? I have become more and more a believer in the fact that you cannot have a 6-inch binder filled with a very detailed instruction plan for dealing with disasters, because during a disaster you don't have time to look at it. During a disaster things unfold in a very chaotic manner. Just because you have a plan for how the last disaster affected you doesn't mean it will work in the next disaster. EdTech: I think you said that it can't be a "break-glass-in-case-of-disaster" plan. That's a good metaphor for it. EdTech: And today there's also so much more emphasis and scrutiny and legal issues about protecting the data. EdTech: Was your system hacked by anyone who thought you were vulnerable at the time? EdTech: You're lucky. I think, though, that we've been developing what we're going to do. We're looking at how we're managing our backup tapes. One of the lessons we learned from the New Orleans schools was that they had taken their data backups offsite, which is good. And they had taken them into downtown New Orleans and stored them in a very safe storage facility on the sixth floor of a building. Unfortunately, the first floor of that building flooded, and no one could get in there for weeks. So I said, where are our backups? And my staff said, "They're on the other side of the river from Baton Rouge." Well, that's no good. So we're taking a look at our normal processes and how we need to change those, being mindful of a very broad-based disaster. The second thing we're doing is looking at some disaster recovery spending that would help us ensure that the very key, critical services and systems are available. Those are the LSU Web space, electronic mail, connections to the Internet and access to some of this data. We're looking for how we could use an offsite facility someplace not in this region to restore those services rather quickly. EdTech: There are some schools, I'm told, that have started to think about using text messaging as a way to communicate in time of disaster. Another piece of the process is that we're assembling a "lifeboat." I told my disaster business continuity planning person to consider: What if we had a few hours' notice that we had to abandon ship? And that we'd have to set up our administration someplace else? What tools would we need to take with us so that we could support the university operating in exile? Given that we could find a facility someplace else, what data would we need so that we could bring back online our institutional data environments, in terms of financial, students, personnel and course systems? These kinds of strategies — taking a look at what we're doing today, spending some money to build a broadened infrastructure to help us prepare for a disaster, and then developing this lifeboat that we would grab and go — are the cornerstones of where we're heading with our disaster recovery plan, or business continuity plan may be the right way to put it. EdTech: Other than revamping some of your older disaster recovery plans, what are some other immediate changes you're going to make? Maybe more on a tactical level with your disaster recovery plan. In fact, when the University of New Orleans staff was trying to figure out how to get its student e-mail back up, we set up a deal with the company that does our outsourcing, and the company brought the UNO students' e-mail back up and gave them new accounts. What we're looking at now is how we can do the same sort of thing as backup for faculty staff e-mail. Things like that, to ensure that those critical services can be taken care of, are the tactical things that we're doing. Having a list of all the key people in our organization, having that on a concise phone list that all of us can carry, is a good tactical thing. It gives us, again, one of the two most valuable things you have in the event of a crisis - your people. And I've heard folks who have lived through the Ground Zero sort of effect say, "You know, IT workers today are becoming first responders." We have to show up because everything is IT dependent. We had better, as an IT organization, have in mind how we're going to take care of our staff. Do we have food, water, blankets, things like that? Can we provide some sort of accommodations for their families? These are new things that we've never had to think of before. On the Saturday before the hurricane a colleague of mine said that he and his wife were going to Dallas and asked whether my wife and I wanted to come. I thought about it briefly, but then said no. It's a good thing, too, because I needed to be here. IT workers are vital to the operation of higher education institutions today — just as vital as key administrators, campus police, facilities people and all those sorts of jobs. Because if we don't show up, they can't do their jobs, either. EdTech: As a part of that, since you play such a vital role, are you also looking at what kind of role will apply in terms of outsourcing your e-mail? Do you want your IT workers to be working on something tactical like getting a server running when there's a crisis, or just making that one call to the outsourcer to make sure it's up and running, and then doing the nine other things they could be doing if they weren't working with tactical technology, uptime issues? EdTech: Disaster can be categorized as almost anything. The same is true for a terrorist attack. Not at all to minimize what happened on Sept. 11, but the physical space that was affected was relatively small. Another such incident certainly could be much larger. We need to be thinking about this regardless of where we are. I don't think there's any place in the country that's safe. EdTech: Agreed. Of everything that's happened since your time there, during Katrina and post-Katrina, what surprised you most from an IT CIO's point of view? Was it how your administrators had to take over almost as civil defense workers? Or how things that you never anticipated would happen, happened? Ultimately the government did show up, represented by the Army units that rolled into New Orleans on Sept. 2 or Sept. 3. And ultimately the government did take over, but what surprised me was just how long it took for some sort of organization to come in and be put into place. From an IT perspective, what surprised me most was that people didn't realize how integrated information technology is into everything that we do. Whenever I talk about the IT vision, I start off by trying to make the case that IT is as critical and as strategic an asset to the institution as its buildings, its faculty, its curricula, its history. IT is a strategic asset because it is involved in everything. I have to make that point over and over again. What surprised me was that it really took this sort of event for people to start to understand just how dependent we are on technology. And people shouldn't be surprised; CIOs have been talking about this for some time. So it just surprised me that we still have a ways to go to convince people in higher education administration that IT is not this luxury item, or this awful expense or this thing to be tolerated. Rather it should be something that's embraced and accepted for what it is — an enabler of everything we do in higher education. EdTech: So you're seeing part of this as an opportunity to revisit the conversations that are being had about the way IT supports the university in achieving its mission. I'm finding it much easier, as I'm writing these things or talking to people about it, to impress on them how critically important it is, simply because of what we went through. So, from my perspective, the timing has never been better. I think that when we talk about IT in general, people now understand how critical it is and how important it is to advancing science, or advancing teaching and learning, or advancing how the university operates and communicates. I think I have a real opportunity, given this horrible thing that happened, to have people listen and perhaps be convinced by my arguments. EdTech: If anything, it probably enhances the argument and allows more CIOs to have a seat at the table. Recently, someone from the university budget office commented in a meeting that our budget is tight, but we're sure hoping that Brian Voss is going to bring us a proposal of how we might better prepare ourselves to continue business in the face of a disaster. And if there was ever an invitation from the money folks, I haven't heard a better one. My goal is to not come back with a number large enough to make them choke, but to give them a very manageable number that I believe they can fund but that will also advance a lot of the things we want to do with regard to business continuity. EdTech: I'm wondering before we wrap up, Brian, whether there are some points that you'd like to make in this interview that we haven't touched upon yet. We need to be reasonable. We need to look at disaster as something that's going to be different, it's going to be chaotic, and we need to prepare ourselves to be flexible with our plans. Concentrate on how are you going to have inventory around, how are you going to reach your people, how are you going to take care of your people? How are those things going to happen? Then go from there as opposed to trying to spend weeks and months writing very detailed scenarios, in case of "x," do "y." I think that's a real key that CIOs need to take into account here. Now's the time to do something and let's do something that puts us a step closer to being able to survive and adapt and improvise and overcome these situations. EdTech: In terms of that short list of really key, actionable things that should be included, is the lifeboat a part of it? That sort of thing, taking a look then at how are we going to handle these priority items. What is priority to me will be different from what's priority to somebody who's the CIO at Rice, or the CIO at Alabama, but to define those sort of fast action items that you need to have, that you would want to have backed up, replicated someplace else and take care of those. And then the third thing is, what if you had to abandon ship? What are you going to need to take with you not only to repopulate the university's information technology environment, but to provide that needed IT support to the university that could be working off of its campus for some time? EdTech: It's the period from six months to a year after the storm when they need aid and support and all that sort of thing. The other thing that I've heard many people repeat is: You really can't understand the devastation in New Orleans until you go there. If you have a chance to go to New Orleans, do that. Somebody at a conference I was at recently casually said, "Well, New Orleans is back up and running, and everything's going to be fine there in a couple of months, right?" The answer was, "No, not even in a couple of decades will it be all right." EdTech: That's where people have a short memory. Lee Copeland is editor in chief and Tom Halligan is editorial director of EdTech. |






Small Wonders
Clean Air
Universities’ Disaster Recovery Powers Up