Data Center

CoSN2023: Schools Share Lessons in Disaster Recovery

Two schools’ disasters highlight the importance of data backups for continued operations.

Twitter

Rebecca Torchia is a web editor for EdTech: Focus on K–12. Previously, she has produced podcasts and written for several publications in Maryland, Washington, D.C., and her hometown of Pittsburgh.

For a K–12 school system, “disaster” can mean many things. Minnesota’s Minnehaha Academy and Oregon’s Beaverton School District dealt with separate disasters in their school systems, and both institutions found their data completely offline as a result.

In Tuesday’s session “This Is Not A Drill: Successfully Navigating A Disaster,” school personnel shared with CoSN2023 attendees a timeline of events, including recovery efforts for data that was lost.

IT Consultant Dan Cummings shared how a fatal natural gas explosion took out Minnehaha Academy’s main data center and onsite data backups on Aug. 2, 2017. The explosion injured the school’s network administrator and sysadmin and took the lives of a receptionist and a custodial worker.

“The damage was extensive from the blast but also from the shock waves that went through the entire building. Every piece of the campus was touched by the percussion,” Cummings said.

A stream of notifications that systems were offline was the first indication to IT consultant Dan Cummings that a gas explosion had occurred at @MinnehahaAcad. pic.twitter.com/nSBB1PDgbK

— EdTech K–12 Magazine 🔜 #CoSN !! (@EdTech_K12) March 21, 2023

Shock waves were also part of the initial problem at Beaverton School District, where faulty valves on halcyon gas cannisters in the data center created a sonic boom three days before school started in 2013. The data center’s hard disc drives, which had been spinning at 15,000 rpm, were knocked offline by the vibrations.

The real disaster came the following day, however, when CIO Steve Langford learned that 100 percent of the district’s HR finance data was lost because of a misconfigured backup. Audience members audibly gasped as Langford explained that the district had no payroll data for its 5,000 employees, no information on their vacation time and no way to know who had stopped working for the district in the past year.

Minnehaha Academy’s Road to a Network and Campus Rebuild

The recovery processes for Minnehaha Academy and Beaverton School District looked very different, but each was fraught with high emotions, big challenges and small victories.

Click the banner to sign up as an Insider and catch all the conference coverage.

Because the explosion rendered Minnehaha Academy’s north campus unusable, the district relocated its main data center and administrative hub to its south campus. Meanwhile, the school needed a temporary campus for additional space. Cummings said the district was fortunate to find a building with relatively modern infrastructure that had been vacated by Brown College two years earlier.

With a lot of helping hands, the new location was turned into a temporary high school in three weeks. The campus was transformed with an Aruba network that included 10 switches, 60 access points and a 20-gigabit fiber backbone; a VoIP phone system; overhead paging; interactive flat panels, projectors and other audiovisual technologies; a point-of-service system for the lunchroom; and security cameras and door access control systems.

Cummings said much of this was possible because the existing infrastructure in the vacated building had been well maintained. It was labeled, documented and well organized, making it easy for Minnehaha Academy to adapt the technology for its own purposes.

By 2019, the school’s original campus had been rebuilt with a focus on hybrid data centers and a move away from on-premises storage. The school also prioritized an ongoing move to cloud-based applications. Other networking technology in the new building included CAT 6A cabling, VMware servers, Aruba switches and access points, and external Wi-Fi.

DIVE DEEPER: Should K–12 schools invest in Backup as a Service?

Beaverton School District’s Road to Data Recovery

While Beaverton School District had to spend less time physically rebuilding its data centers, school leaders like Langford needed a way to solve for missing data, and he needed to do it quickly.

“It’s a million-dollar penalty if you don’t make payroll,” he said.

He and his team started by communicating the problem to the staff members. He remembers painful phone calls in which “it would have been easy to blame the technology.” Instead, he and the team owned up to their mistakes in not having correctly backed up the data.

Langford referred to the recovery process as a roller coaster. Celebrating even a small victory was often followed by a stomach-churning realization or new challenge. A heat-damaged backup tape sparked exaltation, only to be followed by despair as the district began building backups from scratch.

One of the biggest breakthroughs, he said, came when the team discovered an employee had exported HR data. “He didn’t know they weren’t supposed to keep payroll PDFs on their desktop,” Langford said to audience laughter. “We had to say, ‘We’re super mad at you, but thank you so much.’”

Looking back on the disaster recovery process, Langford admitted there were times that he wanted to quit his job and walk away from the challenges. He stressed the importance of resiliency in leadership and communication with team members and school staff. “How are you going to react in a horrible moment of crisis?”

The session resonated with the full audience, who took notes, asked questions and approached the speakers after the session concluded. Both Langford and Cummings urged schools to have a disaster recovery plan that includes offsite backups and to check those backups regularly.

Join EdTech as we provide written coverage of CoSN2023. Bookmark this page and follow us on Twitter @EdTech_K12.

Photo by Rebecca Torchia