Business Continuity Incidents in UK & USA

45

Business Continuity Incident: California DMV – The failover that failed

 

Based on limited information the DMV provided in response to questions submitted by The Associated Press, the experts said it appears the DMV’s technology infrastructure falls short of the best industry practices.

In particular, they said, the DMV should have a plan to recover from a catastrophic failure that involves distinct computer systems that are physically separated with independent power supplies. That way, if one data center — or even one section of a data center — overheats or experiences a power surge, backup systems aren’t affected.

The California DMV ran primary and backup systems side-by-side in the same hardware cabinet.

“If their definition of disaster recovery is having primary and backup systems in the same hardware chassis, that’s grotesque,” said Richard Fichera, a vice president of Forrester Research who advises large companies on servers, storage and data centers. “That is completely inadequate for a critical statewide agency like the DMV.”

The DMV said Friday that three offices were still experiencing problems but it expected all to return to normal operations by the end of the day. The department says its systems were not hacked or targeted and it did not lose customer data.

DMV officials have said the outage was triggered by the failure of hard drives in both primary and backup systems at one of its two server facilities but have not said what caused them to fail.

The DMV’s system was designed to withstand failures in the primary or the backup system but not both, said Jaime Garza, a DMV spokesman. The ability to recover from a disaster was degraded because the systems in two separate locations were used simultaneously, he said.

“The computer system has redundancy,” Garza said. “Unfortunately, with the loss of several hard disks in a short time period, that redundancy was lost.”

Maintaining primary and backup drives within the same physical location and even the same equipment cabinet is “insane,” said Stuart Lipoff, president of IP Action Partners, a technology consultant based in Newton, Massachusetts, who reviewed the DMV’s explanation of the failure.

“It’s inexcusable that the entire system would come down,” Lipoff said. “They were not observing what are good practices and good guidelines that I think any modern IT department would practice.”

The outage began Monday and, at its peak, two-thirds of the DMV’s 188 offices around the state were unable to process driver’s license or vehicle registration matters. By late Thursday, six offices still had limited services available.

The outage forced DMV customers to wait out the troubles. Some said their lives were put on hold while they waited to replace stolen identification cards or renew vehicle registrations on the verge of expiring.

DMV officials say the department may waive late fees for customers affected by the outage. They’ll have to fill out a form or write a letter explaining why they’re late.

Assemblyman Mike Gatto, D-Los Angeles, said the DMV’s computer issue is unlikely to have put personal information at risk, but is unsettling nonetheless.

“This is an agency that starts with a very skeptical public,” Gatto said, but “this is way beyond the typical frustrations that people expect from the DMV.”

 

Business Continuity Incident: Fire destroys hotel & hits nearby businesses

 

Exeter’s business community has begun to rally round to help those affected by the devastating fire which gutted the Royal Clarence Hotel and neighbouring premises.

Many businesses in the area of Cathedral Green will be counting the cost of lost trade as they remained inaccessible to staff and customers on Friday and Saturday while the major incident unfolded.

And those whose premises have been destroyed or damaged are facing weeks and months of disruption. Exeter City Council’s economy team has urged businesses in need of temporary office space to get in touch.

Council chief executive Karime Hassan tweeted that “heartwarming” offers of help had been received from businesses in the city.

He also thanked Greggs, one of several businesses which provided refreshments for emergency services at the scene of the fire.

Mr Hassan added that footfall in the city centre on Saturday was down on the previous week, with ongoing travel disruption and people having been advised to avoid the area of Cathedral Green, but thanked the 44,000 who visited for their support.

Nick Baxter-Sibley, board director of Exeter Chamber of Commerce & Industry, said: “We are saddened by the terrible news about the fire in the Cathedral Yard. Our thoughts are with those affected and it is great relief to know that no one has been injured.

“The emergency services are dealing with the situation and we pay tribute to the experience and skill of the fire fighters who are tackling the on-going incident.

 

Wolverhampton business destroyed by fire

Ten people were evacuated from the three-storey building on Marshalls Industrial Estate, in Blakenhall, at around 7pm yesterday.

One man who worked inside the basement of the building, home to a t-shirt making business, alerted fire crews to the blaze.

Flames were sparked around a conveyor belt machine used to dry the t-shirts inside the basement measuring around 65ft by 65ft.

Around 20 firefighters from Wolverhampton, Fallings Park and Bilston fire stations arrived at the scene to battle the blaze.

People were evacuated from neighbouring businesses inside the building and raods were closed as the fire was tackled.
Read more at http://www.expressandstar.com/news/emergency-services/2016/10/28/wolverhampton-t-shirt-business-wrecked-by-fire-as-worked-evacuated-from-factory-units/#jj44LrfqW7YmgFqM.99