9-1-1 CRASHES
Restoring And Preventing
9-1-1 System Failures
By Nancy J. Rigg
| Nancy J. Rigg is a writer, filmmaker and consultant with an extensive background in Search and Rescue, public safety education, and disaster preparedness. She is a frequent contributor to 9-1-1 Magazine. |
| Photographs: (select thumbnail to view detail image) |
 Ameritech employees monitor the 9-1-1 network for their customers at the Ameritech 9-1-1 Resolution Center in Chicago. This center makes proactive contact with PSAPs in the event of a failure to restore either a crashed system or the network that feeds calls. Ameritech |
 The Alameda County, California, Communications Center. Last January, both the primary and back-up 9-1-1 systems crashed simultaneously rendering 9-1-1 inoperable for two hours and causing dispatchers to hurridly make other arrangements to insure emergency calls reached the Center. Randall Larson |
 EMS Dispatchers for American Medical Response in Fremont (CA) lost all radio, phone, and AVL communications when burglars cut cables at a store next door to keep the alarm bell from going off. Alameda County Comm gave the AMR dispatchers room and radios to continue dispatching until the damage was repaired. Randall Larson |
 Dispatcher Debbie Cortez of the Reno (NV) Police Department's consolidated communication center provides dispatch services to 9 public safety agencies, all of whom depend on consistent and redundant 9-1-1 networks. Alan Mentzer |
This article can be found on
page 28 of the May/June 1998
issue of 9-1-1 Magazine.
A t midnight on New Year's Eve, in San Bernardino County, California, an overly enthusiastic reveler aimed his pistol into the air and started firing. The celebratory bullets that blasted through phone lines somewhere up above also blew holes through the County's 9-1-1 system.
On January 1st, a two hour long effort by Pacific Bell to restore 9-1-1 service for the Los Angeles Police and Fire Departments was hindered after a media alert caused anxious residents to swamp every 9-1-1 center in the county with concerned calls.
On January 22nd, dispatchers in the Alameda County (CA) Sheriff's Office noticed things seemed to be much quieter than usual. A call from the neighboring Hayward Police Department soon alerted them to the fact that all of their 9-1-1 calls were being forwarded, confirming their suspicions about an outage. When Pacific Bell repaired a switcher, bringing the Alameda County Sheriff's 9-1-1 system back on line, the 9-1-1 system in another nearby community, Union City, went down. Two nights later, for the first time ever in Alameda County, both the primary and back-up 9-1-1 systems crashed simultaneously.
When 9-1-1 systems fail at the Public Safety Answer Point (PSAP) level, automatic transfer or equipment back-up systems usually kick in quickly, avoiding any serious disruption of service. But when there is a catastrophic failure of both the primary and back-up systems, lives can be at stake.
Bob Bassett, Emergency Services Dispatch Manager for Alameda County, explained that having a "double failure" of both the primary and back-up 9-1-1 systems "is very, very rare." In the twenty years that there has been a 9-1-1 system in the San Francisco Bay area, the January crash "was the worst failure that we've ever had," said Bassett. "The systems are well built and well monitored. And there are back-ups to the back-up systems. But this time, even the back-up systems failed."
During the countywide outage, Alameda County Sheriff dispatchers initially thought that their calls were again being forwarded to Hayward, as they had been two nights earlier when the switcher failed. "But this time," Basset said, "calls were not being forwarded. In fact, nothing was happening. So we started calling surrounding agencies and found out that it was awfully quiet everywhere. And everyone was getting a quick busy signal when they tested 9-1-1." Even as Pacific Bell was being notified, Bassett recounted how "somebody must have been scanning or something, because the press got hold of the story very quickly."
Unlike in Los Angeles, citizens did not swamp phone lines in an attempt to see if the 9-1-1 system was really down or not, Bassett explained. But a member of the Alameda County Board of Supervisors, who was attending a public function, was unhappily caught off guard when asked by a reporter what was wrong with the 9-1-1 system. Bassett noted that the supervisor "wanted Pacific Bell to put a recorded message on the line, telling people what seven digit numbers to call." This suggestion, although seemingly helpful, posed problems. "If you're dialing 9-1-1, it's because you're having trouble and need help fast," Bassett said. "You're not going to have time to get out a pencil and jot down a seven digit number off of a recorded message. But if you get a fast busy signal, you should know you can't get through because something is wrong, so you use another option."
Access to help was still available through traditional seven digit phone numbers and telephone operators. Cellular phones were able to connect to 9-1-1 through the California Highway Patrol. And at the request of both Pacific Bell and local public safety agencies, the news media began broadcasting seven digit emergency phone numbers for the fourteen affected cities within the county. "The state mandates that we maintain seven digit numbers in addition to 9-1-1 to manage during a situation like this one," Bassett said. "Maybe the public needs to be reminded that losing 9-1-1 does not mean you won't get help in an emergency. Everyone should keep a sticker on their phone that includes 9-1-1, the local seven digit number, and their own address in the event that they can't remember where they are when they call for help."
A press release from Pacific Bell indicated that "the NEC 405 multiplex failed due to a power failure in the hardware, causing the Orinda 5E central office to fail. The back-up power supply also failed. Orinda is the 9-1-1 tandem for Alameda and Contra Costa Counties. Approximately twelve Public Safety Answer Points (PSAPs) were affected. The outage occurred at 6:22 PM and was corrected by 8:25 PM."
"To our knowledge," Bassett reported, "nothing major was intensified by the fact that 9-1-1 was not functioning."
Call-Before-You-Dig:
"All electronic systems can go down," explained Ralph Grimes, 9-1-1 Senior Account Manager for GTE California. "There's no way to prevent failure from time to time of some, or even all, components. But you need to take all logical measures to prevent system failures." For Grimes, the "number one thing that you do is practice good maintenance procedures, build as much redundancy into the system as is economically possible, and keep key pieces of equipment in secure sites. Between secure sites, redundancy, and good maintenance and procedures, we can maintain a fairly secure system from the standpoint of liability."
In addition to building the most secure and technologically advanced systems possible, Southwestern Bell 9-1-1 Account Manager Cindy Clugy encourages her clients to develop detailed contingency plans for a variety of potential sources of disruption. "Unplanned construction outages are among the most common problems," she explained, "where the phone service is affected by somebody with a back-hoe who isn't paying attention to where they're digging."
To offset construction-related outages, Clugy recommends mounting aggressive public awareness campaigns and strengthening local regulations. "There have been two initiatives in the Texas legislature to pass `call-before-you-dig' initiatives," she said. Clugy encourages 9-1-1 administrators and public safety agency managers to "really get behind this kind of effort if you don't already have something in place. Call-before-you-dig legislation is a very important preventive measure."
PSAPs In Partnership
Even with the best efforts at prevention, unplanned outages may still occur due to equipment failure, human error, and other circumstances. To reduce the potentially negative impact on emergency services, Cindy Clugy recommends that dispatchers establish a "buddy system" with a neighboring 9-1-1 center. "A lot of our PSAPs have a buddy system," she explained, "so if they go into failure and can't answer calls, instead of automatically sending calls to their own seven digit phone number, they have an agreement with a neighboring PSAP to take their calls. This works well not only in the event of a catastrophic failure at the equipment level, but in any emergency where the PSAP is isolated or dispatchers might have to evacuate the building."
Clugy recounted a situation where one PSAP, located in the same building as the county jail, had pre-programmed its own seven digit telephone number as its designated alternate answering point. When inmates started rioting, burning mattresses, and backing up toilets, Clugy indicated that the whole building had to be evacuated, including the 9-1-1-answering center. Dispatchers placed a frantic call to the phone company asking to have their calls re-routed outside the building to a neighboring 9-1-1 center. While this request was "not impossible" for the phone company to fulfill, Clugy explained that because the "programming for the alternate answer point is deeply embedded within the network translations of the 9-1-1 system, re-routing calls someplace other than to the pre-programmed destination takes extra time. That's why we recommend that you think very carefully about where you want your calls to roll over to when you set up your alternate answer point."
This decision is rarely a simple one, Clugy acknowledged. For political reasons, as well as convenience, many PSAPs prefer to designate their own seven digit phone number as the alternate answer point. "Public safety agencies can be territorial," she added bluntly. "Some police and fire departments simply don't want anyone else to handle their business, so they roll 9-1-1 calls over to their own seven digit line. You can't anticipate every disaster, but you can factor in the political climate with the potential for having to evacuate the 9-1-1 center and evaluate where you feel it's best to send 9-1-1 calls when a problem arises. The bottom line is that, technically, you have the ability to send your calls to another location. How you choose that location is dictated by politics and local policy. But if someone mixes ammonia and bleach and gasses out the PSAP, you might need to throw the switch on the way out the door and move all your calls to another answering point outside the building."
Bob Bassett supports the buddy system concept, noting that cooperation
and mutual aid are the keys to managing even the most unusual disruptions.
"The paramedic provider for Alameda County is American Medical Response
(AMR)," Bassett said. "Their central dispatch system was completely
shut down a while back by vandals who were trying to rob a store next door.
When they cut cables to keep an alarm bell from going off, they not only
took out the store alarm, but the paramedic radio link, phone link, and
satellite link for the AVL (Automatic Vehicle Location). AMR had no communications
whatever, so we brought them into our office to dispatch from our Comm Center."
Diversity and Redundancy
Tim Jenkins is 9-1-1 Operations Director for Ameritech, which provides 9-1-1 service to 850 PSAPs in a five-state area. In addition to public awareness to reduce incidents like target shooting at telephone poles or uprooting phone cables during construction, Ameritech ensures that there is sufficient diversity and redundancy within the system to offset catastrophic failures. "We have ten selective routing-automatic identification processors (SR/ALI) throughout the Ameritech region, which contain customer and routing data. They're grouped together in paired sets and are completely redundant, so that within each paired set the processors contain identical information."
Ameritech has a special 24-hour per day 9-1-1 Resolution Center that
constantly monitors the network as a whole and makes proactive contact with
public safety agencies to restore either a crashed system or the network
that feeds calls. "At the same time, we are making contact with the
Ameritech media relations office to let them know that they may need to
get in touch with the media to begin broadcasting special announcements,"
Jenkins said. "We will take action based upon what the public safety
agency wants us to do."
Alert The Media?
Asking local television and radio stations to broadcast information about 9-1-1 service disruptions may be necessary, but is also risky, according to GTE's Ralph Grimes. "Agencies would probably be better off not publicizing a disruption if the back-up system has kicked in, since it often impairs the effort to restore service."
Cindy Clugy agrees. "People are concerned about their own safety," she said. "So if we announce in the media that there's a problem with the 9-1-1 system, people are going to start calling 9-1-1 just to see for themselves if it works or not." Clugy works with public safety agencies and Southwestern Bell's media relations staff to determine whether or not the outage is worth reporting. "Generally, my rule of thumb is that if the outage will be cleared before news time, and if we've got the calls re-routed, and they're being handled by trained personnel, I don't say a word. Granted, dispatchers answering non-9-1-1 lines may not automatically get some information, but is it really worth going on the six o-clock news to tell people that for 45-minutes we weren't getting addresses and phone numbers automatically?"
Dennis West, communications shift supervisor San Bernardino (CA) County Fire, indicated that public education efforts need to include a reminder about the presence of seven digit, back-up numbers. "There's a misconception about 9-1-1 that it gets help to you quicker. It does not get help to you quicker, but it is an easier number for you to access the system. It still takes just as long to interrogate someone, and just as long for emergency vehicles to drive down the street to a location."
Tim Jenkins of Ameritech summed it up this way: "In many cases we've
probably over-educated the public to depend on dialing 9-1-1 only."
Advanced Technology:
In addition to prevention and education, improvements in technology and system design are enhancing the reliability of 9-1-1 systems. Ralph Grimes indicated that one major development within the GTE system is upgrading all of their 9-1-1 centers to Intelligent WorkStations, which put the telephones into the dispatcher's computer, affording a great deal more information as well as integration with graphical mapping, AVL, and various databases.
Cindy Clugy is enthusiastic about technology advances, but believes that the strength of the system lies in the continued teamwork of telephone companies and public safety agencies. "We're working within our network to upgrade our facilities to improve prevention," she said. "We look at those areas that have had trouble and target them for upgrade. But the bottom line is that when there's a system problem on 9-1-1, someone calls the phone company. We diligently try to get it fixed. A lot of times we're the hero. But there have been a few times that we've been a little slow. We would like to continue to do better."
