Just a few weeks ago, staff and leaders at Southlake Regional Health Centre held a mock Code Grey, a drill to go over the critical steps needed to keep things running smoothly in the event of a significant infrastructure failure.
The practice stood the hospital in good stead on Friday when they, along with most of the rest of the country on the Rogers telecom network, were left offline in a major network outage.
On July 8, customers across the country were largely disconnected following what Rogers described as a “network system failure following a maintenance update.”
The outage not only left people unable to use their phones, television and internet, but without a means to use their bank cards at businesses, access their money at ATMs, or call for emergency services.
“We know how much our customers rely on our networks and I sincerely apologize,” said Rogers Communications president & CEO Tony Staffieri in a statement. “We’re particularly troubled that some customers could not reach emergency services and we are addressing the issue as an urgent priority.”
It’s an urgent priority across the country and York Region is no exception.
From the start of the outage early Friday morning, frontline and behind-the-scenes teams at Southlake and Central York Fire Services worked hard to reconnect through different means.
“While we do have some electronic mapping in the systems in the trucks, we got around any of the hiccups that we had and we were able to provide any services that were requested for that day,” says Chief Ian Laing of the Central York Fire Services. “It’s a bit of a challenge. Any delay in getting notification of an emergency generally makes the situation worse.”
Impacts were also felt at York Paramedic Services.
“York Region Paramedic Services proactively opened the Incident Management Support Centre on the morning of Friday, July 8, 2022, to provide additional support and ensure continue timely and critical care to residents,” said Linda Gonsalves, general manager, paramedic and senior services, in an email. “Essential staff reported to York Region Paramedic Services Headquarters to ensure service delivery continued during the outage. Ambulance radio operations and York Region’s 911 infrastructure were unaffected. York Region Paramedic Services did experience outages on a couple administrative systems, but these were easily resolved using our backup systems.
“York Region relies on multiple internet and cellular providers. Many Regional facilities also rely on connectivity provided through YorkNet fibre and as such services including landline phone and internet communications were relatively unaffected. York Region Paramedic Services is not aware of any service delays and has not received any complaints to date related to service on Friday, July 8, 2022.”
Responding was a bit more complex for Southlake, as not only were they temporarily unable to connect to important outside information, but staff shift changes were also left up in the air as staff who were not already on site were unable to call in.
“We knew something was up fairly early in the morning when things weren’t necessarily connecting as we would expect to them,” says Robert Bull, vice-president of facilities, IT, and CFO of Southlake Regional Health Centre. “Like any Code Grey, the priority is to figure out what is working as normal and what might be impacted. What are all those core systems? We were able to fairly quickly establish [connection] to the main health information system we use. Things within the building were generally working as they should be expected to. Our internal networks were all working. It was when we needed things to connect to the outside world that it was intermittent depending on what the system was.
“With a Code Grey we were able to establish early on the main systems that weren’t working and understand the implications of those…. We asked people to revert to downtime procedures for systems that don’t work. There were well-established procedures that people have in case these things happen…”
One of the elements that was more difficult to sort out were staffing levels. Staff members on the Rogers network who were sick or otherwise unable to come in were unable to call into the staffing office and, in turn, reaching staff on the same network to fill in was also a challenge.
“Morning shift change is generally 7.30 in the morning, so we knew what the implications were there and it was sort of too early for major impacts then,” says Bull. “The gap in information that we had was later in the day and was going to be for the night shift, which is generally 7.30 p.m. We didn’t have great visibility into how many people had tried to call in sick or report an absence and couldn’t get through to us.
“When we were trying to call out, we weren’t having a good success rate to connect to people. We had plans in place and our command centre is [planned so] people designated to certain roles… and we had plans for a 24-hour command centre all through the weekend. One of the things we had to do heading into the evening as we headed into the 7.30 shift change was huddle with all our leaders, managers and directors to go back to all their units, confirm they were in good shape for the night, that we as leaders [went to] staff to give them the heads-up as obviously we needed to make sure we had adequate staffing throughout the night.”
Eventually, Southlake was able to source alternatives to Rogers that gave them “flexibility in terms of being able to communicate with each other for people outside the building and our offsite locations.”
“We will probably see a little diversification of those communications channels so we can adapt if something like this was to happen again,” says Bull.
Brock Weir is a federally funded Local Journalism Initiative reporter at The Auroran
