Suppliers RFO Posted Below:

Reason for Outage (RFO) Report

Incident: Temporary Loss of Power During UPS Maintenance Date of Incident: 09/09/2024

Prepared by: James Davenport Date of Report: 09/09/2024

‌

Incident Overview:

‌

During scheduled UPS maintenance on 09/09/2024, two temporary power outages occurred when transitioning the system to static bypass mode. The shared static bypass unit, serving all three UPS systems on the B stream, failed during both attempts. After additional checks and troubleshooting, the system was successfully transitioned to bypass, allowing safe work to proceed on replacing the failed units. Power has been stable since the final transition, and there are no long-term impacts on customer operations.

Incident Details:

‌

• Date & Time of First Incident: 09/09/2024, 11:15am
• Date & Time of Second Incident: 09/09/2024, 13:25am
• Date & Time of Successful Transition to Bypass: 09/09/202
• Duration of Power Loss for Each Incident: 50ms first incident, 15 seconds second

Cause of the Outage:

‌

The outages were caused by the failure of a shared static bypass unit during the transition to bypass mode. The static bypass unit, which is part of the aging infrastructure shared by all three UPS systems, failed twice during the switching process. After further checks and troubleshooting, the system was successfully transitioned to bypass, allowing the failed units to be safely bypassed for replacement.

‌

Sequence of Events:

‌

1. First Incident:
   The first power outage occurred during the scheduled maintenance when transitioning the system to static bypass mode. The shared static bypass unit failed, leading to a temporary loss of power to customer equipment. Power was restored quickly by reconfiguring the system.
   
2. Second Incident:
   After power was restored and additional checks were carried out, a second attempt to transition to bypass mode resulted in another failure. Power was once again quickly restored after bypassing the failed unit.
   
3. Successful Transition:
   After further troubleshooting and verification, the system was successfully transitioned to bypass mode without further incident. This allowed safe access to the faulty bypass unit for maintenance and replacement.

Contributing Factors:

‌

• Aging Equipment:

The shared static bypass unit is over 20 years old and showed signs of significant wear. The failure in both instances was related to the deteriorating condition of this equipment and the reason why we embarked on a Critical infrastructure replacement scheme at Reynolds House.

‌

• Legacy System Challenges:

Despite multiple pre-inspections and checks, the legacy infrastructure posed inherent risks during operational transitions.

‌

Resolution:

-Immediate Action Taken:

Power was restored promptly after each failure. After further troubleshooting and testing, the system was safely transitioned to bypass mode, isolating the failed static bypass unit for maintenance.

‌

• Next Steps for System Repair:

The failed static bypass unit will now be safely worked on and replaced as part of the ongoing infrastructure upgrades.

Corrective Actions:

1. Bypassing the Failed Units:
   The failed static bypass unit has been bypassed, allowing for safe work to proceed on its replacement.
   
2. Expedited Replacement of the Static Bypass Unit:
   The aging static bypass unit has been removed and will not be replaced.

‌

Preventative Measures:

1. Complete Replacement of Legacy Systems:

The static bypass unit and associated aging infrastructure has been removed, eliminating future risks during transitions and load transfers.

‌

Conclusion:

The failure of the shared static bypass unit caused two temporary outages during maintenance. After extensive checks and troubleshooting, the system was successfully transitioned to bypass, allowing the faulty units to be isolated and worked on safely. The replacement of the aging static bypass unit is now proceeding, with additional safeguards in place to ensure continued stability.

‌

Next Steps:

1. Complete the replacement of the remaining aging UPS components.
2. Continue enhanced monitoring until the new system is fully operational.
3. Conduct a detailed review of other critical components and prioritize replacements for any remaining legacy systems.
4. Work with customers to make sure each rack has a fully redundant A and B power path.