[Op1st-dc-alerts] [mghpcc-alerts] Final Update re: MGHPCC Utility Power outage Saturday October 14 at 4:33PM

[mghpcc-alerts]

Update: Sunday October 15 at 11:45AM
----------------------------------------------
Power has been restored to all racks in the computer room, and the data center is back to its normal mode of operation.

If there are problems that we can address, please send email to help at mghpcc.org.

Thanks for your patience.


Update: Sunday October 15 at 11:22AM
----------------------------------------------
The power-on sequence for the computer room is now under way.


Update: Sunday October 15 at 10:30AM
----------------------------------------------
Testing has verified the root cause of the problem that prevented the computer room generator from going into service when utility power failed.   

We will stay in touch with HG&E as they diagnose root cause of the Utility power failure.

The final test is expected to complete by approximately 11AM.

Power will then be restored in stages, UPS-backed racks first.

Update: Sunday October 15 at 9:12AM
----------------------------------------------
We believe we have found the root cause of the problem that prevented the computer room generator from going into service when utility power failed.   

One more test to verify the diagnosis, and then we’ll start restoring computer room power. 

Power will be restored in stages.


Update: Saturday October 14 at 11:08PM
----------------------------------------------

Entrance Room Power is up

We have scheduled a technician from our electrical testing contractor to come in on Sunday morning to inspect the equipment that is the leading suspect in the failure of computer room generator power to take over when utility power failed.

This may delay bringup of Computer Room power beyond the 8AM time that was planned earlier, but we will hold as close as we can, balancing time to service with getting an early determination of root cause and remedy.


Update: Saturday October 14 at 8:50PM
----------------------------------------------
Based on feedback received, we will:
   Bring Entrance Room power up now
   Bring Computer Room power up Sunday morning at 8AM   


Update: Saturday October 14 at 7:30PM
----------------------------------------------
Power-on sequence has started.
We will be restoring power to the Entrance Rooms at approximately 7:40PM

Please send mail to help at mghpcc.org if you see current or prospective issues as
power comes back.


Update: Saturday October 14 at 7:29PM
----------------------------------------------
HG&E has restored its power feed.
We are starting the power-on sequence for the data center.

Power will be restored in stages.

Update: Saturday October 14 at 6:10PM
----------------------------------------------

We have confirmed that power to UPS-backed systems in the Computer Room and Entrance Rooms is down.  

Immediate steps under way are:
(1)	Make sure it’s OK to restore utility power. 
(2)	Send an alert warning that utility power is about to be restored.
    (2) Restore utility power. 

Follow up steps:
    (3) Root cause the problem the prevented generator power from reaching UPS-backed 
         racks in the computer room and ERs.
    (4) Stay in touch with HG&E as they root cause the reasons why the North Canal 
         Substation went down when a fault occurred there, instead of failing over to a 
         redundant circuit.



Saturday October 14 at 4:33PM
------------------------------------------

The primary utility power feed to the MGPCC facility was interrupted at 4:33PM today (October 14, 2023).  

UPS-backed power may have been affected.

We will provide updates as we make progress toward restoring power via the backup power feed.