[Op1st-dc-alerts] [mghpcc-alerts] Update re: MGHPCC Utility Power outage Saturday October 14 at 4:33PM

[mghpcc-alerts]

Update: Saturday October 14 at 9:00PM
--------------------------------------------
Based on feedback received, we will:
   Bring Entrance Room power up now
   Bring Computer Room power up Sunday morning at 8AM   


Update: Saturday October 14 at 7:30PM
--------------------------------------------
Power-on sequence has started.
We will be restoring power to the Entrance Rooms at approximately 7:40PM

Please send mail to help at mghpcc.org if you see current or prospective issues as
power comes back.


Update: Saturday October 14 at 7:29PM
--------------------------------------------
HG&E has restored its power feed.
We are starting the power-on sequence for the data center.

Power will be restored in stages.

Update: Saturday October 14 at 6:10PM
--------------------------------------------

We have confirmed that power to UPS-backed systems in the Computer Room and Entrance Rooms is down.  

Immediate steps under way are:
(1)	Make sure it’s OK to restore utility power. 
(2)	Send an alert warning that utility power is about to be restored.
    (2) Restore utility power. 

Follow up steps:
    (3) Root cause the problem the prevented generator power from reaching UPS-backed 
         racks in the computer room and ERs.
    (4) Stay in touch with HG&E as they root cause the reasons why the North Canal 
         Substation went down when a fault occurred there, instead of failing over to a 
         redundant circuit.



Saturday October 14 at 4:33PM
-----------------------------------

The primary utility power feed to the MGPCC facility was interrupted at 4:33PM today (October 14, 2023).  

UPS-backed power may have been affected.

We will provide updates as we make progress toward restoring power via the backup power feed.