[Op1st-dc-alerts] [mghpcc-alerts] Update re: MGHPCC Utility Power outage Saturday October 14 at 4:33PM CCF23667273

[Crown Castle Fiber NOC]

+ CCF23667273

Michael Mackley
NOC Technician II
T: (855) 93-FIBER
CROWN CASTLE FIBER
fibersupport at crowncastle.com
Fiber.CrownCastle.com

-----Original Message-----
From: mghpcc-alerts <mghpcc-alerts at mghpcc.org>
Sent: Saturday, October 14, 2023 9:01 PM
To: mghpcc-alerts <mghpcc-alerts at mghpcc.org>
Subject: [mghpcc-alerts] Update re: MGHPCC Utility Power outage Saturday October 14 at 4:33PM

[You don't often get email from mghpcc-alerts at mghpcc.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Update: Saturday October 14 at 9:00PM
--------------------------------------------
Based on feedback received, we will:
   Bring Entrance Room power up now
   Bring Computer Room power up Sunday morning at 8AM


Update: Saturday October 14 at 7:30PM
--------------------------------------------
Power-on sequence has started.
We will be restoring power to the Entrance Rooms at approximately 7:40PM

Please send mail to help at mghpcc.org if you see current or prospective issues as power comes back.


Update: Saturday October 14 at 7:29PM
--------------------------------------------
HG&E has restored its power feed.
We are starting the power-on sequence for the data center.

Power will be restored in stages.

Update: Saturday October 14 at 6:10PM
--------------------------------------------

We have confirmed that power to UPS-backed systems in the Computer Room and Entrance Rooms is down.

Immediate steps under way are:
(1)     Make sure it's OK to restore utility power.
(2)     Send an alert warning that utility power is about to be restored.
    (2) Restore utility power.

Follow up steps:
    (3) Root cause the problem the prevented generator power from reaching UPS-backed
         racks in the computer room and ERs.
    (4) Stay in touch with HG&E as they root cause the reasons why the North Canal
         Substation went down when a fault occurred there, instead of failing over to a
         redundant circuit.



Saturday October 14 at 4:33PM
-----------------------------------

The primary utility power feed to the MGPCC facility was interrupted at 4:33PM today (October 14, 2023).

UPS-backed power may have been affected.

We will provide updates as we make progress toward restoring power via the backup power feed.
This email may contain confidential or privileged material. Use or disclosure of it by anyone other than the recipient is unauthorized. If you are not an intended recipient, please delete this email.