[Op1st-dc-alerts] [mghpcc-alerts] COMPLETED **Potentially Service-Affecting** Electrical Work December 12-17, 2022

mghpcc-alerts mghpcc-alerts at mghpcc.org
Fri Dec 16 17:40:06 EST 2022


All of the electrical work and generator maintenance that was scheduled
for this week has been completed successfully.

Thanks for your cooperation while we finished several important upgrades
to the facility.


======= December15, 2022 Alert ==========

Power was restored this afternoon (ahead of schedule) to the switchboard that 
was powered down on 12/13.

On 12/15, we will start work that will affect the power feeds to both sides of:
    R1-PA 
    R5-PB

Also, a reminder that Generator maintenance will be taking place on 12/15 and 12/16.
Repeating details from the original alert here for convenience:
   MGHPCC will be performing routine preventive maintenance on the two generators 
   in the transformer yard. This will involve disabling each generator while the maintenance
   work is being done.  In the unlikely event that a utility power outage occurs while one 
   of the generators is unavailable, operation of both UPS-backed and non-UPS-backed 
   equipment in the data center would be disrupted. 

======= December12, 2022 Alert ==========

Work on the switchboard that was powered down on 12/13 continues today.
We currently expect that power will be restored on Thursday 12/13, as long
as there are no unexpected problems today.  Areas affected by this work are:
    Pods
       R6-PA
       R6-PB
       R6-PC
    Entrance Rooms
       ER-01
       ER-02

After the switchboard is powered back up, the final stages of the
project will affect two pods:
     R1-PA
     R5-PB

======= December13, 2022 Alert ==========

The switchboard that was powered down this morning will be 
powered up again on Tuesday morning at approximately 9AM.  
30 minutes later, the second switchboard involved in the construction
activity will be powered down.

Apologies for publishing an incorrect list of affected pods.  The areas 
affected today and tomorrow are:
    Pods
       R6-PA
       R6-PB
       R6-PC
    Entrance Rooms
       ER-01
       ER-02

We will update tomorrow with additional information about activities 
planned for the rest of the week.


======= December12, 2022 Alert ==========

The first of the four switchboards involved in the construction activity 
was powered down at approximately 8:30 this morning.  In addition 
to the pods called out in the November 30 and December 7 alerts, 
this affected the two MGHPCC Entrance Rooms.  Apologies for not 
making that clear in the earlier alerts.

The Comcast modems that support management connections for 
some of you are single-corded, and went down for a brief period 
while we switched them over to the alternate feed.  

We are also aware of two network devices that appear to have 
been affected by loss of one of the two redundant power feeds.  
The networking organizations that maintain those devices will 
communicate status as needed.


======= December 7, 2022 Alert ==========

Notice of two upcoming events that have the potential to affect service.

1. Plan to power down the switchboards that feed power to the following 
    Pods in the MGHPCC computer room at different times during the week 
    of December 12, 2022.
      R1-PA, 
      R1-PB (even side)
      R2-PA, R2-PB
      R3-PA (even side)
      R4-PA (even side)
      R5-PB 

2. Generator maintenance on December 15 and 16, 2022.

All work will take place between 7AM and 6PM.

Tenant Impact
--------------------
There may be brief periods of time when we will ask everyone to leave the 
Computer Room for safety (e.g. when a switchboard is being powered up).

Computing, Storage, and Networking equipment in the pods listed above 
that has been properly configured for operation with power cords attached 
to both the “A” and “B” PDUs in their racks should not be affected. 

Equipment in pods other than the ones listed above should not be affected, 
independent of configuration.

There is elevated risk for the entire Computer Room while the work is 
being done.

Description
—————
The switchboards that feed power to the pods listed above will be powered 
down to install new electrical circuits in the MGHPCC computer room.  The 
switchboards will be powered down one at a time, so there will always be at 
least one bus with active power in every pod.  Each of the four switchboards 
involved in this work will be powered down for approximately two days.

MGHPCC will be performing routine preventive maintenance on the two generators 
in the transformer yard. This will involve disabling each generator while the maintenance
 work is being done.  In the unlikely event that a utility power outage occurs while one 
of the generators is unavailable, operation of both UPS-backed and non-UPS-backed 
equipment in the data center would be disrupted.


More information about the op1st-dc-alerts mailing list