Resolved -
We are happy to report that as of 11 PM MST our support is once again operational and all systems restored. If you are still seeing an issue please reach out to our support team for assistance.
Aug 13, 23:10 MST
Update -
We are happy to report that as of 6:23 pm MST, phoenixNAP services based in the RadiusDC PHX-01 (Phoenix, AZ) Data Center are now back online, including Bare Metal Cloud, Bare Metal Servers, Data Security Cloud and other infrastructure services. This follows an elevated white space temperature issue in the facility, caused by multiple utility bump events which affected the facility's chillers and can be attributed to storms in the Phoenix, Arizona area earlier today.
RadiusDC technicians and their vendors worked throughout today to resolve the situation, restore cooling to their facility customers, and are working to prevent this issue from taking place in the future.
Upon receipt of a formal RFO from RadiusDC, we will distribute a phoenixNAP RFO with further details to our customer base in response to today's outage.
We apologize to our customers and their users for the inconvenience this disruption has caused, and we want to assure you that we are working with our upstream providers to improve and prevent these unexpected disruptions in the future.
Our support portal remains down following this incident. We will post an update to https://status.phoenixnap.com/ once the support portal is fully operational. For any other support issues, you can still reach our support team at 1-855-330-1509.
Thank you,
phoenixNAP Management
Aug 13, 19:30 MST
Monitoring -
We are happy to report that as of 19:03 pm MST, phoenixNAP services based in the RadiusDC PHX-01 (Phoenix, AZ) Data Center are now back online. This follows an elevated white space temperature issue in the facility, caused by multiple utility bump events which affected the facility's chillers and can be attributed to storms in the Phoenix, Arizona area earlier today.
RadiusDC technicians and their vendors worked throughout today to resolve the situation, restore cooling to their facility customers, and are working to prevent this issue from taking place in the future.
Upon receipt of a formal RFO from RadiusDC, we will distribute a phoenixNAP RFO with further details to our customer base in response to today's outage.
We apologize to our customers and their users for the inconvenience this disruption has caused, and we want to assure you that we are working with our upstream providers to improve and prevent these unexpected disruptions in the future.
Thank you,
phoenixNAP Management
Aug 13, 19:02 MST
Update -
Our onsite teams continue to actively work through the affected systems as restoration efforts remain underway. Several services are now partially available, while our teams continue working to restore full functionality.
We are closely monitoring the environment and continuing to address affected systems as part of the ongoing restoration process. We will continue to provide updates on this page as the situation develops.
Our support portal and email ticket support remain offline at this time. For any other support issues, you can still reach our support team at 1-855-330-1509.
Thank you for your continued patience and understanding as we work to restore services.
Aug 13, 19:00 MST
Update -
Our onsite teams remain actively engaged in the restoration effort and continue working through the affected systems. Some services have been restored, and our teams are continuing to work to bring additional services back online.
We are continuing to monitor the environment closely and address systems as they are identified. We will provide further updates on this page as the restoration effort progresses.
Our support portal and email ticket support remain offline at this time. For any other support issues, you can still reach our support team at 1-855-330-1509.
Thank you for your continued patience and understanding while we work through this issue.
Aug 13, 18:31 MST
Update -
Our onsite teams continue working through the environment, physically checking devices and powering up systems as part of the restoration process. Some services have been restored, while work continues across the remaining systems.
Our teams remain actively engaged and are continuing to work through the restoration process. We are closely monitoring the environment and will continue providing updates here as services are restored.
Our support portal and email ticket support remain offline at this time. For any other support issues, you can still reach our support team at 1-855-330-1509.
Thank you for your continued patience and understanding as we work to restore services.
Aug 13, 18:04 MST
Update -
Our onsite teams remain all hands on deck, physically checking devices and powering up those that are offline. Some services have been restored, and our teams are continuing to work through the remaining systems as we work toward full restoration.
We thank you for your continued patience during this process. At present, our support portal and email ticket support remain offline. We will continue posting updates to this page as events progress.
For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 17:30 MST
Update -
Our onsite teams are all hands on deck physically checking devices and powering up offline devices. We thank you for your patience as we work to restore services. Services are beginning to come back online, and some will have already been restored.
At present, our support portal and email ticket support is still offline, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 17:05 MST
Update -
We are continuing to work with our Phoenix node site provider RadiusDC and temperatures are trending toward restoration. We are still working on restoring core support equipment and we have all hands on deck physically checking devices and powering up offline devices. We thank you for your patience as we work to restore services. Services are beginning to come back online, and some will have already been restored.
At present, our support portal and email ticket support is still offline, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 16:25 MST
Update -
Our teams are continuing to work with our Phoenix node site provider RadiusDC and since the third chiller was restored temperatures are stabilizing again. We are working on restoring core support equipment and we have all hands on deck physically checking devices and powering up offline devices to restore services.
At present, our support portal is still offline, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 15:44 MST
Update -
Our teams are continuing to work with our Phoenix node site provider RadiusDC and they are reporting that the third chiller has been restored and temperatures are starting to stabilize again. We are continuing to work with them on restoration of services.
Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 15:05 MST
Update -
Our datacenter provider, RadiusDC is continuing to work with on site vendors to restore temperatures to normal in Phoenix node. We continue to work with them to bring systems back online when possible. For those who missed the previous update, it is duplicated here for visibility:
Our datacenter provider has provided an update on the situation:
Dear Customers,
As we noted in our communication at approximately 7:30 am local time, PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. RadiusDC has on-site vendors working rapidly to resolve the situation and restore adequate cooling. RadiusDC technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional Radius DC remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):
Chiller C restoration (targeted 12:30 pm)
Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)
Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.
We will provide future communications in 20-minute intervals to advise status of the remaining remediation efforts.
Sincerely,
RadiusDC
----
Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 13:39 MST
Update -
Our datacenter provider, RadiusDC is continuing to work with on site vendors to restore temperatures to normal in Phoenix node and temperatures are continuing to fall. We continue to work with them to bring systems back online when possible. For those who missed the previous update, it is duplicated here for visibility:
Our datacenter provider has provided an update on the situation:
Dear Customers,
As we noted in our communication at approximately 7:30 am local time, PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. RadiusDC has on-site vendors working rapidly to resolve the situation and restore adequate cooling. RadiusDC technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional Radius DC remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):
Chiller C restoration (targeted 12:30 pm)
Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)
Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.
We will provide future communications in 20-minute intervals to advise status of the remaining remediation efforts.
Sincerely,
RadiusDC
----
Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 13:05 MST
Update -
Our datacenter provider, RadiusDC is continuing to work with on site vendors to restore temperatures to normal in Phoenix node. We continue to work with them to bring systems back online when possible. For those who missed the previous update, it is duplicated here for visibility:
Our datacenter provider has provided an update on the situation:
Dear Customers,
As we noted in our communication at approximately 7:30 am local time, PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. RadiusDC has on-site vendors working rapidly to resolve the situation and restore adequate cooling. RadiusDC technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional Radius DC remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):
Chiller C restoration (targeted 12:30 pm)
Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)
Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.
We will provide future communications in 20-minute intervals to advise status of the remaining remediation efforts.
Sincerely,
RadiusDC
----
Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 11:59 MST
Update -
As previously mentioned, our datacenter provider, RadiusDC is continuing to work with on site vendors to resolve the cooling issues in Phoenix node. We continue to work with them to bring systems back online when possible. For those who missed the previous update, it is duplicated here for visibility:
Our datacenter provider has provided an update on the situation:
Dear Customers,
As we noted in our communication at approximately 7:30 am local time, PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. RadiusDC has on-site vendors working rapidly to resolve the situation and restore adequate cooling. RadiusDC technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional Radius DC remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):
Chiller C restoration (targeted 12:30 pm)
Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)
Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.
We will provide future communications in 20-minute intervals to advise status of the remaining remediation efforts.
Sincerely,
RadiusDC
----
Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 11:17 MST
Update -
As previously mentioned, our datacenter provider, RadiusDC is continuing to work with on site vendors to resolve the cooling issues in Phoenix node. We continue to work with them to bring systems back online when possible. For those who missed the previous update, it is duplicated here for visibility:
Our datacenter provider has provided an update on the situation:
Dear Customers,
As we noted in our communication at approximately 7:30 am local time, PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. RadiusDC has on-site vendors working rapidly to resolve the situation and restore adequate cooling. RadiusDC technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional Radius DC remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):
Chiller C restoration (targeted 12:30 pm)
Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)
Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.
We will provide future communications in 20-minute intervals to advise status of the remaining remediation efforts.
Sincerely,
RadiusDC
----
Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 10:35 MST
Update -
Our datacenter provider has provided an update on the situation:
Dear Customers,
As we noted in our communication at approximately 7:30 am local time, PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. RadiusDC has on-site vendors working rapidly to resolve the situation and restore adequate cooling. RadiusDC technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional Radius DC remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):
Chiller C restoration (targeted 12:30 pm)
Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)
Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.
We will provide future communications in 20-minute intervals to advise status of the remaining remediation efforts.
Sincerely,
RadiusDC
----
Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 09:58 MST
Update -
All hands are on deck working with our datacenter provider in our Phoenix node. Currently, the datacenter provider has on-site vendors working to resolve the situation and restore adequate cooling capacity. Some capacity appears to have been restored. Their datacenter technicians have established mitigation efforts to reduce rising temperatures. At present, we have no eta on full resolution. Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other support issues, you can still reach our support team at 1-855-330-1509.
Aug 13, 09:36 MST
Update -
Our teams continue to work with our datacenter provider at the Phoenix node. At present, our datacenter provider has on-site vendors working rapidly to resolve the situation and restore adequate cooling. The datacenter technicians have activated mitigation efforts to reduce rising space temperatures. We still have no eta on full resolution. Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other issues, you can still reach our support team at 1-855-330-1509
Aug 13, 09:04 MST
Update -
We continue to work with our datacenter provider at the Phoenix node. At present, the vendor is onsite and is still in the process of repairing the chillers. We still have no eta on full resolution. Our support portal remains down during this incident, however, we will continue posting updates to this page as events progress. For any other issues, you can still reach our support team at 1-855-330-1509
Aug 13, 08:31 MST
Update -
No updates at this time. Our facilities team is actively working with the vendor, who is currently onsite assisting with the issue. There is no ETA for resolution at this time.
Aug 13, 07:21 MST
Update -
The Support Portal(https://support.phoenixnap.com) is affected by this incident, and you will not be able to submit a ticket.
Aug 13, 06:21 MST
Update -
No updates at this time. Our facilities team is actively working with the vendor, who is currently onsite assisting with the issue. There is no ETA for resolution at this time.
Aug 13, 06:04 MST
Update -
We are continuing to work on a fix for this issue.
Aug 13, 05:36 MST
Identified -
Some equipment might be rebooting during this issue with the temperature.
Please for some patience while this is resolved.
Aug 13, 04:28 MST
Investigating -
It iis noticed higher ambient temperature in th ePhoenix DC.
Our teams are working on resolving this and it should be back to normal soon.
Neither one service is critical but some alarms may occur during this.
Aug 13, 03:28 MST