rdc@fri - RDC cooling malfunction – Incident details

Storage experiencing degraded performance

RDC cooling malfunction

Resolved
Major outage
Started 6 months agoLasted 5 months

Affected

Frida

Partial outage from 1:43 PM to 3:13 PM, Operational from 1:43 PM to 3:13 PM, Major outage from 3:13 PM to 2:30 PM, Operational from 2:30 PM to 2:37 PM, Under maintenance from 2:37 PM to 4:52 PM, Operational from 2:37 PM to 4:52 PM, Degraded performance from 4:52 PM to 7:23 AM, Operational from 4:52 PM to 7:23 AM, Major outage from 7:23 AM to 9:05 PM, Operational from 7:23 AM to 9:05 PM

Login

Operational from 1:43 PM to 3:13 PM, Major outage from 3:13 PM to 2:30 PM, Operational from 2:30 PM to 9:05 PM

Storage

Operational from 1:43 PM to 3:13 PM, Major outage from 3:13 PM to 2:30 PM, Operational from 2:30 PM to 9:05 PM

Compute

Partial outage from 1:43 PM to 3:13 PM, Major outage from 3:13 PM to 2:30 PM, Operational from 2:30 PM to 2:37 PM, Under maintenance from 2:37 PM to 4:52 PM, Degraded performance from 4:52 PM to 7:23 AM, Major outage from 7:23 AM to 9:05 PM, Operational from 12:29 PM to 9:05 PM

Updates
  • Resolved
    Resolved

    FRIDA was successfully relocated to a new facility.

  • Monitoring
    Monitoring

    The evaporator of one of the cooling units has been successfully repaired; some repairs will need to be performed at a later time. We're bringing the cluster back online, but will continue monitoring the cooling behaviour.

    We appreciate your patience.

  • Identified
    Identified

    Due to cooling malfunction we need to stop all operations. We're actively working on a fix.

    We appreciate your patience.

  • Update
    Update

    The maintenance of the RDC cooling has finished (partially). Some procedures are more extensive and will need to be performed in the upcoming week. We'll keep the operation of FRIDA under close monitoring and try to keep disruptive actions to the minimum possible.

    We appreciate your patience.

  • Update
    Update

    Due to continual cooling malfunction we're forced to cancel all jobs and shut down the cluster. A larger maintenance of the RDC cooling that will hopefully resolve the issues is planned for tomorrow morning.

    We appreciate your patience and we'll keep you posted of the progress.

  • Monitoring
    Monitoring

    The RDC cooling has returned to normal values, and the cluster has been put back into operation. We will be monitoring the cooling. We are working on resolving the issue. Note that in case of high external temperatures we may be forced to temporarily pause the cluster or cancel jobs to prevent additional fallouts.

    We appreciate your patience.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating

    The RDC cooling is experiencing malfunction. As a preventative measure we're forced to shutdown the cluster; all running jobs were canceled. The cluster has been temporarily put into maintenance, with a planned release after 22:00 or once the datacenter's cooling resumes normal operation and temperatures reach acceptable levels. Due to high external temperatures and regular cooling malfunctions job cancelations of the most demanding jobs may become more frequent. We therefore advise to run demanding jobs overnight and perform regular and frequent checkpoints. Before and when running jobs please consult the dashboard “RDC heat load”.

    We appreciate your patience.