rdc@fri - Zgodovina obvestil

Storage Izkušnje z zmanjšano zmogljivostjo

Docs - Deluje

100% - razpoložljivost
apr. 2025 · 100.0%maj · 100.0%jun. · 100.0%

Previous page

Next page

apr. 2025
maj 2025
jun. 2025
94% - razpoložljivost

Login - Deluje

99% - razpoložljivost
apr. 2025 · 100.0%maj · 100.0%jun. · 96.76%

Previous page

Next page

apr. 2025
maj 2025
jun. 2025

Storage - Poslabšano delovanje

99% - razpoložljivost
apr. 2025 · 100.0%maj · 100.0%jun. · 96.76%

Previous page

Next page

apr. 2025
maj 2025
jun. 2025

Compute - Deluje

84% - razpoložljivost
apr. 2025 · 95.07%maj · 70.26%jun. · 88.58%

Previous page

Next page

apr. 2025
maj 2025
jun. 2025

Zgodovina obvestil

jun. 2025

Storage maintenance
  • Zaključeno
    junij 18, 2025 ob 13:00:00
    Zaključeno
    junij 18, 2025 ob 13:00:00
    Vzdrževanje je uspešno zaključeno
  • V teku
    junij 18, 2025 ob 07:00:01
    V teku
    junij 18, 2025 ob 07:00:01
    Vzdrževanje je v teku
  • Napovedano
    junij 18, 2025 ob 07:00:00
    Napovedano
    junij 18, 2025 ob 07:00:00

    We are planning an upgrade of the storage software; the process should go largely undetected, but in case of issues we may be forced to cancel active jobs.

    We appreciate your patience.

RDC cooling malfunction
  • Odpravljeno
    Odpravljeno

    FRIDA was successfully relocated to a new facility.

  • Spremljanje
    Spremljanje

    The evaporator of one of the cooling units has been successfully repaired; some repairs will need to be performed at a later time. We're bringing the cluster back online, but will continue monitoring the cooling behaviour.

    We appreciate your patience.

  • Napaka odkrita
    Napaka odkrita

    Due to cooling malfunction we need to stop all operations. We're actively working on a fix.

    We appreciate your patience.

  • Nadgradnja
    Nadgradnja

    The maintenance of the RDC cooling has finished (partially). Some procedures are more extensive and will need to be performed in the upcoming week. We'll keep the operation of FRIDA under close monitoring and try to keep disruptive actions to the minimum possible.

    We appreciate your patience.

  • Nadgradnja
    Nadgradnja

    Due to continual cooling malfunction we're forced to cancel all jobs and shut down the cluster. A larger maintenance of the RDC cooling that will hopefully resolve the issues is planned for tomorrow morning.

    We appreciate your patience and we'll keep you posted of the progress.

  • Spremljanje
    Spremljanje

    The RDC cooling has returned to normal values, and the cluster has been put back into operation. We will be monitoring the cooling. We are working on resolving the issue. Note that in case of high external temperatures we may be forced to temporarily pause the cluster or cancel jobs to prevent additional fallouts.

    We appreciate your patience.

  • Napaka odkrita
    Napaka odkrita
    We are continuing to work on a fix for this incident.
  • Raziskovanje
    Raziskovanje

    The RDC cooling is experiencing malfunction. As a preventative measure we're forced to shutdown the cluster; all running jobs were canceled. The cluster has been temporarily put into maintenance, with a planned release after 22:00 or once the datacenter's cooling resumes normal operation and temperatures reach acceptable levels. Due to high external temperatures and regular cooling malfunctions job cancelations of the most demanding jobs may become more frequent. We therefore advise to run demanding jobs overnight and perform regular and frequent checkpoints. Before and when running jobs please consult the dashboard “RDC heat load”.

    We appreciate your patience.

RDC cooling malfunction
  • Odpravljeno
    Odpravljeno

    We implemented a fix, we're bringing the cluster back into operation. We'll be monitoring the cooling. Please make sure to perform regular checkpoints to avoid data loss in case of additional shutdowns.

    We appreciate your patience.

  • Napaka odkrita
    Napaka odkrita

    The RDC cooling is experiencing malfunction. As a preventative measure we're forced to shutdown the cluster; all jobs will be canceled. While working to fix the issue we'll also perform the scheduled FRIDA maintenance to keep the downtime at minimum.

    We appreciate your patience.

maj 2025

Major malfunction of one of the nodes
  • Odpravljeno
    Odpravljeno

    GPU0 on node ixh has been successfully replaced and the node is back in production. Please, benchmark your runs against earlier ones and report any discrepancies.

    Thank you for your patience.

  • Napaka odkrita
    Napaka odkrita

    Vozlišče ixh ne deluje zaradi pregrevanja grafične kartice 0. S podporo iščemo rešitev.

    Hvala za vašo potrpežljivost.

  • Spremljanje
    Spremljanje

    We are reinstating the node and we'll monitor the status.

  • Raziskovanje
    Raziskovanje
    We are currently investigating this incident.

apr. 2025

Regular maintenance
  • Nadgradnja
    maj 06, 2025 ob 20:59:12
    Nadgradnja
    maj 06, 2025 ob 20:59:12

    Maintenance has completed successfully. We performed the full set of updates & upgrades on all cluster nodes. The storage cluster has also undergone the full set of updates & upgrades.

    Most of the cluster is operational and ready to accept jobs. One node is currently kept in maintenance mode due to HW issues that require physical maintenance, it should resume operation in the next few days.

  • Zaključeno
    maj 06, 2025 ob 20:00:00
    Zaključeno
    maj 06, 2025 ob 20:00:00
    Vzdrževanje je uspešno zaključeno
  • V teku
    april 30, 2025 ob 20:00:01
    V teku
    april 30, 2025 ob 20:00:01
    Vzdrževanje je v teku
  • Napovedano
    april 30, 2025 ob 20:00:00
    Napovedano
    april 30, 2025 ob 20:00:00
    We are planning for a scheduled maintenance during that time.
Login is down
  • Odpravljeno
    Odpravljeno
  • Raziskovanje
    Raziskovanje
Cluster outage
  • Odpravljeno
    Odpravljeno
    This incident has been resolved.
  • Raziskovanje
    Raziskovanje

    Cluster cannot be accessed at the moment. This incident was created by an automated monitoring service.

RDC cooling malfunction
  • Odpravljeno
    Odpravljeno

    The RDC cooling was partially fixed, we're bringing the cluster back to production. We'll be monitoring the status. During the week the remaining RDC cooling issues will be resolved.

    We appreciate your patience

  • Napaka odkrita
    Napaka odkrita

    The RDC cooling is experiencing malfunction. As a preventative measure we're forced to shutdown the cluster; all jobs will be canceled.

    We appreciate your patience.

  • Spremljanje
    Spremljanje

    The RDC cooling was fixed, we're currently bringing the cluster back to life, performing the scheduled FRIDA maintenance, and monitoring the status.

  • Napaka odkrita
    Napaka odkrita

    The RDC cooling is experiencing malfunction. As a preventative measure we're forced to shutdown the cluster; all jobs will be canceled. While working to fix the issue we'll also perform the scheduled FRIDA maintenance to keep the downtime at minimum.

    We appreciate your patience.

apr. 2025 do jun. 2025

Naprej