Skip to main content
Table of Contents

System Changes and News

Last Updated On

This page is a log/list of major changes to systems. This page will be updated frequently.

Future/Current

Grid & Fielding

  • Some nodes continue to be offline due to hardware failures, this is likely to extend into Q3 2026 as we chase ongoing issues.
  • 12 May 2026 – Grid Home Directories for Pheno Users (/mt/home and /mt/data-grid) will be offline for up to 4 hours to allow for essential upgrades.
  • 2026 – Some uplink network disruption to allow for better network security.

IPPP & CfAI

  • 2026 (H1) – Replace core networking for IPPP and CfAI – Possible uplift for 2x100Gb/s.
  • 2026 – Data Transfer SSH system to be deployed.
  • 2026 – Plans to implement 2FA on all external SSH connections.

IPPP Only

  • Maple is currently having issues loading on a GUI (xmaple), the command line (maple) still works (March 2026). – We are updating maple to 2025 on the evening of 27th April 2026.

Fielding Only

Nothing.

CfAI Only

  • There may be brief disruption in the future (notice will be given) for some outstanding power infrastructure works.
  • 2026 – Replace core networking for CfAI – Possible uplift for 2x100Gb/s (Currently under review)

Past

2026

April

  • CE1, CE2, CE3 and CE4 were migrated to Nordugrid ARC CE 7 and is now REST submission only.
  • Mathematica site license expired for the IPPP systems on Sunday 19th April and was renewed on the Monday 20th, causing ~36hours of downtime.

March

  • CE-Test was migrated to Nordugrid ARC CE 7 and is now REST submission only.
  • IPPP First ARM WS Node was commissioned – ip3-wsa1 – See WS Nodes.

Feburary

  • IPPP virtual infrastructure was upgraded to new hardware providing more stability, performance and capacity as well as ready for other planned future upgrades.
  • Fielding group was fully migrated to provide a complete working platform on the Grid HPC away from the old hybrid IPPP/GridPP methods (Please note some documentation may be outdated).
  • GridPP local firewall changes to tighten access rules ensuring interoperability with more security.

January

  • HEPForge has undergone some major system upgrades to bring the core infrastructure up-to-date; with more improvements to come during 2026.
  • Applied anti-AI technology to HEPForge to reduce service disruption caused by an distributed AI scraper effectively causing a DDOS.
  • X2Go was retired.

2025

November

  • Fielding Grid home storage migrations.
  • Grid CEs now fully support ARC Rest submission ready for ARC7 in 2026.

October

  • Some minor network disruption for CfAI to migrate required devices off legacy devices to ensure cyber-security.
  • Seafile updated.

September

  • A partial network upgrade for Physics Server Room (IPPP, CfAI, Fielding) to provide redundant 200Gb/s (2x100Gb/s) uplinks from a redundant 160Gb/s (4x 40Gb/s), with each major computing resource (IPPP, CfAI, Astro, etc) having redundant 80Gb/s (2x 40Gb/s) to better secure future requirements.
  • Seafile updated.

July

  • There was a partial shutdown of IPPP resources due to Data Centre cooling issues.
  • The grid was offline due to Data Centre cooling issues.

June

  • AI Bot mitigations were put into place for HEPForge to reduce the DDOS effect they were having.
  • 5 new grid nodes (A9534) were brought online.

May

  • IPPP Remote OnDemand was put into production.
  • IPPP Home storage was replaced.
  • IPPP Batch was upgraded.

April

  • IPPP Desktops were upgraded to Fedora Linux 42.

March

February

  • CfAI Batch storage was brought online.

2024

December

  • A new private WS was brought online
  • WS10 and WS11 were taken offline.

August

  • Remaining GPUs migrated to Rocky 9
  • Remaining WS migrated to Rocky 9
  • CfAI-WS10 brought into test.
  • CfAI Storage migrated to permanent system.

July

  • GPU10, GPU11, GPU12, GPU13 have been reduced to a single GPU system called GPU10.
  • Deployed Environment Modules for gcc-toolset packages.
  • Initial deployment of CfAI home storage is online.

June

  • WS1 and WS2 are now Rocky Linux 9 (EL9).
  • Login is now Rocky Linux 9 (EL9).
  • CPU Batch Queue is no Rocky Linux 9 (EL9).
  • GCC12 and GCC 13 deployed on Rocky 9 WS/Login/CPU Systems. – See usage guide here.
  • CE3 and CE4 had major configuration changes with brief downtime to resolve a ~2 year old bug.
  • CfAI-WS3 brought online.
  • CfAI had brief outage while power infrastructure works were undertaken.

May

  • WS downtime on the 17th and 28th for power infrastructure works.
  • 28th – All WS redundant networking was repaired.
  • 23rd – Login2 is now Rocky Linux 9 (EL9).
  • 23rd – WS5 is now back to staff only and Rocky Linux 9 (EL9).
  • 22nd – WS3 and WS4 are now Rocky Linux 9 (EL9).
  • 14th – WS6 and WS7 are now Rocky Linux 9 (EL9).
  • 2nd – Fielding ‘Long’ Queue is Rocky Linux 9 (EL9) nodes only.
  • 2nd – CE3 is Rocky Linux 9 (EL9) nodes only.
  • 8th – CE2 (Fielding Default) is Rocky Linux 9 (EL9) nodes only.
  • GridUI1s now Rocky Linux 9 (EL9)

April

  • Downtime for power infrastructure work within Physics
  • Downtime for some nodes to enable upgrades to Rocky Linux 9 (EL9)
  • 25th – GridUI3 was upgraded to Rocky Linux 9 (EL9).
  • 24th – CE4 Queue is Rocky Linux 9 (EL9) nodes only.
  • 29th – GridUI2 is now Rocky Linux 9 (EL9).
  • 29th – CE1 Queue is Rocky Linux 9 (EL9) nodes only.)

March

  • Downtime for power infrastructure work within Physics.

February

  • Downtime of Grid test queue to enable upgrades to Rocky Linux 9
  • Downtime of vn0 to upgrade to Rocky Linux 9

January

  • The IPPP GPU 2,3 and 4 systems have had their memory upgraded to double the previous capacity.
  • Brief downtime of CE-TEST, CE1, CE2, CE3 and CE4 to allow for software upgrades.
  • WS5 was upgraded to Rocky 9 as a test deployment.

2023

December

  • The new XGW storage system is currently under going testing after reports of missing data.

November

  • 25 – 27th – Complete downtime for the Grid and some IPPP Systems.
  • SE01 (gsiftp, xrootd, webdavs) was retired, please use the new XGW system – See Grid Storage.

October

  • 4 new grid nodes were brought online, giving an additional 700+ cores.

August

  • Complete downtime for Grid and various IPPP resources.

June

  • Gitlab was offline for 48hours during a system migration.
  • IPPP monitor deployment was 98% completed.

May

April

  • 15 new grid nodes were brought online, replacing 2300 cores and gaining an additional 500 cores.

March

  • WS 3,4,6,12 were replaced.
  • CfAI WS Systems were brought online.

Feb

  • CPU 1,2,3,4,5 were replaced with newer systems (Approx 500 cores to 1200 with a better power efficiency per core – 3.5W down to 1.9W.)
  • CPU 6+7 were retired.
  • WS 1,2,5 were replaced
  • WS 7,13,14,15,16 were brought online.
  • Rollout started for the replacement of all IPPP monitors.
  • Grid Storage was reduced to enable migration to a new system ready for DPM retirement.

Jan

  • GridUI1 failures were investigated and resolved.
  • GridUI1 was replaced with an upgraded system.
  • Scratch SSD was transferred from GridUI2 to Gridui2

2022

  • Upgraded the IPPP networking from 10Gb/s to a redundant 80Gb/s
  • Upgraded the Grid Networking from 10Gb/s to a redundant 40Gb/s
Created On
Last Updated On
byAdam
Tags: