We have also noticed that this issue has occasionally affected compute nodes, so far on savio4_gpu and savio4_htc. If you have a problem accessing scratch on a login node or a compute node, please let us know (including what node you had the problem on) so we can reboot the node in question.
We are in the process of scheduling a downtime to fix the problem.
Posted Oct 07, 2026 - 14:08 PDT
Update
We are continuing to monitor for any further issues.
Posted Sep 16, 2026 - 08:02 PDT
Monitoring
For the past week or so, on a sporadic basis, we have noticed that a login node is not able to access one of the disks that provide the /global/scratch (Lustre) filesystem. When this happens one may see an error message, "Cannot send after transport endpoint shutdown", when running a command that tries to access data on the inaccessible disk.
This is caused by a Lustre bug. A fix is available and will be rolled out when we are able to take a short downtime in coming weeks.
We are monitoring the nodes and rebooting as needed as a short-term work-around, but if you have a problem accessing scratch, please let us know (including what node you had the problem on) so we can reboot the node in question.