MAD Data Storage: Gateway Reboot Completed
The reboot of all 3 gateways has completed successfully and it appears as though normal behavior has returned. Please test your normal use cases to ensure that any problems you may have been experiencing are no longer occurring.
MAD engineering will continue to search for root cause to these latest symptoms and for the original problem from Wednesday, but it does appear that any lingering issues resulting from the remount of storage last week are now ceased. We will look to more closely identify what was wrong so as to avoid a full reboot in the future if at all possible.
We understand there are a few outstanding issues that are in the queue for completion, and with the urgent issues now cleared, MAD engineering will address those open items this afternoon and complete them as soon as possible.
Thank you very much for your patience while we sorted this out!
As you may be aware, the MAD infrastructure encountered issues last Wednesday for customers using non-replicated accounts (see below). We believe we isolated and corrected that issue, but there have been lingering symptoms stemming from correction of this issue that we cannot keep cleared.
We rebooted the gateway servers during the lunch hour on Tuesday, October 6, 2015, as many customers were not able to write to their respective shares.
We advised that if you had any current jobs running against your MAD shares, to please suspend them.
Thanks for your patience while we alleviate this problem!
From Wednesday, September 30, 2015:
Nonreplicated shares on the MAD (Massive Array of Disks) data storage service were reported unavailable yesterday afternoon. Service was fully restored on Thursday, October 1 at approximately 11a.m. EDT. Check any transfers that ran from Wednesday, September 30, 2015 afternoon to the morning of Thursday, October 1, and restart the transfer if necessary.
ERIS technicians are actively exploring the cause of the issue. If you experience any issues with the MAD data storage service, please contact firstname.lastname@example.org.