The ERIS Infrastructure team completed the on the MySQL hosted database Service on March 30, 2019. Please forward any questions to rcc@partners.org.
This website may have been offline for a brief period during the maintenance window for MySQL Server patching on Saturday, March 30 between 7:00a.m. - 10:00a.m. EDT. This website, https://rc.partners.org, may have been unavailable for a few minutes while the system is rebooted to apply patches.
Please contact rcwebmaster@partners.org if you have any questions or concerns
We are upgrading the List Manager (Lyris) system to v12.3.5 on Sunday, April 14, 2019, at 9 PM. We expect this release to take, up to an hour, so please expect the List Manager (Lyris) service to be offline from 9:00 to 10:00 PM. If you have any questions or concerns, relating to this upgrade, please email: rcc@partners.org.
Research Computing Ticketing System Online
The MySQL Hosted Database Services outage impacted the internal ticketing system (Kayako) used by ERIS. The outage started Thursday, March 21 around 10p.m. EDT and services restored around 12:15p.m. EDT on Friday, March 22.
Affected Services
Emails and requests sent to the following addresses were not accessed during this unanticipated downtime:
edcsupport@partners.org - REDCap-generated requests to approve, copy or move projects to production have not been received.
hpcsupport@partners.org
rcc@partners.org
ideasupport@partners.org
Please allow our staff time to catch up on the tickets submitted while the system was offline.
RC Services Online
The outage started Thursday, March 21 and we expect services to be restored around 12:30p.m. EDT on Friday, March 22. This downtime is related to the MySQL Hosted Database Services outage.
We will update this message when service is restored.
MySQL Hosted Database Services Online
The outage started Thursday, March 21 around 10p.m. EDT and was restored around 12:00p.m. EDT on Friday, March 22.
Action Required
If you are the owner of a system on mysql.dipr.partners.org, we strongly encourage you to check your application connection and reboot if necessary.
Affected Services
All systems hosted on mysql.dipr.partners.org
Research Computing ticketing system (Kayako tickets.partners.org)
Research Computing ordering/request system (RC Services rcservices.partners.org)
We will update this posting once service has been restored. Please contact rcc@partners.org with any questions or concerns.
REDCap (https://redcap.partners.org/redcap) upgrades to v8.5.28 scheduled for Tuesday March 19th at 6:30AM EDT are complete REDCap was offline from 6:30AM-6:48AM EDT.
This release includes a number of bug fixes.
If you have any concerns about REDCap being offline at this time, please email edcsupport@partners.org
The hpcwin3.partners.org Windows analysis server has presented several freeze instances in the last few days. To evaluate the usage and logs we need to take the server out of service for a few days. During this window time we will perform test and review the information of the processes that have been causing the system to crash. Future actions needed to be taken after for repair or control will be announced to the users.
Existing remote desktop sessions will be terminated. Please save all open files and exit from your remote desktop session as soon as possible. We apologize for the inconvenience this may cause.
We had to reboot the service. This is an NFS server serving mapped drives fro /nfs1, /nfs2, and /redcap-prod. The fs.dipr.partners.org VM is currently online, however, you may need to remount your NFS mounts this morning.
REDCap (https://redcap.partners.org/redcap) requires server maintenance which is scheduled for Saturday, February 23rd 2019 from 7:00AM to 9:00AM EST. There will be a short interruption of service during the maintenance window.
If you have any concerns about this scheduled maintenance, please email edcsupport@partners.org
There was a short interruption of service for the RISC Website during the maintenance window for MySQL Server patching on Saturday, February 23 between 7:00a.m. - 9:00a.m. The RISC Website, https://rc.partners.org, was unavailable for a few minutes while the system is rebooted during that window.
Please contact rcwebmaster@partners.org if you have any questions or concerns
The ERIS Infrastructure team will perform maintenance on mysql3.research.partners.org and mysql2.dipr.partners.org on Saturday morning February 23, 2019, from 7:00 am - 9:00 am. There will be a short interruption of service during the maintenance window. All databases hosted on mysql3.research.partners.org or mysql2.dipr.partners.org will be offline while we shut down and restart each node.
Action Required
If you are the owner of a database on mysql3.research.partners.org or mysql2.dipr.partners.org we encourage you to check your applications after the maintenance window.
The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.7 to version 11.7.5 and the upgrade is scheduled to take up to 1 hour. During this time, the service will be unavailable between 7 AM to 8 AM on Monday, February 25th. This upgrade will include: New features, critical security patches, and bug fixes
For a complete list of changes, please refer to the GitLab project release notes:
*** This is a major version upgrade of GitLab. As a result, the underlying tools, frameworks, and programming languages that GitLab is built upon have also been updated, most notably the Ruby Rails version. This should only affect functionality in the GitLab user interface. The GitLab API version has not been updated - remains at v4.
We recommend that you test your processes (in the development system, at this link: http://gitlab-dev.dipr.partners.org - to log in, just use your Partners ID and Password). The Development server is currently at version 11.7.10. No projects or data have been copied over from the production server. You can create new projects in the development environment and conduct tests as necessary. Projects created in the development environment may be removed without notice (as we conduct future GitLab version testing). Please complete any necessary testing before the upgrade on Monday, February 25th at 7 AM.
If you have any other questions or concerns, relating to this scheduled downtime, please email: rcc@partners.org
Partners HealthCare GitLab: Scheduled Downtime on 2/04/2019
The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.6 to version 11.5.7. The upgrade is scheduled to take up to 1 hour. During this time, the service will be unavailable between 7 AM to 8 AM on Monday, February 4th.
This upgrade will include:
For a complete list of changes, please refer to the GitLab project release notes:
** Additionally, we want to reach out to the users of GitLab, to complete any necessary testing before the upgrade on Monday, February 4th at 7 AM. Please complete all your testing at this link: http://gitlab-dev.dipr.partners.org (** To login, just use your Partners ID and Password). The Development server is currently at version 11.5.7. Note – no projects or data have been copied over from the production server. You can create new projects in the development environment and conduct tests as necessary. Projects created in the development environment may be removed without notice (as we conduct future GitLab version testing).
If you any questions or concerns relating to this scheduled downtime, please email: rcc@partners.org
UPDATE: 09:45AM: The underlying issue has been corrected and Confluence is back on line. We apologize for the inconvenience
UPDATE 08:45AM: The issue has been identified and ERIS technical teams are now working to resolve the issue. ETA is ~11am EST
There is an issue with the infrastructure upon which Confluence relies causing Confluence to be unavailable. ERIS technical teams are working to identify and fix the issue.
mysql2.dipr.partners.org is up and all the databases are online and available. We experienced a hardware failure on mysql2.dipr.partners.org detected on Monday, January 14, 2019.
If you are a database/application owner using mysql2.dipr.partners.org, we ask that you check your database/application and let us know if you run into any issues.
Please contact rcc@partners.org if you have any questions or concerns.
The enterprise is experiencing an issue with the Mail Gateway. Email sent via the List Manager service may not sent.
This is related to the Email Issue that was communicated on the morning of Monday, January 14 by IS Communications.
UPDATE 22:15 EST: Confluence is back online and available.
UPDATE 16:15 EST: JIRA is now available. The JIRA database was restored from backup to a different mySQL server. Any issues created between 8:00pm 11-JAN-2019 and 8:00pm 12-JAN-2019 must be recreated. We apologize for the inconvenience
UPDATE 9:44 EST: Confluence and JIRA are both still down and unavailable. The issue has been traced back to an underlying database server problem [mysql2]. ERIS Technical teams are working to correct the database issue.
The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.5 to version 11.5.6. The upgrade is scheduled to take up to 1 hour.
During this time, the service will be unavailable between 7 AM to 8 AM on Monday, January 14th.
This upgrade will include:
For a complete list of changes, please refer to the GitLab project release notes:
Please email rcc@partners.org with any questions or concerns.
RESCHEDULED: Confluence and JIRA will be undergoing scheduled maintenance on Sunday, January 27th 13th from 7am-11am and will not be available during this time
