Service Alerts

yellow - pre-downtime, calls to action

Problem on Mysql.dipr.partners.org

April 22, 2019 11:04 am - 11:20 pm

Hi, We have a problem on mysql.dipr.partners.org and need to restart the node to get it online again. We are restarting mysql.dipr.partners.org

green - services online

Completed Emergency Postgres Maintenance

April 20, 2019 9:00 am - 10:15 am

The ERIS Infrastructure Team completed the emergency Postgres maintenance and all Postgres databases are online and available.

green - services online

MySQL Hosted Database Services DownTime on 30 Mar 2019

March 30, 2019 7:00 am - 12:00 pm

The ERIS Infrastructure team completed the on the MySQL hosted database Service on March 30, 2019. Please forward any questions to rcc@partners.org.

green - services online

Research Computing Website Downtime 30 Mar 2019

March 30, 2019 7:00 am - 10:00 am

This website may have been offline for a brief period during the maintenance window for MySQL Server patching on Saturday, March 30 between 7:00a.m. - 10:00a.m. EDT. This website, https://rc.partners.org, may have been unavailable for a few minutes while the system is rebooted to apply patches.

Please contact rcwebmaster@partners.org if you have any questions or concerns

green - services online

List Manager (Lyris): Expected Downtime on 14 April 2019

April 14, 2019 9:00 pm - 10:00 pm

We are upgrading the List Manager (Lyris) system to v12.3.5 on Sunday, April 14, 2019, at 9 PM. We expect this release to take, up to an hour, so please expect the List Manager (Lyris) service to be offline from 9:00 to 10:00 PM. If you have any questions or concerns, relating to this upgrade, please email: rcc@partners.org

green - services online

Research Computing Ticketing System 22 Mar 2019

March 21, 2019 10:04 pm - March 22, 2019 12:30 pm

Research Computing Ticketing System Online

The MySQL Hosted Database Services outage impacted the internal ticketing system (Kayako) used by ERIS. The outage started Thursday, March 21 around 10p.m. EDT and services restored around 12:15p.m. EDT on Friday, March 22.

Affected Services
Emails and requests sent to the following addresses were not accessed during this unanticipated downtime:
edcsupport@partners.org - REDCap-generated requests to approve, copy or move projects to production have not been received.
hpcsupport@partners.org
rcc@partners.org 
ideasupport@partners.org

Please allow our staff time to catch up on the tickets submitted while the system was offline.

green - services online

Research Computing Core Service Catalog 21 Mar 2019

March 21, 2019 10:40 pm - March 22, 2019 12:30 pm

RC Services Online

The outage started Thursday, March 21 and we expect services to be restored around 12:30p.m. EDT on Friday, March 22. This downtime is related to the MySQL Hosted Database Services outage.

We will update this message when service is restored. 

yellow - pre-downtime, calls to action

MySQL Hosted Databases Restored 22 Mar 2019

March 21, 2019 10:58 pm - March 22, 2019 12:30 pm

MySQL Hosted Database Services Online

The outage started Thursday, March 21 around 10p.m. EDT and was restored around 12:00p.m. EDT on Friday, March 22.

Action Required
If you are the owner of a system on mysql.dipr.partners.org, we strongly encourage you to check your application connection and reboot if necessary.

Affected Services
All systems hosted on mysql.dipr.partners.org
Research Computing ticketing system (Kayako tickets.partners.org)
Research Computing ordering/request system (RC Services rcservices.partners.org)

We will update this posting once service has been restored. Please contact rcc@partners.org with any questions or concerns.

green - services online

REDCap Scheduled Downtime 19 Mar 2019

March 18, 2019 4:21 pm

REDCap (https://redcap.partners.org/redcap) upgrades to v8.5.28 scheduled for Tuesday March 19th at 6:30AM EDT are complete REDCap was offline from 6:30AM-6:48AM EDT.

This release includes a number of bug fixes. 

8.5.28 Release Notes

If you have any concerns about REDCap being offline at this time, please email edcsupport@partners.org

green - services online

Emergency Maintenance HPCWIN3

March 6, 2019 5:00 pm - March 11, 2019 9:00 am

The hpcwin3.partners.org Windows analysis server has presented several freeze instances in the last few days. To evaluate the usage and logs we need to take the server out of service for a few days. During this window time we will perform test and review the information of the processes that have been causing the system to crash.  Future actions needed to be taken after for repair or control will be announced to the users. 

Existing remote desktop sessions will be terminated. Please save all open files and exit from your remote desktop session as soon as possible. We apologize for the inconvenience this may cause. 

green - services online

DIPR Hosted VM Reboot 6 Mar 2019

March 6, 2019 1:47 am - 2:47 am

We had to reboot the service. This is an NFS server serving mapped drives fro /nfs1, /nfs2, and /redcap-prod. The fs.dipr.partners.org VM is currently online, however, you may need to remount your NFS mounts this morning. 

green - services online

MySQL Hosted Database Service Reboot 4 Mar 2019

March 4, 2019 8:00 pm - 8:30 pm

We had an emergency issue that impacted mysql2.dipr.partners.org. To resolve this issue we rebooted mysql2.dipr.partners.org at 8:00pm on Monday evening March 04, 2019.

green - services online

PostgreSQL2 Hosted Databases Online 1 Mar 2019

March 1, 2019 9:19 am - March 3, 2019 12:19 pm

pgsql2.research.partners.org is now back online

green - services online

REDCap Service Alert: Scheduled Maintenance

February 22, 2019 10:08 am

REDCap (https://redcap.partners.org/redcap) requires server maintenance which is scheduled for Saturday, February 23rd 2019 from 7:00AM to 9:00AM EST. There will be a short interruption of service during the maintenance window.

If you have any concerns about this scheduled maintenance, please email edcsupport@partners.org

green - services online

rc.partners.org Website Downtime

February 23, 2019 7:00 am - 9:00 am

There was a short interruption of service for the RISC Website during the maintenance window for MySQL Server patching on Saturday, February 23 between 7:00a.m. - 9:00a.m. The RISC Website, https://rc.partners.org, was unavailable for a few minutes while the system is rebooted during that window.

Please contact rcwebmaster@partners.org if you have any questions or concerns

green - services online

MySQL2 Hosted Database Service Maintenance 23 Feb 2019

February 23, 2019 7:00 am - 9:00 am

The ERIS Infrastructure team will perform maintenance on mysql3.research.partners.org and mysql2.dipr.partners.org on Saturday morning February 23, 2019, from 7:00 am - 9:00 am. There will be a short interruption of service during the maintenance window. All databases hosted on mysql3.research.partners.org or mysql2.dipr.partners.org will be offline while we shut down and restart each node. 

Action Required
If you are the owner of a database on mysql3.research.partners.org or mysql2.dipr.partners.org we encourage you to check your applications after the maintenance window.

green - services online

GitLab: Scheduled Downtime 25 Feb 2019

February 25, 2019 7:00 am - 8:00 am

The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.7 to version 11.7.5 and the upgrade is scheduled to take up to 1 hour. During this time, the service will be unavailable between 7 AM to 8 AM on Monday, February 25th. This upgrade will include: New features, critical security patches, and bug fixes

For a complete list of changes, please refer to the GitLab project release notes:

*** This is a major version upgrade of GitLab. As a result, the underlying tools, frameworks, and programming languages that GitLab is built upon have also been updated, most notably the Ruby Rails version. This should only affect functionality in the GitLab user interface. The GitLab API version has not been updated - remains at v4. 

We recommend that you test your processes (in the development system, at this link: http://gitlab-dev.dipr.partners.org - to log in, just use your Partners ID and Password). The Development server is currently at version 11.7.10. No projects or data have been copied over from the production server. You can create new projects in the development environment and conduct tests as necessary. Projects created in the development environment may be removed without notice (as we conduct future GitLab version testing). Please complete any necessary testing before the upgrade on Monday, February 25th at 7 AM
 
If you have any other questions or concerns, relating to this scheduled downtime, please email: rcc@partners.org

green - services online

GitLab: Scheduled Downtime 4 Feb 2019

February 4, 2019 7:00 am - 8:00 am

Partners HealthCare GitLab: Scheduled Downtime on 2/04/2019

The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.6 to version 11.5.7. The upgrade is scheduled to take up to 1 hour. During this time, the service will be unavailable between 7 AM to 8 AM on Monday, February 4th.

This upgrade will include:

  • Critical security patches

For a complete list of changes, please refer to the GitLab project release notes:

 

** Additionally, we want to reach out to the users of GitLab, to complete any necessary testing before the upgrade on Monday, February 4th at 7 AM. Please complete all your testing at this link: http://gitlab-dev.dipr.partners.org (** To login, just use your Partners ID and Password). The Development server is currently at version 11.5.7. Note – no projects or data have been copied over from the production server. You can create new projects in the development environment and conduct tests as necessary. Projects created in the development environment may be removed without notice (as we conduct future GitLab version testing).

 

If you any questions or concerns relating to this scheduled downtime, please email: rcc@partners.org

green - services online

MySQL2 Hosted Databases Issue 16 Jan 2019

January 16, 2019 1:25 am - 9:25 am
We had a problem with mysql2.dipr.partners.org last night. The backup process had an issue and spawned too many processes which increased the load to the point where no one could connect. We killed all the spawned processes and load on mysql2.dipr.partners.org is back in the normal range.
 
Sorry for the inconvenience. 
grey - no status, general alert

Confluence Collaboration Site Issue 16 Jan 2019

January 16, 2019 8:23 am - 11:23 am

UPDATE: 09:45AM: The underlying issue has been corrected and Confluence is back on line.  We apologize for the inconvenience

UPDATE 08:45AM: The issue has been identified and ERIS technical teams are now working to resolve the issue.  ETA is ~11am EST

There is an issue with the infrastructure upon which Confluence relies causing Confluence to be unavailable.   ERIS technical teams are working to identify and fix the issue.

green - services online

Mysql2.dipr.partners.org

January 14, 2019 6:00 am - 10:00 pm

mysql2.dipr.partners.org is up and all the databases are online and available. We experienced a hardware failure on mysql2.dipr.partners.org detected on Monday, January 14, 2019. 

Applies to: All databases and applications connecting to mysql2.dipr.partners.org

Action Required

If you are a database/application owner using mysql2.dipr.partners.org, we ask that you check your database/application and let us know if you run into any issues.

Please contact rcc@partners.org if you have any questions or concerns.

green - services online

List Manager Service Possible Impact

January 14, 2019 7:30 am

The enterprise is experiencing an issue with the Mail Gateway. Email sent via the List Manager service may not sent.

This is related to the Email Issue that was communicated on the morning of Monday, January 14 by IS Communications.

green - services online

Confluence/JIRA Issue 14 Jan 2019

January 14, 2019 8:19 am - 11:19 am

UPDATE 22:15 EST: Confluence is back online and available.  

UPDATE 16:15 EST: JIRA is now available.  The JIRA database was restored from backup to a different mySQL server.  Any issues created between 8:00pm 11-JAN-2019 and 8:00pm 12-JAN-2019 must be recreated.  We apologize for the inconvenience

UPDATE 9:44 EST: Confluence and JIRA are both still down and unavailable.  The issue has been traced back to an underlying database server problem [mysql2].  ERIS Technical teams are working to correct the database issue.

green - services online

GitLab: Scheduled Upgrade 14 Jan 2019

January 14, 2019 7:00 am - 8:00 am

 

The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.5 to version 11.5.6. The upgrade is scheduled to take up to 1 hour.

During this time, the service will be unavailable between 7 AM to 8 AM on Monday, January 14th.

 

This upgrade will include:

  • Critical security patches

 

For a complete list of changes, please refer to the GitLab project release notes:

 

Please email rcc@partners.org with any questions or concerns.

 

yellow - pre-downtime, calls to action

RESCHEDULED: Confluence / JIRA Maintenance

January 27, 2019 7:00 am - 11:00 am

RESCHEDULED: Confluence and JIRA will be undergoing scheduled maintenance on Sunday, January 27th 13th from 7am-11am and will not be available during this time

Pages