Service Alerts

yellow - pre-downtime, calls to action

GitLab: Upcoming Scheduled Downtime 2/25/2019

February 25, 2019 7:00 am - 8:00 am

The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.7 to version 11.7.5 and the upgrade is scheduled to take up to 1 hour. During this time, the service will be unavailable between 7 AM to 8 AM on Monday, February 25th. This upgrade will include: New features, critical security patches, and bug fixes

For a complete list of changes, please refer to the GitLab project release notes:

*** This is a major version upgrade of GitLab. As a result, the underlying tools, frameworks, and programming languages that GitLab is built upon have also been updated, most notably the Ruby Rails version. This should only affect functionality in the GitLab user interface. The GitLab API version has not been updated - remains at v4. 

We recommend that you test your processes (in the development system, at this link: http://gitlab-dev.dipr.partners.org - to log in, just use your Partners ID and Password). The Development server is currently at version 11.7.10. No projects or data have been copied over from the production server. You can create new projects in the development environment and conduct tests as necessary. Projects created in the development environment may be removed without notice (as we conduct future GitLab version testing). Please complete any necessary testing before the upgrade on Monday, February 25th at 7 AM
 
If you have any other questions or concerns, relating to this scheduled downtime, please email: rcc@partners.org

green - services online

GitLab Scheduled Downtime - February 4th, 2019

February 4, 2019 7:00 am - 8:00 am

Partners HealthCare GitLab: Scheduled Downtime on 2/04/2019

The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.6 to version 11.5.7. The upgrade is scheduled to take up to 1 hour. During this time, the service will be unavailable between 7 AM to 8 AM on Monday, February 4th.

This upgrade will include:

  • Critical security patches

For a complete list of changes, please refer to the GitLab project release notes:

 

** Additionally, we want to reach out to the users of GitLab, to complete any necessary testing before the upgrade on Monday, February 4th at 7 AM. Please complete all your testing at this link: http://gitlab-dev.dipr.partners.org (** To login, just use your Partners ID and Password). The Development server is currently at version 11.5.7. Note – no projects or data have been copied over from the production server. You can create new projects in the development environment and conduct tests as necessary. Projects created in the development environment may be removed without notice (as we conduct future GitLab version testing).

 

If you any questions or concerns relating to this scheduled downtime, please email: rcc@partners.org

green - services online

Mysql2.dipr.partners.org Last Night

January 16, 2019 1:25 am - 9:25 am
We had a problem with mysql2.dipr.partners.org last night. The backup process had an issue and spawned too many processes which increased the load to the point where no one could connect. We killed all the spawned processes and load on mysql2.dipr.partners.org is back in the normal range.
 
Sorry for the inconvenience. 
grey - no status, general alert

Confluence is down and unavailable

January 16, 2019 8:23 am - 11:23 am

UPDATE: 09:45AM: The underlying issue has been corrected and Confluence is back on line.  We apologize for the inconvenience

UPDATE 08:45AM: The issue has been identified and ERIS technical teams are now working to resolve the issue.  ETA is ~11am EST

There is an issue with the infrastructure upon which Confluence relies causing Confluence to be unavailable.   ERIS technical teams are working to identify and fix the issue.

green - services online

Mysql2.dipr.partners.org

January 14, 2019 6:00 am - 10:00 pm

mysql2.dipr.partners.org is up and all the databases are online and available. We experienced a hardware failure on mysql2.dipr.partners.org detected on Monday, January 14, 2019. 

Applies to: All databases and applications connecting to mysql2.dipr.partners.org

Action Required

If you are a database/application owner using mysql2.dipr.partners.org, we ask that you check your database/application and let us know if you run into any issues.

Please contact rcc@partners.org if you have any questions or concerns.

green - services online

List Manager Service Possible Impact

January 14, 2019 7:30 am

The enterprise is experiencing an issue with the Mail Gateway. Email sent via the List Manager service may not sent.

This is related to the Email Issue that was communicated on the morning of Monday, January 14 by IS Communications.

green - services online

Confluence/JIRA are both down and unavailable

January 14, 2019 8:19 am - 11:19 am

UPDATE 22:15 EST: Confluence is back online and available.  

UPDATE 16:15 EST: JIRA is now available.  The JIRA database was restored from backup to a different mySQL server.  Any issues created between 8:00pm 11-JAN-2019 and 8:00pm 12-JAN-2019 must be recreated.  We apologize for the inconvenience

UPDATE 9:44 EST: Confluence and JIRA are both still down and unavailable.  The issue has been traced back to an underlying database server problem [mysql2].  ERIS Technical teams are working to correct the database issue.

green - services online

GitLab Scheduled Upgrade January, 14th 2019

January 14, 2019 7:00 am - 8:00 am

 

The GitLab service (https://gitlab.partners.org/) will be upgraded from version 11.5.5 to version 11.5.6. The upgrade is scheduled to take up to 1 hour.

During this time, the service will be unavailable between 7 AM to 8 AM on Monday, January 14th.

 

This upgrade will include:

  • Critical security patches

 

For a complete list of changes, please refer to the GitLab project release notes:

 

Please email rcc@partners.org with any questions or concerns.

 

yellow - pre-downtime, calls to action

RESCHEDULED: Confluence / JIRA Maintenance

January 27, 2019 7:00 am - 11:00 am

RESCHEDULED: Confluence and JIRA will be undergoing scheduled maintenance on Sunday, January 27th 13th from 7am-11am and will not be available during this time

yellow - pre-downtime, calls to action

MobileIron NA1 Cluster Upgrade

December 12, 2018 10:00 pm - December 13, 2018 8:00 am

MobileIron has scheduled an upgrade to the NA1 Cluster used by Partners and thousands of other customers tonight from 10pm through 8am.

The following will be affected during tonight’s MobileIron NA1 Upgrade:

  • You will not be able to enroll new devices
  • Existing MobileIron users will not be able to access the Partners App Catalog

Other functions such as Email, Haiku, Cortext, Receiver, Dropbox, PowerMic, etc are not affected and will continue to work normally

 
yellow - pre-downtime, calls to action

phpMyAdmin Security Advisory

December 12, 2018 12:19 pm

phpMyAdmin is a very popular and widely-used MySQL Database Management system.

A new security advisory was released by the software developers addressing several critical security findings. These are outlined below:

  1. CVE-2018-19968 
    Local file inclusion through transformation feature
    A flaw has been found where an attacker can exploit phpMyAdmin to leak the contents of a local file. The attacker must have access to the phpMyAdmin Configuration Storage tables, although these can easily be created in any database to which the attacker has access. An attacker must have valid credentials to log in to phpMyAdmin; this vulnerability does not allow an attacker to circumvent the login system.
    We consider this vulnerability to be severe.
    https://www.phpmyadmin.net/security/PMASA-2018-6/
     
  2. CVE-2018-19969
    XSRF/CSRF vulnerability in phpMyAdmin
    By deceiving a user to click on a crafted URL, it is possible to perform harmful SQL operations such as renaming databases, creating new tables/routines, deleting designer pages, adding/deleting users, updating user passwords, killing SQL processes, etc.
    We consider this vulnerability to be of moderate severity.
    https://www.phpmyadmin.net/security/PMASA-2018-7/
     
  3. CVE-2018-19970
    XSS vulnerability in navigation tree
    A Cross-Site Scripting vulnerability was found in the navigation tree, where an attacker can deliver a payload to a user through a specially-crafted database/table name.
    We consider this attack to be of moderate severity.
    https://www.phpmyadmin.net/security/PMASA-2018-8/

Please update the software ASAP or take additional appropriate steps to mitigate the possibility of these vulnerabilities being exploited.

 

green - services online

REDCap Service Alert: Scheduled Downtime

December 11, 2018 2:39 pm

REDCap (https://redcap.partners.org/redcap) upgrades to v8.5.24 scheduled for Thursday August 9th at 7:30AM EDT are complete. REDCap was offline from 7:30-8:07AM EST.

 Full release notes can be found here

If you have any concerns about REDCap being offline at this time, please email edcsupport@partners.org


Important: New Method For Composing Survey Invitations!

One of the new features included in the upgrade to v.8.5.19 is the ability to fully customize survey invitations. The standard text and survey link are no longer automatically appended to the text you provide. 

For most users, the transition will be seamless. However, we recommend that users review any projects that utilize Automatic Survey Invitations to ensure their survey text is displaying as expected. 

For groups using the Twilio service, the projects should be reviewed as soon as possible, especially ones using Automated Survey Invitations. You may need to manually add in a reference to [survey-url] or [survey-link] into the text depending on your invitation preference.  

This is how the new feature is detailed in the REDCap release notes:

Improvement/change: New method for composing survey invitation text using Smart Variables for survey link

When composing a survey invitation, the standard text and survey link are no longer automatically appended to the survey invitation text at the time the email is sent. Instead, users must now specify all the entirety of the text of the email (including the stock text and survey link that used to be appended automatically, if they wish) and therefore must supply [survey-url] and/or [survey-link] in the text if they wish to provide the participant with a link to the survey.

If the user forgets to enter the survey URL Smart Variable in the text, REDCap will automatically suggest to them that they should.

If using the Twilio telephony module for sending invitations, the standard instructional text will still be appended in the SMS message as in previous versions EXCEPT for the “Email invitation” and “SMS invitation (contains survey link)” invitation types, which require [survey-url] and/or [survey-link] in the SMS text in order for the participant to receive a survey link.

Note: All survey invitations that were scheduled prior to this upgrade will still have the standard text and survey link appended to their survey text. Additionally, during the upgrade to this version, all saved configurations for Automated Survey Invitations (ASIs) will have the standard text and survey link automatically appended to the saved ASI email text, thus allowing the ASI behavior to remain exactly the same after the upgrade and allowing it to be backward compatible.

green - services online

RISC Website: Unexpected Downtime December 10th, 2018

December 10, 2018 1:10 pm - 2:05 pm

RISC Website (https://rc.partners.org) experienced unexpected downtime today from approximately 1:10 - 2:05 PM EDT. The issues were related to the infrastructure supporting the website. 

If you have any concerns, please email rcwebmaster@partners.org.

green - services online

REDCap Service Alert: Unexpected Downtime December 10th, 2018

December 10, 2018 2:16 pm - 5:16 pm

REDCap (https://redcap.partners.org/redcap) experienced unexpected downtime today from approximately 1:10 - 2:05 PM EDT. The issues were related to the infrastructure supporting REDCap and not related to this morning's upgrade. 

REDCap is back online. 

If you have any concerns, please email edcsupport@partners.org

green - services online

RISC Website Maintenance | rc.partners.org

December 13, 2018 7:02 am - 7:17 am

Research IS & Computing website maintenance was performed on Thursday, December 13 from 7 - 7:15a.m. During this window, the site may have been offline for a few minutes while bug fixes and feature enhancements are implemented.

Please contact rcwebmaster@partners.org if you have any questions or concerns.

green - services online

Partners Healthcare REDCap: Scheduled Downtime

December 6, 2018 10:56 am

REDCap (https://redcap.partners.org/redcap) upgrades to v8.5.19 scheduled for Monday December 10th, 7:00AM EDT are complete. REDCap was offline from 7:00-8:05AM EDT.

This upgrade includes a number of new features and bug fixes. These include:

New feature: Smart Variables: Smart Variables are dynamic variables that can be used in calculated fields, conditional/branching logic, and piping. Similar to using project variable names inside square brackets - e.g., [heart_rate], Smart Variables are also represented inside brackets - e.g., [user-name], [survey-link], [previous-event-name][weight], or [heart_rate][previous-instance]. But instead of pointing to data fields, Smart Variables are context-aware and thus adapt to the current situation. Some can be used with field variables or other Smart Variables, and some are meant to be used as stand-alone. There are many possibilities.
35 Smart Variables are available. They can reference things with regard to users, records, forms, surveys, events/arms, or repeating instances. Documentation and examples for using Smart Variables are included on the Project Setup page, Online Designer, and other places throughout REDCap in a popup and alternatively as a standalone page.

Improvement: Added new "Record ID" column in the Survey Invitation Log table to allow users to find specific invitations more efficiently. Note: If the record name should not be displayed in order to preserve the anonymity of a response (e.g., participant identifier is not used, designated email field is not used), it will instead display an icon indicating that the record name cannot be displayed.

Improvement: The Survey Invitation Log now contains an extra column on the right-hand side to allow users to delete many scheduled invitations at once (rather than having to delete them one at a time).

New feature: Survey-specific email invitation fields: This is a new option on the Survey Settings page that can be enabled for any given survey, in which a user may designate an email field for sending survey invitations for that survey only.

This feature is similar to the project-level email invitation field except that it is employed only for the survey where it has been enabled. This allows users to have data entry workflows that require multiple surveys where the participant is different for each survey. Using this feature, multiple people can be emailed a survey invitation, after which all the survey data they enter goes into the same record in the project.

Improvement/change: New method for composing survey invitation text using Smart Variables for survey link: When composing a survey invitation, the standard text and survey link are no longer automatically appended to the survey invitation text at the time the email is sent. Instead, users must now specify all the entirety of the text of the email (including the stock text and survey link that used to be appended automatically, if they wish) and therefore must supply [survey-url] and/or [survey-link] in the text if they wish to provide the participant with a link to the survey.

 

Release notes highlighting new features, improvements, and major bug fixes are available here.

 

Full release notes available here.

 

If you have any concerns about REDCap being offline at this time or any questions about these new features, please email edcsupport@partners.org

green - services online

REDCap Service Alert: Unexpected Downtime December 5th, 2018

December 5, 2018 1:51 pm

REDCap (https://redcap.partners.org/redcap) was offline intermittently from approximately 11:50-1:00 PM EDT. REDCap is currently back online. 

Please review your projects if surveys were scheduled to send during this time.  Manage Survey Participants > Survey Invitation Log.  You can filter on "Invitations that failed to send" and resend as needed.

Survey data may not have been collected at this time and participants received a web-browser error message when attempting to click on a survey link.

If you have any concerns, please email edcsupport@partners.org

green - services online

Possible Mysql3.research.partners.org Shutdown and Restart Tonight

November 14, 2018 4:34 pm

Applies to: All databases and systems connecting to the MySQL hosted database system mysql3.research.partners.org

The ERIS Infrastructure team has resolved the operational issue on mysql3.research.partners.org and has canceled the shutdown and restart identified for this evening.

 

This is a precautionary communication. 

There is a possibility that we may need to shut down and restart mysql3.research.partners.org tonight (11/14/2018) between the hours of 7:00 pm and9:00 pm. We have an operational issue that is impacting normal maintenance. If the issue is not resolved by early evening, then we will need to shut down and restart mysql3.research.partners.org.

If the issue is resolved and we do not need to restart, we will share a communication with you. If we need to shut down and restart mysql3.research.partners.org tonight, we will send a communication before the shutdown and again when mysql3.research.partners.org is back online.

Impact

All databases hosted on mysql3.research.partners.org will be offline while we shut down and restart mysql3.research.partners.org. 

Action Required

If you are the owner of a database on mysql3.research.partners.org, we encourage you to take your applications offline before the maintenance window and reconnect once the work is complete.

Please let us know if you have any questions by forwarding your questions to rcc@partners.org.

green - services online

RISC Website Maintenance | rc.partners.org

November 1, 2018 7:09 am - 7:12 am

Research IS & Computing website maintenance was performed on Thursday, November 1 starting at 7a.m. The site was offline for approximately 3 minutes while bug fixes and feature enhancements are implemented.

Please contact rcwebmaster@partners.org if you have any questions or concerns.

green - services online

Scheduled Downtime for mysql3.research.partners.org and https://rc.partners.org

October 31, 2018 9:53 am - November 10, 2018 12:53 pm

On Saturday morning November 3, 2018 from 7:00am - 9:00am the ERIS Infrastructure team will perform maintenance on mysql3.research.partners.org and on https://rc.partners.org. There will be a short interruption of service for both mysql3.research.partners.org and for https://rc.partners.org during the maintenance window. 

Please forward any questions to rcc@partners.org.
green - services online

An Update Regarding Apple's macOS Mojave

October 30, 2018 9:00 am

The PEAS team is pleased to inform you that we now recommend an upgrade to macOS Mojave 10.14 in the Enterprise. All issues with Enterprise Services such as email, VPN or Citrix have been resolved.  If you choose to upgrade, please verify the compatibility of any non-enterprise software you use prior to upgrading.
 
Keep in mind that Apple, and therefore Partners, only supports the current OS and two versions prior. With the release of macOS Mojave 10.14, the only other supported macOSs will be macOS High Sierra 10.13 and macOS Sierra 10.12. If you are running an out of date macOS, please view our webpage on how to Update Your Mac, or contact the Service Desk if you need assistance with an upgrade. Older Mac operating systems introduce risk to your data and the enterprise as they are not patched for security vulnerabilities, nor do they comply with Partners policies.

green - services online

Proxy Server Outage

October 24, 2018 10:48 am - 1:48 pm

The proxy server is back online. The /var directory filled and HTTP went down when it could not write to its log file. We cleaned up var and restarted HTTP which resolved the problem. 

The proxy server went down at 10:25 and was back online at 11:05.

Please reach out to rcc@partners.org with any questions.

green - services online

SAS Server Service Back On-line

October 22, 2018 9:11 am - 12:11 pm

The ERISSAS01 Server is now back online.

yellow - pre-downtime, calls to action

Secure File Transfer Service Update

November 1, 2018 10:00 pm - 10:30 pm

We are making an update to the log in process for Secure File Transfer at https://transfer.partners.org. Starting Thursday, November 1st at 10:30 p.m., employees will need to access this system for your individual accounts using your primary work email address and Partners password. This change is only for individual employee accounts.  External collaborators and group/departmental accounts can continue to login the same way.

With the Secure File Transfer Service, you can exchange large files with collaborators both inside and outside of Partners via a web browser. Visit https://rc.partners.org/SFTP for more information about this service.

green - services online

Freezerworks Expected Downtime

October 11, 2018 8:00 pm

Today, October 11th from 5:00 to 5:30 pm, Freezerworks will temporarily be offline to complete unplanned maintenance work. During this timeframe, we ask users to log out of the system so we can complete this necessary maintenance work. We do apologize for any inconvenience this interruption may cause.  

Please contact rcc.partners.org if you have any questions or concerns.

Pages