Service Alerts

yellow - pre-downtime, calls to action

Mysql, Mysql2 and Mysql3 Downtime on Saturday June 23rd

June 23, 2018 9:00 am - 12:00 pm

The ERIS Infrastructure team will perform maintenance on mysql.dipr.partners.org, mysql2.dipr.partners.org and mysql3.research.partners.org on Saturday, June 23, 2018, from 9:00 am -  Noon. There will be a short interruption of service for each server during the maintenance window. Please forward any questions to rcc@partners.org.

green - services online

REDCap Scheduled Downtime

June 14, 2018 4:31 pm

REDCap (https://redcap.partners.org/redcap) upgrades to v8.1.17 scheduled for Tuesday June 19th at 6:30AM EDT are completed. REDCap was offline from 6:30-7:34 AM EDT.

This upgrade included a number of new features and bug fixes. These include:

New action tag: @HIDECHOICE - Hides one or more choices of a multiple choice field. This action tag is useful if you wish to retire a particular choice after utilizing it for a while in data collection, thus allowing you to hide the choice from that point after without orphaning any of the choice's data, which would happen if you simply deleted the choice.

New feature: PDF download for survey respondents - On an instrument's survey settings page, a user may enable the option "Allow participants to download a PDF of their responses at end of survey?". This option will display a button for the survey participant to download a PDF file of their responses for the survey they just completed. Users may also download this same copy of the PDF since it has been added as a new PDF download option at the top of data entry forms.

Improvement: When a calc field's value is changed, it will now display a red border on the right edge of the calc field's text box to subtly indicate that the displayed value is different from the stored value in the database. And if the calculated value gets changed so that its value matches the stored value again, the red border will disappear. This will help users make note of when calculations have been triggered and if there is any disparity between the displayed value and the stored value.

New action tag: @MAXCHECKED - Allows a checkbox field to have a maximum number of checkboxes that can be checked.

New feature: New option to allow respondents to return to a partially completed or fully completed survey response without needing a return code.

List of more new features and fixes available here.

Full release notes available here.

If you have any concerns about REDCap being offline at this time or any questions about these new features, please email edcsupport@partners.org

green - services online

PEAS Service Alert: Improved VPN for Macs!

June 6, 2018 2:00 pm - June 30, 2018 10:00 pm

The PEAS team is pleased to inform you of a service improvement for Apple computers, Java-less VPN! You may now connect directly to the VPN server without requiring Java or allowing your browser to run Java in unsafe mode. To view this new process, please view our knowledge base article HOWTO: Configure and Connect to VPN on a Mac with SMS Authentication. If your device is enrolled in PEAS and running the latest version of Cisco AnyConnect (4.5.02036 as of this alert) it’s as easy as typing the new server address into your AnyConnect application: pvc.partners.org/saml. If you need VPN Access, review KB0018524 for request instructions. If you connect to VPN via software token, please view HOWTO: Configure and Connect to VPN on a Mac with Software Token Authentication. For any question or assistance with VPN, please submit a ticket to the IS Service Desk.

green - services online

Proxy was rebooted and is back online

May 23, 2018 9:49 am - 10:00 am

We are sorry for any inconvenience that occurred with the proxy server outage and we want to share what happened and what we have done to remedy the situation. We were unaware of an increase in usage directed at our servers. All DIPR based HTTP and HTTPS traffic route through our proxy server in the Partners DMZ. We have seen an increase in traffic over the past few weeks and a very large increase in traffic on Wednesday, May 23rd at 9:00am, 2:30pm and 3:50pm.  This increase in traffic caused the proxy server to run out of memory. After the first incident, we increased the memory on the proxy server from 12gb to 16gb and engaged Neteng to try and determine where the increased traffic was coming from. After the third incident on Wednesday, a customer reached out to us concerning the stability of our environment and noted that they had increased their usage by 2500 new engagements to their site. Given this news we increased the memory on the proxy server from 16gb to 32gb, and we adjusted the apache configuration to manage the new load. We are currently monitoring apache, memory usage and load on the proxy server and feel that now we have a good configuration to manage the increased load. Also, we have started plans to replace the current proxy server with a redundant pair of proxy servers, however, that configuration is probably a couple of months away from implementation. 

Proxy users could help us by providing usage estimates for their sites and letting us know when they anticipate new users or additional sites and users coming online.

green - services online

Crimson Database Server is back online

May 22, 2018 2:30 pm - 7:00 pm

The Crimson database server is back online and available. 

green - services online

RISC Website Maintenance | rc.partners.org

May 24, 2018 7:00 am - 7:30 am

Research IS & Computing website maintenance completed on Thursday, May 24 between 7-7:30a.m. EDT. The https://rc.partners.org website was offline from approximately 7:10-7:15a.m. EDT to apply security patches and bug fixes.

green - services online

LimeSurvey Unexpected Downtime

May 22, 2018 12:22 pm

LimeSurvey (https://limesurvey.partners.org/limesurvey) experienced an unexpected outage on May 22nd, 2018. LimeSurvey was offline from 12:00PM-3:57PM EDT.

If you have any concerns, please email edcsupport@partners.org

green - services online

List Manager Issue Fixed

May 21, 2018 9:07 am - 12:07 pm

This morning, there was an issue with our license in Production and our system was not able to send out any mailings, between 12:00 am to 10:40 am (this morning). The issue was corrected and the problem is now, fixed!

** If you sent or scheduled, any mailings between 12:00 to 10:40 am (today), please do not re-send the mailings, again. All mailings were queued and should be sending out as normal. If your mailings, do not get sent out in the next hour or so, please contact: rcc@partners.org for assistance in sending out your mailing.

Additionally, if you have further questions or concerns about this issue, please email: rcc@partners.org

green - services online

REDCap Scheduled Downtime

May 18, 2018 12:36 pm

REDCap (https://redcap.partners.org/redcap) requires server maintenance which is scheduled for Sunday, May 20th from 9:00AM - 11:00AM EDT.   

If you have any concerns about this scheduled maintenance, please email edcsupport@partners.org

green - services online

Mysql2 and Mysql3 Downtime on Sunday - Completed

May 20, 2018 9:00 am - 11:00 am

The ERIS Infrastructure team will perform maintenance on mysql2.dipr.partners.org and mysql3.research.partners.org on Sunday, May 20, 2018, from 9:00 am - 11:00 am. There will be a short interruption of service for each server during the maintenance window. 

green - services online

rc.partners.org Website Downtime

May 20, 2018 9:00 am - 11:00 am

There was a short interruption of service for the RISC Website during the maintenance window for MySQL Server patching conducted during the ERIS Systems Maintenance Window. You might not have been able to connect to the RISC Website, https://rc.partners.org, for approximately 5 minutes between 9-11a.m. EDT while the system was rebooted.

Please contact rcwebmaster@partners.org if you have any questions or concerns.

green - services online

REDCap: New Project Request Improvement

May 17, 2018 10:59 am

EDCSupport is consistently experiencing a high volume of requests. In order to alleviate the number of tickets we receive, we're working towards a few process improvements. As of today, we're allowing certain types of projects to be created in real time and without REDCap Admin approval.

Please be sure to complete as much information on the request to ensure expedited project creation.

If you have any questions, please email edcsupport@partners.org

green - services online

Proxy Server Outage

May 8, 2018 11:30 am - 4:30 pm

The proxy server went down at 11:30am today and was restored at 4:30pm. This impacted all ERIS hosted URL's. The IAAS team will perform a post-mortem and review the results in an effort to reduce the possibility of this event occurring again.

green - services online

RISC Website Maintenance | rc.partners.org

May 2, 2018 7:10 am - 7:20 am

Research IS & Computing website maintenance was scheduled for Wednesday, May 2 between 7-7:30a.m. EDT. The rc.partners.org website was offline for approximately ten minutes, from 7:10-7:20a.m. to apply security patches and bug fixes

 

 

yellow - pre-downtime, calls to action

Partners HealthCare REDCap: Mobile App Bug

April 30, 2018 2:42 pm

If you are using the REDCap Mobile App and are actively collecting data, please DO NOT upgrade to v.2.51.10 of the app on your mobile devices.

There is a known bug that may impact data collection and cause data loss. 

The error occurs when data is saved on the device and the app is then updated. If that happens, the user will receive an error message notifying them that they no longer have access to the project.

To anyone facing this issue, it appears that turning off wifi on the iOS device allows you to open the project so you can view the data stored on the device and re-enter it directly into REDCap. 

After manually entering the data into REDCap, the project should then be removed from the app and added back. 

The issue is specific to iOS devices. Devices using Android do not experience the bug. 

More details to follow.

If you have any concerns about this, please email edcsupport@partners.org

green - services online

Partners HealthCare Freezerworks: Scheduled Downtime on April 30th, 2018

April 30, 2018 5:00 pm - 5:30 pm

In order to prepare for a future upgrade, the Freezerworks System will be temporarily taken offline for less than 30 minutes on Monday, April 30th, 2018 at 5 pm

If you have any questions or concerns, please email: edcsupport@partners.org

green - services online

Freezerworks: Scheduled Downtime

April 18, 2018 4:42 pm

Partners HealthCare FreezerWorks: Expected Downtime and Maintenance

On Thursday April 18th from 8 pm to midnight (EST), the Freezerworks application will be taken offline for maintenance. We apologize for any inconvenience during this interruption.

If you have any questions or concerns about email: edcsupport@partners.org

green - services online

Freezerworks Unexpected Outage: 4/12/2018

April 12, 2018 9:56 am

The system was taken offline from Tuesday, April 11th through Friday, April 13th while our technical team worked with the vendor on identifying and resolving the issues.

ACTION REQUIRED BY FREEZERWORKS USERS  

Freezer inventory data that was entered should be reviewed during the past 7 days (April 5th - April 12th). The data may need to be re-entered. 

We apologize for any inconvenience during this interruption. 

If you have any questions or concerns about email: edcsupport@partners.org

green - services online

Freezerworks: Unexpected Outage: 4/10/2018

April 10, 2018 2:27 pm

Partners HealthCare FreezerWorks: Application Online

Freezerworks users were unable to access this application from approximately 10:30 AM - 2:30 PM. After this unexpected downtime, our technical team was able to rebuild the database.

ACTION REQUIRED BY FREEZERWORKS USERS  

If there are any customized fields that are no longer visible or accessible, please create a ticket and we will address it. 

Again, we apologize for any inconvenience during this interruption.  

If you have any questions or concerns about email: edcsupport@partners.org

green - services online

rc.partners.org Website Maintenance

April 3, 2018 6:30 am - 6:55 am

Research IS & Computing website maintenance was completed on Tuesday, April 3 starting at 6:30a.m. The rc.partners.org will be offline between 6:45-6:55a.m. EDT for bug fixes and security updates.

green - services online

Confluence Downtime and Upgrade

March 25, 2018 7:00 am - 10:00 am

UPDATE: CONFLUENCE MAINTENANCE HAS BEEN COMPLETED. ALL SERVICES ARE RUNNING NOMINALLY (updated 1:25pm)

UPDATE: CONFLUENCE MAINTENANCE EXTENDED UNTIL 1:30PM (updated 12:37pm)

UPDATE: CONFLUENCE IS DOWN FOR SCHEDULE MAINTENANCE (updated 10:30am)

UPDATE: THIS HAS BEEN RESCHEDULED FOR 10:30AM EST - 12:30PM EST SUNDAY, MARCH 25TH

Confluence will be down for maintenance on Sunday, March 25th from 7:00am EST to 10:00am EST as we upgrade from 5.9.11 to 6.7.2

Please see CHG0125598 for more information

green - services online

MySQL Outage

March 17, 2018 1:04 pm - March 20, 2018 12:04 pm

The mysql service went down over the weekend. All mysql databases hosted on mysql3.research.partners.org were down from 1:30pm on Saturday March 17, 2018 through Sunday March 18th at 9:30am. Please check your applications and database connections for all instances on mysql3.research.partners.org.

Please contact rcc@partners.org if you have any questions or concerns.

green - services online

SSH Bastion Update

March 16, 2018 12:45 pm - 12:50 pm

SSH (Secure Shell) Bastion Server, ssh3.partners.org, was unavailable on Friday, March 16, 2018 from 12:45-12:50p.m. to apply patches.

A system reboot interrupted SSH connections during that time. 

Please contact rcc@partners.org if you have any questions or concerns.

yellow - pre-downtime, calls to action

DST Calendar Display Bug in Outlook

March 11, 2018 6:00 am - March 17, 2018 11:00 pm

We are seeing a bug in the Microsoft Outlook full client application where this week’s calendar "Week" view is not displaying the time correctly after Daylight Savings time. This will be resolved after Saturday, March 17, 2018, but you can view the "Work Week", "Daily" or "Monthly" views as workarounds. Learn more about this bug and the workarounds here:

https://www.slipstick.com/outlook/calendar/time-zone-dst-calendar-display-bug-in-outlook/

 

green - services online

REDCap Scheduled Downtime

March 2, 2018 11:02 am

REDCap (https://redcap.partners.org/redcap) required a security patch be applied, which was scheduled for tonight Friday, Mar. 2nd at 8:30 PM EDT.  REDCap was down from 8:41 to 8:45 PM EDT.

If you have any concerns about this scheduled maintenance, please email edcsupport@partners.org

Pages