Service Alerts

green - services online

REDCap Service Alert: Scheduled Downtime

June 12, 2019 2:55 pm

REDCap (https://redcap.partners.org/redcap) upgrades to v8.10.18 scheduled for Tuesday June 18th, 6:30AM EST are complete. REDCap was offline from 6:28 to 6:44 AM EST. 

This upgrade will include a number of new features and bug fixes. These include:

  • New feature: REDCap Messenger
    • REDCap Messenger is a communication platform built directly into REDCap. It allows REDCap users to communicate easily and efficiently with each other in a secure manner. At its core, REDCap Messenger is a chat application that enables REDCap users to send one-on-one direct messages or to organize group conversations with other REDCap users. REDCap Messenger is also the best and easiest way to share documents with other REDCap users, in which you can upload documents and embed pictures inside any given conversation.
    • Watch 10-minute video on REDCap Messenger
  • Improvement: Performance boost – Certain pages in projects with thousands or more records should now load much faster in most cases. This includes the Record Status Dashboard, various pages utilizing Data Access Groups, and certain reports. Reports A and B should especially see significantly faster loading (excluding when viewing “all” pages in report A or B).
  • Change: Changed the text "Manage Survey Participants" to "Survey Distribution Tools," which more clearly describes the pages in that section.
  • New feature: Report Folders - Reports can now be organized into folders in any given project. If a user has "Add/Edit Reports" privileges, they will see an "Organize" link on the left-hand project menu above the project’s reports. They will be able to create folders and then assign their reports to a folder, after which the project's reports will be displayed in collapsible groups on the left-hand menu.
  • New feature: “Edit Access” for reports - In addition to setting "View Access" when creating or editing a report, users can now set the report's "Edit Access" (under Step 1) to control who in the project can edit, copy, or delete the report. This setting will be very useful if one wishes to prevent certain users from modifying or deleting particular reports.
  • Improvement: A project's Record ID field can now be used as a Live Filter in any given report, thus allowing users to easily view the report for a single record.
  • Improvement: New optional parameters added to the API Export Records method to filter data returned based on when a record was created or modified
    • dateRangeBegin – To return only records that have been created or modified *after* a given date/time, provide a timestamp in the format YYYY-MM-DD HH:MM:SS (e.g., '2017-01-01 00:00:00' for January 1, 2017 at midnight server time). If not specified, it will assume no begin time.
    • dateRangeEnd – To return only records that have been created or modified *before* a given date/time, provide a timestamp in the format YYYY-MM-DD HH:MM:SS (e.g., '2017-01-01 00:00:00' for January 1, 2017 at midnight server time). If not specified, it will use the current server time.

Full REDCap 8.10.18 release notes

New Features and Major Bug Fixes

If you have any concerns about REDCap being offline at this time or any questions about these new features, please email edcsupport@partners.org

Posted on June 12, 2019
green - services online

Freezerworks Service Alert: Maintenance

June 6, 2019 4:00 pm to 6:30 pm

Partners Healthcare Freezerworks: Scheduled Downtime

On Thursday June 6th from approximately 6 pm to 6:30 pm (EST), the Freezerworks application will be taken offline for maintenance. During this timeframe, we ask users to log out of the system so we can complete this necessary work. We apologize for any inconvenience during this interruption. 

 

If you have any questions or concerns about email: edcsupport@partners.org

 
Posted on June 6, 2019
green - services online

CrashPlan Access Resolved

June 5, 2019 3:53 pm to June 6, 2019 6:53 pm

The authentication server backup01.partners.org is up and running.  All services are green.  If you are having any issues email us at rcc@partners.org.

Currently, our CrashPlan authentication system (backup01.partners.org) is not functioning correctly.  We are in the process of rebuilding the system.  During this time you will not be able to backup or restore files.  Your backup client might report we are out of space, but that is a false message.   We will up this alert when everything is resolved.

Posted on June 5, 2019
red - services offline, issue, active work

MySQL and DIPR VM Problems

June 1, 2019 1:34 pm to June 3, 2019 5:00 pm

The ERIS MySQL Service and parts of the DIPR Virtual Machine service experienced problems over the weekend and had to be rebooted. The outages were related to the Compellent patching on Saturday, June 1st however, some work extended into Sunday, June 2nd and Monday, June 3rd. If you experience a problem with a DIPR VM please send a note to rcc@partners.org and the ERIS Infrastructure team will investigate.

Posted on June 3, 2019
green - services online

Compellent Outage 1 Jun 2019

June 1, 2019 1:45 pm to 2:45 pm
On Saturday, June 1st, 2019 around 1:45 pm ET, during a scheduled firmware upgrade of our backend Storage Area Network (CHG0145888), we experience an issue that caused some hosted servers that utilize this storage to be impacted.  The servers had to be rebooted, or our engineer team had to run file system consistency check to resolve the issue.
 
 Compellent Storage Area Network (SAN) service was fully restored with completed maintenance on Saturday, June 1 around 2:45p.m. EDT.
 
Many of our services rely on Compellent Storage Area Network (SAN) for underlying storage. Please check the Service Alerts for service-specific instructions and impact. Please make sure you subscribe to receive appropriate email notifications for the services you subscribe to: https://rc.partners.org/subscribe.
Posted on June 3, 2019
yellow - pre-downtime, calls to action

GitLab Unexpected Downtime

June 1, 2019 2:00 pm to June 3, 2019 9:30 am
Partners' GitLab: Intermittent service was unavailable between 2 PM on Saturday, June 01, 2019 until 9:30 AM, June 03, 2019.
Posted on June 3, 2019
green - services online

JIRA is now available

June 3, 2019 8:49 am to 11:49 am

Edit 10:38am: JIRA is now back online.  We apologize for the inconvencience

JIRA is currently down and unavailable.  An update to this alert will be posted shortly

Posted on June 3, 2019
yellow - pre-downtime, calls to action

Research Computing Ticketing System 29 May 2019

May 29, 2019 12:00 pm

Research Computing Ticketing System Now Online

The email component of the internal ticketing system (Kayako) used by ERIS has not been working properly. The issues started Wednesday, May 29 around 12p.m. EDT and are not yet fully resolved. 

The system may be delivering several of the same email or none at all. At times, ERIS was not receiving your tickets at all, so slightly longer than usual response times may be experienced.

Affected Services
Emails and requests sent to the following addresses were not accessed during this service disruption:
edcsupport@partners.org - REDCap-generated requests to approve, copy or move projects to production have not been received.
hpcsupport@partners.org
rcc@partners.org 
ideasupport@partners.org

Please allow our staff time to catch up on the tickets submitted while the system was offline.

Posted on May 31, 2019
yellow - pre-downtime, calls to action

ACTION REQUIRED: Urgent Microsoft Patching

May 23, 2019 8:08 am to 11:08 am

Microsoft has taken the unusual step of releasing sweeping system patches for older or unsupported but widely used Windows operating systems on Tuesday May 14 that include one critical patch that should be applied immediately. Details Here

 

 

Posted on May 23, 2019
yellow - pre-downtime, calls to action

Oracle PPM Database Work Extending Past Noontime

May 11, 2019 10:00 am to 1:00 pm

The Oracle patches scheduled for the Partners Personalized Medicine team on Saturday morning is taking longer than anticipated and will extending past noontime. We anticipate another hour of patching and rebooting as we should complete our efforts by 1:00pm.

Please forward any questions or concerns to rcc@partners.org

Posted on May 11, 2019
green - services online

Completed the ERIS PostgreSQL Service Maintenance on Saturday

April 27, 2019 9:00 am to 12:00 pm

The ERIS Infrastructure team will perform maintenance on the PostgreSQL Service on Saturday, April 27, from 9:00 a.m. - 12 p.m. EST.

Impact: All databases hosted on the ERIS PostgreSQL service will be impacted by one or two short reboots during this maintenance window.

Action Required: If you are the owner of a database on the PostgreSQL Service, we encourage you to check your applications after the maintenance window.

Completed

Posted on April 25, 2019
green - services online

Problem on Mysql.dipr.partners.org

April 22, 2019 11:04 am to 11:20 pm

Hi, We have a problem on mysql.dipr.partners.org and need to restart the node to get it online again. We are restarting mysql.dipr.partners.org

Completed Maintenance 

Posted on April 22, 2019
green - services online

Completed Emergency Postgres Maintenance

April 20, 2019 9:00 am to 10:15 am

The ERIS Infrastructure Team completed the emergency Postgres maintenance and all Postgres databases are online and available.

Posted on April 20, 2019
green - services online

MySQL Hosted Database Services DownTime on 30 Mar 2019

March 30, 2019 7:00 am to 12:00 pm

The ERIS Infrastructure team completed the on the MySQL hosted database Service on March 30, 2019. Please forward any questions to rcc@partners.org.

Posted on March 28, 2019
green - services online

Research Computing Website Downtime 30 Mar 2019

March 30, 2019 7:00 am to 10:00 am

This website may have been offline for a brief period during the maintenance window for MySQL Server patching on Saturday, March 30 between 7:00a.m. - 10:00a.m. EDT. This website, https://rc.partners.org, may have been unavailable for a few minutes while the system is rebooted to apply patches.

Please contact rcwebmaster@partners.org if you have any questions or concerns

Posted on March 27, 2019
green - services online

List Manager (Lyris): Expected Downtime on 14 April 2019

April 14, 2019 9:00 pm to 10:00 pm

We are upgrading the List Manager (Lyris) system to v12.3.5 on Sunday, April 14, 2019, at 9 PM. We expect this release to take, up to an hour, so please expect the List Manager (Lyris) service to be offline from 9:00 to 10:00 PM. If you have any questions or concerns, relating to this upgrade, please email: rcc@partners.org. 

Posted on March 25, 2019
green - services online

Research Computing Ticketing System 22 Mar 2019

March 21, 2019 10:04 pm to March 22, 2019 12:30 pm

Research Computing Ticketing System Online

The MySQL Hosted Database Services outage impacted the internal ticketing system (Kayako) used by ERIS. The outage started Thursday, March 21 around 10p.m. EDT and services restored around 12:15p.m. EDT on Friday, March 22.

Affected Services
Emails and requests sent to the following addresses were not accessed during this unanticipated downtime:
edcsupport@partners.org - REDCap-generated requests to approve, copy or move projects to production have not been received.
hpcsupport@partners.org
rcc@partners.org 
ideasupport@partners.org

Please allow our staff time to catch up on the tickets submitted while the system was offline.

Posted on March 22, 2019
green - services online

Research Computing Core Service Catalog 21 Mar 2019

March 21, 2019 10:40 pm to March 22, 2019 12:30 pm

RC Services Online

The outage started Thursday, March 21 and we expect services to be restored around 12:30p.m. EDT on Friday, March 22. This downtime is related to the MySQL Hosted Database Services outage.

We will update this message when service is restored. 

Posted on March 22, 2019
yellow - pre-downtime, calls to action

MySQL Hosted Databases Restored 22 Mar 2019

March 21, 2019 10:58 pm to March 22, 2019 12:30 pm

MySQL Hosted Database Services Online

The outage started Thursday, March 21 around 10p.m. EDT and was restored around 12:00p.m. EDT on Friday, March 22.

Action Required
If you are the owner of a system on mysql.dipr.partners.org, we strongly encourage you to check your application connection and reboot if necessary.

Affected Services
All systems hosted on mysql.dipr.partners.org
Research Computing ticketing system (Kayako tickets.partners.org)
Research Computing ordering/request system (RC Services rcservices.partners.org)

We will update this posting once service has been restored. Please contact rcc@partners.org with any questions or concerns.

Posted on March 21, 2019
green - services online

REDCap Scheduled Downtime 19 Mar 2019

March 18, 2019 4:21 pm

REDCap (https://redcap.partners.org/redcap) upgrades to v8.5.28 scheduled for Tuesday March 19th at 6:30AM EDT are complete REDCap was offline from 6:30AM-6:48AM EDT.

This release includes a number of bug fixes. 

8.5.28 Release Notes

If you have any concerns about REDCap being offline at this time, please email edcsupport@partners.org

Posted on March 18, 2019
green - services online

Emergency Maintenance HPCWIN3

March 6, 2019 5:00 pm to March 11, 2019 9:00 am

The hpcwin3.partners.org Windows analysis server has presented several freeze instances in the last few days. To evaluate the usage and logs we need to take the server out of service for a few days. During this window time we will perform test and review the information of the processes that have been causing the system to crash.  Future actions needed to be taken after for repair or control will be announced to the users. 

Existing remote desktop sessions will be terminated. Please save all open files and exit from your remote desktop session as soon as possible. We apologize for the inconvenience this may cause. 

Posted on March 7, 2019
green - services online

DIPR Hosted VM Reboot 6 Mar 2019

March 6, 2019 1:47 am to 2:47 am

We had to reboot the service. This is an NFS server serving mapped drives fro /nfs1, /nfs2, and /redcap-prod. The fs.dipr.partners.org VM is currently online, however, you may need to remount your NFS mounts this morning. 

Posted on March 6, 2019
green - services online

MySQL Hosted Database Service Reboot 4 Mar 2019

March 4, 2019 8:00 pm to 8:30 pm

We had an emergency issue that impacted mysql2.dipr.partners.org. To resolve this issue we rebooted mysql2.dipr.partners.org at 8:00pm on Monday evening March 04, 2019.

Posted on March 4, 2019
green - services online

PostgreSQL2 Hosted Databases Online 1 Mar 2019

March 1, 2019 9:19 am to March 3, 2019 12:19 pm

pgsql2.research.partners.org is now back online

Posted on March 1, 2019
green - services online

REDCap Service Alert: Scheduled Maintenance

February 22, 2019 10:08 am

REDCap (https://redcap.partners.org/redcap) requires server maintenance which is scheduled for Saturday, February 23rd 2019 from 7:00AM to 9:00AM EST. There will be a short interruption of service during the maintenance window.

If you have any concerns about this scheduled maintenance, please email edcsupport@partners.org

Posted on February 22, 2019

Pages