Service Alerts

yellow - pre-downtime, calls to action

macOS 10.15 Catalina

October 7, 2019 1:00 pm

On Monday, October 7th, 2019, Apple released their latest version of the System (OS) for Mac laptops & desktops, macOS Catalina. At this time, we recommend you do not upgrade your Mac as this may impact your ability to use enterprise services. 

Like prior Apple OS releases, the expectation is the first minor version of Catalina (version 10.15.1) will be stable for the Enterprise. Further information on compatibility with Partners services will be broadcast in the coming weeks.

Presently we are experiencing issues with Trend Micro Security (Antivirus). We expect an update by Trend in the coming weeks which will be tested by the PEAS team. Further guidance to follow.

If you have upgraded already, there are 3 major items to be aware of: 

  1. You must update Citrix from the PEAS Self Service Application to ensure Partners services delivered from Citrix, such as Epic or Internet Explorer, continue to function. 
    • Citrix - Citrix Receiver should be replaced with Citrix Workspace 19.10.1.31 in Self Service.
  2. VPN - Cisco AnyConnect VPN Client Version 4.800175 also needs to be upgraded from the PEAS Self Service Application.
  3. Any 32-bit applications will no longer run under Catalina.  Prior to upgrading, you will need to check your applications for compatibility.

We also recommend making a backup prior to any upgrade, in case you need to downgrade later. Apple, and therefore Partners, only support the current OS and two versions prior. With the release of macOS Catalina 10.15, the only other supported macOSs will be macOS Mojave 10.14 and macOS High Sierra 10.13 If you are running an out of date macOS, Update Your Mac, or contact the Service Desk if you need assistance with an upgrade. Older Mac operating systems introduce risk to your data and the enterprise as they are not patched for security vulnerabilities, nor do they comply with Partners policies. 

Software Tested Functionality Notes
Crowdstrike 5.15.9504  
Trend Micro 3.0.3067 Visit the Knowledge Base for information about additional antivirus solutions.
Forescout 15.3.15303  
Cisco AnyConnect 4.800175  
Microsoft Office 16.29  
Citrix Workspace 19.10.1.31  
Crestron AirMedia 3.1.3  
Kingston DTVP30 encrypted flash drives Cannot unlock the Drive on Catalina. New App expected ~ November
green - services online

RISC Website Maintenance 28 Sept 2019

September 28, 2019 9:00 am to 2:00 pm

The https://rc.partners.org website was offline for a short period during routine maintenance on Saturday, September 28 from 9a.m.-2p.m. EDT.

Please contact rcwebmaster@partners.org with any questions or concerns.

green - services online

RISC Website Maintenance 22 Sept 2019

September 22, 2019 9:30 pm to September 23, 2019 12:00 am

The https://rc.partners.org website was offline for a short period of time for infrastructure maintenance that took place Sunday, September 22 from 9:30p.m. to midnight EDT.

Please contact rcwebmaster@partners.org with any questions or concerns.

green - services online

Completed MySQL PostgreSQL and Storage Maintenance

September 22, 2019 9:30 pm to September 23, 2019 12:00 am

The MySQL PostgreSQL and Storage Maintenance on Sunday Evening has Completed. 

The MySQL, PostgreSQL, and Storage Maintenance that was scheduled for Saturday, September, 21st 2019 at 12:00 pm have been rescheduled to Sunday evening September 22nd, 2019 at 9:30 pm. The ERIS Infrastructure team will shutdown all hosted PostgreSQL and MySQL servers at 9:45 pm on Sunday, September 22nd, 2019. The window for this maintenance is from 9:30 pm - 12:00 am. All databases hosted on mysql.dipr.partners.org, mysql2.dipr.partners.org, mysql4.research.partners.org, pgsql.dipr.partners.org, and pgsql2.research.partners.org will be offline during this maintenance window.

If you are the owner of a database on the PostgreSQL or MySQL hosted database service, we encourage you to review and validate your application's connection to your database when this maintenance has completed. 

yellow - pre-downtime, calls to action

MobileIron and iOS 13 and 13.1

September 24, 2019 1:00 pm

Applies to: iOS Users enrolled in MobileIron

The Partners MobileIron/EMM team has completed testing on latest GM build of iOS 13.1 Beta. Apple released iOS 13.0 on Thursday, 9/19 at around 1pm EDT and will release 13.1 on Tuesday, 9/24 around 1p.m.

  • If you upgrade to iOS 13.1 from iOS 13.0, no action is required
  • If you upgrade to iOS 13.1 from iOS 12, please note that you will be required to re-enter your Partners Email Password. A pop up will automatically appear that will take the you to the Settings App to re-enter your password.

If you accidentally hit “Cancel”, you can go to Settings > Passwords & Accounts > PARTNERS (OAUTH) to re-enter your Partners password.

Updated MobileIron enrollment instructions have been posted to the Knowledge Base 

 

IS Service Desk https://www.partners.org/ISServiceDesk 

BWH 617-732-5927      BWH-RICS 617-525-0848      CDHC 413-582-5005      DFCI 617-632-3399         
FALK 617-983-7454     LCC 857-307-4150                  MCL 781-416-8940        MGH 617-726-5085     
NHP 617-643-2020       NSMC 978-354-2014               NWH 617-243-6001       PCPO 857-282-5999     
PHH 617-726-0790       PHS 857-282-4357                   SRN 617-952-5555  
 

yellow - pre-downtime, calls to action

Windows 10 Upgrade for Non-standard PCs

September 18, 2019 3:54 pm

Microsoft is ending support for Windows 7. If you are currently running Windows 7 or older, you must upgrade to Windows 10 Professional by Jan. 1, 2020.

Vulnerabilities are discovered frequently in software and they put your computer, data and the enterprise at risk. In order to stay secure and compliant with Partners policies, make sure that you are running a current, supported Operating System (OS) and that your software applications are up to date. 

  • Vendor-provided device:  Work with the vendor. If the vendor device is not compatible with Windows 10.  An approved variance from Information Securityrequired.  
  • Partners Standard Windows devices: Devices are being upgraded by your IS Department. 
  • Non-Standard Windows PC: Follow the instructions below.

​HOWTO: Determine whether you are using a standard or non-standard computer 

Check the eligibility of your PC
Please review the Windows 10 minimum requirements. Check which edition you're using by right-clicking the Start button, clicking System, and finding the Windows Edition.

Which scenario best describes your non-standard Windows PC?
The device runs Windows 7, 8 or 10 Home Edition.

Option 1
Contact your purchasing department to acquire best pricing for Windows 10 Professional upgrade software.

Option 2
Replace device. Contact the IS Service Desk to have a ticket opened and sent to the ERIS site team to discussion options and compatibility.
The device runs Windows 7 or 8 Pro Edition.

Option 1
If you are already running a valid copy of Win7 Pro you can use the link below to download Win10 Pro and perform the upgrade. https://www.microsoft.com/en-gb/software-download/windows10ISO

Option 2
Contact your purchasing department to acquire best pricing for Windows 10 Professional upgrade software.

Contact the IS Service Desk with any questions or for technical assistance.

green - services online

DIPR Hosted Servers OS Patching 28 Sept 2019

September 28, 2019 9:00 am to 2:00 pm

Applies to: All DIPR Servers

Production servers hosted by ERIS were patched on Saturday, September 28 starting at 9AM. If you are the owner of a server hosted by ERIS, we encourage you to connect to your server at the end of our maintenance, verify that your applications restart and reconnect to your database. 

The Discovery Informatics Platform for Research (DIPR) platform hosted by Enterprise Research IS is undergoing several essential upgrades in Fall 2019 to elevate security standards. These changes will include increased auditing, enabling SELinux on all systems and patching of the Linux OS on all servers on a monthly basis. The ERIS Infrastructure team will communicate each change two weeks before deploying. Please subscribe to receive DIPR communications and visit https://rc.partners.org/subscribe for more information. Non-production DIPR servers were patched in waves on Wednesdays in September. 

We understand that this is a significant change. We need your help to ensure an easy transition for your servers, applications and users. Learn more about the new DIPR Servers Monthly OS Patching Schedule.

Service Level Reminders
To ensure premium supportability, security and performance, ERIS strives to keep all systems up to their most current and/or stable revisions. Please visit https://rc.partners.org/sla to review the full ERIS Service Level Agreement.  

Server owners are responsible for maintaining the applications running on DIPR VMs systems and ensuring they are patched and comply with Partners Information Security Practices and Policies. Server owners are responsible for backup of critical configurations in case of a failure.

Please reach out to rcc@partners.org with any questions.

green - services online

Complete ERIS MySQL and PGSQL Maintenance

August 24, 2019 10:00 am to 11:00 am

The ERIS Infrastructure team completed the maintenance on the ERIS MySQL and ERIS PGSQL servers. All servers and databases are up and online. If you are the owners of a ERIS Hosted Database, please check your applications connection to the database. Please contact rcc@partners.org with any questions.

green - services online

Research Computing Website Downtime 24 Aug 2019

August 24, 2019 9:00 am to 12:00 pm

This website was offline for a brief period during the maintenance window for MySQL Server patching on Saturday, August 24 between 9:00a.m. - Noon. EDT. This website, https://rc.partners.org, was unavailable for a few minutes while the system is rebooted to apply patches.

Please contact rcwebmaster@partners.org if you have any questions or concerns.

green - services online

REDCap Scheduled Downtime: August 24th, 2019 9:00-10:00AM EST

August 24, 2019 9:00 am to 10:00 am

REDCap (https://redcap.partners.org/redcap) server maintenance is scheduled for Saturday, August 24th 2019 from 9:00AM to 10:00AM EST. There will be a short interruption of service during the maintenance window.

If you have any concerns about this scheduled maintenance, please email edcsupport@partners.org

yellow - pre-downtime, calls to action

FreezerWorks: Expected Downtime 8/18/2019

August 18, 2019 1:00 am to 5:00 am

On August 18th from approximately 1:00 am to 5:00 am, Freezerworks will temporarily be offline to complete planned server maintenance work. During this timeframe, we ask users to log out of the system so we can complete this necessary maintenance work. We do apologize for any inconvenience this interruption may cause. 

If you have any questions or concerns, please email edcsupport@partners.org

yellow - pre-downtime, calls to action

List Manager (Lyris): Downtime on 18 August 2019

August 18, 2019 8:00 pm to 8:30 pm

We are upgrading the List Manager (Lyris) system to v12.3.6 on Sunday, August 18, 2019, at 8 PM. We expect this release to take, up to 30 minutes, so please expect the List Manager (Lyris) service to be offline from 8:00 PM to 8:30 PM

Applies to:
https://researchlistadmin.partners.org 
https://researchlist.partners.org

This upgrade was to improve system performance and address defects. The release notes can be found in the Knowledge Base

Please contact rcc@partners.org with any questions, concerns or issues with the upgrade.

yellow - pre-downtime, calls to action

Microsoft Windows Patch Tuesday

August 13, 2019 9:00 am

Did you know that Microsoft regularly releases security patches for its software products on the second Tuesday of every month? How frequently do you update your computer?

This month, Microsoft introduces patches for critical flaws and vulnerabilities that require you to patch as soon as possible.

Here is some guidance to keep your device secure and your Microsoft software up to date and running smoothly.

Do I need to make changes on my computer?

  • Not if you use a Partners standard Windows computer (those with the asset tag, yellow lock and Partners' screensaver). The updates are run for you and devices are usually patched the Friday following Patch Tuesday. Both Microsoft and Adobe products are typically updated at this time. This can change if there is a significant freeze in place. 
  • Yes, you need to run updates if you use a non-standard Windows, Macintosh or personally-owned device for Partners' business. ​It is your responsibility to keep your device and software up to date. 

See HOWTO: Run Windows Updates for additional instructions.

green - services online

ERIS Hosted Services

August 6, 2019 10:30 am
Several Enterprise Research IS services are offline starting at 10:30a.m. EDT on Tuesday, August 6, 2019. There was an issue with the underlying storage infrastructure. 

If you find a Virtual Machine (VM) or website that is not responding, please submit a ticket to the IS Service Desk

Additional Impacted Services:

Service Description Status Notes 
Impacted Services
ERIS MySQL hosted database services
  • Online as of 1:00p.m. EDT on 8/6/19 (server)
Research Computing Website (https://rc.partners.org

Online as of 2:49p.m. EDT on 8/6/19

ERIS Web Hosting Services Online as of 3:47p.m. EDT on 8/6/19
RFANFS
Online as of 2:17p.m. EDT on 8/6/19, Rebooting this system on 8/7/19 at 9p.m. EDT
RFAWIN
Online as of 7:00p.m. EDT on 8/6/19
REDCap (https://redcap.partners.org)
 
  • Online as of 1:47p.m. EDT on 8/6/19
  • Issues resolved as of 4:30p.m. EDT on 8/6/19. If you are still experiencing any issues related to logging into REDCap or importing/exporting data, please email edcsupport@partners.org.
Confluence/JIRA
Online as of 1:00p.m. EDT on 8/6/19
PEAS (https://enroll.partners.org)
Online as of 1:00p.m. EDT on 8/6/19
GitLab (https://gitlab.partners.org)
Online as of 1:00p.m. EDT on 8/6/19
RC Services (https://rcservices.partners.org)
Online as of 5:00p.m. EDT on 8/6/19
DIPR Virtual Machines (VMs)
Online as of 9:45p.m. EDT on 8/7/19
DIPR File Server (fs.dipr)
Online as of 9:45p.m. EDT on 8/7/19

If you find a Virtual Machine (VM) or website that is not responding, please submit a ticket to the IS Service Desk

green - services online

Securing your Network Connection at Assembly Row

August 13, 2019 7:00 am

Applies to:        Staff and Visitors at Assembly Row

The Network Access Control (NAC) program updates scheduled for Assembly Row on Tuesday, August 13 are canceled at this time. 

Partners HealthCare continues to enhance the Network Access Control (NAC) program at Assembly Row, strengthening visibility and security of data and devices connected to the Partners network.  

Please review the security requirements and guidance for your device, by visiting rc.partners.org/computer

Thank you in advance.

Jigar Kadakia
Partners Chief Information Security & Privacy Officer
CISPO@partners.org

yellow - pre-downtime, calls to action

Install CrowdStrike on Your Apple Macintosh Device

July 25, 2019 9:00 am

Beginning July 25, 2019, CrowdStrike will be pushed to your corporate/Partners-purchased Macintosh device. Please note the date was moved one week due to the Enterprise Change Freeze.

It will be required to connect to the Partners Corporate network. CrowdStrike is an additional endpoint security tool application that complements antivirus (AV). Both CrowdStrike and antivirus are required to keep your systems safe and secure. Trend Micro antivirus is available for installation on all Mac and Windows devices used for Partners business purposes, including research. Please see Supported Antivirus Software for a full list.

CrowdStrike end point protection and Trend Micro antivirus are available for download from the Partners Enterprise Apple Support (PEAS) Self Service application. 

CrowdStrike can also be self-installed prior to the automatic push using the directions below.

Please note: Apple released the public beta version for the next operating system for Mac desktops & laptops, macOS Catalina (10.15). We recommend you do not upgrade at this time. Traditionally new Mac Operating Systems will initially cause compatibility issues with Partners services.

To download CrowdStrike via Self Service:

  1. Launch “Self Service”
  2. In “Applications”, search for “CrowdStrike Falcon Sensor”
  3. Select Install

View detailed installation instructions on our Knowledge Base
 
For assistance or questions, contact the IS Service Desk and ask to submit a ticket to the Engineering Security Team.
 
Thank you in advance.
 Jigar Kadakia
 
Partners Chief Information Security & Privacy Officer
399 Revolution Dr, Somerville, MA 02145

green - services online

Mysql3 Database Move to Mysql4

July 19, 2019 6:01 pm to July 21, 2019 6:01 pm

This is an update on the ERIS Infrastructure teams efforts to move all databases off of mysql3.research.partners.org and restore them on mysql4.research.partners.org. The team has worked through 50% of the efforts needed to complete this effort and is working diligently on all remaining tasks. We will share another update later tonight and will provide more detail concerning availability.

The final step in this process belongs to the database owners who need to change their MySQL Data Source reference from mysql3.research.partners.org to mysql4.research.partners.org.

Please send a note to rcc@partners.org with any questions.

green - services online

Partners HealthCare REDCap: Scheduled Downtime

July 17, 2019 2:05 pm

IMPORTANT NOTICE: REDCap Database Migration

In order to meet the increased demand, usage, and amount of data stored within the REDCap system, it is necessary to upgrade the database infrastructure.

As a result, a REDCap database migration is scheduled for Saturday July 20th, 2019 at 7:00PM EST.  This will require an extended amount of downtime with REDCap being offline for approximately 14 hours. 

REDCap was offline from 7:00pm-5:55am. 

This will help optimize our REDCap infrastructure, resulting in faster overall performance, and prevent extended downtime for maintenance in the future. 

If you have any concerns about REDCap being offline at either of these times, please email edcsupport@partners.org

green - services online

RISC Website Maintenance 17 Jul 2019

July 17, 2019 7:13 am to 7:20 pm

Research IS & Computing website maintenance completed on Wednesday, July 17 from 7-7:30a.m. EDT. The https://rc.partners.org website was offline from 7:13-7:20a.m. EDT for a database migration. Please see Mysql3 Shutdown and Retirement for details.

green - services online

Mysql3 Shutdown and Retirement

July 19, 2019 6:00 pm to July 21, 2019 6:00 pm

We have an operational event that requires the shutdown of the Mysql Service database server mysql3.research.partners.org. With this shutdown, the ERIS Infrastructure team will migrate all MySQL databases that reside on mysql3.research.partners.org to mysql4.research.partners.org. The ERIS Infrastructure team will start this process on Friday evening July 19th starting at 6:00 pm and the target completion date is Sunday, July 21, 2019, at 6:00 pm. During this period all databases on mysql3.research.partners.org will be down until they are migrated to mysql4.research.partners.org.

This change will require that all database owners update the MySQL datasource definition on their applications and change the reference from mysql3.research.partners.org to mysql4.research.partners.org.

Please forward all questions to rcc@partners.org.

green - services online

Emergancy Reboot of rfanfs.research.partners.org

July 13, 2019 10:01 pm to 11:01 pm

The ERIS Infrastructure team had to reboot rfanfs.research.partners.org on Saturday morning July 13th. The team started at 10:00am and the reboot and maintenance was completed by 11:00am. All applications and systems using a storage device from rfanfs.research.partners.org are asked to double check their storage device and should there be a problem, please send a note to rcc@partners.org.

green - services online

Partners HealthCare REDCap: Scheduled Downtime

July 12, 2019 3:00 pm

REDCap (https://redcap.partners.org/redcap) upgrades to v8.10.20 scheduled for Tuesday July 16th, 2019 at 6:30AM EST are complete. REDCap was offline from 6:30-6:47 AM EST. 

This release includes a number of bug fixes. 

Release notes can be found here.


IMPORTANT NOTICE: REDCap Database Migration

In order to meet the increased demand, usage, and amount of data stored within the REDCap system, it is necessary to upgrade the database infrastructure.

As a result, a REDCap database migration is scheduled for Saturday July 20th, 2019 at 7:00PM EST.  This will require an extended amount of downtime with REDCap being offline for approximately 14 hours

This will help optimize our REDCap infrastructure, resulting in faster overall performance, and prevent extended downtime for maintenance in the future. 

Additional alerts will be sent next week. 

If you have any concerns about REDCap being offline at either of these times, please email edcsupport@partners.org

green - services online

GitLab: Follow-Up to the Outage on July 11, 2019

July 11, 2019 8:15 am

On July 11, 2019 the GitLab service experienced a significant outage. The service was unavailable between 7:15AM and 12:20PM.

 

What lead to the outage:
An upgrade to GitLab 11.11.5 was scheduled. As part of the upgrade tasks, we apply OS-level patches before beginning the application upgrade process. Some OS-level patches require a restart. In this case, a restart was required. After the patches were installed and reported as having completed successfully, the server was restarted. After the restart, the server was never able to come back up fully and on-screen messages indicated a drive failure/issue.

 

NOTE: The drive failure did not affect the storage location where the GitLab/Git repositories are stored!

 

How we recovered:
As part of our recovery procedure, we created a new Virtual Machine, applied all relevant configuration and re-attached the storage where all GitLab/Git repositories are hosted.

 

NOTE: The drive failure did not affect the storage location where the GitLab/Git repositories are stored! We used the nightly backup of the GitLab database and supporting services to recover the additional configuration needed to bring the service back up.

 

What You should look out for:
Our backup process runs at midnight. Therefore, and data entered within the User Interface of GitLab (issues, Wikis, etc) between Midnight and 7:15 AM may have been lost. This does NOT apply to code committed to the remote repositories. Any code that has been committed to GitLab should not have been affected and no data should have been lost.

 

Since we are operating on a new Virtual Server, those of you who use SSH in order to interact with the server will have to update your "known_hosts". In a lot of cases, you will be prompted with a message along the lines of "Remote Host Identification Has Changed" with some additional text regarding a possible Man-in-the-middle attack. This is an expected side-effect of the migration to a new host - please do not be alarmed. To resolve this issue, please refer to the message that you see on the screen and take note of what "known_hosts" file you are being prompted for. These are typically located in your home folder under the ".ssh" hidden folder, but they can be stored elsewhere. Once you know where the know_hosts (or known_hosts2 on some MACs) is located, open that file with a text editor, find any line that has "gitlab.partners.org" or "gitlab.dipr.partners.org" in it and remove it - remove the whole line of text. Then save the file. NOTE: by modifying this file you do NOT risk damaging any configurations you may have for connections to other servers. This file simply contains the fingerprints of servers you have connected to previously.

 

Please check any automated processes and runners to ensure that they are working as expected. If you experience any issues, please feel free to contact us for assistance at rcc@partners.org with the words GitLab in the somewhere in the subject line.

 

 

---------------------

Additional Notes:
Q: Why did it take so long?
A: The outage was prolonged for a number of reasons. Chief among which was the fact that we had not done a full-scale disaster recovery of the service and had to take careful steps to ensure that no data was lost or corrupted.

 

Q: Are there any up-sides to this crash?
A: Actually, yes. We are now running on new-er Operating System and have more resources provisioned to the machine. We can also now use ED25519 keys for SSH communication with GitLab (as opposed to RSA only previously).

 

Q: Would the upgrade to GitLab 11.11.5 take place soon?
A: Yes. The upgrade to 11.11.5 will be re-scheduled. A separate notification will be sent out for that when we are sure that the new system is fully validated by you - the users.

green - services online

GitLab Upgrade Completed on June 28, 2019

June 28, 2019 7:00 am to 8:00 am

The GitLab service (https://gitlab.partners.org/) upgrade is scheduled for Friday, June 28th, 2019 from 7 to 8 AM. We are upgrading from v11.7.11 to v11.11.3 and the upgrade is scheduled to take up to 1 hour to complete. During this time, the service will be unavailable. This upgrade includes: Critical security patches, Bug fixes, and Minor new features.

For a complete list of changes, please refer to the GitLab project release notes:

Version 11.8 -  https://about.gitlab.com/2019/02/22/gitlab-11-8-released/
Version 11.9 -  https://about.gitlab.com/2019/03/22/gitlab-11-9-released/
Version 11.10 -  https://about.gitlab.com/2019/04/22/gitlab-11-10-released/
Version 11.11 - https://about.gitlab.com/2019/05/22/gitlab-11-11-released/

 

For any questions or concerns regarding this upgrade, please contact rcc@partners.org.

red - services offline, issue, active work

All mysql3 database password will expire on June 26, 2019

June 26, 2019 10:30 am to 12:00 pm

All database passwords on mysql3.research.partners.org will expire on June 26, 2019 at 10:30am which will impact all applications that use mysql3.research.partners.org as a database server. Please send a note to rcc@partners.org and identify your database, the database owner, and the mysql UserID's needed and we will coordinate a updated password for your team.

Please send all comments and questions to rcc@partners.org

Pages