REDCap (https://redcap.partners.org/redcap) server maintenance is scheduled for Saturday, August 24th 2019 from 9:00AM to 10:00AM EST. There will be a short interruption of service during the maintenance window.
If you have any concerns about this scheduled maintenance, please email edcsupport@partners.org
On August 18th from approximately 1:00 am to 5:00 am, Freezerworks will temporarily be offline to complete planned server maintenance work. During this timeframe, we ask users to log out of the system so we can complete this necessary maintenance work. We do apologize for any inconvenience this interruption may cause.
If you have any questions or concerns, please email edcsupport@partners.org
We are upgrading the List Manager (Lyris) system to v12.3.6 on Sunday, August 18, 2019, at 8 PM. We expect this release to take, up to 30 minutes, so please expect the List Manager (Lyris) service to be offline from 8:00 PM to 8:30 PM.
Applies to:
https://researchlistadmin.partners.org
https://researchlist.partners.org
This upgrade was to improve system performance and address defects. The release notes can be found in the Knowledge Base.
Please contact rcc@partners.org with any questions, concerns or issues with the upgrade.
Did you know that Microsoft regularly releases security patches for its software products on the second Tuesday of every month? How frequently do you update your computer?
This month, Microsoft introduces patches for critical flaws and vulnerabilities that require you to patch as soon as possible.
Here is some guidance to keep your device secure and your Microsoft software up to date and running smoothly.
Do I need to make changes on my computer?
See HOWTO: Run Windows Updates for additional instructions.
If you find a Virtual Machine (VM) or website that is not responding, please submit a ticket to the IS Service Desk.
Additional Impacted Services:
| Service Description | Status | Notes |
|---|---|---|
| Impacted Services | ||
| ERIS MySQL hosted database services |  |
|
| Research Computing Website (https://rc.partners.org) | |
Online as of 2:49p.m. EDT on 8/6/19 |
| ERIS Web Hosting Services | |
Online as of 3:47p.m. EDT on 8/6/19 |
| RFANFS |
|
Online as of 2:17p.m. EDT on 8/6/19, Rebooting this system on 8/7/19 at 9p.m. EDT |
| RFAWIN |
|
Online as of 7:00p.m. EDT on 8/6/19 |
| REDCap (https://redcap.partners.org) |
 |
|
| Confluence/JIRA |
|
Online as of 1:00p.m. EDT on 8/6/19 |
| PEAS (https://enroll.partners.org) |
|
Online as of 1:00p.m. EDT on 8/6/19 |
| GitLab (https://gitlab.partners.org) |
|
Online as of 1:00p.m. EDT on 8/6/19 |
| RC Services (https://rcservices.partners.org) |
|
Online as of 5:00p.m. EDT on 8/6/19 |
| DIPR Virtual Machines (VMs) |
|
Online as of 9:45p.m. EDT on 8/7/19 |
| DIPR File Server (fs.dipr) |
|
Online as of 9:45p.m. EDT on 8/7/19 |
If you find a Virtual Machine (VM) or website that is not responding, please submit a ticket to the IS Service Desk.
Applies to: Staff and Visitors at Assembly Row
The Network Access Control (NAC) program updates scheduled for Assembly Row on Tuesday, August 13 are canceled at this time.
Partners HealthCare continues to enhance the Network Access Control (NAC) program at Assembly Row, strengthening visibility and security of data and devices connected to the Partners network.
Please review the security requirements and guidance for your device, by visiting rc.partners.org/computer.
Thank you in advance.
Jigar Kadakia
Partners Chief Information Security & Privacy Officer
CISPO@partners.org
Beginning July 25, 2019, CrowdStrike will be pushed to your corporate/Partners-purchased Macintosh device. Please note the date was moved one week due to the Enterprise Change Freeze.
It will be required to connect to the Partners Corporate network. CrowdStrike is an additional endpoint security tool application that complements antivirus (AV). Both CrowdStrike and antivirus are required to keep your systems safe and secure. Trend Micro antivirus is available for installation on all Mac and Windows devices used for Partners business purposes, including research. Please see Supported Antivirus Software for a full list.
CrowdStrike end point protection and Trend Micro antivirus are available for download from the Partners Enterprise Apple Support (PEAS) Self Service application.
CrowdStrike can also be self-installed prior to the automatic push using the directions below.
Please note: Apple released the public beta version for the next operating system for Mac desktops & laptops, macOS Catalina (10.15). We recommend you do not upgrade at this time. Traditionally new Mac Operating Systems will initially cause compatibility issues with Partners services.
To download CrowdStrike via Self Service:
View detailed installation instructions on our Knowledge Base.
For assistance or questions, contact the IS Service Desk and ask to submit a ticket to the Engineering Security Team.
Thank you in advance.
Jigar Kadakia
Partners Chief Information Security & Privacy Officer
399 Revolution Dr, Somerville, MA 02145
This is an update on the ERIS Infrastructure teams efforts to move all databases off of mysql3.research.partners.org and restore them on mysql4.research.partners.org. The team has worked through 50% of the efforts needed to complete this effort and is working diligently on all remaining tasks. We will share another update later tonight and will provide more detail concerning availability.
The final step in this process belongs to the database owners who need to change their MySQL Data Source reference from mysql3.research.partners.org to mysql4.research.partners.org.
Please send a note to rcc@partners.org with any questions.
IMPORTANT NOTICE: REDCap Database Migration
In order to meet the increased demand, usage, and amount of data stored within the REDCap system, it is necessary to upgrade the database infrastructure.
As a result, a REDCap database migration is scheduled for Saturday July 20th, 2019 at 7:00PM EST. This will require an extended amount of downtime with REDCap being offline for approximately 14 hours.
REDCap was offline from 7:00pm-5:55am.
This will help optimize our REDCap infrastructure, resulting in faster overall performance, and prevent extended downtime for maintenance in the future.
If you have any concerns about REDCap being offline at either of these times, please email edcsupport@partners.org
Research IS & Computing website maintenance completed on Wednesday, July 17 from 7-7:30a.m. EDT. The https://rc.partners.org website was offline from 7:13-7:20a.m. EDT for a database migration. Please see Mysql3 Shutdown and Retirement for details.
We have an operational event that requires the shutdown of the Mysql Service database server mysql3.research.partners.org. With this shutdown, the ERIS Infrastructure team will migrate all MySQL databases that reside on mysql3.research.partners.org to mysql4.research.partners.org. The ERIS Infrastructure team will start this process on Friday evening July 19th starting at 6:00 pm and the target completion date is Sunday, July 21, 2019, at 6:00 pm. During this period all databases on mysql3.research.partners.org will be down until they are migrated to mysql4.research.partners.org.
This change will require that all database owners update the MySQL datasource definition on their applications and change the reference from mysql3.research.partners.org to mysql4.research.partners.org.
Please forward all questions to rcc@partners.org.
The ERIS Infrastructure team had to reboot rfanfs.research.partners.org on Saturday morning July 13th. The team started at 10:00am and the reboot and maintenance was completed by 11:00am. All applications and systems using a storage device from rfanfs.research.partners.org are asked to double check their storage device and should there be a problem, please send a note to rcc@partners.org.
REDCap (https://redcap.partners.org/redcap) upgrades to v8.10.20 scheduled for Tuesday July 16th, 2019 at 6:30AM EST are complete. REDCap was offline from 6:30-6:47 AM EST.
This release includes a number of bug fixes.
Release notes can be found here.
IMPORTANT NOTICE: REDCap Database Migration
In order to meet the increased demand, usage, and amount of data stored within the REDCap system, it is necessary to upgrade the database infrastructure.
As a result, a REDCap database migration is scheduled for Saturday July 20th, 2019 at 7:00PM EST. This will require an extended amount of downtime with REDCap being offline for approximately 14 hours.
This will help optimize our REDCap infrastructure, resulting in faster overall performance, and prevent extended downtime for maintenance in the future.
Additional alerts will be sent next week.
If you have any concerns about REDCap being offline at either of these times, please email edcsupport@partners.org
On July 11, 2019 the GitLab service experienced a significant outage. The service was unavailable between 7:15AM and 12:20PM.
What lead to the outage:
An upgrade to GitLab 11.11.5 was scheduled. As part of the upgrade tasks, we apply OS-level patches before beginning the application upgrade process. Some OS-level patches require a restart. In this case, a restart was required. After the patches were installed and reported as having completed successfully, the server was restarted. After the restart, the server was never able to come back up fully and on-screen messages indicated a drive failure/issue.
NOTE: The drive failure did not affect the storage location where the GitLab/Git repositories are stored!
How we recovered:
As part of our recovery procedure, we created a new Virtual Machine, applied all relevant configuration and re-attached the storage where all GitLab/Git repositories are hosted.
NOTE: The drive failure did not affect the storage location where the GitLab/Git repositories are stored! We used the nightly backup of the GitLab database and supporting services to recover the additional configuration needed to bring the service back up.
What You should look out for:
Our backup process runs at midnight. Therefore, and data entered within the User Interface of GitLab (issues, Wikis, etc) between Midnight and 7:15 AM may have been lost. This does NOT apply to code committed to the remote repositories. Any code that has been committed to GitLab should not have been affected and no data should have been lost.
Since we are operating on a new Virtual Server, those of you who use SSH in order to interact with the server will have to update your "known_hosts". In a lot of cases, you will be prompted with a message along the lines of "Remote Host Identification Has Changed" with some additional text regarding a possible Man-in-the-middle attack. This is an expected side-effect of the migration to a new host - please do not be alarmed. To resolve this issue, please refer to the message that you see on the screen and take note of what "known_hosts" file you are being prompted for. These are typically located in your home folder under the ".ssh" hidden folder, but they can be stored elsewhere. Once you know where the know_hosts (or known_hosts2 on some MACs) is located, open that file with a text editor, find any line that has "gitlab.partners.org" or "gitlab.dipr.partners.org" in it and remove it - remove the whole line of text. Then save the file. NOTE: by modifying this file you do NOT risk damaging any configurations you may have for connections to other servers. This file simply contains the fingerprints of servers you have connected to previously.
Please check any automated processes and runners to ensure that they are working as expected. If you experience any issues, please feel free to contact us for assistance at rcc@partners.org with the words GitLab in the somewhere in the subject line.
---------------------
Additional Notes:
Q: Why did it take so long?
A: The outage was prolonged for a number of reasons. Chief among which was the fact that we had not done a full-scale disaster recovery of the service and had to take careful steps to ensure that no data was lost or corrupted.
Q: Are there any up-sides to this crash?
A: Actually, yes. We are now running on new-er Operating System and have more resources provisioned to the machine. We can also now use ED25519 keys for SSH communication with GitLab (as opposed to RSA only previously).
Q: Would the upgrade to GitLab 11.11.5 take place soon?
A: Yes. The upgrade to 11.11.5 will be re-scheduled. A separate notification will be sent out for that when we are sure that the new system is fully validated by you - the users.
The GitLab service (https://gitlab.partners.org/) upgrade is scheduled for Friday, June 28th, 2019 from 7 to 8 AM. We are upgrading from v11.7.11 to v11.11.3 and the upgrade is scheduled to take up to 1 hour to complete. During this time, the service will be unavailable. This upgrade includes: Critical security patches, Bug fixes, and Minor new features.
For a complete list of changes, please refer to the GitLab project release notes:
Version 11.8 - https://about.gitlab.com/2019/02/22/gitlab-11-8-released/
Version 11.9 - https://about.gitlab.com/2019/03/22/gitlab-11-9-released/
Version 11.10 - https://about.gitlab.com/2019/04/22/gitlab-11-10-released/
Version 11.11 - https://about.gitlab.com/2019/05/22/gitlab-11-11-released/
For any questions or concerns regarding this upgrade, please contact rcc@partners.org.
All database passwords on mysql3.research.partners.org will expire on June 26, 2019 at 10:30am which will impact all applications that use mysql3.research.partners.org as a database server. Please send a note to rcc@partners.org and identify your database, the database owner, and the mysql UserID's needed and we will coordinate a updated password for your team.
Please send all comments and questions to rcc@partners.org
REDCap (https://redcap.partners.org/redcap) requires server maintenance. REDCap will be offline tonight Tuesday June 25th, 2019 from approximately 9:00-9:30PM EST.
There will be a short interruption of service during the maintenance window.
If you have any concerns, please email edcsupport@partners.org
Research IS & Computing website maintenance completed on Wednesday, June 26. The https://rc.partners.org website was offline during from 7:15-7:45a.m. EDT to apply security patches and bug fixes.
Applies To: Linux & FreeBSD (SACK Panic)
A new vulnerability has been discovered in the Linux kernel handling TCP Selective Acknowledgments (SACKs). A remote attacker could use this to cause a denial of service attack, interrupting systems operations.
This affects many current distributions being used, as RHEL 4, 6, 7 and 8 and Ubuntu 12.xx to 19.xx (or kernels 2.6.29 and above). This is being tracked as CVE-2019-11477 and is considered important and high impact.
Please note that this might also impact appliances or IoT devices built based on those versions of the Linux kernel.
Some patches are already available and some vendors are still issuing software packages to fix the kernel vulnerability. If you manage Linux or FreeBSD systems please patch asap and reboot accordingly. If you can’t patch a system, please use compensating controls (as sysctl filters disabling tcp probing) as appropriate.
References:
http://packetstormsecurity.com/files/153346/Kernel-Live-Patch-Security-Notice-LSN-0052-1.html
https://access.redhat.com/security/vulnerabilities/tcpsack
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=3b4929f65b0d8249f19a50245cd88ed1a2f78cff
https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md
https://support.f5.com/csp/article/K78234183
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic
Security vulnerabilities fixed in Firefox 67.0.3 and Firefox ESR 60.7.1
Please update your Firefox web browser.
Announced: June 18, 2019
Impact: critical
Products
Firefox, Firefox ESR
Fixed in
Firefox 67.0.3
Firefox ESR 60.7.1
The database was successfully migrated to the new server, all systems are green. If you have any questions or concerns, relating to this change, please email us at rcc@partners.org.
On June 23rd at 8 AM we will be migrating our backend database for List Manager (Lyris) from Microsoft SQL 2008 to Microsoft SQL 2016. During this migration, the List Manager web application (https://researchlist.partners.org and https://researchlistadmin.partners.org) will not be available. We will be moving the database from one system to another. (CHG0147108). Any scheduled emails will be sent after the system comes back online. If you have any questions or concerns, relating to this change, please email us at rcc@partners.org.
REDCap (https://redcap.partners.org/redcap) upgrades to v8.10.18 scheduled for Tuesday June 18th, 6:30AM EST are complete. REDCap was offline from 6:28 to 6:44 AM EST.
This upgrade will include a number of new features and bug fixes. These include:
Full REDCap 8.10.18 release notes
New Features and Major Bug Fixes
If you have any concerns about REDCap being offline at this time or any questions about these new features, please email edcsupport@partners.org
Partners Healthcare Freezerworks: Scheduled Downtime
On Thursday June 6th from approximately 6 pm to 6:30 pm (EST), the Freezerworks application will be taken offline for maintenance. During this timeframe, we ask users to log out of the system so we can complete this necessary work. We apologize for any inconvenience during this interruption.
If you have any questions or concerns about email: edcsupport@partners.org
The authentication server backup01.partners.org is up and running. All services are green. If you are having any issues email us at rcc@partners.org.
Currently, our CrashPlan authentication system (backup01.partners.org) is not functioning correctly. We are in the process of rebuilding the system. During this time you will not be able to backup or restore files. Your backup client might report we are out of space, but that is a false message. We will up this alert when everything is resolved.
