Applies to: All DIPR Servers

Production servers hosted by ERIS were patched on Saturday, September 28 starting at 9AM. If you are the owner of a server hosted by ERIS, we encourage you to connect to your server at the end of our maintenance, verify that your applications restart and reconnect to your database. 

The Discovery Informatics Platform for Research (DIPR) platform hosted by Enterprise Research IS is undergoing several essential upgrades in Fall 2019 to elevate security standards. These changes will include increased auditing, enabling SELinux on all systems and patching of the Linux OS on all servers on a monthly basis. The ERIS Infrastructure team will communicate each change two weeks before deploying. Please subscribe to receive DIPR communications and visit https://rc.partners.org/subscribe for more information. Non-production DIPR servers were patched in waves on Wednesdays in September. 

We understand that this is a significant change. We need your help to ensure an easy transition for your servers, applications and users. Learn more about the new DIPR Servers Monthly OS Patching Schedule.

Service Level Reminders
To ensure premium supportability, security and performance, ERIS strives to keep all systems up to their most current and/or stable revisions. Please visit https://rc.partners.org/sla to review the full ERIS Service Level Agreement.  

Server owners are responsible for maintaining the applications running on DIPR VMs systems and ensuring they are patched and comply with Partners Information Security Practices and Policies. Server owners are responsible for backup of critical configurations in case of a failure.

Please reach out to rcc@partners.org with any questions.

This website was offline for a brief period during the maintenance window for MySQL Server patching on Saturday, August 24 between 9:00a.m. - Noon. EDT. This website, https://rc.partners.org, was unavailable for a few minutes while the system is rebooted to apply patches.

Please contact rcwebmaster@partners.org if you have any questions or concerns.

On August 18th from approximately 1:00 am to 5:00 am, Freezerworks will temporarily be offline to complete planned server maintenance work. During this timeframe, we ask users to log out of the system so we can complete this necessary maintenance work. We do apologize for any inconvenience this interruption may cause. 

If you have any questions or concerns, please email edcsupport@partners.org

Did you know that Microsoft regularly releases security patches for its software products on the second Tuesday of every month? How frequently do you update your computer?

This month, Microsoft introduces patches for critical flaws and vulnerabilities that require you to patch as soon as possible.

Here is some guidance to keep your device secure and your Microsoft software up to date and running smoothly.

Do I need to make changes on my computer?

• Not if you use a Partners standard Windows computer (those with the asset tag, yellow lock and Partners' screensaver). The updates are run for you and devices are usually patched the Friday following Patch Tuesday. Both Microsoft and Adobe products are typically updated at this time. This can change if there is a significant freeze in place. 
• Yes, you need to run updates if you use a non-standard Windows, Macintosh or personally-owned device for Partners' business. ​It is your responsibility to keep your device and software up to date. 

See HOWTO: Run Windows Updates for additional instructions.

Applies to:        Staff and Visitors at Assembly Row

The Network Access Control (NAC) program updates scheduled for Assembly Row on Tuesday, August 13 are canceled at this time. 

Partners HealthCare continues to enhance the Network Access Control (NAC) program at Assembly Row, strengthening visibility and security of data and devices connected to the Partners network.  

Please review the security requirements and guidance for your device, by visiting rc.partners.org/computer. 

Thank you in advance.

Jigar Kadakia
Partners Chief Information Security & Privacy Officer
CISPO@partners.org

This is an update on the ERIS Infrastructure teams efforts to move all databases off of mysql3.research.partners.org and restore them on mysql4.research.partners.org. The team has worked through 50% of the efforts needed to complete this effort and is working diligently on all remaining tasks. We will share another update later tonight and will provide more detail concerning availability.

The final step in this process belongs to the database owners who need to change their MySQL Data Source reference from mysql3.research.partners.org to mysql4.research.partners.org.

Please send a note to rcc@partners.org with any questions.

Research IS & Computing website maintenance completed on Wednesday, July 17 from 7-7:30a.m. EDT. The https://rc.partners.org website was offline from 7:13-7:20a.m. EDT for a database migration. Please see Mysql3 Shutdown and Retirement for details.

The ERIS Infrastructure team had to reboot rfanfs.research.partners.org on Saturday morning July 13th. The team started at 10:00am and the reboot and maintenance was completed by 11:00am. All applications and systems using a storage device from rfanfs.research.partners.org are asked to double check their storage device and should there be a problem, please send a note to rcc@partners.org.

On July 11, 2019 the GitLab service experienced a significant outage. The service was unavailable between 7:15AM and 12:20PM.

What lead to the outage:
An upgrade to GitLab 11.11.5 was scheduled. As part of the upgrade tasks, we apply OS-level patches before beginning the application upgrade process. Some OS-level patches require a restart. In this case, a restart was required. After the patches were installed and reported as having completed successfully, the server was restarted. After the restart, the server was never able to come back up fully and on-screen messages indicated a drive failure/issue.

NOTE: The drive failure did not affect the storage location where the GitLab/Git repositories are stored!

How we recovered:
As part of our recovery procedure, we created a new Virtual Machine, applied all relevant configuration and re-attached the storage where all GitLab/Git repositories are hosted.

NOTE: The drive failure did not affect the storage location where the GitLab/Git repositories are stored! We used the nightly backup of the GitLab database and supporting services to recover the additional configuration needed to bring the service back up.

What You should look out for:
Our backup process runs at midnight. Therefore, and data entered within the User Interface of GitLab (issues, Wikis, etc) between Midnight and 7:15 AM may have been lost. This does NOT apply to code committed to the remote repositories. Any code that has been committed to GitLab should not have been affected and no data should have been lost.

Since we are operating on a new Virtual Server, those of you who use SSH in order to interact with the server will have to update your "known_hosts". In a lot of cases, you will be prompted with a message along the lines of "Remote Host Identification Has Changed" with some additional text regarding a possible Man-in-the-middle attack. This is an expected side-effect of the migration to a new host - please do not be alarmed. To resolve this issue, please refer to the message that you see on the screen and take note of what "known_hosts" file you are being prompted for. These are typically located in your home folder under the ".ssh" hidden folder, but they can be stored elsewhere. Once you know where the know_hosts (or known_hosts2 on some MACs) is located, open that file with a text editor, find any line that has "gitlab.partners.org" or "gitlab.dipr.partners.org" in it and remove it - remove the whole line of text. Then save the file. NOTE: by modifying this file you do NOT risk damaging any configurations you may have for connections to other servers. This file simply contains the fingerprints of servers you have connected to previously.

Please check any automated processes and runners to ensure that they are working as expected. If you experience any issues, please feel free to contact us for assistance at rcc@partners.org with the words GitLab in the somewhere in the subject line.

---------------------

Additional Notes:
Q: Why did it take so long?
A: The outage was prolonged for a number of reasons. Chief among which was the fact that we had not done a full-scale disaster recovery of the service and had to take careful steps to ensure that no data was lost or corrupted.

Q: Are there any up-sides to this crash?
A: Actually, yes. We are now running on new-er Operating System and have more resources provisioned to the machine. We can also now use ED25519 keys for SSH communication with GitLab (as opposed to RSA only previously).

Q: Would the upgrade to GitLab 11.11.5 take place soon?
A: Yes. The upgrade to 11.11.5 will be re-scheduled. A separate notification will be sent out for that when we are sure that the new system is fully validated by you - the users.

All database passwords on mysql3.research.partners.org will expire on June 26, 2019 at 10:30am which will impact all applications that use mysql3.research.partners.org as a database server. Please send a note to rcc@partners.org and identify your database, the database owner, and the mysql UserID's needed and we will coordinate a updated password for your team.

Please send all comments and questions to rcc@partners.org

Research IS & Computing website maintenance completed on Wednesday, June 26. The https://rc.partners.org website was offline during from 7:15-7:45a.m. EDT to apply security patches and bug fixes.

Security vulnerabilities fixed in Firefox 67.0.3 and Firefox ESR 60.7.1

Please update your Firefox web browser.

1. Open Firefox
2. Select About Firefox
3. Select Update

Announced: June 18, 2019
Impact: critical
Products
Firefox, Firefox ESR

Fixed in
Firefox 67.0.3
Firefox ESR 60.7.1

Learn More