Research IS & Computing website maintenance is scheduled for Wednesday, June 26 between 7-8a.m. EDT. The https://rc.partners.org website will be offline for approximately five minutes during this downtime window to apply security patches and bug fixes.
Applies To: Linux & FreeBSD (SACK Panic)
A new vulnerability has been discovered in the Linux kernel handling TCP Selective Acknowledgments (SACKs). A remote attacker could use this to cause a denial of service attack, interrupting systems operations.
This affects many current distributions being used, as RHEL 4, 6, 7 and 8 and Ubuntu 12.xx to 19.xx (or kernels 2.6.29 and above). This is being tracked as CVE-2019-11477 and is considered important and high impact.
Please note that this might also impact appliances or IoT devices built based on those versions of the Linux kernel.
Some patches are already available and some vendors are still issuing software packages to fix the kernel vulnerability. If you manage Linux or FreeBSD systems please patch asap and reboot accordingly. If you can’t patch a system, please use compensating controls (as sysctl filters disabling tcp probing) as appropriate.
References:
http://packetstormsecurity.com/files/153346/Kernel-Live-Patch-Security-Notice-LSN-0052-1.html
https://access.redhat.com/security/vulnerabilities/tcpsack
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=3b4929f65b0d8249f19a50245cd88ed1a2f78cff
https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md
https://support.f5.com/csp/article/K78234183
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SACKPanic
Security vulnerabilities fixed in Firefox 67.0.3 and Firefox ESR 60.7.1
Please update your Firefox web browser.
Announced: June 18, 2019
Impact: critical
Products
Firefox, Firefox ESR
Fixed in
Firefox 67.0.3
Firefox ESR 60.7.1
The database was successfully migrated to the new server, all systems are green. If you have any questions or concerns, relating to this change, please email us at rcc@partners.org.
On June 23rd at 8 AM we will be migrating our backend database for List Manager (Lyris) from Microsoft SQL 2008 to Microsoft SQL 2016. During this migration, the List Manager web application (https://researchlist.partners.org and https://researchlistadmin.partners.org) will not be available. We will be moving the database from one system to another. (CHG0147108). Any scheduled emails will be sent after the system comes back online. If you have any questions or concerns, relating to this change, please email us at rcc@partners.org.
REDCap (https://redcap.partners.org/redcap) upgrades to v8.10.18 scheduled for Tuesday June 18th, 6:30AM EST are complete. REDCap was offline from 6:28 to 6:44 AM EST.
This upgrade will include a number of new features and bug fixes. These include:
Full REDCap 8.10.18 release notes
New Features and Major Bug Fixes
If you have any concerns about REDCap being offline at this time or any questions about these new features, please email edcsupport@partners.org
Partners Healthcare Freezerworks: Scheduled Downtime
On Thursday June 6th from approximately 6 pm to 6:30 pm (EST), the Freezerworks application will be taken offline for maintenance. During this timeframe, we ask users to log out of the system so we can complete this necessary work. We apologize for any inconvenience during this interruption.
If you have any questions or concerns about email: edcsupport@partners.org
The authentication server backup01.partners.org is up and running. All services are green. If you are having any issues email us at rcc@partners.org.
Currently, our CrashPlan authentication system (backup01.partners.org) is not functioning correctly. We are in the process of rebuilding the system. During this time you will not be able to backup or restore files. Your backup client might report we are out of space, but that is a false message. We will up this alert when everything is resolved.
The ERIS MySQL Service and parts of the DIPR Virtual Machine service experienced problems over the weekend and had to be rebooted. The outages were related to the Compellent patching on Saturday, June 1st however, some work extended into Sunday, June 2nd and Monday, June 3rd. If you experience a problem with a DIPR VM please send a note to rcc@partners.org and the ERIS Infrastructure team will investigate.
Research Computing Ticketing System Now Online
The email component of the internal ticketing system (Kayako) used by ERIS has not been working properly. The issues started Wednesday, May 29 around 12p.m. EDT and are not yet fully resolved.
The system may be delivering several of the same email or none at all. At times, ERIS was not receiving your tickets at all, so slightly longer than usual response times may be experienced.
Affected Services
Emails and requests sent to the following addresses were not accessed during this service disruption:
edcsupport@partners.org - REDCap-generated requests to approve, copy or move projects to production have not been received.
hpcsupport@partners.org
rcc@partners.org
ideasupport@partners.org
Please allow our staff time to catch up on the tickets submitted while the system was offline.
Microsoft has taken the unusual step of releasing sweeping system patches for older or unsupported but widely used Windows operating systems on Tuesday May 14 that include one critical patch that should be applied immediately. Details Here
The Oracle patches scheduled for the Partners Personalized Medicine team on Saturday morning is taking longer than anticipated and will extending past noontime. We anticipate another hour of patching and rebooting as we should complete our efforts by 1:00pm.
Please forward any questions or concerns to rcc@partners.org
The ERIS Infrastructure team will perform maintenance on the PostgreSQL Service on Saturday, April 27, from 9:00 a.m. - 12 p.m. EST.
Impact: All databases hosted on the ERIS PostgreSQL service will be impacted by one or two short reboots during this maintenance window.
Action Required: If you are the owner of a database on the PostgreSQL Service, we encourage you to check your applications after the maintenance window.
Completed
The ERIS Infrastructure team completed the on the MySQL hosted database Service on March 30, 2019. Please forward any questions to rcc@partners.org.
This website may have been offline for a brief period during the maintenance window for MySQL Server patching on Saturday, March 30 between 7:00a.m. - 10:00a.m. EDT. This website, https://rc.partners.org, may have been unavailable for a few minutes while the system is rebooted to apply patches.
Please contact rcwebmaster@partners.org if you have any questions or concerns
We are upgrading the List Manager (Lyris) system to v12.3.5 on Sunday, April 14, 2019, at 9 PM. We expect this release to take, up to an hour, so please expect the List Manager (Lyris) service to be offline from 9:00 to 10:00 PM. If you have any questions or concerns, relating to this upgrade, please email: rcc@partners.org.
Research Computing Ticketing System Online
The MySQL Hosted Database Services outage impacted the internal ticketing system (Kayako) used by ERIS. The outage started Thursday, March 21 around 10p.m. EDT and services restored around 12:15p.m. EDT on Friday, March 22.
Affected Services
Emails and requests sent to the following addresses were not accessed during this unanticipated downtime:
edcsupport@partners.org - REDCap-generated requests to approve, copy or move projects to production have not been received.
hpcsupport@partners.org
rcc@partners.org
ideasupport@partners.org
Please allow our staff time to catch up on the tickets submitted while the system was offline.
RC Services Online
The outage started Thursday, March 21 and we expect services to be restored around 12:30p.m. EDT on Friday, March 22. This downtime is related to the MySQL Hosted Database Services outage.
We will update this message when service is restored.
MySQL Hosted Database Services Online
The outage started Thursday, March 21 around 10p.m. EDT and was restored around 12:00p.m. EDT on Friday, March 22.
Action Required
If you are the owner of a system on mysql.dipr.partners.org, we strongly encourage you to check your application connection and reboot if necessary.
Affected Services
All systems hosted on mysql.dipr.partners.org
Research Computing ticketing system (Kayako tickets.partners.org)
Research Computing ordering/request system (RC Services rcservices.partners.org)
We will update this posting once service has been restored. Please contact rcc@partners.org with any questions or concerns.
REDCap (https://redcap.partners.org/redcap) upgrades to v8.5.28 scheduled for Tuesday March 19th at 6:30AM EDT are complete REDCap was offline from 6:30AM-6:48AM EDT.
This release includes a number of bug fixes.
If you have any concerns about REDCap being offline at this time, please email edcsupport@partners.org
The hpcwin3.partners.org Windows analysis server has presented several freeze instances in the last few days. To evaluate the usage and logs we need to take the server out of service for a few days. During this window time we will perform test and review the information of the processes that have been causing the system to crash. Future actions needed to be taken after for repair or control will be announced to the users.
Existing remote desktop sessions will be terminated. Please save all open files and exit from your remote desktop session as soon as possible. We apologize for the inconvenience this may cause.
