Service Alerts

yellow - pre-downtime, calls to action

SSH (Secure Shell) Gateway Server Storage Quota

January 15, 2015 8:00 am

SSH (Secure Shell) Gateway Server - 1GB Storage Quota effective January 15, 2016

This is a reminder that the SSH server is to be used for temporary file staging only and not for storage. Each account is allotted 100MB of storage. Please remove any files that are no longer in use in the SSH server home folders. Beginning Friday, January 15, 2016, the hard quota for storage on the SSH Gateway Server will be set to 1GB.

In 2016, the SSH service will be migrated to a new ResLink platform that allows for security enhancements and a hardware refresh. The hard quota storage limit will be 100MB on the new platform. If you are currently using more than 100MB of storage, please begin to move your files to a separate location. Additional details will be sent regarding the new platform in 2016.

If you have any questions or concerns regarding data storage options at Partners, please visit our website or contact ERIS at rcc@partners.org.   

yellow - pre-downtime, calls to action

Lyris List Manager: Known Issues

November 2, 2015 4:32 pm

We are aware of several issues that you might experience with the Lyris List Manager system (https://researchlist.partners.org). The following issues will be resolved with the next upgrade. 

  • The Image Library does not support an image located at an SSL site
    • Workaround: Contact rcc@partners.org and we can upload your images to the server.
  • List Manager not working in Firefox browser version 39
  • Paste from word option in the HTML editor not working with IE 11
    • Workaround: Use Firefox, Chrome or Safari to use the paste from word option
  • Unable to add a clickable link in a mailing or create a merge tag, when using Windows 7 with IE11
    • Workaround: Use Firefox, Chrome or Safari to create hyperlink or merge tag

The exact upgrade schedule will be posted and communicated to List Administrators as soon as it is confirmed. If you have any questions or additional issues, please email rcc@partners.org and a ticket will be opened for you.

 

green - services online

StudyTRAX Database Issue

October 30, 2015 9:25 am - 10:19 am

The StudyTRAX database experienced an issue and required a restart. StudyTRAX was unavailable from 9:25 - 10:19 AM, October 30, 2015.

The cause of the issue is still being investigated.

green - services online

MySQL connection issues

October 26, 2015 9:50 am - 10:27 am

Applies to: MySQL instance - MySQL2.dipr.partners.org

MySQL.dipr.partners.org experienced a heavy load from 9:50 am to 10:27 am this morning, October 26, 2015, which caused MySQL to be non responsive. MySQL is now running under normal load, and has no issues to report. 

We will be investigating the cause of disruption of service this morning and work to rectify the issue. If you are still experiencing connecting issues, please email us at rcc@partners.org, and we will assist you with your connection issues.

green - services online

REDCap Upgrade 6.5.15

October 28, 2015 6:00 am - 6:38 am

Partners Healthcare REDCap: Upgrades Complete

REDCap (https://redcap.partners.org/redcap) has been upgraded to v6.5.15 on Wednesday, Oct. 28.  Systems were offline from 6:00 AM - 6:38 AM ET.

This upgrade included a number of minor bug fixes.

If you have any concerns about REDCap being offline at this time, please email edcsupport@partners.org 

green - services online

Policy Central is live

November 17, 2015 8:00 am

Partners HealthCare is introducing a new policy and procedures website, Policy Central. This replaces Trove Library. All active policies for Partners HealthCare, Spaulding Rehabilitation Network, and Partners Community Physicians Organization have been migrated to Policy Central.              

How do I access Policy Central?

  • From a Partners-managed Windows device, Go to Partners Applications > Policies and Procedures > Partners Policies and Procedures > You are redirected to Policy Central > Select your institution from dropdown menu.
    • Note: If you do not see Partners Policies and Procedures in your Partners Applications right away, please log off and log back in.
  • From a Mac or a non-standard Partners Windows device, visit: https://grcarcher.partners.org/

Users must have a Partners network ID and password in order to access Policy Central. More information and a user guide is available in the IS Service Desk: https://partnershealthcare.service-now.com/kb_view.do?sysparm_article=KB0023481.   

If you have any questions, please contact the IS Service Desk.

Thank you,

Jigar Kadakia
Chief Information Security and Privacy Officer

green - services online

Monthly IS Maintenance (ISM)

October 19, 2015 2:00 am - 4:00 am

Applies To: All Users

Information Systems Maintenance (ISM) was performed on Monday, October 19 from 2:00 am to 4:00 am.

During this time, planned system maintenance and testing were performed by IS Operations teams. This routine maintenance reduces the risk of unscheduled outages on our computer network and to the Internet.

Impact to systems and applications

  • VPN should be available during the ISM.
  • All Cache Classis applications, except web paging, will be down from 2:00 am to 4:00 am.
  • The Specific applications within Cache Classic that will be down are listed below:
    • Partners-wide: LMR, Medication and Allergy Services, COE (Chemo Order Entry), CAS, EMPI
    • BWH: BICS, Historical Order and eMAR Viewer
    • MGH: eMAR, MGH POE and MGH EDIS, MGH OR Applications
    • All Cache Classic and Ensemble data interfaces and web services will be unavailable from 2:00am-4:00am.

Impact to sites

  • Services or resources hosted in the Partners HealthCare data centers may be unavailable during the ISM.
  • Your site's IS Management team will provide more detailed communication about planned outages at your facility and will coordinate with specific groups that may be impacted.

Need help during the downtime? 

View the Broadcast Online. Please contact your IS Service Desk if you have any questions.

View the 2015 IS System Maintenance (ISM) Schedule in the IS Service Desk Knowledgebase (Partners login required).

red - services offline, issue, active work

RISC Site Team Offices Closed on Tues 10/13/2015

October 13, 2015 8:00 am - 5:00 pm

RISC Site Team Offices are closed on Tues 10/13/2015. Urgent Issue? Contact IS Service Desk: http://www.partners.org/isservicedesk

Closures include:

  • BWH (RICS) Site Team at 221 Longwood
  • MGH Site Team at Bartlett Hall Extension
  • McLean Site Team at the North Belknap Building

 

green - services online

MAD Data Storage: Gateway Reboot Completed

October 6, 2015 1:00 pm - 3:45 pm

The reboot of all 3 gateways has completed successfully and it appears as though normal behavior has returned. Please test your normal use cases to ensure that any problems you may have been experiencing are no longer occurring.

MAD engineering will continue to search for root cause to these latest symptoms and for the original problem from Wednesday, but it does appear that any lingering issues resulting from the remount of storage last week are now ceased.  We will look to more closely identify what was wrong so as to avoid a full reboot in the future if at all possible.

We understand there are a few outstanding issues that are in the queue for completion, and with the urgent issues now cleared, MAD engineering will address those open items this afternoon and complete them as soon as possible.

Thank you very much for your patience while we sorted this out!


As you may be aware, the MAD infrastructure encountered issues last Wednesday for customers using non-replicated accounts (see below). We believe we isolated and corrected that issue, but there have been lingering symptoms stemming from correction of this issue that we cannot keep cleared.

We rebooted the gateway servers during the lunch hour on Tuesday, October 6, 2015, as many customers were not able to write to their respective shares.

We advised that if you had any current jobs running against your MAD shares, to please suspend them.

Thanks for your patience while we alleviate this problem!


From Wednesday, September 30, 2015:

Nonreplicated shares on the MAD (Massive Array of Disks) data storage service were reported unavailable yesterday afternoon. Service was fully restored on Thursday, October 1 at approximately 11a.m. EDT. Check any transfers that ran from Wednesday, September 30, 2015 afternoon to the morning of Thursday, October 1, and restart the transfer if necessary.

ERIS technicians are actively exploring the cause of the issue. If you experience any issues with the MAD data storage service, please contact rcc@partners.org.

green - services online

Mac OS X 10.11 El Capitan in the Enterprise

September 29, 2015 11:45 am

The Partners Enterprise Apple Support (PEAS) team is pleased to inform you that testing is complete and you may now consider an upgrade to Mac OS X 10.11 El Capitan. Note: Apple, and therefore PEAS, supports the current OS and two prior versions. If you’re running Mac OS X 10.8 Mountain Lion or older, you need to upgrade. When upgrading, please review the El Capitan Guide and check the compatibility chart to ensure software you may use is compatible.

The Symantec Enterprise Vault plug-in for Mac Outlook is not compatible with Outlook 2016. The vendor has decide to discontinued future support of Outlook for Mac. You can still access your archived email through the web interface by navigating to https://archive.partners.org/enterprisevault.

If you are running Mac OS X 10.10 or higher on your Mac, you can also install Microsoft Office 2016. Please note that the Enterprise Vault plug-in are not compatible with Microsoft Office 2016. 

To request Microsoft Outlook 2016 

  1. Call the Partners IS Service Desk and request an Office 365 subscription.
  2. Follow step by step instructions in the KnowledgeBase article, HOWTO: Install Microsoft Office 2016 for Mac

Outlook 2016 along with additional updated software, including a Java update, are available in Self Service.

For any questions on El Capitan or technical assistance with upgrading, please submit a request for assistance to the Partners IS Service Desk.

green - services online

Confluence Downtime

September 27, 2015 8:00 am - 10:00 am

Maintenance and upgrades were performed on Confluence (https://confluence.partners.org) on Sunday, September 27 at 8:00a.m. If you have any questions or concerns, please contact rcc@partners.org.

Confluence is an Enterprise Wiki and Collaboration tool used by ERIS and Research for documentation, team calendaring and information dissemination.

green - services online

Monthly IS Maintenance (ISM)

September 28, 2015 12:00 am - 4:00 am

Applies To: All Users

Information Systems Maintenance (ISM) took place on Monday, September 28 from 12:00 am to 4:00 am.

During this time, planned system maintenance and testing were performed by IS Operations teams. This routine maintenance reduces the risk of unscheduled outages on our computer network and to the Internet.

Impact to systems and applications

  • VPN should be available during the ISM.
  • Partners eCare - unavailable from 1:00am until 4:00am
    1. Impacts all PeC applications and related interfaces for MGH, BWH, BWFH, DFCI, PHH, NWH and Newton Wellesley Ambulatory Services.
    2. Impacts PeC dependent applications including Patient Gateway.

Impact to sites

  • Services or resources hosted in the Partners HealthCare data centers may be unavailable during the ISM.
  • Your site's IS Management team will provide more detailed communication about planned outages at your facility and will coordinate with specific groups that may be impacted.

Need help during the downtime? 

View the Broadcast Online. Please contact your IS Service Desk if you have any questions.

View the 2015 IS System Maintenance (ISM) Schedule in the IS Service Desk Knowledgebase (Partners login required).

yellow - pre-downtime, calls to action

MySQL connection issues

September 21, 2015 8:00 pm

Applies to: MySQL instance - MySQL2.dipr.partners.org

A minor connection refusal on mysql2.dipr.partners.org may have caused a small outage for connections to the MySQL instance from around 8p.m. to 9p.m. EST on Monday, September 21, 2015. A hung maintenance job created a backlog of connections, that ultimately created a refusal of new database connections. Once the maintenance job was canceled, connections resumed as normal.

The ERIS database team will investigate the timing of the maintenance job, and attempt to prevent this issue from happening in future runs.

On Tuesday, Sept. 29 from 12:30 - 1:30p.m. EST a small outage for connections to MySQL2.dipr.partners.org also occurred. the server experienced a higher than usual number of concurrent, sustained connections that increased the system load and temporarily maxed out the number of allowed connections. In most cases connections were allowed and the server was queuing up commands, but the execution of those commands took much longer than expected. All site are currently connecting, but the team is continuing to investigate how to prevent a reoccurrence.

If you have any questions, please contact rcc@partners.org, and a ticket will be generated automatically to our database team.

yellow - pre-downtime, calls to action

REDCap Service Alert: Limited Support Sep. 28 - Oct. 1

September 28, 2015 9:00 am - October 1, 2015 5:00 pm

There will be limited support next week while EDC Support attends the annual REDCap Conference. Please plan for delayed response times.  Priorities will be made for Moving Projects to Production, User Access, Creating New Projects, and Change Requests.  General REDCap questions will be best effort. 

Be sure to review the Help&FAQTraining ResourcesGetting Started with REDCap and Knowledge Base pages.  

If you have any questions, please email edcsupport@partners.org.

yellow - pre-downtime, calls to action

REDCap: Known Issues With Upgrade to v6.5

September 17, 2015 1:28 pm

With the recent upgrade to REDCap v6.5.9, we are aware of a number of issues that users are experiencing. These include: 

  • Script errors when creating reports: Some users, especially those using Internet Explorer, have run into script issues where their browsers freeze when creating a report by adding entire instruments. It seems the issue is tied to the number of variables in the project and the number of records. Other browsers, such as Chrome tend to work better. 
  • Issues when moving between authenticated plugins/project bookmarks: If a user selects an authenticated plugin/project bookmark and selects a second one, they receive an error message. The work around for this is to navigate to a different page, and then select the next plugin/bookmark. 
  • REDCap delayed login: As most of you have probably noticed, it now takes 15-20 seconds to log into REDCap. 

We are investigating these issues and will let you know once they are resolved. The resolution will most likely involve an upgrade to apply bug fixes. Schedule to be determined.

If you have any questions, please email edcsupport@partners.org 

yellow - pre-downtime, calls to action

New VPN Client

September 23, 2015 11:00 am

The Partners Network Security team will upgrade the VPN client software (anyconnect) to version 3.1.10010 that is hosted on the VPN servers (www.partners.org/vpn). After 11a.m. on Wednesday, September 23, individuals will be prompted to upgrade the software on their computers when connecting to VPN.

This new client supports these Operating Systems and versions:

  • Windows 7, 8, 10*
  • Mac OS X 10.8, 10.9, 10.10, 10.11

* To use VPN on Windows 10, you will need use IE 11 IE browser icon and not the new IE Edge Edge browser iconbrowser. You will also need to have the latest version of Java installed. 

Contact the IS Service Desk if you have any questions or concerns.

green - services online

StudyTRAX Infrastructure Updates

September 10, 2015 9:00 pm - September 11, 2015 12:00 am

StudyTRAX was offline September 10 at 9pm - 12 midnight for SQL database migration.  Migration was successful.

If you have any concerns, please email edcsupport@partners.org

 

yellow - pre-downtime, calls to action

Internet Explorer 11 Upgrade

September 1, 2015 8:00 am

Applies To: All Users

During September and October, workstations across the Partners network will be upgraded to Internet Explorer 11 (IE11), the most current version available from Microsoft.  IE11 allows a greater number of websites to display and work properly while also improving the security of our network.

Stay tuned for additional details from your site’s IS Planning and Management team, including a specific rollout schedule. If you have any questions, please contact the IS Service Desk.

What do I need to do?

Please be sure to save your work and properly log off your workstation at the end of each day until you have received the IE11 upgrade: 

  1. Save your work and close all applications.
  2. Click the Yellow Lock.
  3. Click New User.

How will the upgrade work?

Workstations that are powered on and connected to the Partners network and logged out will receive IE 11 during designated times. If a workstation is logged in during the automated process, it will be skipped. Any workstations that are skipped will need to be upgraded at a later date.

When a device is upgraded to IE11, it should automatically reboot after the upgrade. If the device does not automatically reboot at the end of the upgrade, you may need to reboot it manually.

How to determine your Partners device build type: 

  • When logged out, the device build type is listed at the top of the Partners screensaver. Example

 . Note: Use the F1 key from the screensaver for additional Device information.

  • When logged in, Go to Partners Applications > Utilities > Desktop Information. Under Workstation Configuration, look for Workstation Type. 

IE 11 Self Install Process for Laptops

Internet Explorer 11 (IE11) is now available for self-installation on laptop computers as well as on private-build workstations that have not yet received the IE11 upgrade via the ongoing automated process. This allows you to install IE11 at your convenience. This does not apply to MGH or DFCI.

  • How do I install IE11?
    • You must be connected to the Partners network in the office or via VPN to run the install. View Installation Instructions (You will need to log in with your Partners Username and Password)

What’s new in IE11?

IE11 is the most current version available from Microsoft. It allows a greater number of websites to display and work properly while also improving the security of our network. Learn more

ServiceNow User Interface Changes in Internet Explorer 11

When Internet Explorer 8 is upgraded to Internet Explorer 11 on Partners standard builds, you’ll notice some minor changes in the user interface (UI). Complete details of all of the user interface changes can be found at the following web page. Note, on this page, the Internet Explorer 8 interface is referred to as “UI 11” and the Internet Explorer 11 interface is referred to as “UI 14”. 

Please find a document detailing the differences between ServiceNow's Internet Explorer 8 and Internet Explorer 11 user interfaces (UIs) in a ServiceNow Knowledgebase

NB – if you use Chrome, Firefox, or Safari to access ServiceNow, you’re already using UI 14, and won’t notice any changes in your experience.

If you have any questions, please contact the IS Service Desk.

green - services online

Partners HealthCare Wireless Network (phswifi3)

September 9, 2015 8:00 am

Additional phswifi3 certificate information for the change announcement box below:

This new intermediary certificate is issued by ‘InCommon RSA Server CA’ by way of the Trusted Certificate ROOT CA ‘USERTrust RSA Certification Authority’.

Applies To: Users Connecting to Partners HealthCare Wireless Network (phswifi3)

On September 9th, Partners Network Security updated the certificate used for accessing our wireless network. This change may impact you if you connect to the Partners wireless network using a laptop, smartphone, or other mobile device.

When connecting to Partners wireless after September 9th, be sure to select ‘trust’ if prompted by the new certificate.

If you have any questions, please contact the IS Service Desk

green - services online

DIPR Server: Software Management Issue

August 19, 2015 12:00 pm

UPDATE 8/21 10:05AM EDT: Current testing shows yum response time has returned to normal thresholds and is fully usable.  ERIS will work with Network Engineering to determine root cause (if possible to determine).

Applies to: Owners of Virtual Machines (VMs) in DIPR

If you are the owner of a Virtual Machine in DIPR, you will be unable to install new packages or software on your server at this time. ERIS technicians are aware of the problem and a ticket has been opened with Network Engineering / Network Security to troubleshoot the issue. 

Issue
DIPR Servers use a software called Yum. Yum is used to manage package/software installation and upgrades. Yum uses a mechanism to find the fastest mirror to download files. In order to do that, it needs to download an XML from a central repository. There is something that is making the process of reading that file takes more than 30 seconds, which is the default timeout for Yum. 

If something of a critical nature comes up, please email rcc@partners.org and a ticket will be opened to put a workaround in place. 

green - services online

MySQL Restart

August 19, 2015 7:00 pm - 9:00 pm

UPDATE - Restart completed successfully.  Thanks for your patience!

We will be restarting the MySQL daemon on MySQL3.research.partners.org on Wednesday, August 19, 2015 at 7p.m. EDT. This will only affect users on MySQL3.research.partners.org. Downtime is expected to be 20 minutes or less.

If you have any questions, please email rcc@partners.org and a ticket will be opened for you.

green - services online

SSH Gateway Server Connections

August 19, 2015 7:00 pm - 9:00 pm

UPDATE 11:55PM EDT: UPDATE - Restart completed successfully.  Thanks for your patience!

UPDATE 4:23PM EDT: Reboot window has been moved to 9PM-11PM EDT to accommodate customer need.
SSH (Secure Shell) Gateway Server Connections may be impacted by the DIPR File Server reboot on Wednesday, August 19 from 7 to 9p.m. EDT. During this window, existing connections should be fine, but new SSH connections may not open.

Impact
ERIS DIPR File Server (fs.dipr.partners.org) is the nfs file server which provides mounted storage to the SSH Gateway Server Home Folders. ERIS technicians will check and remount home folder storage at the end of the reboot for SSH accounts.

Issue
Right now, there are no direct storage problems, but the server itself is not providing ssh or console level access to administer it, and it will thus need to be rebooted.

Please contact rcc@partners.org if you have any questions or concerns about this maintenance.

green - services online

LimeSurvey Scheduled Downtime

August 19, 2015 9:00 pm - 11:55 pm

LimeSurvey (https://limesurvey.partners.org/limesurvey) required urgent infrastructure updates on Wednesday, Aug. 19. LimeSurvey was offline from 9:00 - 11:55 PM EST.  

If you have any concerns, please email edcsupport@partners.org

green - services online

REDCap Scheduled Downtime

August 19, 2015 7:00 pm - 11:55 pm

REDCap (https://redcap.partners.org/redcap) required two urgent infrastructure updates on Wednesday, Aug. 19.  REDCap was offline from 7:00 - 7:50PM and 9:00 - 11:55PM EST.  

 

If you have any concerns, please email edcsupport@partners.org

 

green - services online

DIPR File Server Reboot

August 19, 2015 7:00 pm - 9:00 pm

UPDATE 11:55PM EDT: UPDATE - Restart completed successfully, a good deal of hosts required their storage to be remounted.  Contact ERIS support at rcc@partners.org if you have issues with any of your hosts connected to fs.dipr.partners.org.  Thanks for your patience!

UPDATE 4:23PM EDT: Reboot window has been moved to 9PM-11PM EDT to accommodate customer need.

ERIS DIPR File Server (fs.dipr.partners.org) will be offline for a server reboot. During this downtime window, any applications and systems that have storage associated with this server will also be offline.

Impact

This is the nfs file server which provides mounted storage to our DIPR environment. ERIS technicians will check and remount storage at the end of the reboot for any systems managed by ERIS.

If you are an owner of a virtual machine in DIPR, we strongly encourage you to check the status of your VM after the reboot. If you find one unresponsive, remount the storage. If any issues persist, please contact rcc@partners.org and we'll address the problem.

Issue 

Right now, there are no direct storage problems, but the server itself is not providing ssh or console level access to administer it, and it will thus need to be rebooted. 

Please contact rcc@partners.org if you have any questions or concerns about this maintenance.

Pages