Service Alerts

green - services online

MAD Data Storage: Gateway Reboot Completed

October 6, 2015 1:00 pm - 3:45 pm

The reboot of all 3 gateways has completed successfully and it appears as though normal behavior has returned. Please test your normal use cases to ensure that any problems you may have been experiencing are no longer occurring.

MAD engineering will continue to search for root cause to these latest symptoms and for the original problem from Wednesday, but it does appear that any lingering issues resulting from the remount of storage last week are now ceased.  We will look to more closely identify what was wrong so as to avoid a full reboot in the future if at all possible.

We understand there are a few outstanding issues that are in the queue for completion, and with the urgent issues now cleared, MAD engineering will address those open items this afternoon and complete them as soon as possible.

Thank you very much for your patience while we sorted this out!

As you may be aware, the MAD infrastructure encountered issues last Wednesday for customers using non-replicated accounts (see below). We believe we isolated and corrected that issue, but there have been lingering symptoms stemming from correction of this issue that we cannot keep cleared.

We rebooted the gateway servers during the lunch hour on Tuesday, October 6, 2015, as many customers were not able to write to their respective shares.

We advised that if you had any current jobs running against your MAD shares, to please suspend them.

Thanks for your patience while we alleviate this problem!

From Wednesday, September 30, 2015:

Nonreplicated shares on the MAD (Massive Array of Disks) data storage service were reported unavailable yesterday afternoon. Service was fully restored on Thursday, October 1 at approximately 11a.m. EDT. Check any transfers that ran from Wednesday, September 30, 2015 afternoon to the morning of Thursday, October 1, and restart the transfer if necessary.

ERIS technicians are actively exploring the cause of the issue. If you experience any issues with the MAD data storage service, please contact rcc@partners.org.

Posted on October 1, 2015
green - services online

Mac OS X 10.11 El Capitan in the Enterprise

September 29, 2015 11:45 am

The Partners Enterprise Apple Support (PEAS) team is pleased to inform you that testing is complete and you may now consider an upgrade to Mac OS X 10.11 El Capitan. Note: Apple, and therefore PEAS, supports the current OS and two prior versions. If you’re running Mac OS X 10.8 Mountain Lion or older, you need to upgrade. When upgrading, please review the El Capitan Guide and check the compatibility chart to ensure software you may use is compatible.

The Symantec Enterprise Vault plug-in for Mac Outlook is not compatible with Outlook 2016. The vendor has decide to discontinued future support of Outlook for Mac. You can still access your archived email through the web interface by navigating to https://archive.partners.org/enterprisevault.

If you are running Mac OS X 10.10 or higher on your Mac, you can also install Microsoft Office 2016. Please note that the Enterprise Vault plug-in are not compatible with Microsoft Office 2016. 

To request Microsoft Outlook 2016 

  1. Call the Partners IS Service Desk and request an Office 365 subscription.
  2. Follow step by step instructions in the KnowledgeBase article, HOWTO: Install Microsoft Office 2016 for Mac. 

Outlook 2016 along with additional updated software, including a Java update, are available in Self Service.

For any questions on El Capitan or technical assistance with upgrading, please submit a request for assistance to the Partners IS Service Desk.

Posted on September 29, 2015
green - services online

Confluence Downtime

September 27, 2015 8:00 am - 10:00 am

Maintenance and upgrades were performed on Confluence (https://confluence.partners.org) on Sunday, September 27 at 8:00a.m. If you have any questions or concerns, please contact rcc@partners.org.

Confluence is an Enterprise Wiki and Collaboration tool used by ERIS and Research for documentation, team calendaring and information dissemination.

Posted on September 25, 2015
green - services online

Monthly IS Maintenance (ISM)

September 28, 2015 12:00 am - 4:00 am

Applies To: All Users

Information Systems Maintenance (ISM) took place on Monday, September 28 from 12:00 am to 4:00 am.

During this time, planned system maintenance and testing were performed by IS Operations teams. This routine maintenance reduces the risk of unscheduled outages on our computer network and to the Internet.

Impact to systems and applications

  • VPN should be available during the ISM.
  • Partners eCare - unavailable from 1:00am until 4:00am
    1. Impacts all PeC applications and related interfaces for MGH, BWH, BWFH, DFCI, PHH, NWH and Newton Wellesley Ambulatory Services.
    2. Impacts PeC dependent applications including Patient Gateway.

Impact to sites

  • Services or resources hosted in the Partners HealthCare data centers may be unavailable during the ISM.
  • Your site's IS Management team will provide more detailed communication about planned outages at your facility and will coordinate with specific groups that may be impacted.

Need help during the downtime? 

View the Broadcast Online. Please contact your IS Service Desk if you have any questions.

View the 2015 IS System Maintenance (ISM) Schedule in the IS Service Desk Knowledgebase (Partners login required).

Posted on September 24, 2015
yellow - pre-downtime, calls to action

MySQL connection issues

September 21, 2015 8:00 pm

Applies to: MySQL instance - MySQL2.dipr.partners.org

A minor connection refusal on mysql2.dipr.partners.org may have caused a small outage for connections to the MySQL instance from around 8p.m. to 9p.m. EST on Monday, September 21, 2015. A hung maintenance job created a backlog of connections, that ultimately created a refusal of new database connections. Once the maintenance job was canceled, connections resumed as normal.

The ERIS database team will investigate the timing of the maintenance job, and attempt to prevent this issue from happening in future runs.

On Tuesday, Sept. 29 from 12:30 - 1:30p.m. EST a small outage for connections to MySQL2.dipr.partners.org also occurred. the server experienced a higher than usual number of concurrent, sustained connections that increased the system load and temporarily maxed out the number of allowed connections. In most cases connections were allowed and the server was queuing up commands, but the execution of those commands took much longer than expected. All site are currently connecting, but the team is continuing to investigate how to prevent a reoccurrence.

If you have any questions, please contact rcc@partners.org, and a ticket will be generated automatically to our database team.

Posted on September 22, 2015
yellow - pre-downtime, calls to action

REDCap Service Alert: Limited Support Sep. 28 - Oct. 1

September 28, 2015 9:00 am - October 1, 2015 5:00 pm

There will be limited support next week while EDC Support attends the annual REDCap Conference. Please plan for delayed response times.  Priorities will be made for Moving Projects to Production, User Access, Creating New Projects, and Change Requests.  General REDCap questions will be best effort. 

Be sure to review the Help&FAQ, Training Resources, Getting Started with REDCap and Knowledge Base pages.  

If you have any questions, please email edcsupport@partners.org.

Posted on September 21, 2015
yellow - pre-downtime, calls to action

REDCap: Known Issues With Upgrade to v6.5

September 17, 2015 1:28 pm

With the recent upgrade to REDCap v6.5.9, we are aware of a number of issues that users are experiencing. These include: 

  • Script errors when creating reports: Some users, especially those using Internet Explorer, have run into script issues where their browsers freeze when creating a report by adding entire instruments. It seems the issue is tied to the number of variables in the project and the number of records. Other browsers, such as Chrome tend to work better. 
  • Issues when moving between authenticated plugins/project bookmarks: If a user selects an authenticated plugin/project bookmark and selects a second one, they receive an error message. The work around for this is to navigate to a different page, and then select the next plugin/bookmark. 
  • REDCap delayed login: As most of you have probably noticed, it now takes 15-20 seconds to log into REDCap. 

We are investigating these issues and will let you know once they are resolved. The resolution will most likely involve an upgrade to apply bug fixes. Schedule to be determined.

If you have any questions, please email edcsupport@partners.org 

Posted on September 17, 2015
yellow - pre-downtime, calls to action

New VPN Client

September 23, 2015 11:00 am

The Partners Network Security team will upgrade the VPN client software (anyconnect) to version 3.1.10010 that is hosted on the VPN servers (www.partners.org/vpn). After 11a.m. on Wednesday, September 23, individuals will be prompted to upgrade the software on their computers when connecting to VPN.

This new client supports these Operating Systems and versions:

  • Windows 7, 8, 10*
  • Mac OS X 10.8, 10.9, 10.10, 10.11

* To use VPN on Windows 10, you will need use IE 11 IE browser icon and not the new IE Edge Edge browser iconbrowser. You will also need to have the latest version of Java installed. 

Contact the IS Service Desk if you have any questions or concerns.

Posted on September 17, 2015
green - services online

StudyTRAX Infrastructure Updates

September 10, 2015 9:00 pm - September 11, 2015 12:00 am

StudyTRAX was offline September 10 at 9pm - 12 midnight for SQL database migration.  Migration was successful.

If you have any concerns, please email edcsupport@partners.org

 

Posted on September 3, 2015
yellow - pre-downtime, calls to action

Internet Explorer 11 Upgrade

September 1, 2015 8:00 am

Applies To: All Users

During September and October, workstations across the Partners network will be upgraded to Internet Explorer 11 (IE11), the most current version available from Microsoft.  IE11 allows a greater number of websites to display and work properly while also improving the security of our network.

Stay tuned for additional details from your site’s IS Planning and Management team, including a specific rollout schedule. If you have any questions, please contact the IS Service Desk.

What do I need to do?

Please be sure to save your work and properly log off your workstation at the end of each day until you have received the IE11 upgrade: 

  1. Save your work and close all applications.
  2. Click the Yellow Lock.
  3. Click New User.

How will the upgrade work?

Workstations that are powered on and connected to the Partners network and logged out will receive IE 11 during designated times. If a workstation is logged in during the automated process, it will be skipped. Any workstations that are skipped will need to be upgraded at a later date.

When a device is upgraded to IE11, it should automatically reboot after the upgrade. If the device does not automatically reboot at the end of the upgrade, you may need to reboot it manually.

How to determine your Partners device build type: 

  • When logged out, the device build type is listed at the top of the Partners screensaver. Example

 . Note: Use the F1 key from the screensaver for additional Device information.

  • When logged in, Go to Partners Applications > Utilities > Desktop Information. Under Workstation Configuration, look for Workstation Type. 

IE 11 Self Install Process for Laptops

Internet Explorer 11 (IE11) is now available for self-installation on laptop computers as well as on private-build workstations that have not yet received the IE11 upgrade via the ongoing automated process. This allows you to install IE11 at your convenience. This does not apply to MGH or DFCI.

  • How do I install IE11?
    • You must be connected to the Partners network in the office or via VPN to run the install. View Installation Instructions (You will need to log in with your Partners Username and Password)

What’s new in IE11?

IE11 is the most current version available from Microsoft. It allows a greater number of websites to display and work properly while also improving the security of our network. Learn more

ServiceNow User Interface Changes in Internet Explorer 11

When Internet Explorer 8 is upgraded to Internet Explorer 11 on Partners standard builds, you’ll notice some minor changes in the user interface (UI). Complete details of all of the user interface changes can be found at the following web page. Note, on this page, the Internet Explorer 8 interface is referred to as “UI 11” and the Internet Explorer 11 interface is referred to as “UI 14”. 

Please find a document detailing the differences between ServiceNow's Internet Explorer 8 and Internet Explorer 11 user interfaces (UIs) in a ServiceNow Knowledgebase. 

NB – if you use Chrome, Firefox, or Safari to access ServiceNow, you’re already using UI 14, and won’t notice any changes in your experience.

If you have any questions, please contact the IS Service Desk.

Posted on September 2, 2015
green - services online

Partners HealthCare Wireless Network (phswifi3)

September 9, 2015 8:00 am

Additional phswifi3 certificate information for the change announcement box below:

This new intermediary certificate is issued by ‘InCommon RSA Server CA’ by way of the Trusted Certificate ROOT CA ‘USERTrust RSA Certification Authority’.

Applies To: Users Connecting to Partners HealthCare Wireless Network (phswifi3)

On September 9th, Partners Network Security updated the certificate used for accessing our wireless network. This change may impact you if you connect to the Partners wireless network using a laptop, smartphone, or other mobile device.

When connecting to Partners wireless after September 9th, be sure to select ‘trust’ if prompted by the new certificate.

If you have any questions, please contact the IS Service Desk. 

Posted on September 2, 2015
green - services online

DIPR Server: Software Management Issue

August 19, 2015 12:00 pm

UPDATE 8/21 10:05AM EDT: Current testing shows yum response time has returned to normal thresholds and is fully usable.  ERIS will work with Network Engineering to determine root cause (if possible to determine).

Applies to: Owners of Virtual Machines (VMs) in DIPR

If you are the owner of a Virtual Machine in DIPR, you will be unable to install new packages or software on your server at this time. ERIS technicians are aware of the problem and a ticket has been opened with Network Engineering / Network Security to troubleshoot the issue. 

Issue
DIPR Servers use a software called Yum. Yum is used to manage package/software installation and upgrades. Yum uses a mechanism to find the fastest mirror to download files. In order to do that, it needs to download an XML from a central repository. There is something that is making the process of reading that file takes more than 30 seconds, which is the default timeout for Yum. 

If something of a critical nature comes up, please email rcc@partners.org and a ticket will be opened to put a workaround in place. 

Posted on August 19, 2015
green - services online

MySQL Restart

August 19, 2015 7:00 pm - 9:00 pm

UPDATE - Restart completed successfully.  Thanks for your patience!

We will be restarting the MySQL daemon on MySQL3.research.partners.org on Wednesday, August 19, 2015 at 7p.m. EDT. This will only affect users on MySQL3.research.partners.org. Downtime is expected to be 20 minutes or less.

If you have any questions, please email rcc@partners.org and a ticket will be opened for you.

Posted on August 18, 2015
green - services online

SSH Gateway Server Connections

August 19, 2015 7:00 pm - 9:00 pm

UPDATE 11:55PM EDT: UPDATE - Restart completed successfully.  Thanks for your patience!

UPDATE 4:23PM EDT: Reboot window has been moved to 9PM-11PM EDT to accommodate customer need.
SSH (Secure Shell) Gateway Server Connections may be impacted by the DIPR File Server reboot on Wednesday, August 19 from 7 to 9p.m. EDT. During this window, existing connections should be fine, but new SSH connections may not open.

Impact
ERIS DIPR File Server (fs.dipr.partners.org) is the nfs file server which provides mounted storage to the SSH Gateway Server Home Folders. ERIS technicians will check and remount home folder storage at the end of the reboot for SSH accounts.

Issue
Right now, there are no direct storage problems, but the server itself is not providing ssh or console level access to administer it, and it will thus need to be rebooted.

Please contact rcc@partners.org if you have any questions or concerns about this maintenance.

Posted on August 18, 2015
green - services online

LimeSurvey Scheduled Downtime

August 19, 2015 9:00 pm - 11:55 pm

LimeSurvey (https://limesurvey.partners.org/limesurvey) required urgent infrastructure updates on Wednesday, Aug. 19. LimeSurvey was offline from 9:00 - 11:55 PM EST.  

If you have any concerns, please email edcsupport@partners.org

Posted on August 18, 2015
green - services online

REDCap Scheduled Downtime

August 19, 2015 7:00 pm - 11:55 pm

REDCap (https://redcap.partners.org/redcap) required two urgent infrastructure updates on Wednesday, Aug. 19.  REDCap was offline from 7:00 - 7:50PM and 9:00 - 11:55PM EST.  

 

If you have any concerns, please email edcsupport@partners.org

 

Posted on August 18, 2015
green - services online

DIPR File Server Reboot

August 19, 2015 7:00 pm - 9:00 pm

UPDATE 11:55PM EDT: UPDATE - Restart completed successfully, a good deal of hosts required their storage to be remounted.  Contact ERIS support at rcc@partners.org if you have issues with any of your hosts connected to fs.dipr.partners.org.  Thanks for your patience!

UPDATE 4:23PM EDT: Reboot window has been moved to 9PM-11PM EDT to accommodate customer need.

ERIS DIPR File Server (fs.dipr.partners.org) will be offline for a server reboot. During this downtime window, any applications and systems that have storage associated with this server will also be offline.

Impact

This is the nfs file server which provides mounted storage to our DIPR environment. ERIS technicians will check and remount storage at the end of the reboot for any systems managed by ERIS.

If you are an owner of a virtual machine in DIPR, we strongly encourage you to check the status of your VM after the reboot. If you find one unresponsive, remount the storage. If any issues persist, please contact rcc@partners.org and we'll address the problem.

Issue 

Right now, there are no direct storage problems, but the server itself is not providing ssh or console level access to administer it, and it will thus need to be rebooted. 

Please contact rcc@partners.org if you have any questions or concerns about this maintenance.

Posted on August 18, 2015
green - services online

RFA (Research File Area) Maintenance

August 20, 2015 8:00 pm - August 21, 2015 12:00 am

On Thursday, August 20th, 2015 from 8p.m. - 12a.m. EDT, maintenance was performed on the RFA (Research File Area) storage system, and there was no interruption to service.

Additional free space was added to the system along with a complete upgrade to the servers that run this system. 

There was no downtime on the RFA storage system, but should you identify any problems, please notify rcc@partners.org.

Posted on August 17, 2015
green - services online

Monthly IS Maintenance (ISM)

August 17, 2015 12:00 am - 4:00 am

Applies To: All Users

Information Systems Maintenance (ISM) took place on Monday, August 17th from 12:00 am to 4:00 am.

During this time, planned system maintenance and testing were performed by IS Operations teams. This routine maintenance reduces the risk of unscheduled outages on our computer network and to the Internet.

Impact to systems and applications

  • VPN was available during the ISM.

Impact to sites

  • Services or resources hosted in the Charlestown Navy Yard (East) and Needham (West) data centers may have been unavailable during the ISM.
  • Your site's IS Management team will provide more detailed communication about planned outages at your facility and will coordinate with specific groups that may be impacted.

Need help during the downtime? 

View the IS Broadcast Archive for additional details. Please contact your IS Service Desk if you have any questions.

View the 2015 IS System Maintenance (ISM) Schedule in the IS Service Desk Knowledgebase (Partners login required).

Posted on August 13, 2015
green - services online

DNS issues resolved for Database Services, RFA, DIPR VMs

August 11, 2015 10:00 am - 11:00 am

Multiple services were unreachable on the morning of Tuesday, August 11, 2015 due to DNS (Domain Name System) issues. All services were back up and running by 11:00a.m. the same day.

Root Cause
A DNS Firewall Services Module (RPZ/FW) was implemented on several servers. The RPZ/FW feature caused domains to become unreachable during the time it was activated. Once this was identified as the cause of the issue, Network Engineering deactivated the feature on all DNS servers.

The DNS Firewall Service Module was not supposed to interrupt connections to the domains while activated. Network Engineering is investigating the issue with the vendor to prevent this from reoccurring.

Issue Detail
Some sites were temporarily unable to connect to the following services

  • Database Services (MySQL, Filemaker, Microsoft SQL, Oracle)
  • DIPR VMs (and services hosted on them)
  • RFA (Research Interactive Storage)

If you have any questions, please contact ERIS by emailing rcc@partners.org, a ticket will automatically be opened for you.

 

Posted on August 11, 2015
green - services online

Slow performance on MySQL2.dipr.partners.org

July 31, 2015 3:27 pm

Websites and services hosted on MySQL2.dipr.partners.org were running slower than usual on Friday, July 31 beginning around 2:45p.m. EDT. ERIS is actively investigating and while as of 3:30p.m. performance improved, the cause of the issue is still under investigation. 

Please note - only sites and services hosted on MySQL2.dipr.partners.org were affected. All other MySQL servers are unaffected.

If you have any questions, please contact ERIS by emailing rcc@partners.org, a ticket will automatically be opened for you.

Posted on July 31, 2015
green - services online

REDCap Upgrade v6.5.9

August 25, 2015 7:00 pm - 9:02 pm

REDCap (https://redcap.partners.org/redcap) upgraded to v6.5.9 on Tuesday, Aug. 25. REDCap was offline from 7:00-9:02 PM EST. 

This upgrade will include a large number of new features and bug fixes. Full release notes available.

If you have any concerns about REDCap being offline at this time or any questions about new features, please email edcsupport@partners.org 

Posted on July 29, 2015
green - services online

MySQL.dipr.partners.org reboot

July 28, 2015 10:30 pm

All services on MySQL.dipr.partners.org have been restored.

--------------------------------------------------------------------------------------

MySQL.dipr.partners.org will be rebooted tonight (7/28/15) at 10:30 p.m. EST, to clear some I/O issues, and connectivity issues to the server.  During this time, all databases located on MySQL.dipr.partners.org will not be accessible.  All databases will remain accessible before the reboot, and an all clear email will be sent after the server has been rebooted.

Please note - this will only affect MySQL.dipr.partners.org.  All other MySQL servers will be unaffected.

If you have any questions, or issues connecting after the reboot, please contact ERIS by emailing rcc@partners.org, a ticket will automatically be opened for you.

Posted on July 28, 2015
green - services online

SQL Server 2008 Restart

July 21, 2015 4:00 pm

Applies to:  ERISSQL2K8P.DIPR.PARTNERS.ORG\ERISSQL2K8P

This notice is for all clients using erissql2k8p.dipr.partners.org\erissql2k8p.  We turned on forced encryption for SQL in transit on our SQL production server.  There was a momentary disruption of service, as we needed to restart the SQL service for changes to take effect. This change was implemented at 4 pm today (7/21/15). 

If you have any questions or issues connecting, please contact ERIS by emailing rcc@partners.org, and a ticket will be automatically opened for you.

Posted on July 21, 2015
green - services online

SQL Server 2008 Patching

July 20, 2015 2:00 pm

Applies to: SQL 2008 Users

The SQL 2008 servers (erissql2k8p.dipr.partners.org, erissql2k8d.dipr.partners.org) were offline for patching on Monday, July 20 at 2pm EDT.  There was a slight delay in service, as the SQL service was restarted.  If you have any questions, or issues connecting after the restart of the service, please contact ERIS by emailing rcc@partners.org, and a ticket will be opened for you.

Posted on July 20, 2015

Pages