Service Alerts

green - services online

SSH Gateway Server Connections

August 19, 2015 7:00 pm - 9:00 pm

UPDATE 11:55PM EDT: UPDATE - Restart completed successfully.  Thanks for your patience!

UPDATE 4:23PM EDT: Reboot window has been moved to 9PM-11PM EDT to accommodate customer need.
SSH (Secure Shell) Gateway Server Connections may be impacted by the DIPR File Server reboot on Wednesday, August 19 from 7 to 9p.m. EDT. During this window, existing connections should be fine, but new SSH connections may not open.

Impact
ERIS DIPR File Server (fs.dipr.partners.org) is the nfs file server which provides mounted storage to the SSH Gateway Server Home Folders. ERIS technicians will check and remount home folder storage at the end of the reboot for SSH accounts.

Issue
Right now, there are no direct storage problems, but the server itself is not providing ssh or console level access to administer it, and it will thus need to be rebooted.

Please contact rcc@partners.org if you have any questions or concerns about this maintenance.

green - services online

LimeSurvey Scheduled Downtime

August 19, 2015 9:00 pm - 11:55 pm

LimeSurvey (https://limesurvey.partners.org/limesurvey) required urgent infrastructure updates on Wednesday, Aug. 19. LimeSurvey was offline from 9:00 - 11:55 PM EST.  

If you have any concerns, please email edcsupport@partners.org

green - services online

REDCap Scheduled Downtime

August 19, 2015 7:00 pm - 11:55 pm

REDCap (https://redcap.partners.org/redcap) required two urgent infrastructure updates on Wednesday, Aug. 19.  REDCap was offline from 7:00 - 7:50PM and 9:00 - 11:55PM EST.  

 

If you have any concerns, please email edcsupport@partners.org

 

green - services online

DIPR File Server Reboot

August 19, 2015 7:00 pm - 9:00 pm

UPDATE 11:55PM EDT: UPDATE - Restart completed successfully, a good deal of hosts required their storage to be remounted.  Contact ERIS support at rcc@partners.org if you have issues with any of your hosts connected to fs.dipr.partners.org.  Thanks for your patience!

UPDATE 4:23PM EDT: Reboot window has been moved to 9PM-11PM EDT to accommodate customer need.

ERIS DIPR File Server (fs.dipr.partners.org) will be offline for a server reboot. During this downtime window, any applications and systems that have storage associated with this server will also be offline.

Impact

This is the nfs file server which provides mounted storage to our DIPR environment. ERIS technicians will check and remount storage at the end of the reboot for any systems managed by ERIS.

If you are an owner of a virtual machine in DIPR, we strongly encourage you to check the status of your VM after the reboot. If you find one unresponsive, remount the storage. If any issues persist, please contact rcc@partners.org and we'll address the problem.

Issue 

Right now, there are no direct storage problems, but the server itself is not providing ssh or console level access to administer it, and it will thus need to be rebooted. 

Please contact rcc@partners.org if you have any questions or concerns about this maintenance.

green - services online

RFA (Research File Area) Maintenance

August 20, 2015 8:00 pm - August 21, 2015 12:00 am

On Thursday, August 20th, 2015 from 8p.m. - 12a.m. EDT, maintenance was performed on the RFA (Research File Area) storage system, and there was no interruption to service.

Additional free space was added to the system along with a complete upgrade to the servers that run this system. 

There was no downtime on the RFA storage system, but should you identify any problems, please notify rcc@partners.org.

green - services online

Monthly IS Maintenance (ISM)

August 17, 2015 12:00 am - 4:00 am

Applies To: All Users

Information Systems Maintenance (ISM) took place on Monday, August 17th from 12:00 am to 4:00 am.

During this time, planned system maintenance and testing were performed by IS Operations teams. This routine maintenance reduces the risk of unscheduled outages on our computer network and to the Internet.

Impact to systems and applications

  • VPN was available during the ISM.

Impact to sites

  • Services or resources hosted in the Charlestown Navy Yard (East) and Needham (West) data centers may have been unavailable during the ISM.
  • Your site's IS Management team will provide more detailed communication about planned outages at your facility and will coordinate with specific groups that may be impacted.

Need help during the downtime? 

View the IS Broadcast Archive for additional details. Please contact your IS Service Desk if you have any questions.

View the 2015 IS System Maintenance (ISM) Schedule in the IS Service Desk Knowledgebase (Partners login required).

green - services online

DNS issues resolved for Database Services, RFA, DIPR VMs

August 11, 2015 10:00 am - 11:00 am

Multiple services were unreachable on the morning of Tuesday, August 11, 2015 due to DNS (Domain Name System) issues. All services were back up and running by 11:00a.m. the same day.

Root Cause
A DNS Firewall Services Module (RPZ/FW) was implemented on several servers. The RPZ/FW feature caused domains to become unreachable during the time it was activated. Once this was identified as the cause of the issue, Network Engineering deactivated the feature on all DNS servers.

The DNS Firewall Service Module was not supposed to interrupt connections to the domains while activated. Network Engineering is investigating the issue with the vendor to prevent this from reoccurring.

Issue Detail
Some sites were temporarily unable to connect to the following services

  • Database Services (MySQL, Filemaker, Microsoft SQL, Oracle)
  • DIPR VMs (and services hosted on them)
  • RFA (Research Interactive Storage)

If you have any questions, please contact ERIS by emailing rcc@partners.org, a ticket will automatically be opened for you.

 

green - services online

Slow performance on MySQL2.dipr.partners.org

July 31, 2015 3:27 pm

Websites and services hosted on MySQL2.dipr.partners.org were running slower than usual on Friday, July 31 beginning around 2:45p.m. EDT. ERIS is actively investigating and while as of 3:30p.m. performance improved, the cause of the issue is still under investigation. 

Please note - only sites and services hosted on MySQL2.dipr.partners.org were affected. All other MySQL servers are unaffected.

If you have any questions, please contact ERIS by emailing rcc@partners.org, a ticket will automatically be opened for you.

green - services online

REDCap Upgrade v6.5.9

August 25, 2015 7:00 pm - 9:02 pm

REDCap (https://redcap.partners.org/redcap) upgraded to v6.5.9 on Tuesday, Aug. 25. REDCap was offline from 7:00-9:02 PM EST. 

This upgrade will include a large number of new features and bug fixes. Full release notes available.

If you have any concerns about REDCap being offline at this time or any questions about new features, please email edcsupport@partners.org 

green - services online

MySQL.dipr.partners.org reboot

July 28, 2015 10:30 pm

All services on MySQL.dipr.partners.org have been restored.

--------------------------------------------------------------------------------------

MySQL.dipr.partners.org will be rebooted tonight (7/28/15) at 10:30 p.m. EST, to clear some I/O issues, and connectivity issues to the server.  During this time, all databases located on MySQL.dipr.partners.org will not be accessible.  All databases will remain accessible before the reboot, and an all clear email will be sent after the server has been rebooted.

Please note - this will only affect MySQL.dipr.partners.org.  All other MySQL servers will be unaffected.

If you have any questions, or issues connecting after the reboot, please contact ERIS by emailing rcc@partners.org, a ticket will automatically be opened for you.

green - services online

SQL Server 2008 Restart

July 21, 2015 4:00 pm

Applies to:  ERISSQL2K8P.DIPR.PARTNERS.ORG\ERISSQL2K8P

This notice is for all clients using erissql2k8p.dipr.partners.org\erissql2k8p.  We turned on forced encryption for SQL in transit on our SQL production server.  There was a momentary disruption of service, as we needed to restart the SQL service for changes to take effect. This change was implemented at 4 pm today (7/21/15). 

If you have any questions or issues connecting, please contact ERIS by emailing rcc@partners.org, and a ticket will be automatically opened for you.

green - services online

SQL Server 2008 Patching

July 20, 2015 2:00 pm

Applies to: SQL 2008 Users

The SQL 2008 servers (erissql2k8p.dipr.partners.org, erissql2k8d.dipr.partners.org) were offline for patching on Monday, July 20 at 2pm EDT.  There was a slight delay in service, as the SQL service was restarted.  If you have any questions, or issues connecting after the restart of the service, please contact ERIS by emailing rcc@partners.org, and a ticket will be opened for you.

yellow - pre-downtime, calls to action

Partners MyApps Citrix StoreFront Changes 7/7/2015

July 7, 2015 7:00 am

Starting Tuesday, July 7, 2015, the MacGateway (red apple icon) no longer appears on https://myApps.partners.org (Partners MyApps Citrix StoreFront). To access all of your applications from a Mac, visit https://myApps.partners.org and click the '+' (plus sign) on the left side of your screen (screenshot).
 
Once you select an application from the list, it will be 'pinned' to your main/favorites page. Starting July 7, 2015, Mac users no longer need to request the MacGateway in order to access applications on https://myApps.partners.org.
 
Related Articles:
How to install the Citrix Receiver for Partners MyApps / MacGateway 
Info about Partners myApps Citrix
 
If you need help or assistance please contact the Partners Service Desk at (617) 726-5085 or http://www.partners.org/isservicedesk.

grey - no status, general alert

Remote SSH DNS Name Reminder

June 22, 2015 4:26 pm

This is a reminder that when connecting to Remote SSH at Partners, you should always connect via the DNS name, ssh.research.partners.org, rather than by IP address. The SSH servers IP addresses are subject to change.

Remote SSH is available so technical individuals and those unable to use the VPN software can access the clusters or Linux server. If you have any questions about this service, please contact rcc@partners.org.

green - services online

MAD Data Storage

June 17, 2015 11:01 am

MAD Data Storage service is back online. The cluster became unstable and required node reboots to restore service. 

If you continue to experience any difficulty connecting to your MAD data storage, please contact rcc@partners.org.

red - services offline, issue, active work

HPCWIN Server Retires Soon

July 15, 2015 9:00 am

HPCWIN has provided many years of service and will retire, coinciding with the end-of-support from Microsoft for Windows Server 2003. If you have a HPCWIN account currently, please plan on migrating to HPCWIN3. 

The replacement service, HPCWIN3, is currently available to anyone interested in testing this new server. Request a HPCWIN3 account by completing the sign-up form at https://rc.partners.org/windows-analysis-server-account-request. HPCWIN3 has 32 CPU Cores and 3320 GB of memory running Windows Server 2012.

We will be contacting users who have not yet migrated to schedule that process. After the retirement of HPCWIN, you will no longer be able to login to HPCWIN, but you can request access to your data.

HPCWIN2 is not affected. Visit the website for an overview of the Windows Analysis Server Service

green - services online

IS Service Desk Reboot

June 1, 2015 5:15 pm - 5:45 pm

In order to address lingering performance issues experienced by administrators of the IS Service Desk (http://www.partners.org/isservicedesk), PHS IS Operations performed a reboot of the servers beginning 5:15PM. There was an anticipated 30 minute period (5:15pm – 5:45pm).

View the IS Communications Broadcast online: http://atu.partners.org/ISCommunicationArchives.aspx?msgid=715

yellow - pre-downtime, calls to action

Updates to ERISOne Compute Nodes

July 26, 2015 8:00 am - 8:00 pm

Updates to the operating system are planned for the ERISOne Linux Cluster.  The provisional date set for this change is Sunday July 26th.

Operating system updates are unlikely to change the way jobs on the cluster run, however we recommend testing your applications and providing us feedback prior to Monday July 13th.  A group of compute nodes that have the updates applied are available today for testing, for full details see the Knowledge Base article:

https://rc.partners.org/kb/computational-resources/linux-cluster/using-applications?article=1061

Further notification will be posted before updates are applied generally to all compute nodes.

green - services online

ERISOne Cluster Maintenance

June 28, 2015 8:00 am - 8:00 pm

The ERISOne cluster reopened at 2p.m. on Monday, June 29, 2015. erisone.partners.org SSH access was restored around 3p.m., and permissions issues accessing group folders on erisonefs.partners.org were resolved by 8p.m. on Monday, June 29, 2015.

Systems Affected

  • ERISOne login server
  • ERISOne file transfer server
  • ERISOne virtual desktop nodes
  • ERISOne web application servers

Maintenance Details

In order to ensure premium supportability, security and performance, we utilized this month's maintenance window to perform essential maintenance on the Panasas Storage Servers. On Sunday, June 28, 2015 at 8a.m. EDT, logins to the ERISOne cluster were disabled and the affected data folders were unavailable. 

This maintenance is essential for the upgrading of critical storage. For the affected groups, any running jobs were paused at 8am, Sunday 6/28/2015, and resumed after the maintenance completed on Monday, June 29 at 2p.m. EDT. We highly recommend that affected lab groups avoid submitting jobs that may span the maintenance window as there is a small risk some may not resume successfully. For the full list of systems and folders affected, see the June 28, 2015 maintenance notifcation.

Please email hpcsupport@partners.org if you have any questions or concerns about this scheduled downtime.

 

green - services online

Mailing List System/Lyris

May 13, 2015 11:30 am

The Lyris List Manager Mailing List System, https://researchlist.partners.org, is online and fully operational.

The Administrator Interface for the Lyris List Manager Mailing List System, https://researchlist.partners.org, was unaccessible on Chrome, FireFox & Internet Explorer browsers, when accessing the URL from the Partners network on Wednesday, May 13 from 11:30a.m. to 10p.m.

Messages delivered from email clients (i.e. Outlook) were not affected. 

Contact rcc@partners.org with any questions or concerns. 

yellow - pre-downtime, calls to action

MAD Data Storage Space & Inode Usage

August 15, 2015 9:00 am

In order to keep the system running efficiently, MAD space and inode usage need to be optimized across accounts. Beginning August 15, 2015, a new account policy will limit the amount of inodes consumed on the MAD system. Read more about best practices in the MAD User Guide. This is a reminder that MAD is designed as an archival tier system and file sizes should be kept as large as possible before uploading. Each file or folder object stored on MAD creates an inode, or index node. 
 
Check out your current MAD space and inode usage at http://mad.research.partners.org/usage/; this report is updated hourly. MAD performs most efficiently when each file is over one megabyte in size on average, so that a maximum of 1024 inodes per gigabyte of space is consumed. Accounts consuming more than 1024 inodes/GB are asked to consolidate files into .tar or .zip archives, or move data to a more appropriate service. For assistance with tuning workflows to make better use of MAD, please email rcc@partners.org.

green - services online

REDCap Service Alert - Unexpected Email Delays

April 29, 2015 1:29 pm

Due to a large volume of emails being sent from REDCap yesterday, a backlog was created which delayed the sending of survey invitations and system requests (request new projects, moves to production, change requests). 

The delay was addressed this morning and as of 10:55AM EDT, all queued emails were sent and delay resolved. 

We are working with the Network Security team to optimize our email configuration. Please allow an extra day for replies to service requests. We apologize for the inconvenience. 

If you have any questions, please email edcsupport@partners.org

green - services online

Macgateway: Citrix StoreFront

May 6, 2015 8:00 am

The portal https://myapps.partners.org page and menus received facelifts on Wednesday, May 6, 2015. This change includes a new web interface called StoreFront which will provide you with the ability to “pin” your  favorite applications on the desktop. As of May 6, 2-factor authentification will also be required to access https://myapps.partners.org.

When you log in, you will see your red apple Macgateway icon on the main/favorites page, as well as links on the Macgateway page “pinned” to this area. You can run your applications from either the Macgateway page or from those pinned links.

These links are also available via the lefthand flyout menu activated by the “+”. See the Partners Citrix Storefront Coming for "MY CITRIX APPS" Announcement for additional details.

green - services online

REDCap Upgrade v5.9.18

April 21, 2015 8:00 pm - 8:41 pm

Partners HealthCare: Scheduled Downtime

REDCap (https://redcap.partners.org/redcap) upgrades to v5.9.18 were completed Tuesday, Apr. 21. REDCap was offline from 8:00 - 8:41 p.m. EST.

This upgrade is for bug fixes only, no new features. Full release notes available.

If you have any concerns about REDCap being offline at this time, please email edcsupport@partners.org 

green - services online

Limesurvey Maintenance - Unscheduled

April 17, 2015 11:40 am - 11:50 am

Partners Research Computing - LimeSurvey required unscheduled maintenance on April 17, 11:40 a.m. The systems needed to be rebooted and downtime lasted 10 minutes

Pages