Between 6 am and 7 am EDT Friday (10/25/2013) morning we will perform an upgrade to the system that hosts MiddFiles recommended by our vendor that is a first required step to improve performance. During this period there will be an approximately 15 minute period when MiddFiles will not be available.
We appreciate your patience and our apologies for scheduling this maintenance on short notice and outside our regular maintenance window of Sunday mornings.
This Sunday morning between 7 and 8:30 am we will be upgrade MiddFiles, there will be a less than 15 minute interruption of all connectivity to MiddFiles during this window. We have been continuing to work intensively with our vendor to isolate intermittent disconnects of all client computers to MiddFiles that are occurring due to unexpected and abnormal termination (core dump) of the smb service. When the smb service crashes, all of the client lose their connection to middfiles and need to reconnect, while this process doesn’t take long, it is noticeable and frustrating and particularly so if it is happening often, this is especially the case for database applications that are most sensitive to loss of their access to their underling storage environment. We have been making steady progress working with our vendor to apply patches and tune settings in order to reduce the frequency of these service interruptions, from as many as 40 a day a couple of weeks ago to 15 a day earlier this week to 6 yesterday. While we’re not out of the woods yet, we do believe that we are getting closer to putting this issue behind us.
Please don’t hesitate to contact me with any questions or updates, in particular if we haven’t already touched base related to the impact this is having in your work groups area.
Digital Collections at Middlebury / CONTENTdm (middarchive.middlebury.edu) will be off-line for approximately 4 hours from 6 to 10am this Sunday morning 10/6/2013 in order to upgrade to version 6.5.
We will also be upgrading MiddFiles to apply a patch the system vendor has released and recommended as we work diligently to address the intermittent disconnect issues we have been recently experiencing. Thus between 7 and 8:30 am EST there will be an approximately 15 minute outage of MiddFiles this Sunday morning during our established maintenance window.
This Sunday morning during our established maintenance window from 6 am to 8 am we will be performing the below activities. Please share this message with colleagues as you see appropriate.
(1) Between 7 am and 8 am we will be upgrading our “MiddCloud” system that includes MiddFiles, Augusta, and Crux file servers. There will be a less than 15 minute interruption of access to these file servers during this period.
(2) Between 7:30 am and 9 am we will be upgrading our DNS (Domain Name Service) and DHCP (Dynamic Host Configuration Protocol) servers. We anticipate a less than 5 minute impact of domain name resolution during this period.
We appreciate your patience as we strive to keep our systems operating optimal. Please contact me with any questions.
We will be upgrading Web Help Desk between 8:00 am and 9:00 am this Sunday morning. During this period, Web Help Desk will be off-line for approximately 15 minutes.
This Sunday morning we will be performing the following activities:
(1) Between 6:30 and 7:00 am we will be reconfiguring a component within our network core, while we do not expect any noticeable downtime, a short campus-wide network interruption of a few minutes is possible.
(2) Between 7:00 am and 8:00 am we will be reconfiguring our wifi environment that will require each wireless access point on the Middlebury and Bread Loaf campuses to restart, thus there will be a less than 5 minute interruption of the wireless network campus wide during this period.
(3) Between 7:00 am and 8:00 am we will be upgrading a database server that the below applications are depend on. Thus there will be a less than 10 minute interruption access to these applications during this period.
We appreciate your patience as we perform this essential maintenance and strive to keep our systems and networks functioning optimally.
Update: 6/25/2013, 2:00 PM EDT:
As a result of the change required to isolate the failing component of our email infrastructure late last week, we have discovered that legitimate undeliverable bounce back messages have not been forwarded on to the original senders at Middlebury.edu and MIIS.edu but rather remained queued up. We have made the necessary configuration correction and about 300 legitimately undeliverable messages have now been correctly delivered. You may have received several messages all at once and wondered if there was a problem sending email off-campus generally, there is not, these are legitimate undeliverable bounce messages that have been accumulating for the last several days and now have been correctly forwarded on to you. Our sincere apologies for the delay in the delivery of these messages.
Update as of 6:30 PM EDT:
Testing and monitoring continues to be positive and we are now very confident that the intermittent Outlook disconnect incident has been resolved. Thank you for your patience and support during this event.
—
Update as of 2:10 PM EDT:
At this time we believe we have identified and eliminated the component that was causing the intermittent disconnects from outlook clients to our email Exchange environment. We are monitoring the systems closely and hope that this incident may be behind us.
—
At this time we are experiencing intermittent and isolated disconnects to our email Exchange environment from Outlook clients. You may have noticed warning messages or other pop-ups from time to time indicating that your Exchange client is attempting to contact Microsoft Exchange.
We are diligently working to solve this issue. We are isolating each component of our Exchange infrastructure including load balancing appliances that front-end the Exchange infrastructure and eliminating each as the cause of the problem. This is a time consuming process as it can take several hours to place individual components into maintenance mode and isolate them as client sessions tend to hold on particular servers they are currently working with.
Please note that email messages are flowing and being stored normally. Additionally there have not be any issues with accessing Exchange via the web interface (Outlook web access available at https://mail.middlebury.edu), so if the disconnection errors or warnings you receive within your Outlook client are too disruptive, please consider using the web interface until we are able to completely isolate and resolve the issue.
We appreciate your patience and apologies for the inconvenience, we recognize how critical email is to your daily work activity and our full resources are focused on resolving this issue as quickly as possible.
Best regards,
Jim
This Sunday morning during our established maintenance window between 6 am and 10 am EDT we will be upgrading a number of essential switches in the path between our network core and our Internet service providers. During this period there will be as many 7 Internet connectivity outages each lasting between 5 and 20 minutes.
We will be also be upgrading our wireless network infrastructure this Sunday morning from approximately 7:30 until 9:00 am EDT. During this period each WAP on campus will reboot causing a few minutes service interruption for the area it serves – the WAPs will reboot across campus at different times as their supporting controllers are upgraded.
We appreciate your patience as this essential work is completed.
Best regards,
Jim