Key Survey / WorldApp Update: Message from the CEO

Categories: Midd Blogosphere

Here is the message sent by the CEO of WorldApp, Inc. concerning last Friday’s Key Survey down time.  (Key Survey is a software program used to create and distribute surveys, as well as collect & analyze responses.)

KeySurvey Logo

From: Oleg Matsko
Sent: Monday, May 18, 2015 9:36 AM
Subject: An update on Friday’s disruption – a message from our CEO

Last Friday’s issues have been some of the most severe issues to affect WorldAPP since we launched Key Survey in 2002. As CEO, I take immense pride in serving organizations across the world in fulfilling their requirements and I feel immensely sorry and hurt that we let those customers down. As such, I feel it is only right that we be completely open, honest and transparent about what happened, and what we are doing to make sure it doesn’t happen again.

A few weeks ago we noticed that one of the storage components of our production environment had started to fail. This in itself doesn’t cause an immediate issue, our production environment is built with multiple layers of redundancy, and despite one of the critical elements of this environment not functioning, our applications continued to work in the manner they should, without any impact on availability. It is important though that when these issues occur, we rectify them as quickly as we can, so that should other components of our environment fail, there isn’t any impact on service.

So for the past few weeks we have been preparing our secondary storage components to take over, allowing us to complete the necessary works on the primary components. Our applications collect a lot of data, in fact the equivalent of 11,000 pages of paper an hour, and this amount of data takes a lot of time to transfer. In an absolutely emergency we can complete this transfer in about 12 hours, but as our primary setup was still stable, and the risks of transferring such a huge amount of data in a relatively short amount of time being quite high, we took our time and completed this transfer over a period of a few weeks.

This transfer was completed on Thursday evening, our secondary storage components went live without issue, and our primary storage components were taken offline to allow the required maintenance to be completed. For a few hours, everything worked fine, and then at around 08:00 EDT on Friday morning, without notice our secondary storage components failed. At the moment, the reason why they failed is still unclear, there doesn’t appear to be an obvious cause. We will work hard with our infrastructure partners, to find out why this happened – but the most important thing for us to do on Friday was to get our applications back online.

Key Survey and Form.com are incredibly large and complex applications, and restarting them isn’t a simple operation. The applications are made up of many separate modules, each relating to an area of their functionality, such as reporting, voting or our API. The effort required to restart them is large, so much so that they cannot all be restarted at once. As such, modules were restarted individually, in order of priority. Our main Key Survey and Form.com environments were operational by 15:00 EDT, with all of our reporting modules online by 21:30 EDT and specific instances of our applications for individual customers back online by 00:30 EDT on Saturday morning.

As a result of Friday’s disruption, I have instructed our teams to rebuild our storage infrastructure to include additional layers of redundancy with built in instant failover capabilities. This is no easy challenge, implementing this infrastructure and migrating all our applications will take about a week, but we should be able to complete this without additional disruption. Once these changes are implemented, we will be able to recover our systems in a matter of minutes. This is in addition to the construction of the remote disaster recovery infrastructure which is already underway and estimated to be completed early next year.

Unfortunately, until these changes have been completed, our secondary storage components could fail again, and this leaves us in a precarious position. Whilst the probability of such a failure is low, and we have taken all possible precautions to ensure it doesn’t reoccur, our teams are prepared to restore services as quickly as possible in the event of a second failure. As the amount of data that is migrated to the new infrastructure increases throughout the week, the amount of time to restore services in the event of an issue reduces. This does mean though that should a similar issue occur early this week, we could experience a similar outage as to what happened on Friday.

As mentioned, I want to be transparent about the challenges we face, and honest about what could happen while we take steps to improve our services. We will let you know as soon as this new environment is fully functional and we can be sure that such issues do not cause as much disruption as they have. In the meantime our team are working diligently to monitor and manage our applications to avoid such issues, and are prepared to restore services as quickly as possible in the event of a reoccurrence of Friday’s troubles. I can also assure you that we will investigate thoroughly what caused these components to fail, but for the time being I want to concentrate all our resources on implementing these changes and improving our service to you.

We will support you as much as we can as a result of this disruption – if there is anything WorldAPP can do to assist you from work you weren’t able to complete last week, such as building surveys, forms or reports, please let your account manager know. We’ll endeavour to accommodate as many requests as we can.

Once again I would like to reiterate my thanks for your patience and understanding, and my genuine sorrow that we have let you down. WorldAPP have been a trusted provider of survey, forms and inspection solutions for over 12 years now, and I hope my explanation of what happened, and assurances of the actions we’re taking to ensure it doesn’t happen again, go some way to rebuilding that trust.

Sincerely,
Oleg Matsko
CEO
WorldAPP, Inc.
161 Forbes Rd Ste 300, Braintree, MA, 02184, US

Retirement of Bombay Print Server

Categories: Midd Blogosphere

Middlebury College is retiring the Bombay print server in July 2015 as it is incompatible with our current 64-bit operating systems. Bombay has been replaced by a new print server named Walnut that has all of the same print queues on it. ITS is reviewing Bombay usage and will be contacting individuals who need to transition to the new print server.

If you’re currently using the Bombay Print Server you must perform the following steps before July 1, 2015 so you do not experience an interruption in printing services:

Steps for Windows 7 computers to move from Bombay to Walnut:

  1. look for printers “on Bombay” print server name listed below queue name. Go to Start>Devices and Printers and look for entries that have “on Bombay” in the name, e.g. “LIB125F on Bombay”
  2. record printer queue names
  3. remove old print queues
  4. add printers from Walnut

Instructions for these steps can be found in our printing documentation: http://mediawiki.middlebury.edu/wiki/LIS/Connect_To_Network_Printers_-_Windows

Steps for Macintosh OS X computers to move from Bombay to Walnut:

  1. record printer queue names(the names are posted on the printers)
  2. remove old print queues
  3. add printer from Walnut via Casper’s Self Service(or manually)

Instructions for these steps can be found in our printing documentation: http://mediawiki.middlebury.edu/wiki/LIS/Connect_To_Network_Printers_-_Mac_OS_X

If you have any questions or need further assistance, please contact the Helpdesk at 802. 443.2200 or via email to helpdesk@middlebury.edu.

HR Update: This Week’s Employment Snapshot

Categories: Midd Blogosphere

There are currently 4 faculty positions, 54 external job postings (regular, on-call and temporary), and 5 internal job postings on the Middlebury College employment opportunities web sites.

Employment Quick Links:

Faculty Employment Opportunities:http://www.middlebury.edu/academics/administration/prospective_faculty/employment

Staff Employment Opportunities: go/staff-jobs (on campus),http://go.middlebury.edu/staff-jobs (off campus)

Please note – to view only internal staff postings, please use the internal posting search filter that was highlighted in this MiddPoints article.

On-call/Temporary Staff Employment Opportunities: go/staff-jobs-sh (on campus), http://go.middlebury.edu/staff-jobs-sh (off campus)

The Week’s Headlines

Categories: Midd Blogosphere

Here are the week’s headlines from the News Room:

College Names Kellogg Fellows in Humanities

Quoted this Week: Bill McKibben on Obama’s ‘Catastrophic Climate-Change Denial’

Neil Sinclair Named Men’s Hockey Coach

Local Company Files Request to Build Renewable Natural Gas Plant that Will Bring Middlebury College Close to Carbon Neutrality

New England Review Goes Digital with Publication of Current Issue

View past stories by visiting the News Room Page

End-of-Year Reception!

Categories: Midd Blogosphere

The School in France held its End of Term Reception for students, faculty, staff, host families and foyers at the Madeleine Center on May 5th. It was a lovely, and for some, the last occasion to get together and to celebrate the end of a challenging but thrilling semester and/or academic year in Paris. We profusely thank again the following individuals: our faculty, without whom our academic program wouldn’t be what it is today; EUSA France for its engaging internship program; and our host families and foyers who contribute so much to the success of our students’ cultural and linguistic immersion experience. Merci et bonne fin de séjour à tous nos étudiants!DSCN0381 DSCN0378 DSCN0355DSCN0341DSCN0339DSCN0338DSCN0331DSCN0327DSCN0321

Key Survey / WorldApp Services Restored

Categories: Midd Blogosphere

KeySurvey Logo

As of 8:15 pm today (Fri, 5/15/15), Key Survey functionality has been restored.  WorldApp is conducting a thorough investigation and will be sharing full details with us as soon as they are available.

 

 

 

Island Time

Categories: Midd Blogosphere

IslandWeb

I check my watch again—likely for the 10th time these past two minutes. It’s 6:25 p.m., and the 5:30 “Speedy’s” ferry has yet to leave the dock. I do the math in my head, even though I know there’s no chance I’ll make the connecting ferry to Virgin
Gorda in the British Virgin Islands.

I flip through my notebook, where I’ve written down phone numbers for other ferry services and hotels in the area. I like schedules, efficiency, timeliness. And this night is not going as I’d planned.

I’m about to begin a monthlong internship, an environmental research expedition in the Caribbean. The other ferry passengers around me don’t seem concerned about the lack of timeliness. A baby peeks over the seat, chocolate-brown, sleepy eyes watching me tap my fingers.

Looking for assistance, I ask the man working at the ferry dock when we might be leaving.

He laughs.

“Why are you in such a hurry?” he asks.

Without waiting for an answer, he tells me about “island time.” Apparently island time means nothing is on time.

An hour and fifteen minutes after the scheduled departure, we push away from the dock. We pick up speed, crashing over waves in ways that seem reckless. “Finally,” I sigh.

I’ve always been obsessed with moving forward. In high school I worked endlessly, participating in every imaginable activity to craft the perfect resume to get me into a school like Middlebury. And while I enjoyed these activities—at least I thought I did; in retrospect, I’m not sure I took the time necessary to enjoy them properly— often my primary motivation was to check another item off my mental list: things I needed to do to succeed.

At Middlebury, I’m always working, distributing my hours between athletics, academics, two jobs, and a social life—doing so hoping I’ll find a job after graduation. I have no patience for sitting still. I must always be making progress, always moving forward.

Lately, I’ve been thinking that my exposure to “island time” is starting to change that mindset. While on island time, no matter how badly I wanted to move forward, I couldn’t.

Boxes weren’t checked. And it was okay.

Now, I can’t say that this time of self-
reflection allowed me to “figure everything out.” While gazing out at the beautiful water, I didn’t suddenly realize what I’m supposed to do next; I didn’t figure out how I was going to make an impression upon the world. What I realized—perhaps for the first time—is that trying to figure everything out is a fool’s errand.

When I returned to Middlebury, I resisted the temptation of falling into old habits: I had responsibilities, of course, but I wanted to be responsible for the moment, not the future.

Moving forward may mean a long run down a country road instead of rushing from activity to activity; time doesn’t stand still, but my time does. Instead of devoting countless hours to future plans, I try and turn this devotion to those around me. Instead of worrying about a murky future and trying to blast through the haze, I try to become comfortable with ambiguity.

With graduation approaching, I’m cognizant of the landmark events—graduations, new jobs, promotions—that will mark life’s progression. But if I’m always checking the seconds that go by and focusing on where I need to be next, I’ll forget to notice where I am.

Elizabeth Reed ’15 will graduate this spring as a sociology and anthropology major. She’ll let us know what she plans on doing next—on her own time.