While PCI DSS is not an actual law, it is a standard enforced by the credit card industry, and the banks have stated and upheld the policy that they will no longer accept business from non-PCI compliant merchants. The government has used the PCI DSS as a yardstick by which they have measured such regulations as Gram-Leach-Bliley, Sarbanes-Oxley, and most recently the drafting of the Data Accountability and Trust Act.

We employ a device called a Barracuda here at Middlebury which helps us prevent SPAM from flooding our email system. Just shy of a year ago this system was updated to enable it to filter on cardholder information. By default this feature was turned on. We have left this enabled and have begun reporting on these blocked messages and alerting the senders of outbound messages.  The Barracuda is intended to serve both as a SPAM filter and a compliance tool.

1. those that have moderate coverage
2. those that have minimal coverage, typically in lounges only
3. smaller houses that require only one or two WAPs

As mentioned in a previous blog post, LIS has embarked on an ambitious project to upgrade the campus network infrastructure, including improving wireless in the dorms.  All dorms are currently getting new network switches added to their wire closets to prepare for phase two, the deployment of new WAPs.  Dorms in the first group above (and gradually those in the third group as well) are getting all of their old WAPs replaced with newer equipment, while dorms in the second group are getting new WAPs deployed throughout. Dorms in the first group will be re-visited later on to determine if there are still gaps in the wireless coverage.

As an example, Hadley, Milliken, Kelly-Lang and Stewart are in group 1.  These dorms have had all of their old WAPs upgraded.  Hepburn is an example of group 2.  It has had new WAPs installed throughout the dorm.

Atwater A & B, Gifford, Battell, and Chateau are in progress now.  New switches have also been placed but not yet connected in Coffrin, Pearsons, Starr and Painter, which will also need new WAPs installed.

As you might imagine, this is an enormous task requiring lots of planning and coordination.  Please bear with us as we try to do this with minimal interruption to your access to the network!

• Ian Burke started November 29th as our new Network Security Administrator; he is also on the LIS Security Team.  Ian comes most recently from Amherst, NH and was a Security Engineer for TJX in Framingham MA.
• Jim Stuart started December 8th as System Programmer/Administrator.  Jim was Chief Technology Officer for Qvault, Inc. here in Vermont.   He previously worked here at Middlebury from 1993-1999 in various positions in ITS, prior to the merger which created LIS.

The next step is to upgrade the network infrastructure on campus.  This step can roughly be divided into three phases: 1) replace the core network gear in Voter, Davis Family Library and Carr Hall; 2) replace the edge network switches in academic and residential buildings across campus; and 3) replace existing wireless access points in dorms and add new ones.  Obviously, this is a vast oversimplification of what is in fact an extremely complicated project!  (To give just one example, there are 136 different wire closets to be worked on in phase 2, many containing two or more old switches.)

But we are making progress.  Much of the physical installation of the new core equipment is done.  Now, as each academic and residential building gets upgraded, we turn off the routing function for those subnets in the old core router and turn that function on in the new core router.  Finally, we physically move the fiber cables to connect the new edge switch to the new core router.  Most buildings will see a ten-fold increase in network speed when this is done.

On the evening of November 18th, the Donald E. Axinn ’51, Litt. D. ’89 Center for Literary and Cultural Studies at Starr Library became the first major academic building to be upgraded to the new network environment.  16 new switches were installed in the 6 different wire closets in Axinn, the new fiber cables were connected back to the new core router in Voter, and all of the Ethernet cables were moved from the old switches to the correct ports in the new switches.  The entire process took about 5-6 hours.

Next up is McCardell Bicentennial Hall, which is scheduled to be upgraded over two nights, December 15^th and 16^th.  MBH has 7 wire closets and we’ll be installing 32 new switches and replacing more than 1000 ethernet cables.

A future post will describe our plans for improving wireless in the dorms.

• streamline communications
• remove roadblocks in the workflow
• improve service to the functional areas

Central Systems & Network Services will continue to work closely with Marcy, as we do with all of Enterprise Applications staff.  Marcy will be moving to the Davis Family Library at some point, but the details of that move have not been finalized.

I enjoyed having Marcy in my area and certainly learned a lot about Banner and just how complicated it all is from her!  All of us in CS & NS look forward to continuing to work with Marcy.  In fact, the looming upgrade to Oracle 11G will give us plenty of opportunity.  But that’s a subject for another post…

The College maintains two connections to the Internet: our primary connection is via Level 3 Communications, while Vermont traffic (and failover protection) is supported by a local provider named TelJet.   (Both Seven Days and the Burlington Free Press have recently run favorable articles on TelJet.)

Anticipating continuing, steep growth in our network traffic, LIS requested some time ago that Level 3 increase our bandwidth from 200 Mbps to 500 Mbps.  (Our TelJet connection is at 100 Mbps.)  Just last month, Level 3 completed the upgrade.  While you may not have noticed the increased speed on your desktop, you can rest assured that, at least for the time being, the tubes connecting Middlebury to the outside should be big enough to support your academic needs.

Stay tuned to this blog to learn of other plans we have for improving and expanding the campus network!

LIS will occasionally perform maintenance on systems during a scheduled window of Sunday 8AM-10AM EST. The following systems may not be available during that time this Sunday, May 23, 2010. If you have questions or concerns, please contact the Helpdesk at helpdesk@middlebury.edu or (802) 443-2200.

* BannerWeb
* Internet Native Banner

We’ve migrated from core Drupal-6 to Pressflow, a back-port of Drupal-7 performance features. Using Pressflow allows us to cache anonymous web-requests (about 77% of our traffic) for 5-minutes and return them right from memory. While this vastly improves the amount of traffic we can handle as well as the speed of anonymous page-loads it does mean that anonymous users may not see new versions of content for at most 5 minutes. Traffic for logged-in users will always continue to flow directly through to Drupal/Pressflow and will always be up-to-the-instant-fresh.

Read on for more details about what has change and where we are at with regard to website performance.

Background

When we first launched the new Drupal website back in February we went through some growing pains that necessitated code fixes (Round 1 and Round 2) as well as the addition of an extra web-server host and database changes (Round 2).

These improvements brought our site up to acceptable performance levels, but I was concerned that we might run into performance problems if the college ended up in the news and thousands of people suddenly went to view our site.

At DrupalCon a few weeks ago I attended a Drupal Performance Workshop where I learned a number of techniques that can be used to scale Drupal sites to be able to handle internet-scale traffic — not Facebook or Google-level traffic, but that of The Grammys, Economist, or World Bank.

Since before the launch of the new site we were already making use of optcode-caching via APC to speed code execution and were doing data caching with Memcache to reduce the load on the database. This system-architecture is far more performant than a baseline setup, but we still could only handle a sustained average of 20 requests each second before the web-host started becoming fully loaded. While this double our normal average of 10-requests per second, it is not nearly enough headroom to feel safe from traffic spikes.

Request flow through our Drupal web-host prior to May 13th; using normal Drupal page-caching stored in Memcache. Click for full-size.

Switching to Pressflow

Last week we switched from the standard Drupal-6.16 to Pressflow-6.16.77, a version of Drupal 6 that has had a number of the performance-related improvements from Drupal-7 back-ported to it. Code changes in Pressflow such as dropping legacy PHP4 support and using only MySQL enable Pressflow execute about 27% faster than Drupal, a useful improvement but not enough to make a huge difference were we to get double or triple our normal traffic.

For us, the most important difference between Pressflow and Drupal-6 is that sessions are ‘lazily’ created. This means that rather than creating a new ‘session’ on the server to hold user-specific information on the first page each user sees on the website, Pressflow instead only creates the session when the user hits a page (such as the login page) that actually has user-specific data to store. This change makes it very easy to differentiate between anonymous requests (no session cookies) and authenticated requests (that have session cookies) and enables the next change, Varnish page caching.

Varnish Page Caching

Varnish is a reverse-proxy server that runs on our web hosts and can return pages and images from its own in-memory cache so that they don’t have to execute in Drupal/Pressflow every single time. The default rule in Varnish is that if there are any cookies in the request, then the request is for a particular user and should be transparently passed through to the back-end (Drupal/Pressflow). If there are no cookies in the request, then Varnish assumes correctly that it is an anonymous request and tries to respond from its cache without bothering the back-end.

Request flow through our Drupal/Pressflow web-host after May 13th; using the Varnish proxy-server for caching. Click for full-size.

Since about 77% of our traffic is non-authenticated traffic, Varnish only sends about 30% of the total requests through to Apache/PHP/Drupal: all authenticated requests and anonymous requests where the cache hasn’t been refreshed in the past 5 minutes. Were we to have a large spike in anonymous traffic, virtually all of this increase would be served directly from Varnish’s cache, preventing any load-increase on Apache/PHP/Drupal or the back-end MySQL database. In my tests against our home-page varnish was able to easily handle more than 10,000 requests each second with the limiting factor being network speed rather than Varnish.

A histogram of requests to the website. Y-axis is the number of requests, X-axis is the time to return requests, '|' requests were handled by Varnish's cache and '#' were passed through to Drupal. The majority of our requests are being handled quickly by Varnish while a smaller portion are being passed-through to Drupal.

MySQL Improvements

During the scheduled downtime this past Sunday, Mark updated our MySQL server and installed the InnoBase InnoDB Plugin, a high-performance storage engine for MySQL that can provide twice the performance of the built-in InnoDB engine in MySQL for the types of queries done by Drupal.

Last week Mark and I also went through our database configuration and verified that the important parameters were tuned correctly.

As the MySQL database is not currently the bottleneck that limits our site performance these improvements will likely have a minor (though wide-spread) effect. Were our authenticated traffic to further increase (due to more people editing for instance) these improvements will be more important.

Where We Are Now

At this point the website should be able to handle at least 20,000 requests/second of anonymous users (10,000 on each of two web-hosts) at the same time that it is handling up to 40 requests/second from authenticated users (20 on each of two web-hosts).

While it is impossible to accurately translate these request rates into the number of users we can support visiting the site, a very rough estimation would be to divide the number of requests/second by 10 (a guess at the average number of requests needed for each page view) to get a number of page-views that can be handled each second. (1)

In addition to how many requests can be handled, how fast the requests are returned is also important. Our current response times for un-cached pages usually falls between 0.5 seconds and 2 seconds. If pages take much longer than 2 seconds, the site can “feel slow”. For anonymous pages cached in Varnish response times range from 0.001 seconds to 0.07 seconds, much faster than Apache/Drupal can do and more than fast enough for anything we need.

The last performance metric that we are concerned with is about the time it takes for the page to be usable by the viewer. Even if they receive all of the files for a page in only 0.02 seconds, it may still take their browser several seconds to parse these files, execute javascript code, and turn them into a displayable graphic. Due to these factors, my testing has shown that most pages on our site take between 1 and 3 seconds for users to feel that our pages are loaded. For authenticated users, this stretches to 2-4 seconds.

Finally please be aware that, anonymous users see pages that may be cached for up to 5 minutes. While this is fine for the vast majority of our content, there are a few cases where we may need to have the content shown be up-to-the-second fresh. We will address these few special cases over the coming months.

Future Performance Directions

Now that we have our caching system in place our system architecture is relatively complete for our current performance needs. While we may do a bit of tuning on various server parameters, our focus now shifts to PHP and Javascript code optimization to further improve server-side and client-side performance respectively.

One big impact on javascript performance (and hence perceived load-time) is that we currently have to include two separate versions of the jQuery Javascript Library due to different parts of the site relying on different versions. Phasing out the older version will reduce almost by half the amount of code that the browser has to parse.

Additional Notes

(1) As people browse the site their browser needs to load the main HTML page as well as make separate requests for Javascript files, style-sheet (CSS) files, and every image. After these have been loaded the first time, [most] browsers will cache these files locally and only request them again after 5 minutes or if the user clears their browser cache. CSS files and images that haven’t been seen before will need to be loaded as new pages are browsed to. For example, the first time someone loads the Athletics page, it requires about 40 requests to the server for a variety of files. A subsequent click on the Arts page would require an additional 13 requests, while a click back to the Athletics page would require on 1 additional request as the images would still be cached in the browser.

LIS will occasionally perform maintenance on systems during a scheduled window of Sunday 8AM-10AM EST. The following systems may not be available during that time this Sunday, May 13, 2010. If you have questions or concerns, please contact the Helpdesk at helpdesk@middlebury.edu or (802) 443-2200.

• Museum of Art website
• Undergraduate Research funding database
• Archive of old Content Management System
• Helpdesk knowledgebase
• Blackberry Enterprise Server
• ILLiad
• Key Survey
• Athletics Scoreboard and Roster database
• Sports medicine database
• Online Directory photos
• Papercut

Additionally, editing on the college’s main website (www.middlebury.edu) will be disabled, though we expect the website to be viewable during this time.