We’ve migrated from core Drupal-6 to Pressflow, a back-port of Drupal-7 performance features. Using Pressflow allows us to cache anonymous web-requests (about 77% of our traffic) for 5-minutes and return them right from memory. While this vastly improves the amount of traffic we can handle as well as the speed of anonymous page-loads it does mean that anonymous users may not see new versions of content for at most 5 minutes. Traffic for logged-in users will always continue to flow directly through to Drupal/Pressflow and will always be up-to-the-instant-fresh.

Read on for more details about what has change and where we are at with regard to website performance.

Background

When we first launched the new Drupal website back in February we went through some growing pains that necessitated code fixes (Round 1 and Round 2) as well as the addition of an extra web-server host and database changes (Round 2).

These improvements brought our site up to acceptable performance levels, but I was concerned that we might run into performance problems if the college ended up in the news and thousands of people suddenly went to view our site.

At DrupalCon a few weeks ago I attended a Drupal Performance Workshop where I learned a number of techniques that can be used to scale Drupal sites to be able to handle internet-scale traffic — not Facebook or Google-level traffic, but that of The Grammys, Economist, or World Bank.

Since before the launch of the new site we were already making use of optcode-caching via APC to speed code execution and were doing data caching with Memcache to reduce the load on the database. This system-architecture is far more performant than a baseline setup, but we still could only handle a sustained average of 20 requests each second before the web-host started becoming fully loaded. While this double our normal average of 10-requests per second, it is not nearly enough headroom to feel safe from traffic spikes.

Request flow through our Drupal web-host prior to May 13th; using normal Drupal page-caching stored in Memcache. Click for full-size.

Switching to Pressflow

Last week we switched from the standard Drupal-6.16 to Pressflow-6.16.77, a version of Drupal 6 that has had a number of the performance-related improvements from Drupal-7 back-ported to it. Code changes in Pressflow such as dropping legacy PHP4 support and using only MySQL enable Pressflow execute about 27% faster than Drupal, a useful improvement but not enough to make a huge difference were we to get double or triple our normal traffic.

For us, the most important difference between Pressflow and Drupal-6 is that sessions are ‘lazily’ created. This means that rather than creating a new ‘session’ on the server to hold user-specific information on the first page each user sees on the website, Pressflow instead only creates the session when the user hits a page (such as the login page) that actually has user-specific data to store. This change makes it very easy to differentiate between anonymous requests (no session cookies) and authenticated requests (that have session cookies) and enables the next change, Varnish page caching.

Varnish Page Caching

Varnish is a reverse-proxy server that runs on our web hosts and can return pages and images from its own in-memory cache so that they don’t have to execute in Drupal/Pressflow every single time. The default rule in Varnish is that if there are any cookies in the request, then the request is for a particular user and should be transparently passed through to the back-end (Drupal/Pressflow). If there are no cookies in the request, then Varnish assumes correctly that it is an anonymous request and tries to respond from its cache without bothering the back-end.

Request flow through our Drupal/Pressflow web-host after May 13th; using the Varnish proxy-server for caching. Click for full-size.

Since about 77% of our traffic is non-authenticated traffic, Varnish only sends about 30% of the total requests through to Apache/PHP/Drupal: all authenticated requests and anonymous requests where the cache hasn’t been refreshed in the past 5 minutes. Were we to have a large spike in anonymous traffic, virtually all of this increase would be served directly from Varnish’s cache, preventing any load-increase on Apache/PHP/Drupal or the back-end MySQL database. In my tests against our home-page varnish was able to easily handle more than 10,000 requests each second with the limiting factor being network speed rather than Varnish.

A histogram of requests to the website. Y-axis is the number of requests, X-axis is the time to return requests, '|' requests were handled by Varnish's cache and '#' were passed through to Drupal. The majority of our requests are being handled quickly by Varnish while a smaller portion are being passed-through to Drupal.

MySQL Improvements

During the scheduled downtime this past Sunday, Mark updated our MySQL server and installed the InnoBase InnoDB Plugin, a high-performance storage engine for MySQL that can provide twice the performance of the built-in InnoDB engine in MySQL for the types of queries done by Drupal.

Last week Mark and I also went through our database configuration and verified that the important parameters were tuned correctly.

As the MySQL database is not currently the bottleneck that limits our site performance these improvements will likely have a minor (though wide-spread) effect. Were our authenticated traffic to further increase (due to more people editing for instance) these improvements will be more important.

Where We Are Now

At this point the website should be able to handle at least 20,000 requests/second of anonymous users (10,000 on each of two web-hosts) at the same time that it is handling up to 40 requests/second from authenticated users (20 on each of two web-hosts).

While it is impossible to accurately translate these request rates into the number of users we can support visiting the site, a very rough estimation would be to divide the number of requests/second by 10 (a guess at the average number of requests needed for each page view) to get a number of page-views that can be handled each second. (1)

In addition to how many requests can be handled, how fast the requests are returned is also important. Our current response times for un-cached pages usually falls between 0.5 seconds and 2 seconds. If pages take much longer than 2 seconds, the site can “feel slow”. For anonymous pages cached in Varnish response times range from 0.001 seconds to 0.07 seconds, much faster than Apache/Drupal can do and more than fast enough for anything we need.

The last performance metric that we are concerned with is about the time it takes for the page to be usable by the viewer. Even if they receive all of the files for a page in only 0.02 seconds, it may still take their browser several seconds to parse these files, execute javascript code, and turn them into a displayable graphic. Due to these factors, my testing has shown that most pages on our site take between 1 and 3 seconds for users to feel that our pages are loaded. For authenticated users, this stretches to 2-4 seconds.

Finally please be aware that, anonymous users see pages that may be cached for up to 5 minutes. While this is fine for the vast majority of our content, there are a few cases where we may need to have the content shown be up-to-the-second fresh. We will address these few special cases over the coming months.

Future Performance Directions

Now that we have our caching system in place our system architecture is relatively complete for our current performance needs. While we may do a bit of tuning on various server parameters, our focus now shifts to PHP and Javascript code optimization to further improve server-side and client-side performance respectively.

One big impact on javascript performance (and hence perceived load-time) is that we currently have to include two separate versions of the jQuery Javascript Library due to different parts of the site relying on different versions. Phasing out the older version will reduce almost by half the amount of code that the browser has to parse.

Additional Notes

(1) As people browse the site their browser needs to load the main HTML page as well as make separate requests for Javascript files, style-sheet (CSS) files, and every image. After these have been loaded the first time, [most] browsers will cache these files locally and only request them again after 5 minutes or if the user clears their browser cache. CSS files and images that haven’t been seen before will need to be loaded as new pages are browsed to. For example, the first time someone loads the Athletics page, it requires about 40 requests to the server for a variety of files. A subsequent click on the Arts page would require an additional 13 requests, while a click back to the Athletics page would require on 1 additional request as the images would still be cached in the browser.

Read on for more on the following topics:

• Drupal deployment strategies
• The Chaos tools for Drupal module development
• Drupal in Education
• Searching with Apache Solr
• Recent MySQL happenings

Drupal deployment strategies

The session “Don’t Touch that Server”: A toolkit for zero-touch production environments focused on ways of deploying Drupal servers that don’t require SSHing into each machine and running updates. While this is hugely useful when managing 10-100s of servers, most of the techniques aren’t worth the effort for our little cluster of 4 webservers.

I did learn about a few neat tools and techniques that will likely be useful to improve the type of work that we do:

• Splunk — A tool for aggregating, monitoring and viewing server logs from collected from many systems. This could make it much easier to see trends in the access and error logs of the three main-site web servers.
• Syntax checking with PHPLint — Pre-commit hooks can be set up in our source-control systems just to make sure that we never push a typo to production.

The Chaos tools for Drupal module development

The session Leveraging the Chaos tool suite for module development discussed the variety of abilities included in the CTools module that make it much easier to build a variety of dynamic interfaces in Drupal.

Ajax without Javascript
AJAX Responder allows passing commands back and forth between JS and PHP.
Drupal7 AJAX framework based off of this CTools implementation.

Ajax modal windows, the easy way
Modal Dialogs – built on top of the AJAX responder. Allows building modal windows with forms all with just a bit of PHP. Handles form validation and submission.

Object cache – non-volitile cache useful for ‘unsaved states’ during multi-step forms.

Form Wizard – makes it much easier to create multi step forms. Conceptually, its a workflow of a separate single-page forms. Gives you back, finish, cancel, save buttons and controlling widgets and code.

CSS Tools – disassemble, reassemble, filter by properties/values, reassemble/render-css, compress CSS.

Dependent Fields – add two additional properties to form fields and the form will be dynamically changed based on choices.

Drop-down links – basically a single theme function, theme(‘ctools_dropdown’, …), to create drop-down js menus like the contextual options menus in D7 (panels ‘cogs’).

Drupal in Education

Before lunch Ian and I both went to a “Birds of a Feather” discussion on Drupal usage at Colleges and Universities. I split off with a sub-group to discuss the potential of Drupal as an LMS platform. To kick off efforts in this area, we formed a new LMS group at groups.drupal.org to discuss what features are needed in Drupal for it to replace Blackboard, Moodle, Sakai, and other LMS systems.

In conversations with Amherst developers over lunch we were reminded that their Ed-Tech group has already built a Gradebook and a Quiz module for Drupal. While these modules are currently tied somewhat to Amherst’s ERP system (Datatel), with some work they could likely be generalized to work with our Drupal installation as well as those at other schools.

There is another Quiz Module available for Drupal as well.

Keynote: Tim O’Reilly

The keynote today was a talk by Tim O’Reilly called Open Source in the Cloud Era. This was a nice talk, but not earth-shattering if you’ve heard Tim speak at his Web 2.0 conference or elsewhere. Good stuff, familiar theme.

Searching with Apache Solr

Ian and I both attended the Apache Solr Search Mastery session. We recently set up a test instance of the Apache Solr search engine and Ian tried to get it operational for doing faceted searching of custom content types on our site. Unfortunately the documentation on how to do this is scattered all over the internet and an operational system wasn’t created. This session answered all of our questions and should allow us to proceed with setting up a faceted search system as well as other custom search abilities (like section-scoped search) in the future.

Blog posts from the presenters:
http://acquia.com/blog/advanced-apache-solr-example-ip-based-access
http://evolvingweb.ca/story/apache-solr-mastery-how-add-custom-search-paths-hookmenu
http://acquia.com/blog/understanding-apachesolr-cck-api

Notes

Fixed Fields:
Use the site & hash fields to enable using a single search index for multiple sites.
String type is for exact-matched strings like taxonomy terms rather than partial-matched text.

Dynamic Fields:
Allows you to avoid customizing the the schema for custom content fields. These are set up by having a wild-card field for each data-type used in CCK.
CopyFields for strings allow sorting on string fields.
NodeAccess dynamic field allows restricting results based on permissions for most common node-access modules. Not sure if this will work with MM.

APIs:
hook_apachesolr_update_index: Allows adding extra data (such as thumbnail image URLs) to the search index).

hook_apachesolr_node_exclude: Allows custom logic for excluding nodes from search results.

Custom search paths:
Use hook_menu to build up the nice search paths.
Use hook_menu_alter to change the layout of the search page.

Theme search results with custom theme functions.
Note: solr doesn’t do any security filtering of results.

Indexing CCK field info
http://acquia.com/blog/understanding-apachesolr-cck-api
6.1 branch — Fields captured by default: strings in select/options fields
6.2 branch — Adds date fields.

hook_apachesolr_cck_fields_alter(&$mappings): Used to add/change which fields are indexed and how they are indexed.

Recent MySQL happenings

The The Future Of MySQL: Forks, Patches And Decisions session was a good overview of the state of the MySQL database world now that pluggable storage engines are getting more common, Oracle bought Sun (and by extension MySQL-AB), and other developments.

The Oracle InnoDB plugin
http://www.innodb.com/products/innodb_plugin/
- Higher performance version of the InnoDB engine. “Amazing”. Upgrade if at all possible.

XtraDB plugin from Percona
http://www.mysqlperformanceblog.com/2008/12/16/announcing-percona-xtradb…
- A fork of the Oracle InnoDB plugin.
- A big benefit from splitting Buffer Pool Mutex into typed mutexes for each operation so that non-conflicting operations won’t lock.
- Rewrite of RW Locks.
- More configuration for IO Thread Numbers, IO Capacity.

Ourdelta/Open Query – provides builds of MySQL with patch-sets from various sources (Google, etc).

MySQL 5.1 – What’s new?
- Row level Replication rather than SQL-based replication. Removes the need for a lot of strange locks that were required to get SQL-based replication working. Removes the need for repeatability of SQL statements during replication. Still many stability issues, but hopefully they will be fixed soon.
- InnoDB Plugin

MariaDB/MontyProgram
- Pool of threads like apache rather than forking.

Upcoming stuff:

MySQL5.5 – SemiSynch Replication. Allows you to know that the data has been replicated to at least one slave.

Upcoming in MariaDB 5.2: varchar/blob for heap to prevent temporary tables from going to disk.

Other Notes:

Do not put a UNION inside a view! – Performance nightmare.

Read on if you are interested in the performance and scalability of Drupal, MySQL databases, and web applications in general.

After a two-hour overview of the various areas of application, server, database, and proxy configurations that can affect performance we spent much of the rest of the day deploying a basic Drupal installation on virtual machines and walking through the entirety of the discussed performance optimizations, running benchmarks at each stage.

Benchmarking

While many tools such as Apache JMeter can be used to build complex test plans that more accurately simulate loads seen in production, the Apache Bench (ab) tool is a very simple way to benchmark the effect a change has on the performance of a single page. This is critical for determining if a change that has performance implications increases or decreases the speed at which pages are served. Apache Bench can be run from the command line with arguments for the number of requests to make, the number of concurrent requests to run at the same time, and the URL of the page to test:

ab -n 1000 -c 50 http://host.example.edu/drupal/

As well, a cookie (copied from an authenticated session) can be used to measure the responsiveness for pages that require authentication:

ab -n 1000 -c 50 -C 'SESSd1e57edd95f34cfedb49c219e55ddf26=f35afe5e50caf15cba72c448afe72f81'  http://host.example.edu/drupal/some/page/

Code Profiling

Once slow pages are identified, the location of performance issues can be determined by code profiling.

The FourKitchens wiki has a good page on how to profile Drupal code using XDebugToolkit and generate traces like the one below that help us developers to easily see where in the PHP code most of the processing time is spent.

A code trace weighted by execution time.

MySQL Database Tuning

The first step in setting up a high-performing MySQL database server is to swap out the built-in InnoDB engine with the plug-in version from InnoBase (a subsidiary of Oracle). The InnoDB Plugin performs two or more times faster than the built-in InnoDB engine and is a fully compatible drop-in replacement. (Packages of the InnoDB Plugin for RHEL are available as part of the Remi YUM Repository). Other InnoDB engine implementations such as XtraDB from Percona add additional performance improvements and configuration abilities, but the InnoBase InnoDB plugin is itself a big win.

With the InnoDB Plugin installed, next up is tuning the configuration parameters of the database. In general, all of the configuration tuning is done to ensure that appropriate caches, data, indexes, and table-metadata all remain resident in system RAM rather than requiring the database server to load these from hard disk. For details, see my example my.cnf configuration file with notes as well as these step-by-step tuning instructions from Four Kitchens.

One side note that came up in the workshop was the performance differences between the MyISAM engine and the InnoDB engine. The ‘conventional wisdom’ is that the older, simpler, MyISAM engine is higher performing for read queries than the InnoDB engine since it has less overhead taken up by ensuring data integrity. What was discussed in the workshop is that this supposed edge of the MyISAM engine only actually applies in a tiny subset of cases: where the server is severely memory constrained (such as on small slice of a shared host) or where the data set is massive and cannot be reasonably held in RAM (such as some multi-terabyte reporting databases). Because the MyISAM engine is designed as a file-based engine and the InnoDB engine is designed to hold data in memory, it was said that in cases where much or all of the data and indexes can be held in memory the InnoDB engine will always out-perform the MyISAM engine — even on simple select statements.

Opcode Caching

One performance improvement we already have in place for most of our web applications is to cache compiled PHP ‘opcodes’ in RAM using the PHP APC extension so that PHP scripts don’t need to be read off of the hard disk and compiled each page load.

Having APC enabled also allows very lightweight page-load statistics monitoring to be done with the Drupal Devel-Performance module, giving use summary statistics of the actual page-load times that users are getting from the production webservers.

Note: APC <= 3.0 has an issue where the entire cache is cleared if it becomes full.

Memcache

The second performance improvement that we are already using is to use Memcache and the Drupal Memcache Module to allow cached data to live in memory on the the webservers and not require queries the database server. Since the [MySQL] database server is generally the performance bottle neck and is hard to scale horizontally, even moving simple cache look-ups off of it can be a big performance improvement.

Note: Use the dev version of the Drupal Memcache Module so that one large memory bin (possibly spanning multiple machines) can be used rather than having to separate cache ‘tables’ into multiple bins. The ‘stable’ version of the module didn’t prefix the cache ‘tables’, resulting in excess clearing of cached data when unrelated data was cleared.

PressFlow

PressFlow is a Drupal distribution designed for high performance that has several differences from the main release of Drupal:

• Legacy support for PHP4 is dropped, allowing better performance in PHP5
• Support for databases other than MySQL is dropped, allowing PressFlow to take advantage of higher-performance features than the lowest-common denominator would support
• Rather than being created on the first web request, sessions are only created when data is stored in them. This “Lazy Session Creation” allows proxies like Squid and Varnish to differentiate between anonymous and authenticated traffic and only cache content for anonymous users
• It allows splitting of read-queries to read-only slave databases, taking load off of the read-write master database.

With these changes noted, PressFlow is a drop-in replacement for Drupal Core.
See this page for more information on PressFlow.

Varnish

Varnish is a reverse proxy* that can be run on the web-host in front of the Apache webserver to cache content destined for anonymous users and return that cached data to other anonymous users from memory, without the cached requests ever having to be processed by Drupal code.

* A “Forward Proxy” is a proxy-server that usually sits close to the clients (often at an ISP) and caches requests coming from those clients to the broader internet. Varnish, as a “Reverse Proxy” sits close to the webserver (even on the same machine) and caches requests coming into it from the broader internet.

Since anonymous traffic is often orders of magnitude greater than authenticated traffic, serving pages for anonymous users from a cache before PHP code is even touched can result in huge speed improvements for anonymous users. The cache performance can be as high as ~5,000-10,000 pages per second compared to a max of about 100-300 pages per second for executed Drupal code, even with its internal page caching turned on. Authenticated users get a speed boost as well since the total traffic being processed through the Drupal PHP code drops.

This wiki page has a configuration file for Varnish working with Drupal. The basic idea is that the presence of Google Analytics cookies is ignored for the purpose of determining if the traffic is authenticated or anonymous. If any other cookies are set, then Varnish assumes that the traffic is for an authenticated user and it passes the request through to Apache/Drupal and doesn’t cache the result when sending it back to the client. If no additional cookies are present, then a result from cache is returned if found.

One downside with varnish is that caching is based on Expire headers, meaning that anonymous users may see slightly stale content (often set to be refreshed every 5 minutes or so). The Drupal Varnish module includes options that allow much longer Expire times to be set and pages to be actively cleared from cache when they are changed, but at the cost of additional complexity. Were active cache clearing desired, this would likely involve a lot of testing to ensure that all caches are appropriately cleared in all cases of content change.

Hudson

Hudson is a continuous integration tool that can be used as a kind of cron-on-steroids, allowing nice features like statistics of cron-run execution times and emails sent to admins if cron-runs have failures. It also prevents cron-runs from stepping on each other if previous executions haven’t completed yet.

Other modules and notes related to performance

• Materialized Views
• Boost – Writes out Drupal pages as static HTML. Varnish is a vastly better solution, but not possible on some web hosting providers (not an issue for Middlebury).
• Panels – Can allow caching of blocks and other content pieces even for authenticated users
• Views – Can cache results, just be careful for views where there may be node-access restrictions