Main site and mobile site speed improvements

Our main website and mobile website make fabulous use of data feeds from blogs, calendars, Twitter, and other sources to provide up-to-date news and information to users of the sites. Some pages in the main site have as many as 3 feeds in the carousel at the bottom of the page and another 2 feeds in sidebars. The mobile site displays 7 feeds on its home page for desktop users.

Unfortunately displaying all of these feeds on the site was causing performance problems for users when those feeds needed to be refreshed.  Even if the blog system returns each feed in under a second, that time adds up and was making users wait for 10 seconds or more for some pages to load.

We are happy to announce that we have now resolved this issue by piping all of our data-feed access through a system we call FetchProxy. FetchProxy sits between Drupal and the source of the data feeds and keeps a fresh copy of the feed always available so that Drupal never has to sit and wait for feeds. Pages that once took more than 10 seconds to load, now load in 1-2 seconds. You can read more about FetchProxy on Github.

Campus Map

We’ve updated the interface for our Campus Map with the help of designers in College Communications. New features include a list of locations with quick icons to help you find special locations, a search interface, an improved photo gallery, and more special filters which allow you to scope the locations on the map. For instance, you can get a map of all the locations on campus with Wireless.

MiddMedia

A permalink is now available for each file in MiddMedia. You can access it for a particular video you click the “Embed Code & URLs” next to any video you are browsing in MiddMedia and click on the “Click here to view and/or download this file.”. Alternately you can construct this URL like this “http://middmedia.middlebury.edu/middmedia/view/dir/[the directory where the file is located]/file/[the name of the file with its extension]

Example: http://middmedia.middlebury.edu/middmedia/view/dir/lafrance/file/go.mp4

This is useful if you want to reference your video without embedding it elsewhere.

We’ve migrated from core Drupal-6 to Pressflow, a back-port of Drupal-7 performance features. Using Pressflow allows us to cache anonymous web-requests (about 77% of our traffic) for 5-minutes and return them right from memory. While this vastly improves the amount of traffic we can handle as well as the speed of anonymous page-loads it does mean that anonymous users may not see new versions of content for at most 5 minutes. Traffic for logged-in users will always continue to flow directly through to Drupal/Pressflow and will always be up-to-the-instant-fresh.

Read on for more details about what has change and where we are at with regard to website performance.

Background

When we first launched the new Drupal website back in February we went through some growing pains that necessitated code fixes (Round 1 and Round 2) as well as the addition of an extra web-server host and database changes (Round 2).

These improvements brought our site up to acceptable performance levels, but I was concerned that we might run into performance problems if the college ended up in the news and thousands of people suddenly went to view our site.

At DrupalCon a few weeks ago I attended a Drupal Performance Workshop where I learned a number of techniques that can be used to scale Drupal sites to be able to handle internet-scale traffic — not Facebook or Google-level traffic, but that of The Grammys, Economist, or World Bank.

Since before the launch of the new site we were already making use of optcode-caching via APC to speed code execution and were doing data caching with Memcache to reduce the load on the database. This system-architecture is far more performant than a baseline setup, but we still could only handle a sustained average of 20 requests each second before the web-host started becoming fully loaded. While this double our normal average of 10-requests per second, it is not nearly enough headroom to feel safe from traffic spikes.

Request flow through our Drupal web-host prior to May 13th; using normal Drupal page-caching stored in Memcache. Click for full-size.

Switching to Pressflow

Last week we switched from the standard Drupal-6.16 to Pressflow-6.16.77, a version of Drupal 6 that has had a number of the performance-related improvements from Drupal-7 back-ported to it. Code changes in Pressflow such as dropping legacy PHP4 support and using only MySQL enable Pressflow execute about 27% faster than Drupal, a useful improvement but not enough to make a huge difference were we to get double or triple our normal traffic.

For us, the most important difference between Pressflow and Drupal-6 is that sessions are ‘lazily’ created. This means that rather than creating a new ‘session’ on the server to hold user-specific information on the first page each user sees on the website, Pressflow instead only creates the session when the user hits a page (such as the login page) that actually has user-specific data to store. This change makes it very easy to differentiate between anonymous requests (no session cookies) and authenticated requests (that have session cookies) and enables the next change, Varnish page caching.

Varnish Page Caching

Varnish is a reverse-proxy server that runs on our web hosts and can return pages and images from its own in-memory cache so that they don’t have to execute in Drupal/Pressflow every single time. The default rule in Varnish is that if there are any cookies in the request, then the request is for a particular user and should be transparently passed through to the back-end (Drupal/Pressflow). If there are no cookies in the request, then Varnish assumes correctly that it is an anonymous request and tries to respond from its cache without bothering the back-end.

Request flow through our Drupal/Pressflow web-host after May 13th; using the Varnish proxy-server for caching. Click for full-size.

Since about 77% of our traffic is non-authenticated traffic, Varnish only sends about 30% of the total requests through to Apache/PHP/Drupal: all authenticated requests and anonymous requests where the cache hasn’t been refreshed in the past 5 minutes. Were we to have a large spike in anonymous traffic, virtually all of this increase would be served directly from Varnish’s cache, preventing any load-increase on Apache/PHP/Drupal or the back-end MySQL database. In my tests against our home-page varnish was able to easily handle more than 10,000 requests each second with the limiting factor being network speed rather than Varnish.

A histogram of requests to the website. Y-axis is the number of requests, X-axis is the time to return requests, '|' requests were handled by Varnish's cache and '#' were passed through to Drupal. The majority of our requests are being handled quickly by Varnish while a smaller portion are being passed-through to Drupal.

MySQL Improvements

During the scheduled downtime this past Sunday, Mark updated our MySQL server and installed the InnoBase InnoDB Plugin, a high-performance storage engine for MySQL that can provide twice the performance of the built-in InnoDB engine in MySQL for the types of queries done by Drupal.

Last week Mark and I also went through our database configuration and verified that the important parameters were tuned correctly.

As the MySQL database is not currently the bottleneck that limits our site performance these improvements will likely have a minor (though wide-spread) effect. Were our authenticated traffic to further increase (due to more people editing for instance) these improvements will be more important.

Where We Are Now

At this point the website should be able to handle at least 20,000 requests/second of anonymous users (10,000 on each of two web-hosts) at the same time that it is handling up to 40 requests/second from authenticated users (20 on each of two web-hosts).

While it is impossible to accurately translate these request rates into the number of users we can support visiting the site, a very rough estimation would be to divide the number of requests/second by 10 (a guess at the average number of requests needed for each page view) to get a number of page-views that can be handled each second. (1)

In addition to how many requests can be handled, how fast the requests are returned is also important. Our current response times for un-cached pages usually falls between 0.5 seconds and 2 seconds. If pages take much longer than 2 seconds, the site can “feel slow”. For anonymous pages cached in Varnish response times range from 0.001 seconds to 0.07 seconds, much faster than Apache/Drupal can do and more than fast enough for anything we need.

The last performance metric that we are concerned with is about the time it takes for the page to be usable by the viewer. Even if they receive all of the files for a page in only 0.02 seconds, it may still take their browser several seconds to parse these files, execute javascript code, and turn them into a displayable graphic. Due to these factors, my testing has shown that most pages on our site take between 1 and 3 seconds for users to feel that our pages are loaded. For authenticated users, this stretches to 2-4 seconds.

Finally please be aware that, anonymous users see pages that may be cached for up to 5 minutes. While this is fine for the vast majority of our content, there are a few cases where we may need to have the content shown be up-to-the-second fresh. We will address these few special cases over the coming months.

Future Performance Directions

Now that we have our caching system in place our system architecture is relatively complete for our current performance needs. While we may do a bit of tuning on various server parameters, our focus now shifts to PHP and Javascript code optimization to further improve server-side and client-side performance respectively.

One big impact on javascript performance (and hence perceived load-time) is that we currently have to include two separate versions of the jQuery Javascript Library due to different parts of the site relying on different versions. Phasing out the older version will reduce almost by half the amount of code that the browser has to parse.

Additional Notes

(1) As people browse the site their browser needs to load the main HTML page as well as make separate requests for Javascript files, style-sheet (CSS) files, and every image. After these have been loaded the first time, [most] browsers will cache these files locally and only request them again after 5 minutes or if the user clears their browser cache. CSS files and images that haven’t been seen before will need to be loaded as new pages are browsed to. For example, the first time someone loads the Athletics page, it requires about 40 requests to the server for a variety of files. A subsequent click on the Arts page would require an additional 13 requests, while a click back to the Athletics page would require on 1 additional request as the images would still be cached in the browser.

Here are the highlights from the last day:

• All of the conference sessions are now available to watch online.
• Librarians might be particularly interested in Shh! This is a (Drupal-powered) Library Website, though I suggest skipping the first 27 minutes as those presenters cover topics not relevant to our setup.
• The tech team at the White House is contributing back to the community the work they’ve done to make the Drupal perform better under high load and the NY State Senate is working to provide an out-of-the-box Drupal configuration for State governments. Whatever your personal politics, I hope you’ll agree that it’s neat to see the government using free resources and then improving them for the rest of the community.
• The Node Access module in Drupal 7 includes a feature that will allow us to remove the “core hack” from Monster Menus, which will allow it to be an accepted module in the Drupal community and help it get adopted by other Drupal users.
• Drupal 7 allows you to build form fields with “states”. For instance, a group of field for taking credit card information could have the state “expanded” if a checkbox labeled “pay online” is checked. This will help us build easier to use interfaces.
• We can combine the Apache Solr project with the Apache Nutch project to create a local search crawler and indexer, like the Google Search Appliance, but with a lot more room for us to expand and a lot less configuration to provide faceted search.
• The next database abstraction layer for Drupal 7 uses PHP’s PDO library, which will support MySQL, postgres, SQLite, MSSQL, and Oracle as database back ends. There are also huge improvements in database replication, allowing us to have a true “hot standby” server, support for prepared statements, transactions, a query builder, and a lot of other stuff that our Banner programmers take for granted.
• While still in its very early stages, there are a number of Drupal projects working on support for various NoSQL platforms, including MongoDB. These systems promise to improve performance by removing some of the technical limitations imposed by storing information in “tables” in databases. Not quite ready for wide use in production, but expect to see a lot more of this in a few years.
• Much of the HTML code printed by code modules in Drupal 7 will use RDF markup. This provides additional information about the elements on the page that are intended for the browser and search engines to use. For example, the text “Price: $9.99″ won’t get picked up by a search engine as a price, but “Price: $<span class="field-item even" property="product:listPrice">9.99</span>” allows the search engine to display that information as a price next to the page listing.

Read on for more notes on each of these points.

Node Access in Drupal 7 (Notes by Ian)

Watch the presentation

In the current version of Drupal, changes to the user permissions to access a piece of content can only truly happen through the core of the software, meaning that our custom modules to Drupal are limited in what they can do to control access. With Drupal 7, we get a new hook, hook_node_access(), that allows any module to define CRUD permissions (Create, Read, Update, Delete), on any node and for any other module to then modify those permissions. I’ll back up for a second and explain that when a Drupal page loads, the software core gets first crack at doing whatever it needs to do and then all of the modules get a change to modify it, using hooks, in an order defined by the site administrator. Any module that implements hook_node_access() will run that function on all nodes and, if it is the last module to implement that function, will have the final say on who can do what with a node.

This is very important for us, because this is exactly the type of thing Monster Menus needs to do. MM introduces “pages” to Drupal, which the core of the software doesn’t know anything about. In MM, all nodes are assigned to a page and then permissions are set on the page. So in Drupal 7, MM will be able to run hook_node_access() to tell the system, “these nodes being loaded right now belong to this page and here’s who has each of these permissions on them”. Right now this is all done with a large amount of heavy lifting that taxes the system. The hope is that, by opening up access to these functions, Drupal 7 will improve site performance and let us do more.

There are some costs to doing this. The query builder in Drupal 7 doesn’t know that things need to be run through the node access logic, so it is a requirement that you add a tag of “node_access” to any query that is run against the node table. Failure to do this is a security violation in Drupal and will get your module flagged by the security team. This is a bit silly, since we need to do this for every query against the node table, why doesn’t the system just add it for us? The presenter said that a patch to Drupal to provide that would likely get approved, but the design philosophy behind the decision is that it’s, “not the API’s responsibility to tell you how to code correctly”. The new node_access hook also only works on single nodes, you can’t run a whole list of them through it, so there is a chance of poor performance for pages that need to load a bunch of nodes and need to execute these functions on all of them.

Instant Dynamic Forms with #states (Notes by Ian)

Watch the presentation

Drupal 7 adds the “#states” form API element which allows us to define behavior for a form specific to a current “state” of the form. For instance, a field could become required depending on whether a checkbox elsewhere on the form is checked or unchecked. You can try out an example of this behavior.

Naturally, these examples are really trivial, but this is a framework with a lot of power in it. The other nice thing about using #states to build forms in Drupal 7 is that all of the JavaScript is created by the forms engine. This makes forms easier to build and less prone to bugs. I hate having to debug JavaScript, so I’m happy to hand off that responsibility to the software.

One potential issue is that the form is rebuilt by the server after every action by the user. If you click a checkbox, the server gets a notice, rebuilds the entire form, and sends it back to you. This makes the form more secure: only the server can decide what elements are part of the form, but potentially very resource intensive. Our Page Settings form already takes a while to load the first time. If it has to load multiple times in between user actions I can see people starting to get frustrated. We’ll have to wait and see.

Web Crawling and Search with Nutch and Solr (Notes by Ian)

How to build a Jobs Aggregation Search Engine with Nutch, Apache Solr and Views 3 in about an hour

Solr-Nutch Architecture (Diagram by Adam)

Apache Solr is a search engine and Apache nutch is a crawler that can be used to populate that search engine. Using these tools together, we can build a local search repository that indexes all the same sites our Google Search Appliance does, but allows us to extend the search experience by adding facets and localized search. Both Solr and Nutch are also available in cloud configurations, meaning that we can offload the processing of these actions if they grow beyond what our local staff and servers can manage.

The biggest advantage of using Solr as a search engine with Drupal is that the two are closely integrated through the Apache Solr Drupal module. Rather than use a search crawler that goes through the site, Drupal will periodically send off highly structured data to the Solr search repository, including all of the metadata associated with a node. For instance, we have a node in the site for every staff job descriptions with fields listed for department, position number, and level. A crawler just sees these as plain text, but Drupal sends each field off to Solr as part of the index. So when you search for “Programmer” you can filter by level, department, or location. Though it isn’t mentioned anywhere in the documentation, I learned that these filters are automatically set up for any content type field that is a radio button, checkbox, or drop down menu. For other fields, we can build our own filters.

It’s true that these services emulate what we already have with our Google Search Appliance. We’ll be meeting later this week to discuss our search strategy and determine whether it will be beneficial to set up this infrastructure, extend how we use the GSA, both, or neither. I’m glad I attended this session before trying to make that decision!

Databases: the Next Generation (Notes by Ian)

Watch the presentation

The big announcement in this session was that the database abstraction layer in Drupal 7 will use PHP’s PDO libraries, meaning that Drupal 7 can run with MySQL, postgres, SQLite, MSSQL or Oracle as a database backend. Currently, it can run using MySQL or (with some reservations) postgres. Microsoft also announced at the conference that they have supplied a beta drive for MSSQL to the PHP project and are engaged in improving the driver to access their database, rather than relying on the volunteer community to provide it.

This opens up a lot of features that I bet our Banner programmers would be surprised to learn we didn’t have in Drupal (since these have been common in the PL/SQL environment forever):

• Prepared statements
• Transactions
• Named placeholders for variables in statements
• Merge and truncate
• Return a result set as any object type
• Multi-insert statements
• Full master/slave replication support for multiple failover servers

The importance of that last point can’t be overstated. Right now, if our primary database server fails, we can switch over to our backup server. However, MySQL servers can only replicate data in one way, so we can’t allow you to modify data on the backup server because it will never be written back to the primary server. However, the database abstraction layer is so basic right now that we can’t differentiate between a query that is writing to the database and one that’s just reading from the database in our code. So, we solved this by denying our web server permission to execute write statements on the backup server. When the failover occurs, you can try to write data, but you’ll get a warning and our error log will start piling up the errors too. This isn’t ideal and the added functionality in Drupal 7′s database abstraction layer solves this problem.

This was also one of those sessions where I was impressed by the amount of knowledge in the room. A fairly esoteric question was asked about prepared statements: where are they prepared? In the code? By the database? It turns out that, in PHP, the answer is different for each database driver. The MySQL driver prepares the query before passing it off to the database, so it’s done in C code on the web server. The Oracle driver passes the query to the database to prepare, then fetches the parameters from the web server. The Oracle preparation is more precise and less error prone since the database is doing it, but introduces latency since another request is needed to the web server to get the parameter information.

MongoDB – Humongous Drupal (Notes by Ian)

Watch the presentation

MongoDB is a key-value index database that can be used to improve performance for very, very high volume sites. The claim is that storing information on the filesystem using key-value pairs allows it to be access more quickly than storing information in tables, like RDBMS (MySQL, MSSQL, Oracle) do. Since there is no schema, data can be added easily (just append) and indexed on any key. Since it’s just files, the database can be replicated easily as well. Here’s an example of a few rows from a MongoDB file:

{"name" : "mongo" , "_id" : ObjectId("497cf60751712cf7758fbdbb")}
{"x" : 3 , "_id" : ObjectId("497cf61651712cf7758fbdbc")}
{"x" : 4 , "j" : 1 , "_id" : ObjectId("497cf87151712cf7758fbdbd")}
{"x" : 4 , "j" : 2 , "_id" : ObjectId("497cf87151712cf7758fbdbe")}

The same structure in an RDBMS might use three separate tables, one to store x, one for j, and one for name. Depending on how x and j are related, there might be a fourth table involved. This is important for Drupal because all the fields used by content types are stored in separate tables. For our job descriptions, we have a table to store the title of the description, another to store the department for the description, another to store the level for the description and so on. When the node is printed to the page, all of these tables need to be access by joining them together, which can become a resource intensive task. Under high load, this causes problems, particularly in MySQL.

The work being done on MongoDB for Drupal is still in its very early stages and probably won’t be ready for widespread use for over a year. Right now they have some of the query functions used in Drupal core implemented, but not all of them and not necessarily through the database abstraction layer, meaning that any module using MongoDB with Drupal needs to be rewritten at this time. For those few sites that have enough visitors to warrant using this, that might be an acceptable trade off, but not for us. “NoSQL” systems like MongoDB will be one of the most interesting developments in web software this decade. We’ll have to see how this develops in parallel with the traditional RDBMS systems.

Scalable infrastructure for Whitehouse.gov (Notes by Adam)

Frank Febrarro of Phase 2 technology, part of the team that developed Whitehouse.gov, gave a session titled Providing a Scalable Infrastructure for Whitehouse.gov. This session talked about all of the techniques used to ensure both that the site would be able to handle the huge visitor load, as well as be extremely secure from defacement and highly available.

• Infrastructure build-out took a team of the same size and the same amount of time as the development work.
• Tested by turning off servers and services to see how the system reacts.
• They found that targeting read and write queries to different database servers with MySQL_Proxy worked great in development, but fell down under their heavy load testing. They ended up having to patch Drupal core (by running PressFlow) to enable retargeting from within Drupal.
• They run two complete data centers, a production data-center and a disaster-recovery data-center. The disaster recovery data-center also includes development environments and complete data replication so that they can continue operations with a complete loss of the production data center.
• The hosts are all RHEL and provisioned with Puppet (60+ servers). They use SELinux to lock down access to files and executables within the hosts and use AIDE to report on unauthorized file access. Puppet allows each type of server to be spawned exactly the same (ensuring that new servers are in audit compliance just like existing ones).
• The Akamai content delivery network is used for three services: Site Accelerator (a reverse proxy for handling page caching), Net Storage for file serving, and Live Streaming. 90% of all traffic hits Akamai and doesn’t need to go through Drupal. Since they have under a hundred authenticated Admin and Editor users (no public users can log in) they have very low authenticated traffic and don’t need to scale authenticated access much.
• PHP code and user files are all served from a NAS system mounted via NFS.
• They run Memcache with the consistent hashing strategy that allows a node to fail and cache to still operate.
• The database backend is MySQL Enterprise with InnoDB. The also use a RAM-based filesystem for temp-tables to improve the performance of file-sort operations.

Database Replication:

They use both Master-Master replication as well as Master-Slave replication. The second master is a hot-swap of the primary master as well as handles all of the slave replication. This means that the primary master only has to handle some reads, all writes, and replicating to a single ‘slave’.

Monitoring

MySQL enterprise monitor, Nagios for infrastructure monitoring, Cacti for graphs.

Replication Monitoring:

Constantly writing to a file with a pool of active slaves, allowing PHP to switch where reads are going. Custom scripts remove slaves when replication fails, rebuild them, then move them back into the pool once they are repaired.

Environmental sync:

Changes to servers and files automatically sync to Akami as well as the disaster recovery site.

Hardware scaling:

Goal is to quickly scale horizontally. Puppet handles the provisioning details, new web servers and database slaves can be brought online in minutes.

Data scaling:

They receive 15,000+ Webform submissions every day. These are stored outside of the main Drupal database to make site restores much quick by not requiring the rebuilding of many GBs of data.

Development Process:

They create a branch per issue, then a branch per release — merging in completed issue fixes.

Release Process:

They run a full-featured staging environment that allows testing of all aspects of changes.

For the published data like Whitehouse visitors, data files are imported using the Drush command line tools.

RDF in Drupal 7 (Notes by Adam)

Exclamations of how the “semantic web” is going to revolutionize the internet as we know it have been voiced for a decade. The session “The story of RDF in Drupal 7 and what it means for the Web at large” described how in the upcoming Drupal 7 semantic tags will be added to Drupal markup as HTML attributes (RDFa). These attributes will make it easier for machines to understand the meaning and context of words on the page. One example of this used in the description is of a bit of price text having the an RDF attribute <div class="field-item even" property="product:listPrice">9.99</div> enabling search engines to display that price in their results rather than just assuming that it is unstructured text. In my view, this change won’t be earth shattering for quite a while (another few years), but will make some things we currently do now (such as search and data re-purposing) work better without as much server-side programming. In the long run there may be bigger implications.

The second neat thing was a mention of an RDF Proxy Module that will enable Drupal to fetch and display content from remote sites via RDF searching. This is sort of like displaying RSS feeds on steroids. The big difference is that rather than being limited to an Feed that has one or more items of similar format, the RDF queries can grab individual or multiple data-elements from a remote page or many remote pages. Even better, these queries can be run against the normal human-consumable web pages rather than requiring a separate RSS or XML feed to be generated on the source site.

This session also showed the use of Sindice Inspector, a tool for viewing, navigating, and graphing the RDF data in a web page. They showed a cool example of a blog post with graphs linking the RDF data on multiple sites, but I wasn’t able to find a URL with such complex RDF data.

Read on if you are interested in the performance and scalability of Drupal, MySQL databases, and web applications in general.

After a two-hour overview of the various areas of application, server, database, and proxy configurations that can affect performance we spent much of the rest of the day deploying a basic Drupal installation on virtual machines and walking through the entirety of the discussed performance optimizations, running benchmarks at each stage.

Benchmarking

While many tools such as Apache JMeter can be used to build complex test plans that more accurately simulate loads seen in production, the Apache Bench (ab) tool is a very simple way to benchmark the effect a change has on the performance of a single page. This is critical for determining if a change that has performance implications increases or decreases the speed at which pages are served. Apache Bench can be run from the command line with arguments for the number of requests to make, the number of concurrent requests to run at the same time, and the URL of the page to test:

ab -n 1000 -c 50 http://host.example.edu/drupal/

As well, a cookie (copied from an authenticated session) can be used to measure the responsiveness for pages that require authentication:

ab -n 1000 -c 50 -C 'SESSd1e57edd95f34cfedb49c219e55ddf26=f35afe5e50caf15cba72c448afe72f81'  http://host.example.edu/drupal/some/page/

Code Profiling

Once slow pages are identified, the location of performance issues can be determined by code profiling.

The FourKitchens wiki has a good page on how to profile Drupal code using XDebugToolkit and generate traces like the one below that help us developers to easily see where in the PHP code most of the processing time is spent.

A code trace weighted by execution time.

MySQL Database Tuning

The first step in setting up a high-performing MySQL database server is to swap out the built-in InnoDB engine with the plug-in version from InnoBase (a subsidiary of Oracle). The InnoDB Plugin performs two or more times faster than the built-in InnoDB engine and is a fully compatible drop-in replacement. (Packages of the InnoDB Plugin for RHEL are available as part of the Remi YUM Repository). Other InnoDB engine implementations such as XtraDB from Percona add additional performance improvements and configuration abilities, but the InnoBase InnoDB plugin is itself a big win.

With the InnoDB Plugin installed, next up is tuning the configuration parameters of the database. In general, all of the configuration tuning is done to ensure that appropriate caches, data, indexes, and table-metadata all remain resident in system RAM rather than requiring the database server to load these from hard disk. For details, see my example my.cnf configuration file with notes as well as these step-by-step tuning instructions from Four Kitchens.

One side note that came up in the workshop was the performance differences between the MyISAM engine and the InnoDB engine. The ‘conventional wisdom’ is that the older, simpler, MyISAM engine is higher performing for read queries than the InnoDB engine since it has less overhead taken up by ensuring data integrity. What was discussed in the workshop is that this supposed edge of the MyISAM engine only actually applies in a tiny subset of cases: where the server is severely memory constrained (such as on small slice of a shared host) or where the data set is massive and cannot be reasonably held in RAM (such as some multi-terabyte reporting databases). Because the MyISAM engine is designed as a file-based engine and the InnoDB engine is designed to hold data in memory, it was said that in cases where much or all of the data and indexes can be held in memory the InnoDB engine will always out-perform the MyISAM engine — even on simple select statements.

Opcode Caching

One performance improvement we already have in place for most of our web applications is to cache compiled PHP ‘opcodes’ in RAM using the PHP APC extension so that PHP scripts don’t need to be read off of the hard disk and compiled each page load.

Having APC enabled also allows very lightweight page-load statistics monitoring to be done with the Drupal Devel-Performance module, giving use summary statistics of the actual page-load times that users are getting from the production webservers.

Note: APC <= 3.0 has an issue where the entire cache is cleared if it becomes full.

Memcache

The second performance improvement that we are already using is to use Memcache and the Drupal Memcache Module to allow cached data to live in memory on the the webservers and not require queries the database server. Since the [MySQL] database server is generally the performance bottle neck and is hard to scale horizontally, even moving simple cache look-ups off of it can be a big performance improvement.

Note: Use the dev version of the Drupal Memcache Module so that one large memory bin (possibly spanning multiple machines) can be used rather than having to separate cache ‘tables’ into multiple bins. The ‘stable’ version of the module didn’t prefix the cache ‘tables’, resulting in excess clearing of cached data when unrelated data was cleared.

PressFlow

PressFlow is a Drupal distribution designed for high performance that has several differences from the main release of Drupal:

• Legacy support for PHP4 is dropped, allowing better performance in PHP5
• Support for databases other than MySQL is dropped, allowing PressFlow to take advantage of higher-performance features than the lowest-common denominator would support
• Rather than being created on the first web request, sessions are only created when data is stored in them. This “Lazy Session Creation” allows proxies like Squid and Varnish to differentiate between anonymous and authenticated traffic and only cache content for anonymous users
• It allows splitting of read-queries to read-only slave databases, taking load off of the read-write master database.

With these changes noted, PressFlow is a drop-in replacement for Drupal Core.
See this page for more information on PressFlow.

Varnish

Varnish is a reverse proxy* that can be run on the web-host in front of the Apache webserver to cache content destined for anonymous users and return that cached data to other anonymous users from memory, without the cached requests ever having to be processed by Drupal code.

* A “Forward Proxy” is a proxy-server that usually sits close to the clients (often at an ISP) and caches requests coming from those clients to the broader internet. Varnish, as a “Reverse Proxy” sits close to the webserver (even on the same machine) and caches requests coming into it from the broader internet.

Since anonymous traffic is often orders of magnitude greater than authenticated traffic, serving pages for anonymous users from a cache before PHP code is even touched can result in huge speed improvements for anonymous users. The cache performance can be as high as ~5,000-10,000 pages per second compared to a max of about 100-300 pages per second for executed Drupal code, even with its internal page caching turned on. Authenticated users get a speed boost as well since the total traffic being processed through the Drupal PHP code drops.

This wiki page has a configuration file for Varnish working with Drupal. The basic idea is that the presence of Google Analytics cookies is ignored for the purpose of determining if the traffic is authenticated or anonymous. If any other cookies are set, then Varnish assumes that the traffic is for an authenticated user and it passes the request through to Apache/Drupal and doesn’t cache the result when sending it back to the client. If no additional cookies are present, then a result from cache is returned if found.

One downside with varnish is that caching is based on Expire headers, meaning that anonymous users may see slightly stale content (often set to be refreshed every 5 minutes or so). The Drupal Varnish module includes options that allow much longer Expire times to be set and pages to be actively cleared from cache when they are changed, but at the cost of additional complexity. Were active cache clearing desired, this would likely involve a lot of testing to ensure that all caches are appropriately cleared in all cases of content change.

Hudson

Hudson is a continuous integration tool that can be used as a kind of cron-on-steroids, allowing nice features like statistics of cron-run execution times and emails sent to admins if cron-runs have failures. It also prevents cron-runs from stepping on each other if previous executions haven’t completed yet.

Other modules and notes related to performance

• Materialized Views
• Boost – Writes out Drupal pages as static HTML. Varnish is a vastly better solution, but not possible on some web hosting providers (not an issue for Middlebury).
• Panels – Can allow caching of blocks and other content pieces even for authenticated users
• Views – Can cache results, just be careful for views where there may be node-access restrictions