Here are the highlights from the last day:

• All of the conference sessions are now available to watch online.
• Librarians might be particularly interested in Shh! This is a (Drupal-powered) Library Website, though I suggest skipping the first 27 minutes as those presenters cover topics not relevant to our setup.
• The tech team at the White House is contributing back to the community the work they’ve done to make the Drupal perform better under high load and the NY State Senate is working to provide an out-of-the-box Drupal configuration for State governments. Whatever your personal politics, I hope you’ll agree that it’s neat to see the government using free resources and then improving them for the rest of the community.
• The Node Access module in Drupal 7 includes a feature that will allow us to remove the “core hack” from Monster Menus, which will allow it to be an accepted module in the Drupal community and help it get adopted by other Drupal users.
• Drupal 7 allows you to build form fields with “states”. For instance, a group of field for taking credit card information could have the state “expanded” if a checkbox labeled “pay online” is checked. This will help us build easier to use interfaces.
• We can combine the Apache Solr project with the Apache Nutch project to create a local search crawler and indexer, like the Google Search Appliance, but with a lot more room for us to expand and a lot less configuration to provide faceted search.
• The next database abstraction layer for Drupal 7 uses PHP’s PDO library, which will support MySQL, postgres, SQLite, MSSQL, and Oracle as database back ends. There are also huge improvements in database replication, allowing us to have a true “hot standby” server, support for prepared statements, transactions, a query builder, and a lot of other stuff that our Banner programmers take for granted.
• While still in its very early stages, there are a number of Drupal projects working on support for various NoSQL platforms, including MongoDB. These systems promise to improve performance by removing some of the technical limitations imposed by storing information in “tables” in databases. Not quite ready for wide use in production, but expect to see a lot more of this in a few years.
• Much of the HTML code printed by code modules in Drupal 7 will use RDF markup. This provides additional information about the elements on the page that are intended for the browser and search engines to use. For example, the text “Price: $9.99″ won’t get picked up by a search engine as a price, but “Price: $<span class="field-item even" property="product:listPrice">9.99</span>” allows the search engine to display that information as a price next to the page listing.

Read on for more notes on each of these points.

Node Access in Drupal 7 (Notes by Ian)

Watch the presentation

In the current version of Drupal, changes to the user permissions to access a piece of content can only truly happen through the core of the software, meaning that our custom modules to Drupal are limited in what they can do to control access. With Drupal 7, we get a new hook, hook_node_access(), that allows any module to define CRUD permissions (Create, Read, Update, Delete), on any node and for any other module to then modify those permissions. I’ll back up for a second and explain that when a Drupal page loads, the software core gets first crack at doing whatever it needs to do and then all of the modules get a change to modify it, using hooks, in an order defined by the site administrator. Any module that implements hook_node_access() will run that function on all nodes and, if it is the last module to implement that function, will have the final say on who can do what with a node.

This is very important for us, because this is exactly the type of thing Monster Menus needs to do. MM introduces “pages” to Drupal, which the core of the software doesn’t know anything about. In MM, all nodes are assigned to a page and then permissions are set on the page. So in Drupal 7, MM will be able to run hook_node_access() to tell the system, “these nodes being loaded right now belong to this page and here’s who has each of these permissions on them”. Right now this is all done with a large amount of heavy lifting that taxes the system. The hope is that, by opening up access to these functions, Drupal 7 will improve site performance and let us do more.

There are some costs to doing this. The query builder in Drupal 7 doesn’t know that things need to be run through the node access logic, so it is a requirement that you add a tag of “node_access” to any query that is run against the node table. Failure to do this is a security violation in Drupal and will get your module flagged by the security team. This is a bit silly, since we need to do this for every query against the node table, why doesn’t the system just add it for us? The presenter said that a patch to Drupal to provide that would likely get approved, but the design philosophy behind the decision is that it’s, “not the API’s responsibility to tell you how to code correctly”. The new node_access hook also only works on single nodes, you can’t run a whole list of them through it, so there is a chance of poor performance for pages that need to load a bunch of nodes and need to execute these functions on all of them.

Instant Dynamic Forms with #states (Notes by Ian)

Watch the presentation

Drupal 7 adds the “#states” form API element which allows us to define behavior for a form specific to a current “state” of the form. For instance, a field could become required depending on whether a checkbox elsewhere on the form is checked or unchecked. You can try out an example of this behavior.

Naturally, these examples are really trivial, but this is a framework with a lot of power in it. The other nice thing about using #states to build forms in Drupal 7 is that all of the JavaScript is created by the forms engine. This makes forms easier to build and less prone to bugs. I hate having to debug JavaScript, so I’m happy to hand off that responsibility to the software.

One potential issue is that the form is rebuilt by the server after every action by the user. If you click a checkbox, the server gets a notice, rebuilds the entire form, and sends it back to you. This makes the form more secure: only the server can decide what elements are part of the form, but potentially very resource intensive. Our Page Settings form already takes a while to load the first time. If it has to load multiple times in between user actions I can see people starting to get frustrated. We’ll have to wait and see.

Web Crawling and Search with Nutch and Solr (Notes by Ian)

How to build a Jobs Aggregation Search Engine with Nutch, Apache Solr and Views 3 in about an hour

Solr-Nutch Architecture (Diagram by Adam)

Apache Solr is a search engine and Apache nutch is a crawler that can be used to populate that search engine. Using these tools together, we can build a local search repository that indexes all the same sites our Google Search Appliance does, but allows us to extend the search experience by adding facets and localized search. Both Solr and Nutch are also available in cloud configurations, meaning that we can offload the processing of these actions if they grow beyond what our local staff and servers can manage.

The biggest advantage of using Solr as a search engine with Drupal is that the two are closely integrated through the Apache Solr Drupal module. Rather than use a search crawler that goes through the site, Drupal will periodically send off highly structured data to the Solr search repository, including all of the metadata associated with a node. For instance, we have a node in the site for every staff job descriptions with fields listed for department, position number, and level. A crawler just sees these as plain text, but Drupal sends each field off to Solr as part of the index. So when you search for “Programmer” you can filter by level, department, or location. Though it isn’t mentioned anywhere in the documentation, I learned that these filters are automatically set up for any content type field that is a radio button, checkbox, or drop down menu. For other fields, we can build our own filters.

It’s true that these services emulate what we already have with our Google Search Appliance. We’ll be meeting later this week to discuss our search strategy and determine whether it will be beneficial to set up this infrastructure, extend how we use the GSA, both, or neither. I’m glad I attended this session before trying to make that decision!

Databases: the Next Generation (Notes by Ian)

Watch the presentation

The big announcement in this session was that the database abstraction layer in Drupal 7 will use PHP’s PDO libraries, meaning that Drupal 7 can run with MySQL, postgres, SQLite, MSSQL or Oracle as a database backend. Currently, it can run using MySQL or (with some reservations) postgres. Microsoft also announced at the conference that they have supplied a beta drive for MSSQL to the PHP project and are engaged in improving the driver to access their database, rather than relying on the volunteer community to provide it.

This opens up a lot of features that I bet our Banner programmers would be surprised to learn we didn’t have in Drupal (since these have been common in the PL/SQL environment forever):

• Prepared statements
• Transactions
• Named placeholders for variables in statements
• Merge and truncate
• Return a result set as any object type
• Multi-insert statements
• Full master/slave replication support for multiple failover servers

The importance of that last point can’t be overstated. Right now, if our primary database server fails, we can switch over to our backup server. However, MySQL servers can only replicate data in one way, so we can’t allow you to modify data on the backup server because it will never be written back to the primary server. However, the database abstraction layer is so basic right now that we can’t differentiate between a query that is writing to the database and one that’s just reading from the database in our code. So, we solved this by denying our web server permission to execute write statements on the backup server. When the failover occurs, you can try to write data, but you’ll get a warning and our error log will start piling up the errors too. This isn’t ideal and the added functionality in Drupal 7′s database abstraction layer solves this problem.

This was also one of those sessions where I was impressed by the amount of knowledge in the room. A fairly esoteric question was asked about prepared statements: where are they prepared? In the code? By the database? It turns out that, in PHP, the answer is different for each database driver. The MySQL driver prepares the query before passing it off to the database, so it’s done in C code on the web server. The Oracle driver passes the query to the database to prepare, then fetches the parameters from the web server. The Oracle preparation is more precise and less error prone since the database is doing it, but introduces latency since another request is needed to the web server to get the parameter information.

MongoDB – Humongous Drupal (Notes by Ian)

Watch the presentation

MongoDB is a key-value index database that can be used to improve performance for very, very high volume sites. The claim is that storing information on the filesystem using key-value pairs allows it to be access more quickly than storing information in tables, like RDBMS (MySQL, MSSQL, Oracle) do. Since there is no schema, data can be added easily (just append) and indexed on any key. Since it’s just files, the database can be replicated easily as well. Here’s an example of a few rows from a MongoDB file:

{"name" : "mongo" , "_id" : ObjectId("497cf60751712cf7758fbdbb")}
{"x" : 3 , "_id" : ObjectId("497cf61651712cf7758fbdbc")}
{"x" : 4 , "j" : 1 , "_id" : ObjectId("497cf87151712cf7758fbdbd")}
{"x" : 4 , "j" : 2 , "_id" : ObjectId("497cf87151712cf7758fbdbe")}

The same structure in an RDBMS might use three separate tables, one to store x, one for j, and one for name. Depending on how x and j are related, there might be a fourth table involved. This is important for Drupal because all the fields used by content types are stored in separate tables. For our job descriptions, we have a table to store the title of the description, another to store the department for the description, another to store the level for the description and so on. When the node is printed to the page, all of these tables need to be access by joining them together, which can become a resource intensive task. Under high load, this causes problems, particularly in MySQL.

The work being done on MongoDB for Drupal is still in its very early stages and probably won’t be ready for widespread use for over a year. Right now they have some of the query functions used in Drupal core implemented, but not all of them and not necessarily through the database abstraction layer, meaning that any module using MongoDB with Drupal needs to be rewritten at this time. For those few sites that have enough visitors to warrant using this, that might be an acceptable trade off, but not for us. “NoSQL” systems like MongoDB will be one of the most interesting developments in web software this decade. We’ll have to see how this develops in parallel with the traditional RDBMS systems.

Scalable infrastructure for Whitehouse.gov (Notes by Adam)

Frank Febrarro of Phase 2 technology, part of the team that developed Whitehouse.gov, gave a session titled Providing a Scalable Infrastructure for Whitehouse.gov. This session talked about all of the techniques used to ensure both that the site would be able to handle the huge visitor load, as well as be extremely secure from defacement and highly available.

• Infrastructure build-out took a team of the same size and the same amount of time as the development work.
• Tested by turning off servers and services to see how the system reacts.
• They found that targeting read and write queries to different database servers with MySQL_Proxy worked great in development, but fell down under their heavy load testing. They ended up having to patch Drupal core (by running PressFlow) to enable retargeting from within Drupal.
• They run two complete data centers, a production data-center and a disaster-recovery data-center. The disaster recovery data-center also includes development environments and complete data replication so that they can continue operations with a complete loss of the production data center.
• The hosts are all RHEL and provisioned with Puppet (60+ servers). They use SELinux to lock down access to files and executables within the hosts and use AIDE to report on unauthorized file access. Puppet allows each type of server to be spawned exactly the same (ensuring that new servers are in audit compliance just like existing ones).
• The Akamai content delivery network is used for three services: Site Accelerator (a reverse proxy for handling page caching), Net Storage for file serving, and Live Streaming. 90% of all traffic hits Akamai and doesn’t need to go through Drupal. Since they have under a hundred authenticated Admin and Editor users (no public users can log in) they have very low authenticated traffic and don’t need to scale authenticated access much.
• PHP code and user files are all served from a NAS system mounted via NFS.
• They run Memcache with the consistent hashing strategy that allows a node to fail and cache to still operate.
• The database backend is MySQL Enterprise with InnoDB. The also use a RAM-based filesystem for temp-tables to improve the performance of file-sort operations.

Database Replication:

They use both Master-Master replication as well as Master-Slave replication. The second master is a hot-swap of the primary master as well as handles all of the slave replication. This means that the primary master only has to handle some reads, all writes, and replicating to a single ‘slave’.

Monitoring

MySQL enterprise monitor, Nagios for infrastructure monitoring, Cacti for graphs.

Replication Monitoring:

Constantly writing to a file with a pool of active slaves, allowing PHP to switch where reads are going. Custom scripts remove slaves when replication fails, rebuild them, then move them back into the pool once they are repaired.

Environmental sync:

Changes to servers and files automatically sync to Akami as well as the disaster recovery site.

Hardware scaling:

Goal is to quickly scale horizontally. Puppet handles the provisioning details, new web servers and database slaves can be brought online in minutes.

Data scaling:

They receive 15,000+ Webform submissions every day. These are stored outside of the main Drupal database to make site restores much quick by not requiring the rebuilding of many GBs of data.

Development Process:

They create a branch per issue, then a branch per release — merging in completed issue fixes.

Release Process:

They run a full-featured staging environment that allows testing of all aspects of changes.

For the published data like Whitehouse visitors, data files are imported using the Drush command line tools.

RDF in Drupal 7 (Notes by Adam)

Exclamations of how the “semantic web” is going to revolutionize the internet as we know it have been voiced for a decade. The session “The story of RDF in Drupal 7 and what it means for the Web at large” described how in the upcoming Drupal 7 semantic tags will be added to Drupal markup as HTML attributes (RDFa). These attributes will make it easier for machines to understand the meaning and context of words on the page. One example of this used in the description is of a bit of price text having the an RDF attribute <div class="field-item even" property="product:listPrice">9.99</div> enabling search engines to display that price in their results rather than just assuming that it is unstructured text. In my view, this change won’t be earth shattering for quite a while (another few years), but will make some things we currently do now (such as search and data re-purposing) work better without as much server-side programming. In the long run there may be bigger implications.

The second neat thing was a mention of an RDF Proxy Module that will enable Drupal to fetch and display content from remote sites via RDF searching. This is sort of like displaying RSS feeds on steroids. The big difference is that rather than being limited to an Feed that has one or more items of similar format, the RDF queries can grab individual or multiple data-elements from a remote page or many remote pages. Even better, these queries can be run against the normal human-consumable web pages rather than requiring a separate RSS or XML feed to be generated on the source site.

This session also showed the use of Sindice Inspector, a tool for viewing, navigating, and graphing the RDF data in a web page. They showed a cool example of a blog post with graphs linking the RDF data on multiple sites, but I wasn’t able to find a URL with such complex RDF data.

Read on for more on the following topics:

• Drupal deployment strategies
• The Chaos tools for Drupal module development
• Drupal in Education
• Searching with Apache Solr
• Recent MySQL happenings

Drupal deployment strategies

The session “Don’t Touch that Server”: A toolkit for zero-touch production environments focused on ways of deploying Drupal servers that don’t require SSHing into each machine and running updates. While this is hugely useful when managing 10-100s of servers, most of the techniques aren’t worth the effort for our little cluster of 4 webservers.

I did learn about a few neat tools and techniques that will likely be useful to improve the type of work that we do:

• Splunk — A tool for aggregating, monitoring and viewing server logs from collected from many systems. This could make it much easier to see trends in the access and error logs of the three main-site web servers.
• Syntax checking with PHPLint — Pre-commit hooks can be set up in our source-control systems just to make sure that we never push a typo to production.

The Chaos tools for Drupal module development

The session Leveraging the Chaos tool suite for module development discussed the variety of abilities included in the CTools module that make it much easier to build a variety of dynamic interfaces in Drupal.

Ajax without Javascript
AJAX Responder allows passing commands back and forth between JS and PHP.
Drupal7 AJAX framework based off of this CTools implementation.

Ajax modal windows, the easy way
Modal Dialogs – built on top of the AJAX responder. Allows building modal windows with forms all with just a bit of PHP. Handles form validation and submission.

Object cache – non-volitile cache useful for ‘unsaved states’ during multi-step forms.

Form Wizard – makes it much easier to create multi step forms. Conceptually, its a workflow of a separate single-page forms. Gives you back, finish, cancel, save buttons and controlling widgets and code.

CSS Tools – disassemble, reassemble, filter by properties/values, reassemble/render-css, compress CSS.

Dependent Fields – add two additional properties to form fields and the form will be dynamically changed based on choices.

Drop-down links – basically a single theme function, theme(‘ctools_dropdown’, …), to create drop-down js menus like the contextual options menus in D7 (panels ‘cogs’).

Drupal in Education

Before lunch Ian and I both went to a “Birds of a Feather” discussion on Drupal usage at Colleges and Universities. I split off with a sub-group to discuss the potential of Drupal as an LMS platform. To kick off efforts in this area, we formed a new LMS group at groups.drupal.org to discuss what features are needed in Drupal for it to replace Blackboard, Moodle, Sakai, and other LMS systems.

In conversations with Amherst developers over lunch we were reminded that their Ed-Tech group has already built a Gradebook and a Quiz module for Drupal. While these modules are currently tied somewhat to Amherst’s ERP system (Datatel), with some work they could likely be generalized to work with our Drupal installation as well as those at other schools.

There is another Quiz Module available for Drupal as well.

Keynote: Tim O’Reilly

The keynote today was a talk by Tim O’Reilly called Open Source in the Cloud Era. This was a nice talk, but not earth-shattering if you’ve heard Tim speak at his Web 2.0 conference or elsewhere. Good stuff, familiar theme.

Searching with Apache Solr

Ian and I both attended the Apache Solr Search Mastery session. We recently set up a test instance of the Apache Solr search engine and Ian tried to get it operational for doing faceted searching of custom content types on our site. Unfortunately the documentation on how to do this is scattered all over the internet and an operational system wasn’t created. This session answered all of our questions and should allow us to proceed with setting up a faceted search system as well as other custom search abilities (like section-scoped search) in the future.

Blog posts from the presenters:
http://acquia.com/blog/advanced-apache-solr-example-ip-based-access
http://evolvingweb.ca/story/apache-solr-mastery-how-add-custom-search-paths-hookmenu
http://acquia.com/blog/understanding-apachesolr-cck-api

Notes

Fixed Fields:
Use the site & hash fields to enable using a single search index for multiple sites.
String type is for exact-matched strings like taxonomy terms rather than partial-matched text.

Dynamic Fields:
Allows you to avoid customizing the the schema for custom content fields. These are set up by having a wild-card field for each data-type used in CCK.
CopyFields for strings allow sorting on string fields.
NodeAccess dynamic field allows restricting results based on permissions for most common node-access modules. Not sure if this will work with MM.

APIs:
hook_apachesolr_update_index: Allows adding extra data (such as thumbnail image URLs) to the search index).

hook_apachesolr_node_exclude: Allows custom logic for excluding nodes from search results.

Custom search paths:
Use hook_menu to build up the nice search paths.
Use hook_menu_alter to change the layout of the search page.

Theme search results with custom theme functions.
Note: solr doesn’t do any security filtering of results.

Indexing CCK field info
http://acquia.com/blog/understanding-apachesolr-cck-api
6.1 branch — Fields captured by default: strings in select/options fields
6.2 branch — Adds date fields.

hook_apachesolr_cck_fields_alter(&$mappings): Used to add/change which fields are indexed and how they are indexed.

Recent MySQL happenings

The The Future Of MySQL: Forks, Patches And Decisions session was a good overview of the state of the MySQL database world now that pluggable storage engines are getting more common, Oracle bought Sun (and by extension MySQL-AB), and other developments.

The Oracle InnoDB plugin
http://www.innodb.com/products/innodb_plugin/
- Higher performance version of the InnoDB engine. “Amazing”. Upgrade if at all possible.

XtraDB plugin from Percona
http://www.mysqlperformanceblog.com/2008/12/16/announcing-percona-xtradb…
- A fork of the Oracle InnoDB plugin.
- A big benefit from splitting Buffer Pool Mutex into typed mutexes for each operation so that non-conflicting operations won’t lock.
- Rewrite of RW Locks.
- More configuration for IO Thread Numbers, IO Capacity.

Ourdelta/Open Query – provides builds of MySQL with patch-sets from various sources (Google, etc).

MySQL 5.1 – What’s new?
- Row level Replication rather than SQL-based replication. Removes the need for a lot of strange locks that were required to get SQL-based replication working. Removes the need for repeatability of SQL statements during replication. Still many stability issues, but hopefully they will be fixed soon.
- InnoDB Plugin

MariaDB/MontyProgram
- Pool of threads like apache rather than forking.

Upcoming stuff:

MySQL5.5 – SemiSynch Replication. Allows you to know that the data has been replicated to at least one slave.

Upcoming in MariaDB 5.2: varchar/blob for heap to prevent temporary tables from going to disk.

Other Notes:

Do not put a UNION inside a view! – Performance nightmare.