I think I’m going to move this to a once-a-week posting thing in theory since that’s what it’s already become in reality. Sunday seems like a good time for this too, because I should have all of the code updates for Monday’s push done by now and with nobody else here, I can usually finish up all of my tasks for the day and still have a bit of breathing room at the end, as happened today!

Athletics Recruiting

I’ve finished the Data Analysis, which should really have been completed before any work on the project was started. There are about 700 columns touched by the Web Front End, excluding when the form repeats like when the user enters more than one sport. A single user could bind to more than 1000 Banner columns in the course of filling this thing out. It will be interesting to see if any actually do.

The big trick with this will still be making sure that the default values we use for type codes, such as the home phone number of the recruit, don’t conflict with anything else Admissions is doing. We’ll also have to do some work with the initial population of tables like the questionnaire questions, and person and relative codes that don’t exist yet.

We’ve also made the decision to do this thing in C# rather than PHP, so that we can use the built-in data handling functions to bind to the database, rather than trying to muck with a custom PHP data abstraction layer. I think the savings we’ll see in being able to use built-in form field object, data binding, and native validation exceptions will be well worth abandoning the work that was initially done on the PHP form, much of which would need to be redone at this point anyway.

Athletics Rosters for the CMS

Progress on this has been going faster than I initially thought it would. I’ve completed the code to add in both normal and captain’s profile text to the roster XML. This profile text will be formatted HTML in a CDATA block being generated using the RadEditor control. I had to work with Brett to get the binaries for this in the right folder on the server and we haven’t used the regular RadEditor yet, just the one that binds to CMS placeholders. Because the roster is one big placeholder, we need to use the basic RadEditor so that we can grab the HTML on postback and insert that into an XML element in the big roster placeholder.

I’ve also completed a couple custom data placeholders, including a checkbox to denote whether the student is a captain or not. The checkbox was really tough to do because ASP.NET will return a non-boolean null value from an empty XML field of type “boolean”, which can’t be converted to boolean in the front-end page. This issue was difficult to resolve because all the online examples are of binding to a database backend, rather than XML, and use SQL statements to fetch the data, rather than XPATH or other XML data lookup functions. I finally resolved it by declaring a default value for the field and manually setting the returned data type in the code behind.

I also changed the user interface in a way that I think will make it easier to use. Initially we wanted to have a textbox in which Brad would type the name of an image, then click a button and the CMS would see if the image was in the Resource Gallery. I was having trouble getting an event handler on a button inside a data grid working, so I changed this to just be a drop down box of the elements of the current resource gallery, which is easier for Brad to use anyway. I just need to add some code to handle the event where he returns to edit a record where the image has moved to a new Resource Gallery.

The bulk of the remaining work on this project is creating a rendering script for the captains channels that honors two potential data sources: captains postings from legacy sites and rosters from new sports year channels. With another solid week of programming I could probably have the whole thing done. Of course, there’s no such thing as a solid week of programming :(

Spam Filtering for Form Email

I mentioned off hand to Brett that we’d had a couple users who were getting spammed by email from a web form submission. He told me that the emails from web forms don’t go through the spam filter because they are treated as coming from “on campus”. He suggested that we send emails to shark.middlebury.edu rather than the default smtp address, to force them through the spam filter.

I made this change on plasma and tested it out. It works better than we thought it would because now email addresses are checked for validity when the form is submitted to the server. A non-valid email address will throw a PHP exception, which I can catch and handle. Now we can assure our customers that any emails addresses entered are real, plus give them spam protection. This will also answer a lot of “why didn’t I get a confirmation email?” questions from people who enter their phone number in the email box. Yes, that has happened.

PHP 6 and the Future of PHP

I’ve spent a bit of time this week reading through the PHP Internals Mailing List, which is the one where the core developers for PHP decide on the direction and new features that are going to be available in future releases. There have been two rather large discussions about PHP recently.

First, one of the lead developers randomly decided to create an extension for PHP 6 which will enable “namespaces”. You’ll be able to declare a namespace at the top of any PHP file and then any classes within the file will be considered as part of that namespace. This means that you can have two Exception classes loaded globally if they are logically referenced as part of two different namespaces. The advantages there are obvious to anyone who’s used a language with real namespaces or even something like Python’s packages. It’s looking like this extension will be part of PHP 6’s core, which will be a large functionality improvement for PHP, a language who’s core function library consists of hundreds of functions, all declared globally. Maybe over time these will be moved into namespaces.

And that’s sort of where the second discussion begins. The PHP team would like to retire support of the ereg_* function library into the PECL extension repository, supporting only preg_* regular expression functions in the PHP core from here on. This, of course, triggered a big discussion about what the development team for PHP would continue to support post-PHP 4.

One of the big issues is whether PHP 6 will have an on/off switch for unicode support in the php.ini file. With it, developers will need to write at least two versions of their scripts: one for unicode and one for non-unicode since the string functions in each will work differently. They may need to develop a PHP 5 version as well if they want to continue support for PHP 5. It was mentioned that this support structure is totally unreasonable for large scripts that are expected to work in multiple environments, many times without the users being able to control what is set in php.ini. It remains to be seen if the unicode on/off switch will remain in PHP 6 and what the resolution of these issues means for PHP moving forward.

Abroad View

Looking forward, it is likely that the focus of most of my work this week will be on the Abroad View site design implementation. We’re going to implement this in straight-up XHTML first so that we have templates that can be used on multiple platforms if we’re not able to get the SharePoint site fully operational by the time needed.

Chris has complted work on a unified Photoshop file for the Abroad View design that cleans up a lot of what the vendor left for us. I’ll be using this file and the images that he sliced out of it to create XHTML versions of the three core site templates: front page, aggregation site, and content site. I expect to spend most of Monday and Tuesday working exclusively on this, with deliverables to present at a meeting with the Abroad View folks Wednesday morning.

PaGES Framework

I’ve spent today tuning up the PaGES framework in anticipation of its new use as a frontend to Banner. In order for that to work, the PHP framework really needs to be a well-tuned, well-documented, well-tested machine. We don’t want to get into another Financial Aid web forms support nightmare. The improvements on the PaGES framwork focused on three areas:

1. Eliminate use of cacophony, PEAR, and MiddSettings
2. Comment and test everything
3. Use the new PHP 5.2 features in the SPL and PDO

The first point is an ongoing process. In order to be truly self reliant, we need to get rid of anything that references PEAR and the cacophony project does this all over the place. Fortunately, PaGES was started with some of this in mind and only reverts to relying on PEAR in areas where significant redevelopment is necessary, most notably the reliance on MAuthRA, which uses PEAR::DB_DataObject and PEAR::DB_ldap.

PaGES will continue to rely on MAuthRA for the time being, but anything else is being ported. This means that utility classes, like the Exceptions in cacophony and HTTPQuery need to be redeveloped in PaGES, though this is fairly minor work. I’m also killing off the centralized Singleton class, MiddSettings, and moving any settings into class constants and variables. Includes are done through use of the __autoload magic function in PHP rather than static function calls to MiddSettings::involve.

It’s also important that PaGES have a really solid foundation of code comments and unit tests. The idea for PaGES is that it will be supportable by a team of 2-5 developers within LIS and extensible to meet ongoing needs. For that to work, the library needs to be documented far better than the cacophony project ever was. Unit tests help here too by letting us run a barrage of tests against the whole framework after any change is made to the code base.

The last point focuses on what will be the replacement for some of the utility functionality I was using in PEAR: the standard PHP library functions for container objects and PHP data objects as an engine on top of which the PaGES database abstraction layer will sit. The PHP development team has really done some good work into having a more sensible (though still silly) core library of functions available for PHP 5, so we might as well use them. Hopefully by PHP 6 they’ll have killed off all non-PDO database functions so we won’t have 4 different ways to make a MySQL connection in the core code base. But probably not.

I focused today on porting some of the core classes of the PaGES framework into the not-relying-on-MiddSettings version of the framework with classes like Breadcrumb, Stylesheet, and Header included. The rest of the core classes should more along quickly with more work required for the forms related classes.

Athletics Recruiting
I spent most of last week putting together a spreadsheet of the non-sports-related questions for Athletics Recruiting and their estimated positions in the database schema. There are around 300 affected columns in Banner for these questions and many of the questions, like parent/guardian questions, allow you to add multiple records (i.e. you can continue to dump in parent/guardian information for as many parents/guardians as you like). Of these, there are approximately 10 for which we don’t yet have a column in the database.

There is also the case of many columns for which we don’t yet have a description code. For example, when you add a telephone number you have to specify its type code. There is a type code for “home phone”, but there is a concern that we may want to specify a specific type code for the Athletics Recruiting web front end so that it doesn’t interfere with recruit reporting elsewhere. Hopefully we can find a solution that doesn’t require us going through DIG for approval.

I’d say I’m about 65% done with this spreadsheet with work halted for the moment while I await a revised database schema from Heidi with the changes applied from our discussion last week. We met and made some changes to the original schema to reflect the fact that we’d be gathering data for multiple sports per application on the web form. This was not possible with the original schema which was designed for the tape load process where we only had to worry about a single sport per applicant. There are a number of other revisions, like a way to associate a coach with a sport and adding a relation type field to the relations table for parent, guardians and siblings.

General
I recommend paging through the Software Engineering for Internet Applications book. It’s courseware from an MIT course about designing a web application framework for an online community. There is really good and specific information in there about how to use up user databases and also good philosophical information about how to design for communities. The section on usability is particularly good.

What a day! This one involved a busy morning followed by a productive afternoon. There was also a scheduled update of production today and, as you will see, an unscheduled update of production.

Fall Out from the PHP Upgrade

Yesterday I updated PHP in production after a couple weeks of testing. This included an update to almost all of our PEAR modules. PEAR is an extension repository for PHP that includes a lot of third party scripts that make working with PHP easier. We use PEAR_ldap to make the Directory work and PEAR_DB_DataObject as a database abstraction layer for forms and TouchNet-connected forms.

It was that last PEAR module that really caused problems. Because PHP had been updated so infrequently, we jumped a number of versions of this module and I hadn’t reviewed all of the change notes for all of the different versions. A significant update was made to this package that didn’t show up on the development server, but manifested itself in production.

Each database connection through DB_DataObject is generated by parsing a set of schema files that let the script know what the tables and columns are named and what data type each column is. One of the config files, known as the “link” file can be set up to make joins easier. We never use this because it’s a real pain to write out a custom config file any time you want to add a join. Much easier to just write out the SQL in the query when passing it to the database. Most of the MySQL scripts don’t even use joins. The XML Forms and TouchNet Forms all have only a single table, so joins aren’t necessary for their functionality.

So we never updated the “links” files in production.

And now they’re required.

This caused the production scripts to bomb out with a non-descript error message. The error could not be reproduced in development and the only way to print out the error message in production would have been to start printing the error message, which includes the connection string (i.e. the database username and password). We were told by at least one office that one of their forms was getting hammered, so this wasn’t an option.

After Googling around a bit, I finally landed on a forum post that had to be translated from German that told me what was happening in broken English. I uploaded the file and all was well. Then other forms started breaking, so I uploaded the “links” files for all of the MySQL databases. Now I have to remember to do this whenever the table schema changes for any of the MySQL databases, which happens very frequently for the Forms and TouchNet forms databases.

All in all, this has strengthened my opinion that we just need to move away from using PEAR packages. The PEAR_ldap package in particular is cumbersome and horrendously buggy. The PaGES project is moving away from including anything of the cacophony project, including files that rely on PEAR packages. Hopefully by the time I’m ready to deploy the next version of the Directory, these problems will be a thing of the past.

Athletics Recruiting

I’m nearly done figuring out where all of the information goes, but I need to do a better job of putting it in the correct order. For example, I figured out how to enter information for coaches before I figured out how to enter information for a general high school sport, but the coach questions have to come after the general sports questions.

There are also a lot of these questions which can have multiple answers, such as those regarding siblings, coaches, sports, summer clubs, etc. For these I’ll need to create a mechanism where the student can select to add a new record or go on to the next question. For example, let’s say they have two siblings: a brother and a sister.

1. Select “New Sibling”
2. Select “Brother”
3. Fill in information for brother (name, contact, school)
4. Select “New Sibling”
5. Select “Sister”
6. Fill in information for sister (name, contact, school)
7. Select “No more Siblings”

The process needs to allow the student to repeat it as many times as necessary for the number of siblings they have (or want) to enter into the form, but must also allow the student to move onto the next question at any time. This is very important when we get into multiple stage question groups. For example, for each sport the student is going to have to enter general information, coach information, and statistical information.

This is yet another kind of custom form functionality that I’ll need to develop for the Athletics questionnaire. I don’t think even KeySurvey, which we recently purchased, is able to do repeating branching logic like this. The challenge is to make it transparent to the end user while also coding it in a way that doesn’t break under the strain of added complexity.

Scheduled Production Upgrade
Noting very exciting on this point today. The Fall Family Weekend form mentioned in yesterday’s post was uploaded along with a push of the MCMS project to correct a spelling mistake on one of the error pages.

I’ve been slacking off a lot on writing entires to this thing, so I think I’m going to start picking it up a bit and spending about a half hour at the end of each day writing down what I did during the day. I hope this provides a little insight into what’s going on in Web Services on a day-to-day basis. I fully encourage comments from anyone on how we could be doing these things better or what else you’d like to hear about.

PHP Upgrade to Production

Today marked the first upgrade of PHP in production that we’ve done in a really long time. This one was upgrading PHP 5.1.2 to PHP 5.2.3. There’s really nothing spectacular about this upgrade, except for a pile of security updates. We’re going to be doing many more regular upgrades of PHP than in the past. This updates is a little out of synch of this new philosophy, but we’ll be upgrading to the next-more-recent point version of PHP in the future. Currently PHP 5.2.3 is the most recent version.

This means that if PHP 5.2.4 is released tomorrow, we’ll be waiting until the release of PHP 5.2.5 to upgrade to PHP 5.2.4. If PHP 5.3.0 is released tomorrow, we’ll be waiting until the release of PHP 5.3.1 to upgrade to 5.3.0. The official releases of PHP are typically fully stable by the time the next point release is announced. We want to maintain a balance between keeping the version of PHP current for security purposes with maintaining application stability. We also test the code in development for a week prior to the upgrade to production.

Other than security fixes, PHP 5.2.3 (and the 5.2 branch in general) contains an improved version of the date and time objects, which are now built into the core of the language where they used to require a separate PEAR module. The JSON functionality, used to serialize objects for use in JavaScript, is also built into the language core now. The final big update is really only exciting for hardcore PHP geeks, but the default behavior for an object being printed or echoed is now to call it’s built-in __toString() function.

This last update means that we can do this:

<?php

$addrlist = new AddressList();

$addrlist->add(“Ian McBride”, “imcbride@middlebury.edu”);

print $addrlist;

?>

Instead of this:

<?php

$addrlist = new AddressList();

$addrlist->add(“Ian McBride”, “imcbride@middlebury.edu”);

print $addrlist->__toString();

?>

I’m sure you can see where saving those 13 keystrokes is well worth this massive change to application behavior. Quick note: if the first code block were executed in PHP versions prior to 5.2 the interpreter would print out the Object ID of the AddressList object rather than the return value of the __toString() function.

PHP 4 End of Life

It was recently announced on http://www.php.net that PHP 4 will end its life at the end of this year, meaning that security updates will no longer be issued for the product. For those who don’t know, PHP 4 is the version of PHP currently used on CAT for stuff like the Online Box Office, College Store, and Events Calendar as well as several pro bono hosted sites. While we certainly don’t have the most recent version of PHP 4 installed on CAT, so security updates are something of a moot point anyhow, this should still concern us as it highlights once again our need to remove ourselves from that environment.

Because of the instability of CAT on a weekly basis, the Events Calendar is already targeted for migration to FUSION and PHP 5 as soon as I can get to it. The other applications on this server that we’re responsible for mostly include Business Services applications that may change in nature when the new Point of Sale system is acquired for that functional area. There are also a whole host of web forms on the server that Travis is in the process of migrating to the XML Forms framework on FUSION. If all goes well, we should have major services off CAT by the end of the year, at which point the abandonment of PHP 4 will not concern us.

The impending release of PHP 6 on the other hand….

SharePoint versus MCMS

I answered two CMS related support tickets today as well. The first was a Faculty member who is also an Editor (can make and approve their own changes) to a department site on the CMS. They were asking why a student worker couldn’t add images to the site’s Resource Gallery.

Due to auditing requirements, students working on the CMS are all Authors (can make changes, but not approve them), except for a very small group directly supervised by Tim Etchells. This is to keep students from being able to directly affect the production web site. Unfortunately, while MCMS lets us have Authors able to edit but not approve content, if we give that same person the ability to add multimedia items to the Resource Gallery we have to also give them access to delete multimedia items. This opens the exact issue we’re trying to avoid by making the student an Author: they can now adverseley affect the production site.

Once we’re able to move the CMS content into SharePoint, we will no longer have this issue because the permissions structure of SharePoint is much more granular. Not only can we set permissions at the document level, rather than the container level, but we can set exactly what actions are enabled for a group of users, so we can let students drop their content into the Resource Gallery, but prevent them from deleting any content. SharePoint also provides a multi-stage deletion control system that gives Editor-level users access to a Recycle Bin from which they can restore items without contacting Site Administrators. The Site Admins can set how long items remain in this Recycle Bin after which they’re dumped into another Recycle Bin at the Admin level and then eventually purged.

Fun Note: MCMS never purges data. At all. Once the item is deleted by the admin it’s added to a special place in the database referred to as the “Archive Folder”. There’s no way to get at this data through the API, meaning that you have to violate the MCMS license to delete the data permanently. Consequently, our MCMS database has continued to grow exponentially since implementation.

The ability to set document-level permissions in SharePoint will also help us out with the next support request I closed today. Faculty profiles can be spread accross multiple departments if they teach courses in many disciplines. In order for a Faculty member to add multimedia content to a profile, they have to place this content is a bin where all of the Academic Coordinators have full permissions. With SharePoint, we can just set document level permissions on the posting and related documents and let the Academic Coordinators responsible for the content deal with it.

SharePoint Team Sites

On a less positive SharePoint front, I discovered two problems with the SharePoint Team Site today. First, the custom search box that I had implemented as part of the new design last week doesn’t work correctly. It shows no results for any search, even though search is working correctly. I’ve switched back to the default search box for now until this problem is corrected.

The second issue I discovered is that there is only one RSS feed for each list. This means that you can’t have different feeds for different views in the list. This was brought up because Mike wanted to have a feed for just Banner updates delivered through the Change Log. To do this with the SharePoint list, however, you need to apply filtering on the client end. Coincidently, there was a blog post about this very issue from one of the SharePoint developers today. They recommend using a CQWP to display the custom feed, but that requires writing custom XSLT for your feed unless you want the default view, which is just the title of the item. Interestingly, if you want to get email alerts from a list, you can do that on a per-view basis, so that might be a solution for people not wedded to RSS.

Fall Family Weekend Registration Form

Now that this form has been ported to FUSION, all of the alumni forms have been moved off of CAT. This should help us retire those forms. Heather Cahill had asked us to do so as many of the forms are still linked to through Google and they’ve had a series of registration’s come in for events that were offered years ago. The new XML Forms Framework is much better at handling the on/off nature of these forms.

I’d still like to investigate a better way of handling these items. This was originally part of the Alumni Online Community but was pushed back to the old web forms model because our payment gateway is difficult to integrate with. Each of these takes me 1-2 hours to fully implement correctly and their presence sometimes interferes with features of critical forms like the Giving form. We really need to find a self-service model for this area.

Online Box Office Yearly Maintenance

I discovered today that there’s yet another flaw in the Online Box Office user interface. Debbie Anderson asked for the events from last year to be removed from the database. After asking Chris how we did this in the past, I dumped the database to a backup file and then tried to use the “Delete this Event” option in the admin interface. This dumps you on the image upload page with a bad request and doesn’t remove the event. Now I see why Debbie was asking me to do this.

A quick “delete from events” solved that problem, but with yet another wart on the face of the Online Box Office application. This is one of Tyler’s old development projects. To be fair to him, it has run fairly well considering the business logic it has to bend to. “We’re setting aside this many tickets for online use and this many for in-store use, but adding them together will get you more tickets than we actually have, so we have to limit the number provided to…”, etc. I’ll reiterate my desire that this application reside within the College Book Store, or whatever we brand the combined online Business presence for the College, and tickets be sold just like any other item through that system.

Athletics Recruiting

I’m continuing work on the schema for this questionnaire. Hopefully, when we’ve fully tied each of the questions we want to ask to a field in Banner we’ll all be on common ground when thinking about the implementation of this project. I don’t have a whole lot to say about this right now, but check back for more updates this week as I’ll be working closely with Heidi to square away some of the weirder questions.

One interesting point came up in that we want to tie a recruit’s coach to the High School for which they coach. Coaches are stored in the relations table (SYRRFRL) with siblings and parents. Siblings and parents have college’s associated with them in this table in some columns I proposed named SYRRFRL_SBGI_CODE and SYRRFRL_SBGI_DESC which store the six digit identification number and 30 character school name of the college.

High schools also have SBGI codes! Now, should we shove the coach’s high school code in there and force the application to understand the context of the data in the column? That seems like really bad practice, but the alternative is having separate columns for high schools and colleges and then we get into the case where the coach’s high school isn’t the high school they attended, it’s the high school they work for. Meanwhile the parents and siblings SBGI codes refer to intsitutions which they attended. And what do we do about transfer students?

This project is particularly tricky because of the sheer volume of questions we’re asking of the user. I’m up to thirteen pages of questions and haven’t even got to the sports-specific, athletics-related, extracurriculars, or test score questions yet!