by Michael Roy, Middlebury College

 

A report on the NERCOMP SIG workshop Let No Good Deed Go Unpunished; Setting up Centralized Computational Research Support, 10/25/06

Introduction
Back to the Future of Research Computing

As Clifford Lynch pointed out at a recent CNI taskforce meeting, the roots of academic computing are in research. The formation of computing centers on our campuses was originally driven by faculty and students who needed access to computer systems in order to tackle research questions. It was only years later that the idea of computers being useful in teaching came into play. And once that idea took hold, it seemed that we forgot about the research origins of academic computing.

Lynch argues that the pendulum is swinging back again, as campuses nationwide report an increased interest in having libraries and computer centers provide meaningful, sustainable and programmatic support for the research enterprise across a wide range of disciplines.

At the October 27, 2005 NERCOMP meeting entitled “Let No Good Deed Go Unpunished,” Leo Hill, Leslie Hitch and Glenn Pierce from Northeastern University gave a presentation about how they planned for and implemented a university computer cluster that serves the research agendas of a wide array of Northeastern’s faculty.

The talks provided good information about the technology planning, the politics and the policy questions that arose, and placed the entire project within an economic model that is useful for analyzing a broad range of academic initiatives taking place on our campuses.

Key Questions:

1. To what extent should support for research computing be centralized?
2. If one runs a centralized research computing facility, how does one secure funding for it?
3. What are some technology strategies for keeping these costs to a minimum?
4. How can one justify the establishment of a centralized research facility in language that makes sense to academic administrators?
5. How can this impulse be explained in terms of current trends in computation in particular and research in general?
6. How do you allocate resources to map to institutional priorities?

Part One
On the Ground: Technical Considerations
Speaker: Leo Hill, Academic and Research Technology Consultant, Northeastern University

Slides available at https://myfiles.neu.edu/l.hill/deed/

How do you support research and high performance computing?
As a way into explaining why Northeastern took on the project of building a centralized computer cluster, Hill began his talk by making the claim that faculty are not experts at many of the technologies that are required to provide a robust cluster computer environment (OS, Patches, Security, Networking). He also shared his impression that the National Science Foundation and other funding agencies increasingly look for centralized support as part of the overhead that they pay to universities.

In addition, a major benefit to a centralized facility is that a university can enjoy volume discounts for hardware and software, as well as for the considerable costs associated with creating a space to house a large cluster. These costs primarily revolve around power and air conditioning.

How did the process of designing this space work?
A Research Computing steering committee was created. The group’s job was to understand the needs of the greater community. They conducted interviews about present and future research projects of the Northeastern faculty, as a way to understand what sort of software and computational horsepower they would need. In analyzing the results of these interviews, they asked: Are there consistent terms? How can we resolve conflicting concepts? How do we translate these various desires into a viable service?

Their solution was to build a cluster that had the following features:

1. Job management (queues)
2. Ability to interactively run jobs (required for some users)
3. Ability to support large files
4. Ability to efficiently support large data sets (in excess of 4 gig)

As is true of all centrally-managed computational facilities, they had to factor in (and make trade-offs between) processing power and very large file storage. The list of software that the cluster would be supporting (see slides) was large but did not seem to exceed what most schools support on a range of devices on their network.

Once they had the hardware and software requirements in place, the team chose to develop an RFP (request for proposal) in order to collect bids from multiple vendors. Hill used a web-based service offered by HP (http://linuxhpc.org) for both developing and distributing RFP. As cluster computing has matured into a commodity that one can buy, vendors have begun to provide data on the impact of their systems upon air conditioning and power, allowing a better understanding of the overall set-up costs of a data center.

One of the more alarming aspects of the requirements of this project was that it all had to be accomplished with no new staff. This drove the team to look for a vendor-supported turnkey solution (they ended up choosing Dell with ROCKS as the platform). With no new staff, there has been some impact on existing services. The helpdesk now needs to be able to respond to new types of questions. System administration is accomplished by two existing staff who collectively dedicate roughly four hours per week to this service. They also needed to develop service level agreements around node downtime. How quickly should they respond if a single node goes down? What if the head end of the system is no longer functioning? Implicit in making software available is the support for that software, which has meant that they have also reinstated a dormant training program to explain how to work in this environment, and to provide some support for particular applications.

While the cluster is presently offered as a free service, the work on developing the cluster has triggered interest in and the development of other services at Northeastern. This includes selling rackspace in the datacenter, advanced programming support, and increased (and welcome) consultation on grant writing and equipment specifications.

Part Two
Campus Politics and Process
Speaker: Leslie Hitch, Director of Academic Technology, Northeastern University

Slides available at https://myfiles.neu.edu/l.hill/deed/

While Hill’s presentation provided useful insights into the actual process by which the particular hardware and software were selected, installed and managed, Hitch’s talk focused on the institutional framework in which the project was carried out. Northeastern’s issues should be quite familiar to anyone working in higher ed today. The University’s academic plan calls for an increase in the quantity and quality of faculty research, and the project responds nicely to that area. It also calls for increased undergraduate involvement in research, which can be linked to this project as well. Advocates also linked the project to a possible boost in NEU’s rankings in US News & World Report, suggesting that ignoring research computing is something that one did only at one’s peril.

While the project was driven partially by actual faculty demand, it also anticipated growth in need in the social sciences and humanities, which do not have the traditional funding streams that the scientists enjoy. (For more information, see the draft report on Cyberinfrastructure for the Humanities and Social Sciences, recently published by the American Council of Learned Societies.)

In order to design the system, Hitch’s team set out to find what is common among various scientists and social scientists—a perfectly fine question, and one that those wanting to document the complex working relationships among their various faculties would be well-advised to consider. The act of asking people about what they do with technology, and what they would like to do with technology, almost always reveals useful insights into the nature and structure of their disciplines.

While the list of differences (software, memory requirements, gui v. command line, support requirements) in this case was framed as a means of specifying a particular system, the differences can also be understood in terms of what is called “scholarly infrastructure,” based on Dan Atkins’s recent work for the NSF in describing “cyberinfrastructure.” The slide below—from Indiana University’s recent Educause presentation—suggests a useful way of visualizing what particular disciplines have in common, and how they differ.

Source: “Centralize Research Computing to Drive Innovation, Really,” a presentation by Thomas J. Hacker and Bradley C. Wheeler, Indiana University.

Of course, with increased bandwidth among our schools, the act of centralization need not necessarily stay within the campus. Couldn’t our faculty share infrastructure by discipline in multi-institutional facilities staffed by domain experts who can help with the domain-specific applications? What of the various national supercomputer centers? Why should we build campus-specific clusters if the NSF and others will provide for us through national centers?

One answer to this question lies in governance. For such centers to be sustainable, there needs to be a funding model in place, and a fair and agreed-upon system for allocating computational cycles and provisioning support. (Hitch provides the charge to their user group in her slides.)

Northeastern’s funding model, not yet fully articulated, is to be determined by its users. Northeastern has also decided to allow the users of the system to develop their own policy about the allocation of computational cycles. Since there is no new FTE attached to this project, they do not have to worry about how to allocate the provision of support!

One funding model under discussion links awareness of IT to sponsored research. How can IT be used to bring in more money for research? Is providing this service something that should be part of overhead? If so, how do you go about securing a portion of overhead to dedicate to this sort of facility?

If one believes that the future of academic research lies in the increased use of such facilities, the question of staffing these facilities becomes critical. Is it enough to fund centralized facilities just to avoid the costs of lots of little clusters and to promote outside funding, allowing faculty to raise more money? One needs to more fully understand the support needs of such a transformed enterprise. In the discussion, hard questions arose about who would be providing this sort of support. Who pays for these people? To whom do they report? Even harder, where do they come from? How do you find people who can do this kind of work with/for the faculty? Does shifting the research associate from the department to the central IT universe reduce the amount of freedom, control, and experimental play? How can one build into the structure of these new types of support positions the ability to raise funds, to do research, to stay engaged in the field?

Part Three
Academic Research Process and IT Services
Speaker: Glenn Pierce, Director, IS Strategy and Research, College of Criminal Justice, Northeastern University

The next session moved from the particulars of Northeastern’s technical and political environment to a broader reflection on the implications of centralized research computing support for the academic enterprise. Pierce began by using the history of other enterprises (most notably, banking) to suggest that there are profound changes underway that could (for many disciplines) completely transform their way of conducting research, and eventually affect what happens in the classroom.

Using language more familiar to business school than to the usual IT conference, Pierce described the research process as a value/supply chain heavily dependent on IT investments and support. In this model, any break in the chain disrupts the process, slowing down the rate at which the faculty member can productively produce research, while new efficiencies (faster hardware, better software, training of faculty, hands-on IT support) can improve the efficiency of the process.

 

Source: Weill, Peter and Marianne Broadbent. Leveraging the New Infrastructure: How Market Leaders Capitalize on IT. Boston: Harvard Business School Press, 1998.

In a slide reminiscent of the scholarly cyberinfrastructure slide Hitch used, one is able to see the core question of the day: Where is the cut-off for central services: fast changing local application? shared standard IT applications? shared IT services? For Pierce, central IT should aim to go as high up the pyramid as possible.

While Pierce acknowledges that it is a real challenge to imagine a world in which centralized IT has intimate knowledge about domain-specific applications, he also challenges colleges and universities to re-think what is meant by productivity, and to ask not what it costs to provide central IT support for research computing, but instead to ask what it costs NOT to provide it. He argues that faculty doing their own IT represents a loss in productivity and a lost opportunity, and that traditional measures of academic productivity (like traditional measures of productivity in other industries) do not capture the fact that entire industries can be changed, created, or eliminated altogether through the revolution afforded by the powers of computing.

One concrete example Pierce offers is Softricity, an application (like Citrix) that allows one to run applications locally, taking advantage of local computer resources, without installing the application directly on the local machine. This fundamental change in how software can be distributed would require major changes both organizationally and financially. Pierce argues that the predominant model where all budgets across an institution rise and fall at the same rate gets in the way of fundamental change. In the case of Softricity, in order to meet an increased demand for applications and data, we need more money to make this available, and yet these arguments rarely succeed in an academic culture that approaches change incrementally. It is therefore difficult, if not impossible, to fundamentally re-tool to take advantage of the power and increased productivity enabled by centralized IT services.

If one accepts the argument that investing in central IT makes good business sense, and one is looking for other parts of the academic enterprise where one can point to increased productivity, Pierce suggests that the same productivity gains enjoyed by centrally-supported research initiatives can be (hypothetically) found in student education outcomes. This tantalizing claim, not backed up by examples, certainly seems worthy of further investigation.

So what keeps us from all changing overnight from our distributed model back to something that looks, to many, an awful lot like the old mainframe centralized model? Pierce identifies four major barriers to extending centrally-supported IT for research:

1. The existing perception of IT service (many researchers simply do not believe central IT is up to the task)
2. Current funding models that
   1. are balkanized
   2. measure costs rather than productivity
   3. make it difficult to measure or even see cost of lost opportunities
3. Current planning models that suffer from the same problems as our funding models
4. Anxiety over the loss of local control

Using the scholarly infrastructure model, Pierce made the point that the further one moves away from technical issues of hardware, operating systems and networking, and into the domain of discipline-specific software, the more involved faculty need to be in the planning process. He also makes the point that the sort of financial re-organization required to support this shift toward a centralized model requires a genuine partnership between the IT leadership and academic leadership. All of this is possible only if the campus really and truly believes that IT-supported research can fundamentally change for the better how we conduct research and eventually how we educate our students.

Conclusions
Possible Futures & Implications

What follows is a list of possible changes in the daily operations on campuses that embrace the idea of investing in the support of IT-supported research, and a few ideas for collaboration between campuses (or business opportunities):

1. Change the way you distribute software to allow more ubiquitous access to software, using technologies such as Softricity or Citrix.
2. Fund more aggressively-centralized resources such as clusters.
3. Hire discipline-aware IT support staff who can work with faculty on research problems.

As our campuses become increasingly connected by high-speed networks, one can ask questions such as:

1. Can we negotiate licenses with vendors that would allow us to consortially provide access to software?
2. Can we create local clusters that multiple campuses can fund and support?
3. Can discipline-specific support be organized consortially to allow (for example), an economist at School A in need of help with SAS to get that help from a SAS expert at School B?

What does cluster and research computing have to do with liberal arts education?
One can imagine protests about shifting institutional resources into IT-supported research computing. For some this will be seen as an unwelcome return to the early days of campus computing, when a disproportionate share of the support went to a few faculty from the handful of fields that had discovered how to use computers to facilitate their research. As in the first generation of campus computing, however, this trend may be a harbinger of demands that will arise across campus and across disciplines. If one takes seriously the propositions put forth in the recent American Council of Learned Societies report on cyberinfrastructure for the humanities and social sciences, this re-alignment of resources in support of changing requirements for scientific and quantitative research is very likely one of the re-alignments that will be required to support teaching, research, and scholarly communications in ALL disciplines.

Further Readings

Educause Resource Library on Cyberinfrastructure
http://www.educause.edu/Cyberinfrastructure/645?Parent_ID=803

“The new model for supporting research at Purdue University,” ECAR Publication (requires subscription)
http://www.educause.edu/LibraryDetailPage/666?ID=ECS0507

Beyond Productivity, National Academy of Sciences
William J. Mitchell, Alan S. Inouye, and Marjory S. Blumenthal, Editors, Committee on Information Technology and Creativity, National Research Council, 2003.
http://books.nap.edu/html/beyond_productivity/

Speaker Contact Information

Leo Hill, Academic and Research Technology Consultant, Northeastern University l.hill@neu.edu

Leslie Hitch, Ed.D. Director of Academic Technology, Northeastern University l.hitch@neu.edu

Glenn Pierce, Ph.D, Director-IS Strategy & Research, College of Criminal Justice, Northeastern University g.pierce@neu.edu