High Performance Computing

>> Support for High Performance Computing

"High performance computing(HPC) is not just hardware. It is essential that we understandthat the total package consists of personnel, software, hardware,networking, physical environment and infrastructure, and Universityendorsement and support."

High performance computing is a pervasive,robust, evolving technology that fundamentally changes almostall facets of modern society for the better, including educationand research. HPC will continue to drive these changes for theforeseeable future. In rapidly changing many aspects of our society,supercomputing continually changes even its own definition, sothat many experts define supercomputing as anything within anorder of magnitude of the fastest computing available for a giventask. As high performance becomes ever higher, it forces the restof computing and technology to keep pace by pulling them in itswake. What was state-of-the-art supercomputing power ten yearsago is now a commodity item on every desktop. This trend, too,will continue for the foreseeable future.

Universities are also leaders that drivechange in society, and it is no coincidence that universitiesare deeply involved in HPC. Universities take a leading rolein generating personnel that develop software, hardware, and networking. Universities serve as test beds for new physical and virtualenvironments and infrastructure, and promote and support ever-expandingapplications of computing.

U.S. universities can trace much of theirsuccess in advancing HPC (and as a result, all computing) to theirtreating HPC as they treat their libraries. For example, theuse of books and journals is free in the library at the Universityof Kentucky, and so is use of the HP Superdome computer. Thelibrary is actually merging with HPC over time, as the librarybecomes a large distributed digital database.

Smaller libraries that generally serve thesame purpose as larger ones are able to borrow and exchange resourcesthat one library might not have available through interlibraryloans, broadening the resources available to students, facultyand staff. The linking of distributed computers for HPC on thegrid is like interlibrary loan between campuses, rendering smallercomputers able to perform big jobs for their users at least someof the time.

The collection of the library is large and diverse. The libraryhas one or a few copies of many different titles. In contrast,the campus bookstore has a smaller collection, and a collectionmore specialized for students who want to use their books longer,write in their books, and tear out pages. The library is a databasewith less redundancy than the bookstore, and the library costsfar less than the bookstore for each individual user. The bookstorehas very many copies of a few titles. Similarly, smaller supercomputingclusters, funded by individual investigators or departments usinggrants, are comparable to campus bookstores in their content,cost to individual users, and usage. Users working on a clusterare able to use their own funds to make a supercomputer designedspecifically for their use meet their specific needs. When thegrants expire, a department or individual must leave the "bookstore."They either get more grants, or go back to the library.

It is common for campuses to permit accessto HPC at no cost through allocation boards. The allocation boardsare functionally similar to librarians, who allocate books andreserve materials for specific periods like the allocation boardsregulate computing time. Just as the library covers a broad arrayof topics, no-cost HPC services large areas of science, mathematics,engineering, art and the humanities. Allocation boards attemptto serve clients in all of these areas, and as a result, eachclient usually receives only enough computing cycles for a projectto reach a limited goal.

Institutional

The University of Kentucky's HPC allocationpolicies ensure that they both encourage and facilitate the utilizationof HPC at UK in a manner designed to meet the institutional goalsand priorities. Like the library policies that permit anyoneto check out a small number of books, the allocation policiesfor HPC permit faculty and students to garner a small amount ofHPC time (less than 1000 hours) without a proposal. This mechanismsupports small efforts in new directions in uses of HPC, particularlyin emerging fields that might lead to large, new research grants. There is a well-defined process in place for larger allocationsthat was developed through the university's Subcommittee on ResearchComputing (SRC) in consultation with the office of the Vice Presidentfor Information Technology and Human Services. Any Level 2 (1,000to 10,000 hours per year) request goes to the SRC and one of thefaculty members on the committee reviews and approves the request. The goal is to approve all worthwhile requests within 48 hours. The SRC response is based on basic research potential only. Because of the similarity between HPC and library services, however,a secondary goal is to be able to eventually approve all requestseven if it means educating and working with the PI directly toimprove the proposal. Finally, there are the researchers whowant even larger amounts of resources. The university asks eachof these PIs to go through an outside review through the nationalallocations boards. This review process is no different thangetting an external grant. There is a learning curve on whatthe boards wanted in these proposals, and it is different fromwriting a NSF or DOE grant proposal or a journal article. Still,successful proposals can be written, and there are examples onlineas well as researchers in Kentucky that can help maintain theshared openness of the library model of HPC.

The University of Kentucky allocation reviewprocess is related to that employed by similar HPC centers atCaltech and Boston University, where the goal is to obtain HPCaccess for local researchers. Several classes have used HPC accessfor educational purposes as well, and each course request is discussedin the SRC. Their projects are not typical routine assignments,but instead are interesting and timely scientific problems. Theundergraduates in these classes have generated a sudden, unusuallyhigh number of applications to graduate school (with good success).

If the University of Kentucky attempts todo all of its supercomputing elsewhere, national allocations willbecome a major issue. Not everyone in the country uses the same"library." Instead, libraries have many branches toserve their clientele. In reducing the number of supercomputerson the grid, impediments to usage by the research community willbe increased instead of reduced. For example, Gaussian (a commonprogram for molecular modeling and simulation used in biologyand chemistry) runs best on machines with a large shared memoryarchitecture. The University of Kentucky Superdome is the bestcomputer on the grid for this purpose. Researchers arrive fromall over the Internet to use Gaussian on the University of Kentuckymachine. If Kentucky ceases to support HPC on its own, a numberof Kentucky researchers will suddenly be vying nationally for"a smaller piece of the pie" with a larger number ofother researchers also displaced by the shutdown of HPC in Kentucky. Similar problems would arise with other software packages. Astrong commitment from the administration in support of HPC, usingthe library as a model for providing services, will ensure thatKentuckians take a leading role technologically in the 21st century.

Funding

The University of Kentucky has had a supercomputeras part of its computational resources since 1987 (J. Connolly,High Performance Research Computing at UK, 2002). In the 15 yearssince acquiring that first IBM 3090, the university has movedfrom invisibility in the field of scientific computing to top-10status. Kentucky has joined the national computational programthrough its membership in the National Computational Science Alliance(NCSA). HPC has also made the computational scientists in Kentuckymuch more competitive for extramural funding. According to datafrom the Office of Sponsored Projects Accounting, the top usersof the supercomputer at the University of Kentucky submitted 250proposals and were awarded 137 grants totaling $21.2 million overthe period from 1997-2002. Of course, one cannot simply evaluatethe grant performance of the supercomputer users by comparinga list of the top 25 HPC users to a list of the top 25 fundedPIs, regardless of how tempting and facile that might be. Fundingvaries among the disciplines, in large part because the cost ofworking in different areas varies. Many HPC users get grantsof supercomputer time instead of dollars, so evaluating publicationperformance would be more realistic. However, publication ratesfor different disciplines vary as much as funding does for differentdisciplines.

At $21.2 million over five years, the fundingreceived by the top users greatly exceeds the cost of the supercomputer,which is presently about $1.3 million per year. The amount ofextramural funding awarded to the top users is likely to increasegreatly over the next few years, and having a modern HPC facilityis essential to the university achieving and maintaining top-20status. Withdrawing from HPC would have a detrimental effect onresearch funding and would lead to the loss of many of UK's topresearchers. A state in which 10% of the people cannot read wouldnever consider that as a reason to close the libraries. On thecontrary, illiteracy is a reason to increase educational outreach. The rate of technological and computational illiteracy in thestate is much higher than 10%. Knowing that HPC is a pervasiveevolving technology that fundamentally changes modern societyfor the better, the university cannot withdraw from it and expectto achieve top-20 status. In fact, it is easy to justify investingfar more in HPC than we do now.

The University of Kentucky has decided that open-access HPC shouldfunction like the library for its users, and rightly so. Thelibrary itself is moving into digital collections in computersas individuals realize a library is simply a very large databasebest suited for HPC. Someday, if the university needs funds forHPC it might even consider charging for access to the physicallibrary. Antiquated paper books, magazines, bricks, mortar, parkingand people, cost more per book and more per periodical to deliverthan a digital collection in a high-performance computer. Chargeswould then be used as an instrument of policy to encourage everyoneto utilize the most efficient, complete, and effective means ofdelivering information.

The university does not charge individualusers directly for centrally provided HPC. Cost Accounting Standardsdictate much about recharge schemes for HPC. The university hasoften considered recharge schemes, but has determined that usingthe library as a model for providing services is the best wayto ensure that Kentuckians take a leading role technologicallyin the 21st century. At the same time, the university can andshould recharge (following the campus bookstore model) for departmentaland individual HPC systems adapted to special purposes or in whichaccess is not open to anyone through the standard allocation system.

The University of Kentucky should work withthe state government in Frankfort to secure a national centerfor computational science with federal, state, and industrialfunding. This HPC center would be funded sufficiently to stayin the Top 10 nationally. Such an HPC center, built around theup-to-date hardware platforms needed to do cutting-edge research,and with a yearly increase in its budget (instead of the 14-year-oldline item budget number now in place), would recognize that HPCis a force that fundamentally alters modern society for the betterin a way that propels Kentucky forward.

Staff

As noted earlier, HPC is about more thanjust hardware. The current fad in HPC, computer clusters, lookcheap in part because they do not count the cost of the personnelto set them up and keep them running. Large supercomputers likethe HP Superdome's are cheap in comparison. Peer review panelslike to fund clusters more than bigger general purpose HPCs becausesmall clusters are easier to justify as a special purpose itemappropriate to one grant application. In fact, the HPC packagecomprises personnel, software, hardware, networking, a physicalenvironment and infrastructure, and endorsement and support fromthe University, state, and nation.

An effective effort in HPC must providecompetitive salaries for highly qualified staff. An expansionof support staff to include professionals with specific experiencein use of the various software and visualization tools is necessaryfor HPC outreach. At its present level, the HPC effort is equippedlike a new library without enough personnel to unlock the doorevery day of the week. These new support staff will be able towork on a one-on-one basis with faculty and students, and alsoprovide more general instruction (e.g., seminars and courses)as might be appropriate. The HPC staff will identify areas thatclearly need support, and work aggressively with those users tofind solutions. The HPC staff must be able to work with usersthe way librarians are able to assist their clients.

HPC will continue to drive changes in modernsociety for the better in the near future. As high performancein computing becomes ever higher, it will bring the rest of computingand technology along by pulling them in its wake. Kentucky willreap these benefits from continued investment in HPC.