GridBot

GridBot is a generic system for running multiple Bags of Tasks (BOT) on a mixture (mashup) of multiple opportunistic environments, clusters, and community grids. The paper describing GridBot has been accepted for presentation at the Supercomputing conference SC09 in Portland, Oregon.

GridBot is based on BOINC - Berkeley Open Infrastructure for Network Computing. We modified BOINC main scheduler and added

Classads based matchmaking
Classads based job replication to allow faster BOT turnaround
Bag Of Task as first-class citizen, including the detection of Tail

GridBot comprises two layers: allocation layer - establishes the overlay of modified BOINC clients over different grids depending on the resource demand. The second - scheduling layer - schedules the BOT jobs over the clients of the overlay depending on the classad-based dynamic policy, which depends on the system state, host parameters, BOT properties and others.

The system is operational and employs computers from the Technion, UW Madison, Open Science Grid, EGEE and Superlink@Technion community grid. It spans over 25,000 computers with about 14,000 coming from the community grids, with the peak throughput roughly equivalent to 8,000 dedicated CPUs.

The live monitoring of the system performance can be seen here.