Skip to main content

Federation with Google Cloud

Federation with Google Cloud

OARC has formed a partnership with Google Cloud and SchedMD, the main developer of an open-source job scheduler called SLURM, to develop a federated hybrid storage environment, where some Amarel user data is stored locally and some is stored in the cloud.

The benefits of this arrangement will be many:

  • Allowing many storage resources (including cloud services) to work together seamlessly
  • Allowing OARC to offer a one-stop shop for users, regardless of whether they need local or cloud storage
  • Allowing users to access Amarel compute nodes and storage (either as an owner or a general-access user) from any campus (or other location) at any time.

The initial proof of concept saw OARC take a number of steps:

  • Setting up a federated environment between Rutgers and Google Cloud
  • Setting the Google Cloud instance up in a similar manner as local resources, including how SLURM schedules jobs
  • Establishing a connection through Internet2 between both environments
  • Enabling users to submit jobs directly from Rutgers HPC to Google Cloud using a SLURM submit script
  • Elasticity/bursting capability within the ERN is now being tested, and, looking forward, OARC will extend capabilities to Jetstream and the Open Science Grid (OSG).