GridScale: a journey from Object-Oriented to Functional Programming

Mathieu Leclaire
Jonathan Passerat-Palmbach
Romain Reuillon

Context

Need a library to submit jobs

How does it look?


    val slurmService = new SLURMJobService 
                       with SSHPrivateKeyAuthentication {
      def host: String = "server.domain"
      def user = "user"
      def password = "password"
      def privateKey = new File("/path/to/key/file")
    }

    val description = new SLURMJobDescription {
      def executable = "/bin/echo"
      def arguments = "success > test_success.txt"
      def workDirectory = "/home/user"
      def walltime = 10 minutes
    }

    val j = slurmService.submit(description)

Design

Cake pattern / Mix-ins


trait SLURMJobService extends JobService
                      with SSHHost 
                      with SSHStorage 
                      with BashShell

Problems


trait SLURMJobService extends JobService
                      with SSHHost 
                      with SSHStorage 
                      with SSHSimpleConnection
                      with SSHCachedConnection
                      with BashShell

  • Which Connection is used?
  • Implementation leaks in type
    • SSH
    • Bash

I just want to submit a job!

f: jd => js => job

def submit[D, S](desc: D, jobService: S)

Functional patterns in GridScale

Type classes


trait SSHAuthentication[T] {
  def authenticate(...): ...
}

case class PrivateKey(...)
case class LoginPassword(...)

implicit def privateKeySSH = new SSHAuthentication[PrivateKey] {
  def authenticate(...) = ...
}

def submit[A](a: A, d: Description)
             (implicit c: SSHAuthentication[T])

Reader Monad

  • Dependency injection => connection
  • Pollutes call stack
  • Side effects appear explicitly all the way down the call stack

Free Monad

  • Focus on API
  • Highly composable DSLs
  • Push side-effects to the boundaries of the program
  • Change behaviour using different interpreters
  • Overall pattern contains still a lot of boilerplate (use Freek at the very least)
github.com/ProjectSeptemberInc/freek

FreeMonad - concrete example

  • You've seen the logger already :)
  • Pseudo-Random Number Generation

A Random FreeMonad...

  
object Random {

  def interpreter(random: util.Random) = new Interpreter[Id] {
    def interpret[_] = {
      case nextDouble() => Right(random.nextDouble)
      case nextInt(n) => Right(random.nextInt(n))
      case shuffle(s) => Right(random.shuffle(s))
    }
  }

  def interpreter(seed: Long): Interpreter[Id] = 
    interpreter(new util.Random(seed))

}

@dsl trait Random[M[_]] {
  def nextDouble: M[Double]
  def nextInt(n: Int): M[Int]
  def shuffle[A](s: Seq[A]): M[Seq[A]]
} 
  
  

FreeDSL

  • Macro generating boilerplate on top of Freek
  • Runtime merge function
  • Highly composable DSL
github.com/ISCPIF/freedsl

How will it look? - WIP

  
def interpreter(client: util.Either[ConnectionError, SSHClient]) = 
  new Interpreter[Id] {

  lazy val sFTPClient = client.map(_.newSFTPClient)

  def interpret[_] = {
    case execute(s) ⇒ for {
        c ← client
        r ← SSHClient.exec(c)(s).toEither.leftMap(t ⇒ ExecutionError(t))
      } yield r
    case fileExists(path) ⇒
      sFTPClient.map(_.exists(path))
    case readFile(path, f) ⇒
      sFTPClient.map { c ⇒
          val is = c.readAheadFileInputStream(path)
          try f(is) finally is.close
      }
  }
}

@dsl trait SSH[M[_]] {
    def execute(s: String): M[ExecutionResult]
    def fileExists(path: String): M[Boolean]
    def readFile[T](path: String, f: java.io.InputStream ⇒ T): M[T]
  }
  
  

Usage (merge with UUID)

    
import freek._
import cats.implicits._
import freedsl.random._
import freedsl.util._

val c = freedsl.dsl.merge(Util, SSH, Random)
import c._

def randomData[M[_]](implicit randomM: Random[M]) = randomM.shuffle(Seq(1, 2, 2, 3, 3, 3))

def job(data: String) =
  SSHJobDescription(
    command = s"echo -n $data",
    workDirectory = "/tmp/")

val prg =
  for {
    sData ← randomData[M]
    jobId ← submit[M](job(sData.mkString(", ")))
    _ ← implicitly[Util[M]].sleep(2 second)
    s ← state[M](jobId)
    out ← stdOut[M](jobId)
  } yield s"""Job status is $s, stdout is "$out"."""
  
  

Summary

  • Object didn't work for us
  • Initial design impacts future enhancements
  • Learnt it the hard way...
  • Functional patterns provide (more?) powerful abstractions

So what's next?

  • Massive refactor :)
  • API will change (road to 2.0)
  • Come learn better design with us!
  • Check the issues list to contribute ;)

Useful Links

GridScale Source code openmole/gridscale
FreeDSL Source code ISCPIF/freedsl
OpenMOLE (parent project) openmole/openmole
Announcements @OpenMOLE

Thanks!

romain.reuillon@iscpif.fr
mathieu.leclaire@iscpif.fr
j.passerat-palmbach@imperial.ac.uk