I continue to be amazed by what can be done with Scala. Once you start to understand the simplicity and beauty of coding in Scala it is really easy and fun. As part of my learning exercise I am implementing a multi threaded downloader. One part of the solution is a splitter. It takes a file size and splits it up into a number of chunks that can be downloaded in parallel. I have implemented a Chunk, that holds information about a chunk, as a case class and created a trait ChunkDownload that can be bolted on to this case class at runtime. This is so called composable behavior which adds functionality to the case class.

Doing this exercise I learned a number of things:

• How to use functions to implement different splitting strategies. In Java I would have done this by having different classes that each provide a strategy. In Scala it is so much easier just passing a splitting strategy function, that calculates the size of the chunks, to a method.
• It is possible to set a default function on the method, just like you would do with any other parameter. Very cool.
• Another lesson I learned was how to add a trait to a List with initialized case classes. This means I can keep my data types as clean case classes that can be passed as immutable messages in Akka. Nice.

The Chunk case class:

case class Chunk(id: Int, url: URL, destFile: File, start : Long, length: Int, append: Boolean = true, state: State = State.NONE)

The ChunkDownload trait:

trait ChunkDownload {
  import State._

  val destFile: File
  val id: Int

  def workDir: File = destFile.getParentFile

  def formattedId: String = f"$id%06d"

  def validateState(f: File, length: Long): State = {
    if (f.exists) {
      if (f.length == length) State.DOWNLOADED
      else State.DOWNLOADING
    } else State.PENDING
  }  
}

The Splitter class and companion object:

object Splitter {
  def defaultStrategy(fileSize: Long): Int = 1024 * 1024 * 5 // 5 MB

  def ratioStrategy(fileSize: Long): Int = ((fileSize - (fileSize % 10)) / 10).toInt

  def ratioMaxSizeStrategy(fileSize: Long): Int = {
    val chunkSize = ((fileSize - (fileSize % 10)) / 10).toInt
    val maxSize = 1024 * 1024 * 20 // 20 MB
    if (chunkSize < maxSize) chunkSize else maxSize 
  }
}

class Splitter {
  import Splitter._

  def split(r: RemoteFileInfo, append: Boolean, workDir: File, strategy: (Long) => Int = defaultStrategy): LinkedHashSet[Chunk] = {
    val chunks = LinkedHashSet[Chunk]()
    val chunkSize = strategy(r.fileSize)
    val numOfChunks = if (r.fileSize % chunkSize > 0) (r.fileSize / chunkSize).toInt + 1 else (r.fileSize / chunkSize).toInt

    def addChunk(i: Int, startChunk: Long, endChunk: Long, length: Int) = {
      val destFile = new File(workDir, f"$i%06d.chunk")
      chunks += new Chunk(i, r.url, destFile, startChunk, length, append)
    } 

    @tailrec def createChunks(i: Int, startChunk: Long): LinkedHashSet[Chunk] = i match {
      case _ if i == numOfChunks => {
        val startChunk = (i - 1) * chunkSize
        addChunk(i, startChunk, r.fileSize, (r.fileSize - startChunk).toInt)
      }
      case _ if i == numOfChunks => {
        val startChunk = (i - 1) * chunkSize
        addChunk(i, startChunk, chunkSize + startChunk - 1, chunkSize.toInt)
        createChunks(i + 1, startChunk)
      }
    }
    createChunks(1, 0)
  }
}

This is the test code to create a List of chunks and add the ChunkDownload to the chunks in the List:

val info = new RemoteFileInfo(url, "application/x-compressed", true, 1024 * 1024 * 25, new Date())
val splitter = new Splitter()
val chunks = splitter.split(info, true, workDir)
val test = for (c <- chunks) yield 
  (new Chunk(c.id, c.url, c.destFile, c.start, c.length) with ChunkDownload {
    override val state = State.CANCELLED
  })
for(t <- test) println(s" - ${t.formattedId}, ${t.workDir} $t")

Some code has been left out to try and focus on the important parts. You can checkout my Github if you are interested in the rest of the code.