Streaming large data sets

Michał Bogacz
Jun 20, 2017 · 6 min read

Introduction

Solution 1: Client controls the size — Pagination

case class PageParams(pageNumber: Option[Int], 
pageSize: Option[Int]) {
require(pageNumber.forall(size ⇒ size > 0), "page number must be greater then 0")
require(pageSize.forall(size ⇒ size > 0 && size < 1000), "page size must be greater then 0 and less then 1000")

private val DefaultPageNumber = 1
private val DefaultPageSize = 16

val limit: Int = pageSize.getOrElse(DefaultPageSize)
val skip: Int = (pageNumber.getOrElse(DefaultPageNumber) - 1) * limit
}
trait ServiceDirectives extends Directives {

val pageParams: Directive1[PageParams] =
parameters(('pageNumber.as[Int].?,
'pageSize.as[Int].?)).as(PageParams)
}
val route =
path("resources") {
pageParams { pageParams =>
get {
complete(getData(coll, pageParams)
.map(HttpEntity(ContentTypes.`application/json`, _)))
}
}
}
def getData(coll: MongoCollection[Document], pageParams:  
PageParams): Future[String] =
Source.fromPublisher(coll.find().skip(pageParams.skip)
.limit(pageParams.limit))
.map(_.toJson)
.intersperse("[", ",", "]")
.runFold("")((acc, e) ⇒ acc + e)
[
{
"_id": {
"$oid": "593fc5f3964652a1ffc9836d"
}
},
{
"_id": {
"$oid": "593fc5f4964652a1ffc9836e"
}
}
]

Different requirements

Solution 2: Chunking

def getData(coll: MongoCollection[Document]): Source[ByteString, NotUsed] =
Source.fromPublisher(coll.find())
.map(_.toJson)
.map(ByteString(_))
.intersperse(ByteString("["), ByteString(","), ByteString("]"))
val route =
path("resources") {
get {
complete(HttpEntity(ContentTypes.`application/json`,
getData(coll)))
}
}

Summary

Michał Bogacz

Written by

Software Developer at SAP Hybris. Scala and Akka enthusiast.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade