Starting with Scala ZIO 2 — Building a Data Layer (Part 2)
This is the second part of a series of posts about Getting Started Scala ZIO 2 and how to use it for real by creating a simple web API for solving math challenges following a TDD approach.
You can find the first part here:
About this part
In this post, we are going to:
- Implement a POST endpoint that allows users to answer a challenge
- Create a repository that will store the data in-memory
- Add tests to all the previous functionality
- Setting up CORS in the controller
You can find all the files related to this version here: https://github.com/HyperCodeLab/zio-microservices/tree/main/version-2
Also, to make this a little more interesting, I have created a very ugly UI that will interact with the web app that we are creating in these series of posts. You can find the HTML file in here: https://github.com/HyperCodeLab/zio-microservices/blob/main/version-2/frontend/index.html. Opening it with the browser will be more than enough, as no CSS or fancy JavaScript has been used, just plain old jQuery.
The web will look for the ZIO server at localhost:8080, and it will use the GET and POST endpoint that we will develop during this post.
Adding the new models
For the functionality that we are going to implement, we need 2 new entities in our application:
An entity for User. Later we could add attributes like level, so it will send more complex challenges depending on that attribute:
import zio.json._
case class User(alias: String)
object User:
given JsonEncoder[User] = DeriveJsonEncoder.gen[User]
given JsonDecoder[User] = DeriveJsonDecoder.gen[User]A ChallengeAttempt. It’s what our browser will send when the user tries to solve a challenge. It will hold information of what user is answering, the challenge itself and the attempt that he’s sending, as we want to do server side validation. The check attribute will be Optional and mutable, as we will fill it during the validation:
import ziomicroservices.challenge.model._
import zio.json._
case class ChallengeAttempt(
user: User,
challenge: Challenge,
resultAttempt: Int,
var check: Option[Boolean] = None
)
object ChallengeAttempt:
given JsonEncoder[ChallengeAttempt] = DeriveJsonEncoder.gen[ChallengeAttempt]
given JsonDecoder[ChallengeAttempt] = DeriveJsonDecoder.gen[ChallengeAttempt]Adding new functionality
The Challenge Attempt checker
In the previous post, we implemented a GET endpoint to generate a challenge. Let’s now implement a POST endpoint, so we can send the resolution attempt to the server, that way it will check the answer and return true or false.
Let’s do it following a TDD approach by implementing a falling test first.
- In this test, we’re checking that given a
Challenge, it will return true or false depending on the input answer.
test("ChallengeService should check a ChallengeAttempt and yield true") {
val challenge = ChallengeAttempt(User("TestUser"), Challenge(2, 3), 6)
for {
check <- ChallengeServiceImpl(null).checkAttempt(challenge)
} yield assert(check)(equalTo(true))
}If you try to run it, you will face the following error, as we haven’t implemented the checkAttempt method:
With this failing test, it’s time to do the actual implementation:
- In the
ChallengeServicefile, we add a new method signature in both, the trait and the object. The method will receive an instance of aChallengeAttempt, and validate the answer.
package ziomicroservices.challenge.service
import zio._
import ziomicroservices.challenge.model._
trait ChallengeService:
def checkAttempt(attempt: ChallengeAttempt): Task[Boolean]
def createRandomMultiplication(): UIO[Challenge]
object ChallengeService:
def checkAttempt(attempt: ChallengeAttempt): ZIO[ChallengeService, Throwable, Boolean] = ZIO.serviceWithZIO[ChallengeService](_.checkAttempt(attempt))
def createRandomMultiplication(): ZIO[ChallengeService, Nothing, Challenge] = ZIO.serviceWithZIO[ChallengeService](_.createRandomMultiplication())- Then, add the actual implementation in the
ChallengeServiceImplclass. We only need to check that the valueA * valueB is the same as theresultAttempt.
case class ChallengeServiceImpl(randomGeneratorService: RandomGeneratorService) extends ChallengeService:
def checkAttempt(attempt: ChallengeAttempt): Task[Boolean] = {
ZIO.succeed(attempt.challenge.valueA * attempt.challenge.valueB == attempt.resultAttempt)
}
...Running the tests again, we should see them pass:
Congratulations! You have successfully applied Test Driven Development!
Implementing a POST endpoint
Let’s follow the same approach with the controller for implementing the POST endpoint that will receive the requests with the JSON body with the Challenge Attempt.
We are going to write two tests here:
- In the first test, we are going to check that the result of the attempt is true by sending a right answer.
- In the second one, we are sending a wrong response to get back
false.
test("Controller should return True when validating challenge attempt") {
val app = ChallengeController()
val req = Request.post(Body.fromString("""{"user":{"alias":"TestUser"},"challenge":{"valueA":2,"valueB":2},"resultAttempt":4}"""),
URL(Root / "challenges" / "attempt"))
assertZIO(app.runZIO(req).map(x => x.body))(equalTo(Response.json("true").body))
},
test("Controller should return False when validating challenge attempt") {
val app = ChallengeController()
val req = Request.post(Body.fromString("""{"user":{"alias":"TestUser"},"challenge":{"valueA":2,"valueB":2},"resultAttempt":5}"""),
URL(Root / "challenges" / "attempt"))
assertZIO(app.runZIO(req).map(x => x.body))(equalTo(Response.json("false").body))
},As before, if you run the tests now, you will see that them will fail, as that route doesn’t exist. Let’s fix that and implement the functionality.
In the ChallengeController we need to add a case for routing that POST request:
- First, we try to parse the body of the request.
- If the JSON we are sending is invalid, we return a BadRequest status.
- If it’s a valid body,
uwill be a validChallengeAttemptthat we can pass to thecheckAttempt()method of theChallengeService.
case req@(Method.POST -> Root / "challenges" / "attempt") => (
for {
u <- req.body.asString.map(_.fromJson[ChallengeAttempt]) // Try to parse the request
r <- u match
case Left(e) =>
ZIO
.debug(s"Failed to parse the input: $e")
.as(Response.text(e).withStatus(Status.BadRequest))
case Right(u) =>
ChallengeService.checkAttempt(u).map(out => Response.json(out.toJson))
} yield r).orDieRunning the tests now should return something like this.
Looking good, isn’t it? Let’s try the application now using postman:
- Getting a challenge — GET request.
- Answering the challenge — POST request.
Nice, we have the basic functionality of the application working!!
CORS
Before going into the next big topic, let’s quickly set up the CORS, so our UI can communicate with the backend. That is basically allowing our UI to make requests to the server.
There are a few ways of doing it, but the simplest one is to add to the apply() method in the ChallengeController the CORS configuration. In this case, and to simplify things, we are allowing All Origins to send GET and POST requests to the server. This is something that you don’t want to do in a production environment, but as we are testing here ;)…
object ChallengeController:
def apply(): Http[ChallengeService, Throwable, Request, Response] =
Http.collectZIO[Request] {
...
} @@ cors(CorsConfig(
allowedOrigin = {
case origin@Origin.Value(_, host, _) => Some(AccessControlAllowOrigin.All)
case _ => Some(AccessControlAllowOrigin.All)
},
allowedMethods = AccessControlAllowMethods(Method.GET, Method.POST),
) )With this in place, the UI should be able to send requests to the ZIO server, so give it a try!
Let’s take a look now at another important part of our application, the persistence layer, so we can store the different attempts performed by the users and show a leader board or statistics later on.
Persistence Layer
In this part we’re going to implement an in-memory data layer, so in the next one we can just swap it for a data layer that uses a real database under the hood.
If we were to implement a getAttemptID in the endpoint GET /challenges/{id} using TDD, the best approach would be a top down, that means:
- Write the endpoint in the route service. This will call the
ChallengeService.getAttemptByID(id) - Code the
ChallengeService.getAttemptByID(id)method, that will use aChallengeAttemptRepositoryto get the required data. - Write the
ChallengeAttemptRepositoryfunctionality.
So you can see that by doing TDD, you’re being driven to what you need to code next to be able to achieve the required functionality.
Let’s start coding. We want to get an existing ChallengeAttempt when doing GET /challenges/results/{id}, so following what we said before:
- We start by writing a failing test. When hitting the /challenges/result/1 endpoint, it will return the
ChallengeAttemptwith ID = 1.
test("Get result of a previous attempt") {
val app = ChallengeController()
val req = Request.get(URL(Root / "challenges" / "result" / "1"))
assertZIO(app.runZIO(req).map(x => x.body))(equalTo(Response.json(ChallengeAttempt(User("TestUser"), Challenge(2, 2), 4).toJson).body))
},This test will fail, as we don’t have any route in that endpoint, so let’s add it.
case Method.GET -> Root / "challenges" /"results" / id =>
ChallengeService.getAttemptById(id).map(out => Response.json(out.toJson)).orDieThis code won’t compile now, as the ChallengeService doesn’t have the getAttemptByID method.
That makes us need to modify the Challenge service by adding that new method definition:
package ziomicroservices.challenge.service
import zio._
import ziomicroservices.challenge.model._
trait ChallengeService:
...
def getAttemptById(id: String): Task[ChallengeAttempt]
object ChallengeService:
...
def getAttemptById(id: String): ZIO[ChallengeService, Throwable, ChallengeAttempt] = ZIO.serviceWithZIO[ChallengeService](_.getAttemptById(id))But now the question is… How do we implement this method? Well… We need to introduce a new concept here; The Data Layer or Persistence Layer or Data Repository (lots of different names, but same thing), a specific service in charge of dealing with all the logic of storing and retrieving data.
Let’s start by modifying the implementation of the ChallengeServiceImpl to include this new layer:
- We have added a new argument to our case class, a
ChallengeAttemptRepositorythat will have all the functionality needed to access the persisted data. We also need to provide it in the layers that we’re exposing to our ZIO Environment.
Note: see how we have used the &operator to include multiple dependencies in the environment.
case class ChallengeServiceImpl(randomGeneratorService: RandomGeneratorService, caRepo: ChallengeAttemptRepository) extends ChallengeService:
...
def getAttemptById(id: String): Task[ChallengeAttempt] = {
caRepo.find(id)
.map {
case Some(attempt) => attempt
case _ => ???
}
}
object ChallengeServiceImpl {
def layer: ZLayer[RandomGeneratorService & ChallengeAttemptRepository, Nothing, ChallengeServiceImpl] = ZLayer {
for {
generator <- ZIO.service[RandomGeneratorService]
repo <- ZIO.service[ChallengeAttemptRepository]
} yield ChallengeServiceImpl(generator, repo)
}
}Our code won’t compile again, as we don’t have an implementation for the ChallengeAttempeRepository. At this point, we can go on two different routes:
- Create an in-memory persistence layer that we can use for testing purposes, and then swap the in-memory implementation for a real persistence solution layer on.
- To follow a more purist testing approach, as the repository is a service used by the
ChallengeService, we should mock it, as that isn’t part of what we are testing.
Let’s go with the first approach, as I think it will be more didactic… and we can introduce the concept of mocking later on.
Implementing the Data Layer
We have introduced a new class, the ChallengeAttemptRepository … let’s pause our implementation of the ChallengeService to take a look at it.
To store all the code related to the persistence layer, we’re going to create a package called “repository”. Inside, let’s create the interface that our repository will need to follow, with some basic method for saving and retrieving information. This pattern should be familiar to you by now. We are specifying two main methods:
- save: given a
ChallengeAttempt, it will persist it in the repository and return its ID - find: given an ID, it will retrieve the
ChallengeAttemptfor that ID. Note that we’re returning anOption[A]to include the case in which theChallengeAttemptdoesn’t exist.
package ziomicroservices.challenge.repository
import zio._
import ziomicroservices.challenge.model._
trait ChallengeAttemptRepository:
def save(at: ChallengeAttempt): Task[String]
def find(id: String): Task[Option[ChallengeAttempt]]
object ChallengeAttemptRepository:
def save(at: ChallengeAttempt): ZIO[ChallengeAttemptRepository, Throwable, String] = ZIO.serviceWithZIO[ChallengeAttemptRepository](_.save(at))
def find(id: String): ZIO[ChallengeAttemptRepository, Throwable, Option[ChallengeAttempt]] = ZIO.serviceWithZIO[ChallengeAttemptRepository](_.find(id))Following this interface, our implementation will follow the next structure:
In the layer, we’re creating the Ref as an empty Map[String, ChallengeAttempt] that will hold the state of our application. Initially, it will be empty.
package ziomicroservices.challenge.repository
import zio._
import ziomicroservices.challenge.model._
case class InMemoryChallengeAttemptRepository(map: Ref[Map[String, ChallengeAttempt]]) extends ChallengeAttemptRepository:
def save(att: ChallengeAttempt): UIO[String] = ???
def find(id: String): UIO[Option[ChallengeAttempt]] = ???
object InMemoryChallengeAttemptRepository {
def layer: ZLayer[Any, Nothing, InMemoryChallengeAttemptRepository] =
ZLayer.fromZIO(
Ref.make(Map.empty[String, ChallengeAttempt]).map(new InMemoryChallengeAttemptRepository(_))
)
}- The only important addition we have done here is adding a Ref.
Note: A Ref[A] is a data structure that allow us to share state between different threads (fibers) of the application in a purely functional way.
We have the skeleton of the class in place, so let’s drive the implementation using TDD 😁.
Saving entities
- In this test, we are going to check that the repository saves the entity and returns its ID. For that, we generate a challenge, set a seed (as we’re generating the ID’s randomly) and then we try to save the entity and assert that the given ID is what we’re expecting. You can see that we’re also providing a layer with the In Memory repository.
package ziomicroservices.challenge.repository
import zio._
import zio.test._
import zio.test.Assertion.equalTo
import ziomicroservices.challenge.model._
object InMemoryChallengeAttemptRepositoryTest extends ZIOSpecDefault {
def spec = {
suite("InMemory Challenge Attempt Repository")(
test("Repository should save the entity") {
val entity = ChallengeAttempt(User("TestUser"), Challenge(2, 2), 4)
TestRandom.setSeed(42L)
for {
repository <- ZIO.service[InMemoryChallengeAttemptRepository]
id <- repository.save(entity)
} yield assert(id)(equalTo("b2c8ccb8-191a-4233-9b34-3e3111a4adaf"))
}
)
}.provideLayer(
InMemoryChallengeAttemptRepository.layer
)
}If you try to run the test, you will see that it fails, as we don’t have an implementation for the savemethod.
- InMemory Challenge Attempt Repository - Repository should save the entity
Exception in thread "zio-fiber-277" scala.NotImplementedError: an implementation is missingAs we’re using a Map as our In Memory data structure, the implementation is trivial:
- A
Map[String, ChallengeAttempt]is the simplest data structure that we could use for this task. We generate an ID as the key, and we add the key-value pair to the map.
case class InMemoryChallengeAttemptRepository(map: Ref[Map[String, ChallengeAttempt]]) extends ChallengeAttemptRepository:
def save(att: ChallengeAttempt): UIO[String] =
for {
id <- Random.nextUUID.map(_.toString)
_ <- map.update(_ + (id -> att))
} yield idFinding entities
Let’s now implement the find method.
- The test in this case will use the existing save method to persist an entity, and then use the find method to retrieve it and check if it’s the same as the first one.
test("Repository should find an existing the entity") {
val entity = ChallengeAttempt(User("TestUser"), Challenge(2, 2), 4)
for {
repository <- ZIO.service[InMemoryChallengeAttemptRepository]
id <- repository.save(entity)
expectedEntity <- repository.find(id)
} yield assert(expectedEntity)(equalTo(Some(entity)))
}Now, the implementation of this method is as easy as retrieving the value from the map.
def find(id: String): UIO[Option[ChallengeAttempt]] = map.get.map(_.get(id))We should be able to run both tests now and see them pass.
+ InMemory Challenge Attempt Repository
+ Repository should save the entity
+ Repository should find an existing the entityWith the In Memory repository in place, we can go to the ChallengeServiceImpl tests, and add a scenario for the getAttempt method:
- We save a
ChallengeAttempt, and we check that theChallengeServiceImplworks by trying to retrieve it.
test("ChallengeService should return an existing Attempt back") {
val entity = ChallengeAttempt(User("TestUser"), Challenge(2, 2), 4)
for {
repo <- ZIO.service[ChallengeAttemptRepository]
id <- repo.save(entity)
expectedEntity <- ChallengeServiceImpl(null, repo).getAttemptById(id)
} yield assert(entity)(equalTo(expectedEntity))
}Notice that we also need to modify the provide adding the new layer.
...}.provide(
RandomGeneratorServiceImpl.layer,
InMemoryChallengeAttemptRepository.layer
)Implementing more complex queries
Now that we have the basic functionality of the repository, let’s implement something more challenging. Getting the ChallengeAttempt performed by some user from the endpoint /challenges/users/{userAlias}.
- Using the test as an explanation of the requirement: when we hit the endpoint
GET /challenges/users/TestUser, we want to get back an Array with all the challenges that the user has attempted.
test("Get attempts of users") {
val app = ChallengeController()
val entity = ChallengeAttempt(User("TestUser"), Challenge(2, 2), 4)
for {
repo <- ZIO.service[ChallengeAttemptRepository]
_ <- repo.save(entity)
response <- app.runZIO(Request.get(URL(Root / "challenges" / "users" / "TestUser")) ).map(x => x.body)
} yield assert(response)(equalTo(Response.json(List(entity).toJson).body))
}As before, going from top to bottom modifying the required files:
- The controller. It should be as easy as adding the new route:
case Method.GET -> Root / "challenges" /"users" / userAlias =>
ChallengeService.getAttemptsByUser(userAlias).map(out => Response.json(out.toJson)).orDie- The ChallengeService. In here, we need to add a new method to the interface that will return a
List[ChallengeAttempt]:
trait ChallengeService:
...
def getAttemptsByUser(userAlias: String): Task[List[ChallengeAttempt]]
object ChallengeService:
...
def getAttemptsByUser(userAlias: String): ZIO[ChallengeService, Throwable, List[ChallengeAttempt]] = ZIO.serviceWithZIO[ChallengeService](_.getAttemptByUser(userAlias))And the implementation will be just passing that call to the ChallengeAttemptRepository:
def getAttemptsByUser(userAlias: String): Task[List[ChallengeAttempt]] = {
caRepo.findAttemptsByUser(userAlias)
}- The
ChallengeAttemptRepository.As always, we add the method to the interface definition.
trait ChallengeAttemptRepository:
...
def findAttemptsByUser(userAlias: String): Task[List[ChallengeAttempt]]
object ChallengeAttemptRepository:
...
def findAttemptsByUser(userAlias: String): ZIO[ChallengeAttemptRepository, Throwable, List[ChallengeAttempt]] = ZIO.serviceWithZIO[ChallengeAttemptRepository](_.findAttemptsByUser(userAlias))And for the implementation, here you can notice the importance of using the right data structure to hold the data. In this case, we could have opted for storing the attempts by User rather than by ID. That would have done this query straight forward, but it would have complicated the previous one. So It’s very important to use the right data structure for the use case that you want to solve.
In this case, the easiest solution for finding the ChallengeAttempt of a User would be to iterate through all values of the map and filter those that are related to it.
case class InMemoryChallengeAttemptRepository(map: Ref[Map[String, ChallengeAttempt]]) extends ChallengeAttemptRepository:
...
def findAttemptsByUser(userAlias: String): Task[List[ChallengeAttempt]] =
map.get.map( x => x.values.filter(x => x.user.alias == userAlias).toList )We can add one more test to the repository to check what we have just implemented:
test("Repository should find all attempts for a user") {
val entity1 = ChallengeAttempt(User("TestUser"), Challenge(2, 2), 4)
val entity2 = ChallengeAttempt(User("TestUser"), Challenge(2, 3), 6)
val entity3 = ChallengeAttempt(User("TestUser2"), Challenge(2, 3), 6)
for {
repository <- ZIO.service[InMemoryChallengeAttemptRepository]
id1 <- repository.save(entity1)
id2 <- repository.save(entity2)
id3 <- repository.save(entity3)
expectedEntity <- repository.findAttemptsByUser("TestUser")
} yield assert(expectedEntity)(equalTo((List(entity1, entity2))))
},Wrap-up
Ok, so by now we have a basic working application. You can test all we have done here in the simple UI that I have included in the repo. It should allow you to check that all the functionality that we have covered is working.
In the next post, we’ll take a look at changing the In Memory repository for one that uses a real database to store the data.
Thanks for reading, and I’ll see you in the next post!
