The Scala Standard Library offers a zero-dependency way of processing really, really large files that’s surprisingly reasonable. Implementation Ergonomics 😐 Overall, scala.io.Source provides a decent API for simpler processing tasks. It’s a little awkward because Scala doesn’t have syntactic sugar equivalent to Java’s try-with-resource construct, which would be more of an issue if Source.fromFile didn’t abstract away the nested readers. Source#getLines … Continue reading “Using Scala to Read Really, Really Large Files – Part 2: The Standard Libraries”
Livongo Tech Blog
Using Scala to Read Really, Really Large Files – Part 1: Setup
This post provides background information on how the test data is gathered and implementation details for the curious. To cut down the amount of noise when looking at each implementation and reduce the chance of introducing unneeded variances which might skew the measurements, we’ve extracted out the common elements to narrow the differences as much as possible to the way … Continue reading “Using Scala to Read Really, Really Large Files – Part 1: Setup”
Using Scala to Read Really, Really Large Files – Part 0: Introduction
Every development team has a set of standard practices and approaches to familiar problems. While this is generally a good way to avoid choice paralysis, it can also lead to stagnation, so it’s good to occasionally reevaluate the standard strategies when encountering new techniques or information. One of the familiar problems at Livongo is how to process exceptionally large files, … Continue reading “Using Scala to Read Really, Really Large Files – Part 0: Introduction”
Using Scala to Read Really, Really Large Files – Part 5: FS2 (with IO helpers)
FS2 takes the stance that the Reactive Streams approach is mutable, unsafe, and generally more complicated than it needs to be. To this end, it aims to provide an expressive, safe, and composable DSL for defining and manipulating streaming I/O. It is included primarily because it’s part of the Cats ecosystem, so it plays nicely with our other libraries. Because this is … Continue reading “Using Scala to Read Really, Really Large Files – Part 5: FS2 (with IO helpers)”
