Livongo Tech Blog

Using Scala to Read Really, Really Large Files – Part 3: better-files

better-files is a dependency-free pragmatic thin Scala wrapper around Java NIO README.md Basically: scala.io.Source, done right. It’s simple enough that the user manual fits in the github landing page. Implementation Ergonomics 😀 One of the really nice parts of this library is that the various iterators it creates close themselves when the end of the file is reached or an … Continue reading “Using Scala to Read Really, Really Large Files – Part 3: better-files”

Using Scala to Read Really, Really Large Files – Part 2: The Standard Libraries

The Scala Standard Library offers a zero-dependency way of processing really, really large files that’s surprisingly reasonable. Implementation Ergonomics 😐 Overall, scala.io.Source provides a decent API for simpler processing tasks. It’s a little awkward because Scala doesn’t have syntactic sugar equivalent to Java’s try-with-resource construct, which would be more of an issue if Source.fromFile didn’t abstract away the nested readers. Source#getLines … Continue reading “Using Scala to Read Really, Really Large Files – Part 2: The Standard Libraries”

Using Scala to Read Really, Really Large Files – Part 1: Setup

This post provides background information on how the test data is gathered and implementation details for the curious. To cut down the amount of noise when looking at each implementation and reduce the chance of introducing unneeded variances which might skew the measurements, we’ve extracted out the common elements to narrow the differences as much as possible to the way … Continue reading “Using Scala to Read Really, Really Large Files – Part 1: Setup”

Using Scala to Read Really, Really Large Files – Part 0: Introduction

Every development team has a set of standard practices and approaches to familiar problems. While this is generally a good way to avoid choice paralysis, it can also lead to stagnation, so it’s good to occasionally reevaluate the standard strategies when encountering new techniques or information. One of the familiar problems at Livongo is how to process exceptionally large files, … Continue reading “Using Scala to Read Really, Really Large Files – Part 0: Introduction”