Coffee Data Science
Rethinking Refractometers: VST, Atago, and DiFluid; Part 1
Joe, Jeremy, and Robert
Refractometers have been growing in use over the past few years by people serious about consistently brewing tasty coffee as these devices provide an invaluable and relatively easy objective metric in this largely subjective space. The downside, however, is that to achieve high accuracy and precision, which is particularly important for lower concentration brew methods (e.g., filter), the devices generally cost a lot of money.
VST/MISCO and Atago, with their tangled past, have been the predominant digital refractometer manufacturers in the coffee space. Several devices have attempted to break into the scene, claiming comparable performance at a significantly lower price point — most recently (and aggressively marketed) is the undeniably cute refractometer by DiFluid. As many in the coffee community begin to put this device through its paces (including me), there has been some contention regarding the accuracy and precision of the DiFluid refractometer compared to the more established devices by VST and Atago.
Enter: Socratic Coffee. Socratic Coffee has collected — and continues to collect — a massive amount of data on coffee assessment and preparation devices, to include the DiFluid (abbreviated here as DFT). Working with them, here we present some of their data to see if it can help us get a better picture regarding the performance of the DFT device with respect to assessing the strength of coffee.
As with many discussions in the specialty coffee community, the biggest challenge to arriving at empirically-founded conclusions is access to high quality data. Obtaining the equipment and necessary supplies presents a significant hurdle, as does the amount of effort to setup and execute a well-designed testing protocol. Further, unfamiliarity with experimental methods and data analysis techniques can lead to inappropriate conclusions being drawn.
For example, when it comes to comparison testing between coffee-correlated refractometers, people who have been collecting data typically only have 1 DFT and 1 Atago or VST. In some of my previous studies with refractometers, I suggest we may not fully appreciate the complexities of coffee soluble assessment using refractometry, and this new data leads me to another issue: manufacturing variances.
When doing anything in mass production, one must have quality control to make sure devices are within some specification. This is challenging to do well, and even when done well, there can be variance between devices from the same manufacturer. It is quite possible, for instance, that I may have a DFT device showing good performance relative to my Atago, whereas someone else has a DFT unit that shows bad performance compared to a different Atago or VST.
Who is right? Assuming both sets of data are collected with comparable rigor and methodological control, both can demonstrate their point. It is important to point out that both Atago and VST provide certificates of calibration (Atago to ISO standards; VST to NIST standards). DFT does not appear to offer this.
A brief summary of the data presented here suggests two things:
- Having a variety of devices from the same manufacturer to test for inter-device variability is important.
- We need to better understand and appreciate the limits of using light diffraction as a measure of coffee soluble content.
Because there is a lot of data here to digest, it is broken into a few parts for ease of presentation.
The Data
The data was collected in three batches across 16 DFT devices, 1 VST, 1 Atago Coffee, and 1 Atago RX-5000i. Each batch used a different set of DFT devices (5, 6, and 5 respectively). Additionally, some samples were analyzed with a moisture balance, providing a groundtruth measurement. Several solutions were used, each providing different insight:
- Sucrose solution (the basis for Brix measurement; well-established normative data; a “clean” assessment of hardware)
- Instant Coffee at Espresso Strength (high coffee soluble concentration with minimal interference from non-solubles; increased difficulty from sucrose, requiring software conversion of refractive index reading to coffee solubles)
- Instant Coffee at Filter Strength (low coffee soluble concentration with minimal interference from non-solubles; reduced signal strength compared to instant coffee espresso but relatively low noise compared to real-world solutions as instant coffee is almost entirely coffee solubles — 99.9%)
- Espresso (real-world application at high coffee soluble concentration; a difficult testing solution with increased noise but strong signal)
- Filter Coffee (real-world application at low coffee soluble concentration; the most difficult testing solution with decreased signal and increased noise, testing robustness of both hardware and software)
It should be noted that not all test sets used all of these solutions. In testing, some samples were filtered with a syringe filter and some were not, but it is explicitly stated in the charts. Devices were calibrated between each set of data collected.
In this article, we will look at a sucrose solution, instant coffee for filter strength, and filter coffee.
Analysis
With sucrose, three solutions were made of the same strength, and three samples were taken from each solution. We see that the VST and Atago are very close to groundtruth and have very little variance across samples. The DFT devices consistently show more variance between samples.
We can look at the difference to the mean for each device across all of their samples. Most of the DFT devices have a similar variance to each other, and this is higher than VST or Atago. Sucrose is a great solution to utilize in refractometer testing, as it is comprised of a single soluble (sugar) and allows us to examine the hardware’s ability to provide a refraction reading in the most simple of solutions.
Moving to filter coffee, we have instant coffee at filter strength. In this test, VST and Atago were closer to the groundtruth, but some of the DFT devices had a variance across samples closer to VST’s. DFT II had different readings than the others.
The story changes a little as we go from instant to brewed filter coffee. Interestingly, with respect to groundtruth, one DFT device (DFT II) seems to track the closest. VST and Atago perform similarly to each other, which suggests significant similarity in the refractive index-to-coffee soluble correlation algorithm within the two devices.
Looking at these devices, it could be possible for a person to obtain a device like the DFT II and determine it was significantly off compared strictly to a VST or Atago device. Or one could have obtain a device like the DFT III and determine that it is as accurate as VST or Atago for the test.
The DFT devices had a variability compared to each other, and some had a variance similar to VST or Atago.
Finally for the three tests, we can look more globally at percent error rates. This is the difference in reading divided by the reading, and in this way we can compare all three. Sucrose has the least error while instant coffee and filter coffee has a lot of variation even for Atago and VST.
We can look at standard deviation as a metric of variance, Atago and VST have lower STD except in the case of actual filter coffee. These results are confusing because they are not clear-cut.
Discussion
We intentionally didn’t provide any inferential statistical analysis of the data here as only one VST and one Atago device were used in the testing. Socratic Coffee has previously reported on their testing of multiple VST and Atago devices. One of the biggest take-aways from these tests is that readings can vary within a single sample, even for the “gold standard” devices VST and Atago.
For many though, these error rates may not pose a significant issue because the refractometer is used as an objective metric in combination with other metrics to achieve consistency and desired taste profiles. It is quite possible that the average barista doesn’t need VST- or Atago-level performance, and given the size and price of the DiFluid, it could serve as an attractive option.
However, this is the tip of the data iceberg. Let’s continue looking at more data and dive deeper.
If you like, follow me on Twitter, YouTube, and Instagram where I post videos of espresso shots on different machines and espresso related stuff. You can also find me on LinkedIn. You can also follow me on Medium and Subscribe.
