Thoughts from a Data Consultant — marcin.tustin@gmail.com

Nov 10, 2019

Fun with text generation pt. 1: markov models in awk

Inspired by Really fast Markov chains in ~20 lines of sh, grep, cut and awk this series is intended to explore text generation, first with markov models, then with generative adversarial networks.

Marcin Tustin in Build and Learn

Sep 22, 2019

15 Easy Questions for Easy Change Management

If your systems keep breaking when changes are made, answer these questions

These 15 questions should be very easy for engineers to answer ahead of making a change; and if…

Marcin Tustin in Build and Learn

Aug 17, 2019

Spark: Aggregating your data the fast way

This article is about when you want to aggregate some data by a key within the data, like a sql group by + aggregate function, but you want the whole row of data. It’s easy to do it the right way, but Spark provides lots of wrong ways. I’m going to go over…

1 response

Marcin Tustin in Build and Learn

Aug 19, 2018

Streaming is a myth — Reactive vs Rigid is the real distinction

Tl;dr — Streaming architectures are a special case of reactive architectures (and not actually a myth). The distinction between reactive and rigid distributed architectures is much more important than streaming vs batch…

Marcin Tustin in Build and Learn

Jan 1, 2018

Is Shapeless worth it? What is Shapeless anyway?

Getting started with generic programming in Scala, and best practices for generic programming

Marcin Tustin in Build and Learn

Dec 16, 2017

Perfect table migrations in redshift and hive

Always do your migrations like this

Here’s a neat little trick due to David Heller at Alooma — perform any migration in redshift (or other columnar store) without any need for downtime, table locking…

Marcin Tustin in Build and Learn

I’m not sure this is right. Random access busts caches (slowing reads), and more generally linear access is more capable of being optimized by preloading. This effect is even greater when data are spread across different network nodes (because network IO is relatively slow and relatively easy to saturate).

1 response

Marcin Tustin in Build and Learn

May 18, 2017

Interview research link dump

I’ve used semi-structured interviews for both academic and user research (and you can have a semi-structured questionnaire too). I’ve been reading more into the subject recently, so here are some links of interest. Lawyers will likely find a lot of this familiar.