Advent of Code 2020

 Uh dude, wasn't that in like, December?

Yeah, well, I started it in December and then life happened. I just finished today (ed: 09-OCT).

I don't precisely remember why I started. Maybe someone mentioned it in work Slack and it seemed fun? In any case, when I started I couldn't decide what language to use, so I did each problem twice: once in pure SQL for a challenge, and again in Clojure as an exercise in learning the language.

(Spoilers follow)

Code here: https://github.com/slotrans/advent-of-code-2020

Day 1, "Report Repair": Trivial. There might be a non-brute-force solution but for such tiny input, I couldn't be bothered. Looking back at my solutions, my SQL solution was pretty straightforward (and beautifully declarative), but I was flailing in Clojure.

Day 2,  "Password Philosophy": Also pretty trivial. This was months ago now, but I can remember the SQL solution flowing right out of my fingers. My Clojure was improving but still awkward. Parts of it look downright silly in retrospect.

Day 3, "Toboggan Trajectory": Now we're talkin'. This was fun! The first of many grid-based/spatial puzzles. This one highlighted a couple of interesting things about SQL. First, the fact that the input is ordered requires special handling at load time, because relations are not naturally ordered. Second, the way SQL performs iteration implicitly through consuming input rows (and producing output rows) rather than explicitly through looping or recursion. On the Clojure side, my comfort level was clearly improving, with recursion in particular. I hadn't yet stumbled on the most convenient ways of modeling grids.

Day 4, "Passport Processing": This one is mostly about string processing, specifically handling the irregular input format. Parsing the input in SQL was gnarly. Clojure's string and sequence processing really started to shine here. This is one of the few puzzles that resembles what I'd call "business" code.

Day 5, "Binary Boarding": Pretty straightforward once you get past the obfuscation of the input. Postgres' translate function and bit(N) type helped keep the SQL solution relatively compact. Pleased to find Clojure has a reasonably easy way of parsing binary literals (which I then forgot and had to look up again for a later puzzle).

Day 6, "Custom Customs": Again SQL struggles with wrangling the input. This isn't surprising by the way, since typically the structuring of input into nicely labeled and typed fields is done before data ever hits the SQL layer. It's wild how much shorter the Clojure solution is.

Day 7, "Handy Haversacks":  This is where things start getting a little crazy. The structure of the puzzle is inherently recursive which poses a big challenge for SQL, and I remember having to fiddle with my solution quite a bit and re-read the documentation for with recursive several times. For Clojure, I remember having to think for a while about how to structure the input rules as data.

Day 8,  "Handheld Halting": This one really pushed SQL to the limit, requiring (or at least, I used) both recursive CTEs and lateral joins. The Clojure code is much more straightforward. Also I can see this is the first puzzle for which I included a (def sample-input ...) to test my solution against the provided sample in the same way it would run against the full input.

Day 9, "Encoding Error": The SQL solution to part 1 is wonderfully compact, but part 2 required an odd combination of window functions and a lateral join. The Clojure code for this one is fairly unremarkable, though I do cringe a little at (vec (for ...)), apparently I had not yet discovered mapv.

Day 10,  "Adapter Array": Part 1 is fine, whatever, but part 2... this one pissed me off.

Have you ever heard the riddle "why are manhole covers round?" It's a bad riddle, because if you've heard it you know the answer, and if you haven't heard it you have no way of finding the answer, besides experiencing a miraculous logical leap. Puzzle 10 part 2 is like that, because if you've seen the climbing stairs problem, and notice the correspondence with this "joltage" nonsense, the solution will be obvious and easy. If you haven't, then you'll need either the aforementioned logical leap, or end up doing something dumb like I did (i.e. attacking the problem very literally).

I very nearly ragequit after this one.

Day 11, "Seating System": Another grid puzzle, and the first one involving cellular automata! It's also, a bit sadly, where I gave up on writing SQL solutions, as they were becoming increasingly convoluted. Anyway I found it fun, and it served as a nice base to come back to when tackling future cellular automata puzzles.

Day 12, "Rain Risk": Neat puzzle. The most interesting part was figuring out how to model the instructions and then how to interpret them. This is one of those puzzles where part 2 completely upends what you did for part 1.

Day 13, "Shuttle Search": Not counting Day 10, this is the first one I hit a real performance problem on. More specifically, my naive solution was too slow to ever finish. Again I had to look at others' posted solutions for some inspiration. And again, I don't know that I ever would have figured out the proper technique without being shown. Perhaps for that reason, this is the last one I did in-order in December. At this point I skipped ahead to Day 24 (it was already 26-Dec) to see "the end", and then stopped.

Day 14,  "Docking Data": This is where I resumed, on 31-Aug. It was interesting to see how well Clojure had "stuck" to my brain, after having not used it at all for 8 months. Bit masks are fun.

Day 15, "Rambunctious Recitation": I suspect there's a pattern to this one that can radically simplify things, but I couldn't see it so I forged ahead with brute force and literalism, as usual. Interestingly, part 2 is identical to part 1 except for being pushed to an absurdly large number of iterations. My code wasn't fast enough, so I did peruse a few other solutions for ideas. In the end though I just took an educated guess at where the slowdown was, and turned out to be correct.

Day 16, "Ticket Translation": I enjoyed the deductive reasoning aspect of this puzzle. I definitely felt like the solution-finding routine would have been much more naturally expressed in an imperative language with a "while" loop.

Day 17, "Conway Cubes": This was probably my favorite puzzle. It turned out to be helpful that I'd already done Day 24, which has a lot of similarities. The best part was finding that, other than having to re-write my "get adjacent coordinates" function (which I had been stubbornly unrolling by hand rather than expressing with nested loops), modifying my solution from 3 dimensions to 4 for part 2 required only the most trivial code changes.

Day 18, "Operation Order": This was one of the hardest for me. Part 1 I was able to do, but when part 2 introduced precedence rules I got stuck. I spent a day thinking about it and trying a few approaches, but got nowhere. I found a tutorial on writing a simple calculator-interpreter in Python, but it was brutally stateful. I spent some time trying to adapt it to be functional but again got stuck. I ended up implementing a transliteration of the tutorial's code using atoms for mutable state. It's VERY gross. I would still like to see a proper functional solution to this one.

Day 19, "Monster Messages": This was the hardest. It took me 5 days, and careful study of several other solutions. I'm sure that any CS majors who happen to have written a regex engine had no trouble with this, but I sure did. I was able to solve part 1 without too much trouble, but got very confused when my solution didn't work at all for part 2, despite only minor modifications to the problem. Unlike 10 and 13 though, this one didn't make me angry, because it's a fair problem that can legitimately be figured out with no prior knowledge.

Day 20,  "Jurassic Jigsaw": I enjoyed this one a lot, because it had a surprising amount of depth. The code for this one is the longest by far, almost double the next largest. Like 19, it took me 5 days to complete, but happily I didn't need to consult any other solutions. This is another one where data modeling is key. Parts of this would definitely have been easier in an ordinary mutable-state language. Also I finally used reduce comfortably! Twice!

Day 21,  "Allergen Assessment": Effectively a re-hash of Day 16, with deductive reasoning. I infer that there's an "easier" (less general?) solution to just part 1, but I ended up going straight for the full solution so I needed only a trivial amount of extra code to complete part 2. After the early pre-processing steps, I actually worked out the solution "on paper" and then backed into a code solution. Some of the instructions feel like red herrings... the statement "allergens aren't always marked" is highlighted, but as far as I can tell doesn't actually matter?

Day 22, "Crab Combat": Fun with recursion. Getting part 2 right turned out to be very fiddly. I was happy this didn't turn into another performance problem...

Day 23, "Crab Cups": ...but this one sure did! This is another one where the solution to part 1 should in theory work for part 2 without modification. And I'm sure it would have, if I had a few months to wait for the answer. The direct, sequence-splitting-and-reassembling strategy solves the problem very simply but doesn't turn out to be efficient.

I ended up writing the most solutions for this one. The original slow Clojure solution. A second Clojure solution that leaned on mutating a java.util.LinkedList and was significantly faster, but still too slow. Then a pure Java solution based on that, which was faster yet but still around a full day to run 10M turns. The solution I actually got an answer from -- in ~2.5 hours -- was a pure Java solution that (still quite naively) manipulated an array of integer primitives.

Once I had an honestly-earned answer to my very last puzzle (I had already done 24 and 25), I went looking for performance help. I suspected all along that what I needed was the right data structure, and when I saw it, it was one of those forehead-slapping moments. I had never thought of modeling a linked list as a map of value->next_value. Rewriting the code for that model took some time, but my final Clojure solution ran in 83 seconds.

Day 24, "Lobby Layout": Another favorite. I was thrilled to finally have an excuse to learn how to model a hex grid. Working in a weird coordinate system, and dealing with cellular automata rules in a sparse space, made for a super engaging and rewarding problem. As I mentioned earlier, I skipped ahead to this one after 13, which turned out quite nicely since it was a jump from basically the second-worst to the second-best puzzle.

Day 25, "Combo Breaker": A nice little gimme at the end.

A few general observations...

Since the puzzles don't build on each other, the solution to each one can be treated as throwaway code. Most of the usual best practices can be thrown out the window. When I did look at others' solutions, I observed quite a few single-letter variable names and other signs of code optimized for speed-of-writing rather than clarity. You can certainly do that if you want, but I doubt that typing out readable names is what's going to stop you from solving each puzzle in one day.

Many puzzles are like a shadow of a much more general problem, but the speedy path to a solution lies in adopting as many assumptions about the input as possible. Whatever letters or numbers appear or don't appear, maximum lengths of strings or sizes of numbers, any shortcut you can take, take it.

Sometimes though, that can come back to bite you in part 2. The part 1 -> part 2 transition is really the only way in which these puzzles resemble Actual Software Engineering, because it represents an unforeseen change in requirements. In fact, after working through a handful of problems I found myself trying to anticipate what direction part 2 might go in, and weighed the risk/reward of making some particular function more or less general. In practice one can't actually anticipate part 2 with any useful accuracy, but I did find that "knowing there would be change" generally put some gentle pressure on me to write more flexible code in part 1.

On Clojure...

In theory, functional languages are great for these puzzles, because each one is just a computation. No file I/O, no network I/O, no interactive user input. The only side effect needed is a final print-to-the-screen. And it turned out that none of these puzzles truly needed (e.g. for performance) mutable state either. So it's a reasonable proving ground to take a language like this for a spin.

Clojure has a modest learning curve. You can get going with just a few basics, but to solve non-trivial problems you'll very quickly need to familiarize yourself with a large number of tools. As I've argued elsewhere, Clojure's promoters claim it to have a "small" syntax, which is nonsense. There are a bunch of fiddly special characters to learn, and a zillion standard library functions to memorize. While there are certainly a few obscure things you're not likely to need, like cycle or interpose, you are definitely going to need all of (in random order): for, map, doseq, filter, flatten, some?, every?, loop/recur, contains? (but also .contains), cond, first, rest, take, drop (and/or nthrest), into, merge, count, assoc/dissoc, apply, reverse, identity, nil?, key/val, get-in, conj, disj, remove, repeat, range, subvec, nth... and probably some I'm forgetting. On top of that, you'll have to learn a number of idiomatic compositions of these functions, such as making an equivalent to Python's dictionary comprehension using into {} combined with for or map. It's not intractable, but you will flail for a while, and write dumb stuff that will embarrass you later. It will also be difficult to find the right incantations (e.g. on Stack Overflow) because Clojurists unfortunately tend to post about the most "elegant" or theoretically idiomatic solutions, rather than the most comprehensible ones.

One thing that really shines in the language is string processing. The inputs for Advent puzzles are generally supplied as simple text files, and often there's structure that needs decoding. It's joyous that reading a file into a string is as simple as (slurp "filename"). Splitting on delimiters or patterns, iterating over lines or characters, parsing strings into numbers, all very simple, very easy. The function clojure.edn/read-string is particularly lovely to use, as it solves the problem of "treat this string of numerals as if I had typed it in source code", yielding a long or double as appropriate, without needing to be told.

Overall I really enjoyed using the language and hope I someday have an opportunity to use it for something real.

Tags Are A Bad Data Model

We've all seen tags, right? Twitter, Instagram, Steam, Stack Overflow, Bandcamp, just about every blogging engine or CMS.... all have tags. Sometimes they seem useful but mostly not. Why is that?

Tags are a fundamentally bad data model, because they offer exactly one extremely weak semantic.

Real quick, what do tags look like? Here's a basic relational implementation. Let's assume we have a post table, and we want to have tags and be able to associate tags to posts.

create table tag
(
  tag_id  serial        primary key
, name    varchar(200)  not null
);

create table post_tag_map
(
  post_tag_map_id  serial  primary key
, post_id          int     not null
, tag_id           int     not null
);

create unique index uidx_posttagmap_postidtagid on post_tag_map(post_id, tag_id) ;

There, those are the absolute basics. You would want other stuff like foreign keys, more indexes, created timestamps, and maybe what user created a tag or association, but this is the core of the model.

What can we do with this? Well, we can enumerate all tags that a post has, so that we can show them. Or, given a tag, we can enumerate all posts that have that tag. And... that's it.

To see how weak the semantic really is, let's imagine doing some basic analytics on our tags. How would we summarize the tagging of posts?

select p.post_id
     , p.name
     , max((t.name = 'fiction')::int) as TAG_FICTION
     , max((t.name = 'mysticism')::int) as TAG_MYSTICISM
     , max((t.name = 'politics')::int) as TAG_POLITICS
     , max((t.name = 'wtf')::int) as TAG_WTF
     , max((t.name = 'statistics')::int) as TAG_STATISTICS
     , max((t.name = 'humor')::int) as TAG_HUMOR
     /* many more... */
  from post p
  left join post_tag_map ptm on(p.post_id = ptm.post_id)
  left join tag t on(ptm.tag_id = t.tag_id)
 where 1=1
 group by p.post_id
        , p.name
;

That's the best we can do. A seemingly-endless bit array of 1/0 (or true/false if you like) flags showing whether any particular post has any particular tag. If new tags are added, we need to adjust our query (and table, if we store these results for easy use).

No tag ever conflicts with any other tag. If we have tags for "red" and "blue" and "green", a post can have all of them. If we have tags for "fiction" and "non-fiction" a post can have both of them. Remember, each one is just a flag, and they are all independent of one another.

In fact, we can describe our original data model a different way...

create table post_tag
(
  post_id         int      primary key
, tag_fiction     boolean  not null
, tag_mysticism   boolean  not null
, tag_politics    boolean  not null
, tag_wtf         boolean  not null
, tag_statistics  boolean  not null
, tag_humor       boolean  not null
/* many more... */
);

 ...where the ability to add columns -- at runtime! -- has been delegated to users, whether that be end-users or admins.

That's all the tags data model is. An infinity of boolean flags. No categories. No hierarchies. No key/value pairs or additional detail. This is all you get.

That is an incredibly weak semantic! It's terrible!

 

It only ever works where you, the system designer, fundamentally have no knowledge of what kind of meaning your users might want to impute to Things in your system and never will.

It works on Twitter because the breadth of topics that get discussed on Twitter is up to Twitter users, and changes constantly. If someone wants to try to create a Schelling point around some topic by using a #hashtag, they can. Maybe it'll catch on and maybe it won't. Maybe the word of phrase they chose is awkward or ambiguous or otherwise fails to communicate meaning. Maybe it's disingenuous or an outright lie. It's (arguably) not up to Twitter to manage this, and given the speed at which trending topics morph there's really no way they could.

It works on Bandcamp because artists choose their tags, and they choose them with purpose in mind: self-identifying with genres or styles, to aid discoverability.

It works on Steam for similar reasons. Users tag games to help other users find games they might like. It helps that users have coalesced around a fairly finite set of popular tags which doesn't change much over time, and the Steam UI highlights only the most popular tags (which in turn relies on having a big, engaged user base).

It probably won't work very well on your blog. Absent any conceptual framework, you'll struggle to think of what tags each post should have. The tags I used in the example are real, taken from a blog I've read for years. They don't make a ton of sense, and have never been useful for... anything.

It definitely won't work within your business software, because you need stronger semantics! You need categories, and hierarchies, and sets of mutually-exclusive/collectively-exhaustive values. Maybe some flags, sure, but specific flags with specific meanings. And all of this needs to be designed by the folks that build the software, not left up to users (at least not by default).

Chocolate Chip Banana Bread

This is AB's banana bread recipe from IJHFMF with a few tiny modifications and comments. I've made it probably 10 times over the last year

 

Ingredients

(dry team)

220g AP flour

35g oat flour, which means 35g oats put through a spice grinder or food processor

1 teaspoon salt

1 teaspoon baking soda

(wet team A)

1 stick unsalted butter, melted and cooled

2 eggs

1 teaspoon vanilla extract (or almond extract etc if you're feeling adventurous)

(wet team B)

4 bananas, extremely overripe, like seriously they will be nearly black, it takes a couple of weeks for them to get this way

180-210g sugar, to taste (original recipe says 210). I like to sub a little brown sugar or honey.

 

(misc)

extra butter for pan

dark(!) chocolate chips (only optional if you don't like being awesome)

chopped nuts (pecans or walnuts)

Note: if you have 6 bananas, or whatever, this recipe can be scaled up, but as written it fills a loaf pan so you will need additional pans.

Tools

kitchen scale (we bake by weight, not volume!)

3 bowls

mixing spoon

electric hand mixer (optional)

loaf pan (mine is about 10"x5"x3") or muffin pan(s)

parchment paper

oven (duh)

cooling rack

Procedure

1. Peel the bananas and pile them in a bowl. This step is first so that you can abort if you find that they're moldy.

2. Pre-heat the oven to 350degF

3. Melt the butter and set aside to cool.

4. Assemble wet team B by adding the sugar to the bananas and mashing/mixing thoroughly. I use a hand mixer.

5. Assemble the dry team. Toast the oats before grinding if you're feeling ambitious.

6. Finish wet team A by adding the eggs and vanilla to the butter and mixing gently. Just break the egg membranes and scramble them a bit. If the butter is hot when you do this it will cook the eggs and that is Bad.

7. Add wet team A to wet team B and mix. Again I use a hand mixer.

8. Add the combined wet team to the dry team. Mix only until combined (meaning no pockets of un-moistened flour). If your bananas were huge, or you used more than 4, the batter may seem too wet. Add a bit of extra flour. Getting this right takes practice.

9. Mix in chocolate chips and/or nuts to taste.

10a. If using a loaf pan, rub the inside with butter and then line with parchment paper (only the long sides need to be papered, the short sides will be touched by the batter and this is fine)

10b. If using a muffin pan, use muffin wrappers (or don't, in which case you're on your own)

11. Pour in your batter. For the loaf pan this is simple. For muffins, I haven't yet figured out how much should go in each one. Best of luck.

12a. For a loaf, bake in the center of the oven at 350 for 45 minutes then raise to 380-400 (experiment) for 15 minutes more (this browns the outside and firms the crust). Ovens vary and you may need to tweak times and temps. When it's done a toothpick will come out not-quite-clean (unlike a cake). If a toothpick comes out totally clean it's probably overbaked and the voice of Paul Hollywood will haunt your dreams.

12b. For muffins, bake for uhhhh less time than that? I haven't gotten them right yet.

13. Cool on a rack for 15 minutes in the pan, then remove from the pan and cool for 60 minutes or until you can't stand to wait any longer.

14. I wrap the loaf tightly in plastic, then foil, and keep it on the counter. It will definitely keep for a week. You will almost certainly eat it all before a week goes by.

Far Too Many Words About Airflow

Author's note: I recently wrote the below in nearly-unbroken stream-of-consciousness mode targeted at a specific audience of one. It is reproduced here with just a few minor redactions. The subject/prompt was "why I dislike Airflow".

 

Subject: Airflow

 

I've never used Prefect, but they wrote a detailed piece called "Why Not Airflow?" that hits on many of the relevant issues:

https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4

In my own experience with Airflow I identified three major issues (some of which are covered at the above link):

1. Scheduling is based on fixed points

(docs here https://airflow.apache.org/docs/stable/scheduler.html look how confusing that is!)

When we think about schedules we naturally think of "when is this thing supposed to run?" It might be at a specific time, or it might be an interval description like "every hour" or "every day at 02:30", but it is almost certainly not "...the job instance is started once the period it covers has ended" or "The scheduler runs your job one schedule_interval AFTER the start date, at the END of the period", as the Airflow docs describe it. Our natural conception of scheduling is future-oriented, whereas Airflow's is past-oriented. One way this manifests is that if I have a "daily" job and it first runs at say, 2020-04-01T11:57:23-06:00 (roughly now), its next run will be at 2020-04-02T11:57:23-06:00. That is effectively never what I want. I want to be able to set up a job to run e.g. daily at 11:00, and then since it's a little after 11:00 right now, kick off a manual run now without impacting that future schedule. Airflow can't do this. They try to paper over their weird notion of scheduling by supporting "@daily", "@hourly", and cron expressions, but these are all translated to their bizarre internal interval concept.

(Counterpoint: their schedule model does give rise to built-in backfill support, which is cool)

2. Schedules are optimized for machines, not humans

(see https://airflow.apache.org/docs/stable/timezone.html)

[Upfront weird bias note: I am cursed to trip over every timezone bug present in any system I use. As a result I have become very picky and opinionated about timezone handling.]

We run jobs on a schedule because of human concerns, not machine concerns. Any system that forces humans to bear the load of thinking about the gnarly details of time rather than making the machine do it, is not well designed. Originally, Airflow would only run in UTC. By now they've added support for running in other timezones but they still do not support DST, which basically means they don't actually support timezones. Now, standardizing on UTC certainly makes sense for some use cases at some firms, but for any firm headquartered in the US which mainly does business in the US, DST is a reality that affects humans and that means we have to deal with it. If we deny that, we're going to have problems. For example if I run a job at 05:00 UTC-7 a.k.a Mountain Standard Time, chosen such that it will complete and make data available by 08:00 UTC-7 when employees start arriving to work, I am setting myself up for problems every March when my employees change their clocks and start showing up at 08:00 UTC-6 (which is 07:00 UTC-7!) because they are now on Mountain Daylight Time. If I insist on scheduling in UTC or a fixed UTC offset, I am probably going to have to move half my schedules twice a year. That's crazy! Computers can do this for us!

3. DAGs cannot be dynamic

At the time I was seriously evaluating Airflow at [previous employer], this is what killed it.

A powerful technique in software design is to make our code data-driven. We don't often use that term, but it's a common technique, in fact so common we don't much notice it anymore. The simple way to think of this is I should be able to make my software do things by giving it new input rather than writing new code.

Consider a page like this one (from a former employer): https://shop.example.com/category-slug-foo/product-slug-bar/60774 [link removed, use your imagination]

No doubt you've been to thousands of such pages in your life as an internet user. And as an engineer, you know how they work. See that 60774 at the end? That's an ID, and we can infer that a request router will match against this URL, pull off that ID, and look it up in a database. The results of that lookup will be fed into a template, and the result of that template rendering will be the page that we see. In this way, one request handler and one template can render any product in the system, and the consequence of that is that adding new products requires only that we add data.

Airflow doesn't work this way!

In Airflow's marketing material (for lack of a better term), they say that you build up your DAG with code, and that this is better than specifying static configuration. What they don't tell you is that your DAG-constructing code is expected to evaluate to the same result every time. In order to change the shape of your DAG you must release new code. Sometimes this arguably makes sense. If my DAG at v1 is A -> B, and I change it in v2 to be A -> B -> C, perhaps it makes sense for that to be a new thing, or a new version of a thing. But what if my DAG is A -> B -> C, and I want to parallelize B, perhaps over an unpredictable number of input file chunks, as in A -> {B0, B1, ..., Bn} -> C where N is unknown until runtime? Airflow doesn't allow this, because again our DAG construction code must evaluate to the same shape every run. This means that if we want data to drive our code, that data must be stored inline with the code and we must re-deploy our code whenever that data changes.

This is not good. I have built multiple flows using Luigi that expand at runtime to thousands of dynamically-constructed task nodes, and whose behavior could be adjusted between runs by adding/changing rows in a table. These flows cannot be expressed in Airflow. You will find posts suggesting the contrary (e.g. https://towardsdatascience.com/creating-a-dynamic-dag-using-apache-airflow-a7a6f3c434f3) but note what is going on here: configuration is being fed to the DAG code but that configuration is stored with the code and changing it requires a code push. If you can't feed it input without a code push, it's not dynamic.

Airflow and the team at Airbnb that built it deserve a lot of credit for popularizing the concept of DAG-oriented structuring of data jobs in a way that Luigi (which predates it by years) failed to do. The slick UI, built-in scheduler, and built-in job executor are likewise praiseworthy. Ultimately though I've found that tightly coupling your flow structure to your scheduling system is a mis-feature. The fact that Luigi jobs must be initiated by an outside force is actually a powerful simplification: it means that a Luigi program is just a program which can be run from anywhere and does not (necessarily) require complex execution infrastructure. (Prefect can be used in this way as well, or with its own supplied scheduler.)

I also concede that there is value in wholesale adoption of Airflow (or something like it) as the central unifying structure of one's data wrangling universe. Regardless of the specific tech, having a single central scheduler is a great idea, because it makes the answers to "where is X scheduled?" or "is there anything that runs at time Y?" trivial to find. What's worrisome about Airflow specifically in that role is all the things it prevents you from doing, or allows only through dirty hacks like writing DAGs that use Luigi internally, or using code-generation to push dynamism to "build time".

Lastly, I have to concede that Airflow's sheer popularity is a vote in its favor. There's a lot of enthusiasm and momentum behind it, which bodes well for future feature additions and so on. There are already even managed Airflow-as-a-service products like Astronomer. I think it's still early, though. I've had a serious interest in dependency-structured data workflows since at least 2007, and until I encountered Luigi in 2014 I was aware of zero products that addressed this need, other than giant commercial monsters like Informatica. There's still a great deal of room for innovation and new players in this space.

 

[Original rant concludes here.]

Addendum

For whatever reason this topic keeps turning over in my head, so here are even more words.

 

I recently interviewed with 4 companies, of which 2 are using Airflow and a third is/was planning to adopt it. My current employer also uses it. I have little idea if any of them are actually happy with it, or if they understand the value and/or struggles it's creating for them.

Workflow / pipeline structuring is far from a solved problem. As I noted in my rant, the problem has existed basically forever, but general solutions -- platforms, frameworks -- have only started popping up in the last decade or so (again deliberately ignoring Informatica et al). There seems to be a temptation in the industry to treat Airflow as the de facto standard solution just because it's popular and appears to have a slick UI (the UI is actually clunky as hell, which you will discover within 30 seconds of trying to use it).

The options in this space by my reckoning are:

• Luigi (Spotify, 2012)
• Drake (Factual, 2013)
• Airflow (Airbnb, 2015)
• Dagster (dagster.io, 2018)
• Prefect (prefect.io, 2019)
• AWS Step Functions (AWS, 2016)
• chaining jobs in Jenkins (2011?)
• miscellaneous proprietary shit

These are all very different from each other! This is not a choice like React vs Vue, Flask vs FastAPI, Dropwizard vs Spring Boot, AWS vs GCP vs Azure, or [pick your favorite].

 

These tools aren't even all the same kind of thing. Luigi is a library, Drake is a CLI tool, Airflow and Prefect are libraries and schedulers and distributed task executors, Step Functions is a managed service, and Jenkins is a full server process and plugin ecosystem nominally intended for doing software builds.

 

They also differ markedly in how they model a workflow/pipeline. Luigi has Tasks which produce Targets and depend on other Targets, a design which almost fully externalizes state, with the result that Tasks whose outputs already exist will not be run, even if you ask them to. Airflow tightly couples its tracking of Task state to a server process + database, and requires Tasks to have an execution_date, so whether a Task will run if you ask it to depends on whether the server thinks it has or has not already run for the specified date. Drake, like make, uses output file timestamps to determine what needs to run. Jenkins just does whatever you tell it to (unless some plugin makes it work completely different!).

We don't even have standardized language for talking about these things. Several of these tools use the word "task" to name a major concept in their model. They're usually similar but hardly interchangeable. Perhaps a better example is trying to talk about "dynamic" DAGs, like I mentioned in my rant. I mean something very specific when I say a DAG is dynamic: that the shape of the execution graph is determined at runtime. Other people describe DAGs as dynamic simply because they were assembled by running code rather than specified as configuration data. These definitions are apples and oranges, and the result is a great deal of confusion in discussions of capabilities and alternatives, particularly in the very limited space of public conversation.

I encourage everyone to go out and try this stuff. Build a trivial, dummy pipeline and implement it in 3+ tools. Then repeat that exercise with a small pipeline that does real work and can stand in for the kind of problem you typically tackle. Then start building a solution to a serious problem. You don't have to build the whole thing. If you've gotten this far, simply writing stubbed-out functions/classes and focusing on how they wire up will tell you a great deal. Tasks that sleep for a random time and then touch a file or insert a row are often all you need to simulate your entire data processing world. As a final step think and work through what happens as you change things. Most of these tools don't discuss their implied deployment models, and the devil is in the details.

The bottom line for me is that this remains an active research area, even though I've been working on it for over a decade. I've learned quite a bit in that time but my wisdom remains dwarfed by my ignorance. Don't believe anyone who's trying to tell you that we have this figured out.

Startups Are (Mostly) A Bad Deal For Employees

I had been kicking around the idea of writing a post about how working for a startup is Actually Not Great for most employees, but Dan Luu wrote it for me so just read his instead:
https://danluu.com/startup-tradeoffs/

A Different Take on The Failings of Open Source Software

http://marktarver.com/thecathedralandthebizarre.html

The above is a very good article describing why most of the promises and predictions made in ESR's The Cathedral and the Bazaar (1998,) about the coming triumph of open source over closed source, failed to come true. As someone who read TCatB and fell for its promises -- in 1998, no less -- I cannot help but nod along to the arguments presented. It's helpful to go back to the early promises because while in 2019 open source may feel dominant, it has nevertheless clearly failed to live up to the original hype.

If anything, the author isn't critical enough when it comes to the claim that the world would converge on optimal solutions and avoid duplicate effort. I've got about ten thousand Javascript frameworks waiting outside to have a word with that one.

But I want to come at it from a different angle, and make what may be a novel criticism of open source.

Back in the day, if you wanted software to run your business on, you had to pay for it. Even if you only need the tools, to allow you to build your own software with which to run your business, you still had to pay for it. Operating systems, compilers, databases: money, money, money. The obvious consequence was that running a business on software was rather expensive. (This world sort of still exists, in Microsoft shops.)

But there was a secondary consequence that made things even more expensive than that: you needed to hire people that knew how to use and take care of all this software. This feels a bit like adding insult to injury, but it was actually something that businesses didn't mind at all, because it fit their existing mental model of how industrial systems worked. You see, if I had a factory, it was obviously necessary to hire people to work in it. If I bought a Bridgeport mill, it was only sensible that I needed to hire a machinist to run it. A crane needed an operator, a forklift needed a driver, and so on. Capital was never sufficient on its own, it was always necessary to add labor to get output.

So even as late as 1999, if you forked out a few million dollars a year for Oracle Database, it only made sense to spend a few hundred thousand extra employing a couple of professional Oracle DBAs. Likewise if you had a fleet of Windows NT servers in the racks, you would have a team of administrators trained (and likely Certified) on Microsoft software to look after them. And so it went for all the large proprietary business software vendors.

Then along comes open source software and... it costs nothing. Oh sure, the message is "free as in speech" not "free as in beer", but in practice it's all priced at $0 and if we're honest that's a big part of the attraction. An interesting thing happens psychologically. Paying $100k/yr for a professional DBA to support a $1M/yr Oracle installation feels very reasonable. Paying $100k/yr for a professional DBA to support a $0/yr MySQL installation... somehow does not.

There's another phenomenon developing right around the same time that reinforces this: the amateurization of business software development. Used to be one needed all this expensive software (and hardware!) to get a tech business off the ground. Then suddenly all you need is a cheap x86 server plus the zero-dollar LAMP stack and you're off to the races. For a while it was easy to dismiss this approach as the domain of hobbyists, but then the hobbyists starting launching successful businesses with it, forcing the entire industry to take it seriously. I say "amateurization" because the key driver here was the availability of free (as in beer) software that ran on cheap hardware, which allowed motivated hackers to get experience doing stuff without training, certifications, mentorship, or even (in many cases) college.

This deeply affected the culture of tech companies. In the proprietary high-dollar era, a developer was happy to enlist the help of a DBA, because the DBA was the expert on the database. The DBA was happy to enlist the help of the SysAdmin, becuase the SysAdmin was the expert on the OS and hardware. The SysAdmin was happy to enlist the help of the Network Admin... and so on. In the LAMP era, it's just four guys in a garage, and they all have to do everything just good enough to ship. The hardware, OS, network, database, compiler suite, various server software, and everything else is easy enough to procure, install, and configure that any motivated hacker can do it. There's neither a need nor time for specialized professionals.

This in turn has deeply affected the career development of technologists. Oracle DBA and Microsoft Server Admin used to be stable, high-paying jobs with long-term career prospects. Satellite firms built businesses around selling tools to these folks. These career-slash-cultures had their own conferences, newsletters, even glossy monthly magazines. Almost all of that is absent from the open source world. Do you know anyone who got training on how to install Linux? Anyone who's made a career out of MySQL administration? Someone certified on nginx?

I think it's been about 20 years since this evolution got going in earnest, so it seems reasonable to take a look back, as the author of the opening link did, and ask where it's gotten us.

In the "pro" column, it's a hell of a lot easier to start a company than it ever has been. If you have an idea and the drive to pursue it, it's never been cheaper or easier to try giving it a go.

In the "con" column, we have a systematic loss of expertise and deep understanding. We assume now that any piece of software that's no further away than apt-get install should be something we can run professionally, in production, with real money on the line, with no training, no practice, hell maybe not even a skim of the documentation.

Kubernetes is Anti-DevOps

(bias warning: I think Kubernetes is basically cancer)

So over the last 10 years or so there's been this whole DevOps... movement... thing. The industry got the idea into its collective head that developers should participate in the operation of the software that they build, that operators should adopt practices from development like using source control, and that in general development and operations should work more closely. In the limiting/idealized case, developers are the operators and there's no organizational separation at all.

In general this was a good idea! Developers who have ops responsibilities build software that is more operable, and operators became more effective by using source control and programming languages better than /bin/bash.

There has also been a lot of pretense and bullshit. Companies have undertaken "DevOps transformations" and crap like that which ultimately accomplished nothing at all. System Administrators have had their titled changed to "DevOps Engineers" with zero change in responsibilities or organizational structure. Any company that uses the cloud has declared that they "do DevOps" and left it at that.

And then there's Kubernetes.

Kubernetes runs containers, containers means Docker, and Docker is super DevOps, right?? Yeah, about that...

An interesting thing about containers is they simplify deployment. If I have a Python program that depends on a specific interpreter version and specific versions of a few dozen libraries, I need to be able to manage the deployment target to make sure all that important stuff is there. But if I package it in a container, now it's a singular artifact that I can plop down anywhere and just docker run that action (so the theory goes, anyway).

That simplified deployment can act as an inter-organizational interface. This is a fancy way of saying that it enables that practice we all just decided was bad: developers throwing their code over the wall to be ops' problem. And once they start doing that, the next step is overly complicated systems of containers with elaborate dependencies on each other, and now you need "container orchestration".

Kubernetes thus becomes the icing on the anti-DevOps cake. In theory it enables all this flexibility and solves all these container orchestration problems (that, it should be noted, we didn't have at all just 5 short years ago). In reality it's a hyper-complex operations layer that requires a handful of specialists in order to use at all.

Kubernetes does nothing for the developer, but nor does it hurt the developer. Being just an execution substrate, Kubernetes is irrelevant to the developer. Thus in their ordinary course of business, a developer would have no need to learn and understand how it works. Nor would it be efficient for them to do so, given Kubernetes' off-the-charts complexity. It's reasonable for, say, a Java developer to learn how to manage the JVM as a runtime and what it takes to deploy applications with it. By comparison, learning Kubernetes is like learning how to run an entire private cloud: simply not something it's worth a developer's time to do.

So ultimately, adopting Kubernetes is about the most anti-DevOps move you could make as a software organization. The wall between dev and ops that we've spent the last decade tearing down is going right back up, and we'll set about throwing our code over it. Enjoy!

This all doesn't make the argument as clearly as I would like but hey this is my blog and I get to rant if I want.

Why Engineers Are Grumpy

https://humanwhocodes.com/blog/2012/06/12/the-care-and-feeding-of-software-engineers-or-why-engineers-are-grumpy/

A near-perfect explanation of why engineers are grumpy and say "no" all the time.

Solid Advice

https://rachelbythebay.com/w/2019/10/25/enabler/


"When you're already in a hole, QUIT DIGGING."

Andromeda in the Future Sky

https://kottke.org/19/10/behold-our-dazzling-night-sky-when-the-milky-way-collides-with-andromeda-in-4-billion-years

I have a long-running fascination with cosmology, which is part of how this blog got its name.

If you live in a city, with its attendant light pollution, you likely have never even seen the Milky Way except in photos. If that's true you owe it to yourself to visit somewhere far from city lights where the pale glow of our galactic disk is visible to the unaided eye. Then imagine seeing not just our own galaxy but another galaxy in the night sky... (now click the link)

THIS is what America needs

https://www.politico.com/magazine/story/2019/10/13/america-cultural-divide-red-state-blue-state-228111

Caleb Wright, who’s from Chapel Hill says, “The value is that you can staunchly disagree with someone, but also humanize the person.” Adds Gaby, “It was more to learn about each other than to change people’s minds.”

The point, in other words, is to combat “othering.”

Why Remote War is Bad War

https://www.technologyreview.com/s/614488/why-remote-war-is-bad-war/

Just one link today, and it's not one that's going to make you feel good. About anything.

The moral distance a society creates from the killing done in its name will increase the killing done in its name. We allow technology to increase moral distance; thus, technology increases the killing. More civilians than combatants die in modern warfare, so technology increases worldwide civilian murder at the hands of armies large and small.

Sunday Assorted Links 2019-10-06

https://kevinlynagh.com/notes/pricing-niche-products/
I love keyboards and I love auction theory and this link has both!

https://www.eidel.io/2019/04/24/making-my-own-glasses/
Know thyself.

https://qz.com/1721901/hong-kong-anti-mask-law-a-history-of-mask-bans-around-the-world/
Masks are about inverting power dynamics.

https://www.theatlantic.com/education/archive/2019/10/college-students-dont-want-fancy-libraries/599455/
Books are good.

https://www.youtube.com/watch?v=1OfxlSG6q5Y
Toasters: another thing that's gotten worse.
I will say though that this machine is awesome, and if you need a toaster just forget all the stuff that only toasts bread and get this instead, because it's better at that and it does other stuff (e.g. it reheats pizza like a boss).

https://williamyaoh.com/posts/2019-10-05-you-are-already-smart-enough.html
"...the perception of what tools, libraries, and concepts are important ends up distorted by novelty and excessive cleverness."
Honestly that feels like reason enough to avoid this whole scene.

http://rachelbythebay.com/w/2019/10/05/nxdomain/
Vendor shit is awful. ISPs are awful. Know how your tools work. It's worth spending extra effort to do things properly.

https://towardsdatascience.com/coding-ml-tools-like-you-code-ml-models-ddba3357eace
This is cool and I want to try it.

https://www.slashgeek.net/2016/05/17/cloudflare-is-ruining-the-internet-for-me/
American tech companies pretend most of the rest of the world doesn't exist, film at 11.

React rant

A transcript of something that happened in Slack the other day...

me, trying to avoid starting a rant: [vicious off-topic React rant]
(ed: literally what I wrote, brackets and all)

FE specialist friend & former coworker, stepping in it: ...since you brought it up again, I'm curious why you say it's user-hostile (I'll agree up-front that a lot of its uses can be)

(ed: lightly edited to conceal identities)

me: so like, once upon a time the web was documents, right? and people had slow computers and poor internet connections and retrieving a few KB could be perceptibly slow, as in multiple seconds

me: and then a lot of stuff got better. The experience of browsing a document-based site in 2019, even on a bad machine on a bad connection, is generally lightning-fast. The highest-profile example, though imperfect, is Wikipedia

me: but we don't see that, in general. In general, we see "modern" web development, where, for the convenience of the developer, we routinely ship multiple megabytes of executable code to the client, not just once but repeatedly

me: we consume the client's bandwidth, the client's CPU cycles (and battery power!), to do the same exact operations, over and over again. Instead of sending them the document that results from those operations.

me: And sometimes, this is justified. When you are building an app. Facebook, twitter, google maps, gmail, etc., these are all applications that use the web as a delivery mechanism. But Wikipedia, retailers (including Amazon), newspapers, blogs, these are document sites just like we had in 1995

me: and we are constructing them in such a way that they are hundreds, even thousands of times slower than they could be

me: and we do this in a way that systematically disregards who our client is. We stand at our electrically-raisable desks, with our $3k Macs, 27" monitors, and multi-hundred-Mbps internet connections and revel in the beauty of what we can create. But our client has a 2-year-old mid-range Android phone and intermittent/noisy metered 3G service

me: and we are shipping that client five goddamned megabytes that render into less than a kilobyte of content

me: WE. SHOULD. BE. ASHAMED.

me: </rant>

So yeah, my throwaway "I don't want to rant about this today" comment became a kind of self-fulfilling prophecy and I ranted anyway.

I'm sure I'll have more to say about this in the future, but having transcribed this here I feel it's worth expanding on it a little.

Circa 1997 I got my first Pentium-era PC, which was also my first internet-connected machine. It had a 200MHz CPU, 32MB of RAM (which was somewhat luxurious at the time, 16MB being much more common), a 3.2GB hard disk, and a 33.6Kbps dial-up connection to a local telco ISP. The internet was much less a part of our daily lives like it is today, but being a huge nerd it was certainly important to me even at the time. This comic is from a few years later, but it captures the spirit of what living with dial-up was like.

Fast forward to now. My current PC has a 3.8GHz (up to 4.4 boosted) CPU with 6 cores, 16GB of RAM, a 1TB SSD plus 4TB of spinning rust, and a 200Mbps cable internet connection. Let's review the gains:

• CPU: rounded to 4GHz, a 20x gain (or if we count all the cores, 120x)
• RAM: 512x
• Disk: over 1000x in capacity, speed-wise I don't even know, but if we use bus bandwidth as a proxy, this suggests Ultra SCSI was 160Mbps vs NVMe at 32Gbps, so that would be 200x
• Network: ~5952x, yes that's right, a nearly six thousand-fold gain

Wow! Surely in light of all this hardware improvement, the experience of using a computer, and using the internet in particular, must be much faster now, right?

See, about that...

Let's use Firefox developer tools to measure a few websites:

• nytimes.com: 1.58MB (on the wire), 5.39MB (decompressed), 6.16s, 47 requests
• washingtonpost.com: 4.67MB / 6.15MB, ~8s, 93 requests
• slatestarcodex.com: 1.15MB / 1.47MB, 2.31s, 48 requests
• marginalrevolution.com: 0.98MB / 2.47MB, 2.98s, 35 requests 
• that Wikipedia page linked above: 219KB / 858KB, 1.71s, 22 requests

Out of this entirely non-scientific sample of two major newspapers, two of my favorite blogs, and a Wikipedia article, only Wikipedia comes out looking vaguely close to something possibly approximating fast.

It's so disappointing, so draining, I don't even have the energy to keep ranting about it. All the hard work by hardware engineers over the last two decades has been eaten away by developers making their own lives easier at the expense of their customers, and I just can't fucking stand it.

Thursday Assorted Links 2019-10-03

https://www.theverge.com/2019/10/3/20895798/bird-scooter-fundraising-valuation-unit-economics
Bird managed to lose ~$100M in Q1 on revenue of only ~$15M. Despite that I guess they managed to raise a bunch more money. For some reason The Verge parrots (get it?) Bird's claims about the lifespan of its new scooter models. I get that you can project 15 months of life based on testing, but when the things have only been on the street for 1-4 months you need to say it's a projection.

https://www.delish.com/food-news/a29351591/mcrib-back-mcdonalds-2019/
The McRib is coming back. Does that mean Chipotle Chorizo is next? That wacky theory that McRib runs have something to do with pork futures always struck me as crazy enough to be true.

https://www.objectstyle.com/agile/why-developers-hate-agile
Assertion: Agile Process Bullshit is about legibility, in the Seeing Like A State sense.

https://doisinkidney.com/posts/2019-10-02-what-is-good-about-haskell.html
Look, I get what you're trying to do here, but it's not working. "Let's implement a basic data structure and a sorting algorithm!" is such a Haskell programmer thing to do. If I mostly write code in [general purpose language] I don't need those things, because they're in the standard library, or a popular-consensus third party library.

I think programmers frequently forget just how different other kinds of programming are from whatever it is they do. Web programming is not game programming is not systems programming is not embedded programming is not kernel programming is not [et cetera]. Speaking as a web/data guy, showing me how awesome your thing is at foundational data structures / algorithms stuff is just 100% irrelevant, because I will never implement those things myself.

Ask HN: Who Wants To Be Fired?
You think your job sucks? Reading this is gonna make you feel better.

Microsoft Surface Laptop 3 pre-order
What they want for extra RAM and disk is fucking reprehensible. It's three hundred dollars to go from 128GB to 256GB. Buying 128GB of NVMe at retail is like forty bucks. A nice 1TB NVMe drive can be had for $170. They know this. They know 8GB/128GB is for suckers, and that almost everyone will upgrade. It's Apple-style price anchoring. The margins on this must be fantastic for Microsoft.

...I may buy one anyway.

https://www.pcgamer.com/world-of-warcraft-classic-players-cant-stop-feuding-over-the-abbreviation-for-an-old-dungeon/
Every time I'm in Westfall someone is having this argument and it drives me nuts.

Introducing the blog

It's finally happening. A blog.

Expect a mix of links and rants, starting Very Soon Now.