Saturday, October 23, 2021

Advent of Code 2020

 Uh dude, wasn't that in like, December?

Yeah, well, I started it in December and then life happened. I just finished today (ed: 09-OCT).

I don't precisely remember why I started. Maybe someone mentioned it in work Slack and it seemed fun? In any case, when I started I couldn't decide what language to use, so I did each problem twice: once in pure SQL for a challenge, and again in Clojure as an exercise in learning the language.

(Spoilers follow)

Code here: https://github.com/slotrans/advent-of-code-2020

Day 1, "Report Repair": Trivial. There might be a non-brute-force solution but for such tiny input, I couldn't be bothered. Looking back at my solutions, my SQL solution was pretty straightforward (and beautifully declarative), but I was flailing in Clojure.

Day 2,  "Password Philosophy": Also pretty trivial. This was months ago now, but I can remember the SQL solution flowing right out of my fingers. My Clojure was improving but still awkward. Parts of it look downright silly in retrospect.

Day 3, "Toboggan Trajectory": Now we're talkin'. This was fun! The first of many grid-based/spatial puzzles. This one highlighted a couple of interesting things about SQL. First, the fact that the input is ordered requires special handling at load time, because relations are not naturally ordered. Second, the way SQL performs iteration implicitly through consuming input rows (and producing output rows) rather than explicitly through looping or recursion. On the Clojure side, my comfort level was clearly improving, with recursion in particular. I hadn't yet stumbled on the most convenient ways of modeling grids.

Day 4, "Passport Processing": This one is mostly about string processing, specifically handling the irregular input format. Parsing the input in SQL was gnarly. Clojure's string and sequence processing really started to shine here. This is one of the few puzzles that resembles what I'd call "business" code.

Day 5, "Binary Boarding": Pretty straightforward once you get past the obfuscation of the input. Postgres' translate function and bit(N) type helped keep the SQL solution relatively compact. Pleased to find Clojure has a reasonably easy way of parsing binary literals (which I then forgot and had to look up again for a later puzzle).

Day 6, "Custom Customs": Again SQL struggles with wrangling the input. This isn't surprising by the way, since typically the structuring of input into nicely labeled and typed fields is done before data ever hits the SQL layer. It's wild how much shorter the Clojure solution is.

Day 7, "Handy Haversacks":  This is where things start getting a little crazy. The structure of the puzzle is inherently recursive which poses a big challenge for SQL, and I remember having to fiddle with my solution quite a bit and re-read the documentation for with recursive several times. For Clojure, I remember having to think for a while about how to structure the input rules as data.

Day 8,  "Handheld Halting": This one really pushed SQL to the limit, requiring (or at least, I used) both recursive CTEs and lateral joins. The Clojure code is much more straightforward. Also I can see this is the first puzzle for which I included a (def sample-input ...) to test my solution against the provided sample in the same way it would run against the full input.

Day 9, "Encoding Error": The SQL solution to part 1 is wonderfully compact, but part 2 required an odd combination of window functions and a lateral join. The Clojure code for this one is fairly unremarkable, though I do cringe a little at (vec (for ...)), apparently I had not yet discovered mapv.

Day 10,  "Adapter Array": Part 1 is fine, whatever, but part 2... this one pissed me off.

Have you ever heard the riddle "why are manhole covers round?" It's a bad riddle, because if you've heard it you know the answer, and if you haven't heard it you have no way of finding the answer, besides experiencing a miraculous logical leap. Puzzle 10 part 2 is like that, because if you've seen the climbing stairs problem, and notice the correspondence with this "joltage" nonsense, the solution will be obvious and easy. If you haven't, then you'll need either the aforementioned logical leap, or end up doing something dumb like I did (i.e. attacking the problem very literally).

I very nearly ragequit after this one.

Day 11, "Seating System": Another grid puzzle, and the first one involving cellular automata! It's also, a bit sadly, where I gave up on writing SQL solutions, as they were becoming increasingly convoluted. Anyway I found it fun, and it served as a nice base to come back to when tackling future cellular automata puzzles.

Day 12, "Rain Risk": Neat puzzle. The most interesting part was figuring out how to model the instructions and then how to interpret them. This is one of those puzzles where part 2 completely upends what you did for part 1.

Day 13, "Shuttle Search": Not counting Day 10, this is the first one I hit a real performance problem on. More specifically, my naive solution was too slow to ever finish. Again I had to look at others' posted solutions for some inspiration. And again, I don't know that I ever would have figured out the proper technique without being shown. Perhaps for that reason, this is the last one I did in-order in December. At this point I skipped ahead to Day 24 (it was already 26-Dec) to see "the end", and then stopped.

Day 14,  "Docking Data": This is where I resumed, on 31-Aug. It was interesting to see how well Clojure had "stuck" to my brain, after having not used it at all for 8 months. Bit masks are fun.

Day 15, "Rambunctious Recitation": I suspect there's a pattern to this one that can radically simplify things, but I couldn't see it so I forged ahead with brute force and literalism, as usual. Interestingly, part 2 is identical to part 1 except for being pushed to an absurdly large number of iterations. My code wasn't fast enough, so I did peruse a few other solutions for ideas. In the end though I just took an educated guess at where the slowdown was, and turned out to be correct.

Day 16, "Ticket Translation": I enjoyed the deductive reasoning aspect of this puzzle. I definitely felt like the solution-finding routine would have been much more naturally expressed in an imperative language with a "while" loop.

Day 17, "Conway Cubes": This was probably my favorite puzzle. It turned out to be helpful that I'd already done Day 24, which has a lot of similarities. The best part was finding that, other than having to re-write my "get adjacent coordinates" function (which I had been stubbornly unrolling by hand rather than expressing with nested loops), modifying my solution from 3 dimensions to 4 for part 2 required only the most trivial code changes.

Day 18, "Operation Order": This was one of the hardest for me. Part 1 I was able to do, but when part 2 introduced precedence rules I got stuck. I spent a day thinking about it and trying a few approaches, but got nowhere. I found a tutorial on writing a simple calculator-interpreter in Python, but it was brutally stateful. I spent some time trying to adapt it to be functional but again got stuck. I ended up implementing a transliteration of the tutorial's code using atoms for mutable state. It's VERY gross. I would still like to see a proper functional solution to this one.

Day 19, "Monster Messages": This was the hardest. It took me 5 days, and careful study of several other solutions. I'm sure that any CS majors who happen to have written a regex engine had no trouble with this, but I sure did. I was able to solve part 1 without too much trouble, but got very confused when my solution didn't work at all for part 2, despite only minor modifications to the problem. Unlike 10 and 13 though, this one didn't make me angry, because it's a fair problem that can legitimately be figured out with no prior knowledge.

Day 20,  "Jurassic Jigsaw": I enjoyed this one a lot, because it had a surprising amount of depth. The code for this one is the longest by far, almost double the next largest. Like 19, it took me 5 days to complete, but happily I didn't need to consult any other solutions. This is another one where data modeling is key. Parts of this would definitely have been easier in an ordinary mutable-state language. Also I finally used reduce comfortably! Twice!

Day 21,  "Allergen Assessment": Effectively a re-hash of Day 16, with deductive reasoning. I infer that there's an "easier" (less general?) solution to just part 1, but I ended up going straight for the full solution so I needed only a trivial amount of extra code to complete part 2. After the early pre-processing steps, I actually worked out the solution "on paper" and then backed into a code solution. Some of the instructions feel like red herrings... the statement "allergens aren't always marked" is highlighted, but as far as I can tell doesn't actually matter?

Day 22, "Crab Combat": Fun with recursion. Getting part 2 right turned out to be very fiddly. I was happy this didn't turn into another performance problem...

Day 23, "Crab Cups": ...but this one sure did! This is another one where the solution to part 1 should in theory work for part 2 without modification. And I'm sure it would have, if I had a few months to wait for the answer. The direct, sequence-splitting-and-reassembling strategy solves the problem very simply but doesn't turn out to be efficient.

I ended up writing the most solutions for this one. The original slow Clojure solution. A second Clojure solution that leaned on mutating a java.util.LinkedList and was significantly faster, but still too slow. Then a pure Java solution based on that, which was faster yet but still around a full day to run 10M turns. The solution I actually got an answer from -- in ~2.5 hours -- was a pure Java solution that (still quite naively) manipulated an array of integer primitives.

Once I had an honestly-earned answer to my very last puzzle (I had already done 24 and 25), I went looking for performance help. I suspected all along that what I needed was the right data structure, and when I saw it, it was one of those forehead-slapping moments. I had never thought of modeling a linked list as a map of value->next_value. Rewriting the code for that model took some time, but my final Clojure solution ran in 83 seconds.

Day 24, "Lobby Layout": Another favorite. I was thrilled to finally have an excuse to learn how to model a hex grid. Working in a weird coordinate system, and dealing with cellular automata rules in a sparse space, made for a super engaging and rewarding problem. As I mentioned earlier, I skipped ahead to this one after 13, which turned out quite nicely since it was a jump from basically the second-worst to the second-best puzzle.

Day 25, "Combo Breaker": A nice little gimme at the end.


A few general observations...

Since the puzzles don't build on each other, the solution to each one can be treated as throwaway code. Most of the usual best practices can be thrown out the window. When I did look at others' solutions, I observed quite a few single-letter variable names and other signs of code optimized for speed-of-writing rather than clarity. You can certainly do that if you want, but I doubt that typing out readable names is what's going to stop you from solving each puzzle in one day.

Many puzzles are like a shadow of a much more general problem, but the speedy path to a solution lies in adopting as many assumptions about the input as possible. Whatever letters or numbers appear or don't appear, maximum lengths of strings or sizes of numbers, any shortcut you can take, take it.

Sometimes though, that can come back to bite you in part 2. The part 1 -> part 2 transition is really the only way in which these puzzles resemble Actual Software Engineering, because it represents an unforeseen change in requirements. In fact, after working through a handful of problems I found myself trying to anticipate what direction part 2 might go in, and weighed the risk/reward of making some particular function more or less general. In practice one can't actually anticipate part 2 with any useful accuracy, but I did find that "knowing there would be change" generally put some gentle pressure on me to write more flexible code in part 1.


On Clojure...

In theory, functional languages are great for these puzzles, because each one is just a computation. No file I/O, no network I/O, no interactive user input. The only side effect needed is a final print-to-the-screen. And it turned out that none of these puzzles truly needed (e.g. for performance) mutable state either. So it's a reasonable proving ground to take a language like this for a spin.

Clojure has a modest learning curve. You can get going with just a few basics, but to solve non-trivial problems you'll very quickly need to familiarize yourself with a large number of tools. As I've argued elsewhere, Clojure's promoters claim it to have a "small" syntax, which is nonsense. There are a bunch of fiddly special characters to learn, and a zillion standard library functions to memorize. While there are certainly a few obscure things you're not likely to need, like cycle or interpose, you are definitely going to need all of (in random order): for, map, doseq, filter, flatten, some?, every?, loop/recur, contains? (but also .contains), cond, first, rest, take, drop (and/or nthrest), into, merge, count, assoc/dissoc, apply, reverse, identity, nil?, key/val, get-in, conj, disj, remove, repeat, range, subvec, nth... and probably some I'm forgetting. On top of that, you'll have to learn a number of idiomatic compositions of these functions, such as making an equivalent to Python's dictionary comprehension using into {} combined with for or map. It's not intractable, but you will flail for a while, and write dumb stuff that will embarrass you later. It will also be difficult to find the right incantations (e.g. on Stack Overflow) because Clojurists unfortunately tend to post about the most "elegant" or theoretically idiomatic solutions, rather than the most comprehensible ones.

One thing that really shines in the language is string processing. The inputs for Advent puzzles are generally supplied as simple text files, and often there's structure that needs decoding. It's joyous that reading a file into a string is as simple as (slurp "filename"). Splitting on delimiters or patterns, iterating over lines or characters, parsing strings into numbers, all very simple, very easy. The function clojure.edn/read-string is particularly lovely to use, as it solves the problem of "treat this string of numerals as if I had typed it in source code", yielding a long or double as appropriate, without needing to be told.

Overall I really enjoyed using the language and hope I someday have an opportunity to use it for something real.