Alex C. Viana's Blog

Welcome, Substack Posts

Fri, 27 Mar 2026 00:00:00 GMT

Back in 2020–2021 I ran a short-lived newsletter on Substack called Alex's Newsletter. I wrote 5 posts there before life got in the way, and they've been sitting off on their own ever since.

As part of my ongoing effort to get all my writing in one place (see Hello, Astro), I've now migrated those 5 posts here under the new source-substack tag.

The process was almost embarrassingly easy. I asked an agent to write a Python migration script that pulled the posts from the Substack RSS feed, stripped the boilerplate, downloaded the images locally, and converted everything to Markdown. The whole thing was done in under 10 minutes.

Modern agents are genuinely good at this kind of task.

Hello, cloud-reader

Fri, 27 Mar 2026 00:00:00 GMT

I'm from the generation that still complains about the death of Google Reader. So when I had some surprising quick wins using AI at work this week — wins that genuinely convinced me I could just crank something out in an evening — I decided to spend my Friday night building my own RSS reader.

The result is cloud-reader, running entirely on the Cloudflare developer platform. You can see the live deployment at cloud-reader.alexcostaviana.workers.dev — though I may put it behind Cloudflare Access at some point.

The stack is all Cloudflare under the hood: a Worker for the REST API, D1 (SQLite) for storage, and static assets — all in a single deployment. On top of that: Hono for routing, Drizzle ORM, React 19 + Vite for the frontend, and Cloudflare's Kumo UI components with Tailwind v4.

I used the same agents + worklog approach I've been developing on this blog. Every decision and dead end goes into a WORKLOG.md, which makes it easy to reload context between sessions without losing continuity. I also broke the project into explicit milestones — and the milestones weren't about managing complexity, they were about protecting space at the beginning to carefully think through architecture and design decisions. Things like migration strategy, linting setup, and tool choices. Getting those right up front meant the project moved faster once it had some maturity, not slower.

Like I posted on Twitter: in about two hours I had 60+ commits — tested, linted, type-checked, and documented. I feel like I'm living in the future and way behind everyone at the same time.

Hello, Astro

Thu, 15 Jan 2026 00:00:00 GMT

I made a bunch of big changes to my blog to kick off 2026. I switched to the Astro framework with the Astropaper template and switched hosting providers to Cloudflare Workers (disclosure: I'm currently employed at Cloudflare which recently acquired Astro).

I'm really happy with the update! My blog now has search, a dedicated tags page, better typography, a reading progress bar, and overall better aesthetics. You can find the source code on GitHub.

14 years and 4 blogs in one place

This is my 4th blog going as far back as 2012. I started tech blogging on Tumblr (!!), then transitioned to the Python Pelican framework hosted on GitHub.io, and my last blog was built with the Nextra framework and hosted on Vercel.

I started the process of collecting some of my previous writing in my last blog iteration. As part of this migration I finished that process and have now collected all my previous blog posts in one place for the first time. All my blogs posts are now tagged with source-* so you can see what came from where.

[Update: it's actually now 14 years and 5 blogs — I've since added posts from my old Substack newsletter.]

Coding with agents

This project involved a lot of boilerplate configuration and metadata tweaking. This is not the kind of thing I have a ton of time for so it was the perfect project to check out the latest generation of agents with. I wrote basically no code directly for this project, instead pushing everything to Claude Code through OpenCode.

I'm very happy with both the final result of the blog and the experience of working with Agents at the start of 2026. I want to write more about the experience of working with Agents on this project later - but for now I just wanted to throw up a hello world.

Verifying Git Branch Histories with Diff, Log, and Blame

Tue, 29 Apr 2025 00:00:00 GMT

I recently had to perform some "git surgery" to rebuild the main branch of a repository I was working on. The goal was to recreate the branch with the exact same code but a corrected commit history. This gave me the opportunity to use Git in a few ways I hadn’t before. I took extensive notes to convince myself—and others—that the process worked, which became the basis for this post.

Context

The short backstory is that we had a few commits in the project trunk that needed to be removed. These problematic commits were squashed patches of diffs that had also been correctly merged earlier as unsquashed merge commits. As a result, the branch technically had all the right code, but nearly all the changes were attributed to just two squashed commits. That meant the git log was technically accurate, but git blame became almost useless.

With a combination of branching, merging, and cherry-picking, I was able to create what I believed was a clean, corrected branch. But how do you verify that?

Comparing Branch Heads

The first thing I wanted to check was that the content at the branch tips was identical. Normally, when comparing files, I might use diff or compare md5 hashes. But since we’re working within Git, we can compare the branches directly:

# Check the diff against main—there should be no changes:
> git diff --stat develop-new..main

# Also check against a previous commit on main to confirm that a diff *would* show something
> git diff --stat develop-new..main~1

This obviously isn’t a new command, but previously I’d only used git diff on individual files or within a single branch.

Comparing the Git Log

This part was a bit trickier. I needed to confirm that the Git history looked correct. The first step was to manually inspect the log:

# The graph should look clean and include all expected commits
> git log --graph --oneline --pretty=format:"%h %ad %s" --date=short

I’m not sure there’s a more analytical way to verify this than just reading through and making sure the history makes sense. But there is a way to be more precise when checking whether the commits we intended to exclude actually stayed out:

# These commits should only exist on main—not on the new branch
> git branch --contains 48d14a0
> git branch --contains 4915a48

These commands scan the entire commit tree and list which branches contain the specified commits, helping confirm they weren’t accidentally included in the clean branch.

Git Blame

After confirming that the file contents were identical and the unwanted commits were removed, the last check was to ensure the blame history was corrected. I picked a random file that I knew should have a large number of different commits. As with the git log, you can inspect this visually—but here, there's also a more quantitative approach:

# Count the unique commits that appear in the file's blame output
> git blame .github/workflows/publish-image.yml | awk '{print $1}' | sort | uniq -c | sort -nr

> git blame main -- .github/workflows/publish-image.yml | awk '{print $1}' | sort | uniq -c | sort -nr

This counts how many lines are attributed to each commit. The cleaned-up branch should show a more diverse history—exactly what we were aiming for.

At this point, I was finally convinced we could replace the main branch with this new corrected one.

Thanks for reading, if you enjoyed this article you should check out this post which touches on cleaning git history.

Brian Eno's Oblique Strategies in the Terminal

Tue, 01 Apr 2025 00:00:00 GMT

This post is about a fun little command-line utility I made with ChatGPT's help to randomly generate exercises from Brian Eno's "Oblique Strategies" in an aesthetically pleasing presentation. Here’s what it looks like:

I love seeing one of these every time I open a terminal — it's like a creative spark before doing any work. It recenters me and challenges me to reconsider my assumptions and process.

Here's how I built it. The script is written in Fish shell and works best in terminals with solid Unicode and font rendering support, like Kitty or Alacritty. It reads in the exercises from a source file adapted from this GitHub repo. Here is the source code:

# ~/.config/fish/functions/print_random_oblique_strategy.fish

function print_random_oblique_strategy
    set strategy_file ~/Developer/oblique-strategies.txt

    if test -f $strategy_file
        set strategies (cat $strategy_file)
        set count (count $strategies)
        set index (random 1 $count)
        set strategy $strategies[$index]

        # Terminal width
        set term_width (tput cols)

        # Padding inside the box
        set padding 4
        set strategy_length (string length -- $strategy)
        set content_width (math "$strategy_length + $padding")
        set left_margin (math "floor(($term_width - $content_width) / 2)")

        # Box drawing characters
        set hline "─"
        set vline "│"
        set tl "┌"
        set tr "┐"
        set bl "└"
        set br "┘"

        # Construct the box
        set top (string repeat -n $left_margin " ")$tl(string repeat -n (math "$content_width - 2") $hline)$tr
        set middle (string repeat -n $left_margin " ")$vline (set_color --bold)"$strategy"(set_color normal) $vline
        set bottom (string repeat -n $left_margin " ")$bl(string repeat -n (math "$content_width - 2") $hline)$br

        echo
        echo (set_color yellow)"🎲 Oblique Strategy:"(set_color normal)
        echo
        echo $top
        echo $middle
        echo $bottom
        echo
    end
end

The script defines a function called print_random_oblique_strategy, which can be run anywhere in your Fish shell. To set this up, save the function in ~/.config/fish/functions/print_random_oblique_strategy.fish, and make sure your strategy list is in ~/Developer/oblique-strategies.txt. To have it run whenever a new shell is started, I added it as a command in ~/.config/fish/config.fish.

To get a clean splash page, I needed to do two more things. First, I removed the default "Welcome to Fish..." message by adding set fish_greeting (which sets it to nothing) in the config.fish file. Finally, macOS displays a "last login" message, which you can silence by running touch ~/.hushlogin.

If you enjoy this you might also be interested in my post on practicing jazz from the command line. While you're at it, check out this transcendent 6-hour "time-stretched" version of Eno's "Music for Airports".

2024-12-09 Weekly Note - Upgrading Raspberry Pi OS

Mon, 09 Dec 2024 00:00:00 GMT

Early in the pandemic I got a Raspberry Pi and set it up to run Pi-Hole as an ad blocker on my home network. In theory, it felt like a nice-to-have. But once you get used to surfing the web without walls of ads on almost every page - there’s no going back.

Last weekend I decided to update the operating system on the Pi, Raspberry Pi OS (formerly Rasbian). Because I was going between major release versions this required a full reinstall and reconfiguration of my setup. But, as I’ve mentioned before, I really like rebuilding my tools up from scratch! It's more work but it helps remove old cruft and forces you to really understand how your systems work.

I took some notes so I wouldn't forget what I was doing and in the end I had enough for a quick post. These notes are more of a road map than a detailed guide. I figured between existing how-to guides, forums, and LLMs you should be able to work through any system-specific issues you run into. Good luck!

The Task: I wanted to upgrade my Rasbian OS version (now Raspberry OS) from one based on Debian 10 (”Buster”, released in 2019) to one based on Debian 12 (“Bookworm”, released in 2023). This will wipe my system so I will need to setup ad blocking on the Pi, PADD for visualization, and point my router at the Pi for DNS resolution to complete the process.

My Setup: I’m running a 4 year old CanaKit Raspberry Pi 4 with a 7” touchscreen display. I basically only use the system to run Pi-Hole so I run a distro without a desktop and set it to boot to the PADD utility which gives me a nice ASCII “dashboard” view of the network and system activity.

Every time I walk by my setup in the basement "it sparks joy" as Marie Kondo would say.

Raspberry OS Upgrade: Upgrading the OS requires etching a new ISO image on the MicroSD card. There are very clear warnings on the Raspberry OS site not to try and force an upgrade between versions from the CLI. I of course ignored the warnings and just tried to upgrade directly from CLI — and broke my install. ¯\_(ツ)_/¯

For the new OS I used the 64-bit version of Raspberry OS Lite, which does not include the desktop. Using the Pi-specific disk image etcher from Raspberry Pi you need to enable SSH for remote access and you should pick a non-default a username and password to keep the script kiddies out. From here boot from the new disk image and do the usual software updates with apt.

SSH Access: If your machine now has a new IP address the SSH fingerprint will have changed and your next attempt to SSH will fail. Instead you'll get an error that starts with WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED and contains a warning about a possible man-in-the-middle attack. If you don’t know what this means it’s worth a few minutes of reading to help understand some of the networking steps later on.

You can fix this SSH issue either by just blowing away the old key with something like this:

>  ssh-keygen -R raspberrypi

or you could also just reassign the old static IP address back to your Pi in your router's admin panel.

Pi-Hole and PADD: Now we’ve SSH’ed into the system and we can set up the ad blocker. Both Pi-Hole and PADD install easily with curl commands following the instructions in their docs. Next update the .bashrc script to run PADD on login using the instructions in the repo README. Finally, configure the Pi to automatically boot into your user which you can do with the raspi-config CLI.

Network Configuration: The last step is to configure your router to use the pi-hole as a DNS resolver and block any black-listed ad-related URLS. To do this you use your the DHCP section of your router admin panel to assign a static IP to the Raspberry Pi‘s MAC address. You then use that static IP address as the IP for the router's primary DNS resolver.

If DNS and network configuration is new to you, you can ask your friendly local LLM or if you want a deeper dive I highly recommend Julia Evan's wonderful newsletters and zines on how DNS works.

End Result: You might have to reboot some combination of the router, Pi, and Pi-Hole to get everything to link up but now you should be surfing the internet with few if any 3rd party ads.

Looking at the PADD display I’m usually blocking 10-30% of DNS requests!

2024-11-11 Weekly Update - Python, Inverse Machine Learning, Set Cardinality

Mon, 11 Nov 2024 00:00:00 GMT

I took the last few weeks off of blogging to focus on some other projects. The highlight was a demo at the local AI Tinkerers meetup of the AI medical document summarization tool I've been helping to build. This week, I’m back to blogging and I’ve got a blend of programming, machine learning, notes to share.

This is a longer post, so if you need some music settle into anything by Linenoise.io.

Python Structural Pattern Matching

When I first learned about structural pattern matching in Python (i.e. match and case keywords) I glossed over it as equivalent to if/else statements. After reading this great article from Real Python (more than once) I see there’s a lot more depth here. In particular I learned about deconstruction in pattern matching (similar to *args), conditional pattern execution, and applications to complex nested data structures. I’m not sure how often I would reach for this in production but it’s definitely something I want to experiment with in the future.

Logging for Python Packages

The Python logging module is not the easiest API to grok, but it does have a lot of useful properties once you understand how to hold the tool. It’s also one of those tools where I’ve inherited a logging setup more than I’ve had to set it up myself.

So this week I spent some time digging best practices are for instrumenting logging. The Python standard library docs are pretty rich in examples, patterns, and anti-patterns for logging. I learned a lot of things but my biggest “ah-ha moment” was understanding that logs inherit configurations (e.g. a handler) on import from whatever configuration is executed first. That means that you need to align your logger configurations with the entry point(s) to your code. So for example maybe you define a logger on your main pipeline entry and another in the module’s if __name__ == ”__main__”: block if it can be run as a stand-alone script.

UV in Docker

Like most people in the Python ecosystem uv is my new go-to tool for Python dependency and environment management. First it replaced Poetry for me, and then pyenv, and so now I want to understand how to effectively use it in a Docker container. Astral’s example project as well as this article by Hynek Schlawack are a great place to get started. But knowing my interest in fundamentals I can tell this is going to lead me to a deeper dive on Docker.

While you’re at it you should also check out this uv cheatsheet which translate commands from other tools (pip, vent, poetry) to their equivalent in uv. This framing helped some of the new commands “click” for me.

Inverse Reinforcement learning

I was talking to a local founder and they mentioned “inverse reinforcement learning” (IRL). I had spent some time this year working (unsuccessfully) on reproducing an experiment for Sutton’s “Reinforcement Learning” textbook so I felt (barely) confident enough to ask them to explain IRL. Fortunately, I was able to get it right away — studying the fundamentals, even briefly, always pays off!

In traditional reinforcement learning you are trying to optimize agent behavior to optimize a known reward function, for example in a multi-armed bandit problem. In IRL it's the inverse, you are given agent behaviors and instead have to find the reward function that best explains the agent behaviors. This article has a good summary as well as citations to foundational papers in the area.

Cantor’s Theory of Cardinality

I’m enjoying this second lecture from the MIT open courseware on analysis. This lecture is about using bijections to compare the cardinality of sets, specifically infinite sets. It concluded with Cantor’s proof that there are more real numbers than integers. This builds nicely on the proof work I’ve been doing all year out of “The Book of Proof”, enough so that I was able to follow along while cooking dinner.

Not sure other folks would enjoy this lecture, if you already know the material it’s probably too slow and if you’re not familiar with it (or planning to put in some serious study) it’s probably too fast. But it’s a well-done resource worth highlighting. Free MIT math content online -- what a time to be alive!

Odds and Ends

Updates on some books and audiobooks:

I finished “The MANIAC” audiobook which I thought was fantastic. I especially loved the dark overtones and the bridge to AlphaGo in the last section of the book. It makes a great compliment to the more neutral biography “The Man from the Future: The Visionary Life of John von Neumann” which I listened to earlier this year.
I’m now listening to “A Mind At Play: How Claude Shannon Invented The Information Age”, which is more, maybe I’d call it folksy in tone than ”The MANIC”. It gives a lot of insight into the transition in Electrical Engineering from essentially tinkerers and craftsman to the modern practice which is so intertwined with physics and mathematics.
I’m trying to read “The Missing Billionaires” which is essentially a book-long study of the application of the Kelley Criterion as a risk management tool. I’m making glacial progress (a few pages as I nod off at night) but the book is great if you’re interested in something partway between quantitative finance and personal investing advice.

2024-10-07 Weekly Update - Levenshtein, Hamming, and Jaccard Distances

Mon, 07 Oct 2024 00:00:00 GMT

This week as part of my consulting work I’ve been working on evaluation metrics for LLM-powered text extraction. Essentially, trying to quantify if the LLM grabbed the right text from a document. This means getting hands-on with some common text comparison algorithms. These were largely already familiar to me but I hadn't had a change to use most of them on a real engineering problem before. I collected my notes and turned them into this week's post.

If you need something to listen to while you read may I suggest "Something Will Happen" by house/jazz artist Beloiz. Come for the lush melodies, stay for the encouraging voice-overs by Willem Dafoe.

String Metrics

Reference: https://en.wikipedia.org/wiki/String_metric

Formally, all the string metrics measures here are proper metric spaces in the mathematical sense. A metric space is a set $M$ together with a notion of distance $d$ between set elements that must satisfy the following four conditions:

Self-Distance is Zero: Distance from a point to itself is zero: $d(x,x) = 0$
Positivity: The distance function always returns a positive value for any two distinct points: $x \neq y \Rightarrow d(x,y) > 0$
Symmetry: The distance function is symmetric: $d(x,y) = d(y,x)$
The Triangle Inequality: The following property holds: $d(x,z) \leq d(x,y) + d(y,z)$

Levenshtein Distance

Reference: https://en.wikipedia.org/wiki/Levenshtein_distance

The Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other. For example:

L(bat, cat) = 1

bat : 0
cat : 1 (b -> c)

L(bat, egg) = 3

bat : 0
eat : 1 (b -> e)
egt : 2 (a -> g)
egg : 3 (T -> g)

The Levenshtein distance can be normalized by dividing by the length of the longer of the two text strings. This results in a range of $[0,1]$.

Hamming Distance

Reference: https://en.wikipedia.org/wiki/Hamming_distance

Unlike the Levenshtein distance the Hamming distance is only used to compare strings of the same length. As a result it only measure the number of substitutions. So it is the number of positions at which the two strings are different.

Similar to the Levenshtein distance, the Hamming distance can be normalized by dividing by the length of the strings.

Jaccard Distance

Reference: https://en.wikipedia.org/wiki/Jaccard_index

The Jaccard distance is the complement of the Jaccard coefficient (or index), that is, $1 - J(A,B)$. Where the Jaccard coefficient is the ratio of the intersection of two sets over their union:

$$ J(A,B) = \frac{|{A \cap B}|}{|A \cup B|} $$

Note that the Jaccard coefficient does not define a metric space because $J(A,A) = 1$ not $0$ which is why we subtract $1$ to get the Jaccard distance. For text comparisons set elements can be defined in different ways leading to different results. For example, by characters, by words, or by n-grams.

Odds and Ends

Here are some other things that have caught my interest this week.

Python 3.13 features: Python 3.13 was released on 2024-10-08. I recently wrote about some new (to me) Python features I had missed so I was curious to check out what’s new in this release. I found a nice overview on this Real Python Podcast episode which includes links to all the relevant feature PEPs.
Inline Functions: In another call-back to a recent post I was talking about “taste” in programming, which I loosely defined as being something a little more granular than design patterns. This post on inline functions from John Carmack (and the associated Hacker News discussion) on when to be inconsistent is exactly the kind of programming nuance I’ve been thinking about.
The MANIAC: I’ve been working on this audiobook about the life and work of John von Neumann. Written in a semi-fictional voice, the book covers everything from his work on the incompleteness theorem with Godel through the impact of his work on recent Nobel Prize winner Demis Hassabis and Lee Sedol's legendary Go match against AlphaGo. It's a dark, almost creepy, read on technological progress but I'm still enjoying it.

2024-09-30 Weekly Update - Cleaning git history, blog meta-updates, and new developer line tools

Mon, 30 Sep 2024 00:00:00 GMT

This week's update is again focused on technical content. I've written up some notes on clearing your git history, some updates to this blog, and some new tools I've started using in my daily workflow. I hope you enjoy it and reach out if you want to chat about anything I've written.

Removing Sensitive Data From Git

The other day I was working with a consulting client and trying to live-fix a bug on a call. I was going too fast and I ended up accidentally committing an API key to our repo and pushing the change. Fortunately, we're the only two people who have access to the repo, but I still wanted to properly scrub it from the repo.

I haven't had to scrub a git history for a private key in a few years. Since then, the recommended tool for cleaning up a git history has gone from git-filter-branch to git-filter-repo. I had a pretty easy time finding suggestions on how to delete a word/file from the history but I had a harder time finding a command to verify that the text was removed.

I've captured both in here and subbed <REDACTED> for the sensitive text.

# Check that the API key is in fact in the repo history
bash-3.2$ git log -S <REDACTED> main --name-only --pretty=format: | sort -u
template_notebook.ipynb

# Replace all references to the API key in all commits to ADD_API_KEY
bash-3.2$ uv run git-filter-repo --force --replace-text <(echo "<REDACTED>==>ADD_API_KEY")
Parsed 22 commits
New history written in 0.03 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at 27440ad Replaced unmaintained pdfminer with pdfminer.six (#11)
Enumerating objects: 76, done.
Counting objects: 100% (76/76), done.
Delta compression using up to 10 threads
Compressing objects: 100% (35/35), done.
Writing objects: 100% (76/76), done.
Total 76 (delta 42), reused 69 (delta 39), pack-reused 0 (from 0)
Completely finished after 0.10 seconds.

# Check that the API references are gone
bash-3.2$ git log -S "<REDACTED>" main --name-only --pretty=format: | sort -u

# Confirm that they have been replaced by ADD_API_KEY
bash-3.2$ git log -S "ADD_API_KEY" main --name-only --pretty=format: | sort -u
template_notebook.ipynb

# I'm always scared when I run this one ...
bash-3.2$ git push origin -f main:main
Enumerating objects: 76, done.
Counting objects: 100% (76/76), done.
Delta compression using up to 10 threads
Compressing objects: 100% (32/32), done.
Writing objects: 100% (76/76), 90.13 KiB | 90.13 MiB/s, done.
Total 76 (delta 42), reused 76 (delta 42), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (42/42), done.
To <REDACTED>
 + 4da8e1f...27440ad main -> main (forced update)

Hopefully this is not something I need to do for a few more years but I figured it was worth capturing it in some notes for myself and others.

Blog Updates

Version Update: I updated my blog dependencies including bumping the blog framework to Nextra v3.0. This required massaging some configuration files but overall was not too bad. Most importantly, it seems to have fixed a bug where some new posts were not being added to the main post feed.

Weekly Updates Titles: Now that I have 2 months of weekly notes I realized that I need more descriptive titles for my posts than just the date. I've added more description to the titles and you can now find all those posts under the weekly-update tag.

New (To Me) Developer Tools

At the start of this sabbatical I set up a new laptop and I used that as an excuse to try out some new developer tools. As a result, I ended up replacing a surprising number tools that were previously daily-drivers for me. Here's what's new to me:

Category	Old	New	Notes
Terminal	iTerm2	Kitty	GPU acceleration for LazyVim
Terminal Prompt	Tide	None	Never set it up and didn't miss it
IDE	SublimeText	LazyVim	Trying to live more in the terminal
Theme	Dracula	TokyoNight	Time for an aesthetic change
Git	CLI	LazyGit	Great UI and terminal-based
Python Projects	Poetry	Astral uv	The solution we've been waiting for

One of the themes in my new choices is working more in the terminal. You can see that in my moving from Sublime Text to LazyVim. That prompted me to move from iTerm2 to Kitty for performance reasons. I still love the Fish shell but went for a more stripped back prompt over my previous powerline9000 inspired configuration. Lastly, like everyone else in the Python ecosystem, I've been joining the Astral bandwagon and moving my workflow over to uv from Poetry (and pipenv before that). I suspect I'll migrate off of pyenv in a few weeks as I get more comfortable with uv.

Updates like this feel like a "spring cleaning" of my tech stack and are one of the reasons that I don't backup my config/dotfiles between machines. First of all, I don't tend to use tool that require a ton of configuration out of the box. But more to the point, having to configure each of my machines separately helps me discover new tools, drop tools that I don't miss, and makes sure I really understand my tools well enough to troubleshoot them.

2024-09-23 Weekly Update - Python syntax, Obsidian dataview, vim, and Python notebooks

Mon, 23 Sep 2024 00:00:00 GMT

I didn't get an update out last week because I've been heads down on some projects. But, more time working on code while practicing evergreen note taking means a lot of ready-to-go content for weekly updates.

Easier Blogging with Obsidian's Dataview

I've been using Obsidian as my note-taking app for about a year now. I finally have so much content now that staying on top of all my threads of thinking is becoming challenging. To help with that I've started using the popular dataview plug-in to generate indexes of files for different projects.

Obsidian files are fundamentally and transparently Markdown files and dataview lets you query your file metadata using a SQL-ish dialect. For example, this dataview query I figured out this week returns a Markdown formatted list of all the Obsidian files you've edited in the last week.

LIST
WHERE file.mtime >= date(today) - dur(1 week)

This is useful because looking at all the files I modified in a week is great hack to source content for these weekly updates.

New (To Me) Python Syntax

This week I came across two Python expressions I hadn't seen before, though both are several years old. The first is a shorthand to add dictionaries, the second is a way of marking positional-only arguments in a function.

Dictionary Addition

The pipe operator can be used to combine to dictionaries | like this:

>>> a = {"this": "that"}

>>> b = {"foo": "bar"}

>>> # Regular addition with + doesn't work
... a + b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'dict' and 'dict'

>>> # Use | instead
... a | b
{'this': 'that', 'foo': 'bar'}

This was introduced in PEP-584 - Add Union Operators to dict and released in Python 3.9. Previously you would have had to do something like {**a, **b} (assuming you don't want to modify either dictionary).

Restricting Positional-Only Function Arguments

The second syntax is that you can use the slash symbol / to denote function variables that must be positional-only as opposed to keyword arguments. Here's an example.

>>> def a(arg_1, /, arg_2):
...     print(arg_1, arg_2)
...

>>> a("this", "that")
this that

>>> # This is fine
... a("this", arg_2="that")
this that

>>> # This is not
... a(arg_1="this", arg_2="that")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: a() got some positional-only arguments passed as keyword arguments: 'arg_1'

This was introduced in Python 3.8 back in 2009 along with the "walrus" operator (:=).

These update got me thinking about my first Python version. I've been using Python since 2008 which would put me at about Python 2.5 as my first version. It's pretty neat to watch the language continue to evolve and change after 16 years, even if I am late to the party sometimes.

My First LazyVim Configuration File

During this sabbatical I've been seriously trying to use Vim, more precisely NeoVim (even more precisely LazyVim) as my main text editor and IDE. My big jump this week is that I finally got comfortable enough with my setup to create a configuration file. To LazyVim's credit, the defaults are so sane that I really didn't have much motivation to change anything for months.

Configuration files in LazyVim are fundamentally Lua functions that return maps of the configuration settings. LazyVim's functionality is built around an ecosystem of plugins so to configure your settings you need to know 1) which plugin is controlling the setting you want to configure and then 2) the configuration schema for that particular package.

This is all kind of a steep learning curve but really not that bad once you get the hang of it. For example, here is the config file I wrote to to display hidden dot files in Neotree, the sidebar tree view plugin.

return {
  "nvim-neo-tree/neo-tree.nvim",
  opts = {
    filesystem = {
      filtered_items = {
        visible = true,
        show_hidden_count = true,
        hide_dotfiles = false,
        hide_gitignored = true,
        hide_by_name = {
          -- '.git',
          -- '.DS_Store',
          -- 'thumbs.db',
        },
        never_show = {},
      },
    },
  },
}

One of the things I had to figure out is that, unlike a Python module, the file name isn't special. LazyVim is opening all the Lua files in the config search path and loading all the configurations they return.

Using Editable Modules in Jupyter Notebooks

A somewhat eccentric development pattern I often find myself using is editing a Python module while also importing and using that same module in an Jupyter notebook.

To me, this allows you to have the data exploration tools of a notebook along with the best-practice and tooling of a normal Python project. So as I'm working on a data project I might start migrating functions out of my notebook into my module when they feel done. This forces me to build useful pipelines of code but still allows me an interactive environment to explore the data outputs I'm producing.

To work in this way you need to tell the Jupyter server to refresh the already loaded module. If you're just doing this once you can just restart the notebook server. But if you're doing it constantly you can set a flag to force module reloading. This is one of those syntaxes I have to look up every year and just have to throw at the wall for a bit until it works so I finally decided to take some notes.

# Loads the `autoreload` extension
%load_ext autoreload

# Reload all modules imported with `%aimport` every time before executing the Python code typed. Same as `%autoreload 1`
%autoreload explicit

# Reload all modules (except those excluded by `%aimport`) every time before executing the Python code typed. Same as `%autoreload 2`
%autoreload all

# List modules which are to be automatically imported or not to be imported.
%aimport

# Import modules ‘foo’, ‘bar’ and mark them to be autoreloaded for `%autoreload 1`
%aimport foo, bar

I think this development pattern should manage to upset everyone; data scientists won't want the hassle of pushing data down to a module and developers won't want to use a notebook environment. But to me, it's the best of both worlds.

2024-09-09 Weekly Update - Evergreen notes, LLMs, programming taste

Mon, 09 Sep 2024 00:00:00 GMT

I feel like this weekly update experiment is coming along. Even though I'm a little late on this week's update I still feel:

motivated to share
able to knock out some quick notes
without being a perfectionist
confident enough to share

In part this is from the positive encouragement I receive every week - thank you!

If you need a soundtrack while you read, may I recommend "Obra Akyedzi" an upbeat afrobeat track by Ghanaian Ebo Taylor with production by Ali Shaheed Muhammad (A Tribe Called Quest).

Here's what I've been up to.

Evergreen Notes

As I mentioned last week I've been interested in Andy Matuschak's concept of "Evergreen Notes". In short, he feels notes should be:

atomic
concept-oriented
densely linked
ontological
written for yourself first

I've tried to put this into practice by regularly refining my talking points for my job interviews. It's still early, but I think I like this approach. It feels more like refining my understanding of discrete topics and not just dumping information into yet another unlinked and unread note.

LLMs and Pipelines

My main focus this week has been helping a local founder with creating a data pipeline for an LLM-based medical document summarization tool. As I mentioned last week, I'm using ChatGTP, Python, and the Instructor library for most of this work. What's most striking to me so far about working with LLMs is how ... familiar it all feels.

While the iteration process of prompt engineering is novel, much of the scaffolding for the project centers around familiar data engineering problems. For example, I want to make it really easy for us to run experiments and understand the results. Of course, the LLM is doing incredible and novel work, but most of the work to enable that is familiar data pipelining. This means I spend my time thinking about data schemas, validation and serialization, and function parameters.

Which leads me to my next point.

Taste in Programming

Last week this article was making the rounds on HackerNews by GitHub Co-founder Scott Chacon. He argued that part GitHub "won" because:

It was at the right place at the right time
it had good taste

You can disagree with that conclusion but I still think it's important to develop your personal sense of programming "taste". By taste I don't quite mean programming techniques like TDD or language features like types. I mean something closer to your personal definition of "good".

Lately I've been thinking more about my own taste in programming. In part, this is because I'm writing more code that usual. More specifically, this LLM project has given me a playground where I can both set the style guide and have complete architectural control. At the same time, unlike a toy project, the codebase is starting to get big enough that I'm starting to notice my own inconsistencies and eccentricities. Things like variable names, function scope, abstractions, etc.

There are a lot of best practices and paradigms to choose from, but there's also a certain element of taste at play. This Medium article on "Data pipeline recipes in Python" gives a good sense of some of the design decisions I'm talking about. But most helpful reference I have here is actually not a technical one.

Instead, I've found myself thinking about the men's fashion commentator Derek Guy (aka @DieWorkWear). One the points he continuously makes is that "good taste" is about well-executed coherent ideas. If you want to look like a cowboy, or a 90's rapper, or landed gentry then your outfit should coherently convey that. (Of course you can choose the break all the rules, but this only tends to go well when you intimately understand the rules you are breaking)

I think it's the same with code, you want coherent ideas well executed. As my codebase grows, I find myself asking more and more "why am I doing this?" and maybe more importantly "is it consistent with my other decisions in this codebase?".

Odds and Ends

As I mentioned above Derek Guy (aka @DieWorkWear) is obviously a great follow if you have any interest in men's fashion. But his writing is also an overall excellent example of analysis, especially in how he combines technical details ("this is how a sport coat is constructed") with big-picture framing ("this is the purpose of a sport coat").
Here's another Hacker News post from last month I enjoyed, "Markov chains are Funnier than LLMs". I enjoyed this in part because I enjoy meta-analysis of comedy (I'm really fun at parties). But, I also enjoyed the technical point that Markov chains produce distributions of outcomes (the edges of which an be inadvertently funny) but LLMs only produce the mean outcome (which is not funny).

2024-09-06 Weekly Update - Structured outputs for LLMs and proving the binomal theorem

Fri, 06 Sep 2024 00:00:00 GMT

This is the summary of what I was up to in the 10th week of my sabbatical. This write-up focuses on math proofs, working with LLMs, and wraps up with some odds and ends. A small update, my work on a web scrapper for a friend's company last week helped them close a customer deal!

I continue to really enjoy connecting with everyone who reaches out after reading these posts. I'm talking to folks about everything from career advice, open roles, kicking around data problems, and just catching up. If you're on the fence, please reach out, I would love to hear from you!

If you need a soundtrack to read to, let me suggest "Glass, Concrete & Stone" by David Byrne. You might recognize it from the the 3rd season of "The Bear".

Proving The Binomial Theorem

I finally gave in and looked up the answer to the proof of the binomial theorem I was working on in Hammack's "The Book of Proof". I had the right set-up but never cracked the middle step. But even after looking at the answer I wasn't following how to make the jump.

Because this is a famous proof, I felt confident asking ChatGPT for help. Still, I was genuinely surprised. Not only did ChatGTP produce what appears to be a correct proof (as far as I can tell), when asked it was also able to specifically expand on the step that was confusing me. The trick was connecting the $S_k$ and $S_{k+1}$ statements with a substitution of the form $j = k + 1$. This is is a recurring technique in these induction proofs. In this case, it allows you to change the indexes on a sum and use an identity of the binomial coefficient. You can find the transcript of the chat here.

If you're interested in LLMs and their ability to work on novel proofs (not just repeat famous ones) you should check out this recent talk by Terence Tao on AI and Mathematics.

The Opportunity Cost of Pure Math

I'm glad I now understand this proof, but I go back and forth on if this level of rigor and completeness is a good use of my time. Should I be putting this much effort into being able to solve these proofs? Or should I be trying to get to "good enough" so I can cover more material? It would probably be more helpful in my career to be working on something more applied like standard data science or statistics techniques. And it would be more enjoyable to "fast forward" to more abstract and complex ideas like vector spaces.

There is an opportunity costs to everything we do. Especially as I get older and my family life is changing, time is more precious than ever. And there is also a part of me that is embarrassed that math doesn't come easier to me and that my progress is so gradual. But at the end of the day, I enjoy the work and I feel much "stronger" mathematically than I did 2 years ago.

In practice, I do think I could be better about capping the amount of time I spend working on any one problem before getting help. I have to remind myself the point is to learn, not have a stroke of genius or reinvent math on my own.

LLMs for Document Summarization

My big project this week has been helping a pre-seed founder with some prompt engineering for an LLM-based medical summarization tool. The first thing I added to the project was to start treating each LLM run as an experiment. This means that for every run we need to store not only the results but all the metadata needed to recreate the results. This means data like the model, the data inputs, the prompts, etc.

Once I had that output format down I tried to compare a few results. It only took me a few minutes to realize we needed structured LLM outputs! Fortunately, I had been keeping up with LLM model tools and I know that the Instructor package, built on top of Pydantic, was the go-to solution for this. To quote Jason Liu, the creator of Instructor, "Pydantic is all you need".

I'll give a quick overview of structured outputs and why they're useful. Let's use dentistry as an example. Your typical unstructured text output from a generative LLM might look like this:

The patient came in on Nov. 3rd for a cleaning and we found one cavity.

The point here isn't if the summary is right or wrong, the point is that because this is an unstructured/human-readable response it's not clear how to we would assess the accuracy of this output at scale. For example, if we had 1k records and 5 different prompts we wanted to try, how would be able to parse and compare all those outputs?

We can get a lot closer to a scalable solution just by enforcing a structured response. For example, the above response could become:

{
 "proceedure_date": "2022-11-03",
 "proceedure_type": "cleaning",
 "has_cavity": true 
}

This model is something we could build automated testing and scoring around! There's still a lot of complexity to be addressed but this framework gives us something stable to build against.

Odds and Ends

The web scrapper I worked on last week helped my friend's company win a customer by creating a better search index of the customer content.
I updated the node dependency on this blog which fixed some issues with recent posts appearing in the blog index (PR).
I started actually saving more bookmarks to the my DuckDB bookmark app. Here are some of my favorites from this week:
- LazyVim for Ambitious Developers - The missing NeoVim/LazyVim guide I've been looking for!
- Office Hours with a Geometric Group Theorist - Content like this reminds me why I'm working so hard on my proofs.
- Andy Matuschak on Evergreen Notes - Useful advice on how to take notes that are actually useful for building understanding.

2024-08-30 Weekly Update - Web scrapting and prompt engineering

Fri, 30 Aug 2024 00:00:00 GMT

This is my 3rd weekly sabbatical summary and I'm happy with how this series is shaping up. Writing this week's post started to feel more like the quick and direct process I'm working towards. As with past weeks, I continue to be pleasantly surprised by the amount of engagement these posts generate. This week's post is focused on some advising work I've been doing with some other odds and ends thrown in to round things out. I hope you enjoy it.

If you need a soundtrack while you read, Chance the Rapper dropped one more summer single before Labor Day.

Consulting

My most exiting update this week is that I got to do some hands-on work for two companies. I've been advising to 1-2 companies or investors a week during my sabbatical but this is the first time I gotten to help out with a technical problem. The first was a web scraper and the second is an LLM application.

Web Scraping: The first project was a one-off web scraper for a friend's company to help them gather content for a customer demo. I began my startup career working for a security company that ran a large dark web scraper and it was fun to return to that problem space for a few hours. I originally started the project in Scrapy which is a nice structured crawling and scraping framework. But Scrapy started to feel like overkill for this task and so I wrote a simpler script using just requests and Beautiful Soup. This was pretty quick work and ChatGTP helped out a lot by generating the HTML tag selectors I needed.

Prompt engineering: The second project I'm working on is an on-going effort to help a pre-seed founder with some generative AI prompt engineering and QA validation. The use case is medical document summarization. This is a really interesting project for me. While I've been using generative AI both locally and via ChatGPT for a nearly a year, this is my first time doing so for a commercial use-case. I'm still in the early stages but already learning a lot.

Math and CS

I didn't want to completely skip working on any math this week so I'm working on an induction proof of the Binomial Theorem. I think I have the shape of the proof but I just need to figure out how to connect the dots. As always, I struggle with deciding when I've reached the point of diminished returns of working on my own vs learning from the answer. Whenever I do wrap this up, the next problem starts a series of Fibonacci related questions which I'm looking forward to.

I didn't get to any LeetCode this week but that was the trade off for doing more software projects than usual.

Odds and Ends

Bookmarking Tool: I made a little bit of progress on my bookmark tool project, mostly small updates to the queries and started pushing data up to MotherDuck. This is basically at the point I could start using it to track links.

Neovim: I've been using Neovim casually for most of 2024. But, as I've been telling folks online, watching me use Neovim is like watching a caveman trying to rub two Teslas together to make fire (the follow-up joke writes itself). I'm slowly getting better at learning the key mappings and installing the extensions I need but what I really need to do is rip off the band-aid and start customizing my setup. Soon!

Blog Framework: I've been using my current blog setup for about 2 years now. Which, if you have a blog, you know that means I'm getting the itch to rip it out and start over (and then blog about it). So I've spent a little time trying out different frameworks to see what else is out there. TBD, if I'll actually go through with this!

That's it! Thank you for reading and if you're on the fence about reaching out for any reason, please I'd love to hear from you!

2024-08-23 Weekly Update - More math proofs, LeetCode, DuckDB, and networking

Fri, 23 Aug 2024 00:00:00 GMT

... the most important thing is to have fun!

- Reader feedback on my last post

Unpopular opinion I agree with- learning should not be fun.

- Andrej Karpathy Twitter

Last week I wrote my first weekly update. My only goal was to write something up without belaboring it too much. I was pleasantly surprised to get a wide range of positive feedback from readers (I have readers!?!).

This included encouragement to continue writing from colleagues ranging from investors to former coworkers. I got a thank you from someone I gave helpful career advice to years ago at a conference, connected with someone looking for career advice, and got a couple of intros of folks in my network. I'm humbled and appreciative for all the engagement.

This week, my goals is to see if I can start to make a habit of these updates while I'm on sabbatical. I'm hoping they get easier each week as I build on what I've written before.

This week I'll be talking about more math proofs, some LeetCode, my bookmark tool side project, some networking events, and my job search. If you need a soundtrack while you read, may I suggest "Retreat" a 2023 jazz/funk scorcher by Chicago-based saxophonist Isaiah Collier.

Math

One of the trade-offs I made this week is that I didn't put much time into math work.

I wrapped up the proof I was working on last week which asks you to show that the harmonic series, summed up to $1/2^n \geq 1 + n/2$. This in effect shows that the Harmonic series is unbounded. I got reasonably close to the solution on my own but was missing the final technique to complete the proof. Once I read the answer I got it, but I spent a little more time with the proof until I felt like the missing technique was a logical conclusion and not a flash of insight. This problem in particular, because it's about proving a bound, is a nice set up for the theory of calculus chapters which make up the last 3rd of the book. If you're interested, I was able to eventually nudge ChatGPT to the correct proof.

The other bit of math study I did was wrap up my notes and a few of the problems from Wasserman's "All of Statistics". This included finishing up a proof on the continuity of probabilities (again, ChatGPT can walk you through the proof). Working through this book feels like a bit more of a grind. I'm not sure if it's the subject matter or the book but I'm on the fence on if I'm going to continue working on this right now.

Programming

I did some more LeetCode problems this week and focused on two areas. The first was going back and doing some "medium" difficultly problems in string manipulation and the other one was working through more "easy" dynamic programming problems.

I'm getting better at solving these problems but they're taking more time because I'm now more interested in making sure I really understand all the standard CS techniques. Usually this means doing a problem 2 or 3 times to make sure I can get all the memory and runtime optimizations. This is hard, but it feels like I'm getting the "point" of each exercise.

Just like with my math proofs I'm getting much better at efficiently learning and not just beating my head against the wall when I'm stuck. What that looks like for me is aggressively cutting the problem down until I have identified a missing technique, then checking the answers to find that technique, then trying to apply it from memory.

To be clear, I still get very frustrated when I get stuck! But time boxing myself, taking breaks, and focusing on the big picture helps me keep my cool.

Bookmark Thing

I made a little more progress on my eloquently named bookmark thing side project. This started as an excuse to play with DuckDB some more while tracking interesting links. Because DuckDB is an OLAP database I wanted to see if I could just throw everything into one flat denormalized table.

I wanted to use some of the list parsing functions to modify the tag list associated with each entry. This was working fine until I tried to perform an update on a tag list and started getting unexpected constraint errors. After some digging I realized that currently duckdb treats list updates as an insert followed by an update, which is known to cause these types of errors. If you're interested you can check out the relevant GitHub issue and the documentation explanation.

Instead of trying to figure out a work around (such as handling this in the application code) I decided to just revert to a typical normalized database model and build a denormalized view on top of that.

This new schema has the drawback this database is now more of an OLTP workload, instead of the OLAP workload DuckDB is optimized for. But that's fine, I'm still getting more exposure to DuckDB. I have some ideas for other datasets and other projects I could try that are a better fit for DuckDB's strengths.

I now have working versions of my schema creation as well as the CRUD queries I want to run. I'm eager to get this wrapped up in a Typer CLI interface but I think next week I'll be interacting with my code in SQL while moving it to MotherDuck, DuckDB's hosted offering.

Overall, this project is reaffirming my belief in aggressively using your prototypes as much as possible before writing additional code.

Odds and ends

I finally updated the Twitter card template from the blog default.
I really feel like I'm starting to hit my stride with using ChatGTP to work on what I saw Charlie Marsh call "side quests" - stuff like the changing the Twitter card. I've been squashing tons of small annoying bugs and warnings this way and it's starting to become my go-to.
Lastly been listening to Martin Kleppmann's excellent "Designing Data-Intensive Applications". I finished up the chapters on database replication and partitioning and started the chapter on transactions. I'm learning a lot more than I thought I would by listening to a technical book! That being said, I know I'm only retaining a fraction of the material, but it's more than the 0% I would get by just looking at the book on my shelf and thinking "I should read that".

Networking

This week I started putting more time into networking and exploring job opportunities. In particular going to in-person events and sending out more applications.

I went to two in-person events last week. The first was the monthly meetup of the Chicago AI Tinkerers. This was an awesome event hosted by Drive Capital. I saw some cool demos and made a couple of great connections. The second was a meta-meetup of tech organizations at mHUB, one of our local hard-tech innovation spaces. This was organized around the DNC convention happening downtown and was intended to be a showcase of the Chicago hard tech ecosystem. I got to meet companies working on sharing commercial import/export data, infrastructure for recycling, energy optimization platforms, and even other hard-tech incubators. I also met one of my "neighbors" in my coworking space and got to learn about what they're working on.

Lastly, after talking to so many folks over the last few weeks, I finally feel like I'm at a point where I'm ready to start sending out general applications to job openings. I think my ideal job still rhymes with "I'll know it when I see it", but I feel like I'll be able to spot it and jump on it.

Again, if you want to chat about opportunities or talk shop, please reach out.

2024-08-16 Weekly Update - math proofs, LeetCode, DuckDB, and networking

Fri, 16 Aug 2024 00:00:00 GMT

Like most bloggers, I have a long backlog of unfinished posts. I aspire to write concise well-scoped ideas that I can quickly get out the door. But I tend to reverted to my natural discursive writing style and completionist tendencies which makes things hard to finish.

To work on that I'm trying a "weekly update" format. I'm drawing direct inspiration from Simon Willison's "weeknotes" as well as John D. Cook's short-form technical blogging.

This is my first attempt at this format and it's been a success in that I actually got something published! I hope you enjoy it. If you need a soundtrack while you read, may I suggest Four Tet's latest album Three +

I'm on Sabbatical

My biggest update is that I've left my former job and I'm about 6 weeks into a self-proclaimed sabbatical. While I'm looking for my next role I've been focusing my energy on "sharpening my tools" by studying math and programming fundamentals (phrasing taken from this tweet), networking with interesting folks, and just generally resetting and refocusing. Here are some examples of what's been interesting me.

Mathematical Induction Proofs

For nearly 2 years now I've made a concerted effort to study math again. I spent 2023 working through Sheldon Axler's modestly titled "Linear Algebra Done Right (4th Edition)". The text is an algebraic approach to linear algebra that largely eschews the use of the determinant. I only got a few chapters in but the material was eye-opening. But, as I worked through the book I noticed I was getting too bogged down in the mechanics of completing the proofs and I was losing momentum.

So this year I took a step "back" and have been working through John Hammack's "The Book of Proof (3rd Edition)". This book focuses on teaching the mechanics of proofs using a range of examples from different areas of math. This book is exactly what I needed as an undergrad math major. I've been I've gotten a lot more comfortable with the tools needed to work on the math I'm interested in learning (I even submitted a small typo!).

Currently, I'm working through the chapter on mathematical induction. Most the problems so far have been about elementary number theory or series. I got stuck this week on a problem that implies the divergence of the Harmonic Series. But, I think I've just about unwound my confusion. The rest of the chapter has problems on the Fibonacci sequence, Pascal's triangle, and graph theory.

CS Fundamentals with LeetCode

In keeping with my interest in learning fundamentals, I've also been working through basic CS problems on LeetCode, e.g. linked lists. Because I don't have a CS degree there are a lot of fundamental CS problems I've only read about. While I conceptually understand them I've never worked through them in practice. I used to shrug these off as being academic problems, and maybe they are, but I've had a surprising amount of fun solving them. I feel like I'm solving chess puzzles (something I enjoy) but I also feel like I'm closing some important gaps in my knowledge.

This week I've been working on Binary Search Trees and Binary Search. Coincidentally, this involves a lot of recursion which complements the math proofs I'm working on.

Building a Bookmark Tool with DuckDB

I've also been feeling the itch to do something more applied so I started working on a small bookmarking tool to keep track of interesting links. I decided to do this in DuckDB which is a tool I've wanted to dig into more. It's also giving me a chance to check out MotherDuck, their hosted service.

Right now the "app" is just a series of SQL queries while I prototype out the different operations I want to perform, mostly CRUD operations and trivial analytics. I have some more queries I want to add, specifically around tag manipulations. Shortly, I want to turn this into a python CLI app with the Typer library.

One thing I've been pleasantly surprised by is the number of convenience functions in DuckDB and how pythonic some of the syntax is. There are functions for testing list membership, you can index and slice lists just like in Python, you can declare checks at the column schema level, and there are even list comprehensions!

Networking

I finally shared on LinkedIn that I'm between jobs and looking to use this time to network and as a result I've had tons of interesting meetings. Just this week alone I've talked to:

A couple of VCs who want to connect me to portfolio companies
The investment arm of a family office
The tech transfer office of a national lab
A local data science director
A founder of a stealth mode database company
A few local CTOs

It's been a blast! I'm meeting a ton of smart people and it's helping me better understand what I want to be working on.

If you want to chat about anything related to tech, please reach out.

Migrating Posts From My GitHub Blog

Wed, 26 Jun 2024 00:00:00 GMT

Over the years, I've had four different technical blogs. A Tumblr, a static blog on Github Pages, a substack, and now this blog. For a long time now I wanted to collect all my writing in one place. I've finally started that project by porting all my posts from my long dormant Github Pages blog over here. I still need to clean up some links and images but the content is ready to go. You can find all the posts under the #github-blog tag.

These posts started around 2013 when I was still working for the Hubble Space Telescope and showcase some of my astronomy pipelines and analysis projects I was working on. They end in 2017 when I had a leadership position at a startup and was an organizer for the local Python Meetup group.

Looking back on these posts now, more than anything I'm thankful that I took the time to document where I've been along the way in my professional journey. It's so easy to forget where you've been and how hard you've worked. Looking back on my work gives me a sense of both pride and perspective. It also give me motivation to write even more and raise the bar on my output!

Fumbling Into Floats

Sat, 04 May 2024 00:00:00 GMT

What Do You Mean?

Recently, I was working on a side project writing a multi-armed bandit simulator from scratch. Part of the code requires iteratively appending to a series of numbers and calculating the mean. It turns out that instead recalculating the mean over the entire series each time you can use a more computationally efficient incremental mean. Here what that looks like in Python:

def incremental_mean(mean: float, observation: float, n: int) -> float:
    return mean + ((observation - mean) / n)

The benefit of using this incremental mean formula is that each incremental calculation is $\mathcal{O}(1)$ while calculating the mean of the entire series would be $\mathcal{O}(n)$. To see this, I benchmarked both methods with a series of 1000 randomly generated floats, once using my incremental mean function and once using the statistics.mean function from the Python Standard Library. Here are the results:

%%timeit
[statistics.mean(input_list[0 : index + 1]) for index in range(len(input_list))
]
433 ms ± 62.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
rolling_incremental_mean(input_list)
513 µs ± 28.1 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Fantastic, it's nearly 1000x faster! Great, we're done.

We were not done

What the Float?!?

I swapped in my faster incremental formula into my project, got the same results but faster, and moved on. But, while I was debugging a different error I decided to return to this function just to double-check. I tried to run an assert against the two outputs from my benchmark to confirm that they were the same and the test failed.

No surprise, probably just a typo!

Then I looped over the two outputs and checked them all pairwise. This time, it made it through a few dozen numbers before failing.

That's weird, shouldn't the formula always be right or wrong?

Then tried plotting the residuals (differences) between the two outputs and got this graph

That's weird ... why are the errors quantized? Shouldn't they be - oh right, floats.

We All Float On Alright

Here's what I thought was going on, to first order.

Numbers are infinite but computers are decidedly finite. This means any computational (physical) representation of the number line is going to have certain constraints. The most common representation computers use are called floating-point numbers, or "floats". This represents every number as some real number of finite length raised to an exponent like this $123.456e^{10}$.

Now here's where the constraints come in. Because the number of digits in the prefix to the exponent (also called the "mantissa") is limited, it means we can only construct a finite number of mantissas per exponent. That means that as the exponents get smaller, the numbers we can represent get closer and closer. The exponents themselves have a limited range (much less than the range of the mantissa) which means at some point we get to the smallest interval we can express in our floating point system.

I'm hand-waiving away some details (which we'll return to later) but that's what went through my mind the first time I saw the graph of my residuals. I noticed the errors were on the order of $1e^{-15}$. This is effectively zero for my purposes, and figured I must have hit the floating point limit. I assumed there must be some rounding approximation in the standard library statistics.mean function or something.

Regardless of the details, this meant my function was correct so I could just stop there.

I did not stop there.

Down the Rabbit Hole

I really wanted to get back to my main project, but now my interest was piqued. Could I prove that this was just rounding errors on floating point math?

To start with, I confirmed that both the outputs each had 1000 distinct values which seemed normal. Now, how many different residual values were there?

>>>len(residuals)
1000

>>>len(set(residuals))
18

OK, only 18 distinct values out of 1000. So now my suspicion is that this effect is being introduced when I subtract the two outputs. 18 isn't that many, let's take a look.

>>>sorted(set(residuals))
  [-6.661338147750939e-16,
 -4.440892098500626e-16,
 -3.3306690738754696e-16,
 -2.220446049250313e-16,
 -1.1102230246251565e-16,
 -3.469446951953614e-18,
 0.0,
 5.551115123125783e-17,
 1.1102230246251565e-16,
 2.220446049250313e-16,
 3.3306690738754696e-16,
 4.440892098500626e-16,
 6.661338147750939e-16,
 8.881784197001252e-16,
 1.1102230246251565e-15,
 1.3322676295501878e-15,
 1.5543122344752192e-15]

Many of those seem suspiciously evenly spaced at something like $1.11e^{-16}$. You can see this more clearly if you plot them on a number line.

We can calculate the distances between points to make sure they're really the same.

>>>residuals_set = sorted(set(residuals))
>>>set([
    residuals_set[i] - residuals_set[i + 1] 
    for i in range(len(residuals_set) - 1)
])

{-2.220446049250313e-16,
 -1.1102230246251565e-16,
 -1.0755285551056204e-16,
 -5.551115123125783e-17,
 -3.469446951953614e-18}

That seems promising, there are only 5 distinct distances. Maybe the smallest of those is the minimum floating point step size?

An Epsilon of Delta

It turns out you can figure out the float properties of your system with the sys.float command.

>>>import sys
>>>sys.float_info

sys.float_info(
    max=1.7976931348623157e+308, 
    max_exp=1024, 
    max_10_exp=308, 
    min=2.2250738585072014e-308, 
    min_exp=-1021, 
    min_10_exp=-307, 
    dig=15, 
    mant_dig=53, 
    epsilon=2.220446049250313e-16, 
    radix=2, 
    rounds=1
)

The value we care about is epsilon=2.220446049250313e-16. This value is on of our list of spacings, but confusingly it's not the smallest one. The Python docs define float_info.epsilon as:

difference between 1.0 and the least value greater than 1.0 that is representable as a float.

OK, that definition seems consistent with the idea that we're hitting the lower limit of what we can represent with floats. It doesn't quite explain why I see values smaller than that but still, this seems like a reasonable place to stop!

We're not stopping.

Epsilon Families

If you squint at the list of distinct residuals you might notice a pattern that many of them are offset by the float_info.epsilon value. Actually, it looks like there are two "families" of residuals, each coming at intervals of epsilon, but with one family offset but 1/2 epsilon. This plot helps show that.

The same data appears in both plots with the families are color coded. On the top they all appear on the same number line. On the bottom figure they are separated for clarity and expressed in units of epsilon. You can see the two families, their spacings and offsets, as well as the two points that don't fit into that scheme.

It's Epsilons All The Way Down

At one point while working on this I remember thinking

Boy, it's lucky that these numbers were multiples of roughly 1.11 or I wouldn't have been able to spot the pattern! (e.g. 2.22, 3.33, etc.)

Yeah, so about that. It turns out this ideas of epsilon "families" wasn't quite right, and the clue was in the remaining points that I thought didn't "fit" the pattern. After making lots more plots I finally realized that all the errors can be expressed in units of epsilon. Take a look:

>>>a = [
    -6.661338147750939e-16,
    -4.440892098500626e-16,
    -3.3306690738754696e-16,
    -2.220446049250313e-16,
    -1.1102230246251565e-16,
    -3.469446951953614e-18,
    0.0,
    5.551115123125783e-17,
    1.1102230246251565e-16,
    2.220446049250313e-16,
    3.3306690738754696e-16,
    4.440892098500626e-16,
    6.661338147750939e-16,
    8.881784197001252e-16,
    1.1102230246251565e-15,
    1.3322676295501878e-15,
    1.5543122344752192e-15,
]

>>> [item / sys.float_info.epsilon for item in a]
[-3.0,
 -2.0,
 -1.5,
 -1.0,
 -0.5,
 -0.015625,
 0.0,
 0.25,
 0.5,
 1.0,
 1.5,
 2.0,
 3.0,
 4.0,
 5.0,
 6.0,
 7.0]

OK so epsilon itself is not the smallest difference we can express (that was already clear). But it is the basis for representing all small differences because all our errors are some rational number times epsilon.

To test this I ran some more simulations and cranked up the sample size to 50k. The residuals moved around a lot more as you can see in the plot below. But when I checked every single one was always a multiple of epsilon.

The patterns was something like this.

For values larger than $\pm 2$ the step size is $k \cdot \epsilon$ where $k$ is an integer (or sometimes $\epsilon/2$).
For values between $\pm 2$ and $\pm 1/4$ the step size was $\epsilon/4$.
Smaller than $\pm 1/4$ goes like $\epsilon/2^{k}$ (e.g. $0.015625$ is $1/64$).

Interestingly, I didn't always see the same step sizes and values so I think there's a lot of optimization going on under the hood.

But, all this to say I'm satisfied that epsilon really is the fundamental building block of small number representations (at least close to 0), which explains the quantization effect I saw.

At the Bottom of Everything

This is very much a inductive approach to understanding the problem, one where I'm working backwards from the data. I could come at this from deductive angle but I think that would involve reading IEEE specs, CS textbooks, and Python source code. All that sounds fun (seriously!) but I want to return to my main project of simulating multi-armed bandits. So, at least for now, this is proof enough for me.

Now, we are finally done.

The Data Chief Podcast Episode

Wed, 24 Jan 2024 00:00:00 GMT

Last July I was the guest on a data leadership podcast called "The Data Chief". I've done a decent amount of mid-sized public speaking but I had never been on a professionally produced podcast before. It was a great experience and we got to cover a lot of ground from my diverse work history (astronomy, infosec, health tech, developer tools), to becoming an executive, and some of my thoughts on running a data team at scale.

I’ve had a slow but steady stream of people reaching out to me since then saying they enjoyed the recording and were able to apply some of what they learned. I thought it was a good nudge to write some thoughts down about the experience.

You can find the episode webpage here as well as the embedded Spotify link below.

Highlights From My Conversation

A theme I tried to bring into the conversation was my perspective of a first time junior executive, someone who is on the executive team but not yet in the C-suite. I wanted to share some perspectives on how I got where I am, what I think worked for me, and what problems I’m working on.

One of the ideas I emphasize for folks who want to move into leadership positions is to understanding your unique value as a leader. You can’t be everything to everyone and make it to the top of any hiring pile you have to have the self-awareness to understand what makes you special.

For example, I’ve worked with a lot of security teams in my career. This is unusual and it’s influenced the way I think about risk. As I said on the show:

You can never completely neutralize risk. You want to document the risk and accept it. And I think it's the same thing with data work. You don't want to, for every situation, have the same degree of precision, the same amount of data required. It's sometimes okay to be directionally accurate and to estimate things.

That’s probably not a make-or-break opinion, but it is something that is based on my experience and it sets me apart a little bit. Having the self-reflection to Identify these things in yourself is something aspiring leaders need to be able to do.

There’s lots more in the episode so if that sounds interesting please check it out and let me know what you think.

Sheepishly Helpful

I was actually hesitant to appear on The Data Chief because it feels intimidating, even arrogant, to get in front of a microphone and claim you have something meaningful to say about data leadership. I’m easily taken in by the common cognitive distortion that just because there are people who are more knowledgeable than me, then clearly I can’t have anything meaningful to contribute to the conversation.

I’m glad I was able to ignore that voice in my head. Earlier this week someone reached out to thank me for helping them find a job leading a data team. My contribution was this podcast episode and few other conversations that inspired them to want to come back to data work and gave them a roadmap to do so.

This was kind of a full circle moment because this is the same individual who encouraged me to go on the podcast in the first place. We had met at a local data meetup and I had offered him some career advice. In turn, I told him I had been invited to appear on a data podcast but doubted I had anything interesting to say. He politely corrected my self-perception (a common theme for me) and encouraged me to go ahead.

All this is to say, everyone struggles with being “enough”. If you want to do some public speaking and get a chance to — go for it! And let me know.

Practicing Jazz with the Linux Command Line

Fri, 08 Dec 2023 00:00:00 GMT

With ChatGPT's help, I wrote a command line script that loops over a play-along jazz track starting at a random point each time. Starting at a random point forces me to develop my ear by finding my place in the chord changes, an important skill in Jazz.

Some Background

Over the pandemic I started taking jazz piano lessons. Jazz is improvisational music meaning it's not written out ahead of time like classical music. Even though it's improvised there is a guiding structure, mostly in the form of preset chord changes. To keep their place in the music, Jazz musicians need to be able to hear these changes.

One of the ways to build this skill is to play along with backing audio tracks that simulate playing with a band. This is useful practice, but it's easy to just keep in time and play along without really listening and responding.

To develop your ear, you need randomness. So I decided to see if I could write a script to throw me some musical curve balls by starting the track in random places

Using ChatGTP

Playing an audio file seemed like a good fit for a bash script. I'm sure there are lots of Python libraries I could have used but that felt like overkill.

I chose to use ChatGTP to help me out because bash's syntax is stable and well-documented. It's also an implementation detail and not something I'm not interested in building mastery in. Better to just outsource it to the LLM

The Script

Here's the first pass from ChatGPT 3.5:

#!/bin/bash

if [ -z "$1" ]; then
  echo "Usage: $0 <music_file>"
  exit 1
fi

music_file="$1"

while true; do
  start_time=$(shuf -i 0-$(mpv --no-video --length "$music_file") -n 1)
  mpv --start="$start_time" "$music_file"
done

Like many technical questions I ask ChatGPT, the outline of this answer makes sense at first glance. And while it sets you on the right track, but there were a handful of bugs and the script as written didn't run. I essentially "paired" with ChatGPT for about 20 minutes, asking question about bugs, using alternate tools, and adding an escape key to kill the loop.

The final script looks like this, I'll break it down below.

#!/bin/bash

if [ -z "$1" ]; then
  echo "Usage: $0 <music_file>"
  exit 1
fi

music_file="$1"
track_length=$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$music_file")
echo "Press q twice to quit"

while true; do
  start_time=$(gshuf -i 0-$(echo "$track_length" | cut -d. -f1) -n 1)
  mpv --start="$start_time" "$music_file" --no-audio-display

  read -t 1 -n 1 key
  if [ "$key" == "q" ]; then
    echo "Exiting..."
    break
  fi

done

First we check for a command line argument that we assume gives the location of the audio track. Then we use ffprobe to get the track length from the metadata, gshuf to generate the starting timestamp, and mpv to play the track. Finally it listens for q to quit the program.

This was a fun little evening project that yielded a useful practice tool as well as some more experience working with LLMs. In the future I might want up the complexity by also having it stop at a random time - but in the meantime I need to practice more first.

Notes on Chapter 1 and 2 of What are Embeddings

Sun, 01 Oct 2023 00:00:00 GMT

What are Embeddings is an excellent long-form primer on the topic of embeddings - vector representations of data used for deep learning applications. The material makes a great counterpart to my current self-study of linear algebra, showing how concepts in vector spaces can be applied to deep learning in very concrete way.

This paper is distinct in my opinion in that Vicki goes to great lengths to contextualize the problem of embeddings at a number of different levels, starting from when you might want to use an ML model or a recommender system at all. She also gives historical context of how embeddings evolved and what problems they solved. I encourage you to read to original text.

Below are my highlights and notes from the first 2 chapters, the Introduction and "Recommendations as a Business Problem". Most of my notes focuses on the later chapter on different frameworks for thinking about end-to-end search and recommendation systems.

I look forward to finding more time to dig into the remaining sections which are more technical, as well as read the foundational Word2Vec paper.

1. Introduction

embeddings — deep learning models’ internal representations of their input data (p. 4)

The usage of embeddings to generate compressed, context-specific representations of content exploded in popularity after the publication of Google’s Word2Vec paper (p.4)

[A graph of] Embeddings papers in Arxiv by month. It’s interesting to note the decline in frequency of embeddings-specific papers, possibly in tandem with the rise of deep learning architectures like GPT (p.5 )

Transformer [66] architecture, with its self-attention mechanism, a much more specialized case of calculating context around a given word, has become the de-facto way to learn representations of growing multimodal vocabularies (p.5)

As a general definition, embeddings are data that has been transformed into n-dimensional matrices for use in deep learning computations. (p.5)

3 steps of the embedding process:

Transform multimodal input into representations such as vectors, tensors, or graphs.
Compress inputs for use by an ML task, a fixed input such tagging, labeling, or semantic search.
Create an Embedding Space that is specific to the training space but can be generalized to other tasks and domains. This generalizability is the reason embeddings are so popular.

We often talk about item embeddings being in X dimensions, ranging anywhere from 100 to 1000, with diminishing returns in usefulness somewhere beyond 200-300 in the context of using them for machine learning problems4.

In Systems Thinking, Donella Meadows writes, “You think that because you understand ’one’ that you must therefore understand ’two’ because one and one make two. But you forget that you must also understand ’and.’" (p.9)

2. Recommendations as a Business Problem

How do we solve the problem of what to show in the timeline here so that our users find the content relevant and interesting, and balance the needs of our advertisers and business partners? (p.10)

2.1 Building a Web App

No Notes

2.2 Rules-based systems versus machine learning

In short, the difference between programming and machine learning development is that we are not generating answers through business rules, but business rules through data. (p. 14)

2.3 Building a web app with machine learning

The 4 components of a machine learning system:

Input Data: processing data from a stream or a database
Feature Engineering and Selection: picking which features (attributes of the data) to use as inputs to machine learning. Embeddings are used as inputs here.
Model Building: Select the important features and train the model, iterating to optimize performance. Embeddings are also the output of this step and be (re)used downstream.
Model Serving: Put the model in prod.

The 3 highest-level types of ML tasks:

Supervised: when we have training data that can tell us if the model predictions are correct (e.g. regression, SVM, NN)
Unsupervised: when this is no single ground-truth answer, patterns can be detected but have to be interpreted. (e.g. clustering, PCA)
Reinforcement Learning: Akin to a game theory problem. We have an agent moving through a model and we want to iteratively learn an optimal strategy. (e.g. PCA and Word2Vec).

2.4 Formulating a machine learning problem

Again, that attributes of the data are called the features. In general, this is a really good section to review as an overview of a ML learning process.

We use linear regression to train a model but then use gradient descent to minimize the cost function (in this example MSE). As a reminder:

$$ MSE = \frac{1}{N} \sum^{n}{i=1}(y{i} - (mx_{i} + b))^{2} $$

2.4.1 The Task of Recommendations

The general task of recommender systems is information retrieval which is focused on pulling relevant information from large corpuses. Within information retrieval we have 2 complimentary approaches:

Search: Direction information seeking, well established at this point.
Recommendation: The query is not directly given but inferred based on learned taste and preferences.

The first industrial recommender systems were created to filter messages in email and newsgroups [22] at the Xerox Palo Alto Research Center based on a growing need to filter incoming information from the web. The most common recommender systems today are those at Netflix, YouTube, and other large-scale platforms that need a way to surface relevant content to users.

Common approaches to recommender systems:

Collaborative Filtering: "finding missing user-item interactions in a given set of user-item interaction history", e.g. most people who watch 8 Star Wars movies watch the 9th and you still have not watched the 9th. There are two main approaches:
- Neighborhood Models: Finding similar users based on similarity functions.
- Matrix Factorization: "the process of representing users and items in a feature matrix made up of low-dimensional factor vectors, which in our case, are also known as embeddings, and learning those feature vectors through the process of minimizing a cost function" (p. 21) This is similar to Word2Vec.
Content Filtering: "This approach uses metadata available about our items (for example in movies or music, the title, year released, genre, and so on) as initial or additional features input into models and work well when we don’t have much information about user activ- ity" (p. 21)
Learn to Rank: pair-wise rankings of items. "This step normally takes place after candidate generation, in a filtering step, because it’s computationally expensive to rank extremely large lists" (p.21)
Neural Recommendations: NN capture the same relationships as matrix factorization and this is were we see deep learning networks like Word2Vec and BERT. An example is convolutional and recurrent NN for sequential recommendation such as in music playlists.

Recommender systems tend to have a specialized architecture with 4 stages:

Candidate Generation: A first-pass model that generates a smaller list of candidates down from millions to thousands or hundreds.
Ranking: Ordering the list of candidate recommendations based on predicted user preference.
Filtering: Remove unwanted items e.g. NSFW content or sale items
Retrieval: Hit the modal endpoint to get final list.

Embeddings play a role in search and recommendation systems similar to that of databases in backend architectures.

Embeddings are a type of machine learning feature — or model input data — that we use first as input into the feature engineering stage, and the first set of results that come from our candidate generation stage, that are then incorporated into downstream processing steps of ranking and retrieval to produce the final items the user sees.

2.4.2 Machine Learning Features

As a general rule, the creation of the correct formulation of input data is perhaps the heart of machine learning. I.e. if we have bad input, we will get bad output.

The process of formatting data correctly to feed into a model is called fea- ture engineering [...] when we have textual data, we need to turn it into numerical representations so that we can compare these representations.

2.5 Numerical Feature Vectors

This section just explains encoding text as a numerical value so we can represent it as a vector.

2.6 From Words to Vectors in Three Easy Pieces

In deep learning architectures and NLP-related tasks you repeatedly see the following core concepts:

Encoding: Representing non-numerical multimodal data as numbers.
Vectors: We store our data as vectors
Lookup Matrices: hash tables to allow for efficient lookups between words and numbers

Finite Dimensiona Vector Spaces Ch. 1 Problem 2(c)

Sat, 23 Sep 2023 00:00:00 GMT

I'm 9 months into my self-directed study of math and I wanted to start writing about some of the problems I enjoyed working on. First up is problem 2(c) from Paul Halmos's foundational 1942 text "Finite Dimensional Vector Spaces".

I was excited by this problem because of the technique I had to use to solve it. It's the first to me I've been given a problem that can't be solved in the original domain, but can be solved the mapping your set to a new domain with different properties. It's a technique I've seen before, but never one I've been asked to implement myself.

The Setup

The first two parts of this problem are quick to verify and on-par with what you would see in any other introductory abstract algebra text covering fields.

Briefly, the set of all positive integers ($\mathbb{Z}+$) is not a field because there is no additive inverse ($\nexists -x \in \mathbb{Z}{+}$ such that $x + (-x) = 0$ for $\forall x \in \mathbb{Z}_{+}$).

Similarly, the set of all integers ($\mathbb{Z}$), while it does have an additive inverse, is still not a field because there is no multiplicative inverse ($\nexists x^{-1} \in \mathbb{Z}$ such that $xx^{-1} = 1$ for $\forall x \in \mathbb{Z}$).

This is all pretty standard stuff but Part (c) was interesting to me though.

The Problem

Can the answers to the question be changed by re-defining addition or multiplication (or both)?

After have no luck with my usual bag of tricks (modulo addition, trig functions, even/odd numbers) I found some help on Math Stackexchange .

We can, in fact, redefine addition (here $\oplus$ instead of $+$) and multiplication ($\odot$ instead of $\cdot$) such that $\mathbb{Z}{+}$ (or $\mathbb{Z}$) are fields $\mathfrak{F}(\mathbb{Z}+,\oplus,\odot)$.

We want to take advantage of the fact that the rational numbers ($\mathbb{Q}$) are a field. Note that $\mathbb{Z}{+}$ is infinite but has the same cardinality as $\mathbb{Q}$ -- they are both countably infinite (written as $\aleph{0}$). Using this fact, we can define a function $f: \mathbb{Z}+ \rightarrow \mathbb{Q}$ . That is, a bijection (a mapping that is one-to-one and onto) that maps each member of $\mathbb{Z}{+}$ onto a unique member of $\mathbb{Q}$.

Then we can use our function $f$ (and its inverse $f^{-1}$) to build a new definition of addition and multiplication for $x \in \mathbb{Z}_+$ such that it now meets the properties of a field: $$ \begin{align} &\oplus : f^{-1}(f(x) + f(y)) & \ &\odot : f^{-1}(f(x) \cdot f(y)) & \end{align} $$

Why It Works

These new definitions of addition and multiplication satisfy the properties of a field by moving our binary functions to $\mathbb{Q}$, performing the operation, and then mapping the result back to $\mathbb{Z}_{+}$. Note that we never actually define the function, and there could be infinitely many such mappings, it's just enough to show that it exists.

The general technique here is to use a function ($f$) to map a set ($\mathbb{Z}{+}$ or $\mathbb{Z}$) into a codomain ($\mathbb{Q}$), using the properties we need in the new codomain (additive and multiplicative inverse), and then using an inverse function ($f^{-1}$) to return to the original space ($\mathbb{Z}{+}$ or $\mathbb{Z}$).

Conceptually, this idea of mapping to a new space with useful properties reminds me of related techniques from other areas of applied math. For example, a Fourier transform to transform a waveform from time-space to function-space, or a Laplace transform to go from a real-space to a complex-space.

Why It's Interesting

Like most people, most of my math career has been focused on calculation, I was given formulas and told to apply them. The motivation and reasoning behind the techniques was explained, which is how I recognized some of the parallels I pointed out. But these were always presented a tools to be used, not general techniques that we could also apply.

So to wrap things up, even though this is an elementary result it got me excited because this is the type deeper mathematical tool set I would like to build.

The Best Math Advice I Ever Got

Sat, 12 Aug 2023 05:00:00 GMT

Hieroglyphics and Pictures

Math is just a picture which you can understand hidden in hieroglyphics which you don't understand. Your job is to turn the hieroglyphics back into a picture.

- Prof. David Griffeath, UW - Madison

That quote is the best pedagogical math advice I ever I got. It was an off-hand remark in 1-credit elective on cellular automaton (as in Conway's Game of Life), an esoteric course even by math standards. I doubt any of the other students remember hearing it, and probably professor Griffeath doesn't remember saying it. You never know the impact your words are going to have!

But offhand remark or not, this idea of "translating" concepts into pictures resonated with me. Case-in-point, the only concept that I vividly remember from my senior-level abstract algebra class are the diagrams I drew for myself to understand how 3 major classes of functions map between sets; injective (one-to-one), surjective (on-to), and bijective (both surjective and bijective).

Pictures can be powerful tools. When I say this I'm not trying to say I'm a in some way a "visual learner" (the popular theory of "Learning Styles" is widely debunked). Instead, to me drawing a picture is a shorthand for saying you have a deeper understanding of a concept that just parroting a definition. It's when you understand the meaning of an idea enough to translate it into different forms, such as a picture.

This quote has been kicking around in my head a lot over the past 8 months and I've been taking a second pass at some of the subjects from my undergrad coursework, linear algebra in particular. I don't always draw a literal picture, but I am going more slowly than I have in the past and striving for that same level of deeper understanding that allows me to the express concepts in my own language and shorthand. Being able to rewrite something has proven a pretty good heuristic for how well I understand something. And realizing my "pictures" are wrong has been a pretty effective way to correct my understanding!

This brings me to the final thing I've always liked about this quote, which is that I find it encouraging. On the other side of whatever difficult concept you're banging your head against is something friendly and accessible, a picture.

Your job is just to get to that picture.

Hello, Manjaro!

Wed, 12 Jul 2023 05:00:00 GMT

Hello, Manjaro!!

Hello from a fresh Manjaro Linux install on my 2014 Macbook Air 6,2!

This is my first time running an Arch-based Linux distro and my first time running Linux on Apple hardware. I wrote out a few thoughts on picking Manjaro and getting everything setup.

But first, let's talk about dorking around with Linux distros while everyone else is playing with AI.

Under Pressure

If you're even moderately connected to the more "online" parts of the tech world, it's hard not to feel pressure to constantly be doing more. Pressure to build a public persona, pressure to ship, pressure to be up-to-date, pressure to get the next job, pressure to join "the discourse", to monetize your side hustle, to turn your project into a company.

I was thinking about this pressure as I was an hour into sitting cross-legged on the floor tethered to my router with a 3-foot Ethernet cable trying to set up a Wi-Fi drivers on an 8 year-old machine when I have four other perfectly good laptops around the house. As I finally got the Wi-Fi icon to light up I couldn't help thinking ...

I did it! ... But should I have?

The rest of the tech world is exploring a brave new world of AI and LLMs. I've got work I could be doing for my job or math I could be studying to further my career. Instead, I'm losing the feeling in my legs while tweaking Linux settings and then doubling down on that decision by blogging about it for the 2 (maybe) people who read this blog.

My point is that playing with Linux is not what productivity culture tells me I should be doing. But, I think it's important to learn to let go of that. This is something I've written about before, but sometimes, it's nice just to fiddle for the sake of fiddling. To write for the sake of writing. To practice both as a craft, or if nothing else just so the ideas will stop rattling around in my head.

So let's talk about messing around with Linux, AI will still be there when we get back.

Going Beyond Ubuntu

The machine I set this up on is an old MacBook Air that hasn't been used in a few years. It's about 8 years old at this point, which is well-past the hardware cutoff for the latest MacOS updates. But, with the right OS, a quardcore i5 CPU and 4 GB of RAM is more than enough for the programming and websurfing I do. So I decided to give this machine a second life as Linux box, but which flavor?

Ubuntu was my first choice. I have been running it for 10+ years across two older IMB/Lenovo laptops so I was comfortable with it and I knew it met my needs. At first I was worried it wouldn't work well on Apple hardware, but from what I've read that doesn't seem to be an issue. And in case it isn't obvious from the MacBook model date, this is still an Intel-based CPU. So I'm not worried about compiling against the new generation of Apple CPUs, just general hardware support.

But, as I mentioned, this machine only has 4 GB of memory. That barely meets the required minimums for my standard Ubuntu + Gnome desktop setup. I could address this by running a lighter-weight Ubuntu distro like Xubuntu or Kubuntu. Or, if I wanted to stay in the Debian family, I got an old laptop up and running for my dad on ZornOS last year.

But, none of those felt like compelling options to me. I was doing this purely for fun and after 10 years on Ubuntu I wanted to try something completely new.

Settling on Manjaro and Arch

When I first started running Linux on my own machines Ubuntu felt like the front-runner for folks who just wanted something FOSS "that just worked" in a desktop environment. But as I was looking around now in 2023 it seemed like there were a lot more well-supported options that had taken the same "batteries included" approach. On distro that I had never heard of seemed to be on everyone's list: Manjaro.

Manjaro (man-JAR-o), is a popular and relatively user-friendly distro based on Arch Linux. This immediately caught my eye because Arch Linux is a notoriously difficult distro to set up. It comes with essentially nothing included, not even a desktop, so you start from a command line and you have to really know your way around Linux to set it up. The payoff is a high level of customization. Manjaro tries to bridge that gap by providing a user-friendly Arch setup.

Honestly, they had me at notoriously difficult. I decided to give Manjaro a shot.

First Steps with Manjaro

All-in-all this was a pretty easy system to get set up. The one major exception, as I mentioned in the beginning of the post, was getting the Wi-Fi drivers to work. But that likely has nothing to do with Manjaro or Arch; Apple hardware is proprietary. As I understand it, that means that they do not release open-source drivers for some of their hardware such as the webcam and, crucially, their wireless network card.

The solution to this is install the OS with a wired connection and then install the drivers needed to support the Broadcom network hardware on my machine. Fortunately, this is a well-documented problem on the Manjaro / Arch message boards and after about 90 minutes I had a working Wi-Fi connection.

The steps are well-documented elsewhere but just to give you sense of the scale of solution I had to:

Figure out how to read the system configuration with inxi.
Decide I wanted to go with the boardcom-dkms driver (broadcom-wl-dkms and wpa_supplicanat).
Learn how to use pacman (the Arch package manager).
Find my kernal version and install the corresponding linux header files.
Use ip link set to enable the network interface.

And then it worked! So not terrible, but definitely something you need at least intermediate Linux knowledge to troubleshoot. And this is an issue if you're running this on a Mac.

All-in-all a fun weekend project and a decent excuse to crank out a blog post. Thanks for reading!

Hello, LaTeX!

Mon, 22 May 2023 05:00:00 GMT

This year I've working on gaining a deeper understanding of AI and ML tools. To do this I've been studying a lot of math, specifically vector spaces. As I've been getting increasingly excited about the progress I'm making, I figured some blog posts were in my future. This seemed like a good enough reason to go on a side quest to set up equation rendering.

Fortunately, the blogging framework I'm using right now, Nextra, comes with $\LaTeX$ (nice) support built in. After migrating some configuration files, I was able get it set up and wanted to do a "little hello" world post with some equations!

So, as a proof-of-concept, here is the definition a subspace, of a core concept I've been studying in Sheldon Axler's text Linear Algebra Done Right.

Subspaces

For a vector space $V$ over a field $F$ (nominally either $\mathbb{R}$ the real numbers or $\mathbb{C}$ the complex numbers), the subset $U \subseteq V$ is said to be a subspace of V if and only if ($\Leftrightarrow$) the following 3 conditions are met:

Additive Identity: The additive identity $0 \in V$ is also $\in U$. That is, $0 \in U \cap V$, where $0$ is an element in $V$ such that $0 + x = x + 0 = x$ for all $x \in V$.
Closure Under Addition: For any $x,y \in U$ we also have $x+y \in U$.
Closure Under Scalar Multiplication: For $a \in F$ and $x \in U$ we have $ax \in U$.

Update 2023-06-25: Fixed the definition of a subspace to include the trivial case $U = V$ i.e. $U \subseteq V$ instead of $U \subset V$. Thanks Dad :)

Journal Club: Talking About Large Language Models

Mon, 16 Jan 2023 06:00:00 GMT

I want to do more intention self-directed study in 2023. One of the ways I'm working on this is by reading academic and technical papers. Taking notes on what I'm reading gives me some more direction and helps organize my thoughts. Hopefully, I'll continue commenting on articles as time and inspiration allows.

My first paper of the year is “Talking About Large Language Models” by Murray Shanahan. It was a good first pick! The topic of Large Language Models (LLMs) is basically inescapable in tech right now and the fact that the article is fairly philosophical and less technical in nature made it an accessible entry point for trying to understand these systems. Overall, I think the article gives a useful framing of LLM systems that I'll continue to use as I try to understand LLMs and AI generally.

What follows are my notes but the paper itself isn't that long so you should consider reading the original if this seems interesting to you.

With apologies to Raymond Carver, let's dive in.

What We Talk About When We Talk About Talking About Large Language Models

The paper stars by reminding us that despite the incredible and evolving accomplishments of LLMs they are fundamentally still "next [text] token prediction" machines. As such, they lack any internal structures that could be mapped to anything like the human experience of "knowing" or "understanding". Because LLMs and other forms of "exotic, mind-like entities" are poised to become a fixture in our everyday lives we need to be precise in talking about what these systems are actually doing, both as practitioners and in the general public.

Unreasonable Effectiveness

The article starts by pointing out 3 ways in which large language models (LLMs) are “unreasonably” effective. As an aside, this is a reference to Halevy, Norvig, and Pereira's 2009 article "The Unreasonable Effectiveness of Data". This is in turn is a reference to Wagner's 1960 article "The Unreasonable Effectiveness of Mathematics in the Natural Sciences" (additional background).

The third of there three points is the focus of the article and the most relevant to the understanding LLMs:

… a great many tasks that demand intelligence in humans can be reduced to next token prediction with a sufficiently performant model.

The more you sit with this statement the more remarkable it becomes! However, in spite of this (or rather because of this) the rest of the article focuses on reminding us that this "effectiveness" is not in fact anything resembling “intelligence”.

Defining LLMS

Continuing on, we get a definition of LLMs:

LLMs are generative mathematical models of the statistical distribution of tokens in the vast public corpus of human generated text …

And from there the author reminds us that, under the hood, all LLM systems are doing is something like the following:

Here’s a fragment of text. Tell me how this fragment might go on. According to your model of the statistics of human language, what words are likely to come next?

Reiterating and tying those points together more explicitly:

There are two important takeaways here. First, the basic function of a large language model, namely to generate statistically likely continuations of word sequences, is extraordinarily versatile. Second, notwithstanding this versatility, at the heart of every such application is a model doing just that one thing: generating statistically likely continuations of word sequences. Having established how LLMs work (probabilistic prediction on sequences of text tokens) the article now focuses on the fact that, by design, LLMs can not “know” or “understand” anything. They say what is likely to come next.

These concepts are largely preliminary to the main point of the article, which is the limitations of these systems and how our language should reflect this. But, as technical reader who is not an AI practitioner, these preliminary definitions are an extremely useful framework to begin to think about all the AI and LLM models news I see everyday.

LLMs, Knowledge, and Anthropomorphism

The remainder of the paper presses the issue that these systems, by nature of their scope, can not be said to "think" or "know". And further, it's important not to reach for convenient idiomatic expressions that suggest they can. This precise language is important for an accurate long-term societal understanding of what these systems are doing.

The author provides a nice analogy to illustrate these limitations:

An encyclopedia doesn't literally “know” or “believe” anything, in the way that a human does, and neither does a bare-bones LLM.

The author supports this position by arguing that LLMs operate in a context completely outside of that of human "communicative intent"; meaning it knows nothing about person asking the question, people in general, or the effect of its response. The author then preempts the argument that such knowledge could arise as an emergent phenomenon from a neural network like system because sequence prediction (the essence of LLMs) will never contain a notion of communicative intent. I'm omitting the details because neither argument is particularly lengthy and interested readers would be better served to read the source material.

Wrapping Up

To conclude, the author asserts (and I largely agree) that these arguments are neither pedantic nor purely philosophical. LLMs, AI, and other “exotic, mind-like entities” are not going anywhere and nor are they well understood. The linguistic choices we make now could potentially impact the ability of practitioners, law makers, and the general public to reason about the limitations and abilities of these new and exciting systems.

This paper was an interesting and relevant start to the year and I'm glad I found time to jot down my thoughts. I'm hoping the next paper I pick will come from the citations in this paper so I can start to build a deeper understanding of this space.

I Love Computers

Sun, 01 Jan 2023 06:00:00 GMT

I'm coming off an 11-day break for the winter holidays. I came into the holidays this year really run down and needing a break. But weirdly, one of the things I did to recharge from my job working with computers all day was ... tinkering with and maintaining various computers in my house. I have 3 different generations of MacBooks in my house but I spent most of my time on my favorite machine, a 9 year old ThinkPad X1 Carbon running Ubuntu.

I worked on everything from updating software, installing new utilities, tweaking custom GUI aesthetics, fixing nagging warning messages, and fiddling with configuration files. It was a nice change of pace from my work life where I have to prioritize efficiency over say, getting a mouse pointer size just right.

It was also a nice reminder that ever since I was a kid I have loved computers for their own sake. Not just for building useful stuff, I just enjoy the act of working on them, even in the absence of a project. There's something very rewarding to me about taking a system that's nearly a decade old and getting it to be almost as nice to work on as the MacBook M1 Pro I use for work.

Happy new year and I hope we all get to do more of the things that make us happy in 2023.

To start of the year with some festive energy please enjoy Masayoshi Takanaka's samba cover of the Star Wars theme. It's completely irreverent, wildly fun, and so much better than it has any right to be.

One Day Of Advent Of Code

Sun, 01 Jan 2023 06:00:00 GMT

This year I finally accomplished a dream of mine - I only did a single day of the annual Advent of Code challenge. While most people aspire to complete the entire 25 day exercise, I've learned Advent of Code is as much about me wrestling with my own perfectionism as it is about solving coding puzzles.

I've been doing Advent of Code for 4 years now. 3 years ago, I did the most I had ever done and I was miserable. For 13 days AOC was eating up what little time and energy I had after work. I was caught in a cycle where I was ashamed of myself for not solving the problems more easily. But, I was unable to step away because I felt like I was so close to finishing. This made me invest even more time which made me even more ashamed.

Last year, I only did 4 days of puzzles but I also stopped trying to solve as many problems as I could. Part of this is because I stopped feeling like I needed to use AOC to prove my programming ability to myself and others. This allowed me to start using AOC as a sandbox to try new (to me) Python tools and features like type hints, mypy, and to build out workflows in Make. I had way more fun working this way and ended up with some tools and patterns I was able to use for the rest of the year.

I discovered that last part, up-to-date knowledge of how to build complete modern python projects, is really valuable to me. As a then manager and now executive, I'm not coding regularly. So when I do have a chance to get my hands dirty, for fun or for profit, it's a huge accelerator to have a go-to template to quickly build out a project with up-to-date tools and practices.

I don't mean "template" metaphorically. I have a cookiecutter template I use to build my projects and now I use Advent of Code as my chance to give it an annual refresh.

This year, I leaned even more into "doing more with less" and only did one day's worth of puzzles. I then spent the remaining time making updates to my solution workflow including switching from flake8 to ruff, adding mypy into my workflows, and adding a template documentation page with Sphinx driven by docstrings and typehints with autodoc. With the exception of Ruff this was all stuff I'd done before, but had never taken the time to systematize.

I then started porting the ideas I enjoyed into my template. Checking for bugs in your template can be tricky because the template itself is full of jinja-style template variables (e.g. {{project-name}}) so you can't run it directly. To help with this I created a new project with a Makefile that pulls my cookiecutter template and runs a "hello world" build to check for obvious bugs.

I really wish I had the time to do all 25 of the AOC puzzles every year, but the truth is I don't. Instead, I'm glad I've gotten to the point in my life where I have the confidence to make use of something in the way I want to.

The Coconut Programing Language

Mon, 24 Oct 2022 05:00:00 GMT

A friend of mine tipped me off to a neat little project called the Coconut programming language. From the docs:

Coconut is a functional programming language that compiles to Python. Since all valid Python is valid Coconut, using Coconut will only extend and enhance what you're already capable of in Python.

Because I have a math degree and use functions in my Python code I like to pretend I'm a functional programmer (mustache twirl). However, other than my annual 30-minute attempt to learn Haskell on the first day of Advent of Code and the occasional Python type hint -- I haven't made much progress on that front.

On one hand, I doubt I'll spend much time with Coconut. But on the other, I do think this would be the first project I would reach for now if I wanted to get my hands dirty working in a functional framework.

My last thought here is that this project kind of reminds me of CoffeeScript. That effort seems to have run out of steam but developers of a certain age will remember it as another Haskell-influence language that compiles down to a "mainstream" language.

Where The Water Goes by David Owen

Sat, 17 Sep 2022 05:00:00 GMT

David Owen's "Where The Water Goes" follows the author along the length of the Colorado River as he narrates the complex legal, economic, and environmental issues around water usage.

What stood out to me in this book is the cascading of unintended consequences, even from what seem like sensible environmentally conscious decisions.

For example:

Allowing man-made lakes to dry up releases captured pollution into the air
Water waste provides neccissary "slack" in water system capacity and replenishes the water table
Efficient water consumption induces more water utilization
Removing water-intensive plants increases dust and creates "heat islands"
Man-made water ways become critical support systems for endangered species

The ways in which decisions, compromises, and accidents continued to change the social, environmental, and literal landscape were mind-boggling. Overall, it made me interested in learning more about the state of the waterways in the upper Midwest where I live.

The Crystal Programming Language

Tue, 06 Sep 2022 05:00:00 GMT

I recently came across a Hacker News thread on the Crystal programming language that piqued my interest. Reading around, it seems to hit many of the language features I'm interested in.

From what I can gather, Crystal is an imperative, compiled, and statically-typed language with a syntax heavily influenced by Ruby. Types can often be inferred by the compiler resulting in a syntax more similar to dynamic languages while static typing leads to more efficient compiled code.

Here's a sample webserver from their documentation page:

# A very basic HTTP server
require "http/server"

server = HTTP::Server.new do |context|
  context.response.content_type = "text/plain"
  context.response.print "Hello world, got #{context.request.path}!"
end

puts "Listening on http://127.0.0.1:8080"
server.listen(8080)

This syntax looks very clean to me. It's just different enough to Python to be interesting but not so much that I couldn't quickly be productive.

Maybe I'll try it out on one or two Advent of Code puzzles this year.

Thanks for reading, please enjoy this recording of a Beethoven late string quartet (No. 12, Op. 127) by the LeSalle Quartet.

SQLite and DuckDB

Mon, 05 Sep 2022 05:00:00 GMT

I recently came across a great blog post by Simon Willison (creator of Datasette) digging into a recent academic paper comparing OLAP workloads on SQLite against DuckDB.

DuckDB is a newer file-based database that's generating a lot of buzz and is nicknamed "the SQLite for analytics".

If you're into that sort of thing you should check out the original paper. It's pretty readible and gives an overview of SQLite's history, architecture, OLAP benchmarks, bottlenecks, and potential optimizations.

I love this type of well-organized reserach and I feel like I've learned a lot more about my favorite database even though I'm only 1/3 of the way through the paper.

Thanks for reading and please enjoy this wonderful Georgia O'Keeffe's painting that has moved me tears before.

Hello, Darkness

Thu, 01 Sep 2022 05:00:00 GMT

I figured out how to turn on a "dark mode" toggle. It's somewhat buried in the docs but you can add a darkMode: true flag in theme.config.js to turn this feature on.

I was a little confused at first because the app doesn't default to dark mode, it just adds a sun/moon icon in the upper right the page where you can toggle your color preference.

It's a pretty intense high-contrast dark mode though, which I'm not sure if I like it enough to keep.

But this is a neat little app and I'm enjoying playing with it.

Please enjoy this documentary on the Helvetica typeface.

Hello, World!

Wed, 31 Aug 2022 05:00:00 GMT

I'm starting this blog with some experiments with 100-word posts.

I'm not witty enough for Twitter, but I mostly don't want to argue with strangers. Writing a newsletter was fun, but my tiny ideas keep exploding into multi-page essays. The proofreading and editing buried me for weeks.

So I'm trying for ~100 words at a time. About the length of a few tweets. I mostly just want a place to jot down my thoughts. Some code, some math, some music.

Maybe I should just text my friends more.

But I like playing around with Markdown so here we are.

Please enjoy this video of Jaco Pastorius playing bass.

Prototyping for Data Teams

Fri, 23 Jul 2021 23:17:47 GMT

Hello, welcome back, and thanks for reading! If you enjoy this issue please consider sharing or subscribing.

We're Back!

I took a break from this newsletter to work on some of the projects I wrote about in this issue. I'm trying to make time to write again and part of that process is trying to nudge this newsletter towards a shorter and more sustainable format.

I think the tone I'm aiming for is less like the comprehensive Medium-style guides of my first issues and instead more like informal sketches of ideas I'm kicking around. Something more like an email you would send a friend when you come across an interesting article or have an idea you want to share.

You're that friend. We're data friends now.

Prototyping for Data Teams

The main project I've been working on for the last few months is organizing some thoughts around the role of prototyping for the modern data practitioner.

My thesis goes something like this. The expanding role of modern data teams, powered by executive-level buy-in and self-service SaaS tools, had allowed data teams to take on projects that are higher impact and more complex then their previous work. This is interesting and exciting but because the stakes are now higher the costs of building the wrong things are also higher.

One of the ways to reduce project risk is to create a prototype. But to prototype a data product (even something as basic as a dashboard) you often have to use data that is somehow approximate, simulated, or outright faked. I think this flies in the face of the natural tendency of data teams to obsesses over data accuracy and as a result we tend to skip this step. The result is that we don't really start validating if we've built the right thing until after we've built it, at which point we're already dealing with a sunk cost.

My first public iteration of this idea came last month when I spoke at the Data Quality Meetup hosted by DataFold. It's a great event I encourage you to check out. You can read a write-up of the entire event and see my section in the video below.

Hitting the Conference Circuit

I read a blog post years ago by someone who was a frequent speaker at tech conferences about how they approached the conference submission process (unfortunately, I can't find the link anymore). The author's method was to pick one topic at the beginning of the year, make the best presentation they could around that topic, and then submit iterations of that talk to every conference for the rest of the year. It's a simple concept but it was mind-blowing to me at the time.

This newsletter has already gotten me the habit of publicly articulating my ideas and then revisiting and rehashing them. From there, it was a natural progression to bring the same approach to my conference presentations. Case in point, I submitted an updated version of my Data Quality Meetup talk to the dbt Coalesce Conference in December. And then again to the Fivetran Modern Data Stack Conference in September.

You can read my Coalesce Conference submission here and watch my pitch video here.

Submitting to conferences is scary. You could get rejected, you could get accepted and not be happy with your talk. And there's nothing like watching a video of yourself speaking to make you realize what a weirdly twitchy and awkward human you are. 🤪

Submitting to conferences means opening yourself up to failure. So let's talk about failure.

Failure as a Goal and Failure as a Process

2 years ago when I was last on the job market, I read a blog post about someone trying to get to 100 rejections in their job hunt. I can't find the original post, but it was certainly inspired by this TED Talk. I know TED talks are something of a meme these days but this was a useful nudge for me.

I realized that if I wasn't getting rejections then I wasn't being ambitious enough in my job search. So even thought it left me anxious to the point of insomnia, I applied to everything I was even remotely interested in: from seed stage startups to billion-dollar companies, and from VC firms to boutique analytics consultancies.1 It made my job hunt much scarier but also more interesting and fulfilling. I think I ended up feeling much more settled in my final decision as well.

It's in this same spirit I'm intentionally telling you about the conferences I've applied to without knowing if my talks were accepted.

I'm doing this because I think it's healthy to make sure we're "failing" just a little bit, and even to loosen up our concept of what counts as success. I'm doing it because I want to choose to celebrate the hard work I put into this process and what I was able to produce.

Let me be clear, I hate failure and rejection. I'm naturally anxious and risk-adverse. I mean, I leave the house with two raincoats. But I've been learning to let some of that go when it doesn't serve me.

Lastly, I hope it's not presumptuous of me to assume that you might need the same reminder to put yourself out there a little more.

Seeing as we're data friends now.

Odds & Ends

Around the Internet:

3 Things: This newsletter is increasingly influenced by the format of my friend Elaine's "3 Things" newsletter. Concise, original, and consistent, Elaine is a VC who digs into 3 business rabbit holes each week she finds interesting. The breadth of her ideas are astounding to me and I admire her delivery.

Listening:

Flow State: If I have your phone number and you're into music I've probably already texted you about this one. "Flow State" is a substack that comes out every work day with "2 hours of music perfect for working". They lean heavily into ambient, experimental, electronic, and jazz artists, which minimal lyrics and more upbeat selections on Friday. My only complaint is that its too much content!

Reading:

Irrational Exuberance: Robert J. Schiller is a noble laureate in economics and Irrational Exuberance is probably his most famous book about behavioral economics and specifically the wisdom/folly of crowds in pricing assets such as stocks and real estate. The topic is fascinating but this book is a textbook in disguise. Incredibly exhaustive and discursive I was barely able to get though it even though I'm glad I did.
The Box: I enthusiastically read a whole book about shipping containers in case you wanted to know exactly how much of a nerd I can be.
90% of Everything: Another book about logistics, this time about the shipping industry. I thought this was a nearly perfect non-fiction book. Informative but not overly serious and with a narrative arc that doesn't overshadow the content.

I should acknowledge there's an abundance of research that shows that as a straight white male with a nondescript name that obscures my immigrant identity and a traditional STEM background, I'm generally taken much more seriously as a candidate for a broad range of positions than folks from other backgrounds.

Configuring Your OSX Terminal

Fri, 07 May 2021 12:30:05 GMT

Hello and welcome back! This edition of my newsletter is about how to configure an attractive and functional terminal environment. I'll be walking through my current setup (pictured below) which uses iTerm2, the Dracula color theme, Meslo LGS Nerd Font, and the Fish shell with the Tide theme. This tutorial is geared towards OSX so your milage may vary on Linux and Windows systems.

Why Should You Care?

As always, why should we bother with this in this the first place?

Given the justifiable momentum right now to low-code and no-code solutions in the data space, the command line might not be a high on many people's list of favorite tools. But, even as we gain new SaaS tools the command line remains an incredibly powerful systems programming tool. For folks in the analytics space, the command line is useful for everything from parsing and peeking at data to setting up your Python environment. Using the terminal becomes essential as your work gets to closer to the Data Engineering and DevOps domains (even if those words aren't in your job title).

But for a lot of people who work in data, and even some engineers, the terminal is their least favorite tool. The steep learning curve and dramatic failure modes aside, it doesn't help that terminal environment itself can feel like stepping back in time -- forget linting and autocomplete, there aren't even any colors!

But check this out …

Sharp fonts, contrasting colors, fun glyphs -- all this could be yours!

That's my current terminal setup pictured above. From left to right you can see my system type, my current working directory, my git branch, the number of staged and untracked files in that repo, which virtual environment I'm using, and the timestamp of the last command. On the next line I've started to type git add and tests/ is being suggested as an autocomplete option based on the modified files in the repo.

These aren't just neat tricks, all this information clearly displayed helps prevent many of my most frequent command line mistakes. Notice that form is as important as function here. The font face is clear down to the details in the file icon. The colors clearly separate out different information. There's visual whitespace between each command. Theses are little things but when you are staring at a terminal for hours a day these little things add up.

Lastly, I think there's something to be said for the educational value that comes from taking ownership of your tools; customizing them, breaking them along the way, but ultimately learning how they work.

Let's get started!

iTerm2

lolcat, figlet, and cowsay are some of very serious tools you can use in the command line

Let's start by downloading the iTerm terminal emulator for OSX. iTerm is a feature-rich alternative to the built-in OSX terminal application. Some of the features that I use regularly are:

Better control of font size, font, and transparency
Tabs, Split screens, and moving panes
A configurable status bar (visible at the bottom of the screen shot above) that can show things like location, git info, CPU and memory use.

Even just this handful of features is a quality of life improvement over the OSX Terminal app. Note that iTerm only runs on OSX so Windows and Linux users can skip this step. The good news is that the Linux terminal (or at least on Ubuntu) can already do most of what I use iTerm for. Windows users should look at setting up WSL 2 which gives you a native Linux terminal your Windows environment.

Colors

Credit: https://draculatheme.com/

Now that we have a better terminal environment let's get rid of the default black and white color pallet. We want our color theme to provide helpful highlighting as well as reduce eyestrain and be aesthetically pleasing. iTerm already comes with a few built-in color presets which you can find under Preferences > Profiles > Colors > Color Presets. Personally, for years I used the popular built-in Solarized Dark theme and would switch to Solarized Light in bright lighting conditions like outside. Lately, I've switched to the Dracula color theme (the background story is worth a read).

If none of these appeal to you can grab all the iTerm2 color files you want from this repo. They're a little hard to interpret but you can find plenty of "best of" lists on web that can guide you toward the most popular ones. After you download them you can import and select them from the same Color Presets menu.

These colors might not look like much right now, but as we keep going they'll start to pop a bit more as our terminal makes better use of them.

Fonts

Source: https://www.nerdfonts.com/

Typefaces are a rich world of nerdiness unto themselves with their own subculture of designers. There's a documentary on Helvetica. There are books about type faces (including one by some good friends of mine). All this is to say that while you might never have given the font in your terminal much thought -- some people have.

OK, but what if don't want a research project and just want a good font? From what I've read, a safe bet for OSX users __ is the Meslo LG font family which you can read more about here. I picked Meslo because it's a monospaced font designed for software applications and is specifically optimized for OSX systems and displays. But this optimization means it doesn't always look great on my external monitors, in which case I switch to the Source Code Pro family.

But wait! These are not exactly the typefaces you want to install. We're going to make our terminals fancy (🧐 🎩) by choosing variants of our fonts that include glyphs from the Nerd Font family. For the font families I mentioned these are the Meslo LGS Nerd Font and Source Code Pro Nerd Font (aka "Sauce Code Pro" for namespace reasons). We'll make use of these new glyphs in the next section.

Lastly, make your font bigger. While reading small font won't permanently damage your eyes, it does cause eye strain. Many of us correct for this by hunching over and squinting without even realizing it, leading to muscle strain and headaches. Here is an experiment you can try; make your font just a little bit bigger than you think it needs to be and then see if you even remember to change it back.

Fish Shell and Themes

Source: https://github.com/IlanCosman/tide

So far, we have a modern terminal emulator, a nice color theme, and a sharp font. Now let's bring them all together. Within your terminal you are always working in a "shell". You can think of a shell as a scripting language for interacting with your operating system. The most common shell is called bash; though the OSX default recently switched to zsh, a more modern bash-compatible option.

For the past few years though I've been using Fish Shell, a user-friendly shell that come set up with things like color highlighting and extensive auto-completion for everything from git to unix commands ("Finally, a command line shell for the 90s").

There is a trade off. Fish achieves at least some of this by not being bash compliant. I've found this is rarely an issue as Fish will respect a #!/bin/bash header and otherwise I can drop in to a bash shell anytime I need to. In contrast, almost every command I run in Fish gives me some kind of value such autocompleting file paths.

Fish works great out of the box but to the get the best experience I prefer setting a theme with a plug-in manager. All of these themes will make heavy use of the Nerd fonts we installed earlier. For the past few years I've been using the Oh My Fish plugin manager with the default theme. I briefly used the bobthefish theme but the Powerline-inspired aspects were too distracting for me.

In the process of writing this up I came across the Tide theme which is not supported by OMF but you can install using the Fisher plugin manager. I like this theme quite a bit, in part because of the built-in setup manager that walks you through a customization workflow. This is the setup I have pictured at the top of the article and my new default on my personal and work computers.

Whatever you pick, there is a a fork in the road here because you can't have both OMF and Fisher installed on the same system because they try to edit the same Fish configuration files -- so you do have to pick one.

That's it! Hopefully you have a more functional and attractive setup now or have otherwise learned a little bit more about the tools available to you.

Odds and Ends

It's been a little bit since I've put out a newsletter. I've spent a good portion of that time playing with some the frameworks I wrote about in my last edition including a Dagster MVP built around an Advent of Code challenge that I want to write up at some point.

And while all that has kept me busy the truth is that writing is hard and I've gotten stuck on a lot of the drafts I've started. But I think I'm learning how to write more tightly-scoped newsletters that should be easier to get out the door.

Anyway, here are some odds and ends for you.

Around the Internet:

What I Learned in My First 6 Months on the Data Team at HealthJoy - One of my teammates recently started blogging and wrote about his great article about his first 6 months on our Data Team. He talks a lot about how we're trying to build a modern team around collaboration and data as a product.
This is Why Your Holiday Travel is Awful - A long read on debacle that is Penn Station in NYC, the politics of government projects, and why we can't seem to build anything in America.

Reading:

Go Ahead in the Rain: Notes to a Tribe Called Quest: I finished this wonderful book, part history of ATCQ and part personal memoir of the author as he was growing up with the group. Probably my favorite hip-hop book since I read the 33 1/3 series on Nas's Illmatic.
Thick: And Other Essays: I've been following recent McCarther Genius Award recipient Dr. Tressie McMillan Cottom for years so I'm glad to finally put some money in her pocket. Her first volume of her essays is effecting and profound. She also has a great substack where she talks about everything from her writing process to Peloton to Dolly Parton.

Listening:

I've been bouncing around a lot but lately. Not really loving anything too much but this is what's been at the top of my rotation:

On a recommendation I checked out Benny the Butcher's The Plugs I Met 2 (mostly for the Harry Fraud production)
Floated around some SoCal Lo-Fi pop by Little Monarch
Went down a Chicago blues history rabbit hole listening to Alligator Records first artist Hound Dog Taylor
My personal favorites The Hold Steady put out a pretty decent studio album

6 Python Tools I'd Like To Try In 2021

Tue, 02 Mar 2021 14:43:39 GMT

I recently discovered most of what I've arrived at (in addition to a bunch of other stuff) is covered by Claudio Jolowicz's excellent (and wonderfully illustrated) Hypermodern Python series. My ideal project setup includes "hypermodern" tools like Black, pyenv, and Poetry but also more traditional tools like PyTest and Flake8, build pipelines in Makefiles and GitLab CI/CD config files, as well as more mundane things like README layouts.

At work, I've been slowly adding these things to a template project but this is mostly used as a reference implementation for individual tools that we gradually back port to other projects as time allows. In previous roles, we tried to take this a step further by using a GitHub-style development model to fork new projects off of a base project but this never quite got traction.

I thought about building a separate template project for my personal projects on GitHub but something about it just felt inefficient. I could hear Raymond Hettinger's catch phrase from his PyCon talks in the back of my mind "There MUST be a better way!"

Enter Cookiecutter. From their website:

A command-line utility that creates projects from cookiecutters (project templates), e.g. creating a Python package project from a Python package project template.

This looks very promising and is the top project I want to try this year. I'm an advocate of getting folks out of notebooks workflows and into a proper project format when work starts to move beyond exploratory analysis. Giving everyone a dynamic template to start with is a huge enabler for that. You can even find maintained templates for things like Python packages and data science.

pre-commit

pre-commit is another newish package that feels like an emerging standard to me (checkout the Hypermodern post). Currently, I wrap all my "pre-commit" steps like Black and Flake8 into a Makefile. This works pretty well, but you still have to remember to run a make pre-commit before you push. While this isn't a huge issue in my team's current workflow, I don't like anything that relies on the user having to remember something.

One solution to this issue to write a pre-commit hook in git. These are scripts that are automatically run by git before committing code. I've known about pre-commit hooks for a long time but never quite felt compelled to write my own.

Enter pre-commit. From their website:

We built pre-commit to solve our hook issues. It is a multi-language package manager for pre-commit hooks. You specify a list of hooks you want and pre-commit manages the installation and execution of any hook written in any language before every commit.

I learned about pre-commit sometime last year and it seemed like something nice to get around to eventually. But what I'm now realizing is that people are using pre-commit as a way to share git-hook recipes. For example, instead of showing everyone how to set up their text editors to remove trailing whitespace I can just turn on the trailing-whitespace hook like in this Hypermodern post. This connected the dots for me with cookiecutter as a way to increase standardization and reusability at my job. I think the possibilities really became clear to me when I read about the 3rd project on my list.

pre-commit-dbt

Our data warehouse at work is centered around the dbt package, its recommended design patterns, and even the concept of an Analytics Engineer. I'll be honest, my team has a pretty good handle on dbt which means I'm largely hands-off on this project. But, from our team meetings my engineering sense was telling me that we could be doing more in terms of enforcing things like testing and documentation. So I was excited to connect the dots back to the pre-commit project (maybe framework is a better word here?) in the latest dbt community newsletter.

From the pre-commit-dbt GitHub page:

List of pre-commit hooks to ensure the quality of your dbt projects.

You've got my attention! It continues:

dbt is awesome, but when a number of models, sources, and macros grow it starts to be challenging to maintain quality. People often forget to update columns in schema files, add descriptions, or test. Besides, with the growing number of objects, dbt slows down, users stop running models/tests (because they want to deploy the feature quickly), and the demands on reviews increase.

Perfect, and these are exactly the kinds of code/project quality issues I'm trying to improve on my current team and I think using some of these dbt pre-commit hooks could really help improve the overall quality of our data warehouse as a platform.

And thinking about our data warehouse as a platform leads me to the next interesting project on our list.

Dagster

One of the benefits of doing the Advent of Code puzzles this last December is that it really helped me understand my personal software design preferences. Getting into a daily rhythm of completing a mini-project and then discussing them in a Discord channel with my friends allowed me to refine and articulate some programming ideas I had been taking for granted (link to my code repo). Upon reflection, also I realized that these were ideas that I was failing to communicate effectively while mentoring other team members.

From my point of view, the Advent of Code problems are all essentially mini data pipelines with an input file and a single summary output. This had me advocating to my friends about the merits of highly-decoupled and functional programming paradigms assembled using composition and with strong test coverage (there's a draft of a newsletter just about this).

You can almost directly scale those ideas of composing decoupled functions with no side-effects from toy python pipelines into full data architectures. At a certain scale this argument runs into pushback about the practical (not computational) scalability of microservices architectures. However, I think there's still a lot of value in the concept of highly independent chunks of code that do exactly one thing perfectly.

I was thinking about these types of patterns when I remembered a recent issue of Tristan Handy's Data Science Roundup (Tristan is the founder of Fishtown Analytics which maintains dbt core) talking about a new data orchestration tool called Dagster. From their website:

Dagster is a data orchestrator for machine learning, analytics, and ETL

Continuing:

With Dagster's pluggable execution, the same pipeline can run in-process, against your local file system or on a distributed work queue, against your production data lake. You can set up Dagster's web interface in a minute on your laptop, or deploy it on-premise or in any cloud.

You can read more from one of the project organizers on this Medium post including a some helpful sample code. I've scanned their docs and there seems to be a lot of overlap with how I'm already thinking about pipelines. I'd have to read more to see how this would compare to something like Airflow, Databricks, or Matillin but I could see this being a very cool project. In fact, this is the project that has me the most excited of anything on my list. I see a lot of possibilities for composing various scripts and processes that are starting to pop up in our team's work.

There's even typing:

Dagster models data dependencies between steps in your orchestration graph and handles passing data between them. Optional typing on inputs and outputs helps catch bugs early.

Type hints in Python is something I'm increasing pulling into my work. In working on my Advent of Code puzzles I finally started using mypy and type hints and loved the structure and constraints they added to my code. Unfortunately, everything I've read and tried has led to believe the type hint integration just isn't quite there yet in terms of full integration with Pandas + MyPy. Nonetheless, the next two projects on my list for 2021 are interesting in-and-of-themselves but also because of how they leverage type hints for autogeneration of functionality and documentation.

Typer

I've been writing command line interfaces since the beginning of my career when I was creating image analysis pipelines for the Hubble Space Telescope and I'll always have a soft spot for a solid CLI framework. I started out like everyone just manually parsing sys.agrv, then move to the now deprecated optparse, then argparse, and finally moved on to Click a few years ago which is the modern Python standard and a great tool.

Typer is a newish CLI framework and the self described "little sibling" of FastAPI, the next and final project on this list. From their website:

Typer is a library for building CLI applications that users will love using and developers will love creating. Based on Python 3.6+ type hints

Check out this "Hello world" example:

import typer


def main(name: str):
    typer.echo(f"Hello {name}")


if __name__ == "__main__":
    typer.run(main)

That's it. That's enough to generate a CLI complete with unix style --help flags and everything:

python main.py --help

Usage: main.py [OPTIONS] NAME

Arguments:
  NAME  [required]

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or customize the installation.
  --help                Show this message and exit.

Amazing! This reminds me of the Flask API hello world! Which brings us to our final project I've got on my radar for 2021.

FastAPI

I spent a lot of time early in my engineering career bouncing back and forth between Django and Flask. Both are great projects with their own merits. Django is the go-to batteries-included option but a lot of times my projects ended up needing something much more lightweight. Flask offered us that minimal code footprint (at first) but as our projects scaled I found we had to spend more and more time thinking about which design principles we wanted to impose on our app. Or, to be more honest, thinking about how to reconcile the various patterns we had half-implemented throughout our app.

Some of this is inevitable as projects start to scale but Flask's hands-off nature seems to make the issue especially acute from the get-go. Often times, while I still felt like I didn't want to go full Django with it's CRM and ORM, I did still wish Flask was making more decisions for me.

Since then I've been increasingly interested in how opinionated projects can enforce a certain structure which can help guide your work and this is one of the things that drew to me to FastAPI. From their website:

FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.

FastAPI is Typer's (older?) sibling. The idea is the same, using type hints allows you to more richly define boilerplate for your API. In the same way that Typer generates a command line help interface, FastAPI generates rich Swagger docs.

As that name would suggest, FastAPI is also fast. Actually a subclass of another project called starlette it's one of the fastest Python Frameworks. I don't see myself building too many APIs in the near future but if I do I'm certainly going to reach for this project.

Odds and Ends

Around the Internet:

Welcome to the Center of the Universe: This a long read about the Deep Space Network, the network of satellite dishes that has given us 24/7 contact with all of our deep space missions for over 50 years. It's an incredible feat of engineering and commitment characteristic of our space program. It also had me reminiscing of my first career in astronomy when I was doing overnight shifts of thermal vacuum testing on the Hubble WFC3 camera at Goddard Space Flight Center.

Reading:

Nature's Metropolis: I finally wrapped up this long but fantastic read about the economic history of Chicago and the greater midwest. I learned about the many historic economic connections between the places I've lived, studied, visited, and vacationed, down to individual buildings I've been in. It changed the way I think of my city and the region it's situated in.
Go Ahead in the Rain: Notes to A Tribe Called Quest: I'm looking forward to poet and writer Hanif Abdurraqib's part history and part personal memoir of ATCQ. I haven't done a deep dive on the group's history since a tumultuous 2016 when in a few short months I watched the Beats, Rhymes & Life documentary for the first time, only to learn of Phife Dawg's untimely death a few months later, closely followed by the release of their last album.

Listening:

Chick Corea: It's unfortunate that so recently after losing MF DOOM I have to say goodbye to another treasured musician: jazz pianist Chick Corea. I've been fascinated by Chick's playing since I came across Return to Forever's Romantic Warrior in high school. Here he is with his long-time collaborator Gary Burton in in a 2016 Tiny Desk Concert:

And here he is 1992 at the Tokyo Blue Note with his New Akoustic Band trio.

Chick knew he was on his way out with a rare form of cancer soon after it was discovered so I'll close out this newsletter with an excerpt from his goodbye message to his fans.

It is my hope that those who have an inkling to play, write, perform or otherwise, do so. If not for yourself then for the rest of us.

My Python Documentation Toolchain

Sun, 07 Feb 2021 19:03:47 GMT

Let's start with why we bothered to build a documentation system in the first place instead of just taking ad-hoc notes.

Why The Fuss About Documentation?

I think most people who write software would agree that projects should have at least some documentation and that major projects, like Pandas, should have exhaustive documentation. But for an internal medium-sized project with only a few users isn't just a README file, sane variable names, and a few comment strings more than enough? More to the point, real projects have deadlines. Wouldn't developer effort be better spent getting code out the door and not on comprehensive documentation?

In my opinion, for code to be truly high-quality it needs to be maintainable. Solid design principles can make it easier to understand what your code is doing. Tests can ensure your code is behaving as you intended. But documentation helps you understand why you are doing what you are doing. It adds context, and data work in particular is heavily contextual relying on nuanced understandings of upstream and downstream data flows.

While cutting corners on documentation is a pragmatic approach, and certainly something I've done myself, I don't think it works well in the long run. In my experience it's just another form of technical debt. In the extreme but not uncommon case of a single project maintainer years of system knowledge can be lost when that individual leaves the organization.

These are well-known arguments but I want to push this idea of the value of documentation beyond just creating maintainable code. In a modern data team it's common to have folks that are new to writing software in the form of a project and not just one-off scripts or notebooks. This is a transition I can't encourage enough but one that can be initially very confusing for the practitioner. Forcing these team members to slow down, think about what they're doing, and be explicit about their intentions and understanding in the documentation, is beneficial not only for code quality but also for their own professional development.

Let me close this section by saying that I've never met a technical team, mine included, that was satisfied with their documentation -- or their testing for that matter. But if you can agree as an organization that documentation is a critical part of delivering long-term value then you can at least strive to make sure your documentation is constantly improving. But to do that at scale you need a solid repeatable structure, and for that you need tools.

Why So Many Tools?

Even if you buy my argument about documentation being important, why do we need so many tools? Can't you just … keep the documentation up-to-date? My argument is that setting up a toolchain for your documentation gives your project structure and a direction, just like with a testing or a deployment system. As long as you have structure and direction you can iterate and evolve. Without it, you're lost and "documentation" will be scattered throughout your project, often completely outdated and irrelevant.

I see more parallels here between documentation and testing. One of the main arguments in favor of testing is that beyond the immediate value of tests as code validation is that tests also encourage people to write code that is testable by design. That is, code is intended to be intelligible and verifiable and which is overall better code. I think the same thing happens with documentation. When documentation becomes part of the structure in your projects, people will be inherently concerned with communicating all the ideas surrounding their code and their work will start to become more documentable (and intelligible) by design.

But that structure needs to start from the ground up.

Documentation with Python Docstrings

Python has the concept of a docstring, which is a multiline string at the start of a module, function, class, or method. If you're not familiar it looks like this:

def raise_to_power(base, exponent):
    """This is a docstring"""
    # This is a comment
    return base ** exponent

Like their name says, docstrings are intended to hold documentation. This might seem like a trivial feature at first; after all, we can also put important information in the README or comments. So why should we use docstrings?

I've found that you get the most out of tools when you lean into their opinionated design decisions. Python as a language has made an intentional decision that documentation belongs in docstrings and as a result tools and best practices have been built up around this assumption. Aligning to these conventions means you can take advantage of all these additional tools. For example, if you load the previous function into a Python REPL you'll find it already knows what to do with the docstring:

>>> help(raise_to_power)

Help on function raise_to_power in module __main__:

raise_to_power()
    This is a docstring

You can get the same response with the ? operator in IPython or a Jupyter notebook. This is just a hint at what we can do with information once it's collected into docstrings. But, to do more we need to standardize our docstrings so other tools can make use of them.

Docstring Guidelines with the Google Style Guide

There are a couple of different conventions for Python docstrings but I strongly prefer the conventions from the Google Python Style Guide. I'm a big fan of the Google syntax because it strikes a balance between human and machine readable; similar in my opinion to Markdown.

We'll leverage the fact that this format is machine readable when we start building webpages for our documentation but first let's look at a sample. Here's the same function we just saw but now with a Google style docstring:

def raise_to_power(base, exponent):
    """
    Raise a base to an exponent, i.e. base ** exponent.

    Args:
        base (int): The base value
        exponent (int): The exponent value

    Returns:
        int: The output of the calculation
    """
    return base ** exponent

I think most people will agree that this is readable but some people might question if simple functions should be documented this extensively. Have we really added anything to our project in this case? I would argue yes.

First of all, never overestimate the extent to which you have made assumptions about the context and clarity of intent in your code. In the case of our example, I would want this docstring to explain why we are wrapping functionality that already exists in the Python standard library into a function. There's a useful side effect of this, forcing myself to write out assumptions in docstrings has become a form of rubber duck debugging for me that has frequently caused me to change my mind about a particular implementation.

Second, as a practical matter, I already have enough decisions to make in writing my code. Having no input on when and how I add docstrings actually frees me to focus on the decisions that do matter. And as we're about to see, having a uniform documentation standard not only frees me to think about what matters, it allows us to leverage automation.

Automation with darglint, Flake8, and Merge Templates

One of the pain points about maintaining docstrings is that it's hard to keep them up-to-date. One of the reasons I like to keep documentation right in the code is that it make it easier to remember to update both. But having to remember something is always an unreliable plan. Fortunately, we can lean on some automation to help.

We use GitLab CI/CD in the cloud and Makefiles locally for all our code quality checks including testing and linting (though I have my eye on migrating the later to the Python pre-commit tool). For linting, we first run auto-formatting with the Black code formatter and then do a pass with Flake8. To make sure our docstrings are accurate we use a tool call darglint which can run as a Flake8 extension. Darglint ("document argument linter") checks to make sure that your function call signature and docstrings match. If you change one without updating the other Flake8 will fail.

It's worth taking just a moment to talk about tool selection here. Darglint is a small project that seems pretty stable. Nonetheless, it's much earlier in its development that anything else in our projects and I've been burnt before by teammates including trendy new tools into our code, only to find a few years later that we have a critical dependency on a seemingly abandoned piece of technology.

A key consideration for me when evaluating an architecture or tool is making sure I have a plan, or at least the ability, to migrate off of it. In this case, if we had to rip out darglint tomorrow, we would loose the ability to lint our docstrings which would only be an annoyance at worst and only require changing a few lines in a some config files. I decided this was worth the risk to help maintain our docstrings.

Another way we keep the docs up-to-date is to use GitLab merge templates which includes the ability to create a blocking markdown checkbox for the developer to confirm they've updated the docs. This is a helpful nudge but it doesn't help if you forget what you changed or don't realize the scope of the impact of your changes. Ultimately, I don' think there is a solution for that problem. But I do think making high-impact docs that people want to use will help you find the inevitable documentation problems sooner.

So let's make our docs high-impact by building some webpages from our docstrings.

Documentation Webpages with Sphinx

At this point in the process if we've written our docstrings with even a moderate amount of care we now have an important body of knowledge about how our code works. And because we've written them using strict standards we can now export our documentation into webpages. To do this we use a popular Python library called Sphinx which generates static HTML pages. Sphinx was developed to create the documentation for the Python core library so the formatting out-of-the-box looks pretty similar to the historical python.org docs.

You can build a perfectly usable documentation page using just vanilla Sphinx; but with a few plug-ins we can make a much nicer site with much less work. First, we add the autodoc extension which will automatically find and collect all the docstrings in our package. Then we add the viewcode extension which will allow us to click through to the raw source code within the doc pages themselves. The napoleon extension allows Sphinx to parse our Google style docstrings as opposed to Sphinx's default reStructured text format. Finally, and this is just personal preference, we can set our HTML theme to the ReadTheDocs theme.

Like I said, this is not intended to be a tutorial so I've skipped a few steps such as setting up your conf.py file, your base index.rst file, and making sure your package is installable and not just a script. Fortunately, there are plenty of resources on these steps. Setting this stuff up is honestly a pain, but you only have to do it once and after that it's pretty stable. And once you've figured it our for one project it's easy to repeat for all your projects.

So now that we have a nice looking document on our local machine, how do we publish it? After all, telling someone to RTFM just doesn't have the same bite to it when they have to go build the docs from source on their local machine.

Going Live with GitLab Pages

We use GitLab as our version control platform which has a Pages feature that allows you to host static webpages from pipeline build artifacts (GitHub has similar functionality). The GitLab examples include a Python + Sphinx example that you can pretty much just cut and paste and into your CI/CD and you've now got live automatically documenting project.

Nice work!

Closing Thoughts

It's worth pointing out that something interesting happens at this point. When docs are published and automatically updated I've noticed they become a lot more real to teams. People are proud of them and want to keep the up-to-date. Questions get directed to the docs, the docs get updated to be more helpful, and you start to get a feedback loop.

There were a number of other directions I wanted to take this article as I was writing it including building a documentation ecosystem by connecting the other self-documenting services in our cloud warehouse such as Looker and DBT. Or how by following the example of the FastAPI project data pipelines could use Python type hints to build even more automatic documentation. However, I eventually realized these topics were way too much for a single article so I split them out into drafts for future articles. I'm quickly (re)learning that in writing just like in programming, scoping and structure are everything.

Odds and Ends

Around the Internet:

Overdiagnosis (The Limits of Inference) - If you love data and philosophy you should check out my friend Clare's substack. Her last article in particular on false positives and what they mean for widespread COVID testing is an excellent reminder of the importance of core statistical principles.

Currently Listening:

Portrait of Wes (Wes Montgomery Trio): A friend of mine recently turned me on to under-appreciated Indiana jazz guitar player Wes Montgomery. Opting for Melvin Rhyne on organ instead of a bass player, his trios have an especially interesting sound. Check out the up-tempo cover of Art Blakey's "Moanin'"

Currently Reading:

Death's End / Three-Body Problem Trilogy: Just finished up this series. If you like your science fiction to take on a truly cosmological scale, are pretty comfortable with plot devices that rely heavily on physics concepts like the speed of light or the strong atomic force, and can handle characters that act more like platonic ideas of humans than actual people - then this is the trilogy for you. It's also interesting to think about how the author's Chinese cultural perspective comes through in his writing. If that last bit seems interesting I'd recommend China in Ten Words.
Nature's Metropolis: Absolutely blown away by this economic and business history of Chicago - I see why it was a Pulitzer Prize finalist. For example the chapter on grain could not sound more boring and yet ends up being an amazing primer for all the current stock market upheaval around GameStop.

Welcome!

Thu, 31 Dec 2020 23:50:10 GMT

It seems that every 4 years I get the itch to write a tech blog. Recently, I was getting ready to breath some life back into my long dormant static web blog when my friend informed me it's 2021, blogs are out and newsletters are back in -- time is a flat circle.

Now it's the last few hours of 2020 and some of my friends have already begun subscribing to the empty URL where I have indicated there will one day be a newsletter. This seems like as good a reason as any to let go of my perfectionism and kick this thing off.

So … here we go!

Who's Writing This Thing?

Whether you just wandered in from Twitter or if I've known you for years, I thought it would be good to start things off with some background about myself to give you an of idea where I'm coming from with this newsletter.

My parents are both biostatisticians so I grew up around discussions about how statistics isn't magic. I went to UW-Madison where I majored in Astronomy and Math. I spent the first 7 years of my career working for the Hubble Space Telescope at STScI cutting my teeth as a self-taught software developer and writing data analysis pipelines. Following that I spent 4.5 years at a dark web information security startup. I was their first hire starting as an engineer, then VP of Engineering, and finally VP of Technical Products. Since 2019 I've been the Director of Data at HeathJoy, a Chicago-based healthcare startup, where I lead our Data Team. In my current role my projects span traditional analytics, data engineering, and data science as well as management across those fields (my opinions here are not the opinions of my employer etc.).

My career has always spanned data and software. I was always the engineer who cared the most about benchmarks and monitoring or I was the analyst who cared the most about software and automation. Throughout my career I've been deeply interested in the craft of writing quality software and producing data analysis that's fundamentally sound. I tend to ask why we're doing things which for better of worse has ended up with me leading new projects or teams. I've learned some things about how to build and work in changing organizations.

What's the Plan?

I want to write about a mix of software, data, technology, math and science, and how to lead teams that work in those areas. This isn't strictly a software newsletter but there will be some posts -- wait no, _issues -- _that focus pretty heavily on software development. But within that the emphasis will be less on tips and tricks and instead focus on the theme of building stable maintainable software. Overall though, there should be plenty of non-software topics.

So what exactly does that look like? Here are some drafts I have on the back burner:

What I learned doing Advent of Code 2020 with 5 of my friends
How I try to create reusable software project structures myself and the teams I lead
How to structure data team interactions with stakeholders
A long read about chess, deep learning, and sports
Some grumpy thoughts about deep learning in industry
Side projects to measure my home's internet connection speed with a RaspberryPi, generate perfect picture hanging arrangements, and analyze my chess performance
General odds and ends about tech, data, and some bits of culture to round things out.

But, to quote the title of another substack I read I am absolutely going to bail on this in a month.

Odds and Ends

Around the Internet:

I've started using SpotMenu for Spotify on OSX to display current track information and as mini-controller in my menu bar. It's such an obvious tool, I'm amazed I've gone so long without it. It makes me little bit nostalgic for WinAmp.
After getting tired of my Solarized Dark color scheme as my terminal and editor color scheme I've switched over to Dracula (created by a fellow Brazilian!).

Current Reading:

Nature's Metropolis: Deepening my love affair with Chicago after moving back here in 2017. This was the most recommended book in a McMansion Hell Twitter thread asking for Chicago book recs.
Death's End: Wrapping up the Three Body Problem trilogy. My love/hate relationship with this series has tilted towards love as the themes have taken on a cosmological scale.
Regression And Other Stories: I've been looking for a text that balances theory and practice like this for a while. I'm sure I'll be writing more about it in the future.

Currently Listening:

Getting into Knives - The Mountain Goats: Trying to catch up on 2020 releases I missed.
Operation: Doomsday - MF DOOM: While writing this I was deeply saddened to learn that one of my favorite MCs, MF DOOM, has passed away.

On Doomsday!, ever since the womb 'til I'm back where my brother went, that's what my tomb will say
Right above my government; Dumile
Either unmarked or engraved, hey, who's to say?

Rest in Power DOOM

An Intro to Regex

Wed, 10 May 2017 00:00:00 GMT

How to Talk to your Analysts about Regex

Alex Viana - 05/05/17

The following is a presentation I gave to some non-techincal collegues at work to help them get started on reading and writing regex expressions. Sharing in case others find it useful as well.

Regular Expressions (Regexs) are a way of defining search patterns that can be applied to text. Throughout this document regexes will be written in Python syntax, quoted and preceded by a letter r, like this r"a regex pattern", to differentiate it from raw text, like this "some raw text".

A regex can be as simple as a literal match like this:

r"match this string exactly"

Which would only match against the literal text "match this string exactly". By assigning special properties to certain characters regex can also search for text that matches complicated patters like this:

r"10.(6[4-9]|[7-9]\d|1\d\d|2[0-4]\d|25[0-5]).(1?\d\d?|2[0-4]\d|25[0-5]).(1?\d\d?|2[0-4]\d|25[0-5])"

This regex would match against an IP address range that begins with 10., has a second octect in in the 64-255 range, and accepts any values for the third and fourth octets.

Complicated regex patterns like this can be pretty inscrutable even when they are working correctly and hence the old joke:

You have a problem. You try to solve it with Regular Expressions. Now you have two problems.

Still, regexes are very useful and you can get a lot of mileage out of just a few parts of the regex syntax.

Testing Regexes

I highly recommend using the Regex101 website to test regexes, both to understand existing expressions and to test expressions you are writing. Try it out to play with the example searches in this tutorial.

Special Characters

As we saw, matching string literals, i.e. patterns that have only one possible match, is trivial:

Regex: r"this" Matches: "this" Misses: "that"

The regex syntax is largely built around assigning special properties to punctuation marks. As a result, if you ever want to literally search for any of these characters you will have to escape those characters with a backslash (\). So if you wanted to search for a backslash you would have to escape the backslash like this:

Regex: r"\" Matches: "" Misses: "Something else"

Pretty meta right? Similarly if you wanted to search for "google.com":

Regex: r"google.com" Matches: "google.com" Misses: "anything else"

Replacing a Single Character

String literals without special characters aren't very useful thought, the power of regex comes from matching patterns. One of the simplest regex patterns is a period (.), which means "match any single character in this position". For example:

Regex: r"1.3" Matches: "123", "1.3" Misses: "13"

Because the period is a special character, as we saw in the last section you would have to escape it if you literally wanted to search for "1.3".

Regex: r"1.3" Matches: "1.3" Misses: "123"

We can constrain the possible characters we match on by using square brackets ([]). For example if we only wanted to match on the numbers 1, 2, or 3:

Regex: r"1[123]3" Matches: "113", "123", "133" Misses: "13", "143"

A common pattern in regex is matching against any letter. This can be done by specifying a range like this [1-3]. So any lowercase letter would be [a-z], uppercase would be [A-Z], both would be [a-zA-Z], and adding letters would be [a-zA-Z0-9].

Regex: r"[a-zA-Z0-9]" Mathes: Any uppercase letter, lowercase letter, or number Misses: "!!!"

Replacing Multiple Characters

Using just square brackets will match only character one time, we can also match against a character or set of characters repeatedly by using the curly braces {}, where the number in the curly braces indicates the number of times, or range of times, to match.

For starters we can rewrite our last regex which, matches only once, as r"1[123]{1}3" and produce the same search results (try it!).

Repeating the set match twice then looks like this:

Regex: r"1[123]{2}3" Matches: "1113", "1123", "1333" Misses: "1243"

We can also repeat a variable number of times. To repeat a set between x and y times looks like {x,y}. For example this regular expression will match from the set [123] either once or twice (but not zero times).

Regex: r"1[123]{1,2}3" Matches: "113", "123", "133", "1113", "1123", "1333" Misses: "13", "143", "1243"

Word Boundaries

Another useful pattern is defining the boundaries of what you want to match against. You can do this with any string literal like we did in the section on replacing a single character where the search terms were all bound by a 1 and a 3. You could bound a match with whitespace using a space ``. More generally you can use the \b for any word break. For example

Regex: r"\bthis\b" Matches: "that this that" Misses: "thatthisthat"

Logical Operators

The vertical bar (|) is a logical or operator. For example:

Regex: r"tyler|perry" Matches: "Katie Perry", "Steve Tyler", "Tyler Perry", "Perry Como" Misses: "something else"

Next Steps

There are a lot more you can do with regexes and a lot more syntax to learn but hopefully this was a useful starting point.

If you really want to dive into the deep end check out one of my favorite regex problems, trying to write a regex that matches any valid URL

Congratulations, you now have two problems!

What The F*ck Is The Internet Vol. 1 - Bits, Encoding, Packets, and Protocols

Fri, 17 Mar 2017 00:00:00 GMT

I organize a weekly series of tech talks at work. It's a good way to stay up on what technologies people are using in and out of the office. Recently, I've started presenting at these talks on various computer science fundamentals I was never exposed to in college, using the presentation as a driver to read up on these topics.

Last winter I read a short book on cryptography and gave a presentation on various historical cyphers as a way to provide a context for modern cryptographic techniques (there were no slides). The theme was basically, "we tied this and it didn't work for this reason so now we do this instead".

I followed that up a few weeks ago with a presentation on basic networking. And when I say basic, I really mean from the from the ground up starting with binary encoding and work up only as far as protocols. Because we sometimes have non technical members of our team sit in on these talks I made en effort to really walk through all the fundamentals step by step. Here are the slides:

Hopefully, I'll be able to follow this up later in the spring with another presentation digging into how network messages are routed using IP address and sockets.

Meanwhile, as my experience of being a self-taught software engineer becomes increasingly common it's good to see new resources springing up as more developers try to fill in our knowledge gaps. I particularly liked this site which popped up on Hacker News this week: https://teachyourselfcs.com/

Special Methods in Python

Wed, 15 Mar 2017 00:00:00 GMT

This post is intended to accompany a presentation I gave to the Baltimore Python Meetup on Wednesday, March 15th 2017.

This post assumes some familiarity with object-oriented programming as it relates to classes and inheritance in Python.

The purpose of this post is to explore Python's special methods, also called "magic", "built-in", "double underscore", or "dunder" methods. A common example is the __init__ method. These methods are considered special because they are referenced by Python to determine class behavior. Note that the double underscore notation is just a naming convention for indicating which methods Python considers special. Adding a double underscore to any other method (e.g. __foo__) doesn't give it any special properties.

Special Methods

To explore how these methods work we're going to pretend we're building a Scrabble-type game. A natural object for this type of project would be an object that encapsulates a word and it's score. Let's start with a super basic class:

In [1]: class ScrabbleWord():
  ...:     pass
  ...:
In [2]: my_word = ScrabbleWord()
In [3]: my_word.word = 'cat'
In [4]: my_word.score = 5

We start with an "empty" class, and manually assigned it two attributes, a word, and a score which I calculated by consulting a table of scrabble letter values. Using Python's dir() function we can inspect the attributes and methods on the my_word object.

In [1]: dir(my_word)
Out[1]: ['__class__', '__delattr__', '__dict__', '__doc__',
  '__format__', '__getattribute__', '__hash__', '__init__',
  '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
  '__setattr__', '__sizeof__', '__str__', '__subclasshook__',
  '__weakref__', 'score', 'word']

At the very end, we can see the score and word attributed I added as well as about a dozen other methods and attributes which all Python classes have. These other attributes, which all happen to be special attributes and methods, came from the Python base class. These are the minimum set of methods that Python will need to call to be able to use an object so it injects them into all its classes. Let's start looking at what we can do with these special methods.

The init Method

One of the first things we can change about our ScrabbleWord class is that we have to manually add the the word and the score. Since we will always want to do this for every class instance we should just make it part of the object creation.

To define the way an object is created (also referred to as "instantialized") we can create an __init__ method. If we look at the output of the dir() function though we can see that ScrabbleWord already has an __init__ method. Because we are replacing an existing method inherited from the parent class, this is called operator overloading. In this case the parent, implicit in our class definition, is the Python base class for all objects.

As it turns out, all classes have an __init__ method. This is because Python always uses the __init__ method to create class instances. This is an example of what we mean when we say __init__ is a "special" method; these methods are referenced by Python to determine class behavior. Our replacement looks like this:

class ScrabbleWords(object):

    letter_values = {
        'a': 1, 'b': 3, 'c': 3, 'd': 2, 'e': 1, 'f': 4, 'g': 2, 'h': 4,
        'i': 1, 'j': 8, 'k': 5, 'l': 1, 'm': 3, 'n': 1, 'o': 1, 'p': 3,
        'q': 10, 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 4, 'q': 10,
        'w': 4, 'x': 8, 'y': 4, 'z': 10
    }

    def __init__(self, word):
        self.word = word
        self.score = sum(
            [ScrabbleWords.letter_values[letter] for letter in self.word]
	      )

Our __init__ method now takes an argument called word which it assigns as an attribute to the class. I added a class attribute called letter_values that is accessed as an attribute on the class, i.e. ScrabbleWords.letter_values. This attribute is a dictionary that maps letters to their Scrabble score. __init__ uses this dictionary in a list comprehension to turn the input word into a list of letter scores, sum those up, and then assign the sum to the score attribute. Calling the class now looks like this:

In [1]: my_word = ScrabbleWords('cat')

In [2]: my_word.word
Out[2]: 'cat'

In [3]: my_word.score
Out[3]: 5

Not bad, we now provide the word at the same time as we create the object and both the word the calculated score are saved on the object. Let's see what else we can do.

The repr and str and Methods

Let's take a look at what happens when we try to inspect or print our class instance [^1].

In [1]: my_word
Out[1]: <__main__.ScrabbleWord instance at 0x1084c80e0>

In [2]: print(my_word)
Out[2]: <__main__.ScrabbleWord instance at 0x1084c80e0>

This tells us that the my_word object is an instance of the ScrabbleWord class from the __main__ namespace and is located at memory address 0x1084c80e0. While this is all correct it's the default representation of any Python object and that information is not as useful as it could be for building our Scrabble game

We can make the results more descriptive by overloading the __repr__ and __str__ methods to fit our use case. This could look like this.

    def __repr__(self):
        return '{}: {} points'.format(self.word, self.score)

    def __str__(self):
        return '{}: {} points'.format(self.word, self.score)

Now interacting with our object will look like this:

In [1]: my_word
Out[1]: cat: 5 points

In [2]: print(my_word)
cat: 5 points

You can read more about the difference between the __repr__ and __str__ methods in this Stack Overflow Question.

The eq and Other Operator Methods

At some point we're probably going to want some type of score optimization algorithm for our game, maybe for a simple computer opponent. To do that it would be helpful to have a way of comparing the value of two ScrabbleWord objects. Right now we can do that manually by explicitly comparing the score attribute.

In [1]: cat_word  = ScrabbleWords('cat')

In [2]: dog_word = ScrabbleWords('dog')

In [3]: cat_word
Out[3]: cat: 5 points

In [4]: dog_word
Out[4]: dog: 5 points

In [5]: cat_word.score == dog_word.score
Out[5]: True

In [6]: cat_word.score > dog_word.score
Out[6]: False

But if we start to implicitly compare the words Python doesn't know we mean to compare the score and we get unexpected behavior.

In [5]: cat_word == dog_word
Out[5]: False

In [8]: cat_word > dog_word
Out[8]: True

It turns out we can define the behavior of these comparison operators ==, !=, <, >, <=, and >= using the __eq__, __ne__, __lt__, __gt__, __le__, and __ge__ respectively. If we want to make our comparisons implicitly use the score attribute to prevent us from having to write it out every time our definitions could look like this:

def __eq__(self, y):
    if self.score == y.score:
        return True
    else:
        return False

def __ne__(self, y):
    if self.score != y.score:
        return True
    else:
        return False

def __lt__(self, y):
    if self.score < y.score:
        return True
    else:
        return False

def __gt__(self, y):
    if self.score > y.score:
        return True
    else:
        return False

def __le__(self, y):
    if self.score <= y.score:
        return True
    else:
        return False

def __ge__(self, y):
    if self.score >= y.score:
        return True
    else:
        return False

Now using our object would go like this:

In [1]: cat_word == dog_word
Out[1]: True

In [2]: cat_word != dog_word
Out[2]: False

In [3]: cat_word < dog_word
Out[3]: False

In [4]: cat_word > dog_word
Out[4]: False

In [5]: cat_word <= dog_word
Out[5]: True

In [6]: cat_word >= dog_word
Out[6]: True

It's worth pointing out what's happening here. Taking the __eq__ method as an example. Where Python sees the == operator between two instance of our ScrabbleWords class it'll evaluate the __eq__ method to determine the result.

But looking at our code and you can see the comparison being performed is self.score == y.score which is something like <int> == <int>. So what determines the answer of that statement? Because everything in python is an object, including integers, Python uses the integer classes __eq__ method to evaluate this.

The iter and len and Methods

When we play a word on the board in Scrabble we might care about how long a word is (to see if it'll fit on the board) and we might need to look at a word letter-by-letter (to evaluate things like bonus letter scores). If we try to do either of these things with Our class in it's current state we'll see neither are not supported.

	In [1]: len(cat_word)
	---------------------------------------------------------------------------
	AttributeError                            Traceback (most recent call last)
	<ipython-input-48-7ee5d4e3efb8> in <module>()
	----> 1 len(cat_word)

	AttributeError: ScrabbleWords instance has no attribute '__len__'

	In [2]: [letter for letter in cat_word]
	---------------------------------------------------------------------------
	TypeError                                 Traceback (most recent call last)
	<ipython-input-49-b747de29668a> in <module>()
	----> 1 [letter for letter in cat_word]

	TypeError: iteration over non-sequence

Again we can get around this by directly accessing the attribute we what to work with, in this case the word attribute.

In [1]: len(cat_word.word)
Out[1]: 3

In [2]: [letter for letter in cat_word.word]
Out[2]: ['c', 'a', 't']

But, following the theme of this post, we can do this implicitly with the special methods, in this case the __len__ and __iter__ methods.

def __iter__(self):
    for letter in self.word:
        yield letter

def __len__(self):
    return len(self.word)

Now our class works like this:

In [43]: len(cat_word)
Out[43]: 3

In [44]: [letter for letter in cat_word]
Out[44]: ['c', 'a', 't']

Conclusion

Python special methods become increasingly important as you start working on more involved projects. Either you will be writing your own libraries, in which case these are common "pythonic" programming patterns. Or even if you are leveraging existing libraries, at some point you will need to understand exactly what some special methods are doing in order to understanding some bug or quirk in the library.

Just doing some quick poking around on the github repos for some of my favorite open source projects turns up a couple of examples of classes with lots of special methods in the wild: BaseRowProxy in SQLAlchemy, SiteRegistry in Astropy.

[^1]: Note I'm using the Python 3 style print statement. To use this in Python 2 run from __future__ import print_function

Setting Up Pelican with GitHub Pages

Sat, 25 Feb 2017 00:00:00 GMT

This is a short post about how I set up my pelican project to deploy with GitHub pages. Specifically, I want to version control both the output static pages that are being served as well as the source Markdown files used to generate the output. What I ended up isn't novel but it did take me a while to wrap my head around and wasn't highlighted in any of the documentation or blog posts I found so I wanted to share. Hopefully, it can save you some time.

Introduction

Pelican is a static blogging tool written in Python that generates static webpages from Markdown. GitHub Pages is a site hosting feature from GitHub that will serve the content of a repo in your account named $USERNAME.github.io to that same URL - super easy. This makes for a great blogging workflow, just write some Markdown, peep the output, commit, push, done! The only complication in setting up this workflow is that you project actually contains two different types of content, the source and the output, which you want to store separately.

The Problem

The problem I ran into is that when you run the pelican quickstart command you get a file tree that looks roughly like this.

   ├── Makefile
   ├── content/
   ├── output/
   ├── pelicanconf.py
   └── publishconf.py

With most new projects you just initialize a new repo at the root level and go. But, I didn't want everything in the same place. I didn't want the source Markdown files, config files, etc. to be served to my GitHub pages site, I only want the contents of the output/ folder. At the same time though, I do want to keep track of the source Markdown files. These are critical if I ever want to regenerate the output files to do something like fix a typo or change the site theme.

What's worse is that I already had two repos set up like this on GitHub from when I was last actively blogging. But over the course of the 3 years I had forgotten Pelican workflow, missed a few releases, and changed laptops so I was struggling to remember how to reconnect them. I tried playing around with having two parallel projects with GitHub submodules, git branches, symbolic links, and shell scripting to sync the files over but all of these felt really hacky. Finally, I remembered what I had done.

My Solution

What I ended up doing was cloning my pages git repo into the content/ folder and the source file repo into the output/ folder.

   ├── Makefile
   ├── content/
   │   ├── 2013/
   │   ├── 2014/
   │   ├── 2017/
   │   ├── images
   │   ├── pages
   │   └── .git
   ├── output/
   │   ├── index.html
   │   ├── ...
   │   └── .git
   ├── pelicanconf.py
   └── publishconf.py

It seems obvious in hindsight but because it's a break from the traditional model of one repo per project that it took a while for me to think of it.

Sources

So initially, about 4 years ago, I modeled a lot of my pelican configuration and setup around this post by Jake Vanderplas. Rereading it now I recognized lots of the configuration decisions I had made in the past but couldn't find anything about how he set up his repos. Finally, I think what I must have done all those years ago is looked at his github source for his blog and copied the pattern from there.

Actually, what Jake does is better than my setup. He has a git repo at the root level so all his config settings are saved, with a .gitignore file the ignores the output/ directory which I'm assuming contains another git repo which he uses for deployment. Nested git repos, neat!

Like I said, this isn't anything most people couldn't have figured out on their own eventually it took me longer than it should have so hopefully this saves you some time. s

Exploring Python's 'in' operator

Fri, 17 Feb 2017 00:00:00 GMT

I spent 12 hours on a plane this week traveling between the East Coast and San Fransisco for the RSA conference. I've appreciated the way flying forces me to disconnect and spend a little bit more time on side projects such as this blog. On this particular flight picked up an idea I've been wanting to play with for a while, overloading Python's special object methods.

Magic Methods

These are the object methods that begin with a double underscore such as __init__ and are also known as "magic", "double-underscore", or "dunder" methods. These methods define a lot of the inherent behavior of Python objects, including the Python built-ins such as lists and dictionaries. These methods also define how objects interact with operators such as + (the __add__ method) and <= (the __leq__ operator).

Because so much of how Python works is wrapped up in these method I figured messing with them would be a good way to gain a deeper understanding of the language. So I started making little toy classes that implemented these methods in weird ways and everything was going as I expected until I got to the __contains__ method. The __contains__ method is the method used to check membership using the in operator. For example:

In [1]: 'foo' in ['foo', 'bar']
Out[1]: True

In [2]: ['foo', 'bar'].__contains__('foo')
Out[2]: True

In [3]: 1 in ['foo', 'bar']
Out[3]: False

In [4]: ['foo', 'bar'].__contains__(1)
Out[4]: False

In each case the __contains__ method of Python's built-in list object is being called to evaluate the in operator. As it turns out this is an over simplification. I realized this when I took my first pass at building my own __contains__ method and I did something a little weird:

class MyObject(object):

    def __contains__(self, y):
        return '__contains__'

This is a non-sense method, returning a string instead of a Boolean is unexpected behavior and doesn't make sense in the context of evaluating an object membership. But I just figured it would always evaluate to False. A little more exploration showed that there was something special about the in operator.

In [1]: mo = MyObject()

In [2]: mo.__contains__(1)
Out[2]: '__contains__'

In [3]: mo.__contains__(1) == True
Out[3]: False

In [4]: 1 in mo
Out[4]: True

I was expecting 1 in mo to evaluate to something like mo.__contains__(1) == True which evaluates to __contains__ == True which should be False. Instead I'm got True, so there was something I didn't understand going on.

A little but of searching on StackOverflow led me to this hint:

First, in always casts the result of __contains__ to a bool

So this answers my question but doesn't really teach me much more about Python. This is the beauty about being on a plane. Normally, if I ran into something like this at work I would just take this answer at face value and moved on. But instead being stuck on a plane I had the time and incentive to go a little further down rabbit hole and read the cpython source.

Reading the Source

I've only looked at the cpython source code a few times before this. I've never written a line of C in my life but I've been programming long enough that I can kind of squint and make out the flow of most things.

It didn't take me too long to get oriented enough to find an example of how the __contains__ object is implemented in an function called list_contains in the listobject.c file. This function iterates over the object contents calling the c function on each one.

The comment on the PyObject_RichCompareBool function reads:

> /* Perform a rich comparison with integer result.  This wraps
   PyObject_RichCompare(), returning -1 for error, 0 for false, 1 for true. */

OK so our casting as of everything as a boolean must happen in here. Digging in a little more this function drops down into the PyObject_IsTrue function. And here we finally have our answer. PyObject_IsTrue checks if the input is True, False, None and then a handful of sequence types. If none of these are true the function returns 1, which cascades up to __contains__ and is then interpreted as True.

Writing a FITS File Bigger Than Your Memory

Tue, 08 Apr 2014 00:00:00 GMT

<center>I've listed Erik Bray(GitHub) as a co-author on this post. Erik is one of the PyFITS developers and this post was born out of an email chain where he explained most of what follows to me, several times.</center>

"I’ve always thought that one of the the great things about physics is that you can add more digits to any number and see what happens and nobody can stop you."

Randall Munroe, What If?

I've been working a lot lately on my HST WFC3 PSF project and recently had to solve a challenging scaling problem that forced me to deal with the hardware limits of my machine. I needed to create a FITS file containing a 4,000,000 x 11 x 11 data cube [^1]. This ends up being more than my 16 GB machine can handle and it resulted in paging to the virtual memory which killed performance. As I was trying to find a solution to this problem a play on Randall Munroe's quote from the beginning of this post kept popping up in my head:

"The annoying thing about writing software is that people can just add zeros and break everything and you can't stop them."

So even thought it was exciting that my dataset had grown to the point that it couldn't all fit in memory at once, I was now faced with the problem of how to create a FITS file from a NumPy array that's too big to fit in memory?

Reading the Docs

I work with FITS files using the Astropy io.fits module. For those of you familiar with PyFITS you'll find that the code in Astropy has been wholly migrated over from PyFITS so the functionality is currently identical. So much so that you can still use the PyFITS docs to understand Astropy's io.fits. This is great because the PyFITS FAQ explicitly answers this question: How can I create a very large fits file from scratch?. Go ahead and read that section.

With that FAQ as a starting point I ended up with this code snippet:

data = np.zeros((1, 1, 1), dtype=np.float64)
hdu = fits.PrimaryHDU(data=data)
header = hdu.header
header['NAXIS1'] = 11
header['NAXIS2'] = 11
header['NAXIS3'] = 4000000
header.tofile(fits_file_name, clobber=True)
header_length = len(header.tostring())
data_length = math.ceil(11 * 11 * 4000000 * 8 / 2880.0)
with open(fits_file_name, 'rb+') as fobj:
    fobj.seek(header_length + data_length  - 1)
    fobj.write('\0')

The FAQ got me 80% of the way there and Erik helped me connect the dots. It's worth walking though this for this last 20% as well as an explanation of what exactly is going on.

Hacking the Header

data = np.zeros((1, 1, 1), dtype=np.float32)
hdu = fits.PrimaryHDU(data=data)
header = hdu.header
header['NAXIS1'] = 11
header['NAXIS2'] = 11
header['NAXIS3'] = record_count
header.tofile(fits_file_name, clobber=True)

To being with I create a dummy NumPy array just to get the FITS dimensionality right. As we keep going you'll see why we didn't just create the HDU with an array the size of our expected output in the first place. Then I create a PrimaryHDU instance with that NumPy array. Under the hood the fits module is using this to set up some basic elements of the FITS file format. But this is all being done in-memory, I haven't written anything to the disk yet. Next I changes the NAXIS keywords required by the FITS standard to match those of our expected output and not those of our dummy NumPy array. Changing this keyword doesn't do anything to actually change the dimensions or size of the file, those were set by our initial NumPy array. But it does update our header to match the data we'll be putting in. To wrap this up I use the tofile method to write only the FITS header to the disk with not actual data following it. PyFITS generally won't let you write an invalid FITS file, in this case it would complain that the NAXIS values don't match the 1 x 1 x 1 array we used to create the file, but we can write just the header as we're doing here. The clobber option overwrites the file if it already exists.

Hacking the Data

header_length = len(header.tostring())
data_length = math.ceil(11 * 11 * 4000000 * 8 / 2880.0)
with open(fits_file_name, 'rb+') as fobj:
    fobj.seek(header_length + data_length  - 1)
    fobj.write('\0')

Now comes the interesting part. One specific advantage Python offers scientists without a programming background is that Python is a "high level" programming language. The term "high" is subjective but the point is that Python takes care of many of the "low-level" aspects of programming such as memory allocation, pointers, and garbage collection. However, as you get further into the language you'll find that you have to learn how these concepts are implemented to take solve more complicated problems. This was one of those times for me.

Let's jump to the middle where I use the "new" standard Python convention of using with to open a file object. In this case I open it with the rb+ setting which means I'm going to be read (r) and update (+) the file in binary mode (b). Binary mode means that rather than trying to decode strings in something like UTF-8 the file object will read raw bytes.

When you open a file object in binary update mode your current position is the beginning of the file. Meaning if you tell python to start reading or writing to the file it will start right a the beginning of the file. In my case I don't want that so I first use the seek function to tell Python how far ahead on the disk to skip in units of bytes. In this case I want to seek ahead by the length of my final FITS file in units of bytes. It's worth noting that I'm taking advantage of the fact that seek will, by design, seek past the end of the file. It's worth noting that this is all being done at a very high level, I'm not actually seeking on the disk or even memory locations.

For clarity I've broken up this calculation into two variables, header_length and data_lenght. I use the tostring method to return a string representation of the header. Because each character can be represented by one byte the string length is the same as the byte length.

Next I figure out how many bytes my data is going to be. I do this by multiplying the number of elements in my array times the number of bytes required to store each element. This is interesting and something I haven't really thought about before; I can tell Python exactly how big my file will be before I write anything to it. I'm using a double precision, or 64bit, floating point, which can be represented by 8 bytes. So my data will take 11 x 11 x 4,000,000 x 8 byes or a little over 3.6 GB.

Now you can start to see why we needed to go through all this trouble. This data would have first have to have been read into memory from a different format (SQL in this case) and then copied into a NumPy array, that's 7+ GB of memory right there. Additionally, there's a couple of GB of metadata I need to load in as well, plus what's being used by system. And my data set is growing constantly. Suddenly, it's clear why we couldn't do this all in memory.

Actually, you'll notice it's a little more complicated then that. I'm actually rounding up to the next multiple of 2880 bytes. This is because, and I'm trying to say this with a straight face, of historical limitations on how tape machines used to read FITS data. So just like the FORTRAN line limit, this is a historical artifact we just have to live with. If you read the documentation carefully for the header.tostring method you'll see this was automatically done for the header when it was turned into a string.

Also notice that I never had to tell Python what was going to be in those bytes. An array of complex decimals takes up just as much space as an array of zeros if they're both stored as the same data type. This is the reason I couldn't just create our original HDU with a 4,000,000 x 11 x 11 NumPy array of zeros - that would take up just as much room as a NumPy array of the real data!

But what about the - 1 at the end? Well I go back just one byte from the end to write the final character of the FITS file, the \0 byte. In this case it acts as kind of like a place holder staking out how big the file is going to be. So what was the point of all that? Well, without even needing to create anything in memory anywhere near the size of our dataset we've now created a file exactly big enough to hold all our data.

Note that what we've created is called a sparse file and how it's implemented will depend on your file system. Certain file systems will just add some metadata letting the system know that it's just blank space, whereas other file systems such as Apple's HFS+ will go through and write each "blank" byte which will take just as long as writing real data.

(Finally) Writing the Data

with fits.open(fits_file_name, mode='update') as hdlu:
    for bottom_index, top_index in zip(bottom_index_list, top_index_list):
        numpy_data_cube = get_data_chunk(bottom_index, top_index)
        hdul[0].data[bottom_index:top_index,:,:] = numpy_data_cube 
hdul.close()

Now that I've created the output file of the correct size and dimensionality I can start writing data to the file. Fist I use fits.open with the with convention to open the FITS file. But wait, what's going on here? Won't opening this file just read all the data into memory - the exact thing we're trying to avoid?

What's happening is that the fits module is by default opening the file with mmap. What this means is that the file is read in a "lazy" or "on-demand" mode, data is only read into memory as needed. You can find more info in both the Astropy and PyFITS docs.

Next I step through the data I want to add to the FITS file in chunks that easily fit in memory. I wrote some generic code that does this by stepping over some indices and passing them to a function get_data_chunk that returns the next "chunk" of records as a NumPy array. I index the FITS data just like a NumPy array (because it is one) and then update it with the chunk from my current data cube. Iterating over this eventually write the entire file without ever needing to store the entire FITS data in memory at once.

One More Thing

Erik pointed out that there is a little-known feature in PyFITS called the StreamingHDU class. It's an alternative way to make a FITS file on disk just by outputting the header and then writing the data one chunk at a time: http://pyfits.readthedocs.org/en/latest/api_docs/api_hdus.html#streaminghdu

[^1]: FITS is a standard data file format in astronomy.

Working with NumPy Arrays and SQL

Mon, 07 Apr 2014 00:00:00 GMT

Lately I've been doing a lot (millions) of calculations involving small NumPy arrays of HST PSFs. Naturally, I wanted to save the output of these calculations to for later analysis. I put all the results in a MySQL database so I could easily select subsets of the data for future work (by filter, image, date, etc.). However, sometimes the outputs of these calculations are arrays themselves. This left me searching for a good way to save these NumPy arrays to a SQL database.

Before I dive into this it's worth noting that there are non-SQL storage options that are specifically designed for use cases like this such as PyTables or HDF5. But, my project was already pretty tightly integrated with SQLAlchemy and I wasn't concerned with having readable, hierarchical, or queryable array information, which are the strengths of these other storage systems as I understand them. The queries I'm going to write are going to be constructed on other fields and the data is only going to analyzed once it had been read back in as a Numpy array in Python. So, all I really needed was a way to go between NumPy and some SQL data type.

Starting with String

So my first thought was to just flatten the array into a string and then write that to the database as a VARCHAR field. So something like this:

a = np.array([[1,2],[3,4]])

Which gives us:

[[1 2]
 [3 4]]

And then transform it into something like this:

'1,2,3,4'

And then I would just code up some logic in Python that would know to convert it back into a 2x2 array. The problem is then you start getting really awkward obtuse Python like this:

string_array = str(numpy_array.flatten().tolist())[1:-1]

And on top of that you have to convince yourself that you are always reading and writing your strings in the correct order in terms of left/right and up/down, which means writing more tests. This quickly started to not feel right to me, especially if the end result was a human-readable SQL field that was never going to be read by a human while in the database.

Moving to Bytecode

After some digging and things I switched to bytecode. This isn't human readable (which is fine) but it easily and consistently goes in and out of numpy arrays with built-in methods and sits nicely in a SQL BLOB field. Writing looks like this:

byte_array = numpy_array.tostring()

Which gives me:

\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00'

And converting back to numpy is just as easy:

numpy_array = numpy.fromstring(byte_array)

Ta-Da! This does what I wanted and in my opinion has that intangible "pythonic" feel to it. The default NumPy datatype is numpy.float64 but you can specify others with the dtype parameter. While this solution met my needs that are probably many other ways to accomplish this, feel free to tell me about them in the comments.

Fitting 2D Gaussians with agpy

Thu, 30 Jan 2014 00:00:00 GMT

Update 01/30/2014: Adam has split his gaussfitter code off into it's own GitHub repository here ("PR's Welcome!"). This removes some dependencies and changes the import statement but as of right now everything else is the same. I've maintained the old links to the original agpy repo in the post below but please use the above repo for the latest version.

After some initial work with fitting WFC3 UVIS PSFs with 1D Gaussians through the x and y axis I decided to look at 2d Guassian fitting as well. I was disappointed to find there wasn't already a canned procedure to do this in something like SciPy. But after some digging I decided to use Adam Ginsburg's personal agpy library. I briefly met Adam at the dotAstronomy conference last year in Boston. He's a contributor to AstroPY, AstroQuery, and AplPy so I had a hunch I could trust his code and it's worked out great.

You can clone the repo here. There are a couple of dependencies but I only satisfied the AstroPy and Numpy requirements and that was enough to run the gaussfit function.

from agpy import gaussfitter

mpfit, psf_fit = gaussfitter.gaussfit(
    psf_array,
    returnmp=True, 
    returnfitimage=True
)

Using gaussfit without the returnmp or returnfitimage parameters just returns a list with the following model parameters (in order):

height
amplitude
x
y
width_x
width_y
rotation angle.

Adding returnfitimage=True will also return a NumPy array of the model with the same dimensions as the input data. Lastly, setting returnmp=True will return a mpfit instance, which is the class used to generate the fit. The class is defined in the agpy.mpfit_custom module. The mpfit instance contains two useful attributes, mpfit.params which is the same list of parameters that guassfit returns by default, and mpfits.covar which is a 7x7 covariance matrix for the 7 model parameters.

It took me a little bit of work to figure out all these outputs but they were exactly what I needed so I followed up with Adam and submitted my first FOSS PR on GitHub with some documentation improvements. It's a small contribution but still a personal milestone.

Finally, I made a plot of the input data, the model, and the residual (difference) at two different scales. I'm definitely happy with this and am looking forward to digging into the covariance matrix a little more to really understand how well I'm fitting these PSFs.

<imgß style="width: 800px; max-width: 100%; height: auto;" alt="Oops, something broke." src="/images/2d-gaussians.png" />

Faster File Existence Testing with Sets

Sun, 19 Jan 2014 00:00:00 GMT

It's Time to Think about Performance

Lately at work I've been thinking a lot about the performance of my code. In the past most of my work fell into one of two performance categories: (roughly) overnight or (roughly) right now. In either case I didn't really care about performance. Either the task was going to take so long I had time to go do something else, in which case I didn't care if it took 1 hour or 10. Or it was going to be done fast enough I could immediately start iterating on the results, again in which case I didn't really care if was going to take 1 second or 10. I think this is indicative of the scientific computing mindset where you are both the programmer and the user: fast means fast enough for you.

But recently my datasets have been getting bigger (which is awesome) which has forced me to be more careful about my programming. I'm routinely finding my scripts out-growing both of my performance "categories" and either taking several minutes to run or several days. Both scenarios leave me waiting around, which is the real problem. While I always try, to the best of my abilities, to write high-quality code my time is more scarce and expensive than CPU time. This means that I optimize my time, not the CPU's. However, when I do find myself waiting around for some code to run, it's time to roll up my sleeves and find some speedups.

The work I do is very I/O intensive involving lots of databases and data files. I/O is extremely expensive in terms of latency so reducing trips to the disk can yield sizable speedups. Here's an example I found today that includes an introduction to a handy (and I would argue underutilized) Python type called sets.

The Slow Way

I was working on a project where I wanted to verify that all the files I had listed in a database actually existed in my file system [^1]. To do this I wrote a SQL query in SQLAlchemy to grab all the file names listed in the database. Then I looped over the the records returned by the query and used os.path.exists to test the existence of each file in the file system.

%%timeit
for record in database_query:
    if os.path.exists(record.fits_file) == False:
        print 'Missing {}'.format(record.fits_file)

There were 3,096 iterations (records) in this loop and the IPython %%timeit cell magic gave the following result:

1 loops, best of 3: 103 s per loop

This is a bit too long of a wait for me. It's long enough for me to get distracted by Facebook or maybe writing a blog post. I kid but task switching does have a real mental overhead. I'm not advocating optimizing every task that makes you sit around for a few minutes, but in this case the solution was trivial and applicable to lots of my projects.

The Fast Way

It occurred to me that I was making 3,096 separate trips to the disk. It's my understanding that there is some overhead for each disk read so I thought maybe it would be faster to read everything I needed at once and then work with the result in memory. To do this I used glob and create a list of all the files in my file system I wanted to check my query against. This gave me all the data I wanted in memory from one SQL query and one glob command. That reduced the problem to a membership testing problem and Python has a great built-in type for this, sets. Sets are unordered hash tables which means their average performance for a lookup operation is the holy grail of speed, O(1). Incorporating all this into my code looks like this:

%%timeit
file_set = set(glob.glob(file_search_string))
for record in database_query:
    if record.fits_file in file_set == False:
        print 'Missing {}'.format(record.fits_file)

It turns out I was right, this is almost a full order of magnitude faster than my original code.

1 loops, best of 3: 10.6 s per loop

Caveats

So I think the principles behind this speed up are solid but, as always, your mileage my vary and there are some caveats I can think of.

First of all, the file system I am searching is a network file system that I'm connecting to over VPN, this makes each disk read exceptionally expensive. Secondly, the glob operation is very expensive, almost all the run time is spent in that step. So if you're only checking a few files it might be faster to just look them up one-by-one than to use wildcards to scan a file tree. I'm not sure where the tipping point is, but it's certainly worthwhile if you're checking every file like I am.

I've just starting thinking about these topics so if I missed something in my code or my explanation I would love to hear about it the comments.

[^1]: If you're wondering why I want to do this, yes, it's because I screwed up and put the wrong files in the database.

2014 Summer Internship Opportunity

Thu, 16 Jan 2014 00:00:00 GMT

The Moving Target Pipeline group at the Space Telescope Science Institute (STScI) in Baltimore, Maryland seeks a graduate or undergraduate student for a paid summer internship through the Institute's Space Astronomy Summer Program (SASP). The intern will work with our group to develop, test, and run our software pipeline which processes Solar System object observations ("moving targets") taken with the Hubble Space Telescope.

The Moving Target Pipeline is the first attempt in the 22-year history of the Hubble Space Telescope to create a generalized data processing pipeline to meet the unique requirements of Solar System data. Using Python and MySQL the pipeline creates calibrated science data, dynamically scaled preview images, and predicts possible detections of "incidental" or serendipitous objects such as moons and asteroids. These data products and software techniques are of broad interest to the solar system research community as well as STScI, even influencing plans for the new James Webb Space Telescope (JWST). More information on the project can be found here.

This is an ideal opportunity for a student interested in software development, whose goal is either a career in software or wishes to add programming to his or her highly-desirable skill set. Our intern will be exposed to many aspects of professional software development including documentation, testing, code reviews, distributed version control, and remote software execution. Last year's intern in our group (a college senior) reported the internship experience helpful in both job interviews and subsequently, on the job.

Strong candidates for this position will have experience in:

Working in the Linux command line
The Python programming language
SQL databases, specifically MySQL
Version control, specifically Git and GitHub

A demonstrated ability to work and learn independently as well as attention to detail are all valued. A background in astronomy or physics is desirable but not required. Applicants with other relevant qualifications, such as experience in other programming languages or success in prior internships, are encouraged to demonstrate how they could contribute to our work.

Candidates are invited apply by follow this link. Women and minorities are especially encouraged to apply. The deadline for applications is January 31st, 2014.

Please note that all applications go into a general applicant pool and will be considered for all internship opportunities available through the SASP. If you are specifically interested in an internship with our Moving Target project please include the words "Moving Target" somewhere in your application to ensure our group reviews your application.

We look forward to your application,

The Moving Target Team
Space Telescope Science Institute
Baltimore, Maryland

Guilty As Charged

Thu, 19 Dec 2013 00:00:00 GMT

Earlier this month the wonderful Randall Munroe at XKCD posted this great strip poking fun at the decrease in commit message quality as a project ages.

As the meme goes, "nailed it".

(I did finally get everything to work BTW.)

A Basic Automation Setup for Astronomy - Part 2

Tue, 10 Dec 2013 00:00:00 GMT

Why Log Files?

Let's put on our imagining caps. Pretend that you have already set up an automated pipeline like the one I described in the first post in this series (In fact some people already have). One day your boss walks into your office and asks for some details on about something you pipeline made 6 months ago. Maybe they want to know what input files were used. Or what settings and options were used. Or Maybe they want to reproduce a figure. How would you do that?

Something that I think sometimes gets lost when you start to transfer you work to a more automated work flow is our old friend the lab notebook. For any data products your automation platform produces you should be able to tell someone exactly what inputs were used, what software, and even what version of your software. Ideally, someone should even be able to figure this out for themselves. Log files are a great way to accomplish this type of detailed record keeping, essentially generating an automated lab notebook. Here is an example output from one of my scripts:

2013-11-22 10:08:52,719 - INFO - User: jstrummer
2013-11-22 10:08:52,720 - INFO - Host: casbah
2013-11-22 10:08:52,745 - INFO - Machine: x86_64
2013-11-22 10:08:52,747 - INFO - Platform: Linux-2.6.32-358.18.1.el6.x86_64-x86_64-with-redhat-6.4-Santiago
2013-11-22 10:08:52,747 - INFO - Command-line arguments used:
2013-11-22 10:08:52,747 - INFO - reproc: False
2013-11-22 10:08:52,747 - INFO - filelist: /*_saturn/*single_sci.fits
2013-11-22 10:08:52,820 - INFO - Processing 2722 files.
2013-11-22 10:08:52,821 - INFO - Now running on u2ona109t_c0m_center_single_sci.fits
...

With the command line grep utility you can quickly start to look for trends. If you want to do even more you can explore the Pandas Python package, more on that in a later post.

Getting Started with the Python Logger

Python has a great logging module. You can find a basic tutorial here. There are lots of ways to invoke and setup the Python logger. I usually define a function something like this and import it throughout my project.

import logging
import os
import datetime

def setup_logging(module):
    """Set up the logging."""
    log_file = os.path.join('/my-project/logs/', module,
        module + '_' + datetime.datetime.now().strftime('%Y-%m-%d-%H-%M') + '.log')
    logging.basicConfig(filename = log_file,
        format = '%(asctime)s %(levelname)s: %(message)s',
        datefmt = '%m/%d/%Y %H:%M:%S %p',
        level = logging.INFO)

What this does is it will create a separate log directory for each module and the log the outputs in a date-stamped file, e.g. /my-project/logs/my_module/my_module_2013-12-22-15-41.log. Every line in the log file will then begin with a date and time stamp, followed by the level name and then the logging message, similar to the output I showed in the last section. Lastly, I tell it to log all statements down to the "INFO" level.

A Logging Recipe

def log_info(func):
    """Decorator to log some useful environment information."""
    def wrapped(*a, **kw):

     # Log user, system, and Python metadata
        logging.info('User: ' + getpass.getuser())
        logging.info('System: ' + socket.gethostname())
        logging.info('Python Version: ' + sys.version.replace('\n', ''))
        logging.info('Python Executable Path: ' + sys.executable)

        # Log PyRAF data
        logging.info('PyRAF Version: ' + pyraf.__version__)
        logging.info('PyRAF Path: ' + pyraf.__path__[0])

        # Call the function and log the execution time.
        t1_time = time.time()
        func(*a, **kw)
        t2_time = time.time()
        hours_time, remainder_time = divmod(t2_time - t1_time, 60 * 60)
        minutes_time, seconds_time = divmod(remainder_time, 60)
        logging.info('Elapsed Real Time: {0:.0f}:{1:.0f}:{2:f}'.\
         format(hours_time, minutes_time, seconds_time))
    return wrapped

You script would then looks like this:

@log_info
def main():
 """My main function."""
 make_science()

if __name__ == '__main__':
 main()

That Time I Made a Metaclass

Wed, 04 Dec 2013 00:00:00 GMT

"If you don't know what a metaclass is you don't need to use one."

David Beazley

I was shooting some messages back and forth this morning with some current and former coworkers on Twitter on the topic of Python Metaclasses. One coworker said metaclasses was something he'd never really got around to using. I mentioned that I had used them exactly once to generate database Object Relational Models (ORMs). My second coworker said that was a common use case and that it would be nice to see an example. Since a tech blog is a shining example of a hammer searching for a nail I immediately got to work on this post.

Some Background

I took David Beazley's Python Master class in 2009 (?) and I still remembered the quote from the start of this post, so for years I didn't worry about metaclasses because I knew I didn't need them. Finally though, I did need them.

One of my favorite Python modules, despite its near vertical learning curve, is the SQLAlchemy database toolkit. One of the features in this module is a very nice Object Relational Mapper (ORM) which maps database tables to Python classes. The ORM can be used in a number of ways and I prefer to use what's called the Declarative Base syntax. The basic idea is that you create a parent class called Base that contains information about your database connection and metadata. All your ORM classes are then child classes of Base and you use them to work with your tables. Here is a basic example:

class MyTable(Base):
	"""Defines a SQLAlchemy ORM"""

	def __init__(self, init_dict):
		self.__dict__.update(init_dict)

	__tablename__ = 'my_table'
	id = Column(Integer, primary_key=True, index=True)
	foo1 = Column(String(50))
	foo2 = Column(String(50))
	foo3 = Column(String(50))
	...

You could imagine an application that would need to dynamically define several of these tables, but you don't have to because I'm about to tell you about one.

My Problem

The most common image file format in astronomy is called FITS. FITS files have multiple layers (called "extensions") each with it's own set of metadata (called "headers"). For one of my projects we have over a million FITS files and we index these files with a MySQL database that maps the header keywords in the extensions to fields in SQL tables. We have about a dozen different file types, each with a handful of extensions, and each of those has 10s of header keywords. If you spelled out every ORM explicitly with a class and an attribute for every column like we do above we would literally have thousands of rows of ORM definitions. I'm a big proponent of the DRY principle (Don't Repeat Yourself) for the sake of readability and maintainability so this was a pretty big red flag in my opinion.

My Solution

Notice that we don't need to dynamically create many instances of the same class. Instead we need to dynamically create many class definitions. This is the specific need the drove me to use a metaclass.I ended up with something like the code snippet below.

def orm_factory(class_name):
	"""Creates SQLA ORM Classes."""
	def __init__(self, init_dict):
		self.__dict__.update(init_dict)

	class_attributes_dict = {}
	class_attributes_dict['__init__'] = __init__
	class_attributes_dict['id'] = Column(Integer, primary_key=True, index=True)
	class_attributes_dict['__tablename__'] = class_name.lower()s
	class_attributes_dict.__update__(get_column_defs(class_name))

	return type(class_name.upper(), (Base,), class_attributes_dict)

You could then call orm_factory like this:

Class1 = orm_factory('Class1')
Class2 = orm_factory('Class2')
Class3 = orm_factory('Class3')
...

And there you have your classes, dynamically created using metaclasses.

Solution Breakdown

Let's walk through this. First, let's look at the last line for the orm_factory function. This is maybe the "craziest" part of the whole function. That's because type is actually a metaclass constructor. That's right, the thing that tells you type(1) is int is also used as a metaclass constructor to maintain backward comparability [^1]. (If you want to tickle your brain check the type of type). To really wrap your head around metaclasses and type check out Jake VanderPlas's excellent post on the subject.

The basic idea is you pass type the string name you would to give the constructed class, a tuple of parent classes, and a dictionary of any other attributes for the class. Looking up the code block you'll see that I create a dictionary for these attributes called class_attribute_dict.

Notice I'm doing a little bit of magic by creating a get_column_defs() function. This function will dynamically add the appropriate column definitions, for example by pulling them from the FITS headers. The implementation of this function isn't import to the topic of this post, what matters is that these is some dynamic aspect to the column definition (and hence the class creation) that necessitates the use of a metaclass.

Also, notice that the attributes in class_attribute_dict includes an __init__ method defined as a function. This is one of those weird moments in python when you define a function inside of another function. We do this because we're never going to use the __init__ function outside of the orm_factory function it's nested in so there is no reason to globally scope it. In this case we define __init__ just like it was a method with a reference to self and everything. Even though there is nothing special about this function it will still take on the properties of a method after it's passed to type. I personally think is pretty cool and give you some insight into how classes are built.

So that's my example of metaclasses. It took me a couple of long days to figure this all out but I learned a lot about the inner workings of Python in the process. It's not so hard once you see it, but it's also not something I anticipate having to do again soon.

[^1]: I swear I read this somewhere but I'm still hunting for the source.

The Joy of "Screen"

Sun, 24 Nov 2013 00:00:00 GMT

Because I frequently work remotely as well as run long processes I recently gave up my desktop system at work in exchange for a laptop and a couple of virtual machines. I use the laptop for day-to-day work but offload longer computations or automated processes to my VMs. But before I could fully take advantage of this setup I had to solve a new problem I didn't have to worry about with my desktop, how do I run a process remotely without maintaining an open SSH connection?

I went to our IT department with this question and my favorite sys admin[^1] turned me onto the joy of Screen. Since then I've been spreading the word around my branch about this awesome little tool. Screen is a robust little piece of software that allows you to manage your shell session in a variety of ways. Screen can do a lot and it's worth taking the time to read through some tutorials but I'll explain my basic workflow in this post to start a process on a remote host and leave it running after closing the SSH connection.

First you SSH into you remote machine as normal and then type

$ screen -ls

This is going to return a list of all the Screen[^2] sessions you have active on your machine. If this is your first time using screen you should see something like No Sockets found in .... Great, now we're going to start up our first Screen session, I'm going to call this one making_science, so then I type:

$ screen -S making_science

And then you'll get a new command line prompt, you are now "in" your Screen session, or as Screen calls it "attached". If you run screen -ls again you'll now get something like this:

there is a screen on:
        48625.making_science    (Attached)
1 Socket in blah-blah

Now go ahead and launch your long script, I usually run it in the background by appending script with an ampersand (&) like this:

$ make_some_science.py &

And now we finally get to the good part, we're going to detach the session without killing the process, even if you close the SSH connection. You can do this either by typing ctrl + a then :detach, or if you don't have those key bindings, screen -D. Anytime you want to check back in one it just reattach your screen session on the same host with screen -r and everything will be just as you left it, even your command history. When you're finally done with your session you can just kill it with exit and it'll be removed from your list of screens.

Lastly, if like me you're suspicious by nature, you're going to want to check to make sure your process is actualy still running so you don't die a little inside when you come back to work the next morning expecting a pile of fresh data and instead get a stack trace. There are a variety of tools that allow you to do this, the two command line solutions I use are top [^3] and ps. Either one will list the processed currently running on your machine. I usually start up a process with screen, detach the session, confirm that the process is still running with top or ps and then close the SSH connection. If I'm feeling extra careful I'll check the log files from another host after closing th SSH to make sure things are still humming along.

And that's it. Go ask your IT department very nicely if they can build you a VM and then unleash your codebase while you go about your life. Horray!

[^1]: If you're going to do any serious DevOps work make friends with you IT staff. Ask them for help nicely and thank them profusely. Buy them beers. A good relationship with your sys admins in invaluable to getting sh*t done. This really shouldn't even be a footnote.

[^2]: Screen is inconsistently capitalized in the websites and blogs I saw. I decided to follow the convention in the GNU docs and treat it as a proper noun. I bring this up here because this is the first place in this post where Screen isn't capitalized because it starts a sentence. I also bring it up because I'm a nerd.

[^3]: Depending on your version of top you can type u on the main screen and then your username to view all the processes being run by your user name. Given that top shows all the system processes this helps remove all the ones you don't care about.

A Basic Automation Setup for Astronomy - Part 1

Sat, 23 Nov 2013 00:00:00 GMT

For one of my projects at work I engineered an automation platform for one of our instrument teams. This platform allows us to automatically execute 20+ daily scripts, written in a variety of programming languages, as data is coming down from the telescope. All the scripts for our team, from the downloading the data, copying and indexing the data in an SQL database, running in-house scripts, and system self-diagnostics run on the same automation platform. Adding a script to this platform requires as little as 4 lines of code. Our codebase is updated with hourly builds from our team of 6 developers and all execution and maintenance is performed via a service account on a Linux Red Hat virtual machine.

This is going to be a series of posts where I'm going to cover all the odds and ends I had to stick together to build this system. This is far from a generalized solution but hopefully you can learn from my mistakes and build something to suit your own needs faster and better than I did.

As an aside, this is what I consider to be the DevOps side of my job. It's only in the last year that this has become part of my work and it's only now that I'm starting to identify this as a valuable skill in response to a hard problem. Previously, I was just embarrassed I kept breaking things. But looking back on it, automating this system in this way is one of the hardest things I've done in my job. I'm fortunate I have a team that didn't tell me to stop wasting my time and do it the old way by running everything by hand. And now that it's up and running I haven't had to fix it in weeks.

In this first post we'll just cover combining the automated execution solution (cron) with the environment configuration solution (Ureka).

Cron: Running Your Code

At the heart of our system, like many automation solutions, is the Unix job scheduler cron. There are other applications our team considered using to fill this role such as Condor a distributed computing platform, Jenkins CI a java-based web frontend for continuous integration, launchd an OSX job scheduler, and even Celery a distributed job queue. In the end we chose cron because, once we got it working, it was the most direct and simple solution for our architecture. However, as you'll see later, that simplicity made it incredibly difficult to use with IRAF/PyRAF. But first let's start with some cron basics.

If you look at some cron tutorials you'll see that cron jobs are scheduled with a syntax like this:

00 11 * * * my_script.py

This will execute my_script.py every day at 11 am. This is nothing you can't find in any cron tutorial but here are some tricks I found useful that I had to dig around for a little bit. You can also run multiple scripts sequentially on one line by separating them with a semicolon (;) like this:

00 11 * * * my_first_script.py; my_second_script.py

Each script will run once after the preceding script finishes regardless of the preceding script's exit status. Alternatively, you can make the execution of the second script dependent on the successful completion of the first script with a double ampersand (&&) like this:

00 11 * * * my_first_script.py && my_second_script.py

Now cron is likely going to try to be helpful by emailing you any outputs from your code. The recipient of this email can be defined by setting the MAILTO variable before the job definitions like this:

MAILTO = 'my_team_list@my_institution.edu'
00 11 * * * my_first_script.py && my_second_script.py

You can also define any other variables or alias you want in this same manner. But let's say that you only want to hear from cron when something breaks. You can do this by redirecting STDOUT just as you would from the command line:

MAILTO = 'my_team_list@my_institution.edu'
00 11 * * * my_first_script.py && my_second_script.py > /dev/null

Now you will only get an email when something gets passed to STDERR. For our setup, this is all the cron syntax we needed to understand. Now onto setting up you environment.

Ureka: Setting Up Your Environment

Right now you might be thinking, "My environment is already set up! Right?". This is when using cron starts to become a little non-trivial; cron does not know about your enviorment variables, like any of them. In a lot of applications this is not a big deal, you can just define some environment variables just like I defined MAILTO above, and you're set.

But, if you're in astronomy one of your default software tools is likely IRAF/PyRAF. This takes installing software and declaring environment variables to an entirely new level of difficulty. I spent weeks working on the problem of getting cron to run in an IRAF/PyRAF compatible environment without any success. I tried half a dozen different approaches and talked to several people, all of who confessed to giving up due to the same complication. In the end, I partnered with one of our best IT people and we came up with a solution. The first step of that solution is to use Ureka.

Ureka is a software package developed by STScI and Gemini. From the website:

Ureka is a collection of useful astronomy software that is generally centered around Python and IRAF. The software provides everything you need to run the data reduction packages provided by STScI and Gemini.

Ureka is great, it builds a completely isolated IRAF and Python environment in minutes and loads or unloads the environment with a single command. You can run ds9, IRAF, PyRAF, and Python. Plus Python comes loaded with IPython, the IPython notebook, matplotlib, numpy, scipy, and pandas. If you need more than that you can immediately run pip install because your paths have already been set up for you. You're on your own for IDL though.

Whether you're like me and work on an institute machine with pre-built libraries or if you're running everything on your own machine and have root, Ureka is worth looking into because it just works. I spent a lot of time learning to use package managers, building from source into different prefixes, virtualenv, and the ins and outs of pip, but when I got a new virtual machine last month I used Ureka and literally set up everything I needed in 3 commands. I was sold.

So now we have our automation tool, cron, and our environment setup with Ureka. Now it's time to combine them.

Cron + Ureka: Automatic Environment Setup

Like we just saw, you can run more than one script on a single line in cron. You can start the Ureka environment with the command ur_setup and exit with ur_forget. So I thought the following command would have been enough to run our scripts:

00 11 * * * ur_setup && my_second_script.py > /dev/null

But it doesn't work. Somehow this does not run my_second_script.py using the environment set up by ur_setup, my guess is that each script is launched in an independent shell that doesn't propagate variables back to the parent cron environment. This independence is generally a desirable feature so that makes sense, though it makes life hard in our case. This is where everyone I talked to crashed and burned when trying to use cron to automate astronomy software, whether they were using Ureka or not. But eventually one of our IT specialists worked out a wrapper script:

:::sh
#!/bin/tcsh
ur_setup
"$*"

It's non-intuitive at first but what it does it runs ur_setup and then takes a script name as a command line argument and runs that script. Because this is all done in the same shell session the script is launched in the Ureka environment - finally. I can't tell you how happy I was to finally get this to work. The execution looks like this:

00 11 * * * cron_setup.sh my_second_script.py > /dev/null

Done, Right?

That was a bit of a long post, and you might be tempted to call it quits and just run with this setup, but I would encourage you not to. We still have to talk about a deployment solution for your code using your version control system (because your code is version controlled right?), using Python to wrap code written in other languages, using the Python logging module to generate logs. Doesn't that sound nice? I'll put the link to all that right [here] once it's ready.

The Data are Inconclusive

Thu, 21 Nov 2013 00:00:00 GMT

"Give me a point and I can draw a line. Give me two points and I can draw a curve. That's astronomy." - Anonymous Professor.

Ran the full pipeline the PSF data for the first time. The data are inconclusive.

Haha, oh error bars. Who hasn't made a plot just like this before?

The First Thousand PSFs

Wed, 20 Nov 2013 00:00:00 GMT

In my last post on my PSF Project I characterized the PSF shape by fitting a 1-D Gaussian to the a row and column slice through the central pixel. In this post I start to think about how to characterize entire images with thousands of stars.

Adding Some Zeros

Many applications for the PSF database involve looking at PSF changes over a time series. One way to do this is to move from characterizing individual PSFs to characterizing the PSFs in an entire image and seeing how that changes over time. [^1] In other words, we need to add some zeros to our lone star and start working our way towards our final dataset of 10 million. I decided to do this by digging into the first stellar field I could find. This happened to be iabj01a2q_flt.fits an image of NGC104 which adds 3 more zeros to our star total, bringing it to roughly 1,500 stars. Here is the image, it's actually quite nice (sorry for the large file):

In case you want to play along at home, this image is in the public domain and can be found here.

Plotting the PSF Fits

So I went ahead and fitted a Gaussian in both the row and column directions for all the stars in that image. Then I studied the distribution by plotting the fit parameters against each other in each slice.

First of all, in hindsight what I wanted was a scatter plot matrix but this is exploratory work so I can go back and make that plot in the future.

In terms of the actual data, we don't see anything noticeably different between the row and column slices, which is expected at this point, though there there may be directionally-dependent instrumental effects, such as the charge transfer efficiency, which may come into play later. Looking at the individual fit parameters there's not much going on with the amplitude but things get interesting when we plot the standard deviation against the mean (mu). However, the correlation we're seeing in the plot is actually a sampling effect.

Interpreting the Results

Let's start by explaining what sigma and mu mean in this context (it might be helpful to refer to my last post). First of all, mu, the mean of the Gaussian fit is almost always between 4.5 and 5.5. This is by design because the algorithm that centers the PSF cutouts does a good job of picking the brightest pixel. sigma then is a measure of the width of the PSF.

The actual energy distribution of a star is a continuous distribution. However, detectors take discrete samples from this distribution, pretty much literally making a histogram. If you've played around with histograms much you might see where this is going. If a star happens to land exactly in the middle of a pixel (mu = 5) most of the light from the PSF will land in that pixel. This will result in a narrow sigma. But is the same PSF lands right in the middle of two pixels (equidistant from their centers) then the same amount of flux is split between those two pixels. The total amount of flux is still the same, but the distribution is different. This distribution is less peaked, which is to say wider and with a larger sigma.

I think this is supported by the fact that as you move away from the middle of the central pixel (mu = 5) the distribution of sigma for a given mean moves up (gets wider) in an absolute sense, but not in a relative sense. Put another way, the width of you PSF increases the further the peak is from the central pixel, but the difference between a peaked and broad PSF at any given mean is about the same.

What's Next?

The next step is pretty clear, we need to use this distribution of parameters to characterize the "average" PSF shape in each image and plot that as a time series. This will almost certainly yield nothing but noise, but I'm confident that as we tease the data out such as separating each filter or different parts of the detector we'll start to see some real trends.

But, this sampling effect is bothering me. If we just take a mean and standard distribution of all the sigma parameters in each image those results are going to be heavily influenced by the sampling effect (I'm deliberately trying to avoid saying the sigma of the sigmas.). However, after looking at a sample set of plots I think this is likely characteristic of all our data. Consistency will have to suffice for now because as you'll see in my next post there are more pressing problems as we start to add zeros to our star count.

[^1]: Most astronomers talk about PSF widths in terms of the full width at half maximum (FWHM). I'm using the standard deviation (sigma) in my work but they only differ by a coefficient; FWHM = sigma x 2 x (2 x ln(s)) ^ (1/2). Sorry about the lack of nice math symbols, I haven't played with that plugin yet.

Counting to 10 Million Stars

Mon, 18 Nov 2013 00:00:00 GMT

I've started a new project working with 10 million stellar PSFs. In my first few steps in the project I performed some model fitting and made a pretty visualization of the individual data points.

My New (Little) Big Data Project

I am starting a new project that I'm pretty excited, it's one of the reasons I decided to start this blog. about because it is pushing me more in the direction of "Big Data". Lots of people throw the term big data around with different meanings. Personally, I consider something to be Big Data when the complexity of the task is dominated by complications from the size of the data. The "task" could be anything related to the data including storage, computation, or visualization. Specifically, this project is going to push the computation and database aspects of my work into the Big Data zone.

The dataset for this project is 10 million stellar PSFs observations taken with the HST WFC3 UVIS instrument. These PSF were data mined from the total on-orbit data set of roughly 35 thousand WFC3 UVIS observations using a colleague's specialized FORTRAN code which extracted a 11x11 array centered on each PSF. This is an especially powerful method of constructing our dataset because it allows us to use any incidental PSFs observations when the target was not a star or stellar field.

Fitting 1-D Gaussian Distributions

After some initial work I was able to create a reader that takes the outputs text files from my colleague's code and transforms it into a numpy array. Next, we decided we wanted to start by characterizing the PSFs with two 1-D Gaussian fits through the center pixel, one in the row direction and another in the column.

First of all, I was shocked to learn, after an hour of googling and popping my head into people's offices, that the definition of a Gaussian distribution isn't tucked away somewhere in NumPy or SciPy. Thinking about it, it guess makes sense because it's not clear what format your inputs and outputs should be, but I'm still a little surprised that all the tutorials I found on this subject began with defining the Gaussian distribution. Anyway, once I got past that the rest wasn't too hard.

import numpy as np
from scipy.optimize import curve_fit


def gaussian(x, A, mu, sigma):
    """Definintion of the Guassian function."""
    return A*np.exp(-(x-mu)**2/(2.*sigma**2))


def get_gaussian_dict(data):
    """Use curve fit to return a dictionary with all the model 
    information."""
    p0 = [data.max(), np.where(data == data.max())[0][0], 1]
    coeff, var_matrix = curve_fit(gaussian, range(len(data)), data, p0)
    A, mu, sigma = coeff
    output_dict = {}
    output_dict['amplitude'] = float(A)
    output_dict['mu'] = float(mu)
    output_dict['sigma'] = float(sigma)
    output_dict['var_matrix'] = var_matrix
    output_dict['model_data'] = gaussian(resample_range(data, 10), A, mu, sigma)
    return output_dict

I use scipy's impressive curve_fit function to perform the model fitting. The last argument curve_fit takes is p0, the initial guess for the fitting parameters. Fortunately, our data is very well behaved so we can easily do a good job guessing the initial parameters from the input data. Because I like to make my functions as general as possible I return all the possible information from the fit in a dictionary. For example, in the future I'll probably want to dig into the covariance matrix that the curve_fit returns to calculate a goodness of fit estimator and I'll be able to do that with the same function.

Eye Candy

Finally, all this is all visualized in the 4-panel figure below.

The bottom row contains the row and column slices and the Gaussian fits with the model parameters printed in the upper corners. The upper row contains a heat map, and just for fun, a 3D wire frame for the PSF. I could make some tweaks here and there such as matching the wire frame and heat map color bars but this is already more than enough to visualize a single data point, I need to start working my way up to 10 million.

The Moving Target Pipeline

Mon, 18 Nov 2013 00:00:00 GMT

A few weeks ago I was awarded an research grant to continue working on a prototype software pipeline for HST moving target (solar system) observations. The grant came from an internal source called the Director's Discretionary Research Fund (DDRF). My project, called the Moving Target Pipeline, was fully funded at $21,000 and allows me to buy back 25% of my time for one year to work on the project. Here is the proposal abstract:

"We propose a moving target pipeline for the WFC3 and ACS instruments based our existing WFPC2 software to produce properly drizzled FITS images, dynamically scaled preview images, and predicted ephemeris positions. Such a pipeline is relevant to ongoing HST scientific observations, the Hubble Legacy Archive (HLA), and serves to lay the design groundwork for JWST’s moving target processing. We request funds to support a senior RIA for our software development activities. [^1]"

Continuing from the proposal:

"Our WFPC2 pipeline addresses the 4 main issues that impede performing Solar System astronomy with HST archival data: (1) identifying cosmic rays, (2) drizzling, (3) scaled preview images, and (4) identifying incidental ephemeris observations."

This image gives an attractive visual of what we can already accomplish for WFPC2 data and will expand to the WFC3 and ACS cameras:

You can find the full proposal here. Our project will be open source and available on GitHub. It will be an extension of our existing work on a citizen science project for WFPC2 which you can browse here. This builds off a number of other grants and existing work in this area [^2].

I hope that this phase of the project will be useful for planetary scientists using HST.

[^1]: Full disclosure, I misspelled "activities" in the actual abstract. facepalm [^2]: Humblebrag / scavenger hunt, spot the astronaut co-investigator in the parent proposal :-)

The Trouble with Tech Blogs

Sun, 17 Nov 2013 00:00:00 GMT

the trouble with poetry is
that it encourages the writing of more poetry
- Billy Collins, The Trouble with Poetry

To paraphrase one of my favorite poets, the trouble with tech blogs is that they encourage the writing of more tech blogs. The Internet probably doesn't need another tech blog but I think this will be a fun way for me to both share and keep track of what I'm working on. Hopefully, some people will at least find it entertaining if not helpful.

I tried to start this blog over a year ago with Octopress. I'd never worked with ruby before but I immediately feel in love with the idea of static HTML generated from Markdown. It seemed so elegant yet customizable, definitely a hacker's blogging platform. Of course I almost immediately broke the deploy step to GitHub and got pulled away to other projects. Then I stumbled across Pelican earlier this year, which is the same idea as Octopress but in Python! After several false starts I now finally have a working blog again!

And so another tech blog begins!

All other languages were for some reason inferior, and as a Python programmer, I was the member of an elite cabal of superhuman...

Fri, 14 Sep 2012 21:14:48 GMT

All other languages were for some reason inferior, and as a Python programmer, I was the member of an elite cabal of superhuman ultranerds, smarter than those childish Rails/JavaScript/PHP/whatever developers that couldn’t write a bubble sort or comprehend even basic algorithmic complexity, but more in touch with reality than the grey-bearded wizards of Lisp/Haskell/whatever that sat in their caves/towers/whatever solving contrived, nonexistent problems for people that don’t exist, or those insane Erlang programmers who are content writing sumerian cuneiform all day long. http://jordanorelli.tumblr.com/post/31533769172/why-i-went-from-python-to-go-and-not-node-js

2012 DjangoCon Slides

Thu, 06 Sep 2012 16:34:22 GMT

**DjangoCon Lightning Talk: Hello from Hubble ** from alexcostaviana

Here are my slides from my lightning talk at the 2012 DjangoCon in Washington DC. It was my first talk at a conference so I was glad I got such a warm response. It would be nice to return next year with some working django sites!

Overly honest commenting on hitting 'accept' on a GitHub pull request for the first time.

Sun, 02 Sep 2012 18:28:54 GMT

Overly honest commenting on hitting 'accept' on a GitHub pull request for the first time.

Making sure we stay at the forefront of space exploration is a big priority for my administration. The passing of Neil Armstrong...

Thu, 30 Aug 2012 09:09:26 GMT

Making sure we stay at the forefront of space exploration is a big priority for my administration. The passing of Neil Armstrong this week is a reminder of the inspiration and wonder that our space program has provided in the past; the curiosity probe on mars is a reminder of what remains to be discovered. The key is to make sure that we invest in cutting edge research that can take us to the next level - so even as we continue work with the international space station, we are focused on a potential mission to a asteroid as a prelude to a manned Mars flight. President Barak Obama

The academic system does not respect practitioners’ knowledge (or timescales). Practitioners don’t understand that computer...

Mon, 27 Aug 2012 14:03:00 GMT

The academic system does not respect practitioners' knowledge (or timescales). Practitioners don't understand that computer scientists don't care about building software.

Taken from Greg Wilson's "Two Solitudes" presentation. Slide 78.

And this is why I don't see myself ever going to grad school.

I'm on-site today at Southern Illinois University - Edwardsville working on Space Telescope's collaboration with...

Tue, 21 Aug 2012 16:28:09 GMT

I'm on-site today at Southern Illinois University - Edwardsville working on Space Telescope's collaboration with CosmoQuest on an exciting crowd-sourcing project for citizen-scientists.

I'm providing the back-end pipeline to produce the images for the web app (I'll share the GitHub onces it's a little further along). Right now I'm using a little imaging tool I created (below) to examine the steps in the image scaling. The three row are the original image, the image after the step was applied (in this case the log of original image) and a delta image. Each row has an image and a histogram. In this case you can see there is single pixel pulling the stretch way out of whack so I'm working a function to trim the lowest outliers.

On Leaving Academia « Ars Experientia

Tue, 24 Jul 2012 17:54:26 GMT

On Leaving Academia « Ars Experientia

I saw this blog post being passed around Twitter this afternoon by the likes of David Beazley and Greg Wilson and the timing was perfect for post I was thinking about writing.

One of the things I noticed last week at SciPy was how many smart people have left academia or were weighing the benefits of leaving. I found out a friend of mine, who just got tenure at the age of 39, makes a only few hundred dollars more a year than I do as a 27 year-old analyst. He said he basically has to talk himself into staying in academia every summer.

There are of course unhappy people in the private sector. And there are many smart people still going into academia. But most of the current and aspiring academics I know or have had a chance to talk to do not paint a pretty picture of the journey to academia, or the what happens when you get there. It seems to me that even in the last 5 years since I finished undergrad the game has changed.

What I am personally seeing is a lot of very smart and motivated people look around and change their minds. And as Terran Lane points on in his blog post, this is a very disturbing trend.

Benchmarking Python Sets, Comprehensions, and Loops

Sat, 14 Jul 2012 14:17:00 GMT

I learned how to use Python's cProfile module last week and thought it was pretty cool. So at work this week when we came across a couple across a couple of lines at code I thought could be optimized I used it as an opportunity do a little profiling.

I compared loops against list comprehensions and then against the set object. I knew loops would be slower in both cases, but like a homework problem I wanted to go through the steps and get a better handle on numbers.

Comprehension vs Loops

I started by comparing a list comprehension against an equivalent for loop to pull all the even numbers out of the first 10,000 integers.

def lists_with_for_loop():
    evens = []
    for item in range(10000): 
        if item % 2 == 0:
        evens.append(item)

def lists_with_comprehensions(): 
    evens = [x for x in range(10000) if x % 2 == 0]

Here are the results:

function	ncalls	tottime	percall	cumtime	percal
lists_with_for_loop	1	0.013	0.013	0.022	0.022
lists_with_comprehensions	1	0.002	0.002	0.002	0.002

List comprehensions win by about an order of magnitude.

Sets vs Loops

To compare sets and loops I made a loop that generated a list of all the unique entries in a list of 10,000 integers between 0 and 1,000. It occurred to me that this was a relatively "sparse" case so I also made a "dense" one where the integers are distributed between 0 and 10. Here is the code:

SPARSE_INPUT_LIST = N.random.random_integers(0,1000,10000) 
DENSE_INPUT_LIST = N.random.random_integers(0,10,10000)

def unique_with_for_loop_dense():
    output = [] 
    for item in DENSE_INPUT_LIST:
            if item not in output: 
                output.append(item)

def unique_with_for_loop_sparse():
    output = [] 
    for item in SPARSE_INPUT_LIST:
        if item not in output:
            output.append(item)

def unique_with_set_dense():
    output = set(DENSE_INPUT_LIST)

def unique_with_set_sparse():
    output = set(SPARSE_INPUT_LIST)

And here are the results:

function	ncalls	tottime	percall	cumtime	percal
unique_with_for_loop_sparse	1	0.515	0.515	0.518	0.518
unique_with_set_dense	1	0.024	0.024	0.024	0.024
unique_with_for_loop_dense	1	0.013	0.013	0.013	0.013
unique_with_set_sparse	1	0.008	0.008	0.008	0.008

So sets always beat loops, but the difference is more dramatic the sparser the input list is. This makes sense since the set object uses a hash table while the for loop I write just does a list look up (though I'm not exactly sure how it's implemented). The larger the output list/set will be the more efficient the hash is over the list.

I made a plot of this for a a non-negative set of 1,000,000 integers of varying densities.

A couple of notes:

The bench marks and the plots were generated on different computers so the numbers don't match up, though the trends should be the same. I just didn't feel like entering all the numbers in the html table tags again.
To generate the plot I used the time module and took the difference of two time instances to find the run time.
You can view the source code for the benchmarks and the plot (and any other future code from this blog) on my Github Page

Pain with the Windows Command Prompt

Wed, 04 Jul 2012 16:54:06 GMT

[I wrote this post a few months but never posted it for some reason. Ah, the joys of trying to develop on Windows.]

I love my trusty IBM Thinkpad. It's coming up on 5 years old and other than a some fan buzz and a RAM card that needed to be replaced it runs just fine.

But the Windows Command Prompt can just ruin my day.

So I run my code to ingest the data into the database. No problem. But then I need to see if it made it into the database. I try just typing sqlite3 in the Command Prompt. No dice. No big deal, I just download the SQLite3 windows shell. But the problem with this tool is is just launches. You can't do something like:

sqlite3 mydatabse.db

Bummer. In hindsight I should have tried to called the full path to sqlite3.exe in the folder with my database. But since SQLite was just firing up in the folder where sqlite3.exe lives I focused on trying to open the database from within the SQLite3 shell.

Enter my next problem. There is no tab completion or copy/paste functionality in the SQLite3 shell. So now I have to type the full path out. No big deal right?

Wrong. My database lives with my code in a folder in my dropbox area. On a windows machine the Dropbox root folder is called My Dropbox, with the space. This creates a minor headache because now I have to get SQLite to accept a path name with space in it. After some playing with quotes, and forward and back slashes I decided that the space in the My Dropbox path element was annoying enough in general I should just change it altogether (it's just Dropbox on my OSX system at work).

It turns out this is a known 'non-issue' with Dropbox that is intentionally configured that was to avoid confusing non-technical Dropbox users. Around the time I was reading about scripts and redownloading instructions to change this I realized I had gone much too far afield wresting with this.

I the end I hacked a solution by just making a copy of sqlite3.exe in the same folder as my database. Then I can use the .restore command inside the SQLite3 shell without worrying about the full path. Definitely a hack but it worked and I was able to get it to work.

Benchmarking the Python SQLite3 Connection

Wed, 04 Jul 2012 16:51:00 GMT

We have several SQLite calls in our pipeline that look something like this

for file in file_list:
    conn = sqlite3.connect(database)
    c = conn.cursor()
    command = 'SELECT * FROM table'
    c.execute(command)
    conn.close()

I originally wrote our database calls this way because at the time (this was about 2 years ago), I was worried about the code crashing with an open database connection. I thought this would be vaguely "bad" so I was over enthusiastic about keeping connections closed. The design pattern stuck.

But someone pointed out to us that it would be faster to do something like this:

conn = sqlite3.connect(database)
c = conn.cursor()
for file in file_list:
    command = 'SELECT * FROM table'
    c.execute(command)
conn.close()

The reason being that there is some overhead involved when opening a Python SQLite3 connection. When you loop over the connection opening/closing steps you multiply this overhead.

My officemate went ahead and tested the overhead for running both design patterns over a test set of queries. Based on our results you can infer a connection overhead of 0.0134s. I took that data and assumed a linear increase for the overhead time for each additional query to extrapolate the results up to scale of our filesystem. The results are below. The first row are the real results and the next two are the extrapolated predictions.

Multiple	Single	Delta	Delta %	Records	Factor	Notes
0:02:23	0:02:05	0:00:18	12.59%	9300	1	wfc3g flt files
0:19:13	0:16:48	0:02:25	12.59%	75000	8.0645	All flt files
3:31:25	3:04:48	0:26:36	12.59%	825000	88.7097	All fits files

As you can see, we achieve a 13% speed up, which if we are querying the entire database saves us almost 30 min. Since we are running this process overnight this isn't a huge deal. Going forward it would be smart to use the faster form. However, I'm not sure if it's worth it to go back and fix our old SQLite calls as we have more pressing issues.

All in all, this was an interesting exercise and will probably lead to some more detailed profiling of our pipeline in the future.

Hipster Passwords

Wed, 04 Jul 2012 16:37:16 GMT

K: I can appreciate using "12345" as a password, like ironically. You know, like Spaceballs. A: Passwords aren't t-shirts.

_Finding the Right Plot_ I really like data visualization, even if I can't claim to be any an expert at it. Today I wanted to...

Tue, 03 Jul 2012 18:31:13 GMT

Finding the Right Plot

I really like data visualization, even if I can't claim to be any an expert at it. Today I wanted to plot some data in a flat file a coworker had generated. I first tried this with Google Docs because I just wanted something quick and dirty. But while I discovered some nice new features I wasn't quite able to do what I wanted; so I switched to Python with the Matplotlib module.

In Matplotlib I was pretty quickly able to get what I wanted, but then there was the matter of deciding what I wanted. I went through the 4 plots above.

The first was the standard scatter plot, this is no good because close fluctuations in February just look scattered and hard to follow. Then I tried a line plot, but now I felt that it was hard to pick out the individual points. Since the data points are non-uniformly distributed this is key information. Next I tried a plot with both lines and points (actually the two plots overlaid). This produced the best of both worlds, you could see the individual data points as well as follow their progression.

But then I noticed there was a problem with this chart as well. The use of the line implies an interpolation between the data points that might not be true. For example the increase shown on the plot from mid-May to mid-June actually occurred suddenly, not gradually as the line implies.

Finally I settled on a bar plot. This allows us to see that the data is non-uniformly distributed. We can pick out each individual point, as well as follow the flow of data. Lastly, nothing about data markers is misleading about the nature of the data. Areas with no data look "empty", which is exactly what they are.

Fun little project for the end of the workday.

Just registered for [SciPy2012](http://conference.scipy.org/). This will be my first programming conference of any kind. I'm all...

Mon, 02 Jul 2012 16:39:25 GMT

Just registered for SciPy2012. This will be my first programming conference of any kind. I'm all around excited but I'm the most excited for the opportunity to hack during the sprints during the last two days. Plus, Austin is a sweet town!

I just wanted to share a quick and dirty visualization I made to analyze some an processing pipeline I'm working on. What...

Tue, 26 Jun 2012 12:11:58 GMT

I just wanted to share a quick and dirty visualization I made to analyze some an processing pipeline I'm working on.

What you are looking at is a GIF of a the same image displayed with a linear stretch from 11 stretch cut-offs. The cutoffs are created by flattening the top and bottom pixels. It's a little quick and dirty but I've noticed it works across a broad range of image types.

However, the top % of pixels will often include data from the target. This is good strategy when the target is saturated since you aren't losing any information by scaling it down. But when the target is not saturated you end up "smoothing" out real features.

To test this I tried varying the %-clip levels. For each image I made 11 png files. Each of these pgn files is a XX% to YY% stretch. They start at 99% to 1% and step down by a 1/10 of a percent to 99.9% and 0.1%.

Each of these png files contains 4 images. The first is the before image, then the after image. The 3rd image is the most interesting, it shows the top pixels which were flattened in red, and the bottom in blue. The last image is the histogram of the after image. The title shows the stretch range and the title of the "flagged" image shows the value of the top % cut-off.

The "flagged" is definitely the most interesting - it shows the compromises you make to create a acceptable linear scaling. Still thinking about how to improve the end product but I'm happy with the visualization I created to get there.

I'm excited I finally have a reason to learn JavaScript at work. The first thing, per the suggestion of my friend who does UX,...

Mon, 25 Jun 2012 13:33:00 GMT

I'm excited I finally have a reason to learn JavaScript at work. The first thing, per the suggestion of my friend who does UX, was have work order me a copy of JavaScript: The Good Parts.

Excited to read through it but couldn't help laughing when I stumbled upon the size comparison between JavaScript: The good Parts and JavaScript: The Definitive Guide.

Widget Factory vs Film Crew Organizational Models

Wed, 20 Jun 2012 14:58:18 GMT

Widget Factory vs Film Crew Organizational Models

You need to do something spectacular before anyone knows you;re doing anything at all.

Wed, 30 May 2012 11:47:03 GMT

You need to do something spectacular before anyone knows you;re doing anything at all. One of my bosses. Everything I've done at work that's really stood out has been in that vein.

My First List Comprehension

Wed, 02 May 2012 13:50:00 GMT

I've been programming in Python for over 4 years. That's how long it took me to use list comprehensions. I'm a little embarrassed about this.

To be fair I've known about list comprehensions for years but just never used them. It's one of those nice bits of syntactic sugar that you can get by without. This is especially true if you're off in the corner reinventing the wheel, as I was the first couple of years I was programming. I knew they existed, I had a rough Idea of what they were for, but I just didn't see the use for them. Like learning most things in software development, it was just a matter of having the right problem.

In my case I was refactoring some of my code so my officemate could more effectively collaborate with me and I didn't like the way I was modifying lists. Let's say I had a list of names and wanted to remove all the names that didn't start with b. I was doing something like this:

name_list = ['bill', 'bob', 'brian', 'betty', 'farnsworth']
b_name_only_list = []
for name in name_list:
    if name[0] == 'b':
        b_name_only_list.append(name)

This would create a list of the names that start with 'b'. For smaller projects this worked. But it felt dumb to have to make a new list to do this. What if I didn't want two lists? What I just wanted to chuck the non-b names? I tried to do this by modifing the list "in-place" with something like this:

for name in name_list:
    if name[0] != 'b':
        name_list.remove(name)

But modifying a list as you are iterating over it creates errors and other weirdness such as this little guy:

a = [1, 2, 3]
for item in a:
    a.remove(item)
print a

Which prints 2. (Explained here.)

I got the feeling I was barking up the wrong tree. So after a little searching on StackOverflow I remembered, "oh yeah, list comprehensions, I should try that." Sure enough I can just do something like this:

 name_list = [name for name in name_list if name[0] == 'b']

Sweet.

Just had the same thought this morning. (via Yahoo's genius content strategy - The Oatmeal)

Tue, 24 Apr 2012 16:11:28 GMT

Just had the same thought this morning.

(via Yahoo's genius content strategy - The Oatmeal)

Sure, I trust Google to index the contents of all my files. Why not?

Tue, 24 Apr 2012 12:33:38 GMT

Sure, I trust Google to index the contents of all my files. Why not? Daring Fireball Linked List: Introducing Google Drive

`ls` in the Windows Command Prompt

Fri, 13 Apr 2012 15:59:51 GMT

Right now to interact with Windows Command Prompt I use a wrapper called Console2. It adds some basic functionality to the Command Prompt like tabs, and copy/paste. It's an improvement but not great because it's still just a wrapper around the primitive command prompt.

So I've been slowly transitioning over to Cygwin. Last week one of my buddies showed me what to add to my Windows PATH variable to to allow Cygwin to see the local filespace. I was, and still am, confused about why adding paths inside of the Cygwin filespace allowed Cygwin to see the rest of the drive, but it worked.

But today I noticed something interesting. One of the minor annoyances of having to use the Windows Command Prompt at home and OSX at work is that in windows you have to use dir instead of ls to view the contents of a folder and I have to remember to switch back and forth.

But today, ls worked! It was such a little thing that I didn't even notice till I had done it a few times. I realized (I think) this is because the paths I added to the PATH variable mapped the functions from Cygwin to rest of the Windows environment.

Neat! A nice surprise.

Update 04/12/12

Thu, 12 Apr 2012 16:25:56 GMT

It's been a productive few days.

Got some stuff done:

Joined GitHub and have my eyes on a project I want to contribute to.
Asked my first question on Stack Overflow (got my answer a few hours later).
Working my way through JavaScript examples on Code Academy.
Got IPython Notebook to work on my laptop, excited to play with that.

Still scratching my head on:

Getting IPython Notebook to work on my work computer because of some complications between the IPython in our central Python source and my local site-packages.
Getting Cygwin on my laptop to see my python install outside of the Cygwin drive tree.

Got some good programming projects lined up a work as well.

Feeling good, feeling like I'm moving along.

In December 2006, Palm CEO Ed Colligan summarily dismissed the idea that a traditional personal computing company could compete...

Mon, 09 Apr 2012 20:35:25 GMT

In December 2006, Palm CEO Ed Colligan summarily dismissed the idea that a traditional personal computing company could compete in the smartphone business. “We’ve learned and struggled for a few years here figuring out how to make a decent phone,” he said. “PC guys are not going to just figure this out. They’re not going to just walk in.”
In January 2007, Microsoft CEO Steve Ballmer laughed off the prospect of an expensive smartphone without a keyboard having a chance in the marketplace as follows: “Five hundred dollars? Fully subsidized? With a plan? I said that’s the most expensive phone in the world and it doesn’t appeal to business customers because it doesn’t have a keyboard, which makes it not a very good e-mail machine.”
In March 2007, computing industry pundit John C. Dvorak argued that “Apple should pull the plug on the iPhone” since “There is no likelihood that Apple can be successful in a business this competitive.” Dvorak believed the mobile handset business was already locked up by the era’s major players. “This is not an emerging business. In fact it’s gone so far that it’s in the process of consolidation with probably two players dominating everything, Nokia Corp. and Motorola Inc.”

http://www.forbes.com/sites/adamthierer/2012/04/01/bye-bye-blackberry-how-long-will-apple-last/

RSS in Chrome

Mon, 09 Apr 2012 20:32:29 GMT

Why does clicking on an RSS feed link take you to XML source code for the feed? I'm pretty sure that's never what anyone wants when they click the RSS button. Shouldn't there be something like the <mailto> tag?

Paying Bills Online

Mon, 09 Apr 2012 11:59:00 GMT

I just spent an hour paying a series of small bills, parking tickets, vehicle registration, toll fees, medical bills, etc. All were <$25. About half the time I just gave up and called the organization, sat on hold, and paid over the phone. Some trends I saw:

Why are these pages so ugly? Look at a clean page, look at your page. Recognize the problem.
Why do the names of things change across your pages and documents. Account number, member number, user number, etc.
Why do different divisions have different billing pages. You have one company. Give me one billing portal.
You already have my email. I give it to you every time I interact with you.
Why am I hunting through menus and sidebars for what I want. Guide me, don't send me on what you think is a logical hunt.
Why do I have to keep typing in my credit card number. There has to be a better way. Hell, why isn't there a card reader on my laptop. I'd rather scan it every time than have the numbers sitting in a cache somewhere.
Why do you send anything in bulk mail ever?

Inbox Zero

Sun, 08 Apr 2012 22:41:43 GMT

I just watched this Google Tech Talk on "Inbox Zero" by Merlin Mann. It wasn't as interesting as I expected but I did get some useful ideas from it. The biggest immediate change is that I put everything from my inbox, all 9k+ messages, into a folder called DMZ. I'll slowly go through and sort those guys out. But in the meantime, my inbox has zero messages. A fresh slate. Now I have to set up my system to keep it at zero.

Email is not a messaging protocol, it’s a to-do list, right? Or at least my inbox is a to-do list and email is the protocol for...

Sun, 08 Apr 2012 21:45:58 GMT

Email is not a messaging protocol, it's a to-do list, right? Or at least my inbox is a to-do list and email is the protocol for putting things on it. Here's the problem, it is a shitty to-do list. Any one of you can put something on my to-do list and I don't want that. Paul Graham of YCombinator on Email at PyCon 2012. Video and text.

The Greedy Algorithm

Sun, 08 Apr 2012 21:07:00 GMT

I was reading a fun bit of analysis of value of a pound of coins on Dan Kozikowski's blog and stumbled upon the greedy algorithm (via wikipedia):

A greedy algorithm is an algorithm that follows the problem solving heuristic of making the locally optimal choice at each stage with the hope of finding a global optimum.

So, you pick to local max (the highest value coin in this example) and it ends up getting you the global max (the fewest number of coins to make the total value).

Knowing the names of little things like the greedy algorithm is definitely something I'm trying to work on these days. It just doesn't look good on interviews when I have to go back and forth a few times before going "ohhhh, right that thing. I've used that before but I didn't know that's what it was called."

Getting Cygwin to see my local file system.

Thu, 05 Apr 2012 15:41:09 GMT

My friend showed me how to see my local file system on my windows Vista machine. I added the following paths to my windows PATH variable:

C:\Cygwin\bin;
C:\Cygwin\usr\bin;
C:\Cygwin\usr\local\bin;
C:\Cygwin\lib;
C:\Cygwin\usr\lib

Then I can see the local file system with:

cd c:/

I'm not sure what adding these path to this variable changed though. Cygwin was already working so I'm not sure why adding paths that point to it's own directory structure allows me to see the local file structure.

On the other side of the screen, it all looks so easy.

Thu, 05 Apr 2012 15:26:25 GMT

On the other side of the screen, it all looks so easy. Tron (1982)