Documenting What I Didn't Build: The Decisions Log Behind My NYC Pipeline

#Why keep a log of decisions nobody asked for?

Because a portfolio project with no stakeholders has no one to say no, except you. My NYC taxi supply-demand pipeline ran from June to October 2026, and every few days there was a new temptation: another data source, a fancier library, a more "enterprise" setup. Without something to push back, the project would have grown sideways instead of finishing.

So the repository has a file called DECISIONS.md. It now holds 52 entries, numbered D-001 to D-060, and the first line sets the rule: these decisions are not to be reopened without reading this file first. One entry exists purely to enforce that: any temptation to add scope has to be logged, with the reason it was rejected.

It turned out to do more than control scope. It became the project's memory, its interview preparation and its most honest documentation.

#What does a decision entry look like?

Short, and always with a reason. Each entry has an ID, a date, the decision, the rationale, and when relevant the trade-off accepted, a lock or a status. Here is one, condensed:

### D-055 | Rate Aggregation: Aggregate Totals, Never Average Ratios
**Date:** 2026-06-11 (Day 8)
**Status:** LOCKED
**Context:** The zone mart produced unmet-demand rates as low as -41.26.
**Decision:** Compute rates from totals: (sum(demand) - sum(supply)) / sum(demand).
**Guardrail:** dbt range tests on the rate block regressions at build time.
**Principle:** Never average a ratio.

The format is close to the architecture decision records Michael Nygard described in 2011: context, decision, status, consequences. I didn't need a template tool. A Markdown file next to the code was enough, because it is read in the same place the code is reviewed.

Open full size

#What did I decide not to build?

This is the part I'd keep if I could keep only one section. A good share of the log is about things that were considered, priced and turned down.

Not built	Why not	Entry
Trip data from the city's open-data API	Sampled data and rate limits; direct monthly Parquet files are complete	D-005
Census or land-use data as a demand proxy	Too little value for the added complexity	D-006
S2 cells or geohash instead of H3	H3's six equidistant neighbours suit supply-demand comparison	D-008
Apache Sedona for spatial joins	Overkill and the wrong bottleneck: trips carry zone IDs, not coordinates, so a 260-row lookup was enough	D-041
Loading OpenStreetMap straight into PostGIS during ingestion	Would break the lake pattern, add 12–15 hours and risk the free-tier database limit	D-031
A raster or elevation layer	No analytical use case in New York's flat terrain; it would have been a checkbox feature	D-039
Precise polygon clipping of the road network	A bounding box was good enough for version one	D-034
Preemptible workers and autoscaling	Premature before the job itself was stable	D-038
More than one year of data	One full year answers the seasonality question; scope locked	D-003a
A time-series forecast	Parked for later evaluation rather than added early	D-004

Open full size

#Why does writing down the "no" matter as much as the "yes"?

Because a rejected option without a reason will be proposed again, by someone else or by you in three months. A rejected option with a reason is a design decision.

It also shows judgement in a way a finished stack can't. Anyone can list the tools they used. The log shows that I knew the alternatives and chose not to use them for stated reasons. One entry goes further: when the Spark cluster couldn't install Python packages because its machines had no public IP, I pre-built a small lookup file instead, and the entry also lists the production alternatives I didn't implement, such as a NAT gateway, a custom machine image or a private package mirror. Knowing the enterprise pattern and choosing not to need it is part of the skill.

#How do amendments keep the log honest?

By never editing history. When a decision changes, the original stays and a new entry amends or supersedes it.

• Scope changes are visible. The data scope began as six months and became a full year in D-003a, because the seasonality question needed a whole cycle. The trade-off, double the processing cost, is written next to it.
• Wrong explanations get corrected in public. D-045 described a "mid-2023" schema change. A later check across all twelve files showed January was the only outlier, so D-045a corrected the framing and the lesson. I wrote that story up in The First File Lied.
• Incidents become rules. A storage lifecycle rule meant to expire old raw files was applied to the whole bucket and quietly deleted a hand-made reference file the pipeline couldn't download again. D-060 records the fix and the lesson: a lifecycle rule is a data-deletion policy, so scope it to the prefix that is meant to expire.

Open full size

An amended log is less flattering than a clean one. It is also the only kind anyone can trust.

#What would I tell a team starting a decisions log?

• Write the reason, not just the choice. "We use H3" is a fact; "we use H3 because neighbours are equidistant" is a decision.
• Log the rejected options. They save the most time later.
• Amend, never edit. Keep the original and add a new entry that supersedes it.
• Record incidents as decisions. The rule you adopt after a failure is the most valuable line in the file.
• Keep it next to the code. A log in the repository is read during review; a log in a wiki is read during onboarding, once.

I use the same habit on this website, where architecture decisions live in the project's documentation and are superseded rather than edited. I described that process in how I built this portfolio spec-first.

#Where can you see the pipeline running?

Everything the log describes is public. The live dashboard shows the result: the H3 hexagon map, KPIs and zone table. The GitHub repository holds the code and the full DECISIONS.md, so every entry quoted here can be checked against the original, and the project page summarises the problem, architecture and results.

#FAQ

#What is an architecture decision record?

A short document that captures one decision: the context, the decision itself, its status and its consequences. Michael Nygard popularised the format in 2011 as a lightweight alternative to large design documents.

#Is a decisions log worth it for a solo project?

Yes, arguably more than for a team. With no colleagues to challenge you, the log is the only thing that makes you state a reason before adding scope, and it is the best interview preparation you can have.

#Should rejected options really be documented?

Yes. They are the part people forget fastest and propose again most often. A one-line reason prevents a repeat discussion.

#References

• Documenting Architecture Decisions, Michael Nygard, 2011
• NYC Taxi Supply-Demand Analysis, project repository (DECISIONS.md)
• Finding $81.6M of unmet taxi demand in NYC with H3 and PostGIS, the project write-up on this site
• Live dashboard, NYC taxi supply-demand gap
• Project page, on this site

Last updated: 30 October 2026.