the accretion of complexity
No one knows how to make a pencil from scratch. Reality contains a surprising amount of detail. We stand on the shoulders of giants. World War IV will be fought with sticks and stones. These all feel connected, and as the world feels the fragility of functioning society, I happened to run into an concrete lesson today in the tradeoffs of accretive progress, in software.
I first built a website in 1999 - hand-written HTML, and a little "DHTML" (read: JavaScript) for some onMouseover button styling. I did professional web development primarily in the 2007-2012 years, after XmlHTTPRequest begat Web 2.0 and "apps" became the thing. And now, fully into the maturity of Single Page Apps with Server Side Rendering, I came back to build northcrownhill.com.
First, I built it by hand - the fastest, simplest way to get pixels on screen. Old school coupled with the modern magic of Netlify, which has automated away SSL, CDNs, and hosting itself! But as I reached the 3rd page, I was already making divergence errors amidst my copy-pasta. By my honor as a seasoned developer, it was time to do it right.
"Right" in 2020 looks something like a JAMStack app using the Gatsby framework (on top of React and JSX) with SSR leveraging GraphQL and Webpack and a bunch of other Node modules at build time to... produce static files that are just as fast as hand-coded. Well, except then after that, it's a full SPA! For... a tiny static site. But (slaps roof of meme) this bad boy scores 100/100 on a performance audit, and can handle any new use case! ...as long as there's an npm/react module for it.
I spent probably 10 hours learning my way through these several new layers of complexity: the CLI of Gatsby, the mental model of how JSX compiles, React components and lifecycles, CSS modules, GraphQL queries, Gatsby plugins and transformers... phew! And that was me moving fast, folks - I didn't even touch Yarn, didn't bat an eye at TOML being the new YAML, adopted VSCode and npm and Git instantly from just ambient knowledge... and I learn fast and only had 7 years of web dev progress to catch up on!
The time investment will likely break even in a couple months for me. But the interesting part is how all of these layers just smacked me in the face all at once. This towering edifice of two decades of technological accretion is amazing; I can do in a few hours what was barely possible with a full team working months a decade ago! And yet it has its costs. I had to add a dependency written by the National Football League just to stick something inside a <head> tag. The live "gatsby development" websocket server is buttery magical goodness, and yet I've already tripped over npm packages that work in debug but fail the SSR build because they naively reference the "window" global. I get markdown-to-html for free, and yet I had to figure out GraphQL filtering syntax just to reference a particular filename. The gatsby-image component gives world-class perf for free, but gosh if it didn't play nicely with something, I wouldn't even want to know how deep that rabbit hole goes.
It's always the "wait, what" problems and debugging that reveals the fragility of our software house of cards. Everything I've mentioned above is just front-end development, on top of the world's most complex and powerful VM (browser engines beat the JVM on that front, right?), on top of OSes and drivers and hardware and oh gosh I'm flashing back to that op-amps, diodes, and transistors class in college. Forget all of those other layers though, just think about the leaky abstractions piled on top of HTML. Writing a Gatsby site as my entry to web dev would expose me to HTML, technically, but all those layers would mean that I would be totally dependent on the tower of complexity that provides those other layers of magic! I remember when Ruby on Rails was black magic, but this now, this is less "spooky action at a distance" and more "no one knows how to build a pencil" territory times 100-1000.
If we lost it all, if it were just us, Apache, and Notepad... we'd be in the stone age, know what I mean? Sure we could rebuild civilization faster the second time around, but only if we survived the shock long enough to survive and scratch out enough of a meager existence to start the compounding of wealth again.
And... it wouldn't rebuild the same way. We probably wouldn't go so document-oriented with Web 1.0 if we had it to do over again, eh? And wouldn't we say the same thing about healthcare in the US, now? Or how about elections, or presidential powers, or even capitalism? (Galaxy-brain level: or human biology.) It's just that the sunk costs in these things are so, so huge, and we mostly improve by hill-climing! It takes a disruption or lots of guts to jump between local maxima. Maybe the shock of Covid-19 will be enough for us to switch hills re detaching health care from employment... just maybe? I hope?
Anyway, that's how modern programming relates to a global pandemic. Look under the surface, and the evolved layers get intractable real quick. But... it's all. Just. Bits. Just choices. The complex system is undeniable, but the accretion is incidental. To change is not impossible - all we have to do is work together. In software, that's been mostly decentralized. In society... we have no idea how to manage it yet.