Notes from an abandoned project

I made a couple of New Year's resolutions back in January. One was to blog much more than I have been the past couple of years, the other was to tidy my damn desk; little did I suspect that the latter would give me easy fodder for the former.

Under a pile of diagrams, doodles, books I'd forgotten I owned (sorry A Programmer's Guide to Computer Science, vol. 1 - I will read you one day!), and other detritus best left unmentioned, I found a pre-pandemic notebook, the first 20-odd pages chock full with notes and diagrams followed by a few pages of Post-Its I'd saved from a whiteboard.

My handwriting being what it is, and me being a software developer, if almost anyone else had found it it might as well have been the Voynich Manuscript, but I'm here to decode and it makes for some nice, easy #content.

Background

In the latter half of 2019 and early 2020 I was involved in a bunch of conversations, design discussions, and eventually a workshop that I expected would be leading to working in the lead role on a new project, building something truly from scratch for the first time in a long time.

Sadly, for boring organisational reasons I couldn't join it after all, but in the weeks prior I gathered a lot of thoughts about what I was going to be working on and how I would like to approach it.

I won't write about the specific details of the project itself, that would be quite inappropriate, but going through the post-its I saw many which give a good idea of how I like to approach software development and architecture, technically and organisationally.

I've since taken a big left turn in my career, leaving behind my previous thwarted ambitions to become a software architect and instead swtiched to being a data engineer back at the end of 2020 (a change which has done me a lot of good), and it's also interesting for me to look through this in that light.

No. More. Scala.

A contentious note (literally and figuratively) to start on, and no doubt I've already lost some people, but there it is.

By the start of 2020 I'd been programming primarily in Scala professionally for over 7 years, and started dabbling in it almost 9 years previously. This was long enough.

Do I hate Scala? Actually, no. As a language I like it a lot - it's flexible and expressive, and I recently published an article about going frameworkless in the language where I attest to the strengths I see in it. However, that was a solo hackday project, not a professional system built by a team, which is where I see the problem.

This is in part due to that flexibility - there are lots of ways to code in Scala and lots of very strong opinions on the right way to code in Scala. It's a recipe for conflict that runs from programming pairs all the way up to the level of the Scala community (or communities plural).

Even when the discourse is healthy, it's rarely more useful than bike-shedding, from pair programmers looking over their code when the tests are green and debating for an hour over how they can make it more idiomatic and/or functional (for whatever definitions of those you can find), to entire teams spending hours each week agreeing on guidelines and ensuring that code meets them.

Contrary to what the devotees will tell you, this is rarely worth it. Better to use a simpler language, no matter how "impure" or "inelegant", and Get Things Done.

Somewhat sadly these days, there's also the long-term future of Scala to consider. Lightbend, who oversee Scala, have been seeing rough days and major frameworks like Play are even falling out of maintenance. Starting a new system in Scala is more risky now than five years ago. I have fond memories of my early days with Scala, I learnt a lot from it and met some really great devs through it, but it's time to let go.

Kotlin?

Kotlin has some heavy usage at my current workplace, and I did a little of it after leaving behind Scala. It's fine. You are unlikely to get very passionate about it, but that can be a good thing.

Again taking the long-term view, I have a feeling that Kotlin is to Java what CoffeeScript was to JavaScript in some ways - a selection of syntactic improvements, with a few new (to Java) ideas thrown in, which will tide us over until Java's momentum inevitably allows it to catch up then surpass it (and Java is improving a lot).

In it's favour, Kotlin is more likely to be consistent and easy-to-read than Scala, with a much shallower learning curve for someone joining a team fresh who isn't experienced in the language and/or the system, and is also a lot less likely to spark any holy wars. It should also be easier to port back to Java if the need arises.

I think I would have assessed both Kotlin and Java, but given what I know now, I would cast the net a little wider as I now have experience of serverless, and of the problems that can come with slow cold-start times in JVM cloud functions (I'm also not sure that traditional Java et al programming styles are best suited to serverless functions in cloud architectures, and it might be easier to leave behind old habits by using a new language). From this perspective, golang would be on the list for sure, and currently working as a data engineer I'm increasingly a fan of Python (I'm also willing to come out as Clojure-curious, but to be honest I've never even experimented).

DDD

One area of product development I've longed for an opportunity to make more use of is Domain-Driven Design.

A lot of people talk about it as if it's a way of structuring a codebase and applying naming, but that's like saying Continuous Integration is about having a build server. When applied properly it pulls a team together, not just the software developers, but everyone involved in product development, driving alignment and understanding, facilitating communication, and hopefully even helping foster a greater sense of purpose.

My practical experience to that point had been somewhat limited - a couple of years prior I envisioned using it on a recently formed, significant project, going as far as planning and booking an event-storming session, but my role on that project changed before I could run it.

I did at least manage to push through the idea of starting a glossary, which we kicked off by holding full cross-functional team sessions to fill it out and hopefully push us into using a ubiquitous language. These sessions in themselves were incredibly valuable in increasing the team's understanding of processes of which we often only had small parts of the picture.

ADRs

Architectural Decision Records (ADRs) are something I've been using for a while now after a colleague introduced them into the company in 2018-19. They're one of the rare innovations that have come up during my career which turned out to be invaluable in the long run (though no doubt someone was "already doing this in the seventies").

Logging the architectural decisions made while building a system (and you can start keeping these at any time, not just for greenfield projects) sounds like a very simple idea, and that's because it is, but it is also highly effective.

They're great not just for the purpose of the team being able to look up why something is the way it is after some time has passed, but also as a tool for onboarding - effectively giving new hires the chance to read the story of the system - and as an aid for non-technical team members - we got positive feedback from our product owner on how our ADRs helped them to understand why we did some of the things we did and also communicate that out to stakeholders who might sometimes question the value of purely technical-sounding chunks of work, or wonder why something which sounds simple takes so much effort.

It's also not a prescriptive process, there's plenty of space for experimentation. At one point we followed a suggestion from someone on Twitter and added a field where we captured team sentiment at the time of a decision. You can also use them as a format for proposing changes, or use the structure to structure a meeting where a decision needs to be made, and you naturally get an ADR popping out at the end.

I find it hard to imagine developing a software system without them now.

Documentation!!!

The importance of written documentation has certainly increased with the sudden rise of remote working, but it's still a topic that people tend to both over-and-under state the importance of. Hitting the sweet spot of the right level of documentation, targeted at (and accessible to) the right audiences, and kept up to date is very hard to do.

The best way to do this in my opinion is build it into the process, and be as methodical about it as any other aspect. No story is "done" until relevant documentation is updated / created / deleted, where applicable, just as no story is "done" until it has tests and has proved its value in production.

This is easier said than, well, done; it will always be among the first corners to be cut when time pressure grows, and like many of those other corners the damage of cutting it won't be felt until much later. So you need both discipline and a safe environment. I'm lucky enough to usually have the latter, if not often enough the former, personally speaking.

You can go overboard when it comes to documentation as part of the process - it's not very many years since an engineering manager told me "we should ditch all this wasteful pairing and replace it with rigorous and extensive documentation". For me this missed so many points I could write a whole article on it, but I'd rather spend the time on collaboratively building something...

Visualise!!!

In many respects this is just another aspect of documentation, but I felt the need to call it out specifically in my Post-Its, and so out of respect for past me I'll maintain the distinction here.

For all that written documentation and written code is great, I've always found it incredibly useful to have visualisations to hand, whether it's a system diagram showing the software components and how they interact, a sequence diagram to help understand the processing of messages in a system, logical flow charts, or state transition diagrams. I often draw them for my own benefit while working, and they tend to stick in my head for a long time afterwards.

Since switching to data engineering, we've found such visualisations extremely useful in understanding data pipelines and planning out building and updating them, and I feel like if I were to return to backend engineering this is something I'd take with me (though useful automatically-produced visualisations of backend code are harder to come by than for data pipelines).

The Bigger Picture

I'm not 100% certain what I specifically meant when writing this, but I do think it's inherently important for all team members to be aware of the wider context - having a good understanding of what the org is about, understanding the customer, having knowledge of both up-and-downstream systems, having at least a high-level understanding of other teams.

Having all that documented and visualised is also an essential onboarding resource, worth the effort in that context alone.

Team PR

One of the many excellent ideas in Team Topologies that I like is team APIs. This is not that, but there is some overlap.

In a big enough org a team's reputation can be important. I've often seen inexplicably poorly-regarded teams a number of times (and been on a couple, though I swear my presence was just a coincidence...) - inexplicable from the point of view that the team was made up of good people reliably delivering good work, but this didn't translate into the way the team was perceived. Often this was down to historical reasons, and sometimes political ones, but even when it was identified as a problem it was hard to address when there are so many other priorities.

So if I were building a team from scratch I would attempt to build managing the public image of the team into our ways of working from early on. Following the Team Topologies Team API model would be a good start, but it should extend further than that, actively showing a friendly face to the organisation.

Open showcase and feedback sessions can be great, especially for an enabling or platform team.

The team I'm currently on has been producing weeknotes since the beginning of 2022, capturing our learnings and achievements. Writing these as you go throughout the week then tidying them up and publishing them at the end is a relatively cheap way of regularly producing content for internal (or even external) blogs, and they've also turned out to be useful in onboarding.

I had the good fortune of, a few years back, being the lead on a team with a welcoming, open (within the org), Slack channel which was nominally for internal team communication but where lots of non-members were happy to join and just hang out with us. On that same team we experimented quite early on with mob / ensemble programming, and held our sessions in an open space where devs from other teams might drop by and join in on our work, which often happened. This was awesome, and I'd love to be in a position to do it again (though maybe there's a little more to it in the remote working world, it should still be doable).

#NoNoSQL

The software engineering community does have a tendency to the faddish, easily picking up rallying cries and then getting all fundamentalist. One of the most damaging of those is surely the #NoSQL movement.

(I've been told it originally stood for Not Only SQL, but I can't find a reliable citation for that and it sounds like an after-the-fact etymology.)

"The Right Tool for the Job" is not placard material unfortunately, but sometimes data really is relational and benefits from typing and consistent structure in the datastore, in which case don't just throw everything in a document store and hope for the best.

I feel like the landscape has slowly started shifting back somewhere sensible now, but it still bugs me that I've sat in team discussions about choosing datastores where relational DBs were rejected solely on the point that they were relational DBs, and then any old document store was chosen based on whatever was then shiniest or whichever the team already knew.

Also, SQL gets a worse rap than it deserves. It's a useful query language that continues to evolve in interesting ways, and there is an enormous amount of experience out there to exploit. Since shifting to data engineering I've gained an even greater appreciation for what can be done with it by people who know how.

TDD

The one thing I've learnt during my whole career that positively impacted my coding, productivity, and design it's Test-Driven Development.

I strongly feel this ought to be taught very close to the beginning of learning how to develop software. Pretty much all aspects of good code design emerge naturally from this simple (though it comes with a learning curve) way of working; to my mind it's far more valuable than learning some acronym-d set of principles from a crusty old developer (or another set of acronym-d princples from a younger, attention-seeking iconoclast).

If I were to join a team and be told "we don't do TDD here" I'd do it anyway - the code would be better for it and if I'm working by myself, how would they know? (not that I would join such a team. Except maybe if the money was especially good.)

There's not much I could write about it which isn't said better elsewhere. It just works; do it.

Monorepo

At the time I had little-to-no experience of monorepos, but knew a lot of people and teams who did, and wanted to give it a go.

I've had some experience since, and I'm not completely sold on them, there are pros and cons, and they don't always play naturally with existing tooling and IDEs. There are definitely some upsides, but it's not been a revelatory approach to codebase management for me. That said, the kind of codebase I'm currently working on might not (yet) be one which benefits most from the approach.

Ensemble by default?

As mentioned previously, I've been lucky enough to be on-and-off practising ensemble (AKA mob) programming for a long time now, and when it works it's amazing. It requires discipline, and a lot of buy-in, not to mention psychological safety (though practising it may also help foster that), but it's worth it and I would always at least keep it in the tool box for situations where there's a clear benefit, in particular kicking off new projects and starting out using new technologies and patterns.

We've done quite a lot of it on my current team, which has been a great fit for the work as I've had a lot to learn in the switch to data engineering, and as a team we have a lot to explore and experiment with. This is a situation in which it shines, but I believe it can shine for software development teams generally where the conditions are good.

The non-negotiables

I regret calling these non-negotiables, as that's not how good teams work - healthy teams are always negotiating about how best they can work in their context (especially as no team is truly "stable"), and it's not how I like to work either, but it does impress how important I have found these over the years.

TDD I've already talked about, I'm not sure why it's in here as well.

TBD, or Trunk-Based Development I have talked about at length before in other articles, frequently on social media, a couple of talks given internally at my employer (I also released a git wrapper I created to help teach it), so I won't bang on about it too much here. To me, a team which has full ownership of a codebase rarely has a reason not to do TBD, whether they're co-located or distributed. Sometimes teams will do it out of habit - it's just all they know - or because of a top-down directive, sure, but if a self-organised team chooses to go this way it can be a symptom of other problems like a lack of psychological safety. The best time to start practicing TBD is at the start of a project, the second best time is right now - it's very easy to acknowledge it then kick it down the road to "when we have less tech debt" or "when we have more tests". Don't fall into that trap! (I have before)

FSM? Is that what it says? Not sure what this is about... The only FSM I can think of is Finite State Machines, but that doesn't belong in a group with more "ways of working" type things, nor on a Post-It titled "non-negotiable". I can only imagine it says or means something else, but what that was is now lost in the mists of time...

Conclusion

I'm not sure there is any major conclusion to draw here, it's a collection of notes highlighting how I like to work which I hope might be interesting to others. For me, reviewing and writing about some notes from over two years ago, it has been interesting to see how much I myself have learnt in that time. Working in software development really is about continuous learning, and making sure to have a yearly-or-so exercise in high-level self-reflection like this is a note I will have to make.