Six Years of Professional Clojure

TL;DR

• Clojure is a great programming languages due to its functional nature, lack of objects / concentration on primitive values and the vast JVM eco system available via its seamless Java interop
• Recruiting and building engineering teams of Clojure engineers is challenging compared to other programming languages due to the lack of its popularity and the absence of a large pool of experienced engineers

Preface

After years of working mainly with Ruby I arrived to Nanit. I didn’t really know Clojure back then, so in my first stages I did mostly Ruby work to provide quick value. Chen, Nanit’s SVP R&D, had already implemented some services in Clojure and that’s how I was introduced to Clojure as a language. More than 6 years have passed since and today Clojure is one of the strongest tools in my toolbox and the language I feel most productive with.

During these years Nanit’s backend group became larger and the question regarding choosing Clojure as our main programming language rose over and over again mainly due to the lack of experienced Clojure engineers which affected recruiting and introduced a longer onboarding process until a new engineer could be productive.
When I tried to provide an answer to this question I always felt like I have to recollect my thoughts and organize them into coherent arguments even though Clojure’s strengths were very clear to me all along. I came to a decision that one day I’ll pour my thoughts about Clojure out on a blog post.
This day has arrived :)

I always like to say that even though I’ve been working with Clojure for more than half a decade now, I’m in no way a “Clojure expert”. Since I consider myself someone that tends to dive deep into topics I think this reflects more on Clojure as a language rather than on me as a software engineer. It’s just that compared to other programming languages, Clojure is quite simple, and simplifying a topic turns expertise to esoteric. In other words, Clojure allows you to achieve a lot with little knowledge since there is not a lot to know, which is really great.

Simplicity should not be confused with weakness. On the contrary, Clojure’s simplicity is its main strength, as you can achieve everything you could have achieved with other languages like Ruby, Java or Python with less overhead and accidental complexity in your code.

I want to try and avoid the “language wars” with an absolute conclusion that Clojure is the best language on earth. Clojure is another tool in my toolbox and might fit better to certain use cases than others. Instead I will try to list the objective parameters that made my life easier working with Clojure along with some topics that I’ve had difficulties with both technically with Clojure as a language and building a team of software engineers practicing mainly Clojure as their tool.

Functional Programming

Clojure is a functional programming (FP) language. For me, as a software developer, FPs biggest advantage is that most of the codebase is composed out of “pure functions”. Pure functions have two traits that make them easier to test, refactor and compose together into more complex functions:

1. They are free of side effects. Side effects include network IO, disk interaction or mutating the system state.
2. Their output relies solely on their arguments. They are not dependent in external state to compute their return value.

When I think of how I spend my time creating software I can divide it to 4 main activities:
I read the existing code and try to understand it
I refactor code that requires refactoring
I design new code before I write it
I write new code with tests — this code probably re-uses existing code

The combination of the two traits above makes any of the listed activities easier for me:

1. Pure functions make code design easier: In fact, there’s very little design to be done when your codebase consists mostly of pure functions. You don’t have to build class hierarchy with interfaces, extensions and implementations. There’s no need for advanced design tricks like composition over inheritances or the visitor pattern. You don’t have to find creative solutions to the multiple inheritance problem or the dreaded diamond diagram. I haven’t dealt with any of these for the last 6 years and yet I produced well crafted, tested, maintainable, readable, extensible production-grade code (or at least that’s what I would like to believe :) ).
2. Pure functions are easier to re-use: I can use a pure function as many times as I’d like without having to take how it affects the system into consideration since there aren’t any side effects. It’s like the WYSIWYG of computer programming — the function follows its body and nothing else. No hidden considerations to be accounted for.
   Pure functions encourage code re-use by removing the extra overhead of having to investigate whether the code I’m going to re-use affects the system and if so what are the implications of that.
3. Pure functions are easier to read and understand: Each pure function is an isolated, consistent and predictable piece of code that only relies on its arguments. You don’t need to be familiar with the database schema or the RabbitMQ architecture to reason about the code — it’s all about the arguments and the data transformations done in the function body.
4. Pure functions are easier to test: Since they don’t rely on external state all you have to do to test the function is to apply it on its arguments. There’s no need to create fixtures on the database or mock an HTTP request. Also, since pure functions don’t apply any changes on the system all you have to test is the return value.
5. Pure functions are easier to refactor: Their lack of external dependencies and their statelessness turns them into an isolated building blocks that are easily replaceable and composable.

Only Values, Only Primitives

Clojure does not have “objects”. I mean, it does, but most of the time you won’t feel any need for those. Instead, Clojure relies on primitive values and collections of those (arrays, dictionaries, sets etc). 99% of what I do in Clojure is working with arrays and dictionaries that contain primitive values.

Dealing with primitive values is easier for me as a software engineer:

1. My code focuses on business logic and data transformations rather than describing the domain and its relations. Every line of code is executing business logic and by this the business logic is very prominent across the codebase.
2. I don’t have to be familiar with hundreds of unique objects and the behaviors coded into them to be effective:
   An incoming HTTP request? it is a plain Clojure dictionary.
   You want to form an SQL query? build a dictionary and pass it on to the SQL library for formatting.
   You want to return an HTTP response? You return a dictionary with the keys of status code, header and body.
   Want to read a message from a RabbitMQ queue? Yep, you guessed it — you get a dictionary.
   If you’re familiar with Clojure operations on its basic data structures like dictionaries you become effective in HTTP, SQL, RabbitMQ and every other domain specific part of your system.
   It reduces the complexity and the level of familiarity you need to have in a domain to the minimum required since from the software side, all you do is repeatedly building, transforming and moving dictionaries from one function to the other.

Minimal Syntax

Clojure’s syntax is built out of its own data types. This property is called homoiconicity. It sounds strange at first but I’ll try to demonstrate:

Clojure vectors (arrays in other languages) look like this [1 2 3 4]
Clojure lists look like this: (1 2 3 4)

To define a function you would write:

(defn my-sum [arg1 arg2] (+ arg1 arg2))

As you see, the code is a Clojure list with the symbol defn, the function name and then comes a vector of arguments. The body is a list with the function as the first member (+) and the arguments follow.

Why is that a good thing you might ask yourself? Good question!

1. Generating code via macros feels very natural. Since most of what we do in Clojure is transforming and generating data structures in favor of business logic, doing the same, with the same data structures, to generate code is mostly unnoticeable.
2. It reduces the number of special symbols and characters you have to be familiar with to minimum. Code and data become one as they share the same data structures, behaviors and syntax.

Concurrency

Concurrency feels like a non-issue when working with Clojure mainly due to 2 reasons:

1. The majority of Clojure’s values are immutable which prevents race conditions and allows code that is free from shared access controls like mutexes and locks.
   Those who are not immutable (atoms for example) provide safe ways to manipulate the data they store.
2. Clojure has a great collection of tools for concurrent programming called clojure.async. The highlights of these tools, at least from my experience, is Channels, which allow safe inter-thread communication and selection over a set of channels much like Golang’s select directive.

Java Interop

Clojure is not a widespread programming language, and as a such, many libraries are missing for common use cases. Fortunately, Clojure’s interop with Java is seamless so in practice the vast eco system of Java is at your fingertips. This way you can enjoy working with Clojure but do not suffer from its lack of popularity and libraries.

It’s not all flowers and butterflies

Yes, Clojure is great, but like most decisions we make in life, the decisions that were made with Clojure are also tradeoffs.

The first aspect of Clojure that made my hard life is the JVM and for 3 reasons:

1. The JVM is a known memory eater and it is very hard to predict your application memory requirements. Also, it always seems to require more memory than needed to run the application. I am sure that the same applications would take significantly less memory on other runtimes (although I never took the time to prove it).
2. Debugging memory leaks and heap size in remote servers is very hard. We tried VisualVM but since Clojure memory consists mostly of primitives (Strings, Integers etc) it was very hard to understand which data of the application is being accumulated and why. I assume that in common Java based application most of the memory consists of Java objects so the memory profiling would be easier.
3. The boot time for Clojure projects might become very long as the project is growing in size. Although there are solutions like GraalVM I haven’t had the chance to experience them in production to testify on their matureness and robustness.

To sum it up, I’m not a fan of the JVM, but I do understand the reasoning behind the decision of targeting Clojure’s runtime to the JVM.

The second topic I find difficulty with when working in large, unfamiliar Clojure codebases is Typing. Clojure is a dynamic language which has its advantages but not once I stumbled upon a function that received a dictionary argument and I found myself spending a lot of time to find out what keys it holds. Sometimes I had to put a log in our integration environment to see what message it receives and what are the fields that are available for me in that message. Sometimes I would go to the tests for that function and look for the example argument value we used in the tests but that might not be enough because there might be other fields that exists in that dictionary and are just not being used in the function at the moment so they might be missing from the test value as well. Sometime I would look at the function’s call site to understand what argument has been passed and how it was built.
There are solutions to that as well, like core.typed but I never experienced them myself and I am not sure of how comprehensive and usable they are.

The last thing that feels hard with Clojure, and I’ve already touched it earlier in this post, is recruiting and onboarding. Recruiting is hard because the pool of existing Clojure engineers is very small and some engineers deliberately refrain from working with unpopular languages due to career advancement considerations. Other engineers gain expertise with specific languages and would like to continue and work with these languages so Clojure is not an option for them.
Onboarding also requires more attention and guidance since most engineers arrive with little to no knowledge of Clojure and its eco system. When a NodeJS engineer joins a company, they already know javascript, they’re familiar with the eco system, they have a favorite IDE and plugins and they know what tools makes their local development environment as productive as it can be.
When engineers join an organization that works with Clojure without prior knowledge, they have to learn the language, find an IDE they’re comfortable with, adapt a new development flow and configure their development environment to be able to be productive. It’s almost like learning to tie your showed all over again and it has to come with the right amount of guidance and availability from existing engineers.

Another interesting issue that the lack of Clojure popularity introduces is that it is hard for new engineers to bring Clojure specific knowledge from outside into the company and enrich the existing team. Going to the NodeJS example again, an engineer with vast experience that joins a new team may introduce new tools / libraries / work methodologies / development flows they gained expertise with in prior companies. Engineers that come with no prior experience with a specific domain cannot really enrich the team in the same way so the team has to rely on self learning and improvement rather than bringing knowledge from the outside.

Summary

I think that every software engineer needs to at least get theirselves familiar with one functional programming language just to open their mind and see outside the OOP paradigm. Learning Clojure made me doubt everything I practiced before as a software engineer and ask questions on the very basics of how I spend my energy on the right direction to provide the company I’m working at value.

I think that Clojure, being a mature, production ready and a simple programming language, is a great candidate for that exploration. You may choose to use it professionally, for side projects or not at all, but the experience of exposing yourself to this language will surely enrich the way you think of programming and make you a better developer.