As a Senior Software Engineer 2 on FlightAware's Predictive Tech crew, Andrew Brooks works to maintain, analyze, & improve FA's predictive models as well as the software & infrastructure that supports them. Andrew is also the tech lead of the Engineering Productivity crew, which designs and maintains internal tools and standards to support FlightAware's engineers.

Overview

At the time of writing, most articles on the Nix package manager are written with individuals or small teams in mind. Unfortunately, these articles do not offer much insight or advice on adopting Nix across an entire engineering organization. This blog post is our attempt to fix that.

In this blog post, we’ll provide an account of the problems that motivated FlightAware to adopt Nix and how we’ve used Nix to solve them at scale. Additionally, we’ll share some insight into the patterns, infrastructure, and practices that allow us to use it effectively, and offer some tips for both individuals and teams that are considering working with Nix.

Although this blog post assumes no familiarity with Nix, having some prior Nix experience may help readers get the most from our advice.

FlightAware Circa 2020

Our earliest experiments with Nix occurred around 2020 in response to a few day-to-day problems that FlightAware engineers didn’t have a good answer for.

Some languages we used had no package manager

FlightAware has a substantial amount of legacy code written in Tcl. Unfortunately, Tcl did not have an actively maintained package manager. This means that if you want to install a Tcl package, you’re on the hook for figuring out how to install it and its entire dependency tree. To make matters worse, Tcl packages often don’t clearly state their dependencies, so even identifying what needs to be installed often proved to be a hassle.

FlightAware also has several performance-critical libraries and applications that are written in C++. Unfortunately, the package management landscape C++ is bleak: even though there are a few C++ package managers, none are widely accepted. When building internal C++ projects, engineers often resorted to doing git operations with CMake to pull in any dependencies.

Creating Docker images was miserable

If you’ve ever written a nontrivial Dockerfile, you’ll know that building a Docker image for an application with a complex dependency tree would be a tedious endeavor without a package manager. Furthermore, Docker builds were often painfully slow due to the large number of repositories and build steps involved.

Changing any steps early in a complex Dockerfile was especially painful: Docker can’t tell whether running one step will affect the outcome of another, so it’s forced to re-run every step after the one you changed. Leveraging multi-stage builds is a common way to address this problem, and it works very well when installing something results in a manageable number of obvious paths to copy between stages. Unfortunately, this was not the case for many FlightAware libraries. In practice, Dockerfiles using multi-stage builds were sometimes so much more verbose than their naïve alternatives that they drove engineers back to less efficient, conventional Dockerfiles.

One approach taken by some FlightAware crews was to assemble one giant Docker image containing almost every FlightAware package, suffer through its long build time, and use it as a base for all applications. Unfortunately, this had a few drawbacks of its own. In addition to producing enormous images, you were still potentially stuck waiting on a very long build if you needed to change some component in the base image and your application in tandem. Some packages also proved challenging to install on top of a “comprehensive base image” because they required a conflicting version of a library or compiler present in the base image.

What we really needed was a way to say “go build/install application X and its dependencies – without repeating work unnecessarily – and give me a Docker image” without being drowned in an ocean of problems and lengthy build times.

Multi-language projects could be challenging to support

FlightAware had (and still has!) a “just use the best programming language for the problem” policy. Accordingly, many of our projects make use of native extensions to achieve performance goals or enable interoperability with native libraries. From a package management perspective, mixing languages could be a hassle because it’s frequently necessary to jump around between different package managers and accommodate languages that might not have a well-established one. Additionally, some languages in use at FlightAware tend to compile slowly (for example, one of our mission-critical Haskell projects typically takes about 40 minutes to compile its dependency tree), which exacerbated our frustrations with long Docker image build times.

What’s the Deal with Nix, Anyway?

Before we can describe how we’ve used Nix to address these problems, we first need to offer a quick overview of Nix and some basic concepts needed to understand why it’s been helpful at FlightAware.

In short: Nix is a package manager, a domain-specific language (DSL) used by that package manager, and a command line toolset based around the previous two. You may also have heard of NixOS, a Nix-centric Linux distribution (but you can comfortably use Nix without using NixOS)[1]. You’ll also hear “nixpkgs” (the Nix packages collection) mentioned alongside Nix. Nixpkgs is a git repository that offers common packaging functions and Nix package definitions for over 70,000 packages (to put that number in perspective, that’s currently more than twice as many packages as are available in the entire Ubuntu 22.10 repository).

At this point, you might be imagining Nix as “another apt/rpm/pip.” If so, think again! Nix provides several features and guarantees that more conventional package managers are unable to, requiring some surprising technical decisions to enable them:

·      Nix is a “multi-language package manager”: it supports a wide variety of programming languages and build systems. Conveniently, it’s sometimes able to reuse other package managers’ package formats and give you a Nix package “for free.” Supporting projects in multiple languages means that you can avoid jumping pip/apt/etc. to install your software.

·      Nix packages are built reproducibly: no matter where you build them, you’ll always get the same result[2]. There are several package managers that claim to provide reproducible builds, but they’re generally limited in what they can control for in comparison to Nix. Very few of them use a build sandbox, manage the entire build toolchain, account for native dependencies, or handle dependencies from other languages like Nix does.

·      Nix does not install packages into FHS paths, like /usr/bin or /opt, but instead installs into a special directory called the “Nix store,” usually placed at /nix/store. The exact paths inside the store are determined by a cryptographic hash of the instructions used to build each package (which includes source code and store paths of dependencies) and the package’s human-readable name[3]. It’s important to note that paths inside the store are immutable: once written, their contents never change.

·      Nix makes strong isolation guarantees for packages: installing one package can never break another. Additionally, you can simultaneously install as many different versions or customized variants of a package as you’d like without worrying about conflicts between them. Nix treats each of these versions/variants as distinct packages and installs them to distinct paths inside the store. If you change a build instruction, alter any inputs to the build sandbox, or change any of the input sources, you create a different package and therefore a different path in the store. Combined with Nix’s reproducibility guarantees, this means that if the same path is present inside the Nix store on two different hosts, it will have the same contents.

·      In Nix, the notion of a “package” is slightly different than in other package managers. To simplify somewhat, a “package” in Nix is a function of all inputs needed to build paths in the Nix store, which you (or another host) may already have built[4]. The fact that Nix always carries around a description of how to build a package is extremely powerful for one-off experiments and package customization, as we’ll show later.

·      Nix has a DSL for defining these packages (unfortunately, this DSL is also called Nix, a point of confusion for new users). The Nix language facilitates succinct, declarative package definitions[5], enables flexible customization of already-defined packages, and allows defining arbitrary functions to encapsulate common packaging patterns and logic. Although the Nix language is very capable, it is not intended as a general purpose programming language.

This is not an all-encompassing list: Nix boasts an extensive (and growing) set of features and guarantees that we’re not going to attempt to list exhaustively.

Solving Our Problems With Nix

Here’s where things get interesting: at this point, Nix seemed like a promising package manager choice for FlightAware. While Nix eventually proved to be a powerful solution for the problems we were facing, the road to using Nix effectively across FlightAware took a few surprising turns.

Could Nix be the package manager we were missing for Tcl and C++?

Nix already supports several languages’ common build/install patterns, package managers, and language-specific package formats. Unfortunately, at the time, Tcl was not among the languages supported by nixpkgs. However, because Nix already supports several other interpreted languages, it wasn’t difficult to add some conveniences for packaging code written in another. In late 2020, I implemented Tcl packaging support for nixpkgs, which wound up being more straightforward than I’d expected. FlightAware later contributed this work to nixpkgs, and this functionality is included in any recent nixpkgs release as the tcl.mkTclDerivationfunction.

Unsurprisingly, C/C++ were already very well supported by Nix. Nixpkgs contained an extensive collection of C/C++ packages and adding your own tended to be easy: packaging programs or libraries with a commonplace autotools or CMake configure/build/install process was often as simple as specifying immediate dependencies, where to find the source code, and basic package metadata (like a package name and description).

At this point, the case for Nix was compelling: not only did we have a package manager for two extensively used languages that didn’t really have one, but we could use the same package manager for any other languages at FlightAware. This made it easy for us to get a comprehensive picture of the dependency tree for FlightAware projects -- even across languages -- when doing so used to be impractical. As frosting on the cake, the Nix CLI includes several tools for querying and visualizing dependencies in addition to allowing inspection of packages from within the Nix language.

Building the FlightAware package overlay

Now that we had support for common FlightAware languages figured out, we needed to actually write new Nix packages for FlightAware internal libraries and applications. For a more conventional “system-level” package manager (like apt), you’d generally accomplish this by creating a new package repository and populating it with packages built by a CI server based on some spec.

Nix, however, is a little different: to add your own tweaks and custom packages, you generally define something called an overlay. In simplest terms, an overlay is just a function, written in the Nix language, that describes how to extend or modify some base package set (typically nixpkgs). Conveniently, those modifications can be “fed back into” the base set of packages as inputs (i.e., if you modify the “sqlite” package in your overlay, any packages in the base set would be redefined to build against your customized sqlite).

Both nixpkgs itself and the internal package overlay we applied on top of it are “just” git repositories containing Nix expressions. This turns out to be very powerful: you can upgrade or roll back the set of packages in use for a project just by specifying a different git branch and commit of the FlightAware overlay and nixpkgs itself. The packages you use are just code like everything else. Engineers could experiment with upgrading/tweaking/modifying packages on their own git branch and install them without worrying about stepping on other engineers’ toes or having to coordinate uploading packages to a shared apt repository.

One important distinction to note is that these git repositories don’t contain “already built packages,” only the Nix expressions that describe how to build them. However, this doesn’t necessarily mean that you’re forced to recompile and reinstall software every time you install a package. We’ll talk more about how this works later. In the meantime, take our word for it.

With support for the most common languages at FlightAware and several new FlightAware packages at our fingertips, we were off to the races.

Creating Docker images without Dockerfiles

Ordinarily, when building a Docker container, you’d write a Dockerfile with one or more RUN steps that invoke the package manager of your choosing to install whatever packages you need or build any software that you need to. Surprisingly, this isn’t how you’d usually create a Docker image with Nix. As a matter of fact, you generally don’t need a Dockerfile at all.

Nixpkgs defines several functions for building Docker images with just a Nix expression. Behind the scenes, Nix builds any packages or paths that you’d need inside the image as “ordinary” Nix store paths in the usual Nix build sandbox. These paths are then copied from the Nix store into a layer of the image being built. Although copying in files from the build host like this might sound dangerous, Nix’s reproducibility and isolation guarantees make it safe – after all, if you get the same result anywhere you build a Nix package, who cares if you build it inside a Docker container, outside of one, or somewhere else entirely? Finally, any additional shell commands needed to set up the image are run in either the Nix build sandbox or a disposable VM.

Nix offers us a pleasant API for building Docker images – you don’t have to do much beyond specifying the packages that need to go in each layer and any shell commands that need to run to finish setting up that layer.  The arrangement of the layers in the final image isn’t determined by “steps” as it would be with a Dockerfile -- Nix allows you to split package installation / setup across layers however you’d like. In fact, Nix’s dockerTools can optionally leverage Nix’s understanding of packages’ runtime dependencies to guess how to split the contents of your image across layers in the hope of maximizing layer reuse across images.

For example, here’s how a Nix language expression would look like to define a single layer Docker image with some application, an interactive shell for debugging, and basic user setup, from scratch:

When building an image with Nix, the Nix store acts a bit like the Docker layer cache. However, it caches not only image layers but also built packages themselves, their dependencies, and some intermediate build artifacts (like downloaded source code). This allows Nix-based image builds to achieve much finer grained caching than would generally be practical with a Dockerfile using a multistage build. For example, if installing an NPM package from a node2nix-generated Nix package, the Nix store would separately “cache” the downloaded source tarball for each NPM dependency. The only way to achieve a comparable level of caching in a multistage Docker build would be to create a separate stage describing the download of each dependency source tarball. For a nontrivial Node application, writing a multistage Dockerfile like that would be completely impractical.

Docker images produced by Nix tend to be very small: Nix knows exactly which store paths an installed package depends on at runtime and pulls in only those paths to the created Docker image. In our experience, Nix does this a little too well, and we occasionally need to manually pull in coreutils/bash/etc. so that we’ll have enough basic tools inside each image to effectively debug running containers when necessary.

In short, Nix’s dockerTools offered us an immense quality of life improvement when creating Docker images by making them simple to define, fast to build, small, and easy to iterate on quickly.

Setting up a private binary cache server

There’s one important and nuanced detail that we’ve glossed over so far: we’ve said very little about how already built Nix packages are distributed -- after all, we don’t want to waste time recompiling packages if we can avoid it. This is where binary cache servers come into play.

Basically, a binary cache server has a large Nix store with paths that are likely to be useful to other Nix installations and serves those paths to other Nix installations. When trying to install or use a package, another Nix installation will first check against a list of binary caches to see if it can find an already built store path for that package. If so, and if the paths are signed by a key that it considers trustworthy, it’ll download and install the package from the binary cache instead of building it. Otherwise, it just builds the package itself.

With a more conventional package manager, you might build and upload packages from a dedicated server or build farm. However, Nix enables a more decentralized approach: you can have any Nix installation automatically sign and upload paths to the binary cache as soon as they’re built, allowing them to be automatically and transparently reused elsewhere. This works especially well given how many variations on a package may need to be cached at once because of how easily and frequently packages wound up overridden or tweaked(e.g., from an engineer applying our overlay to a different nixpkgs release or modifying a package when testing something).

We’ve found that an internal binary cache deployment like this has some compelling advantages:

·      For our predictive tech crew, it allows data science environments to be quickly, reproducibly, and effortlessly shared between developers and development machines. This is especially useful given that some of them customize large swaths of the package set (e.g., by swapping out the BLAS implementation that linear-algebra-heavy packages are built against), which could otherwise result in disruptive, long build times.

·      In general, it makes jumping between development hosts easy: any packages you built while developing on one server should be trivially reusable on another, with no additional work or compiling required.

·      It can make Docker image builds incredibly fast: if a package being installed into a Docker image was built anywhere in the past and uploaded to the cache, it can just be downloaded from the binary cache and immediately copied into a Docker layer. If you’re experimenting with something outside of Docker on a dev host, then build a production Docker image on a CI/CD server using Nix, you won’t have to re-compile anything when building the Docker image (assuming both environments are submitting to/pulling from the same binary cache).

“The hard part” of setting up a binary cache

When productionizing our binary cache infrastructure, we found that there was a little more plumbing involved than we’d initially anticipated:

·      The automatic upload is triggered by the Nix daemon’s post-build-hook setting. Unfortunately, you’ll need to implement the actual build hook script responsible for signing/uploading the package (the Nix manual has a trivial example).

·      As noted by the Nix manual, the post-build-hook is blocking and fails the build if it doesn’t succeed. To prevent slowing down or breaking builds, our post-build hook merely submits a path for upload to a dedicated cache upload service we created that runs alongside each Nix installation. This allows the post-build hook to run quickly and get out of the way while the upload service does the “heavy lifting.”

·      It’s probably a mistake to blindly upload everything you build to your binary cache. For example, uploading Docker layers or source code to the cache isn’t actually very helpful: they tend to be large and can be downloaded from sources other than the binary cache anyway. The upload service that we install alongside Nix uses a configurable set of heuristics for excluding store paths that are either too expensive to cache or seem unlikely to be useful.

·      To prevent the binary cache’s Nix store from growing endlessly larger, you’ll want to configure automatic garbage collection (we’ll discuss this later).

If setting this up seems daunting, keep in mind that you don’t need to set up your own binary cache to make effective use of Nix. Note that you can also manually move built packages between servers without a binary cache if desired. However, for us, investing the time to set up an internal binary cache and automatic package submission has been well worth it.

On a related note, Nix has good support for transparent, multiplatform, distributed builds. While we don’t currently use this functionality, other teams or organizations might benefit from this flexibility.

Solving More Than Known Problems

In addition to offering elegant solutions to problems that we’d recognized, Nix proved to be useful for several reasons that we didn’t initially expect.

Unified Tooling and Infrastructure for Multi-Language Development

Although package management for Tcl and C++ motivated our initial adoption of Nix, we soon found that Nix still has a lot to offer when developing in languages with well-established package managers, especially for multi-language projects and services. Nix has been very effective as a unified package management and build solution, used across 12 teams for languages spanning C++, Tcl, Python, Haskell, Rust, Golang, and more. For languages with well-established package managers and practices, Nix is often able to trivially leverage existing language-specific package formats, which makes it “cheap” to add to a project.

Conveniently, Nix also has good support for describing and entering development environments with temporary installations of specific versions of compilers, interpreters, and other development tools. This means that engineers don’t need to waste time learning and tangling with language-specific tools to manage per-user or per-project installations of these tools. When engineers work on a project with a well-written shell.nix, you seldom hear the infamous phrase, “well, it worked on my machine.”

Nix (the language) is impressively powerful

Once you’re familiar with the Nix DSL, it’s surprisingly easy to arbitrarily tweak or experiment with variations on packages. It’s particularly easy to describe a new package in terms of an existing one within the Nix language. For example, let’s say you’d forked the kubectl GitHub repo and made a few tweaks. If you want a Nix package for your “custom kubectl” you can define it in terms of the existing one, keeping all old build/configure/install steps, except the source code will come from your fork instead:

Additionally, Nix’s overlay mechanism makes it easy to experiment with changes that span large swaths of the entire package set in a way that’s inconvenient or even unthinkable to do with a more conventional package manager. For example, suppose that you want to test the performance impact of enabling the AVX ISA extension across all C/C++ packages. Nix makes this trivial: in just a few lines of the Nix language, you can define and apply a package overlay that modifies the compiler flags used in the standard build environment and let Nix call all existing package definitions again with that new build environment.

Seriously, it only takes a few lines of Nix, and you can refer to AVX/non-AVX package definitions side by side:

If your team has internal conventions or team specific patterns in build/install processes, it’s easy to create and reuse functions in the Nix language that encapsulate them. For example, at FlightAware, we have convenience functions for pulling sources from our GitHub Enterprise deployment with reasonable defaults or offering shortcuts to allow commonly used repositories to be easily overridden during development.

Cross-compiling doesn’t have to be hard

Nixpkgs uses the Nix language very effectively to make cross-compiling relatively painless. Nixpkgs defines cross-compiled package sets for several common systems out of the box, and working with them is often as simple as just changing a variable assignment (check out the NixOS cross-compilation guide for more details). If you’ve been eyeing those ARM instances in AWS EC2, you’ll generally have an easier time experimenting with them if you’re packaging your application with Nix.

The Nix community is easy to work with

At this point, FlightAware engineers have contributed PRs to nixpkgs and to Nix itself. It’s generally been easy to navigate the PR process, and any interactions we’ve had with maintainers have been positive and constructive.

The Ugly Parts

Until now, we’ve painted a rosy picture of Nix. Make no mistake: while Nix has been immensely useful at FlightAware, there are a few aspects of Nix that have proved challenging to work with. We’d be remiss if we didn’t mention some of the “ugly parts” of Nix that we’ve encountered and briefly discuss how we’ve been able to address them.

The intended audience for Nix documentation varies considerably

Like any documentation, Nix documentation is written with a specific audience in mind, and that intended audience can vary significantly depending on what you’re reading. Unfortunately, that audience isn’t always clearly identified, and unknowingly encountering something intended for a different audience can be off-putting for engineers with little Nix experience. For example, the Nix Pills are well written but can be divisive among new users. Although they’re very effective at introducing many of the design decisions behind the Nix language and nixpkgs, explaining Nix “from first principles” can seem far removed from the day to day realities of working with it. This may frustrate new users if they were expecting documentation describing the latter.

Nix is simple, but really weird

When learning how to use Nix, the refrain seems to be “simple, but weird.” Nix isn’t complicated so much as it is unusual. Most package managers don’t treat packages as functions or have their own DSL (much less a functional one with non-eager evaluation semantics). Unfortunately, while oddities like these in Nix generally exist for good reason, they can make the learning “curve” look a lot more like a step function if you’re not careful.

From the perspective of someone unacquainted with Nix, “simple, but weird” is intimidating because it initially doesn’t look any different from “really complicated.” Any organization trying to achieve widespread Nix adoption will need to invest time both into helping engineers climb the learning curve quickly and demonstrating why they’d want to. There’s not any one way to do this effectively, and you’ll need to adapt any internal docs/explanations/examples based on the backgrounds of the engineers you’re working with. For example, the Nix language would likely be relatively easy for someone with Clojure experience to pick up. However, it may require careful explanation to someone with a background in exclusively imperative, non-expression-oriented languages.

It’s worth mentioning that Nix’s upcoming “flakes” feature helps address this issue by making the CLI more consistent and standardizing some patterns commonly used for multi-repository projects (more on this later).

MacOS support is second-rate

Nix officially supports MacOS, which is convenient for FlightAware (most FlightAware employees’ development machines run MacOS). Unfortunately, if you’re expecting to replace homebrew with Nix, you might be a bit disappointed: if you use Nix on MacOS for any extended period, expect to brush shoulders with a few broken packages. However, the landscape is quickly improving, and we’d be surprised if this remains a problem for long.

Handling private repos can be clumsy

Unsurprisingly, most Nix examples and packages are built with publicly visible open source repositories in mind. However, as you add your own internal packages, you’ll likely need to check out sources that come from private repositories. This means that you may need to use “impure” fetchers (e.g., like using builtins.fetchGit instead of pkgs.fetchgit in order to prevent the build sandbox from blocking SSH agent forwarding) on occasion. Some of Nix’s functions for handling other package managers’ package formats may not know to do this, which is an occasional headache.

Thankfully, in practice, this is little more than a minor inconvenience: it’s generally not difficult to override any offending functions to use a private repo-friendly built-in when needed (and if your team sets up a dedicated overlay, that’s a great place to make that override). The upcoming “flakes” feature can also be helpful here: repositories fetched as flake inputs “just work” (more on flakes later).

“Angle-bracket paths” are troublesome

In many Nix tutorials/blog posts, you’ll see Nix language expressions that start out with something like:

Roughly, this means, “when evaluating the expression that follows, bring into scope everything defined in the package set loaded from the ‘nixpkgs’ channel.” This is a convenient shortcut for one-off experimentation, but it has a serious drawback that’s very easy to overlook: the specific release and revision of the “‘nixpkgs’ channel” is environment dependent. Different hosts and different users may have different channels configured called ‘nixpkgs.’ Practically speaking, this means that if you share code that uses angle-bracket paths, you’ll eventually see behavior that looks nonreproducible because you may be unknowingly asking Nix to build different packages on different hosts. It’s a shame that it’s so easy for Nix’s reproducible builds to appear otherwise to new users.

Fortunately, once you know to avoid this, it’s a nonissue: you can easily “pin” the package set in use to a specific branch/commit. Our internal Nix standards require avoiding angle bracket paths altogether and pinning the source set. Nonetheless, many Nix articles and examples that start this way assume that you’re already aware of the drawbacks (unfortunately, Nix newbies usually aren’t).

Conveniently, Nix’s upcoming “flakes” feature helps address this problem.

Flakes are still an experimental feature

We’ve mentioned “flakes” as being useful for addressing some of these concerns. To make a long story short, a flake is a composable unit for packaging Nix code that standardizes the structure of Nix-based projects and adds several conveniences for a Nix-based workflow.

Flakes are tremendously helpful, and we could dedicate an entire blog post to discussing how they work and why they’re useful. However, we’ll stay on topic and stick to the bottom line. Concretely, flakes improve a Nix-based workflow in several respects:

·      Flakes make it much harder to shoot yourself in the foot with angle-bracket paths – your Nix packages/expressions would generally reference a “flake input” instead. With flakes, Nix handles pinning any such inputs for you.

·      Updating or overriding inputs to the flake (like nixpkgs, the repo that holds your team’s overlay or some other repo) can be done entirely from the command line without manually editing any Nix expressions.

·      The Nix CLI is overhauled to support new operations on flakes and in the process becomes more consistent and intuitive.

·      Repositories with Nix sources follow a standard structure and have a standard entry point (flake.nix), which makes it a lot easier to pick up and understand others’ Nix-based projects.

·      Flakes generally improve the performance of evaluating Nix expressions.

Several new and immensely useful Nix features are also based on flakes. For example, you can use the Nix CLI to generate new projects using templates from any flake that provides them (this is a great way to ensure that new projects adhere to your team’s standards from the get-go) or automatically figure out how to create a .deb, .rpm, or Docker image from a Nix package.

Flakes do a lot more to improve the Nix experience, and we’re glossing over a lot here in the interest of keeping this blog post accessible and focused. However, if you’re looking for more details about flakes and the motivation behind them, we’d recommend reading Eelco Dolstra’s introduction , this tutorial from the Serokell Labs blog, or even just the Nix manual’s section on flakes.

This brings us to the only thing that we don’t like about flakes: at the time of writing, they’re still considered an experimental feature. Although the flake CLI seems to have stabilized significantly as of late, there’s still no guarantee that there won’t be breaking changes. Obviously, this makes flakes regrettably difficult to recommend for production use just yet.

Other Nix Tips and Tricks

At this point, if you’re considering adopting Nix at your company, we have a few useful tips to share based on what’s worked well for us and what we wish we’d known before adopting Nix.

Flatten the learning curve

Nix is a powerful tool, but you should expect to invest time in flattening the learning curve to help your team learn to use it effectively. What this looks like will vary between teams, but we’ve generally had luck with writing thoroughly commented example projects, documenting common development tasks, establishing best practices/guidelines for using Nix, and when appropriate, pair programming. Having a dedicated overlay for your team can also be helpful for flattening the learning curve by serving as a shared source of convenience functions for team specific, common development patterns and addressing complaints raised by other engineers.

You should also make sure that your fellow engineers know about nix repl, which is altogether too easy for new Nix users to overlook. Being able to interactively examine expressions and packages is a great way to experiment with the Nix language and get your hands dirty. Experimenting with the REPL made Nix “click” for more than a few FlightAware engineers.

Care and feeding of your Nix store

When installing Nix and setting up the Nix store, there are a few tips and tricks that can make your Nix experience a lot more pleasant:

·      Set up automatic garbage collection: in addition to installed packages, the Nix store holds (some) intermediate build artifacts and downloaded packages that aren’t currently in use. It may also continue to hold previously installed packages that aren’t currently needed by anything. While this can be a real time saver for future builds, it means that /nix/store will tend to keep increasing in size unless garbage-collected (if you’ve ever run “docker prune,” it’s roughly the same idea). Thankfully, Nix can do this for you if free space drops below a configurable threshold, but you’ll have to tell it to.

·      Give /nix its own volume when feasible: this is helpful for making the most of automatic GC (it won’t be triggered by other files on disk taking up a lot of space) and gives you the flexibility to select a filesystem well suited to the Nix store.

·      Choose a good filesystem for the Nix store: using a filesystem that supports transparent compression, like btrfs or zfs, is a good idea. Even if your compiled packages often don’t compress well, transparent compression can still be helpful to compress sources or scripts that are unpacked into the Nix store while building a package. Note that Nix does use the store for locking and maintains some state in a sqlite database under /nix, so it’d be wise to avoid a distributed/networked filesystem.

·      Consider enabling automatic store “optimization”: this automatically checks for and replaces paths in the Nix store with identical contents with hard links to only one copy[6]. This cuts down on disk usage when you have several different variations/versions of a package in the store at once.

Addressing non-NixOS quirks

Using Nix on a Linux distribution other than NixOS is a great way to try out Nix without uprooting any existing Linux installation. However, when you’re not using Nix on NixOS, there are a few quirks that are easy to overlook compared to a NixOS environment. We haven’t seen very many tips on managing such installations, so we’ll provide some basic tips that we wish someone had told us:

·      The Nix installer selects the “unstable” nixpkgs channel by default. There are fewer surprises on the latest stable channel, so we’d recommend switching ASAP unless you have a good reason to prefer the very latest nixpkgs.

·      On NixOS, you upgrade your system and its configuration with nixos-rebuild. On non-NixOS Nix installation it’s a little different: you pull the nixpkgs channel and do a nix-env --upgrade as root to upgrade the default profile. If Nix itself was upgraded, you may also need to restart the nix-daemon service manually.

·      Running Nix commands as root (eg, to upgrade packages in the system-wide profile) may require using the -i flag for sudo.

·      Consider adjusting/dev/kvm permissions: some features of Nix’s dockerTools can leverage KVM for acceleration when building images if you need to run any commands “inside the image” to prepare a layer. Unfortunately, Linux distributions vary in the permissions that they assign to /dev/kvm, and some will prevent the Nix build users from using it by default. If practical, we’d recommend setting up a udev rule to grant +rw on /dev/kvm to benefit from KVM acceleration. If this isn’t an option for you, don’t worry: dockerTools can fall back on unaccelerated qemu if you’re using nixpkgs-22.05 or later.

One Last Thing: Don’t Rush Into It

We hope that this article was able to shed some light on how we’ve been able to use Nix effectively across the entire FlightAware engineering team, and we hope that it proves helpful to anyone thinking of doing the same. In closing, we’d like to offer one last piece of advice: Although Nix has served us well at FlightAware, for you and your team, its value proposition will vary depending on what problems your team is solving, which features are already available with your current tooling, and the backgrounds and experience levels of the engineers that you work with. At the end of the day, package management has a significant social dimension, and whether/where to adopt Nix should be a discussion with your team.

_________________________________________________________

[1] It’s trivial to install Nix on Ubuntu or most other Linux distributions, and Nix thankfully won’t interfere with more conventional package managers on your system.

[2] Nix goes to great lengths to ensure reproducibility, but it’s technically still possible to cause non-reproducible behavior in specific circumstances (eg, if your compiler isn’t deterministic or if the kernel’s behavior changes in a way that affects a build). However, Nix provides tools to verify reproducibility and upholds this guarantee quite well in practice.

[3] This is a simplification: there are situations where the contents of something are hashed and used in determining the path inside the store. For example, some Nix built-in functions that fetch source code behave this way.

[4] Nix experts may take issue with how this blog post uses the word “package.” Formally, we’re using “package” to describe a derivation, a Nix language expression that produces a derivation, or a store path built from that derivation. We’re deliberately glossing over the distinction because we think it obscures the basic ideas in this blog post. If you don’t know what any of this means, don’t worry about it.

[5] The Nix language is “so declarative” that new Nix users often mistake Nix sources for configuration!

[6] It’s a good thing that store paths are immutable – this would otherwise be dangerous!