Introducing Nix

Declarative builds and deployments

codgician

2024.05.20

Problems

  • Software should still work when distributed among machines.
  • Not the reality. Following challenges:
    • Environment issues
    • Managability issues

Environment issues

  • Components have dependencies.
  • Dependencies need to be compatible.
  • Dependencies should be discoverable.
  • Components may depend on non-software artifacts (e.g. configurations).

Manageability issues

  • Uninstall / Upgrade: should not induce failures to another part of the system (e.g. DLL hell).
  • Administrator queries: file ownership? disk space consumption? source?
  • Rollbacks: able to undo effects of upgrades.
  • Variability: build / deployment configurations may differ.
  • … and they scale for a huge fleet of machines with different SKUs.

Industry solutions?

Idea #1. Global package management

  • Systematically manage packages (e.g. apt, yum, pacman, etc).
  • Each component provide a set of constraints:
    • A is installed \rightarrow B (>= 1.0) must be installed.
    • A is installed \rightarrow C should NOT be installed.
    • Success deployment \rightarrow pkg1, pkg2, … is installed.
  • Solve: success deployment.

B-SAT problem (NP-complete).

Implement pseudo-solvers.

When you try to manage the global system,

you lose isolation.

  • Two components want different versions of the same dependency?
  • Two components providing files on the same location?
  • Upgrading is destructive, and not atomic.
  • Files are usually overwritten, making rollbacks non-trivial.

Idea #2. Go monolithic!

  • Environment issues?
    • Resolving undeterministic dependency is hard.
    • Why not bundle everything?
  • Managability issues?
    • Isolation between bundles.
    • Only load dependencies which are bundled inside.
  • Self-contained packaging: AppImage, most Windows/macOS apps, etc.
  • Containers, and virtualization based technologies.

Wait, won’t build + deployment be complex due to monolithic?

Chunking.

  • Break big software into multiple parts.
    • Inside each part, we go monolithic.
    • Between parts, apply simpler dependency management.
  • Break complex build and deployment into multiple stages.
    • Inside each stage, we go imperative.
    • Between xstages, we declaratively define dependencies.
  • Break huge docker image into multiple layers.
    • Inside each layer, we go imperative.
    • Between layers, apply simpler dependency management.

But we sacrifice sharing between components.

  • How many Electrons do you have in Windows/macOS?
  • How many Linux base images do you have in your K8s cluster?

Idea #3

Can we marry isolation with fine-grained package management?

  • Environment issues:
    • Do we really need SAT model for dependency management?
    • Much simpler if dependency is just a deterministic tree.
  • Managability issues:
    • Store components isolately without bundling?
    • Let components see dependencies in separate views?

image/svg+xml Nix

Originating from The Purely Functional Software Deployment Model (2006),

Nix solves all above problems,

while achieving decalrative builds & deployments.

# Works for any Linux distribution and macOS
curl https://nixos.org/nix/install | sh

Nix: build system

Deterministic software building makes dependencies deterministic.

\begin{CD} \text{Inputs} @> f>> \text{Output} \end{CD}

\begin{CD} x = 1, y = 2 @> f(x, y) = x + y >> 1 + 2 = 3 \end{CD}

\begin{CD} \text{gcc, libc, source, ...} @> \text{./configure, make, make install} >> \text{binary} \end{CD}

Derivation

  • In Nix, we call the smallest unit of compliation as derivation.
  • A derivation is written in a Purely Functional Programming Language (Nix).
  • A derivation must have all inputs explictly specified.
  • A derivation is built inside a sandbox to avoid global state.
  • Therefore, the build output of a derivation is deterministic.

Nix store

  • Derivations are stored in Nix Store, and builds output to Nix Store.

  • A derivation’s build inputs are also managed in Nix Store.

  • Nix store paths made unique with cryptographic hash:

    /nix/store/zrwzkd3szh13zd3wrlzj0kdkgiv1xzjn-hello.drv
/nix/store/rq6w0k38h7kbh2s9snwpysk5yph2fqbf-hello

  • The output path’s hash is generated by derivation content.

    • Any slight change to build process is reflected in hash.

  • Nix store is read-only, only modifiable by nix.

Sandboxing

  • Builds only see specified inputs, and no other files.
    • Assume nothing in global paths like /lib, /usr/bin, etc.
  • Private version of /proc, /dev, /dev/shm and /dev/pts (Linux-only).
    • Therefore, private PID, mount, IPS, UTS namespace, etc.
    • No networking access during build.

An example “hello world” program:

/* ./src/main.c */
#include <stdio.h>

int main() {
    printf("Hello, World!");
    return 0;
}

… and how we write derivation for it:

{ pkgs, ... }:

pkgs.stdenv.mkDerivation {
  name = "hello";
  src = ./src;

  nativeBuildInputs = with pkgs; [ gcc ];
  buildPhase = ''
    gcc main.c -o hello
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp hello $out/bin
  '';
}
{
  "/nix/store/87zf1q5dx3dkn597lqq17f1g83y116l6-hello.drv": {
    "args": [ "-e", "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh" ],
    "builder": "/nix/store/0c337gsdfjf3162avbkchh0yh4qbs2s3-bash-5.2-p15/bin/bash",
    "env": {
      "buildPhase": "gcc main.c -o hello\n",
      "builder": "/nix/store/0c337gsdfjf3162avbkchh0yh4qbs2s3-bash-5.2-p15/bin/bash",
      "installPhase": "mkdir -p $out/bin\ncp hello $out/bin\n",
      "name": "hello",
      "nativeBuildInputs": "/nix/store/hc326d04c91h73ndqdx3qkggsk730kf2-gcc-wrapper-12.3.0",
      "out": "/nix/store/8ygc7ks9ggj7p2q0b98w1axc3mkyi68c-hello",
      "outputs": "out",
      "src": "/nix/store/92m3yxqi2hfmj75b053zvj0kkhv9bplq-src",
      "stdenv": "/nix/store/iszb73m627pq8v3gwf7zl6xaw01ln2hj-stdenv-linux",
      "system": "aarch64-linux"
    },
    // to be continued...
}}
{{ // continuing 
    "inputDrvs": {
      "/nix/store/f27pfz65b77lby39rrr48ps21pa6mbxj-gcc-wrapper-12.3.0.drv": {
        "outputs": [ "out" ]
      },
      "/nix/store/hq9032m10smw5qbig1b1cvvqirv61j54-stdenv-linux.drv": {
        "outputs": [ "out" ]
      },
      "/nix/store/nsn38mpj8j5h9861w5chg40f2vz4blq3-bash-5.2-p15.drv": {
        "outputs": [ "out" ]
      }
    },
    "inputSrcs": [
      "/nix/store/92m3yxqi2hfmj75b053zvj0kkhv9bplq-src",
      "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh"
    ],
    "name": "hello",
    "outputs": {
      "out": { "path": "/nix/store/8ygc7ks9ggj7p2q0b98w1axc3mkyi68c-hello" }
    },
    "system": "aarch64-linux"
  }
}

Another example of hello world with ncurses

/* ./src/main.c */
#include <ncurses.h>

int main() {
    initscr();
    printw("Hello, World!");
    refresh();
    getch();
    endwin();
    return 0;
}

… ncurses needs to be explictly specified as buildInputs:

{ pkgs, ... }:

pkgs.stdenv.mkDerivation {
  name = "hello";
  src = ./src;

  nativeBuildInputs = with pkgs; [ gcc ];
  buildInputs = with pkgs; [ ncurses ];
  buildPhase = ''
    gcc main.c -o hello -lncurses
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp hello $out/bin
  '';
}
{
  "/nix/store/2w3jmr0s30ylyvpri0m2kb91q4c6wvcb-hello.drv": {
    "args": [ "-e", "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh" ],
    "builder": "/nix/store/0c337gsdfjf3162avbkchh0yh4qbs2s3-bash-5.2-p15/bin/bash",
    "env": {
      "buildInputs": "/nix/store/k7wlgnj0d7fp3862gy0s5s6vphkm48k1-ncurses-6.4-dev",
      "buildPhase": "gcc main.c -o hello -lncurses\n",
      "builder": "/nix/store/0c337gsdfjf3162avbkchh0yh4qbs2s3-bash-5.2-p15/bin/bash",
      "installPhase": "mkdir -p $out/bin\ncp hello $out/bin\n",
      "name": "hello",
      "nativeBuildInputs": "/nix/store/hc326d04c91h73ndqdx3qkggsk730kf2-gcc-wrapper-12.3.0",
      "out": "/nix/store/rmkrazrqfy8zpk1h52qnjrj9qlcyh9mv-hello",
      "src": "/nix/store/yf2fijnfz19kqh8finky3n2rk11217r9-src",
      "stdenv": "/nix/store/iszb73m627pq8v3gwf7zl6xaw01ln2hj-stdenv-linux",
      "system": "aarch64-linux"
    },
}}
{{ // continuing 
    "inputDrvs": {
      "/nix/store/5l9mg0nlx3j0nf08hlaspnnx592acfm1-ncurses-6.4.drv": {
        "outputs": [ "dev" ]
      },
      "/nix/store/f27pfz65b77lby39rrr48ps21pa6mbxj-gcc-wrapper-12.3.0.drv": {
        "outputs": [ "out" ]
      },
      "/nix/store/hq9032m10smw5qbig1b1cvvqirv61j54-stdenv-linux.drv": {
        "outputs": [ "out" ]
      },
      "/nix/store/nsn38mpj8j5h9861w5chg40f2vz4blq3-bash-5.2-p15.drv": {
        "outputs": [ "out" ]
      }
    },
    "inputSrcs": [
      "/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh",
      "/nix/store/yf2fijnfz19kqh8finky3n2rk11217r9-src"
    ],
    "name": "hello",
    "outputs": {
      "out": { "path": "/nix/store/rmkrazrqfy8zpk1h52qnjrj9qlcyh9mv-hello" }
    },
    "system": "aarch64-linux"
  }
}

What if we call other binaries in source?

#include <stdlib.h>

int main() {
    system("cowsay 'Hello World!'");
    return 0;
}

Create a wrapper to set PATH before actually executing the binary

{ pkgs, ... }:

pkgs.stdenv.mkDerivation {
  name = "hello";
  src = ./src;

  nativeBuildInputs = with pkgs; [ gcc makeWrapper ];
  buildPhase = ''
    gcc main.c -o hello
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp hello $out/bin
  '';
  postFixup = ''
    wrapProgram $out/bin/hello \
      --prefix PATH : "${pkgs.lib.makeBinPath [ pkgs.cowsay ]}"
  '';
}

/bin/hello is a wrapper script instead of real binary

$ cat /nix/store/z7i77wwagy58f6svxc8ksm5snsc8wnrm-hello/bin/hello

#! /nix/store/0c337gsdfjf3162avbkchh0yh4qbs2s3-bash-5.2-p15/bin/bash -e
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/klqwsfd2xn14bb977d5dvjqdjpp6ka74-cowsay-3.7.0/bin'':'/':'}
PATH='/nix/store/klqwsfd2xn14bb977d5dvjqdjpp6ka74-cowsay-3.7.0/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
exec -a "$0" "/nix/store/z7i77wwagy58f6svxc8ksm5snsc8wnrm-hello/bin/.hello-wrapped"  "$@"

Binary cache

  • Note that the output hash can be calculated without building the derivation.
  • Meaning, we can cache the builds easily.
  • Serve nix store with a file server, which is the binary cache.
  • Packages can be signed before being added to binary cache or on the fly as they are served.

NixPkgs

There are many languages/frameworks, complicated.

  • Why not unite the efforts in building software?
  • Nixpkgs provides not only compiler toolchains, but also infrastructure for packaging applications written in various language and frameworks.
  • And of course, it comes with an official binary cache.

It seems to restrictive for majority packages to onboard?

Nix Search - Packages

Repology

Take rust-analyzer as a real-life example.

Nix: package manager

Build makes little sense if we cannot install it.

  • Search for references recursively for a package’s derivation,
  • we get a package’s closure,
  • that is, itself and all its direct and transitive runtime dependencies.

Installation

Two possibilities:

  • The closure is already in store or binary cache,
  • It has to be built fresh (then we infer the closure).

How to make the package accessible?

  • Activation script: an idempotent script making things accessible.

  • e.g. add all package outputs to $PATH.

  • Executed while initializing profile.

    nix-shell -p hello

Removal

Garbage collection.

  • Register it as gcroot when installing a derivation.

  • Deregister after uninstalling.

  • Periodically, enumerate all reachable store paths from the gcroots, and remove all unreachable paths.

    $ nix-collect-garbage

Advantages

  • Deterministic dependencies. No SAT solver.
  • Different versions / variants of the same package can be installed together.
  • Zero assumption about system global state.
  • Atomic installs / upgrades.

NixOS

A working system = Software + Configurations

Why separate packages and configurations?

NixOS leverages Nix to manage both altogether.

Filesystem Hierarchy Standard

/
├── boot
├── bin
├── etc
├── lib
├── usr
│  ├── bin
│  ├── include
│  ├── lib
│  └── share

However, in NixOS …

/
├── boot
├── nix
│  ├── store
│  │  ├── acr...kl-nixos-system-...
│  │  ├── 11w...cv-nixos-system-...
│  │  │  ├── activate
│  │  │  ├── kernel -> /nix/store/bcd...3h-linux-6.6.30/bzImage
│  │  │  ├── systemd -> /nix/store/9cx...8s-systemd-255.4
│  │  │  ├── ...
│  │  ├── bcd...3h-linux-6.6.30
│  │  ├── 9cx...8s-systemd-255.4
│  │  ├── ...
│  ├── var/nix/profiles
│  │  ├── system -> system-250-link
│  │  ├── system-249-link -> /nix/store/acr...kl-nixos-system-...
│  │  ├── system-250-link -> /nix/store/11w...cv-nixos-system-...
  • Activation script: /nix/var/nix/profiles/system/activate

Declaratively describe your system in Nix:

{ config, lib, pkgs, ...}: {
    # Use systemd-boot EFI bootloader
    boot.loader.systemd-boot.enable = true;
    # Kernel configurations
    boot.supportedFilesystems = [ "bcachefs" ];
    boot.initrd.availableKernelModules = [ "xhci_pci" "sr_mod" ];
    boot.kernelPackages = pkgs.linuxPackages_latest;
    # Use zsh for shell
    programs.zsh.enable = true; 
    programs.zsh.enableCompletion = true;
    # Enable the OpenSSH daemon.
    services.openssh.enable = true;
    # Configure users
    users.users.codgi = {
        name = "codgician";
        home = "/home/codgi";
        shell = pkgs.zsh;
        openssh.authorizedKeys.keys = [ "..." ];
    };
}

The power of NixOS roots in the Nix language.

  • Validate configurations before deployment.
  • Reference values accross modules of configurations.
  • Effeciently reuse configurations.
  • Unified language interface for any system component.

And even more

  • Trivial remote deployment: just push closure over ssh
  • Home manager: Manage dotfiles under /home declaratively using Nix.
  • Impermanence: Trivially handle persistent states on systems with ephemeral root storage

Ending

  • Although I am a Nix enthusiastic, I still realize:
    • Nix is hard to adopt in industry due to steep learning curve.
    • Even if Nix has a novel model, implementations consist many workarounds: sometimes you “hack” to make a derivation work on Nix.
    • Many existing infra may be “acceptable” enough.
      • Completely refactor everything with Nix is usually not necessary.

Nix/NixOS is still gaining increasing visibility and popularity.

Google trends: NixOS

Embracing Nix without fully switching

Thank you!

To get started, or learn further about Nix/NixOS:

Slides are

generated by pandoc,

rendered by reveal.js,

and managed by Nix.

Fully open-sourced.

References

Supplementaries

Slides not shown by default

Wait, for hello-ncurses, doesn’t gcc do dynmaic linking by default?

  • Nix has a patched version of dynamic linker in stdenv.
    • It never searches global library directories, like /lib, /usr/lib, etc.
    • The linker adds -rpath flag for every library directory mentioned through -L flags.
  • Won’t this possibly include unnecessary dependencies?
    • We don’t know in advance if a library is actually used by the linker.
    • Leverage fix up stage at the end to patchelf and shrink rpath.

Wait, how can Nix magically know runtime dependencies?

“Runtime dependencies must be a subset of build time dependencies”.

  • Build time dependencies are explictly specified.
  • Runtime dependencies are automatically inferred by:
    • Serializing store paths into NAR,
    • Then search for references to other store paths within it.

How can this be true and how can it work?

It just works! 🤪

Wait, what if I have secrets in my configurations?

Nix store is globally readable, any user has R/O access.

  • Store encrypted secrets in Nix Store.
  • Decrypt secrets with user private key / host private key before service load.
  • Example solutions: agenix | sops-nix.