Photo - Dylan McLeod
Photo

Configurable Nix packages

The vim and neovim packages in Nixpkgs allow users to set custom configuration, including their customRC and any plugins they might want.

How do they accomplish it?

In this article, we'll explore how to create packages with similar behavior. We'll create a simple Rust app that consumes a configuration file, then create a Nix flake containing both an unwrapped binary package as well as a configurable package.

Observing our goal

My neovimConfigured derivation demonstrates the desired experience for our users:

{ neovim, vimPlugins }:

neovim.override {
  vimAlias = true;
  viAlias = true;
  configure = {
    customRC = ''
      luafile ${../config/nvim/init.lua}
    '';
    packages.myVimPackage = with vimPlugins; {
      start = [
        LanguageClient-neovim
        # ...
      ];
    };
  };
}

We're able to override the neovim package with some custom attributes which are then loaded when the nvim from that package is invoked. In theory, virtually any Nix user could run the following and get nearly my exact configuration:

nix run github:hoverbear-consulting/flake#neovimConfigured

In order to accomplish this, we'll make use of nixpkgs.lib.makeOverridable.

Interacting with makeOverridable

The makeOverridable (source) function allows us to attach an override behavior to a given function.

Let's open our nix repl and try it:

nix-repl> makeOverridable = (builtins.getFlake "nixpkgs").lib.makeOverridable

nix-repl> pretendPkg = args@{ ... }: args

nix-repl> pretendPkg { foo = 1; }
{ foo = 1; }

nix-repl> overriddenPretendPkg = makeOverridable pretendPkg { default = true; }

nix-repl> overriddenPretendPkg
{ default = true; override = { ... }; overrideDerivation = «lambda @ /nix/store/31jbhzh2pj5zsr5ip983qbknv23kf7d4-source/lib/customisation.nix:84:32»; }

When we used makeOverridable with the pretendPkg function it creates a result (overriddenPretendPkg) as if the original function was called with the given attribute set ({ default = true; }), as well as two additional attributes: override and overrideDerivation.

Let's check out override:

nix-repl> overriddenPretendPkg.override
{ __functionArgs = { ... }; __functor = «lambda @ /nix/store/31jbhzh2pj5zsr5ip983qbknv23kf7d4-source/lib/trivial.nix:346:19»; }

nix-repl> overriddenPretendPkg.override { }
{ default = true; override = { ... }; overrideDerivation = «lambda @ /nix/store/31jbhzh2pj5zsr5ip983qbknv23kf7d4-source/lib/customisation.nix:84:32»; }

nix-repl> overriddenPretendPkg.override { default = false; }
{ default = false; override = { ... }; overrideDerivation = «lambda @ /nix/store/31jbhzh2pj5zsr5ip983qbknv23kf7d4-source/lib/customisation.nix:84:32»; }

So override is a function which accepts an attribute set which it merges with the attribute set the makeOverride function is invoked with.

Note: overrideDerivation is deprecated, so we won't discuss it.

We can use makeOverridable on any function that accepts an attribute set, namely, a Nix package created through something like mkDerivation or mkShell.

Let's make a quick Rust application, after, we can make a Nix flake, package, and overridable package for it!

A test application

First, let's quickly bootstrap a Rust package to play with. Make a demo directory and create the following two files:

# demo/Cargo.toml
[package]
name = "demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
structopt = { version = "0.3", default-features = false }
toml = { version = "0.5", default-features = false }
serde = { version = "1", default-features = false, features = [ "derive" ] }
color-eyre = { version = "0.5", default-features = false }
eyre = { version = "0.6", default-features = false }

// demo/src/main.rs
use eyre::WrapErr;
use serde::Deserialize;
use std::{fs::File, io::Read, path::PathBuf};
use structopt::StructOpt;

#[derive(StructOpt, Debug)]
#[structopt(name = "demo", about = "A binary with a configuration file.")]
struct CliArgs {
    /// A path to a configuration file
    #[structopt(
        short,
        long,
        parse(from_os_str),
        env = "DEMO_CONFIG",
        default_value = "/etc/demo.toml"
    )]
    pub config: PathBuf,
}

#[derive(Deserialize, Debug)]
struct Config {
    /// A togglable switch
    #[allow(dead_code)]
    #[serde(default)]
    switch: bool,
}

fn main() -> color_eyre::eyre::Result<()> {
    color_eyre::install()?;

    let cli_args = CliArgs::from_args();
    println!("{:#?}", cli_args);

    let mut config_file = File::open(&cli_args.config)
        .wrap_err_with(|| format!("Failed to open config {}", cli_args.config.display()))?;

    let mut config_string = String::default();
    config_file
        .read_to_string(&mut config_string)
        .wrap_err_with(|| format!("Failed to read config {}", cli_args.config.display()))?;

    let config: Config = toml::from_str(&config_string)
        .wrap_err_with(|| format!("Failed to parse config {}", cli_args.config.display()))?;
    println!("{:#?}", config);

    Ok(())
}

This will create a program which accepts a given config, loads it, and prints it out to stdout. That should be sufficient to tell if we've succeeded in our task later!

If you already have a global Rust environment set up, you can give this a test with cargo run or DEMO_CONFIG=/some/path cargo run. If you don't have one, you'll be able to do this after the next step.

Flexing Nix

Now our little crate needs a Nix package. We'll use github:oxalica/rust-overlay and github:nix-community/naersk today, so our demo crate is built like so:

# Chunk of a Flake.nix
overlay = final: prev: {
  demo-unwrapped = naersk.lib."${final.hostPlatform.system}".buildPackage rec {
    pname = "demo";
    version = "0.0.1";
    src = gitignore.lib.gitignoreSource ./.;
  };
  # ...

Note: We suffix it with -unwrapped so the configurable package can be demo. We call it 'unwrapped' since we will be 'wrapping' the package to make it configurable.

Then we'll define a wrapDemo function which takes some given non-overridable demo package and makes it overridable.

wrapDemo = demo-unwrapped: final.lib.makeOverridable ({ configuration ? null }:
  if configuration == null then
    demo-unwrapped
  else
    let
      configurationFile = final.writeTextFile {
        name = "demo-config";
        text = configuration;
      }; in
    final.symlinkJoin {
      name = "demo-${final.lib.getVersion final.demo-unwrapped}";
      paths = [ ];
      nativeBuildInputs = with final; [ makeWrapper ];
      postBuild = ''
        makeWrapper ${demo-unwrapped}/bin/demo \
          ${placeholder "out"}/bin/demo \
          --set-default DEMO_CONFIG ${configurationFile}
      '';
    });

The { configuration ? null } (on the first line) is the available options to override, here you could add, remove, or change what users can manipulate.

symlinkJoin (source) is a builder like mkDerivation, and produces what nix run (and the rest) expect.

The placeholder function, a Nix builtin, returns a placeholder for one of the outputs of a package.

In the postBuild step of the derivation we'll make use of makeWrapper (source), which allows us to do things like set arguments and populate environment variables.

With that, we can define the 'wrapped' package:

demo = final.wrapDemo final.demo-unwrapped { };

So whole flake looks like so:

# demo/flake.nix
{
  description = "An example of a configurable Nix package.";

  inputs = {
    nixpkgs.url = "nixpkgs";
    rust-overlay.url = "github:oxalica/rust-overlay";
    rust-overlay.inputs.nixpkgs.follows = "nixpkgs";
    naersk.url = "github:nix-community/naersk";
    naersk.inputs.nixpkgs.follows = "nixpkgs";
    gitignore.url = "github:hercules-ci/gitignore.nix";
    gitignore.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, rust-overlay, naersk, gitignore }:
    let
      supportedSystems = [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
      forAllSystems = f: nixpkgs.lib.genAttrs supportedSystems (system: f system);
      pkgsForSystem = system: (import nixpkgs {
        inherit system;
        overlays = [
          self.overlay
          rust-overlay.overlay
          (self: super: { inherit (self.rust-bin.stable.latest) rustc cargo rustdoc; })
        ];
      });
    in
    {
      overlay = final: prev: {
        demo-unwrapped = naersk.lib."${final.hostPlatform.system}".buildPackage rec {
          pname = "demo";
          version = "0.0.1";
          src = gitignore.lib.gitignoreSource ./.;
        };
        wrapDemo = demo-unwrapped: final.lib.makeOverridable ({ configuration ? null }:
          if configuration == null then
            demo-unwrapped
          else
            let
              configurationFile = final.writeTextFile {
                name = "demo-config";
                text = configuration;
              }; in
            final.symlinkJoin {
              name = "demo-${final.lib.getVersion final.demo-unwrapped}";
              paths = [ ];
              nativeBuildInputs = with final; [ makeWrapper ];
              postBuild = ''
                makeWrapper ${demo-unwrapped}/bin/demo \
                  ${placeholder "out"}/bin/demo \
                  --set-default DEMO_CONFIG ${configurationFile}
              '';
            });
        demo = final.wrapDemo final.demo-unwrapped { };
      };

      defaultPackage = forAllSystems (system:
        (pkgsForSystem system).demo
      );

      packages = forAllSystems (system:
        let
          pkgs = pkgsForSystem system;
        in
        {
          inherit (pkgs) demo demo-unwrapped;
          demoConfigured = pkgs.demo.override {
            configuration = ''
              switch = true
            '';
          };
        });

      devShell = forAllSystems (system:
        let
          pkgs = pkgsForSystem system;
        in
        pkgs.mkShell {
          inputsFrom = [ pkgs.demo ];
          buildInputs = with pkgs; [ nixpkgs-fmt cargo rust-bin.stable.latest.default ];
        });

      checks = forAllSystems (system:
        let
          pkgs = pkgsForSystem system;
        in
        {
          format = pkgs.runCommand "check-format"
            { } ''
            ${pkgs.nixpkgs-fmt}/bin/nixpkgs-fmt --check ${./.}
            touch $out # it worked!`
          '';
          inherit (pkgs) demo;
          demoConfigured = self.packages."${system}".demoConfigured;
        });
    };
}

If you use direnv you can add this too:

# demo/.envrc
# reload when these files change
watch_file flake.nix
watch_file flake.lock
# load the flake devShell
eval "$(nix print-dev-env)"

A scientist doing an experiment. - @nci
A scientist doing an experiment.

Experimenting

Let's validate that this works!

In our Flake.nix we also defined a demoConfigured package which uses override and sets a custom configuration:

packages = forAllSystems (system:
  let
    pkgs = pkgsForSystem system;
  in
  {
    inherit (pkgs) demo demo-unwrapped;
    demoConfigured = pkgs.demo.override {
      configuration = ''
        switch = true
      '';
    };
  });

We can run it as well as the default demo and validate the configuration file does get overriden:

$ nix run .#demo
CliArgs {
    config: "/etc/demo.toml",
}
Error: 
   0: Failed to open config /etc/demo.toml
   1: No such file or directory (os error 2)

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Now the configured one:

$ nix run .#demoConfigured
CliArgs {
    config: "/nix/store/9q4hmdspzh4riga0k3xsy1zsg2s33q09-demo-config",
}
Config {
    switch: true,
}

Fantastic! We can see switch got set in our demoConfigured package, and the config file it reads from is part of our Nix store.

As you can see, Nix lets us make configurable wrappers around our packages quite easily, allowing your users to take their configuration with them.

6c3fee57ca834cb9f133106444ebe3bf12a9628e