Nixpkgs Contributors Guide

Version 17.03pre-git


Table of Contents

1. Introduction
1.1. Overview of Nixpkgs
2. Quick Start to Adding a Package
3. The Standard Environment
3.1. Using stdenv
3.2. Tools provided by stdenv
3.3. Attributes
3.4. Phases
3.4.1. Controlling phases
3.4.2. The unpack phase
3.4.3. The patch phase
3.4.4. The configure phase
3.4.5. The build phase
3.4.6. The check phase
3.4.7. The install phase
3.4.8. The fixup phase
3.4.9. The installCheck phase
3.4.10. The distribution phase
3.5. Shell functions
3.6. Package setup hooks
3.7. Purity in Nixpkgs
3.8. Hardening in Nixpkgs
4. Multiple-output packages
4.1. Introduction
4.2. Installing a split package
4.3. Using a split package
4.4. Writing a split derivation
4.4.1. File type groups
4.4.2. Common caveats
5. Cross-compilation
5.1. Introduction
5.2. Packaging in a cross-friendly manner
5.2.1. Platform parameters
5.2.2. Specifying Dependencies
5.3. Cross-building packages
5.4. Cross-compilation infrastructure
6. Global configuration
6.1. Modify packages via packageOverrides
7. Functions reference
7.1. Overriding
7.1.1. <pkg>.override
7.1.2. <pkg>.overrideAttrs
7.1.3. <pkg>.overrideDerivation
7.1.4. lib.makeOverridable
7.2. Generators
7.3. buildFHSUserEnv
7.4. pkgs.dockerTools
7.4.1. buildImage
7.4.2. pullImage
7.4.3. exportImage
7.4.4. shadowSetup
8. Library functions
8.1. Top level functions
8.2. All library functions
9. Meta-attributes
9.1. Standard meta-attributes
9.2. Licenses
10. Support for specific programming languages and frameworks
10.1. Beam Languages (Erlang & Elixir)
10.1.1. Introduction
10.1.2. Build Tools
10.1.2.1. Rebar3
10.1.2.2. Mix & Erlang.mk
10.1.3. How to install Beam packages
10.1.4. Packaging Beam Applications
10.1.4.1. Erlang Applications
10.1.5. How to develop
10.1.5.1. Accessing an Environment
10.1.5.2. Creating a Shell
10.1.6. Generating Packages from Hex with Hex2Nix
10.2. Bower
10.2.1. bower2nix usage
10.2.2. buildBowerComponents function
10.2.3. Troubleshooting
10.3. Coq
10.4. Go
10.5. User’s Guide to the Haskell Infrastructure
10.5.1. How to install Haskell packages
10.5.2. How to create a development environment
10.5.2.1. How to install a compiler
10.5.2.2. How to install a compiler with libraries
10.5.2.3. How to install a compiler with libraries, hoogle and documentation indexes
10.5.2.4. How to build a Haskell project using Stack
10.5.2.5. How to create ad hoc environments for nix-shell
10.5.3. How to create Nix builds for your own private Haskell packages
10.5.3.1. How to build a stand-alone project
10.5.3.2. How to build projects that depend on each other
10.5.4. Miscellaneous Topics
10.5.4.1. How to build with profiling enabled
10.5.4.2. How to override package versions in a compiler-specific package set
10.5.4.3. How to recover from GHC’s infamous non-deterministic library ID bug
10.5.4.4. How to use the Haste Haskell-to-Javascript transpiler
10.5.4.5. Builds on Darwin fail with math.h not found
10.5.4.6. Builds using Stack complain about missing system libraries
10.5.4.7. Creating statically linked binaries
10.5.4.8. Building GHC with integer-simple
10.5.5. Other resources
10.6. Idris packages
10.6.1. callPackage
10.6.2. builtins
10.6.3. build-idris-package
10.6.4. build-builtin-package
10.6.5. with-packages
10.7. Java
10.8. Lua
10.9. Node.js packages
10.10. Perl
10.10.1. Generation from CPAN
10.11. Python
10.11.1. User Guide
10.11.1.1. Using Python
10.11.1.2. Developing with Python
10.11.1.3. Handling dependencies
10.11.1.4. Organising your packages
10.11.1.5. Including a derivation using callPackage
10.11.2. Reference
10.11.2.1. Interpreters
10.11.2.2. Building packages and applications
10.11.2.3. Development mode
10.11.2.4. Tools
10.11.3. FAQ
10.11.3.1. How can I install a working Python environment?
10.11.3.2. How to solve circular dependencies?
10.11.3.3. How to override a Python package?
10.11.3.4. python setup.py bdist_wheel cannot create .whl
10.11.3.5. install_data / data_files problems
10.11.3.6. Rationale of non-existent global site-packages
10.11.3.7. How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?
10.11.4. Contributing
10.11.4.1. Contributing guidelines
10.12. Qt
10.12.1. Libraries
10.12.2. Programs
10.12.3. KDE
10.13. R packages
10.13.1. Installation
10.13.2. RStudio
10.13.3. Updating the package set
10.13.4. Testing if the Nix-expression could be evaluated
10.14. Ruby
10.15. TeX Live
10.15.1. User's guide
10.15.2. Known problems
10.16. User’s Guide to Vim Plugins/Addons/Bundles/Scripts in Nixpkgs
10.16.1. dependencies by Vim plugins
10.16.2. HOWTO
10.16.3. Important repositories
11. Package Notes
11.1. Linux kernel
11.2. X.org
11.3. Eclipse
11.4. Elm
11.5. Autojump
11.6. Steam
11.6.1. Steam in Nix
11.6.2. How to play
11.6.3. Troubleshooting
11.6.4. steam-run
12. Overlays
12.1. Installing Overlays
12.2. Overlays Layout
13. Coding conventions
13.1. Syntax
13.2. Package naming
13.3. File naming and organisation
13.3.1. Hierarchy
13.3.2. Versioning
13.4. Fetching Sources
13.5. Patches
14. Submitting changes
14.1. Making patches
14.2. Submitting changes
14.3. Hotfixing pull requests
14.4. Commit policy
14.4.1. Master branch
14.4.2. Staging branch
14.4.3. Stable release branches
15. Reviewing contributions
15.1. Package updates
15.2. New packages
15.3. Module updates
15.4. New modules
15.5. Other submissions
15.6. Merging pull-requests
16. Contributing to this documentation

List of Examples

7.1. Docker build
7.2. Docker pull
7.3. Docker export
7.4. Shadow base files
10.1. bower.json
10.2. buildBowerComponents
10.3. Example build script (gulpfile.js)
10.4. Full example — default.nix
10.5. buildGoPackage
10.6. deps.nix
15.1. Sample template for a package update review
15.2. Sample template for a new package review
15.3. Sample template for a module update review
15.4. Sample template for a new module review

Chapter 1. Introduction

Table of Contents

1.1. Overview of Nixpkgs

The Nix Packages collection (Nixpkgs) is a set of thousands of packages for the Nix package manager, released under a permissive MIT/X11 license. Packages are available for several platforms, and can be used with the Nix package manager on most GNU/Linux distributions as well as NixOS.

This manual primarily describes how to write packages for the Nix Packages collection (Nixpkgs). Thus it’s mainly for packagers and developers who want to add packages to Nixpkgs. If you like to learn more about the Nix package manager and the Nix expression language, then you are kindly referred to the Nix manual.

1.1. Overview of Nixpkgs

Nix expressions describe how to build packages from source and are collected in the nixpkgs repository. Also included in the collection are Nix expressions for NixOS modules. With these expressions the Nix package manager can build binary packages.

Packages, including the Nix packages collection, are distributed through channels. The collection is distributed for users of Nix on non-NixOS distributions through the channel nixpkgs. Users of NixOS generally use one of the nixos-* channels, e.g. nixos-16.03, which includes all packages and modules for the stable NixOS 16.03. The purpose of stable NixOS releases are generally only given security updates. More up to date packages and modules are available via the nixos-unstable channel.

Both nixos-unstable and nixpkgs follow the master branch of the Nixpkgs repository, although both do lag the master branch by generally a couple of days. Updates to a channel are distributed as soon as all tests for that channel pass, e.g. this table shows the status of tests for the nixpkgs channel.

The tests are conducted by a cluster called Hydra, which also builds binary packages from the Nix expressions in Nixpkgs for x86_64-linux, i686-linux and x86_64-darwin. The binaries are made available via a binary cache.

The current Nix expressions of the channels are available in the nixpkgs-channels repository, which has branches corresponding to the available channels. There is also the Nixpkgs Monitor which keeps track of updates and security vulnerabilities.

Chapter 2. Quick Start to Adding a Package

To add a package to Nixpkgs:

  1. Checkout the Nixpkgs source tree:

    $ git clone git://github.com/NixOS/nixpkgs.git 
    $ cd nixpkgs

  2. Find a good place in the Nixpkgs tree to add the Nix expression for your package. For instance, a library package typically goes into pkgs/development/libraries/pkgname, while a web browser goes into pkgs/applications/networking/browsers/pkgname. See Section 13.3, “File naming and organisation” for some hints on the tree organisation. Create a directory for your package, e.g.

    $ mkdir pkgs/development/libraries/libfoo

  3. In the package directory, create a Nix expression — a piece of code that describes how to build the package. In this case, it should be a function that is called with the package dependencies as arguments, and returns a build of the package in the Nix store. The expression should usually be called default.nix.

    $ emacs pkgs/development/libraries/libfoo/default.nix
    $ git add pkgs/development/libraries/libfoo/default.nix

    You can have a look at the existing Nix expressions under pkgs/ to see how it’s done. Here are some good ones:

    Some notes:

    • All meta attributes are optional, but it’s still a good idea to provide at least the description, homepage and license.

    • You can use nix-prefetch-url (or similar nix-prefetch-git, etc) url to get the SHA-256 hash of source distributions. There are similar commands as nix-prefetch-git and nix-prefetch-hg available in nix-prefetch-scripts package.

    • A list of schemes for mirror:// URLs can be found in pkgs/build-support/fetchurl/mirrors.nix.

    The exact syntax and semantics of the Nix expression language, including the built-in function, are described in the Nix manual in the chapter on writing Nix expressions.

  4. Add a call to the function defined in the previous step to pkgs/top-level/all-packages.nix with some descriptive name for the variable, e.g. libfoo.

    $ emacs pkgs/top-level/all-packages.nix

    The attributes in that file are sorted by category (like “Development / Libraries”) that more-or-less correspond to the directory structure of Nixpkgs, and then by attribute name.

  5. To test whether the package builds, run the following command from the root of the nixpkgs source tree:

    $ nix-build -A libfoo

    where libfoo should be the variable name defined in the previous step. You may want to add the flag -K to keep the temporary build directory in case something fails. If the build succeeds, a symlink ./result to the package in the Nix store is created.

  6. If you want to install the package into your profile (optional), do

    $ nix-env -f . -iA libfoo

  7. Optionally commit the new package and open a pull request, or send a patch to nix-dev@cs.uu.nl.

Chapter 3. The Standard Environment

The standard build environment in the Nix Packages collection provides an environment for building Unix packages that does a lot of common build tasks automatically. In fact, for Unix packages that use the standard ./configure; make; make install build interface, you don’t need to write a build script at all; the standard environment does everything automatically. If stdenv doesn’t do what you need automatically, you can easily customise or override the various build phases.

3.1. Using stdenv

To build a package with the standard environment, you use the function stdenv.mkDerivation, instead of the primitive built-in function derivation, e.g.

stdenv.mkDerivation {
  name = "libfoo-1.2.3";
  src = fetchurl {
    url = http://example.org/libfoo-1.2.3.tar.bz2;
    sha256 = "0x2g1jqygyr5wiwg4ma1nd7w4ydpy82z9gkcv8vh2v8dn3y58v5m";
  };
}

(stdenv needs to be in scope, so if you write this in a separate Nix expression from pkgs/all-packages.nix, you need to pass it as a function argument.) Specifying a name and a src is the absolute minimum you need to do. Many packages have dependencies that are not provided in the standard environment. It’s usually sufficient to specify those dependencies in the buildInputs attribute:

stdenv.mkDerivation {
  name = "libfoo-1.2.3";
  ...
  buildInputs = [libbar perl ncurses];
}

This attribute ensures that the bin subdirectories of these packages appear in the PATH environment variable during the build, that their include subdirectories are searched by the C compiler, and so on. (See Section 3.6, “Package setup hooks” for details.)

Often it is necessary to override or modify some aspect of the build. To make this easier, the standard environment breaks the package build into a number of phases, all of which can be overridden or modified individually: unpacking the sources, applying patches, configuring, building, and installing. (There are some others; see Section 3.4, “Phases”.) For instance, a package that doesn’t supply a makefile but instead has to be compiled “manually” could be handled like this:

stdenv.mkDerivation {
  name = "fnord-4.5";
  ...
  buildPhase = ''
    gcc foo.c -o foo
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp foo $out/bin
  '';
}

(Note the use of ''-style string literals, which are very convenient for large multi-line script fragments because they don’t need escaping of " and \, and because indentation is intelligently removed.)

There are many other attributes to customise the build. These are listed in Section 3.3, “Attributes”.

While the standard environment provides a generic builder, you can still supply your own build script:

stdenv.mkDerivation {
  name = "libfoo-1.2.3";
  ...
  builder = ./builder.sh;
}

where the builder can do anything it wants, but typically starts with

source $stdenv/setup

to let stdenv set up the environment (e.g., process the buildInputs). If you want, you can still use stdenv’s generic builder:

source $stdenv/setup

buildPhase() {
  echo "... this is my custom build phase ..."
  gcc foo.c -o foo
}

installPhase() {
  mkdir -p $out/bin
  cp foo $out/bin
}

genericBuild

3.2. Tools provided by stdenv

The standard environment provides the following packages:

  • The GNU C Compiler, configured with C and C++ support.

  • GNU coreutils (contains a few dozen standard Unix commands).

  • GNU findutils (contains find).

  • GNU diffutils (contains diff, cmp).

  • GNU sed.

  • GNU grep.

  • GNU awk.

  • GNU tar.

  • gzip, bzip2 and xz.

  • GNU Make. It has been patched to provide nested output that can be fed into the nix-log2xml command and log2html stylesheet to create a structured, readable output of the build steps performed by Make.

  • Bash. This is the shell used for all builders in the Nix Packages collection. Not using /bin/sh removes a large source of portability problems.

  • The patch command.

On Linux, stdenv also includes the patchelf utility.

3.3. Attributes

Variables affecting stdenv initialisation

NIX_DEBUG

If set, stdenv will print some debug information during the build. In particular, the gcc and ld wrapper scripts will print out the complete command line passed to the wrapped tools.

Variables specifying dependencies

nativeBuildInputs

A list of dependencies used by the new derivation at build-time. I.e. these dependencies should not make it into the package's runtime-closure, though this is currently not checked. For each dependency dir, the directory dir/bin, if it exists, is added to the PATH environment variable. Other environment variables are also set up via a pluggable mechanism. For instance, if buildInputs contains Perl, then the lib/site_perl subdirectory of each input is added to the PERL5LIB environment variable. See Section 3.6, “Package setup hooks” for details.

buildInputs

A list of dependencies used by the new derivation at run-time. Currently, the build-time environment is modified in the exact same way as with nativeBuildInputs. This is problematic in that when cross-compiling, foreign executables can clobber native ones on the PATH. Even more confusing is static-linking. A statically-linked library should be listed here because ultimately that generated machine code will be used at run-time, even though a derivation containing the object files or static archives will only be used at build-time. A less confusing solution to this would be nice.

propagatedNativeBuildInputs

Like nativeBuildInputs, but these dependencies are propagated: that is, the dependencies listed here are added to the nativeBuildInputs of any package that uses this package as a dependency. So if package Y has propagatedBuildInputs = [X], and package Z has buildInputs = [Y], then package X will appear in Z’s build environment automatically.

propagatedBuildInputs

Like buildInputs, but propagated just like propagatedNativeBuildInputs. This inherits buildInputs's flaws of clobbering native executables when cross-compiling and being confusing for static linking.

Variables affecting build properties

enableParallelBuilding

If set, stdenv will pass specific flags to make and other build tools to enable parallel building with up to build-cores workers.

preferLocalBuild

If set, specifies that the package is so lightweight in terms of build operations (e.g. write a text file from a Nix string to the store) that there's no need to look for it in binary caches -- it's faster to just build it locally. It also tells Hydra and other facilities that this package doesn't need to be exported in binary caches (noone would use it, after all).

Special variables

passthru

This is an attribute set which can be filled with arbitrary values. For example:

passthru = {
  foo = "bar";
  baz = {
    value1 = 4;
    value2 = 5;
  };
}

Values inside it are not passed to the builder, so you can change them without triggering a rebuild. However, they can be accessed outside of a derivation directly, as if they were set inside a derivation itself, e.g. hello.baz.value1. We don't specify any usage or schema of passthru - it is meant for values that would be useful outside the derivation in other parts of a Nix expression (e.g. in other derivations). An example would be to convey some specific dependency of your derivation which contains a program with plugins support. Later, others who make derivations with plugins can use passed-through dependency to ensure that their plugin would be binary-compatible with built program.

3.4. Phases

The generic builder has a number of phases. Package builds are split into phases to make it easier to override specific parts of the build (e.g., unpacking the sources or installing the binaries). Furthermore, it allows a nicer presentation of build logs in the Nix build farm.

Each phase can be overridden in its entirety either by setting the environment variable namePhase to a string containing some shell commands to be executed, or by redefining the shell function namePhase. The former is convenient to override a phase from the derivation, while the latter is convenient from a build script.

3.4.1. Controlling phases

There are a number of variables that control what phases are executed and in what order:

Variables affecting phase control

phases

Specifies the phases. You can change the order in which phases are executed, or add new phases, by setting this variable. If it’s not set, the default value is used, which is $prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase $preDistPhases distPhase $postPhases.

Usually, if you just want to add a few phases, it’s more convenient to set one of the variables below (such as preInstallPhases), as you then don’t specify all the normal phases.

prePhases

Additional phases executed before any of the default phases.

preConfigurePhases

Additional phases executed just before the configure phase.

preBuildPhases

Additional phases executed just before the build phase.

preInstallPhases

Additional phases executed just before the install phase.

preFixupPhases

Additional phases executed just before the fixup phase.

preDistPhases

Additional phases executed just before the distribution phase.

postPhases

Additional phases executed after any of the default phases.

3.4.2. The unpack phase

The unpack phase is responsible for unpacking the source code of the package. The default implementation of unpackPhase unpacks the source files listed in the src environment variable to the current directory. It supports the following files by default:

Tar files

These can optionally be compressed using gzip (.tar.gz, .tgz or .tar.Z), bzip2 (.tar.bz2 or .tbz2) or xz (.tar.xz or .tar.lzma).

Zip files

Zip files are unpacked using unzip. However, unzip is not in the standard environment, so you should add it to buildInputs yourself.

Directories in the Nix store

These are simply copied to the current directory. The hash part of the file name is stripped, e.g. /nix/store/1wydxgby13cz...-my-sources would be copied to my-sources.

Additional file types can be supported by setting the unpackCmd variable (see below).

Variables controlling the unpack phase

srcs / src

The list of source files or directories to be unpacked or copied. One of these must be set.

sourceRoot

After running unpackPhase, the generic builder changes the current directory to the directory created by unpacking the sources. If there are multiple source directories, you should set sourceRoot to the name of the intended directory.

setSourceRoot

Alternatively to setting sourceRoot, you can set setSourceRoot to a shell command to be evaluated by the unpack phase after the sources have been unpacked. This command must set sourceRoot.

preUnpack

Hook executed at the start of the unpack phase.

postUnpack

Hook executed at the end of the unpack phase.

dontMakeSourcesWritable

If set to 1, the unpacked sources are not made writable. By default, they are made writable to prevent problems with read-only sources. For example, copied store directories would be read-only without this.

unpackCmd

The unpack phase evaluates the string $unpackCmd for any unrecognised file. The path to the current source file is contained in the curSrc variable.

3.4.3. The patch phase

The patch phase applies the list of patches defined in the patches variable.

Variables controlling the patch phase

patches

The list of patches. They must be in the format accepted by the patch command, and may optionally be compressed using gzip (.gz), bzip2 (.bz2) or xz (.xz).

patchFlags

Flags to be passed to patch. If not set, the argument -p1 is used, which causes the leading directory component to be stripped from the file names in each patch.

prePatch

Hook executed at the start of the patch phase.

postPatch

Hook executed at the end of the patch phase.

3.4.4. The configure phase

The configure phase prepares the source tree for building. The default configurePhase runs ./configure (typically an Autoconf-generated script) if it exists.

Variables controlling the configure phase

configureScript

The name of the configure script. It defaults to ./configure if it exists; otherwise, the configure phase is skipped. This can actually be a command (like perl ./Configure.pl).

configureFlags

A list of strings passed as additional arguments to the configure script.

configureFlagsArray

A shell array containing additional arguments passed to the configure script. You must use this instead of configureFlags if the arguments contain spaces.

dontAddPrefix

By default, the flag --prefix=$prefix is added to the configure flags. If this is undesirable, set this variable to true.

prefix

The prefix under which the package must be installed, passed via the --prefix option to the configure script. It defaults to $out.

dontAddDisableDepTrack

By default, the flag --disable-dependency-tracking is added to the configure flags to speed up Automake-based builds. If this is undesirable, set this variable to true.

dontFixLibtool

By default, the configure phase applies some special hackery to all files called ltmain.sh before running the configure script in order to improve the purity of Libtool-based packages[1]. If this is undesirable, set this variable to true.

dontDisableStatic

By default, when the configure script has --enable-static, the option --disable-static is added to the configure flags.

If this is undesirable, set this variable to true.

preConfigure

Hook executed at the start of the configure phase.

postConfigure

Hook executed at the end of the configure phase.

3.4.5. The build phase

The build phase is responsible for actually building the package (e.g. compiling it). The default buildPhase simply calls make if a file named Makefile, makefile or GNUmakefile exists in the current directory (or the makefile is explicitly set); otherwise it does nothing.

Variables controlling the build phase

dontBuild

Set to true to skip the build phase.

makefile

The file name of the Makefile.

makeFlags

A list of strings passed as additional flags to make. These flags are also used by the default install and check phase. For setting make flags specific to the build phase, use buildFlags (see below).

makeFlagsArray

A shell array containing additional arguments passed to make. You must use this instead of makeFlags if the arguments contain spaces, e.g.

makeFlagsArray=(CFLAGS="-O0 -g" LDFLAGS="-lfoo -lbar")

Note that shell arrays cannot be passed through environment variables, so you cannot set makeFlagsArray in a derivation attribute (because those are passed through environment variables): you have to define them in shell code.

buildFlags / buildFlagsArray

A list of strings passed as additional flags to make. Like makeFlags and makeFlagsArray, but only used by the build phase.

preBuild

Hook executed at the start of the build phase.

postBuild

Hook executed at the end of the build phase.

You can set flags for make through the makeFlags variable.

Before and after running make, the hooks preBuild and postBuild are called, respectively.

3.4.6. The check phase

The check phase checks whether the package was built correctly by running its test suite. The default checkPhase calls make check, but only if the doCheck variable is enabled.

Variables controlling the check phase

doCheck

If set to a non-empty string, the check phase is executed, otherwise it is skipped (default). Thus you should set

doCheck = true;

in the derivation to enable checks.

makeFlags / makeFlagsArray / makefile

See the build phase for details.

checkTarget

The make target that runs the tests. Defaults to check.

checkFlags / checkFlagsArray

A list of strings passed as additional flags to make. Like makeFlags and makeFlagsArray, but only used by the check phase.

preCheck

Hook executed at the start of the check phase.

postCheck

Hook executed at the end of the check phase.

3.4.7. The install phase

The install phase is responsible for installing the package in the Nix store under out. The default installPhase creates the directory $out and calls make install.

Variables controlling the install phase

makeFlags / makeFlagsArray / makefile

See the build phase for details.

installTargets

The make targets that perform the installation. Defaults to install. Example:

installTargets = "install-bin install-doc";

installFlags / installFlagsArray

A list of strings passed as additional flags to make. Like makeFlags and makeFlagsArray, but only used by the install phase.

preInstall

Hook executed at the start of the install phase.

postInstall

Hook executed at the end of the install phase.

3.4.8. The fixup phase

The fixup phase performs some (Nix-specific) post-processing actions on the files installed under $out by the install phase. The default fixupPhase does the following:

  • It moves the man/, doc/ and info/ subdirectories of $out to share/.

  • It strips libraries and executables of debug information.

  • On Linux, it applies the patchelf command to ELF executables and libraries to remove unused directories from the RPATH in order to prevent unnecessary runtime dependencies.

  • It rewrites the interpreter paths of shell scripts to paths found in PATH. E.g., /usr/bin/perl will be rewritten to /nix/store/some-perl/bin/perl found in PATH.

Variables controlling the fixup phase

dontStrip

If set, libraries and executables are not stripped. By default, they are.

dontMoveSbin

If set, files in $out/sbin are not moved to $out/bin. By default, they are.

stripAllList

List of directories to search for libraries and executables from which all symbols should be stripped. By default, it’s empty. Stripping all symbols is risky, since it may remove not just debug symbols but also ELF information necessary for normal execution.

stripAllFlags

Flags passed to the strip command applied to the files in the directories listed in stripAllList. Defaults to -s (i.e. --strip-all).

stripDebugList

List of directories to search for libraries and executables from which only debugging-related symbols should be stripped. It defaults to lib bin sbin.

stripDebugFlags

Flags passed to the strip command applied to the files in the directories listed in stripDebugList. Defaults to -S (i.e. --strip-debug).

dontPatchELF

If set, the patchelf command is not used to remove unnecessary RPATH entries. Only applies to Linux.

dontPatchShebangs

If set, scripts starting with #! do not have their interpreter paths rewritten to paths in the Nix store.

forceShare

The list of directories that must be moved from $out to $out/share. Defaults to man doc info.

setupHook

A package can export a setup hook by setting this variable. The setup hook, if defined, is copied to $out/nix-support/setup-hook. Environment variables are then substituted in it using substituteAll.

preFixup

Hook executed at the start of the fixup phase.

postFixup

Hook executed at the end of the fixup phase.

separateDebugInfo

If set to true, the standard environment will enable debug information in C/C++ builds. After installation, the debug information will be separated from the executables and stored in the output named debug. (This output is enabled automatically; you don’t need to set the outputs attribute explicitly.) To be precise, the debug information is stored in debug/lib/debug/.build-id/XX/YYYY…, where XXYYYY… is the build ID of the binary — a SHA-1 hash of the contents of the binary. Debuggers like GDB use the build ID to look up the separated debug information.

For example, with GDB, you can add

set debug-file-directory ~/.nix-profile/lib/debug

to ~/.gdbinit. GDB will then be able to find debug information installed via nix-env -i.

3.4.9. The installCheck phase

The installCheck phase checks whether the package was installed correctly by running its test suite against the installed directories. The default installCheck calls make installcheck.

Variables controlling the installCheck phase

doInstallCheck

If set to a non-empty string, the installCheck phase is executed, otherwise it is skipped (default). Thus you should set

doInstallCheck = true;

in the derivation to enable install checks.

preInstallCheck

Hook executed at the start of the installCheck phase.

postInstallCheck

Hook executed at the end of the installCheck phase.

3.4.10. The distribution phase

The distribution phase is intended to produce a source distribution of the package. The default distPhase first calls make dist, then it copies the resulting source tarballs to $out/tarballs/. This phase is only executed if the attribute doDist is set.

Variables controlling the distribution phase

distTarget

The make target that produces the distribution. Defaults to dist.

distFlags / distFlagsArray

Additional flags passed to make.

tarballs

The names of the source distribution files to be copied to $out/tarballs/. It can contain shell wildcards. The default is *.tar.gz.

dontCopyDist

If set, no files are copied to $out/tarballs/.

preDist

Hook executed at the start of the distribution phase.

postDist

Hook executed at the end of the distribution phase.

3.5. Shell functions

The standard environment provides a number of useful functions.

makeWrapper executable wrapperfile args

Constructs a wrapper for a program with various possible arguments. For example:

# adds `FOOBAR=baz` to `$out/bin/foo`’s environment
makeWrapper $out/bin/foo $wrapperfile --set FOOBAR baz

# prefixes the binary paths of `hello` and `git`
# Be advised that paths often should be patched in directly
# (via string replacements or in `configurePhase`).
makeWrapper $out/bin/foo $wrapperfile --prefix PATH : ${lib.makeBinPath [ hello git ]}

There’s many more kinds of arguments, they are documented in nixpkgs/pkgs/build-support/setup-hooks/make-wrapper.sh.

wrapProgram is a convenience function you probably want to use most of the time.

substitute infile outfile subs

Performs string substitution on the contents of infile, writing the result to outfile. The substitutions in subs are of the following form:

--replace s1 s2

Replace every occurence of the string s1 by s2.

--subst-var varName

Replace every occurence of @varName@ by the contents of the environment variable varName. This is useful for generating files from templates, using @...@ in the template as placeholders.

--subst-var-by varName s

Replace every occurence of @varName@ by the string s.

Example:

substitute ./foo.in ./foo.out \
    --replace /usr/bin/bar $bar/bin/bar \
    --replace "a string containing spaces" "some other text" \
    --subst-var someVar

substitute is implemented using the replace command. Unlike with the sed command, you don’t have to worry about escaping special characters. It supports performing substitutions on binary files (such as executables), though there you’ll probably want to make sure that the replacement string is as long as the replaced string.

substituteInPlace file subs

Like substitute, but performs the substitutions in place on the file file.

substituteAll infile outfile

Replaces every occurence of @varName@, where varName is any environment variable, in infile, writing the result to outfile. For instance, if infile has the contents

#! @bash@/bin/sh
PATH=@coreutils@/bin
echo @foo@

and the environment contains bash=/nix/store/bmwp0q28cf21...-bash-3.2-p39 and coreutils=/nix/store/68afga4khv0w...-coreutils-6.12, but does not contain the variable foo, then the output will be

#! /nix/store/bmwp0q28cf21...-bash-3.2-p39/bin/sh
PATH=/nix/store/68afga4khv0w...-coreutils-6.12/bin
echo @foo@

That is, no substitution is performed for undefined variables.

Environment variables that start with an uppercase letter or an underscore are filtered out, to prevent global variables (like HOME) or private variables (like __ETC_PROFILE_DONE) from accidentally getting substituted. The variables also have to be valid bash “names”, as defined in the bash manpage (alphanumeric or _, must not start with a number).

substituteAllInPlace file

Like substituteAll, but performs the substitutions in place on the file file.

stripHash path

Strips the directory and hash part of a store path, outputting the name part to stdout. For example:

# prints coreutils-8.24
stripHash "/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24"

If you wish to store the result in another variable, then the following idiom may be useful:

name="/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24"
someVar=$(stripHash $name)

wrapProgram executable makeWrapperArgs

Convenience function for makeWrapper that automatically creates a sane wrapper file It takes all the same arguments as makeWrapper, except for --argv0.

It cannot be applied multiple times, since it will overwrite the wrapper file.

3.6. Package setup hooks

The following packages provide a setup hook:

GCC wrapper

Adds the include subdirectory of each build input to the NIX_CFLAGS_COMPILE environment variable, and the lib and lib64 subdirectories to NIX_LDFLAGS.

Perl

Adds the lib/site_perl subdirectory of each build input to the PERL5LIB environment variable.

Python

Adds the lib/${python.libPrefix}/site-packages subdirectory of each build input to the PYTHONPATH environment variable.

pkg-config

Adds the lib/pkgconfig and share/pkgconfig subdirectories of each build input to the PKG_CONFIG_PATH environment variable.

Automake

Adds the share/aclocal subdirectory of each build input to the ACLOCAL_PATH environment variable.

Autoconf

The autoreconfHook derivation adds autoreconfPhase, which runs autoreconf, libtoolize and automake, essentially preparing the configure script in autotools-based builds.

libxml2

Adds every file named catalog.xml found under the xml/dtd and xml/xsl subdirectories of each build input to the XML_CATALOG_FILES environment variable.

teTeX / TeX Live

Adds the share/texmf-nix subdirectory of each build input to the TEXINPUTS environment variable.

Qt 4

Sets the QTDIR environment variable to Qt’s path.

gdk-pixbuf

Exports GDK_PIXBUF_MODULE_FILE environment variable the the builder. Add librsvg package to buildInputs to get svg support.

GHC

Creates a temporary package database and registers every Haskell build input in it (TODO: how?).

GStreamer

Adds the GStreamer plugins subdirectory of each build input to the GST_PLUGIN_SYSTEM_PATH_1_0 or GST_PLUGIN_SYSTEM_PATH environment variable.

paxctl

Defines the paxmark helper for setting per-executable PaX flags on Linux (where it is available by default; on all other platforms, paxmark is a no-op). For example, to disable secure memory protections on the executable foo:

      postFixup = ''
        paxmark m $out/bin/foo
      '';
    

The m flag is the most common flag and is typically required for applications that employ JIT compilation or otherwise need to execute code generated at run-time. Disabling PaX protections should be considered a last resort: if possible, problematic features should be disabled or patched to work with PaX.

3.7. Purity in Nixpkgs

[measures taken to prevent dependencies on packages outside the store, and what you can do to prevent them]

GCC doesn't search in locations such as /usr/include. In fact, attempts to add such directories through the -I flag are filtered out. Likewise, the linker (from GNU binutils) doesn't search in standard locations such as /usr/lib. Programs built on Linux are linked against a GNU C Library that likewise doesn't search in the default system locations.

3.8. Hardening in Nixpkgs

There are flags available to harden packages at compile or link-time. These can be toggled using the stdenv.mkDerivation parameters hardeningDisable and hardeningEnable.

Both parameters take a list of flags as strings. The special "all" flag can be passed to hardeningDisable to turn off all hardening. These flags can also be used as environment variables for testing or development purposes.

The following flags are enabled by default and might require disabling with hardeningDisable if the program to package is incompatible.

format

Adds the -Wformat -Wformat-security -Werror=format-security compiler options. At present, this warns about calls to printf and scanf functions where the format string is not a string literal and there are no format arguments, as in printf(foo);. This may be a security hole if the format string came from untrusted input and contains %n.

This needs to be turned off or fixed for errors similar to:

/tmp/nix-build-zynaddsubfx-2.5.2.drv-0/zynaddsubfx-2.5.2/src/UI/guimain.cpp:571:28: error: format not a string literal and no format arguments [-Werror=format-security]
         printf(help_message);
                            ^
cc1plus: some warnings being treated as errors
    
stackprotector

Adds the -fstack-protector-strong --param ssp-buffer-size=4 compiler options. This adds safety checks against stack overwrites rendering many potential code injection attacks into aborting situations. In the best case this turns code injection vulnerabilities into denial of service or into non-issues (depending on the application).

This needs to be turned off or fixed for errors similar to:

bin/blib.a(bios_console.o): In function `bios_handle_cup':
/tmp/nix-build-ipxe-20141124-5cbdc41.drv-0/ipxe-5cbdc41/src/arch/i386/firmware/pcbios/bios_console.c:86: undefined reference to `__stack_chk_fail'
    
fortify

Adds the -O2 -D_FORTIFY_SOURCE=2 compiler options. During code generation the compiler knows a great deal of information about buffer sizes (where possible), and attempts to replace insecure unlimited length buffer function calls with length-limited ones. This is especially useful for old, crufty code. Additionally, format strings in writable memory that contain '%n' are blocked. If an application depends on such a format string, it will need to be worked around.

Addtionally, some warnings are enabled which might trigger build failures if compiler warnings are treated as errors in the package build. In this case, set NIX_CFLAGS_COMPILE to -Wno-error=warning-type.

This needs to be turned off or fixed for errors similar to:

malloc.c:404:15: error: return type is an incomplete type
malloc.c:410:19: error: storage size of 'ms' isn't known
    
strdup.h:22:1: error: expected identifier or '(' before '__extension__'
    
strsep.c:65:23: error: register name not specified for 'delim'
    
installwatch.c:3751:5: error: conflicting types for '__open_2'
    
fcntl2.h:50:4: error: call to '__open_missing_mode' declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
    
pic

Adds the -fPIC compiler options. This options adds support for position independant code in shared libraries and thus making ASLR possible.

Most notably, the Linux kernel, kernel modules and other code not running in an operating system environment like boot loaders won't build with PIC enabled. The compiler will is most cases complain that PIC is not supported for a specific build.

This needs to be turned off or fixed for assembler errors similar to:

ccbLfRgg.s: Assembler messages:
ccbLfRgg.s:33: Error: missing or invalid displacement expression `private_key_len@GOTOFF'
    
strictoverflow

Signed integer overflow is undefined behaviour according to the C standard. If it happens, it is an error in the program as it should check for overflow before it can happen, not afterwards. GCC provides built-in functions to perform arithmetic with overflow checking, which are correct and faster than any custom implementation. As a workaround, the option -fno-strict-overflow makes gcc behave as if signed integer overflows were defined.

This flag should not trigger any build or runtime errors.

relro

Adds the -z relro linker option. During program load, several ELF memory sections need to be written to by the linker, but can be turned read-only before turning over control to the program. This prevents some GOT (and .dtors) overwrite attacks, but at least the part of the GOT used by the dynamic linker (.got.plt) is still vulnerable.

This flag can break dynamic shared object loading. For instance, the module systems of Xorg and OpenCV are incompatible with this flag. In almost all cases the bindnow flag must also be disabled and incompatible programs typically fail with similar errors at runtime.

bindnow

Adds the -z bindnow linker option. During program load, all dynamic symbols are resolved, allowing for the complete GOT to be marked read-only (due to relro). This prevents GOT overwrite attacks. For very large applications, this can incur some performance loss during initial load while symbols are resolved, but this shouldn't be an issue for daemons.

This flag can break dynamic shared object loading. For instance, the module systems of Xorg and PHP are incompatible with this flag. Programs incompatible with this flag often fail at runtime due to missing symbols, like:

intel_drv.so: undefined symbol: vgaHWFreeHWRec
    

The following flags are disabled by default and should be enabled with hardeningEnable for packages that take untrusted input like network services.

pie

Adds the -fPIE compiler and -pie linker options. Position Independent Executables are needed to take advantage of Address Space Layout Randomization, supported by modern kernel versions. While ASLR can already be enforced for data areas in the stack and heap (brk and mmap), the code areas must be compiled as position-independent. Shared libraries already do this with the pic flag, so they gain ASLR automatically, but binary .text regions need to be build with pie to gain ASLR. When this happens, ROP attacks are much harder since there are no static locations to bounce off of during a memory corruption attack.

For more in-depth information on these hardening flags and hardening in general, refer to the Debian Wiki, Ubuntu Wiki, Gentoo Wiki, and the Arch Wiki.



[1] It clears the sys_lib_*search_path variables in the Libtool script to prevent Libtool from using libraries in /usr/lib and such.

Chapter 4. Multiple-output packages

4.1. Introduction

The Nix language allows a derivation to produce multiple outputs, which is similar to what is utilized by other Linux distribution packaging systems. The outputs reside in separate nix store paths, so they can be mostly handled independently of each other, including passing to build inputs, garbage collection or binary substitution. The exception is that building from source always produces all the outputs.

The main motivation is to save disk space by reducing runtime closure sizes; consequently also sizes of substituted binaries get reduced. Splitting can be used to have more granular runtime dependencies, for example the typical reduction is to split away development-only files, as those are typically not needed during runtime. As a result, closure sizes of many packages can get reduced to a half or even much less.

Note

The reduction effects could be instead achieved by building the parts in completely separate derivations. That would often additionally reduce build-time closures, but it tends to be much harder to write such derivations, as build systems typically assume all parts are being built at once. This compromise approach of single source package producing multiple binary packages is also utilized often by rpm and deb.

4.2. Installing a split package

When installing a package via systemPackages or nix-env you have several options:

Warning

Currently nix-env almost always installs all outputs until https://github.com/NixOS/nix/pull/815 gets merged.

  • You can install particular outputs explicitly, as each is available in the Nix language as an attribute of the package. The outputs attribute contains a list of output names.

  • You can let it use the default outputs. These are handled by meta.outputsToInstall attribute that contains a list of output names.

    TODO: more about tweaking the attribute, etc.

  • NixOS provides configuration option environment.extraOutputsToInstall that allows adding extra outputs of environment.systemPackages atop the default ones. It's mainly meant for documentation and debug symbols, and it's also modified by specific options.

    Note

    At this moment there is no similar configurability for packages installed by nix-env. You can still use approach from Section 6.1, “Modify packages via packageOverrides to override meta.outputsToInstall attributes, but that's a rather inconvenient way.

4.3. Using a split package

In the Nix language the individual outputs can be reached explicitly as attributes, e.g. coreutils.info, but the typical case is just using packages as build inputs.

When a multiple-output derivation gets into a build input of another derivation, the dev output is added if it exists, otherwise the first output is added. In addition to that, propagatedBuildOutputs of that package which by default contain $outputBin and $outputLib are also added. (See Section 4.4.1, “File type groups”.)

4.4. Writing a split derivation

Here you find how to write a derivation that produces multiple outputs.

In nixpkgs there is a framework supporting multiple-output derivations. It tries to cover most cases by default behavior. You can find the source separated in <nixpkgs/pkgs/build-support/setup-hooks/multiple-outputs.sh>; it's relatively well-readable. The whole machinery is triggered by defining the outputs attribute to contain the list of desired output names (strings).

outputs = [ "bin" "dev" "out" "doc" ];

Often such a single line is enough. For each output an equally named environment variable is passed to the builder and contains the path in nix store for that output. By convention, the first output should contain the executable programs provided by the package as that output is used by Nix in string conversions, allowing references to binaries like ${pkgs.perl}/bin/perl to always work. Typically you also want to have the main out output, as it catches any files that didn't get elsewhere.

Note

There is a special handling of the debug output, described at separateDebugInfo.

4.4.1. File type groups

The support code currently recognizes some particular kinds of outputs and either instructs the build system of the package to put files into their desired outputs or it moves the files during the fixup phase. Each group of file types has an outputFoo variable specifying the output name where they should go. If that variable isn't defined by the derivation writer, it is guessed – a default output name is defined, falling back to other possibilities if the output isn't defined.

$outputDev

is for development-only files. These include C(++) headers, pkg-config, cmake and aclocal files. They go to dev or out by default.

$outputBin

is meant for user-facing binaries, typically residing in bin/. They go to bin or out by default.

$outputLib

is meant for libraries, typically residing in lib/ and libexec/. They go to lib or out by default.

$outputDoc

is for user documentation, typically residing in share/doc/. It goes to doc or out by default.

$outputDevdoc

is for developer documentation. Currently we count gtk-doc in there. It goes to devdoc or is removed (!) by default. This is because e.g. gtk-doc tends to be rather large and completely unused by nixpkgs users.

$outputMan

is for man pages (except for section 3). They go to man or doc or $outputBin by default.

$outputDevman

is for section 3 man pages. They go to devman or $outputMan by default.

$outputInfo

is for info pages. They go to info or doc or $outputMan by default.

4.4.2. Common caveats

  • Some configure scripts don't like some of the parameters passed by default by the framework, e.g. --docdir=/foo/bar. You can disable this by setting setOutputFlags = false;.

  • The outputs of a single derivation can retain references to each other, but note that circular references are not allowed. (And each strongly-connected component would act as a single output anyway.)

  • Most of split packages contain their core functionality in libraries. These libraries tend to refer to various kind of data that typically gets into out, e.g. locale strings, so there is often no advantage in separating the libraries into lib, as keeping them in out is easier.

  • Some packages have hidden assumptions on install paths, which complicates splitting.

Chapter 5. Cross-compilation

5.1. Introduction

"Cross-compilation" means compiling a program on one machine for another type of machine. For example, a typical use of cross compilation is to compile programs for embedded devices. These devices often don't have the computing power and memory to compile their own programs. One might think that cross-compilation is a fairly niche concern, but there are advantages to being rigorous about distinguishing build-time vs run-time environments even when one is developing and deploying on the same machine. Nixpkgs is increasingly adopting this opinion in that packages should be written with cross-compilation in mind, and nixpkgs should evaluate in a similar way (by minimizing cross-compilation-specific special cases) whether or not one is cross-compiling.

This chapter will be organized in three parts. First, it will describe the basics of how to package software in a way that supports cross-compilation. Second, it will describe how to use Nixpkgs when cross-compiling. Third, it will describe the internal infrastructure supporting cross-compilation.

5.2. Packaging in a cross-friendly manner

5.2.1. Platform parameters

The three GNU Autoconf platforms, build, host, and cross, are historically the result of much confusion. https://gcc.gnu.org/onlinedocs/gccint/Configure-Terms.html clears this up somewhat but there is more to be said. An important advice to get out the way is, unless you are packaging a compiler or other build tool, just worry about the build and host platforms. Dealing with just two platforms usually better matches people's preconceptions, and in this case is completely correct.

In Nixpkgs, these three platforms are defined as attribute sets under the names buildPlatform, hostPlatform, and targetPlatform. All are guaranteed to contain at least a platform field, which contains detailed information on the platform. All three are always defined at the top level, so one can get at them just like a dependency in a function that is imported with callPackage:

{ stdenv, buildPlatform, hostPlatform, fooDep, barDep, .. }: ...

Warning

These platforms should all have the same structure in all scenarios, but that is currently not the case. When not cross-compiling, they will each contain a system field with a short 2-part, hyphen-separated summering string name for the platform. But, when when cross compiling, hostPlatform and targetPlatform may instead contain config with a fuller 3- or 4-part string in the manner of LLVM. We should have all 3 platforms always contain both, and maybe give config a better name while we are at it.

buildPlatform

The "build platform" is the platform on which a package is built. Once someone has a built package, or pre-built binary package, the build platform should not matter and be safe to ignore.

hostPlatform

The "host platform" is the platform on which a package is run. This is the simplest platform to understand, but also the one with the worst name.

targetPlatform

The "target platform" is black sheep. The other two intrinsically apply to all compiled software—or any build process with a notion of "build-time" followed by "run-time". The target platform only applies to programming tools, and even then only is a good for for some of them. Briefly, GCC, Binutils, GHC, and certain other tools are written in such a way such that a single build can only compiler code for a single platform. Thus, when building them, one must think ahead about what platforms they wish to use the tool to produce machine code for, and build binaries for each.

There is no fundamental need to think about the target ahead of time like this. LLVM, for example, was designed from the beginning with cross-compilation in mind, and so a normal LLVM binary will support every architecture that LLVM supports. If the tool supports modular or pluggable backends, one might imagine specifying a set of target platforms / backends one wishes to support, rather than a single one.

The biggest reason for mess, if there is one, is that many compilers have the bad habit a build process that builds the compiler and standard library/runtime together. Then the specifying target platform is essential, because it determines the host platform of the standard library/runtime. Nixpkgs tries to avoid this where possible too, but still, because the concept of a target platform is so ingrained now in Autoconf and other tools, it is best to support it as is. Tools like LLVM that don't need up-front target platforms can safely ignore it like normal packages, and it will do no harm.

Note

If you dig around nixpkgs, you may notice there is also stdenv.cross. This field defined as hostPlatform when the host and build platforms differ, but otherwise not defined at all. This field is obsolete and will soon disappear—please do not use it.

5.2.2. Specifying Dependencies

As mentioned in the introduction to this chapter, one can think about a build time vs run time distinction whether cross-compiling or not. In the case of cross-compilation, this corresponds with whether a derivation running on the native or foreign platform is produced. An interesting thing to think about is how this corresponds with the three Autoconf platforms. In the run-time case, the depending and depended-on package simply have matching build, host, and target platforms. But in the build-time case, one can imagine "sliding" the platforms one over. The depended-on package's host and target platforms (respectively) become the depending package's build and host platforms. This is the most important guiding principle behind cross-compilation with Nixpkgs, and will be called the sliding window principle. In this manner, given the 3 platforms for one package, we can determine the three platforms for all its transitive dependencies.

Some examples will probably make this clearer. If a package is being built with a (build, host, target) platform triple of (foo, bar, bar), then its build-time dependencies would have a triple of (foo, foo, bar), and those packages' build-time dependencies would have triple of (foo, foo, foo). In other words, it should take two "rounds" of following build-time dependency edges before one reaches a fixed point where, by the sliding window principle, the platform triple no longer changes. Indeed, this happens with cross compilation, where only rounds of native dependencies starting with the second necessarily coincide with native packages.

Note

The depending package's target platform is unconstrained by the sliding window principle, which makes sense in that one can in principle build cross compilers targeting arbitrary platforms.

How does this work in practice? Nixpkgs is now structured so that build-time dependencies are taken from from buildPackages, whereas run-time dependencies are taken from the top level attribute set. For example, buildPackages.gcc should be used at build time, while gcc should be used at run time. Now, for most of Nixpkgs's history, there was no buildPackages, and most packages have not been refactored to use it explicitly. Instead, one can use the four attributes used for specifying dependencies as documented in ???. We "splice" together the run-time and build-time package sets with callPackage, and then mkDerivation for each of four attributes pulls the right derivation out. This splicing can be skipped when not cross compiling as the package sets are the same, but is a bit slow for cross compiling. Because of this, a best-of-both-worlds solution is in the works with no splicing or explicit access of buildPackages needed. For now, feel free to use either method.

5.3. Cross-building packages

Note

More information needs to moved from the old wiki, especially https://nixos.org/wiki/CrossCompiling, for this section.

Many sources (manual, wiki, etc) probably mention passing system, platform, and, optionally, crossSystem to nixpkgs: import <nixpkgs> { system = ..; platform = ..; crossSystem = ..; }. system and platform together determine the system on which packages are built, and crossSystem specifies the platform on which packages are ultimately intended to run, if it is different. This still works, but with more recent changes, one can alternatively pass localSystem, containing system and platform, for symmetry.

One would think that localSystem and crossSystem overlap horribly with the three *Platforms (buildPlatform, hostPlatform, and targetPlatform; see stage.nix or the manual). Actually, those identifiers are purposefully not used here to draw a subtle but important distinction: While the granularity of having 3 platforms is necessary to properly *build* packages, it is overkill for specifying the user's *intent* when making a build plan or package set. A simple "build vs deploy" dichotomy is adequate: the sliding window principle described in the previous section shows how to interpolate between the these two "end points" to get the 3 platform triple for each bootstrapping stage. That means for any package a given package set, even those not bound on the top level but only reachable via dependencies or buildPackages, the three platforms will be defined as one of localSystem or crossSystem, with the former replacing the latter as one traverses build-time dependencies. A last simple difference then is crossSystem should be null when one doesn't want to cross-compile, while the *Platforms are always non-null. localSystem is always non-null.

5.4. Cross-compilation infrastructure

To be written.

Note

If one explores nixpkgs, they will see derivations with names like gccCross. Such *Cross derivations is a holdover from before we properly distinguished between the host and target platforms —the derivation with "Cross" in the name covered the build = host != target case, while the other covered the host = target, with build platform the same or not based on whether one was using its .nativeDrv or .crossDrv. This ugliness will disappear soon.

Chapter 6. Global configuration

Nix packages can be configured to allow or deny certain options.

To apply the configuration edit ~/.config/nixpkgs/config.nix and set it like

{
  allowUnfree = true;
}

and will allow the Nix package manager to install unfree licensed packages.

The configuration as listed also applies to NixOS under nixpkgs.config set.

  • Allow installing of packages that are distributed under unfree license by setting

    allowUnfree =
        true;

    or deny them by setting it to false.

    Same can be achieved by setting the environment variable:

    $ export NIXPKGS_ALLOW_UNFREE=1
    

  • Whenever unfree packages are not allowed, single packages can still be allowed by a predicate function that accepts package as an argument and should return a boolean:

    allowUnfreePredicate = (pkg: ...);
    

    Example to allow flash player and visual studio code only:

    allowUnfreePredicate = with builtins; (pkg: elem (parseDrvName pkg.name).name [ "flashplayer" "vscode" ]);
    

  • Whenever unfree packages are not allowed, packages can still be whitelisted by their license:

    whitelistedLicenses = with stdenv.lib.licenses; [ amd wtfpl ];
    

  • In addition to whitelisting licenses which are denied by the allowUnfree setting, you can also explicitely deny installation of packages which have a certain license:

    blacklistedLicenses = with stdenv.lib.licenses; [ agpl3 gpl3 ];
    

A complete list of licenses can be found in the file lib/licenses.nix of the nix package tree.

6.1. Modify packages via packageOverrides

You can define a function called packageOverrides in your local ~/.config/nixpkgs/config.nix to overide nix packages. It must be a function that takes pkgs as an argument and return modified set of packages.

{
  packageOverrides = pkgs: rec {
    foo = pkgs.foo.override { ... };
  };
}

Chapter 7. Functions reference

The nixpkgs repository has several utility functions to manipulate Nix expressions.

7.1. Overriding

Sometimes one wants to override parts of nixpkgs, e.g. derivation attributes, the results of derivations or even the whole package set.

7.1.1. <pkg>.override

The function override is usually available for all the derivations in the nixpkgs expression (pkgs).

It is used to override the arguments passed to a function.

Example usages:

pkgs.foo.override { arg1 = val1; arg2 = val2; ... }

import pkgs.path { overlays = [ (self: super: {
    foo = super.foo.override { barSupport = true ; };
  })]};

mypkg = pkgs.callPackage ./mypkg.nix {
    mydep = pkgs.mydep.override { ... };
  }

In the first example, pkgs.foo is the result of a function call with some default arguments, usually a derivation. Using pkgs.foo.override will call the same function with the given new arguments.

7.1.2. <pkg>.overrideAttrs

The function overrideAttrs allows overriding the attribute set passed to a stdenv.mkDerivation call, producing a new derivation based on the original one. This function is available on all derivations produced by the stdenv.mkDerivation function, which is most packages in the nixpkgs expression pkgs.

Example usage:

helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec {
    separateDebugInfo = true;
  });

In the above example, the separateDebugInfo attribute is overriden to be true, thus building debug info for helloWithDebug, while all other attributes will be retained from the original hello package.

The argument oldAttrs is conventionally used to refer to the attr set originally passed to stdenv.mkDerivation.

Note

Note that separateDebugInfo is processed only by the stdenv.mkDerivation function, not the generated, raw Nix derivation. Thus, using overrideDerivation will not work in this case, as it overrides only the attributes of the final derivation. It is for this reason that overrideAttrs should be preferred in (almost) all cases to overrideDerivation, i.e. to allow using sdenv.mkDerivation to process input arguments, as well as the fact that it is easier to use (you can use the same attribute names you see in your Nix code, instead of the ones generated (e.g. buildInputs vs nativeBuildInputs, and involves less typing.

7.1.3. <pkg>.overrideDerivation

Warning

You should prefer overrideAttrs in almost all cases, see its documentation for the reasons why. overrideDerivation is not deprecated and will continue to work, but is less nice to use and does not have as many abilities as overrideAttrs.

Warning

Do not use this function in Nixpkgs as it evaluates a Derivation before modifying it, which breaks package abstraction and removes error-checking of function arguments. In addition, this evaluation-per-function application incurs a performance penalty, which can become a problem if many overrides are used. It is only intended for ad-hoc customisation, such as in ~/.config/nixpkgs/config.nix.

The function overrideDerivation creates a new derivation based on an existing one by overriding the original's attributes with the attribute set produced by the specified function. This function is available on all derivations defined using the makeOverridable function. Most standard derivation-producing functions, such as stdenv.mkDerivation, are defined using this function, which means most packages in the nixpkgs expression, pkgs, have this function.

Example usage:

mySed = pkgs.gnused.overrideDerivation (oldAttrs: {
    name = "sed-4.2.2-pre";
    src = fetchurl {
      url = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2;
      sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k";
    };
    patches = [];
  });

In the above example, the name, src, and patches of the derivation will be overridden, while all other attributes will be retained from the original derivation.

The argument oldAttrs is used to refer to the attribute set of the original derivation.

Note

A package's attributes are evaluated *before* being modified by the overrideDerivation function. For example, the name attribute reference in url = "mirror://gnu/hello/${name}.tar.gz"; is filled-in *before* the overrideDerivation function modifies the attribute set. This means that overriding the name attribute, in this example, *will not* change the value of the url attribute. Instead, we need to override both the name *and* url attributes.

7.1.4. lib.makeOverridable

The function lib.makeOverridable is used to make the result of a function easily customizable. This utility only makes sense for functions that accept an argument set and return an attribute set.

Example usage:

f = { a, b }: { result = a+b; }
  c = lib.makeOverridable f { a = 1; b = 2; }

The variable c is the value of the f function applied with some default arguments. Hence the value of c.result is 3, in this example.

The variable c however also has some additional functions, like c.override which can be used to override the default arguments. In this example the value of (c.override { a = 4; }).result is 6.

7.2. Generators

Generators are functions that create file formats from nix data structures, e. g. for configuration files. There are generators available for: INI, JSON and YAML

All generators follow a similar call interface: generatorName configFunctions data, where configFunctions is a set of user-defined functions that format variable parts of the content. They each have common defaults, so often they do not need to be set manually. An example is mkSectionName ? (name: libStr.escape [ "[" "]" ] name) from the INI generator. It gets the name of a section and returns a sanitized name. The default mkSectionName escapes [ and ] with a backslash.

Note

Nix store paths can be converted to strings by enclosing a derivation attribute like so: "${drv}".

Detailed documentation for each generator can be found in lib/generators.nix.

7.3. buildFHSUserEnv

buildFHSUserEnv provides a way to build and run FHS-compatible lightweight sandboxes. It creates an isolated root with bound /nix/store, so its footprint in terms of disk space needed is quite small. This allows one to run software which is hard or unfeasible to patch for NixOS -- 3rd-party source trees with FHS assumptions, games distributed as tarballs, software with integrity checking and/or external self-updated binaries. It uses Linux namespaces feature to create temporary lightweight environments which are destroyed after all child processes exit, without root user rights requirement. Accepted arguments are:

name

Environment name.

targetPkgs

Packages to be installed for the main host's architecture (i.e. x86_64 on x86_64 installations). Along with libraries binaries are also installed.

multiPkgs

Packages to be installed for all architectures supported by a host (i.e. i686 and x86_64 on x86_64 installations). Only libraries are installed by default.

extraBuildCommands

Additional commands to be executed for finalizing the directory structure.

extraBuildCommandsMulti

Like extraBuildCommands, but executed only on multilib architectures.

extraOutputsToInstall

Additional derivation outputs to be linked for both target and multi-architecture packages.

extraInstallCommands

Additional commands to be executed for finalizing the derivation with runner script.

runScript

A command that would be executed inside the sandbox and passed all the command line arguments. It defaults to bash.

One can create a simple environment using a shell.nix like that:

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
  name = "simple-x11-env";
  targetPkgs = pkgs: (with pkgs;
    [ udev
      alsaLib
    ]) ++ (with pkgs.xorg;
    [ libX11
      libXcursor
      libXrandr
    ]);
  multiPkgs = pkgs: (with pkgs;
    [ udev
      alsaLib
    ]);
  runScript = "bash";
}).env

Running nix-shell would then drop you into a shell with these libraries and binaries available. You can use this to run closed-source applications which expect FHS structure without hassles: simply change runScript to the application path, e.g. ./bin/start.sh -- relative paths are supported.

7.4. pkgs.dockerTools

pkgs.dockerTools is a set of functions for creating and manipulating Docker images according to the Docker Image Specification v1.0.0 . Docker itself is not used to perform any of the operations done by these functions.

Warning

The dockerTools API is unstable and may be subject to backwards-incompatible changes in the future.

7.4.1. buildImage

This function is analogous to the docker build command, in that can used to build a Docker-compatible repository tarball containing a single image with one or multiple layers. As such, the result is suitable for being loaded in Docker with docker load.

The parameters of buildImage with relative example values are described below:

Example 7.1. Docker build

  buildImage {
    name = "redis"; 1
    tag = "latest"; 2

    fromImage = someBaseImage; 3
    fromImageName = null; 4
    fromImageTag = "latest"; 5

    contents = pkgs.redis; 6
    runAsRoot = '' 7
      #!${stdenv.shell}
      mkdir -p /data
    '';

    config = { 8
      Cmd = [ "/bin/redis-server" ];
      WorkingDir = "/data";
      Volumes = {
        "/data" = {};
      };
    };
  }
  

The above example will build a Docker image redis/latest from the given base image. Loading and running this image in Docker results in redis-server being started automatically.

1

name specifies the name of the resulting image. This is the only required argument for buildImage.

2

tag specifies the tag of the resulting image. By default it's latest.

3

fromImage is the repository tarball containing the base image. It must be a valid Docker image, such as exported by docker save. By default it's null, which can be seen as equivalent to FROM scratch of a Dockerfile.

4

fromImageName can be used to further specify the base image within the repository, in case it contains multiple images. By default it's null, in which case buildImage will peek the first image available in the repository.

5

fromImageTag can be used to further specify the tag of the base image within the repository, in case an image contains multiple tags. By default it's null, in which case buildImage will peek the first tag available for the base image.

6

contents is a derivation that will be copied in the new layer of the resulting image. This can be similarly seen as ADD contents/ / in a Dockerfile. By default it's null.

7

runAsRoot is a bash script that will run as root in an environment that overlays the existing layers of the base image with the new resulting layer, including the previously copied contents derivation. This can be similarly seen as RUN ... in a Dockerfile.

Note

Using this parameter requires the kvm device to be available.

8

config is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the Docker Image Specification v1.0.0 .

After the new layer has been created, its closure (to which contents, config and runAsRoot contribute) will be copied in the layer itself. Only new dependencies that are not already in the existing layers will be copied.

At the end of the process, only one new single layer will be produced and added to the resulting image.

The resulting repository will only list the single image image/tag. In the case of Example 7.1, “Docker build” it would be redis/latest.

It is possible to inspect the arguments with which an image was built using its buildArgs attribute.

Note

If you see errors similar to getProtocolByName: does not exist (no such protocol name: tcp) you may need to add pkgs.iana_etc to contents.

Note

If you see errors similar to Error_Protocol ("certificate has unknown CA",True,UnknownCa) you may need to add pkgs.cacert to contents.

7.4.2. pullImage

This function is analogous to the docker pull command, in that can be used to fetch a Docker image from a Docker registry. Currently only registry v1 is supported. By default Docker Hub is used to pull images.

Its parameters are described in the example below:

Example 7.2. Docker pull

  pullImage {
    imageName = "debian"; 1
    imageTag = "jessie"; 2
    imageId = null; 3
    sha256 = "1bhw5hkz6chrnrih0ymjbmn69hyfriza2lr550xyvpdrnbzr4gk2"; 4

    indexUrl = "https://index.docker.io"; 5
    registryVersion = "v1";
  }
  

1

imageName specifies the name of the image to be downloaded, which can also include the registry namespace (e.g. library/debian). This argument is required.

2

imageTag specifies the tag of the image to be downloaded. By default it's latest.

3

imageId, if specified this exact image will be fetched, instead of imageName/imageTag. However, the resulting repository will still be named imageName/imageTag. By default it's null.

4

sha256 is the checksum of the whole fetched image. This argument is required.

Note

The checksum is computed on the unpacked directory, not on the final tarball.

5

In the above example the default values are shown for the variables indexUrl and registryVersion. Hence by default the Docker.io registry is used to pull the images.

7.4.3. exportImage

This function is analogous to the docker export command, in that can used to flatten a Docker image that contains multiple layers. It is in fact the result of the merge of all the layers of the image. As such, the result is suitable for being imported in Docker with docker import.

Note

Using this function requires the kvm device to be available.

The parameters of exportImage are the following:

Example 7.3. Docker export

  exportImage {
    fromImage = someLayeredImage;
    fromImageName = null;
    fromImageTag = null;

    name = someLayeredImage.name;
  }
  

The parameters relative to the base image have the same synopsis as described in Section 7.4.1, “buildImage”, except that fromImage is the only required argument in this case.

The name argument is the name of the derivation output, which defaults to fromImage.name.

7.4.4. shadowSetup

This constant string is a helper for setting up the base files for managing users and groups, only if such files don't exist already. It is suitable for being used in a runAsRoot 7 script for cases like in the example below:

Example 7.4. Shadow base files

  buildImage {
    name = "shadow-basic";

    runAsRoot = ''
      #!${stdenv.shell}
      ${shadowSetup}
      groupadd -r redis
      useradd -r -g redis redis
      mkdir /data
      chown redis:redis /data
    '';
  }
  

Creating base files like /etc/passwd or /etc/login.defs are necessary for shadow-utils to manipulate users and groups.

Chapter 8. Library functions

This paragraph describes the available library functions.

8.1. Top level functions

These functions are available under pkgs.lib

lib.concatImapStrings

Like `concatMapStrings' except that the f functions also gets the position as a parameter.

Example:

concatImapStrings (pos: x: "${toString pos}-${x}") ["foo" "bar"]
=> "1-foo2-bar"
lib.concatMapStrings

Map a function over a list and concatenate the resulting strings.

Example:

concatMapStrings (x: "a" + x) ["foo" "bar"]
=> "afooabar"
lib.concatMapStringsSep

First maps over the list and then concatenates it.

Example:

concatMapStringsSep "-" (x: toUpper x)  ["foo" "bar" "baz"]
=> "FOO-BAR-BAZ"
lib.concatStrings

Concatenate a list of strings

Example:

concastStrings ["foo "bar"]
=> "foobar"
lib.concatStringsSep

Concatenate a list of strings with a separator between each element

Example:

concatStringsSep "/" ["usr" "local" "bin"]
=> "usr/local/bin"
lib.intersperse

Place an element between each element of a list.

Example:

intersperse "/" ["usr" "local" "bin"]
=> ["usr" "/" "local" "/" "bin"].

8.2. All library functions

These functions are available under pkgs.$category.$name

lib.strings.concatImapStrings

Like `concatMapStrings' except that the f functions also gets the position as a parameter.

Example:

concatImapStrings (pos: x: "${toString pos}-${x}") ["foo" "bar"]
=> "1-foo2-bar"
lib.strings.concatMapStrings

Map a function over a list and concatenate the resulting strings.

Example:

concatMapStrings (x: "a" + x) ["foo" "bar"]
=> "afooabar"
lib.strings.concatMapStringsSep

First maps over the list and then concatenates it.

Example:

concatMapStringsSep "-" (x: toUpper x)  ["foo" "bar" "baz"]
=> "FOO-BAR-BAZ"
lib.strings.concatStrings

Concatenate a list of strings

Example:

concastStrings ["foo "bar"]
=> "foobar"
lib.strings.concatStringsSep

Concatenate a list of strings with a separator between each element

Example:

concatStringsSep "/" ["usr" "local" "bin"]
=> "usr/local/bin"
lib.strings.intersperse

Place an element between each element of a list.

Example:

intersperse "/" ["usr" "local" "bin"]
=> ["usr" "/" "local" "/" "bin"].

Chapter 9. Meta-attributes

Nix packages can declare meta-attributes that contain information about a package such as a description, its homepage, its license, and so on. For instance, the GNU Hello package has a meta declaration like this:

meta = {
  description = "A program that produces a familiar, friendly greeting";
  longDescription = ''
    GNU Hello is a program that prints "Hello, world!" when you run it.
    It is fully customizable.
  '';
  homepage = http://www.gnu.org/software/hello/manual/;
  license = stdenv.lib.licenses.gpl3Plus;
  maintainers = [ stdenv.lib.maintainers.eelco ];
  platforms = stdenv.lib.platforms.all;
};

Meta-attributes are not passed to the builder of the package. Thus, a change to a meta-attribute doesn’t trigger a recompilation of the package. The value of a meta-attribute must be a string.

The meta-attributes of a package can be queried from the command-line using nix-env:

$ nix-env -qa hello --json
{
    "hello": {
        "meta": {
            "description": "A program that produces a familiar, friendly greeting",
            "homepage": "http://www.gnu.org/software/hello/manual/",
            "license": {
                "fullName": "GNU General Public License version 3 or later",
                "shortName": "GPLv3+",
                "url": "http://www.fsf.org/licensing/licenses/gpl.html"
            },
            "longDescription": "GNU Hello is a program that prints \"Hello, world!\" when you run it.\nIt is fully customizable.\n",
            "maintainers": [
                "Ludovic Court\u00e8s <ludo@gnu.org>"
            ],
            "platforms": [
                "i686-linux",
                "x86_64-linux",
                "armv5tel-linux",
                "armv7l-linux",
                "mips64el-linux",
                "x86_64-darwin",
                "i686-cygwin",
                "i686-freebsd",
                "x86_64-freebsd",
                "i686-openbsd",
                "x86_64-openbsd"
            ],
            "position": "/home/user/dev/nixpkgs/pkgs/applications/misc/hello/default.nix:14"
        },
        "name": "hello-2.9",
        "system": "x86_64-linux"
    }
}


nix-env knows about the description field specifically:

$ nix-env -qa hello --description
hello-2.3  A program that produces a familiar, friendly greeting

9.1. Standard meta-attributes

It is expected that each meta-attribute is one of the following:

description

A short (one-line) description of the package. This is shown by nix-env -q --description and also on the Nixpkgs release pages.

Don’t include a period at the end. Don’t include newline characters. Capitalise the first character. For brevity, don’t repeat the name of package — just describe what it does.

Wrong: "libpng is a library that allows you to decode PNG images."

Right: "A library for decoding PNG images"

longDescription

An arbitrarily long description of the package.

branch

Release branch. Used to specify that a package is not going to receive updates that are not in this branch; for example, Linux kernel 3.0 is supposed to be updated to 3.0.X, not 3.1.

homepage

The package’s homepage. Example: http://www.gnu.org/software/hello/manual/

downloadPage

The page where a link to the current version can be found. Example: http://ftp.gnu.org/gnu/hello/

license

The license, or licenses, for the package. One from the attribute set defined in nixpkgs/lib/licenses.nix. At this moment using both a list of licenses and a single license is valid. If the license field is in the form of a list representation, then it means that parts of the package are licensed differently. Each license should preferably be referenced by their attribute. The non-list attribute value can also be a space delimited string representation of the contained attribute shortNames or spdxIds. The following are all valid examples:

  • Single license referenced by attribute (preferred) stdenv.lib.licenses.gpl3.

  • Single license referenced by its attribute shortName (frowned upon) "gpl3".

  • Single license referenced by its attribute spdxId (frowned upon) "GPL-3.0".

  • Multiple licenses referenced by attribute (preferred) with stdenv.lib.licenses; [ asl20 free ofl ].

  • Multiple licenses referenced as a space delimited string of attribute shortNames (frowned upon) "asl20 free ofl".

For details, see Section 9.2, “Licenses”.

maintainers

A list of names and e-mail addresses of the maintainers of this Nix expression. If you would like to be a maintainer of a package, you may want to add yourself to nixpkgs/lib/maintainers.nix and write something like [ stdenv.lib.maintainers.alice stdenv.lib.maintainers.bob ].

priority

The priority of the package, used by nix-env to resolve file name conflicts between packages. See the Nix manual page for nix-env for details. Example: "10" (a low-priority package).

platforms

The list of Nix platform types on which the package is supported. Hydra builds packages according to the platform specified. If no platform is specified, the package does not have prebuilt binaries. An example is:

meta.platforms = stdenv.lib.platforms.linux;

Attribute Set stdenv.lib.platforms in nixpkgs/lib/platforms.nix defines various common lists of platforms types.

hydraPlatforms

The list of Nix platform types for which the Hydra instance at hydra.nixos.org will build the package. (Hydra is the Nix-based continuous build system.) It defaults to the value of meta.platforms. Thus, the only reason to set meta.hydraPlatforms is if you want hydra.nixos.org to build the package on a subset of meta.platforms, or not at all, e.g.

meta.platforms = stdenv.lib.platforms.linux;
meta.hydraPlatforms = [];

broken

If set to true, the package is marked as “broken”, meaning that it won’t show up in nix-env -qa, and cannot be built or installed. Such packages should be removed from Nixpkgs eventually unless they are fixed.

updateWalker

If set to true, the package is tested to be updated correctly by the update-walker.sh script without additional settings. Such packages have meta.version set and their homepage (or the page specified by meta.downloadPage) contains a direct link to the package tarball.

9.2. Licenses

The meta.license attribute should preferrably contain a value from stdenv.lib.licenses defined in nixpkgs/lib/licenses.nix, or in-place license description of the same format if the license is unlikely to be useful in another expression.

Although it's typically better to indicate the specific license, a few generic options are available:

stdenv.lib.licenses.free, "free"

Catch-all for free software licenses not listed above.

stdenv.lib.licenses.unfreeRedistributable, "unfree-redistributable"

Unfree package that can be redistributed in binary form. That is, it’s legal to redistribute the output of the derivation. This means that the package can be included in the Nixpkgs channel.

Sometimes proprietary software can only be redistributed unmodified. Make sure the builder doesn’t actually modify the original binaries; otherwise we’re breaking the license. For instance, the NVIDIA X11 drivers can be redistributed unmodified, but our builder applies patchelf to make them work. Thus, its license is "unfree" and it cannot be included in the Nixpkgs channel.

stdenv.lib.licenses.unfree, "unfree"

Unfree package that cannot be redistributed. You can build it yourself, but you cannot redistribute the output of the derivation. Thus it cannot be included in the Nixpkgs channel.

stdenv.lib.licenses.unfreeRedistributableFirmware, "unfree-redistributable-firmware"

This package supplies unfree, redistributable firmware. This is a separate value from unfree-redistributable because not everybody cares whether firmware is free.

Chapter 10. Support for specific programming languages and frameworks

Table of Contents

10.1. Beam Languages (Erlang & Elixir)
10.1.1. Introduction
10.1.2. Build Tools
10.1.2.1. Rebar3
10.1.2.2. Mix & Erlang.mk
10.1.3. How to install Beam packages
10.1.4. Packaging Beam Applications
10.1.4.1. Erlang Applications
10.1.5. How to develop
10.1.5.1. Accessing an Environment
10.1.5.2. Creating a Shell
10.1.6. Generating Packages from Hex with Hex2Nix
10.2. Bower
10.2.1. bower2nix usage
10.2.2. buildBowerComponents function
10.2.3. Troubleshooting
10.3. Coq
10.4. Go
10.5. User’s Guide to the Haskell Infrastructure
10.5.1. How to install Haskell packages
10.5.2. How to create a development environment
10.5.2.1. How to install a compiler
10.5.2.2. How to install a compiler with libraries
10.5.2.3. How to install a compiler with libraries, hoogle and documentation indexes
10.5.2.4. How to build a Haskell project using Stack
10.5.2.5. How to create ad hoc environments for nix-shell
10.5.3. How to create Nix builds for your own private Haskell packages
10.5.3.1. How to build a stand-alone project
10.5.3.2. How to build projects that depend on each other
10.5.4. Miscellaneous Topics
10.5.4.1. How to build with profiling enabled
10.5.4.2. How to override package versions in a compiler-specific package set
10.5.4.3. How to recover from GHC’s infamous non-deterministic library ID bug
10.5.4.4. How to use the Haste Haskell-to-Javascript transpiler
10.5.4.5. Builds on Darwin fail with math.h not found
10.5.4.6. Builds using Stack complain about missing system libraries
10.5.4.7. Creating statically linked binaries
10.5.4.8. Building GHC with integer-simple
10.5.5. Other resources
10.6. Idris packages
10.6.1. callPackage
10.6.2. builtins
10.6.3. build-idris-package
10.6.4. build-builtin-package
10.6.5. with-packages
10.7. Java
10.8. Lua
10.9. Node.js packages
10.10. Perl
10.10.1. Generation from CPAN
10.11. Python
10.11.1. User Guide
10.11.1.1. Using Python
10.11.1.2. Developing with Python
10.11.1.3. Handling dependencies
10.11.1.4. Organising your packages
10.11.1.5. Including a derivation using callPackage
10.11.2. Reference
10.11.2.1. Interpreters
10.11.2.2. Building packages and applications
10.11.2.3. Development mode
10.11.2.4. Tools
10.11.3. FAQ
10.11.3.1. How can I install a working Python environment?
10.11.3.2. How to solve circular dependencies?
10.11.3.3. How to override a Python package?
10.11.3.4. python setup.py bdist_wheel cannot create .whl
10.11.3.5. install_data / data_files problems
10.11.3.6. Rationale of non-existent global site-packages
10.11.3.7. How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?
10.11.4. Contributing
10.11.4.1. Contributing guidelines
10.12. Qt
10.12.1. Libraries
10.12.2. Programs
10.12.3. KDE
10.13. R packages
10.13.1. Installation
10.13.2. RStudio
10.13.3. Updating the package set
10.13.4. Testing if the Nix-expression could be evaluated
10.14. Ruby
10.15. TeX Live
10.15.1. User's guide
10.15.2. Known problems
10.16. User’s Guide to Vim Plugins/Addons/Bundles/Scripts in Nixpkgs
10.16.1. dependencies by Vim plugins
10.16.2. HOWTO
10.16.3. Important repositories

The standard build environment makes it easy to build typical Autotools-based packages with very little code. Any other kind of package can be accomodated by overriding the appropriate phases of stdenv. However, there are specialised functions in Nixpkgs to easily build packages for other programming languages, such as Perl or Haskell. These are described in this chapter.

10.1. Beam Languages (Erlang & Elixir)

10.1.1. Introduction

In this document and related Nix expressions we use the term Beam to describe the environment. Beam is the name of the Erlang Virtial Machine and, as far as we know, from a packaging perspective all languages that run on Beam are interchangable. The things that do change, like the build system, are transperant to the users of the package. So we make no distinction.

10.1.2. Build Tools

10.1.2.1. Rebar3

By default Rebar3 wants to manage it's own dependencies. In the normal non-Nix, this is perfectly acceptable. In the Nix world it is not. To support this we have created two versions of rebar3, rebar3 and rebar3-open. The rebar3 version has been patched to remove the ability to download anything from it. If you are not running it a nix-shell or a nix-build then its probably not going to work for you. rebar3-open is the normal, un-modified rebar3. It should work exactly as would any other version of rebar3. Any Erlang package should rely on rebar3 and thats really what you should be using too.

10.1.2.2. Mix & Erlang.mk

Both Mix and Erlang.mk work exactly as you would expect. There is a bootstrap process that needs to be run for both of them. However, that is supported by the buildMix and buildErlangMk derivations.

10.1.3. How to install Beam packages

Beam packages are not registered in the top level simply because they are not relevant to the vast majority of Nix users. They are installable using the beamPackages attribute set. You can list the avialable packages in the beamPackages with the following command:

$ nix-env -f "<nixpkgs>" -qaP -A beamPackages
beamPackages.esqlite    esqlite-0.2.1
beamPackages.goldrush   goldrush-0.1.7
beamPackages.ibrowse    ibrowse-4.2.2
beamPackages.jiffy      jiffy-0.14.5
beamPackages.lager      lager-3.0.2
beamPackages.meck       meck-0.8.3
beamPackages.rebar3-pc  pc-1.1.0
  

To install any of those packages into your profile, refer to them by their attribute path (first column):

$ nix-env -f "<nixpkgs>" -iA beamPackages.ibrowse
  

The attribute path of any Beam packages corresponds to the name of that particular package in Hex or its OTP Application/Release name.

10.1.4. Packaging Beam Applications

10.1.4.1. Erlang Applications

10.1.4.1.1. Rebar3 Packages

There is a Nix functional called buildRebar3. We use this function to make a derivation that understands how to build the rebar3 project. For example, the epression we use to build the hex2nix project follows.

        {stdenv, fetchFromGitHub, buildRebar3, ibrowse, jsx, erlware_commons }:

          buildRebar3 rec {
            name = "hex2nix";
            version = "0.0.1";

            src = fetchFromGitHub {
              owner = "ericbmerritt";
              repo = "hex2nix";
              rev = "${version}";
              sha256 = "1w7xjidz1l5yjmhlplfx7kphmnpvqm67w99hd2m7kdixwdxq0zqg";
            };

          beamDeps = [ ibrowse jsx erlware_commons ];
        }
      

The only visible difference between this derivation and something like stdenv.mkDerivation is that we have added erlangDeps to the derivation. If you add your Beam dependencies here they will be correctly handled by the system.

If your package needs to compile native code via Rebar's port compilation mechenism. You should add compilePort = true; to the derivation.

10.1.4.1.2. Erlang.mk Packages

Erlang.mk functions almost identically to Rebar. The only real difference is that buildErlangMk is called instead of buildRebar3

        { buildErlangMk, fetchHex,  cowlib, ranch }:
          buildErlangMk {
            name = "cowboy";
            version = "1.0.4";
            src = fetchHex {
              pkg = "cowboy";
              version = "1.0.4";
              sha256 =
                "6a0edee96885fae3a8dd0ac1f333538a42e807db638a9453064ccfdaa6b9fdac";
            };
            beamDeps  = [ cowlib ranch ];

            meta = {
              description = ''Small, fast, modular HTTP server written in
                            Erlang.'';
              license = stdenv.lib.licenses.isc;
              homepage = "https://github.com/ninenines/cowboy";
            };
        }
      
10.1.4.1.3. Mix Packages

Mix functions almost identically to Rebar. The only real difference is that buildMix is called instead of buildRebar3

        { buildMix, fetchHex, plug, absinthe }:
        buildMix {
          name = "absinthe_plug";
          version = "1.0.0";
          src = fetchHex {
            pkg = "absinthe_plug";
            version = "1.0.0";
            sha256 =
              "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
          };
          beamDeps  = [ plug absinthe];

          meta = {
            description = ''A plug for Absinthe, an experimental GraphQL
                            toolkit'';
            license = stdenv.lib.licenses.bsd3;
            homepage = "https://github.com/CargoSense/absinthe_plug";
         };
       }
      

10.1.5. How to develop

10.1.5.1. Accessing an Environment

Often, all you want to do is be able to access a valid environment that contains a specific package and its dependencies. we can do that with the env part of a derivation. For example, lets say we want to access an erlang repl with ibrowse loaded up. We could do the following.

      ~/w/nixpkgs ❯❯❯ nix-shell -A beamPackages.ibrowse.env --run "erl"
      Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

      Eshell V7.0  (abort with ^G)
      1> m(ibrowse).
      Module: ibrowse
      MD5: 3b3e0137d0cbb28070146978a3392945
      Compiled: January 10 2016, 23:34
      Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam
      Compiler options:  [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"},
      debug_info,debug_info,nowarn_shadow_vars,
      warn_unused_import,warn_unused_vars,warnings_as_errors,
      {i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}]
      Exports:
      add_config/1                  send_req_direct/7
      all_trace_off/0               set_dest/3
      code_change/3                 set_max_attempts/3
      get_config_value/1            set_max_pipeline_size/3
      get_config_value/2            set_max_sessions/3
      get_metrics/0                 show_dest_status/0
      get_metrics/2                 show_dest_status/1
      handle_call/3                 show_dest_status/2
      handle_cast/2                 spawn_link_worker_process/1
      handle_info/2                 spawn_link_worker_process/2
      init/1                        spawn_worker_process/1
      module_info/0                 spawn_worker_process/2
      module_info/1                 start/0
      rescan_config/0               start_link/0
      rescan_config/1               stop/0
      send_req/3                    stop_worker_process/1
      send_req/4                    stream_close/1
      send_req/5                    stream_next/1
      send_req/6                    terminate/2
      send_req_direct/4             trace_off/0
      send_req_direct/5             trace_off/2
      send_req_direct/6             trace_on/0
      trace_on/2
      ok
      2>
    

Notice the -A beamPackages.ibrowse.env.That is the key to this functionality.

10.1.5.2. Creating a Shell

Getting access to an environment often isn't enough to do real development. Many times we need to create a shell.nix file and do our development inside of the environment specified by that file. This file looks a lot like the packaging described above. The main difference is that src points to project root and we call the package directly.

{ pkgs ? import "<nixpkgs"> {} }:

with pkgs;

let

  f = { buildRebar3, ibrowse, jsx, erlware_commons }:
      buildRebar3 {
        name = "hex2nix";
        version = "0.1.0";
        src = ./.;
        erlangDeps = [ ibrowse jsx erlware_commons ];
      };
  drv = beamPackages.callPackage f {};

in
 drv
    
10.1.5.2.1. Building in a shell

We can leveral the support of the Derivation, regardless of which build Derivation is called by calling the commands themselv.s

# =============================================================================
# Variables
# =============================================================================

NIX_TEMPLATES := "$(CURDIR)/nix-templates"

TARGET := "$(PREFIX)"

PROJECT_NAME := thorndyke

NIXPKGS=../nixpkgs
NIX_PATH=nixpkgs=$(NIXPKGS)
NIX_SHELL=nix-shell -I "$(NIX_PATH)" --pure
# =============================================================================
# Rules
# =============================================================================
.PHONY= all test clean repl shell build test analyze configure install \
        test-nix-install publish plt analyze

all: build

guard-%:
        @ if [ "${${*}}" == "" ]; then \
                echo "Environment variable $* not set"; \
                exit 1; \
        fi

clean:
        rm -rf _build
        rm -rf .cache

repl:
        $(NIX_SHELL) --run "iex -pa './_build/prod/lib/*/ebin'"

shell:
        $(NIX_SHELL)

configure:
        $(NIX_SHELL) --command 'eval "$$configurePhase"'

build: configure
        $(NIX_SHELL) --command 'eval "$$buildPhase"'

install:
        $(NIX_SHELL) --command 'eval "$$installPhase"'

test:
        $(NIX_SHELL) --command 'mix test --no-start --no-deps-check'

plt:
        $(NIX_SHELL) --run "mix dialyzer.plt --no-deps-check"

analyze: build plt
        $(NIX_SHELL) --run "mix dialyzer --no-compile"

    

If you add the shell.nix as described and user rebar as follows things should simply work. Aside from the test, plt, and analyze the talks work just fine for all of the build Derivations.

10.1.6. Generating Packages from Hex with Hex2Nix

Updating the Hex packages requires the use of the hex2nix tool. Given the path to the Erlang modules (usually pkgs/development/erlang-modules). It will happily dump a file called hex-packages.nix. That file will contain all the packages that use a recognized build system in Hex. However, it can't know whether or not all those packages are buildable.

To make life easier for our users, it makes good sense to go ahead and attempt to build all those packages and remove the ones that don't build. To do that, simply run the command (in the root of your nixpkgs repository). that follows.

$ nix-build -A beamPackages
    

That will build every package in beamPackages. Then you can go through and manually remove the ones that fail. Hopefully, someone will improve hex2nix in the future to automate that.

10.2. Bower

Bower is a package manager for web site front-end components. Bower packages (comprising of build artefacts and sometimes sources) are stored in git repositories, typically on Github. The package registry is run by the Bower team with package metadata coming from the bower.json file within each package.

The end result of running Bower is a bower_components directory which can be included in the web app's build process.

Bower can be run interactively, by installing nodePackages.bower. More interestingly, the Bower components can be declared in a Nix derivation, with the help of nodePackages.bower2nix.

10.2.1. bower2nix usage

Suppose you have a bower.json with the following contents:

Example 10.1. bower.json

{
  "name": "my-web-app",
  "dependencies": {
    "angular": "~1.5.0",
    "bootstrap": "~3.3.6"
  }
}


Running bower2nix will produce something like the following output:

{ fetchbower, buildEnv }:
buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [
  (fetchbower "angular" "1.5.3" "~1.5.0" "1749xb0firxdra4rzadm4q9x90v6pzkbd7xmcyjk6qfza09ykk9y")
  (fetchbower "bootstrap" "3.3.6" "~3.3.6" "1vvqlpbfcy0k5pncfjaiskj3y6scwifxygfqnw393sjfxiviwmbv")
  (fetchbower "jquery" "2.2.2" "1.9.1 - 2" "10sp5h98sqwk90y4k6hbdviwqzvzwqf47r3r51pakch5ii2y7js1")
]; }

Using the bower2nix command line arguments, the output can be redirected to a file. A name like bower-packages.nix would be fine.

The resulting derivation is a union of all the downloaded Bower packages (and their dependencies). To use it, they still need to be linked together by Bower, which is where buildBowerComponents is useful.

10.2.2. buildBowerComponents function

The function is implemented in pkgs/development/bower-modules/generic/default.nix. Example usage:

Example 10.2. buildBowerComponents

bowerComponents = buildBowerComponents {
  name = "my-web-app";
  generated = ./bower-packages.nix; 1
  src = myWebApp; 2
};


In Example 10.2, “buildBowerComponents”, the following arguments are of special significance to the function:

1

generated specifies the file which was created by bower2nix.

2

src is your project's sources. It needs to contain a bower.json file.

buildBowerComponents will run Bower to link together the output of bower2nix, resulting in a bower_components directory which can be used.

Here is an example of a web frontend build process using gulp. You might use grunt, or anything else.

Example 10.3. Example build script (gulpfile.js)

var gulp = require('gulp');

gulp.task('default', [], function () {
  gulp.start('build');
});

gulp.task('build', [], function () {
  console.log("Just a dummy gulp build");
  gulp
    .src(["./bower_components/**/*"])
    .pipe(gulp.dest("./gulpdist/"));
});

Example 10.4. Full example — default.nix

{ myWebApp ? { outPath = ./.; name = "myWebApp"; }
, pkgs ? import <nixpkgs> {}
}:

pkgs.stdenv.mkDerivation {
  name = "my-web-app-frontend";
  src = myWebApp;

  buildInputs = [ pkgs.nodePackages.gulp ];

  bowerComponents = pkgs.buildBowerComponents { 1
    name = "my-web-app";
    generated = ./bower-packages.nix;
    src = myWebApp;
  };

  buildPhase = ''
    cp --reflink=auto --no-preserve=mode -R $bowerComponents/bower_components . 2
    export HOME=$PWD 3
    ${pkgs.nodePackages.gulp}/bin/gulp build 4
  '';

  installPhase = "mv gulpdist $out";
}

A few notes about Example 10.4, “Full example — default.nix:

1

The result of buildBowerComponents is an input to the frontend build.

2

Whether to symlink or copy the bower_components directory depends on the build tool in use. In this case a copy is used to avoid gulp silliness with permissions.

3

gulp requires HOME to refer to a writeable directory.

4

The actual build command. Other tools could be used.

10.2.3. Troubleshooting

ENOCACHE errors from buildBowerComponents

This means that Bower was looking for a package version which doesn't exist in the generated bower-packages.nix.

If bower.json has been updated, then run bower2nix again.

It could also be a bug in bower2nix or fetchbower. If possible, try reformulating the version specification in bower.json.

10.3. Coq

Coq libraries should be installed in $(out)/lib/coq/${coq.coq-version}/user-contrib/. Such directories are automatically added to the $COQPATH environment variable by the hook defined in the Coq derivation.

Some libraries require OCaml and sometimes also Camlp5. The exact versions that were used to build Coq are saved in the coq.ocaml and coq.camlp5 attributes.

Here is a simple package example. It is a pure Coq library, thus it only depends on Coq. Its makefile has been generated using coq_makefile so we only have to set the $COQLIB variable at install time.

{stdenv, fetchurl, coq}:
stdenv.mkDerivation {
  src = fetchurl {
    url = http://coq.inria.fr/pylons/contribs/files/Karatsuba/v8.4/Karatsuba.tar.gz;
    sha256 = "0ymfpv4v49k4fm63nq6gcl1hbnnxrvjjp7yzc4973n49b853c5b1";
  };

  name = "coq-karatsuba";

  buildInputs = [ coq ];

  installFlags = "COQLIB=$(out)/lib/coq/${coq.coq-version}/";
}

10.4. Go

The function buildGoPackage builds standard Go programs.

Example 10.5. buildGoPackage

deis = buildGoPackage rec {
  name = "deis-${version}";
  version = "1.13.0";
  
  goPackagePath = "github.com/deis/deis"; 1
  subPackages = [ "client" ]; 2

  src = fetchFromGitHub {
    owner = "deis";
    repo = "deis";
    rev = "v${version}";
    sha256 = "1qv9lxqx7m18029lj8cw3k7jngvxs4iciwrypdy0gd2nnghc68sw";
  };

  goDeps = ./deps.nix; 3

  buildFlags = "--tags release"; 4
}

Example 10.5, “buildGoPackage” is an example expression using buildGoPackage, the following arguments are of special significance to the function:

1

goPackagePath specifies the package's canonical Go import path.

2

subPackages limits the builder from building child packages that have not been listed. If subPackages is not specified, all child packages will be built.

In this example only github.com/deis/deis/client will be built.

3

goDeps is where the Go dependencies of a Go program are listed as a list of package source identified by Go import path. It could be imported as a separate deps.nix file for readability. The dependency data structure is described below.

4

buildFlags is a list of flags passed to the go build command.

The goDeps attribute can be imported from a separate nix file that defines which Go libraries are needed and should be included in GOPATH for buildPhase.

Example 10.6. deps.nix

[ 1
  {
    goPackagePath = "gopkg.in/yaml.v2"; 2
    fetch = {
      type = "git"; 3
      url = "https://gopkg.in/yaml.v2";
      rev = "a83829b6f1293c91addabc89d0571c246397bbf4";
      sha256 = "1m4dsmk90sbi17571h6pld44zxz7jc4lrnl4f27dpd1l8g5xvjhh";
    };
  }
  {
    goPackagePath = "github.com/docopt/docopt-go";
    fetch = {
      type = "git";
      url = "https://github.com/docopt/docopt-go";
      rev = "784ddc588536785e7299f7272f39101f7faccc3f";
      sha256 = "0wwz48jl9fvl1iknvn9dqr4gfy1qs03gxaikrxxp9gry6773v3sj";
    };
  }
]

1

goDeps is a list of Go dependencies.

2

goPackagePath specifies Go package import path.

3

fetch type that needs to be used to get package source. If git is used there should be url, rev and sha256 defined next to it.

buildGoPackage produces Multiple-output packages where bin includes program binaries. You can test build a Go binary as follows:

    $ nix-build -A deis.bin
  

or build all outputs with:

    $ nix-build -A deis.all
  

bin output will be installed by default with nix-env -i or systemPackages.

You may use Go packages installed into the active Nix profiles by adding the following to your ~/.bashrc:

for p in $NIX_PROFILES; do
    GOPATH="$p/share/go:$GOPATH"
done

To extract dependency information from a Go package in automated way use go2nix. It can produce complete derivation and goDeps file for Go programs.

10.5. User’s Guide to the Haskell Infrastructure

10.5.1. How to install Haskell packages

Nixpkgs distributes build instructions for all Haskell packages registered on Hackage, but strangely enough normal Nix package lookups don’t seem to discover any of them, except for the default version of ghc, cabal-install, and stack:

$ nix-env -i alex
error: selector ‘alex’ matches no derivations
$ nix-env -qa ghc
ghc-7.10.2

The Haskell package set is not registered in the top-level namespace because it is huge. If all Haskell packages were visible to these commands, then name-based search/install operations would be much slower than they are now. We avoided that by keeping all Haskell-related packages in a separate attribute set called haskellPackages, which the following command will list:

$ nix-env -f "<nixpkgs>" -qaP -A haskellPackages
haskellPackages.a50         a50-0.5
haskellPackages.abacate     haskell-abacate-0.0.0.0
haskellPackages.abcBridge   haskell-abcBridge-0.12
haskellPackages.afv         afv-0.1.1
haskellPackages.alex        alex-3.1.4
haskellPackages.Allure      Allure-0.4.101.1
haskellPackages.alms        alms-0.6.7
[... some 8000 entries omitted  ...]

To install any of those packages into your profile, refer to them by their attribute path (first column):

$ nix-env -f "<nixpkgs>" -iA haskellPackages.Allure ...

The attribute path of any Haskell packages corresponds to the name of that particular package on Hackage: the package cabal-install has the attribute haskellPackages.cabal-install, and so on. (Actually, this convention causes trouble with packages like 3dmodels and 4Blocks, because these names are invalid identifiers in the Nix language. The issue of how to deal with these rare corner cases is currently unresolved.)

Haskell packages who’s Nix name (second column) begins with a haskell- prefix are packages that provide a library whereas packages without that prefix provide just executables. Libraries may provide executables too, though: the package haskell-pandoc, for example, installs both a library and an application. You can install and use Haskell executables just like any other program in Nixpkgs, but using Haskell libraries for development is a bit trickier and we’ll address that subject in great detail in section How to create a development environment.

Attribute paths are deterministic inside of Nixpkgs, but the path necessary to reach Nixpkgs varies from system to system. We dodged that problem by giving nix-env an explicit -f "<nixpkgs>" parameter, but if you call nix-env without that flag, then chances are the invocation fails:

$ nix-env -iA haskellPackages.cabal-install
error: attribute ‘haskellPackages’ in selection path
       ‘haskellPackages.cabal-install’ not found

On NixOS, for example, Nixpkgs does not exist in the top-level namespace by default. To figure out the proper attribute path, it’s easiest to query for the path of a well-known Nixpkgs package, i.e.:

$ nix-env -qaP coreutils
nixos.coreutils  coreutils-8.23

If your system responds like that (most NixOS installations will), then the attribute path to haskellPackages is nixos.haskellPackages. Thus, if you want to use nix-env without giving an explicit -f flag, then that’s the way to do it:

$ nix-env -qaP -A nixos.haskellPackages
$ nix-env -iA nixos.haskellPackages.cabal-install

Our current default compiler is GHC 7.10.x and the haskellPackages set contains packages built with that particular version. Nixpkgs contains the latest major release of every GHC since 6.10.4, however, and there is a whole family of package sets available that defines Hackage packages built with each of those compilers, too:

$ nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc6123
$ nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc763

The name haskellPackages is really just a synonym for haskell.packages.ghc7102, because we prefer that package set internally and recommend it to our users as their default choice, but ultimately you are free to compile your Haskell packages with any GHC version you please. The following command displays the complete list of available compilers:

$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler
haskell.compiler.ghc6104        ghc-6.10.4
haskell.compiler.ghc6123        ghc-6.12.3
haskell.compiler.ghc704         ghc-7.0.4
haskell.compiler.ghc722         ghc-7.2.2
haskell.compiler.ghc742         ghc-7.4.2
haskell.compiler.ghc763         ghc-7.6.3
haskell.compiler.ghc784         ghc-7.8.4
haskell.compiler.ghc7102        ghc-7.10.2
haskell.compiler.ghcHEAD        ghc-7.11.20150402
haskell.compiler.ghcNokinds     ghc-nokinds-7.11.20150704
haskell.compiler.ghcjs          ghcjs-0.1.0
haskell.compiler.jhc            jhc-0.8.2
haskell.compiler.uhc            uhc-1.1.9.0

We have no package sets for jhc or uhc yet, unfortunately, but for every version of GHC listed above, there exists a package set based on that compiler. Also, the attributes haskell.compiler.ghcXYC and haskell.packages.ghcXYC.ghc are synonymous for the sake of convenience.

10.5.2. How to create a development environment

10.5.2.1. How to install a compiler

A simple development environment consists of a Haskell compiler and one or both of the tools cabal-install and stack. We saw in section How to install Haskell packages how you can install those programs into your user profile:

$ nix-env -f "<nixpkgs>" -iA haskellPackages.ghc haskellPackages.cabal-install

Instead of the default package set haskellPackages, you can also use the more precise name haskell.compiler.ghc7102, which has the advantage that it refers to the same GHC version regardless of what Nixpkgs considers default at any given time.

Once you’ve made those tools available in $PATH, it’s possible to build Hackage packages the same way people without access to Nix do it all the time:

$ cabal get lens-4.11 && cd lens-4.11
$ cabal install -j --dependencies-only
$ cabal configure
$ cabal build

If you enjoy working with Cabal sandboxes, then that’s entirely possible too: just execute the command

$ cabal sandbox init

before installing the required dependencies.

The nix-shell utility makes it easy to switch to a different compiler version; just enter the Nix shell environment with the command

$ nix-shell -p haskell.compiler.ghc784

to bring GHC 7.8.4 into $PATH. Alternatively, you can use Stack instead of nix-shell directly to select compiler versions and other build tools per-project. It uses nix-shell under the hood when Nix support is turned on. See How to build a Haskell project using Stack.

If you’re using cabal-install, re-running cabal configure inside the spawned shell switches your build to use that compiler instead. If you’re working on a project that doesn’t depend on any additional system libraries outside of GHC, then it’s even sufficient to just run the cabal configure command inside of the shell:

$ nix-shell -p haskell.compiler.ghc784 --command "cabal configure"

Afterwards, all other commands like cabal build work just fine in any shell environment, because the configure phase recorded the absolute paths to all required tools like GHC in its build configuration inside of the dist/ directory. Please note, however, that nix-collect-garbage can break such an environment because the Nix store paths created by nix-shell aren’t alive anymore once nix-shell has terminated. If you find that your Haskell builds no longer work after garbage collection, then you’ll have to re-run cabal configure inside of a new nix-shell environment.

10.5.2.2. How to install a compiler with libraries

GHC expects to find all installed libraries inside of its own lib directory. This approach works fine on traditional Unix systems, but it doesn’t work for Nix, because GHC’s store path is immutable once it’s built. We cannot install additional libraries into that location. As a consequence, our copies of GHC don’t know any packages except their own core libraries, like base, containers, Cabal, etc.

We can register additional libraries to GHC, however, using a special build function called ghcWithPackages. That function expects one argument: a function that maps from an attribute set of Haskell packages to a list of packages, which determines the libraries known to that particular version of GHC. For example, the Nix expression ghcWithPackages (pkgs: [pkgs.mtl]) generates a copy of GHC that has the mtl library registered in addition to its normal core packages:

$ nix-shell -p "haskellPackages.ghcWithPackages (pkgs: [pkgs.mtl])"

[nix-shell:~]$ ghc-pkg list mtl
/nix/store/zy79...-ghc-7.10.2/lib/ghc-7.10.2/package.conf.d:
    mtl-2.2.1

This function allows users to define their own development environment by means of an override. After adding the following snippet to ~/.config/nixpkgs/config.nix,

{
  packageOverrides = super: let self = super.pkgs; in
  {
    myHaskellEnv = self.haskell.packages.ghc7102.ghcWithPackages
                     (haskellPackages: with haskellPackages; [
                       # libraries
                       arrows async cgi criterion
                       # tools
                       cabal-install haskintex
                     ]);
  };
}

it’s possible to install that compiler with nix-env -f "<nixpkgs>" -iA myHaskellEnv. If you’d like to switch that development environment to a different version of GHC, just replace the ghc7102 bit in the previous definition with the appropriate name. Of course, it’s also possible to define any number of these development environments! (You can’t install two of them into the same profile at the same time, though, because that would result in file conflicts.)

The generated ghc program is a wrapper script that re-directs the real GHC executable to use a new lib directory — one that we specifically constructed to contain all those packages the user requested:

$ cat $(type -p ghc)
#! /nix/store/xlxj...-bash-4.3-p33/bin/bash -e
export NIX_GHC=/nix/store/19sm...-ghc-7.10.2/bin/ghc
export NIX_GHCPKG=/nix/store/19sm...-ghc-7.10.2/bin/ghc-pkg
export NIX_GHC_DOCDIR=/nix/store/19sm...-ghc-7.10.2/share/doc/ghc/html
export NIX_GHC_LIBDIR=/nix/store/19sm...-ghc-7.10.2/lib/ghc-7.10.2
exec /nix/store/j50p...-ghc-7.10.2/bin/ghc "-B$NIX_GHC_LIBDIR" "$@"

The variables $NIX_GHC, $NIX_GHCPKG, etc. point to the new store path ghcWithPackages constructed specifically for this environment. The last line of the wrapper script then executes the real ghc, but passes the path to the new lib directory using GHC’s -B flag.

The purpose of those environment variables is to work around an impurity in the popular ghc-paths library. That library promises to give its users access to GHC’s installation paths. Only, the library can’t possible know that path when it’s compiled, because the path GHC considers its own is determined only much later, when the user configures it through ghcWithPackages. So we patched ghc-paths to return the paths found in those environment variables at run-time rather than trying to guess them at compile-time.

To make sure that mechanism works properly all the time, we recommend that you set those variables to meaningful values in your shell environment, too, i.e. by adding the following code to your ~/.bashrc:

if type >/dev/null 2>&1 -p ghc; then
  eval "$(egrep ^export "$(type -p ghc)")"
fi

If you are certain that you’ll use only one GHC environment which is located in your user profile, then you can use the following code, too, which has the advantage that it doesn’t contain any paths from the Nix store, i.e. those settings always remain valid even if a nix-env -u operation updates the GHC environment in your profile:

if [ -e ~/.nix-profile/bin/ghc ]; then
  export NIX_GHC="$HOME/.nix-profile/bin/ghc"
  export NIX_GHCPKG="$HOME/.nix-profile/bin/ghc-pkg"
  export NIX_GHC_DOCDIR="$HOME/.nix-profile/share/doc/ghc/html"
  export NIX_GHC_LIBDIR="$HOME/.nix-profile/lib/ghc-$($NIX_GHC --numeric-version)"
fi

10.5.2.3. How to install a compiler with libraries, hoogle and documentation indexes

If you plan to use your environment for interactive programming, not just compiling random Haskell code, you might want to replace ghcWithPackages in all the listings above with ghcWithHoogle.

This environment generator not only produces an environment with GHC and all the specified libraries, but also generates a hoogle and haddock indexes for all the packages, and provides a wrapper script around hoogle binary that uses all those things. A precise name for this thing would be ghcWithPackagesAndHoogleAndDocumentationIndexes, which is, regrettably, too long and scary.

For example, installing the following environment

{
  packageOverrides = super: let self = super.pkgs; in
  {
    myHaskellEnv = self.haskellPackages.ghcWithHoogle
                     (haskellPackages: with haskellPackages; [
                       # libraries
                       arrows async cgi criterion
                       # tools
                       cabal-install haskintex
                     ]);
  };
}

allows one to browse module documentation index not too dissimilar to this for all the specified packages and their dependencies by directing a browser of choice to ~/.nix-profiles/share/doc/hoogle/index.html (or /run/current-system/sw/share/doc/hoogle/index.html in case you put it in environment.systemPackages in NixOS).

After you’ve marveled enough at that try adding the following to your ~/.ghc/ghci.conf

:def hoogle \s -> return $ ":! hoogle search -cl --count=15 \"" ++ s ++ "\""
:def doc \s -> return $ ":! hoogle search -cl --info \"" ++ s ++ "\""

and test it by typing into ghci:

:hoogle a -> a
:doc a -> a

Be sure to note the links to haddock files in the output. With any modern and properly configured terminal emulator you can just click those links to navigate there.

Finally, you can run

hoogle server -p 8080

and navigate to http://localhost:8080/ for your own local Hoogle. Note, however, that Firefox and possibly other browsers disallow navigation from http: to file: URIs for security reasons, which might be quite an inconvenience. See this page for workarounds.

10.5.2.4. How to build a Haskell project using Stack

Stack is a popular build tool for Haskell projects. It has first-class support for Nix. Stack can optionally use Nix to automatically select the right version of GHC and other build tools to build, test and execute apps in an existing project downloaded from somewhere on the Internet. Pass the --nix flag to any stack command to do so, e.g.

$ git clone --recursive http://github.com/yesodweb/wai
$ cd wai
$ stack --nix build

If you want stack to use Nix by default, you can add a nix section to the stack.yaml file, as explained in the Stack documentation. For example:

nix:
  enable: true
  packages: [pkgconfig zeromq zlib]

The example configuration snippet above tells Stack to create an ad hoc environment for nix-shell as in the below section, in which the pkgconfig, zeromq and zlib packages from Nixpkgs are available. All stack commands will implicitly be executed inside this ad hoc environment.

Some projects have more sophisticated needs. For examples, some ad hoc environments might need to expose Nixpkgs packages compiled in a certain way, or with extra environment variables. In these cases, you’ll need a shell field instead of packages:

nix:
  enable: true
  shell-file: shell.nix

For more on how to write a shell.nix file see the below section. You’ll need to express a derivation. Note that Nixpkgs ships with a convenience wrapper function around mkDerivation called haskell.lib.buildStackProject to help you create this derivation in exactly the way Stack expects. All of the same inputs as mkDerivation can be provided. For example, to build a Stack project that including packages that link against a version of the R library compiled with special options turned on:

with (import <nixpkgs> { });

let R = pkgs.R.override { enableStrictBarrier = true; };
in
haskell.lib.buildStackProject {
  name = "HaskellR";
  buildInputs = [ R zeromq zlib ];
}

You can select a particular GHC version to compile with by setting the ghc attribute as an argument to buildStackProject. Better yet, let Stack choose what GHC version it wants based on the snapshot specified in stack.yaml (only works with Stack >= 1.1.3):

{nixpkgs ? import <nixpkgs> { }, ghc ? nixpkgs.ghc}:

with nixpkgs;

let R = pkgs.R.override { enableStrictBarrier = true; };
in
haskell.lib.buildStackProject {
  name = "HaskellR";
  buildInputs = [ R zeromq zlib ];
  inherit ghc;
}

10.5.2.5. How to create ad hoc environments for nix-shell

The easiest way to create an ad hoc development environment is to run nix-shell with the appropriate GHC environment given on the command-line:

nix-shell -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [mtl pandoc])"

For more sophisticated use-cases, however, it’s more convenient to save the desired configuration in a file called shell.nix that looks like this:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
let
  inherit (nixpkgs) pkgs;
  ghc = pkgs.haskell.packages.${compiler}.ghcWithPackages (ps: with ps; [
          monad-par mtl
        ]);
in
pkgs.stdenv.mkDerivation {
  name = "my-haskell-env-0";
  buildInputs = [ ghc ];
  shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)";
}

Now run nix-shell — or even nix-shell --pure — to enter a shell environment that has the appropriate compiler in $PATH. If you use --pure, then add all other packages that your development environment needs into the buildInputs attribute. If you’d like to switch to a different compiler version, then pass an appropriate compiler argument to the expression, i.e. nix-shell --argstr compiler ghc784.

If you need such an environment because you’d like to compile a Hackage package outside of Nix — i.e. because you’re hacking on the latest version from Git —, then the package set provides suitable nix-shell environments for you already! Every Haskell package has an env attribute that provides a shell environment suitable for compiling that particular package. If you’d like to hack the lens library, for example, then you just have to check out the source code and enter the appropriate environment:

  $ cabal get lens-4.11 && cd lens-4.11
  Downloading lens-4.11...
  Unpacking to lens-4.11/

  $ nix-shell "<nixpkgs>" -A haskellPackages.lens.env
  [nix-shell:/tmp/lens-4.11]$

At point, you can run cabal configure, cabal build, and all the other development commands. Note that you need cabal-install installed in your $PATH already to use it here — the nix-shell environment does not provide it.

10.5.3. How to create Nix builds for your own private Haskell packages

If your own Haskell packages have build instructions for Cabal, then you can convert those automatically into build instructions for Nix using the cabal2nix utility, which you can install into your profile by running nix-env -i cabal2nix.

10.5.3.1. How to build a stand-alone project

For example, let’s assume that you’re working on a private project called foo. To generate a Nix build expression for it, change into the project’s top-level directory and run the command:

$ cabal2nix . >foo.nix

Then write the following snippet into a file called default.nix:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
nixpkgs.pkgs.haskell.packages.${compiler}.callPackage ./foo.nix { }

Finally, store the following code in a file called shell.nix:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
(import ./default.nix { inherit nixpkgs compiler; }).env

At this point, you can run nix-build to have Nix compile your project and install it into a Nix store path. The local directory will contain a symlink called result after nix-build returns that points into that location. Of course, passing the flag --argstr compiler ghc763 allows switching the build to any version of GHC currently supported.

Furthermore, you can call nix-shell to enter an interactive development environment in which you can use cabal configure and cabal build to develop your code. That environment will automatically contain a proper GHC derivation with all the required libraries registered as well as all the system-level libraries your package might need.

If your package does not depend on any system-level libraries, then it’s sufficient to run

$ nix-shell --command "cabal configure"

once to set up your build. cabal-install determines the absolute paths to all resources required for the build and writes them into a config file in the dist/ directory. Once that’s done, you can run cabal build and any other command for that project even outside of the nix-shell environment. This feature is particularly nice for those of us who like to edit their code with an IDE, like Emacs’ haskell-mode, because it’s not necessary to start Emacs inside of nix-shell just to make it find out the necessary settings for building the project; cabal-install has already done that for us.

If you want to do some quick-and-dirty hacking and don’t want to bother setting up a default.nix and shell.nix file manually, then you can use the --shell flag offered by cabal2nix to have it generate a stand-alone nix-shell environment for you. With that feature, running

$ cabal2nix --shell . >shell.nix
$ nix-shell --command "cabal configure"

is usually enough to set up a build environment for any given Haskell package. You can even use that generated file to run nix-build, too:

$ nix-build shell.nix

10.5.3.2. How to build projects that depend on each other

If you have multiple private Haskell packages that depend on each other, then you’ll have to register those packages in the Nixpkgs set to make them visible for the dependency resolution performed by callPackage. First of all, change into each of your projects top-level directories and generate a default.nix file with cabal2nix:

$ cd ~/src/foo && cabal2nix . >default.nix
$ cd ~/src/bar && cabal2nix . >default.nix

Then edit your ~/.config/nixpkgs/config.nix file to register those builds in the default Haskell package set:

  {
    packageOverrides = super: let self = super.pkgs; in
    {
      haskellPackages = super.haskellPackages.override {
        overrides = self: super: {
          foo = self.callPackage ../src/foo {};
          bar = self.callPackage ../src/bar {};
        };
      };
    };
  }

Once that’s accomplished, nix-env -f "<nixpkgs>" -qA haskellPackages will show your packages like any other package from Hackage, and you can build them

$ nix-build "<nixpkgs>" -A haskellPackages.foo

or enter an interactive shell environment suitable for building them:

$ nix-shell "<nixpkgs>" -A haskellPackages.bar.env

10.5.4. Miscellaneous Topics

10.5.4.1. How to build with profiling enabled

Every Haskell package set takes a function called overrides that you can use to manipulate the package as much as you please. One useful application of this feature is to replace the default mkDerivation function with one that enables library profiling for all packages. To accomplish that, add configure the following snippet in your ~/.config/nixpkgs/config.nix file:

{
  packageOverrides = super: let self = super.pkgs; in
  {
    profiledHaskellPackages = self.haskellPackages.override {
      overrides = self: super: {
        mkDerivation = args: super.mkDerivation (args // {
          enableLibraryProfiling = true;
        });
      };
    };
  };
}

Then, replace instances of haskellPackages in the cabal2nix-generated default.nix or shell.nix files with profiledHaskellPackages.

10.5.4.2. How to override package versions in a compiler-specific package set

Nixpkgs provides the latest version of ghc-events, which is 0.4.4.0 at the time of this writing. This is fine for users of GHC 7.10.x, but GHC 7.8.4 cannot compile that binary. Now, one way to solve that problem is to register an older version of ghc-events in the 7.8.x-specific package set. The first step is to generate Nix build instructions with cabal2nix:

$ cabal2nix cabal://ghc-events-0.4.3.0 >~/.nixpkgs/ghc-events-0.4.3.0.nix

Then add the override in ~/.config/nixpkgs/config.nix:

{
  packageOverrides = super: let self = super.pkgs; in
  {
    haskell = super.haskell // {
      packages = super.haskell.packages // {
        ghc784 = super.haskell.packages.ghc784.override {
          overrides = self: super: {
            ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {};
          };
        };
      };
    };
  };
}

This code is a little crazy, no doubt, but it’s necessary because the intuitive version

haskell.packages.ghc784 = super.haskell.packages.ghc784.override {
  overrides = self: super: {
    ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {};
  };
};

doesn’t do what we want it to: that code replaces the haskell package set in Nixpkgs with one that contains only one entry,packages, which contains only one entry ghc784. This override loses the haskell.compiler set, and it loses the haskell.packages.ghcXYZ sets for all compilers but GHC 7.8.4. To avoid that problem, we have to perform the convoluted little dance from above, iterating over each step in hierarchy.

Once it’s accomplished, however, we can install a variant of ghc-events that’s compiled with GHC 7.8.4:

nix-env -f "<nixpkgs>" -iA haskell.packages.ghc784.ghc-events

Unfortunately, it turns out that this build fails again while executing the test suite! Apparently, the release archive on Hackage is missing some data files that the test suite requires, so we cannot run it. We accomplish that by re-generating the Nix expression with the --no-check flag:

$ cabal2nix --no-check cabal://ghc-events-0.4.3.0 >~/.nixpkgs/ghc-events-0.4.3.0.nix

Now the builds succeeds.

Of course, in the concrete example of ghc-events this whole exercise is not an ideal solution, because ghc-events can analyze the output emitted by any version of GHC later than 6.12 regardless of the compiler version that was used to build the ghc-events executable, so strictly speaking there’s no reason to prefer one built with GHC 7.8.x in the first place. However, for users who cannot use GHC 7.10.x at all for some reason, the approach of downgrading to an older version might be useful.

10.5.4.3. How to recover from GHC’s infamous non-deterministic library ID bug

GHC and distributed build farms don’t get along well:

https://ghc.haskell.org/trac/ghc/ticket/4012

When you see an error like this one

package foo-0.7.1.0 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91

then you have to download and re-install foo and all its dependents from scratch:

# nix-store -q --referrers /nix/store/*-haskell-text-1.2.0.4 \
  | xargs -L 1 nix-store --repair-path

If you’re using additional Hydra servers other than hydra.nixos.org, then it might be necessary to purge the local caches that store data from those machines to disable these binary channels for the duration of the previous command, i.e. by running:

rm /nix/var/nix/binary-cache-v3.sqlite
rm /nix/var/nix/manifests/*
rm /nix/var/nix/channel-cache/*

10.5.4.4. How to use the Haste Haskell-to-Javascript transpiler

Open a shell with haste-compiler and haste-cabal-install (you don’t actually need node, but it can be useful to test stuff):

$ nix-shell -p "haskellPackages.ghcWithPackages (self: with self; [haste-cabal-install haste-compiler])" -p nodejs

You may not need the following step but if haste-boot fails to compile all the packages it needs, this might do the trick

$ haste-cabal update

haste-boot builds a set of core libraries so that they can be used from Javascript transpiled programs:

$ haste-boot

Transpile and run a Hello world program:

$ echo 'module Main where main = putStrLn "Hello world"' > hello-world.hs
$ hastec --onexec hello-world.hs
$ node hello-world.js
Hello world

10.5.4.5. Builds on Darwin fail with math.h not found

Users of GHC on Darwin have occasionally reported that builds fail, because the compiler complains about a missing include file:

fatal error: 'math.h' file not found

The issue has been discussed at length in ticket 6390, and so far no good solution has been proposed. As a work-around, users who run into this problem can configure the environment variables

export NIX_CFLAGS_COMPILE="-idirafter /usr/include"
export NIX_CFLAGS_LINK="-L/usr/lib"

in their ~/.bashrc file to avoid the compiler error.

10.5.4.6. Builds using Stack complain about missing system libraries

--  While building package zlib-0.5.4.2 using:
  runhaskell -package=Cabal-1.22.4.0 -clear-package-db [... lots of flags ...]
Process exited with code: ExitFailure 1
Logs have been written to: /home/foo/src/stack-ide/.stack-work/logs/zlib-0.5.4.2.log

Configuring zlib-0.5.4.2...
Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: zlib.h
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

When you run the build inside of the nix-shell environment, the system is configured to find libz.so without any special flags – the compiler and linker just know how to find it. Consequently, Cabal won’t record any search paths for libz.so in the package description, which means that the package works fine inside of nix-shell, but once you leave the shell the shared object can no longer be found. That issue is by no means specific to Stack: you’ll have that problem with any other Haskell package that’s built inside of nix-shell but run outside of that environment.

You can remedy this issue in several ways. The easiest is to add a nix section to the stack.yaml like the following:

nix:
  enable: true
  packages: [ zlib ]

Stack’s Nix support knows to add ${zlib.out}/lib and ${zlib.dev}/include as an --extra-lib-dirs and extra-include-dirs, respectively. Alternatively, you can achieve the same effect by hand. First of all, run

$ nix-build --no-out-link "<nixpkgs>" -A zlib
/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8

to find out the store path of the system’s zlib library. Now, you can

  1. add that path (plus a /lib suffix) to your $LD_LIBRARY_PATH environment variable to make sure your system linker finds libz.so automatically. It’s no pretty solution, but it will work.

  2. As a variant of (1), you can also install any number of system libraries into your user’s profile (or some other profile) and point $LD_LIBRARY_PATH to that profile instead, so that you don’t have to list dozens of those store paths all over the place.

  3. The solution I prefer is to call stack with an appropriate –extra-lib-dirs flag like so:

    $ stack –extra-lib-dirs=/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8/lib build

Typically, you’ll need –extra-include-dirs as well. It’s possible to add those flag to the project’s stack.yaml or your user’s global ~/.stack/global/stack.yaml file so that you don’t have to specify them manually every time. But again, you’re likely better off using Stack’s Nix support instead.

The same thing applies to cabal configure, of course, if you’re building with cabal-install instead of Stack.

10.5.4.7. Creating statically linked binaries

There are two levels of static linking. The first option is to configure the build with the Cabal flag --disable-executable-dynamic. In Nix expressions, this can be achieved by setting the attribute:

enableSharedExecutables = false;

That gives you a binary with statically linked Haskell libraries and dynamically linked system libraries.

To link both Haskell libraries and system libraries statically, the additional flags --ghc-option=-optl=-static --ghc-option=-optl=-pthread need to be used. In Nix, this is accomplished with:

configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ];

It’s important to realize, however, that most system libraries in Nix are built as shared libraries only, i.e. there is just no static library available that Cabal could link!

10.5.4.8. Building GHC with integer-simple

By default GHC implements the Integer type using the GNU Multiple Precision Arithmetic (GMP) library. The implementation can be found in the integer-gmp package.

A potential problem with this is that GMP is licensed under the ​GNU Lesser General Public License (LGPL), a kind of copyleft license. According to the terms of the LGPL, paragraph 5, you may distribute a program that is designed to be compiled and dynamically linked with the library under the terms of your choice (i.e., commercially) but if your program incorporates portions of the library, if it is linked statically, then your program is a derivative–a work based on the library–and according to paragraph 2, section c, you must cause the whole of the work to be licensed under the terms of the LGPL (including for free).

The LGPL licensing for GMP is a problem for the overall licensing of binary programs compiled with GHC because most distributions (and builds) of GHC use static libraries. (Dynamic libraries are currently distributed only for OS X.) The LGPL licensing situation may be worse: even though ​The Glasgow Haskell Compiler License is essentially a free software license (BSD3), according to paragraph 2 of the LGPL, GHC must be distributed under the terms of the LGPL!

To work around these problems GHC can be build with a slower but LGPL-free alternative implemention for Integer called integer-simple.

To get a GHC compiler build with integer-simple instead of integer-gmp use the attribute: pkgs.haskell.compiler.integer-simple."${ghcVersion}". For example:

$ nix-build -E '(import <nixpkgs> {}).pkgs.haskell.compiler.integer-simple.ghc802'
...
$ result/bin/ghc-pkg list | grep integer
    integer-simple-0.1.1.1

The following command displays the complete list of GHC compilers build with integer-simple:

$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler.integer-simple
haskell.compiler.integer-simple.ghc7102  ghc-7.10.2
haskell.compiler.integer-simple.ghc7103  ghc-7.10.3
haskell.compiler.integer-simple.ghc722   ghc-7.2.2
haskell.compiler.integer-simple.ghc742   ghc-7.4.2
haskell.compiler.integer-simple.ghc763   ghc-7.6.3
haskell.compiler.integer-simple.ghc783   ghc-7.8.3
haskell.compiler.integer-simple.ghc784   ghc-7.8.4
haskell.compiler.integer-simple.ghc801   ghc-8.0.1
haskell.compiler.integer-simple.ghc802   ghc-8.0.2
haskell.compiler.integer-simple.ghcHEAD  ghc-8.1.20170106

To get a package set supporting integer-simple use the attribute: pkgs.haskell.packages.integer-simple."${ghcVersion}". For example use the following to get the scientific package build with integer-simple:

$ nix-build -A pkgs.haskell.packages.integer-simple.ghc802.scientific

10.5.5. Other resources

  • The Youtube video Nix Loves Haskell provides an introduction into Haskell NG aimed at beginners. The slides are available at http://cryp.to/nixos-meetup-3-slides.pdf and also – in a form ready for cut & paste – at https://github.com/NixOS/cabal2nix/blob/master/doc/nixos-meetup-3-slides.md.

  • Another Youtube video is Escaping Cabal Hell with Nix, which discusses the subject of Haskell development with Nix but also provides a basic introduction to Nix as well, i.e. it’s suitable for viewers with almost no prior Nix experience.

  • Oliver Charles wrote a very nice Tutorial how to develop Haskell packages with Nix.

  • The Journey into the Haskell NG infrastructure series of postings describe the new Haskell infrastructure in great detail:

    • Part 1 explains the differences between the old and the new code and gives instructions how to migrate to the new setup.

    • Part 2 looks in-depth at how to tweak and configure your setup by means of overrides.

    • Part 3 describes the infrastructure that keeps the Haskell package set in Nixpkgs up-to-date.

10.6. Idris packages

This directory contains build rules for idris packages. In addition, it contains several functions to build and compose those packages. Everything is exposed to the user via the idrisPackages attribute.

10.6.1. callPackage

This is like the normal nixpkgs callPackage function, specialized to idris packages.

10.6.2. builtins

This is a list of all of the libraries that come packaged with Idris itself.

10.6.3. build-idris-package

A function to build an idris package. Its sole argument is a set like you might pass to stdenv.mkDerivation, except build-idris-package sets several attributes for you. See build-idris-package.nix for details.

10.6.4. build-builtin-package

A version of build-idris-package specialized to builtin libraries. Mostly for internal use.

10.6.5. with-packages

Bundle idris together with a list of packages. Because idris currently only supports a single directory in its library path, you must include all desired libraries here, including prelude and base.

10.7. Java

Ant-based Java packages are typically built from source as follows:

stdenv.mkDerivation {
  name = "...";
  src = fetchurl { ... };

  buildInputs = [ jdk ant ];

  buildPhase = "ant";
}

Note that jdk is an alias for the OpenJDK.

JAR files that are intended to be used by other packages should be installed in $out/share/java. The OpenJDK has a stdenv setup hook that adds any JARs in the share/java directories of the build inputs to the CLASSPATH environment variable. For instance, if the package libfoo installs a JAR named foo.jar in its share/java directory, and another package declares the attribute

buildInputs = [ jdk libfoo ];

then CLASSPATH will be set to /nix/store/...-libfoo/share/java/foo.jar.

Private JARs should be installed in a location like $out/share/package-name.

If your Java package provides a program, you need to generate a wrapper script to run it using the OpenJRE. You can use makeWrapper for this:

buildInputs = [ makeWrapper ];

installPhase =
  ''
    mkdir -p $out/bin
    makeWrapper ${jre}/bin/java $out/bin/foo \
      --add-flags "-cp $out/share/java/foo.jar org.foo.Main"
  '';

Note the use of jre, which is the part of the OpenJDK package that contains the Java Runtime Environment. By using ${jre}/bin/java instead of ${jdk}/bin/java, you prevent your package from depending on the JDK at runtime.

It is possible to use a different Java compiler than javac from the OpenJDK. For instance, to use the Eclipse Java Compiler:

buildInputs = [ jre ant ecj ];

(Note that here you don’t need the full JDK as an input, but just the JRE.) The ECJ has a stdenv setup hook that sets some environment variables to cause Ant to use ECJ, but this doesn’t work with all Ant files. Similarly, you can use the GNU Java Compiler:

buildInputs = [ gcj ant ];

Here, Ant will automatically use gij (the GNU Java Runtime) instead of the OpenJRE.

10.8. Lua

Lua packages are built by the buildLuaPackage function. This function is implemented in pkgs/development/lua-modules/generic/default.nix and works similarly to buildPerlPackage. (See Section 10.10, “Perl” for details.)

Lua packages are defined in pkgs/top-level/lua-packages.nix. Most of them are simple. For example:

fileSystem = buildLuaPackage {
  name = "filesystem-1.6.2";
  src = fetchurl {
    url = "https://github.com/keplerproject/luafilesystem/archive/v1_6_2.tar.gz";
    sha256 = "1n8qdwa20ypbrny99vhkmx8q04zd2jjycdb5196xdhgvqzk10abz";
  };
  meta = {
    homepage = "https://github.com/keplerproject/luafilesystem";
    hydraPlatforms = stdenv.lib.platforms.linux;
    maintainers = with maintainers; [ flosse ];
  };
};
  

Though, more complicated package should be placed in a seperate file in pkgs/development/lua-modules.

Lua packages accept additional parameter disabled, which defines the condition of disabling package from luaPackages. For example, if package has disabled assigned to lua.luaversion != "5.1", it will not be included in any luaPackages except lua51Packages, making it only be built for lua 5.1.

10.9. Node.js packages

To add a package from NPM to nixpkgs:

  1. Install node2nix: nix-env -f '<nixpkgs>' -iA node2nix.

  2. Modify pkgs/development/node-packages/node-packages.json, to add, update, or remove package entries.

  3. Run the script: cd pkgs/development/node-packages && sh generate.sh.

  4. Build your new package to test your changes: cd /path/to/nixpkgs && nix-build -A nodePackages.<new-or-updated-package>. To build against a specific node.js version (e.g. 5.x): nix-build -A nodePackages_5_x.<new-or-updated-package>

  5. Add, commit, and share your changes!

10.10. Perl

Nixpkgs provides a function buildPerlPackage, a generic package builder function for any Perl package that has a standard Makefile.PL. It’s implemented in pkgs/development/perl-modules/generic.

Perl packages from CPAN are defined in pkgs/top-level/perl-packages.nix, rather than pkgs/all-packages.nix. Most Perl packages are so straight-forward to build that they are defined here directly, rather than having a separate function for each package called from perl-packages.nix. However, more complicated packages should be put in a separate file, typically in pkgs/development/perl-modules. Here is an example of the former:

ClassC3 = buildPerlPackage rec {
  name = "Class-C3-0.21";
  src = fetchurl {
    url = "mirror://cpan/authors/id/F/FL/FLORA/${name}.tar.gz";
    sha256 = "1bl8z095y4js66pwxnm7s853pi9czala4sqc743fdlnk27kq94gz";
  };
};

Note the use of mirror://cpan/, and the ${name} in the URL definition to ensure that the name attribute is consistent with the source that we’re actually downloading. Perl packages are made available in all-packages.nix through the variable perlPackages. For instance, if you have a package that needs ClassC3, you would typically write

foo = import ../path/to/foo.nix {
  inherit stdenv fetchurl ...;
  inherit (perlPackages) ClassC3;
};

in all-packages.nix. You can test building a Perl package as follows:

$ nix-build -A perlPackages.ClassC3

buildPerlPackage adds perl- to the start of the name attribute, so the package above is actually called perl-Class-C3-0.21. So to install it, you can say:

$ nix-env -i perl-Class-C3

(Of course you can also install using the attribute name: nix-env -i -A perlPackages.ClassC3.)

So what does buildPerlPackage do? It does the following:

  1. In the configure phase, it calls perl Makefile.PL to generate a Makefile. You can set the variable makeMakerFlags to pass flags to Makefile.PL

  2. It adds the contents of the PERL5LIB environment variable to #! .../bin/perl line of Perl scripts as -Idir flags. This ensures that a script can find its dependencies.

  3. In the fixup phase, it writes the propagated build inputs (propagatedBuildInputs) to the file $out/nix-support/propagated-user-env-packages. nix-env recursively installs all packages listed in this file when you install a package that has it. This ensures that a Perl package can find its dependencies.

buildPerlPackage is built on top of stdenv, so everything can be customised in the usual way. For instance, the BerkeleyDB module has a preConfigure hook to generate a configuration file used by Makefile.PL:

{ buildPerlPackage, fetchurl, db }:

buildPerlPackage rec {
  name = "BerkeleyDB-0.36";

  src = fetchurl {
    url = "mirror://cpan/authors/id/P/PM/PMQS/${name}.tar.gz";
    sha256 = "07xf50riarb60l1h6m2dqmql8q5dij619712fsgw7ach04d8g3z1";
  };

  preConfigure = ''
    echo "LIB = ${db}/lib" > config.in
    echo "INCLUDE = ${db}/include" >> config.in
  '';
}

Dependencies on other Perl packages can be specified in the buildInputs and propagatedBuildInputs attributes. If something is exclusively a build-time dependency, use buildInputs; if it’s (also) a runtime dependency, use propagatedBuildInputs. For instance, this builds a Perl module that has runtime dependencies on a bunch of other modules:

ClassC3Componentised = buildPerlPackage rec {
  name = "Class-C3-Componentised-1.0004";
  src = fetchurl {
    url = "mirror://cpan/authors/id/A/AS/ASH/${name}.tar.gz";
    sha256 = "0xql73jkcdbq4q9m0b0rnca6nrlvf5hyzy8is0crdk65bynvs8q1";
  };
  propagatedBuildInputs = [
    ClassC3 ClassInspector TestException MROCompat
  ];
};

10.10.1. Generation from CPAN

Nix expressions for Perl packages can be generated (almost) automatically from CPAN. This is done by the program nix-generate-from-cpan, which can be installed as follows:

$ nix-env -i nix-generate-from-cpan

This program takes a Perl module name, looks it up on CPAN, fetches and unpacks the corresponding package, and prints a Nix expression on standard output. For example:

$ nix-generate-from-cpan XML::Simple
  XMLSimple = buildPerlPackage rec {
    name = "XML-Simple-2.22";
    src = fetchurl {
      url = "mirror://cpan/authors/id/G/GR/GRANTM/${name}.tar.gz";
      sha256 = "b9450ef22ea9644ae5d6ada086dc4300fa105be050a2030ebd4efd28c198eb49";
    };
    propagatedBuildInputs = [ XMLNamespaceSupport XMLSAX XMLSAXExpat ];
    meta = {
      description = "An API for simple XML files";
      license = with stdenv.lib.licenses; [ artistic1 gpl1Plus ];
    };
  };

The output can be pasted into pkgs/top-level/perl-packages.nix or wherever else you need it.

10.11. Python

10.11.1. User Guide

Several versions of Python are available on Nix as well as a high amount of packages. The default interpreter is CPython 3.5.

10.11.1.1. Using Python

10.11.1.1.1. Installing Python and packages

It is important to make a distinction between Python packages that are used as libraries, and applications that are written in Python.

Applications on Nix are installed typically into your user profile imperatively using nix-env -i, and on NixOS declaratively by adding the package name to environment.systemPackages in /etc/nixos/configuration.nix. Dependencies such as libraries are automatically installed and should not be installed explicitly.

The same goes for Python applications and libraries. Python applications can be installed in your profile, but Python libraries you would like to use to develop cannot. If you do install libraries in your profile, then you will end up with import errors.

10.11.1.1.2. Python environments using nix-shell

The recommended method for creating Python environments for development is with nix-shell. Executing

$ nix-shell -p python35Packages.numpy python35Packages.toolz

opens a Nix shell which has available the requested packages and dependencies. Now you can launch the Python interpreter (which is itself a dependency)

[nix-shell:~] python3

If the packages were not available yet in the Nix store, Nix would download or build them automatically. A convenient option with nix-shell is the --run option, with which you can execute a command in the nix-shell. Let’s say we want the above environment and directly run the Python interpreter

$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"

This way you can use the --run option also to directly run a script

$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"

In fact, for this specific use case there is a more convenient method. You can add a shebang to your script specifying which dependencies Nix shell needs. With the following shebang, you can use nix-shell myscript.py and it will make available all dependencies and run the script in the python3 shell.

#! /usr/bin/env nix-shell
#! nix-shell -i python3 -p python3Packages.numpy

import numpy

print(numpy.__version__)

Likely you do not want to type your dependencies each and every time. What you can do is write a simple Nix expression which sets up an environment for you, requiring you only to type nix-shell. Say we want to have Python 3.5, numpy and toolz, like before, in an environment. With a shell.nix file containing

with import <nixpkgs> {};

(pkgs.python35.withPackages (ps: [ps.numpy ps.toolz])).env

executing nix-shell gives you again a Nix shell from which you can run Python.

What’s happening here?

  1. We begin with importing the Nix Packages collections. import <nixpkgs> import the <nixpkgs> function, {} calls it and the with statement brings all attributes of nixpkgs in the local scope. Therefore we can now use pkgs.

  2. Then we create a Python 3.5 environment with the withPackages function.

  3. The withPackages function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages numpy and toolz from the package set.

  4. And finally, for in interactive use we return the environment by using the env attribute.

10.11.1.2. Developing with Python

Now that you know how to get a working Python environment on Nix, it is time to go forward and start actually developing with Python. We will first have a look at how Python packages are packaged on Nix. Then, we will look how you can use development mode with your code.

10.11.1.2.1. Python packaging on Nix

On Nix all packages are built by functions. The main function in Nix for building Python packages is buildPythonPackage. Let’s see how we would build the toolz package. According to python-packages.nix toolz is build using

toolz = buildPythonPackage rec{
  name = "toolz-${version}";
  version = "0.7.4";

  src = pkgs.fetchurl{
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
};

What happens here? The function buildPythonPackage is called and as argument it accepts a set. In this case the set is a recursive set (rec). One of the arguments is the name of the package, which consists of a basename (generally following the name on PyPi) and a version. Another argument, src specifies the source, which in this case is fetched from an url. fetchurl not only downloads the target file, but also validates its hash. Furthermore, we specify some (optional) meta information.

The output of the function is a derivation, which is an attribute with the name toolz of the set pythonPackages. Actually, sets are created for all interpreter versions, so python27Packages, python34Packages, python35Packages and pypyPackages.

The above example works when you’re directly working on pkgs/top-level/python-packages.nix in the Nixpkgs repository. Often though, you will want to test a Nix expression outside of the Nixpkgs tree. If you create a shell.nix file with the following contents

with import <nixpkgs> {};

pkgs.python35Packages.buildPythonPackage rec {
  name = "toolz-${version}";
  version = "0.8.0";

  src = pkgs.fetchurl{
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "e8451af61face57b7c5d09e71c0d27b8005f001ead56e9fdf470417e5cc6d479";
  };

  doCheck = false;

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
}

and then execute nix-shell will result in an environment in which you can use Python 3.5 and the toolz package. As you can see we had to explicitly mention for which Python version we want to build a package.

The above example considered only a single package. Generally you will want to use multiple packages. If we create a shell.nix file with the following contents

with import <nixpkgs> {};

( let
    toolz = pkgs.python35Packages.buildPythonPackage rec {
      name = "toolz-${version}";
      version = "0.8.0";

      src = pkgs.fetchurl{
        url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
        sha256 = "e8451af61face57b7c5d09e71c0d27b8005f001ead56e9fdf470417e5cc6d479";
      };

      doCheck = false;

      meta = {
        homepage = "http://github.com/pytoolz/toolz/";
        description = "List processing tools and functional utilities";
      };
    };

  in pkgs.python35.withPackages (ps: [ps.numpy toolz])
).env

and again execute nix-shell, then we get a Python 3.5 environment with our locally defined package as well as numpy which is build according to the definition in Nixpkgs. What did we do here? Well, we took the Nix expression that we used earlier to build a Python environment, and said that we wanted to include our own version of toolz. To introduce our own package in the scope of withPackages we used a let expression. You can see that we used ps.numpy to select numpy from the nixpkgs package set (ps). But we do not take toolz from the nixpkgs package set this time. Instead, toolz will resolve to our local definition that we introduced with let.

10.11.1.3. Handling dependencies

Our example, toolz, doesn’t have any dependencies on other Python packages or system libraries. According to the manual, buildPythonPackage uses the arguments buildInputs and propagatedBuildInputs to specify dependencies. If something is exclusively a build-time dependency, then the dependency should be included as a buildInput, but if it is (also) a runtime dependency, then it should be added to propagatedBuildInputs. Test dependencies are considered build-time dependencies.

The following example shows which arguments are given to buildPythonPackage in order to build datashape.

datashape = buildPythonPackage rec {
  name = "datashape-${version}";
  version = "0.4.7";

  src = pkgs.fetchurl {
    url = "mirror://pypi/D/DataShape/${name}.tar.gz";
    sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
  };

  buildInputs = with self; [ pytest ];
  propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ];

  meta = {
    homepage = https://github.com/ContinuumIO/datashape;
    description = "A data description language";
    license = licenses.bsd2;
    maintainers = with maintainers; [ fridh ];
  };
};

We can see several runtime dependencies, numpy, multipledispatch, and dateutil. Furthermore, we have one buildInput, i.e. pytest. pytest is a test runner and is only used during the checkPhase and is therefore not added to propagatedBuildInputs.

In the previous case we had only dependencies on other Python packages to consider. Occasionally you have also system libraries to consider. E.g., lxml provides Python bindings to libxml2 and libxslt. These libraries are only required when building the bindings and are therefore added as buildInputs.

lxml = buildPythonPackage rec {
  name = "lxml-3.4.4";

  src = pkgs.fetchurl {
    url = "mirror://pypi/l/lxml/${name}.tar.gz";
    sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
  };

  buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ];

  meta = {
    description = "Pythonic binding for the libxml2 and libxslt libraries";
    homepage = http://lxml.de;
    license = licenses.bsd3;
    maintainers = with maintainers; [ sjourdois ];
  };
};

In this example lxml and Nix are able to work out exactly where the relevant files of the dependencies are. This is not always the case.

The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as FFTW. On Nix we have separate packages of FFTW for the different types of floats ("single", "double", "long-double"). The bindings need all three types, and therefore we add all three as buildInputs. The bindings don’t expect to find each of them in a different folder, and therefore we have to set LDFLAGS and CFLAGS.

pyfftw = buildPythonPackage rec {
  name = "pyfftw-${version}";
  version = "0.9.2";

  src = pkgs.fetchurl {
    url = "mirror://pypi/p/pyFFTW/pyFFTW-${version}.tar.gz";
    sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
  };

  buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];

  propagatedBuildInputs = with self; [ numpy scipy ];

  # Tests cannot import pyfftw. pyfftw works fine though.
  doCheck = false;

  LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
  CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
  '';

  meta = {
    description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
    homepage = http://hgomersall.github.com/pyFFTW/;
    license = with licenses; [ bsd2 bsd3 ];
    maintainer = with maintainers; [ fridh ];
  };
};

Note also the line doCheck = false;, we explicitly disabled running the test-suite.

10.11.1.3.1. Develop local package

As a Python developer you’re likely aware of development mode (python setup.py develop); instead of installing the package this command creates a special link to the project code. That way, you can run updated code without having to reinstall after each and every change you make. Development mode is also available. Let’s see how you can use it.

In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using

src = ./path/to/source/tree;

If we create a shell.nix file which calls buildPythonPackage, and if src is a local source, and if the local source has a setup.py, then development mode is activated.

In the following example we create a simple environment that has a Python 3.5 version of our package in it, as well as its dependencies and other packages we like to have in the environment, all specified with propagatedBuildInputs. Indeed, we can just add any package we like to have in our environment to propagatedBuildInputs.

with import <nixpkgs>;
with pkgs.python35Packages;

buildPythonPackage rec {
  name = "mypackage";
  src = ./path/to/package/source;
  propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
};

It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode.

10.11.1.4. Organising your packages

So far we discussed how you can use Python on Nix, and how you can develop with it. We’ve looked at how you write expressions to package Python packages, and we looked at how you can create environments in which specified packages are available.

At some point you’ll likely have multiple packages which you would like to be able to use in different projects. In order to minimise unnecessary duplication we now look at how you can maintain yourself a repository with your own packages. The important functions here are import and callPackage.

10.11.1.5. Including a derivation using callPackage

Earlier we created a Python environment using withPackages, and included the toolz package via a let expression. Let’s split the package definition from the environment definition.

We first create a function that builds toolz in ~/path/to/toolz/release.nix

{ pkgs, buildPythonPackage }:

buildPythonPackage rec {
  name = "toolz-${version}";
  version = "0.7.4";

  src = pkgs.fetchurl{
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
};

It takes two arguments, pkgs and buildPythonPackage. We now call this function using callPackage in the definition of our environment

with import <nixpkgs> {};

( let
    toolz = pkgs.callPackage ~/path/to/toolz/release.nix { pkgs=pkgs; buildPythonPackage=pkgs.python35Packages.buildPythonPackage; };
  in pkgs.python35.withPackages (ps: [ ps.numpy toolz ])
).env

Important to remember is that the Python version for which the package is made depends on the python derivation that is passed to buildPythonPackage. Nix tries to automatically pass arguments when possible, which is why generally you don’t explicitly define which python derivation should be used. In the above example we use buildPythonPackage that is part of the set python35Packages, and in this case the python35 interpreter is automatically used.

10.11.2. Reference

10.11.2.1. Interpreters

Versions 2.6, 2.7, 3.3, 3.4 and 3.5 of the CPython interpreter are available as respectively python26, python27, python33, python34 and python35. The PyPy interpreter is available as pypy. The aliases python2 and python3 correspond to respectively python27 and python35. The default interpreter, python, maps to python2. The Nix expressions for the interpreters can be found in pkgs/development/interpreters/python.

All packages depending on any Python interpreter get appended out/{python.sitePackages} to $PYTHONPATH if such directory exists.

10.11.2.1.1. Missing tkinter module standard library

To reduce closure size the Tkinter/tkinter is available as a separate package, pythonPackages.tkinter.

10.11.2.1.2. Attributes on interpreters packages

Each interpreter has the following attributes:

  • libPrefix. Name of the folder in ${python}/lib/ for corresponding interpreter.

  • interpreter. Alias for ${python}/bin/${executable}.

  • buildEnv. Function to build python interpreter environments with extra packages bundled together. See section python.buildEnv function for usage and documentation.

  • withPackages. Simpler interface to buildEnv. See section python.withPackages function for usage and documentation.

  • sitePackages. Alias for lib/${libPrefix}/site-packages.

  • executable. Name of the interpreter executable, e.g. python3.4.

  • pkgs. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing packageOverrides.

10.11.2.2. Building packages and applications

Python libraries and applications that use setuptools or distutils are typically build with respectively the buildPythonPackage and buildPythonApplication functions. These two functions also support installing a wheel.

All Python packages reside in pkgs/top-level/python-packages.nix and all applications elsewhere. In case a package is used as both a library and an application, then the package should be in pkgs/top-level/python-packages.nix since only those packages are made available for all interpreter versions. The preferred location for library expressions is in pkgs/development/python-modules. It is important that these packages are called from pkgs/top-level/python-packages.nix and not elsewhere, to guarantee the right version of the package is built.

Based on the packages defined in pkgs/top-level/python-packages.nix an attribute set is created for each available Python interpreter. The available sets are

  • pkgs.python26Packages

  • pkgs.python27Packages

  • pkgs.python33Packages

  • pkgs.python34Packages

  • pkgs.python35Packages

  • pkgs.pypyPackages

and the aliases

  • pkgs.python2Packages pointing to pkgs.python27Packages

  • pkgs.python3Packages pointing to pkgs.python35Packages

  • pkgs.pythonPackages pointing to pkgs.python2Packages

10.11.2.2.1. buildPythonPackage function

The buildPythonPackage function is implemented in pkgs/development/interpreters/python/build-python-package.nix

The following is an example:

twisted = buildPythonPackage {
  name = "twisted-8.1.0";

  src = pkgs.fetchurl {
    url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2;
    sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl";
  };

  propagatedBuildInputs = [ self.ZopeInterface ];

  meta = {
    homepage = http://twistedmatrix.com/;
    description = "Twisted, an event-driven networking engine written in Python";
    license = stdenv.lib.licenses.mit; };
  };

The buildPythonPackage mainly does four things:

  • In the buildPhase, it calls ${python.interpreter} setup.py bdist_wheel to build a wheel binary zipfile.

  • In the installPhase, it installs the wheel file using pip install *.whl.

  • In the postFixup phase, the wrapPythonPrograms bash function is called to wrap all programs in the $out/bin/* directory to include $PATH environment variable and add dependent libraries to script’s sys.path.

  • In the installCheck phase, ${python.interpreter} setup.py test is ran.

As in Perl, dependencies on other Python packages can be specified in the buildInputs and propagatedBuildInputs attributes. If something is exclusively a build-time dependency, use buildInputs; if it’s (also) a runtime dependency, use propagatedBuildInputs.

By default tests are run because doCheck = true. Test dependencies, like e.g. the test runner, should be added to buildInputs.

By default meta.platforms is set to the same value as the interpreter unless overriden otherwise.

10.11.2.2.1.1. buildPythonPackage parameters

All parameters from mkDerivation function are still supported.

  • namePrefix: Prepended text to ${name} parameter. Defaults to "python3.3-" for Python 3.3, etc. Set it to "" if you’re packaging an application or a command line tool.

  • disabled: If true, package is not build for particular python interpreter version. Grep around pkgs/top-level/python-packages.nix for examples.

  • setupPyBuildFlags: List of flags passed to setup.py build_ext command.

  • pythonPath: List of packages to be added into $PYTHONPATH. Packages in pythonPath are not propagated (contrary to propagatedBuildInputs).

  • preShellHook: Hook to execute commands before shellHook.

  • postShellHook: Hook to execute commands after shellHook.

  • makeWrapperArgs: A list of strings. Arguments to be passed to makeWrapper, which wraps generated binaries. By default, the arguments to makeWrapper set PATH and PYTHONPATH environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"].

  • installFlags: A list of strings. Arguments to be passed to pip install. To pass options to python setup.py install, use --install-option. E.g., `installFlags=[–install-option=–cpp_implementation].

  • format: Format of the source. Valid options are setuptools (default), flit, wheel, and other. setuptools is for when the source has a setup.py and setuptools is used to build a wheel, flit, in case flit should be used to build a wheel, and wheel in case a wheel is provided. In case you need to provide your own buildPhase and installPhase you can use other.

  • catchConflicts If true, abort package build if a package name appears more than once in dependency tree. Default is true.

  • checkInputs Dependencies needed for running the checkPhase. These are added to buildInputs when doCheck = true.

10.11.2.2.2. buildPythonApplication function

The buildPythonApplication function is practically the same as buildPythonPackage. The difference is that buildPythonPackage by default prefixes the names of the packages with the version of the interpreter. Because with an application we’re not interested in multiple version the prefix is dropped.

10.11.2.2.3. python.buildEnv function

Python environments can be created using the low-level pkgs.buildEnv function. This example shows how to create an environment that has the Pyramid Web Framework. Saving the following as default.nix

with import <nixpkgs> {};

python.buildEnv.override {
  extraLibs = [ pkgs.pythonPackages.pyramid ];
  ignoreCollisions = true;
}

and running nix-build will create

/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env

with wrapped binaries in bin/.

You can also use the env attribute to create local environments with needed packages installed. This is somewhat comparable to virtualenv. For example, running nix-shell with the following shell.nix

with import <nixpkgs> {};

(python3.buildEnv.override {
  extraLibs = with python3Packages; [ numpy requests2 ];
}).env

will drop you into a shell where Python will have the specified packages in its path.

10.11.2.2.3.1. python.buildEnv arguments
  • extraLibs: List of packages installed inside the environment.

  • postBuild: Shell command executed after the build of environment.

  • ignoreCollisions: Ignore file collisions inside the environment (default is false).

10.11.2.2.4. python.withPackages function

The python.withPackages function provides a simpler interface to the python.buildEnv functionality. It takes a function as an argument that is passed the set of python packages and returns the list of the packages to be included in the environment. Using the withPackages function, the previous example for the Pyramid Web Framework environment can be written like this:

with import <nixpkgs> {};

python.withPackages (ps: [ps.pyramid])

withPackages passes the correct package set for the specific interpreter version as an argument to the function. In the above example, ps equals pythonPackages. But you can also easily switch to using python3:

with import <nixpkgs> {};

python3.withPackages (ps: [ps.pyramid])

Now, ps is set to python3Packages, matching the version of the interpreter.

As python.withPackages simply uses python.buildEnv under the hood, it also supports the env attribute. The shell.nix file from the previous section can thus be also written like this:

with import <nixpkgs> {};

(python33.withPackages (ps: [ps.numpy ps.requests2])).env

In contrast to python.buildEnv, python.withPackages does not support the more advanced options such as ignoreCollisions = true or postBuild. If you need them, you have to use python.buildEnv.

10.11.2.3. Development mode

Development or editable mode is supported. To develop Python packages buildPythonPackage has additional logic inside shellPhase to run pip install -e . --prefix $TMPDIR/for the package.

Warning: shellPhase is executed only if setup.py exists.

Given a default.nix:

with import <nixpkgs> {};

buildPythonPackage { name = "myproject";

buildInputs = with pkgs.pythonPackages; [ pyramid ];

src = ./.; }

Running nix-shell with no arguments should give you the environment in which the package would be built with nix-build.

Shortcut to setup environments with C headers/libraries and python packages:

$ nix-shell -p pythonPackages.pyramid zlib libjpeg git

Note: There is a boolean value lib.inNixShell set to true if nix-shell is invoked.

10.11.2.4. Tools

Packages inside nixpkgs are written by hand. However many tools exist in community to help save time. No tool is preferred at the moment.

10.11.3. FAQ

10.11.3.1. How can I install a working Python environment?

As explained in the user’s guide installing individual Python packages imperatively with nix-env -i or declaratively in environment.systemPackages is not supported. However, it is possible to install a Python environment with packages (python.buildEnv).

In the following examples we create an environment with Python 3.5, numpy and ipython. As you might imagine there is one limitation here, and that’s you can install only one environment at a time. You will notice the complaints about collisions when you try to install a second environment.

10.11.3.1.1. Environment defined in separate .nix file

Create a file, e.g. build.nix, with the following expression

with import <nixpkgs> {};

pkgs.python35.withPackages (ps: with ps; [ numpy ipython ])

and install it in your profile with

nix-env -if build.nix

Now you can use the Python interpreter, as well as the extra packages that you added to the environment.

10.11.3.1.2. Environment defined in ~/.nixpkgs/config.nix

If you prefer to, you could also add the environment as a package override to the Nixpkgs set.

  packageOverrides = pkgs: with pkgs; {
    myEnv = python35.withPackages (ps: with ps; [ numpy ipython ]);
  };

and install it in your profile with

nix-env -iA nixpkgs.myEnv

We’re installing using the attribute path and assume the channels is named nixpkgs. Note that I’m using the attribute path here.

10.11.3.1.3. Environment defined in /etc/nixos/configuration.nix

For the sake of completeness, here’s another example how to install the environment system-wide.

environment.systemPackages = with pkgs; [
  (python35.withPackages(ps: with ps; [ numpy ipython ]))
];

10.11.3.2. How to solve circular dependencies?

Consider the packages A and B that depend on each other. When packaging B, a solution is to override package A not to depend on B as an input. The same should also be done when packaging A.

10.11.3.3. How to override a Python package?

We can override the interpreter and pass packageOverrides. In the following example we rename the pandas package and build it.

with import <nixpkgs> {};

let
  python = let
    packageOverrides = self: super: {
      pandas = super.pandas.override {name="foo";};
    };
  in pkgs.python35.override {inherit packageOverrides;};

in python.pkgs.pandas

Using nix-build on this expression will build the package pandas but with the new name foo.

All packages in the package set will use the renamed package. A typical use case is to switch to another version of a certain package. For example, in the Nixpkgs repository we have multiple versions of django and scipy. In the following example we use a different version of scipy and create an environment that uses it. All packages in the Python package set will now use the updated scipy version.

with import <nixpkgs> {};

(
let
  packageOverrides = self: super: {
    scipy = super.scipy_0_17;
  };
in (pkgs.python35.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
).env

The requested package blaze depends on pandas which itself depends on scipy.

If you want the whole of Nixpkgs to use your modifications, then you can use overlays as explained in this manual. In the following example we build a inkscape using a different version of numpy.

let
  pkgs = import <nixpkgs> {};
  newpkgs = import pkgs.path { overlays = [ (pkgsself: pkgssuper: {
    python27 = let
      packageOverrides = self: super: {
        numpy = super.numpy_1_10;
      };
    in pkgssuper.python27.override {inherit packageOverrides;};
  } ) ]; };
in newpkgs.inkscape

10.11.3.4. python setup.py bdist_wheel cannot create .whl

Executing python setup.py bdist_wheel in a nix-shellfails with

ValueError: ZIP does not support timestamps before 1980

This is because files are included that depend on items in the Nix store which have a timestamp of, that is, it corresponds to January the 1st, 1970 at 00:00:00. And as the error informs you, ZIP does not support that. The command bdist_wheel takes into account SOURCE_DATE_EPOCH, and nix-shell sets this to 1. By setting it to a value corresponding to 1980 or later, or by unsetting it, it is possible to build wheels.

Use 1980 as timestamp:

nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"

or the current time:

nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"

or unset:

nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"

10.11.3.5. install_data / data_files problems

If you get the following error:

could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
Permission denied

This is a known bug in setuptools. Setuptools install_data does not respect --prefix. An example of such package using the feature is pkgs/tools/X11/xpra/default.nix. As workaround install it as an extra preInstall step:

${python.interpreter} setup.py install_data --install-dir=$out --root=$out
sed -i '/ = data\_files/d' setup.py

10.11.3.6. Rationale of non-existent global site-packages

On most operating systems a global site-packages is maintained. This however becomes problematic if you want to run multiple Python versions or have multiple versions of certain libraries for your projects. Generally, you would solve such issues by creating virtual environments using virtualenv.

On Nix each package has an isolated dependency tree which, in the case of Python, guarantees the right versions of the interpreter and libraries or packages are available. There is therefore no need to maintain a global site-packages.

If you want to create a Python environment for development, then the recommended method is to use nix-shell, either with or without the python.buildEnv function.

10.11.3.7. How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?

This is an example of a default.nix for a nix-shell, which allows to consume a virtualenv environment, and install python modules through pip the traditional way.

Create this default.nix file, together with a requirements.txt and simply execute nix-shell.

with import <nixpkgs> {};
with pkgs.python27Packages;

stdenv.mkDerivation { 
  name = "impurePythonEnv";
  buildInputs = [
    # these packages are required for virtualenv and pip to work:
    #
    python27Full
    python27Packages.virtualenv
    python27Packages.pip
    # the following packages are related to the dependencies of your python 
    # project. 
    # In this particular example the python modules listed in the 
    # requirements.tx require the following packages to be installed locally 
    # in order to compile any binary extensions they may require.
    #
    taglib
    openssl
    git
    libxml2
    libxslt
    libzip
    stdenv
    zlib ];
  src = null;
  shellHook = ''
  # set SOURCE_DATE_EPOCH so that we can use python wheels
  SOURCE_DATE_EPOCH=$(date +%s)
  virtualenv --no-setuptools venv 
  export PATH=$PWD/venv/bin:$PATH
  pip install -r requirements.txt
  '';
}

Note that the pip install is an imperative action. So every time nix-shell is executed it will attempt to download the python modules listed in requirements.txt. However these will be cached locally within the virtualenv folder and not downloaded again.

10.11.4. Contributing

10.11.4.1. Contributing guidelines

Following rules are desired to be respected:

  • Python libraries are supposed to be called from python-packages.nix and packaged with buildPythonPackage. The expression of a library should be in pkgs/development/python-modules/<name>/default.nix. Libraries in pkgs/top-level/python-packages.nix are sorted quasi-alphabetically to avoid merge conflicts.

  • Python applications live outside of python-packages.nix and are packaged with buildPythonApplication.

  • Make sure libraries build for all Python interpreters.

  • By default we enable tests. Make sure the tests are found and, in the case of libraries, are passing for all interpreters. If certain tests fail they can be disabled individually. Try to avoid disabling the tests altogether. In any case, when you disable tests, leave a comment explaining why.

  • Commit names of Python libraries should include pythonPackages, for example pythonPackages.numpy: 1.11 -> 1.12.

10.12. Qt

The information in this section applies to Qt 5.5 and later.

Qt is an application development toolkit for C++. Although it is not a distinct programming language, there are special considerations for packaging Qt-based programs and libraries. A small set of tools and conventions has grown out of these considerations.

10.12.1. Libraries

Packages that provide libraries should be listed in qt5LibsFun so that the library is built with each Qt version. A set of packages is provided for each version of Qt; for example, qt5Libs always provides libraries built with the latest version, qt55Libs provides libraries built with Qt 5.5, and so on. To avoid version conflicts, no top-level attributes are created for these packages.

10.12.2. Programs

Application packages do not need to be built with every Qt version. To ensure consistency between the package's dependencies, call the package with qt5Libs.callPackage instead of the usual callPackage. An older version may be selected in case of incompatibility. For example, to build with Qt 5.5, call the package with qt55Libs.callPackage.

Several environment variables must be set at runtime for Qt applications to function correctly, including:

  • QT_PLUGIN_PATH

  • QML_IMPORT_PATH

  • QML2_IMPORT_PATH

  • XDG_DATA_DIRS

To ensure that these are set correctly, the program must be wrapped by invoking wrapQtProgram program during installation (for example, during fixupPhase). wrapQtProgram accepts the same options as makeWrapper.

10.12.3. KDE

Many of the considerations above also apply to KDE packages, especially the need to set the correct environment variables at runtime. To ensure that this is done, invoke wrapKDEProgram program during installation. wrapKDEProgram also generates a ksycoca database so that required data and services can be found. Like its Qt counterpart, wrapKDEProgram accepts the same options as makeWrapper.

10.13. R packages

10.13.1. Installation

Define an environment for R that contains all the libraries that you’d like to use by adding the following snippet to your $HOME/.config/nixpkgs/config.nix file:

{
    packageOverrides = super: let self = super.pkgs; in
    {

        rEnv = super.rWrapper.override {
            packages = with self.rPackages; [ 
                devtools
                ggplot2
                reshape2
                yaml
                optparse
                ];
        };
    };
}

Then you can use nix-env -f "<nixpkgs>" -iA rEnv to install it into your user profile. The set of available libraries can be discovered by running the command nix-env -f "<nixpkgs>" -qaP -A rPackages. The first column from that output is the name that has to be passed to rWrapper in the code snipped above.

However, if you’d like to add a file to your project source to make the environment available for other contributors, you can create a default.nix file like so:

let
  pkgs = import <nixpkgs> {};
  stdenv = pkgs.stdenv;
in with pkgs; {
  myProject = stdenv.mkDerivation {
    name = "myProject";
    version = "1";
    src = if pkgs.lib.inNixShell then null else nix;

    buildInputs = with rPackages; [
      R
      ggplot2
      knitr
    ];
  };
}

and then run nix-shell . to be dropped into a shell with those packages available.

10.13.2. RStudio

RStudio by default will not use the libraries installed like above. You must override its R version with your custom R environment, and set useRPackages to true, like below:

{
    packageOverrides = super: let self = super.pkgs; in
    {

        rEnv = super.rWrapper.override {
            packages = with self.rPackages; [ 
                devtools
                ggplot2
                reshape2
                yaml
                optparse
                ];
        };
        rstudioEnv = super.rstudio.override {
            R = rEnv;
            useRPackages = true;
        };
    };
}

Then like above, nix-env -f "<nixpkgs>" -iA rstudioEnv will install this into your user profile.

10.13.3. Updating the package set

Rscript generate-r-packages.R cran  > cran-packages.nix.new
mv cran-packages.nix.new cran-packages.nix

Rscript generate-r-packages.R bioc  > bioc-packages.nix.new
mv bioc-packages.nix.new bioc-packages.nix

Rscript generate-r-packages.R irkernel  > irkernel-packages.nix.new
mv irkernel-packages.nix.new irkernel-packages.nix

generate-r-packages.R <repo> reads <repo>-packages.nix, therefor the renaming.

10.13.4. Testing if the Nix-expression could be evaluated

nix-build test-evaluation.nix --dry-run

If this exits fine, the expression is ok. If not, you have to edit default.nix

10.14. Ruby

There currently is support to bundle applications that are packaged as Ruby gems. The utility "bundix" allows you to write a Gemfile, let bundler create a Gemfile.lock, and then convert this into a nix expression that contains all Gem dependencies automatically.

For example, to package sensu, we did:

$ cd pkgs/servers/monitoring
$ mkdir sensu
$ cd sensu
$ cat > Gemfile
source 'https://rubygems.org'
gem 'sensu'
$ nix-shell -p bundler --command "bundler package --path /tmp/vendor/bundle"
$ $(nix-build '<nixpkgs>' -A bundix)/bin/bundix
$ cat > default.nix
{ lib, bundlerEnv, ruby }:

bundlerEnv rec {
  name = "sensu-${version}";

  version = (import gemset).sensu.version;
  inherit ruby;
  # expects Gemfile, Gemfile.lock and gemset.nix in the same directory
  gemdir = ./.;

  meta = with lib; {
    description = "A monitoring framework that aims to be simple, malleable, and scalable";
    homepage    = http://sensuapp.org/;
    license     = with licenses; mit;
    maintainers = with maintainers; [ theuni ];
    platforms   = platforms.unix;
  };
}

Please check in the Gemfile, Gemfile.lock and the gemset.nix so future updates can be run easily.

Resulting derivations also have two helpful items, env and wrapper. The first one allows one to quickly drop into nix-shell with the specified environment present. E.g. nix-shell -A sensu.env would give you an environment with Ruby preset so it has all the libraries necessary for sensu in its paths. The second one can be used to make derivations from custom Ruby scripts which have Gemfiles with their dependencies specified. It is a derivation with ruby wrapped so it can find all the needed dependencies. For example, to make a derivation my-script for a my-script.rb (which should be placed in bin) you should run bundix as specified above and then use bundlerEnv lile this:

let env = bundlerEnv {
  name = "my-script-env";

  inherit ruby;
  gemfile = ./Gemfile;
  lockfile = ./Gemfile.lock;
  gemset = ./gemset.nix;
};

in stdenv.mkDerivation {
  name = "my-script";

  buildInputs = [ env.wrapper ];

  script = ./my-script.rb;

  buildCommand = ''
    mkdir -p $out/bin
    install -D -m755 $script $out/bin/my-script
    patchShebangs $out/bin/my-script
  '';
}

10.15. TeX Live

Since release 15.09 there is a new TeX Live packaging that lives entirely under attribute texlive.

10.15.1. User's guide

  • For basic usage just pull texlive.combined.scheme-basic for an environment with basic LaTeX support.

  • It typically won't work to use separately installed packages together. Instead, you can build a custom set of packages like this:

    texlive.combine {
      inherit (texlive) scheme-small collection-langkorean algorithms cm-super;
    }
          

    There are all the schemes, collections and a few thousand packages, as defined upstream (perhaps with tiny differences).

  • By default you only get executables and files needed during runtime, and a little documentation for the core packages. To change that, you need to add pkgFilter function to combine.

    texlive.combine {
      # inherit (texlive) whatever-you-want;
      pkgFilter = pkg:
        pkg.tlType == "run" || pkg.tlType == "bin" || pkg.pname == "cm-super";
      # elem tlType [ "run" "bin" "doc" "source" ]
      # there are also other attributes: version, name
    }
          

  • You can list packages e.g. by nix-repl.

    $ nix-repl
    nix-repl> :l <nixpkgs>
    nix-repl> texlive.collection-<TAB>
          

10.15.2. Known problems

  • Some tools are still missing, e.g. luajittex;

  • some apps aren't packaged/tested yet (asymptote, biber, etc.);

  • feature/bug: when a package is rejected by pkgFilter, its dependencies are still propagated;

  • in case of any bugs or feature requests, file a github issue or better a pull request and /cc @vcunat.

10.16. User’s Guide to Vim Plugins/Addons/Bundles/Scripts in Nixpkgs

You’ll get a vim(-your-suffix) in PATH also loading the plugins you want. Loading can be deferred; see examples.

VAM (=vim-addon-manager) and Pathogen plugin managers are supported. Vundle, NeoBundle could be your turn.

10.16.1. dependencies by Vim plugins

VAM introduced .json files supporting dependencies without versioning assuming that using latest version is ok most of the time.

10.16.2. HOWTO

First create a vim-scripts file having one plugin name per line. Example:

"tlib"
{'name': 'vim-addon-sql'}
{'filetype_regex': '\%(vim)$', 'names': ['reload', 'vim-dev-plugin']}

Such vim-scripts file can be read by VAM as well like this:

call vam#Scripts(expand('~/.vim-scripts'), {})

Create a default.nix file:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
nixpkgs.vim_configurable.customize { name = "vim"; vimrcConfig.vam.pluginDictionaries = [ "vim-addon-vim2nix" ]; }

Create a generate.vim file:

ActivateAddons vim-addon-vim2nix
let vim_scripts = "vim-scripts"
call nix#ExportPluginsForNix({
\  'path_to_nixpkgs': eval('{"'.substitute(substitute(substitute($NIX_PATH, ':', ',', 'g'), '=',':', 'g'), '\([:,]\)', '"\1"',"g").'"}')["nixpkgs"],
\  'cache_file': '/tmp/vim2nix-cache',
\  'try_catch': 0,
\  'plugin_dictionaries': ["vim-addon-manager"]+map(readfile(vim_scripts), 'eval(v:val)')
\ })

Then run

nix-shell -p vimUtils.vim_with_vim2nix --command "vim -c 'source generate.vim'"

You should get a Vim buffer with the nix derivations (output1) and vam.pluginDictionaries (output2). You can add your vim to your system’s configuration file like this and start it by vim-my:

my-vim =
 let plugins = let inherit (vimUtils) buildVimPluginFrom2Nix; in {
      copy paste output1 here
 }; in vim_configurable.customize {
   name = "vim-my";

   vimrcConfig.vam.knownPlugins = plugins; # optional
   vimrcConfig.vam.pluginDictionaries = [
      copy paste output2 here
   ];

   # Pathogen would be
   # vimrcConfig.pathogen.knownPlugins = plugins; # plugins
   # vimrcConfig.pathogen.pluginNames = ["tlib"];
 };

Sample output1:

"reload" = buildVimPluginFrom2Nix { # created by nix#NixDerivation
  name = "reload";
  src = fetchgit {
    url = "git://github.com/xolox/vim-reload";
    rev = "0a601a668727f5b675cb1ddc19f6861f3f7ab9e1";
    sha256 = "0vb832l9yxj919f5hfg6qj6bn9ni57gnjd3bj7zpq7d4iv2s4wdh";
  };
  dependencies = ["nim-misc"];

};
[...]

Sample output2:

[
  ''vim-addon-manager''
  ''tlib''
  { "name" = ''vim-addon-sql''; }
  { "filetype_regex" = ''\%(vim)$$''; "names" = [ ''reload'' ''vim-dev-plugin'' ]; }
]

10.16.3. Important repositories

  • vim-pi is a plugin repository from VAM plugin manager meant to be used by others as well used by

  • vim2nix which generates the .nix code

Chapter 11. Package Notes

This chapter contains information about how to use and maintain the Nix expressions for a number of specific packages, such as the Linux kernel or X.org.

11.1. Linux kernel

The Nix expressions to build the Linux kernel are in pkgs/os-specific/linux/kernel.

The function that builds the kernel has an argument kernelPatches which should be a list of {name, patch, extraConfig} attribute sets, where name is the name of the patch (which is included in the kernel’s meta.description attribute), patch is the patch itself (possibly compressed), and extraConfig (optional) is a string specifying extra options to be concatenated to the kernel configuration file (.config).

The kernel derivation exports an attribute features specifying whether optional functionality is or isn’t enabled. This is used in NixOS to implement kernel-specific behaviour. For instance, if the kernel has the iwlwifi feature (i.e. has built-in support for Intel wireless chipsets), then NixOS doesn’t have to build the external iwlwifi package:

modulesTree = [kernel]
  ++ pkgs.lib.optional (!kernel.features ? iwlwifi) kernelPackages.iwlwifi
  ++ ...;

How to add a new (major) version of the Linux kernel to Nixpkgs:

  1. Copy the old Nix expression (e.g. linux-2.6.21.nix) to the new one (e.g. linux-2.6.22.nix) and update it.

  2. Add the new kernel to all-packages.nix (e.g., create an attribute kernel_2_6_22).

  3. Now we’re going to update the kernel configuration. First unpack the kernel. Then for each supported platform (i686, x86_64, uml) do the following:

    1. Make an copy from the old config (e.g. config-2.6.21-i686-smp) to the new one (e.g. config-2.6.22-i686-smp).

    2. Copy the config file for this platform (e.g. config-2.6.22-i686-smp) to .config in the kernel source tree.

    3. Run make oldconfig ARCH={i386,x86_64,um} and answer all questions. (For the uml configuration, also add SHELL=bash.) Make sure to keep the configuration consistent between platforms (i.e. don’t enable some feature on i686 and disable it on x86_64).

    4. If needed you can also run make menuconfig:

      $ nix-env -i ncurses
      $ export NIX_CFLAGS_LINK=-lncurses
      $ make menuconfig ARCH=arch

    5. Copy .config over the new config file (e.g. config-2.6.22-i686-smp).

  4. Test building the kernel: nix-build -A kernel_2_6_22. If it compiles, ship it! For extra credit, try booting NixOS with it.

  5. It may be that the new kernel requires updating the external kernel modules and kernel-dependent packages listed in the linuxPackagesFor function in all-packages.nix (such as the NVIDIA drivers, AUFS, etc.). If the updated packages aren’t backwards compatible with older kernels, you may need to keep the older versions around.

11.2. X.org

The Nix expressions for the X.org packages reside in pkgs/servers/x11/xorg/default.nix. This file is automatically generated from lists of tarballs in an X.org release. As such it should not be modified directly; rather, you should modify the lists, the generator script or the file pkgs/servers/x11/xorg/overrides.nix, in which you can override or add to the derivations produced by the generator.

The generator is invoked as follows:

$ cd pkgs/servers/x11/xorg
$ cat tarballs-7.5.list extra.list old.list \
  | perl ./generate-expr-from-tarballs.pl

For each of the tarballs in the .list files, the script downloads it, unpacks it, and searches its configure.ac and *.pc.in files for dependencies. This information is used to generate default.nix. The generator caches downloaded tarballs between runs. Pay close attention to the NOT FOUND: name messages at the end of the run, since they may indicate missing dependencies. (Some might be optional dependencies, however.)

A file like tarballs-7.5.list contains all tarballs in a X.org release. It can be generated like this:

$ export i="mirror://xorg/X11R7.4/src/everything/"
$ cat $(PRINT_PATH=1 nix-prefetch-url $i | tail -n 1) \
  | perl -e 'while (<>) { if (/(href|HREF)="([^"]*.bz2)"/) { print "$ENV{'i'}$2\n"; }; }' \
  | sort > tarballs-7.4.list

extra.list contains libraries that aren’t part of X.org proper, but are closely related to it, such as libxcb. old.list contains some packages that were removed from X.org, but are still needed by some people or by other packages (such as imake).

If the expression for a package requires derivation attributes that the generator cannot figure out automatically (say, patches or a postInstall hook), you should modify pkgs/servers/x11/xorg/overrides.nix.

11.3. Eclipse

The Nix expressions related to the Eclipse platform and IDE are in pkgs/applications/editors/eclipse.

Nixpkgs provides a number of packages that will install Eclipse in its various forms, these range from the bare-bones Eclipse Platform to the more fully featured Eclipse SDK or Scala-IDE packages and multiple version are often available. It is possible to list available Eclipse packages by issuing the command:

$ nix-env -f '<nixpkgs>' -qaP -A eclipses --description

Once an Eclipse variant is installed it can be run using the eclipse command, as expected. From within Eclipse it is then possible to install plugins in the usual manner by either manually specifying an Eclipse update site or by installing the Marketplace Client plugin and using it to discover and install other plugins. This installation method provides an Eclipse installation that closely resemble a manually installed Eclipse.

If you prefer to install plugins in a more declarative manner then Nixpkgs also offer a number of Eclipse plugins that can be installed in an Eclipse environment. This type of environment is created using the function eclipseWithPlugins found inside the nixpkgs.eclipses attribute set. This function takes as argument { eclipse, plugins ? [], jvmArgs ? [] } where eclipse is a one of the Eclipse packages described above, plugins is a list of plugin derivations, and jvmArgs is a list of arguments given to the JVM running the Eclipse. For example, say you wish to install the latest Eclipse Platform with the popular Eclipse Color Theme plugin and also allow Eclipse to use more RAM. You could then add

packageOverrides = pkgs: {
  myEclipse = with pkgs.eclipses; eclipseWithPlugins {
    eclipse = eclipse-platform;
    jvmArgs = [ "-Xmx2048m" ];
    plugins = [ plugins.color-theme ];
  };
}

to your Nixpkgs configuration (~/.config/nixpkgs/config.nix) and install it by running nix-env -f '<nixpkgs>' -iA myEclipse and afterward run Eclipse as usual. It is possible to find out which plugins are available for installation using eclipseWithPlugins by running

$ nix-env -f '<nixpkgs>' -qaP -A eclipses.plugins --description

If there is a need to install plugins that are not available in Nixpkgs then it may be possible to define these plugins outside Nixpkgs using the buildEclipseUpdateSite and buildEclipsePlugin functions found in the nixpkgs.eclipses.plugins attribute set. Use the buildEclipseUpdateSite function to install a plugin distributed as an Eclipse update site. This function takes { name, src } as argument where src indicates the Eclipse update site archive. All Eclipse features and plugins within the downloaded update site will be installed. When an update site archive is not available then the buildEclipsePlugin function can be used to install a plugin that consists of a pair of feature and plugin JARs. This function takes an argument { name, srcFeature, srcPlugin } where srcFeature and srcPlugin are the feature and plugin JARs, respectively.

Expanding the previous example with two plugins using the above functions we have

packageOverrides = pkgs: {
  myEclipse = with pkgs.eclipses; eclipseWithPlugins {
    eclipse = eclipse-platform;
    jvmArgs = [ "-Xmx2048m" ];
    plugins = [
      plugins.color-theme
      (plugins.buildEclipsePlugin {
        name = "myplugin1-1.0";
        srcFeature = fetchurl {
          url = "http://…/features/myplugin1.jar";
          sha256 = "123…";
        };
        srcPlugin = fetchurl {
          url = "http://…/plugins/myplugin1.jar";
          sha256 = "123…";
        };
      });
      (plugins.buildEclipseUpdateSite {
        name = "myplugin2-1.0";
        src = fetchurl {
          stripRoot = false;
          url = "http://…/myplugin2.zip";
          sha256 = "123…";
        };
      });
    ];
  };
}

11.4. Elm

The Nix expressions for Elm reside in pkgs/development/compilers/elm. They are generated automatically by update-elm.rb script. One should specify versions of Elm packages inside the script, clear the packages directory and run the script from inside it. elm-reactor is special because it also has Elm package dependencies. The process is not automated very much for now -- you should get the elm-reactor source tree (e.g. with nix-shell) and run elm2nix.rb inside it. Place the resulting package.nix file into packages/elm-reactor-elm.nix.

11.5. Autojump

autojump needs the shell integration to be useful but unlike other systems, nix doesn't have a standard share directory location. This is why a autojump-share script is shipped that prints the location of the shared folder. This can then be used in the .bashrc like this:

  source "$(autojump-share)/autojump.bash"

11.6. Steam

11.6.1. Steam in Nix

Steam is distributed as a .deb file, for now only as an i686 package (the amd64 package only has documentation). When unpacked, it has a script called steam that in ubuntu (their target distro) would go to /usr/bin . When run for the first time, this script copies some files to the user's home, which include another script that is the ultimate responsible for launching the steam binary, which is also in $HOME.

Nix problems and constraints:

  • We don't have /bin/bash and many scripts point there. Similarly for /usr/bin/python .

  • We don't have the dynamic loader in /lib .

  • The steam.sh script in $HOME can not be patched, as it is checked and rewritten by steam.

  • The steam binary cannot be patched, it's also checked.

The current approach to deploy Steam in NixOS is composing a FHS-compatible chroot environment, as documented here. This allows us to have binaries in the expected paths without disrupting the system, and to avoid patching them to work in a non FHS environment.

11.6.2. How to play

For 64-bit systems it's important to have

hardware.opengl.driSupport32Bit = true;

in your /etc/nixos/configuration.nix. You'll also need

hardware.pulseaudio.support32Bit = true;

if you are using PulseAudio - this will enable 32bit ALSA apps integration. To use the Steam controller, you need to add

services.udev.extraRules = ''
    SUBSYSTEM=="usb", ATTRS{idVendor}=="28de", MODE="0666"
    KERNEL=="uinput", MODE="0660", GROUP="users", OPTIONS+="static_node=uinput"
  '';

to your configuration.

11.6.3. Troubleshooting

Steam fails to start. What do I do?

Try to run

strace steam

to see what is causing steam to fail.

Using the FOSS Radeon drivers
  • The open source radeon drivers need a newer libc++ than is provided by the default runtime, which leads to a crash on launch. Use

    environment.systemPackages = [(pkgs.steam.override { newStdcpp = true; })];

    in your config if you get an error like

    libGL error: unable to load driver: radeonsi_dri.so
    libGL error: driver pointer missing
    libGL error: failed to load driver: radeonsi
    libGL error: unable to load driver: swrast_dri.so
    libGL error: failed to load driver: swrast
  • Steam ships statically linked with a version of libcrypto that conflics with the one dynamically loaded by radeonsi_dri.so. If you get the error

    steam.sh: line 713: 7842 Segmentation fault (core dumped)

    have a look at this pull request.

Java
  1. There is no java in steam chrootenv by default. If you get a message like

    /home/foo/.local/share/Steam/SteamApps/common/towns/towns.sh: line 1: java: command not found

    You need to add

     steam.override { withJava = true; };

    to your configuration.

11.6.4. steam-run

The FHS-compatible chroot used for steam can also be used to run other linux games that expect a FHS environment. To do it, add

pkgs.(steam.override {
          nativeOnly = true;
          newStdcpp = true;
        }).run

to your configuration, rebuild, and run the game with

steam-run ./foo

Chapter 12. Overlays

This chapter describes how to extend and change Nixpkgs packages using overlays. Overlays are used to add layers in the fix-point used by Nixpkgs to compose the set of all packages.

12.1. Installing Overlays

The set of overlays is looked for in the following places. The first one present is considered, and all the rest are ignored:

  1. As an argument of the imported attribute set. When importing Nixpkgs, the overlays attribute argument can be set to a list of functions, which is described in Section 12.2, “Overlays Layout”.

  2. In the directory pointed to by the Nix search path entry <nixpkgs-overlays>.

  3. In the directory ~/.nixpkgs/overlays/.

For the second and third options, the directory should contain Nix expressions defining the overlays. Each overlay can be a file, a directory containing a default.nix, or a symlink to one of those. The expressions should follow the syntax described in Section 12.2, “Overlays Layout”.

The order of the overlay layers can influence the recipe of packages if multiple layers override the same recipe. In the case where overlays are loaded from a directory, they are loaded in alphabetical order.

To install an overlay using the last option, you can clone the overlay's repository and add a symbolic link to it in ~/.nixpkgs/overlays/ directory.

12.2. Overlays Layout

Overlays are expressed as Nix functions which accept 2 arguments and return a set of packages.

self: super:

{
  boost = super.boost.override {
    python = self.python3;
  };
  rr = super.callPackage ./pkgs/rr {
    stdenv = self.stdenv_32bit;
  };
}

The first argument, usually named self, corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of rr in the example above come from self, as well as the overriden dependencies used in the boost override.

The second argument, usually named super, corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs. For example, the original recipe of boost in the above example, comes from super, as well as the callPackage function.

The value returned by this function should be a set similar to pkgs/top-level/all-packages.nix, which contains overridden and/or new packages.

Chapter 13. Coding conventions

13.1. Syntax

  • Use 2 spaces of indentation per indentation level in Nix expressions, 4 spaces in shell scripts.

  • Do not use tab characters, i.e. configure your editor to use soft tabs. For instance, use (setq-default indent-tabs-mode nil) in Emacs. Everybody has different tab settings so it’s asking for trouble.

  • Use lowerCamelCase for variable names, not UpperCamelCase. TODO: naming of attributes in all-packages.nix?

  • Function calls with attribute set arguments are written as

    foo {
      arg = ...;
    }
    

    not

    foo
    {
      arg = ...;
    }
    

    Also fine is

    foo { arg = ...; }
    

    if it's a short call.

  • In attribute sets or lists that span multiple lines, the attribute names or list elements should be aligned:

    # A long list.
    list =
      [ elem1
        elem2
        elem3
      ];
    
    # A long attribute set.
    attrs =
      { attr1 = short_expr;
        attr2 =
          if true then big_expr else big_expr;
      };
    
    # Alternatively:
    attrs = {
      attr1 = short_expr;
      attr2 =
        if true then big_expr else big_expr;
    };
    

  • Short lists or attribute sets can be written on one line:

    # A short list.
    list = [ elem1 elem2 elem3 ];
    
    # A short set.
    attrs = { x = 1280; y = 1024; };
    

  • Breaking in the middle of a function argument can give hard-to-read code, like

    someFunction { x = 1280;
      y = 1024; } otherArg
      yetAnotherArg
    

    (especially if the argument is very large, spanning multiple lines).

    Better:

    someFunction
      { x = 1280; y = 1024; }
      otherArg
      yetAnotherArg
    

    or

    let res = { x = 1280; y = 1024; };
    in someFunction res otherArg yetAnotherArg
    

  • The bodies of functions, asserts, and withs are not indented to prevent a lot of superfluous indentation levels, i.e.

    { arg1, arg2 }:
    assert system == "i686-linux";
    stdenv.mkDerivation { ...
    

    not

    { arg1, arg2 }:
      assert system == "i686-linux";
        stdenv.mkDerivation { ...
    

  • Function formal arguments are written as:

    { arg1, arg2, arg3 }:
    

    but if they don't fit on one line they're written as:

    { arg1, arg2, arg3
    , arg4, ...
    , # Some comment...
      argN
    }:
    

  • Functions should list their expected arguments as precisely as possible. That is, write

    { stdenv, fetchurl, perl }: ...
    

    instead of

    args: with args; ...
    

    or

    { stdenv, fetchurl, perl, ... }: ...
    

    For functions that are truly generic in the number of arguments (such as wrappers around mkDerivation) that have some required arguments, you should write them using an @-pattern:

    { stdenv, doCoverageAnalysis ? false, ... } @ args:
    
    stdenv.mkDerivation (args // {
      ... if doCoverageAnalysis then "bla" else "" ...
    })
    

    instead of

    args:
    
    args.stdenv.mkDerivation (args // {
      ... if args ? doCoverageAnalysis && args.doCoverageAnalysis then "bla" else "" ...
    })
    

13.2. Package naming

In Nixpkgs, there are generally three different names associated with a package:

  • The name attribute of the derivation (excluding the version part). This is what most users see, in particular when using nix-env.

  • The variable name used for the instantiated package in all-packages.nix, and when passing it as a dependency to other functions. This is what Nix expression authors see. It can also be used when installing using nix-env -iA.

  • The filename for (the directory containing) the Nix expression.

Most of the time, these are the same. For instance, the package e2fsprogs has a name attribute "e2fsprogs-version", is bound to the variable name e2fsprogs in all-packages.nix, and the Nix expression is in pkgs/os-specific/linux/e2fsprogs/default.nix.

There are a few naming guidelines:

  • Generally, try to stick to the upstream package name.

  • Don’t use uppercase letters in the name attribute — e.g., "mplayer-1.0rc2" instead of "MPlayer-1.0rc2".

  • The version part of the name attribute must start with a digit (following a dash) — e.g., "hello-0.3.1rc2".

  • If a package is not a release but a commit from a repository, then the version part of the name must be the date of that (fetched) commit. The date must be in "YYYY-MM-DD" format. Also append "unstable" to the name - e.g., "pkgname-unstable-2014-09-23".

  • Dashes in the package name should be preserved in new variable names, rather than converted to underscores (which was convention up to around 2013 and most names still have underscores instead of dashes) — e.g., http-parser instead of http_parser.

  • If there are multiple versions of a package, this should be reflected in the variable names in all-packages.nix, e.g. json-c-0-9 and json-c-0-11. If there is an obvious “default” version, make an attribute like json-c = json-c-0-9;. See also Section 13.3.2, “Versioning”

13.3. File naming and organisation

Names of files and directories should be in lowercase, with dashes between words — not in camel case. For instance, it should be all-packages.nix, not allPackages.nix or AllPackages.nix.

13.3.1. Hierarchy

Each package should be stored in its own directory somewhere in the pkgs/ tree, i.e. in pkgs/category/subcategory/.../pkgname. Below are some rules for picking the right category for a package. Many packages fall under several categories; what matters is the primary purpose of a package. For example, the libxml2 package builds both a library and some tools; but it’s a library foremost, so it goes under pkgs/development/libraries.

When in doubt, consider refactoring the pkgs/ tree, e.g. creating new categories or splitting up an existing category.

If it’s used to support software development:
If it’s a library used by other packages:

development/libraries (e.g. libxml2)

If it’s a compiler:

development/compilers (e.g. gcc)

If it’s an interpreter:

development/interpreters (e.g. guile)

If it’s a (set of) development tool(s):
If it’s a parser generator (including lexers):

development/tools/parsing (e.g. bison, flex)

If it’s a build manager:

development/tools/build-managers (e.g. gnumake)

Else:

development/tools/misc (e.g. binutils)

Else:

development/misc

If it’s a (set of) tool(s):

(A tool is a relatively small program, especially one intented to be used non-interactively.)

If it’s for networking:

tools/networking (e.g. wget)

If it’s for text processing:

tools/text (e.g. diffutils)

If it’s a system utility, i.e., something related or essential to the operation of a system:

tools/system (e.g. cron)

If it’s an archiver (which may include a compression function):

tools/archivers (e.g. zip, tar)

If it’s a compression program:

tools/compression (e.g. gzip, bzip2)

If it’s a security-related program:

tools/security (e.g. nmap, gnupg)

Else:

tools/misc

If it’s a shell:

shells (e.g. bash)

If it’s a server:
If it’s a web server:

servers/http (e.g. apache-httpd)

If it’s an implementation of the X Windowing System:

servers/x11 (e.g. xorg — this includes the client libraries and programs)

Else:

servers/misc

If it’s a desktop environment:

desktops (e.g. kde, gnome, enlightenment)

If it’s a window manager:

applications/window-managers (e.g. awesome, compiz, stumpwm)

If it’s an application:

A (typically large) program with a distinct user interface, primarily used interactively.

If it’s a version management system:

applications/version-management (e.g. subversion)

If it’s for video playback / editing:

applications/video (e.g. vlc)

If it’s for graphics viewing / editing:

applications/graphics (e.g. gimp)

If it’s for networking:
If it’s a mailreader:

applications/networking/mailreaders (e.g. thunderbird)

If it’s a newsreader:

applications/networking/newsreaders (e.g. pan)

If it’s a web browser:

applications/networking/browsers (e.g. firefox)

Else:

applications/networking/misc

Else:

applications/misc

If it’s data (i.e., does not have a straight-forward executable semantics):
If it’s a font:

data/fonts

If it’s related to SGML/XML processing:
If it’s an XML DTD:

data/sgml+xml/schemas/xml-dtd (e.g. docbook)

If it’s an XSLT stylesheet:

(Okay, these are executable...)

data/sgml+xml/stylesheets/xslt (e.g. docbook-xsl)

If it’s a game:

games

Else:

misc

13.3.2. Versioning

Because every version of a package in Nixpkgs creates a potential maintenance burden, old versions of a package should not be kept unless there is a good reason to do so. For instance, Nixpkgs contains several versions of GCC because other packages don’t build with the latest version of GCC. Other examples are having both the latest stable and latest pre-release version of a package, or to keep several major releases of an application that differ significantly in functionality.

If there is only one version of a package, its Nix expression should be named e2fsprogs/default.nix. If there are multiple versions, this should be reflected in the filename, e.g. e2fsprogs/1.41.8.nix and e2fsprogs/1.41.9.nix. The version in the filename should leave out unnecessary detail. For instance, if we keep the latest Firefox 2.0.x and 3.5.x versions in Nixpkgs, they should be named firefox/2.0.nix and firefox/3.5.nix, respectively (which, at a given point, might contain versions 2.0.0.20 and 3.5.4). If a version requires many auxiliary files, you can use a subdirectory for each version, e.g. firefox/2.0/default.nix and firefox/3.5/default.nix.

All versions of a package must be included in all-packages.nix to make sure that they evaluate correctly.

13.4. Fetching Sources

There are multiple ways to fetch a package source in nixpkgs. The general guidline is that you should package sources with a high degree of availability. Right now there is only one fetcher which has mirroring support and that is fetchurl. Note that you should also prefer protocols which have a corresponding proxy environment variable.

You can find many source fetch helpers in pkgs/build-support/fetch*.

In the file pkgs/top-level/all-packages.nix you can find fetch helpers, these have names on the form fetchFrom*. The intention of these are to provide snapshot fetches but using the same api as some of the version controlled fetchers from pkgs/build-support/. As an example going from bad to good:

  • Bad: Uses git:// which won't be proxied.

    src = fetchgit {
      url = "git://github.com/NixOS/nix.git";
      rev = "1f795f9f44607cc5bec70d1300150bfefcef2aae";
      sha256 = "1cw5fszffl5pkpa6s6wjnkiv6lm5k618s32sp60kvmvpy7a2v9kg";
    }
    

  • Better: This is ok, but an archive fetch will still be faster.

    src = fetchgit {
      url = "https://github.com/NixOS/nix.git";
      rev = "1f795f9f44607cc5bec70d1300150bfefcef2aae";
      sha256 = "1cw5fszffl5pkpa6s6wjnkiv6lm5k618s32sp60kvmvpy7a2v9kg";
    }
    

  • Best: Fetches a snapshot archive and you get the rev you want.

    src = fetchFromGitHub {
      owner = "NixOS";
      repo = "nix";
      rev = "1f795f9f44607cc5bec70d1300150bfefcef2aae";
      sha256 = "04yri911rj9j19qqqn6m82266fl05pz98inasni0vxr1cf1gdgv9";
    }
    

13.5. Patches

Only patches that are unique to nixpkgs should be included in nixpkgs source.

Patches available online should be retrieved using fetchpatch.

patches = [
  (fetchpatch {
    name = "fix-check-for-using-shared-freetype-lib.patch";
    url = "http://git.ghostscript.com/?p=ghostpdl.git;a=patch;h=8f5d285";
    sha256 = "1f0k043rng7f0rfl9hhb89qzvvksqmkrikmm38p61yfx51l325xr";
  })
];

Chapter 14. Submitting changes

14.1. Making patches

  • Read Manual (How to write packages for Nix).

  • Fork the repository on GitHub.

  • Create a branch for your future fix.

    • You can make branch from a commit of your local nixos-version. That will help you to avoid additional local compilations. Because you will receive packages from binary cache.

      • For example: nixos-version returns 15.05.git.0998212 (Dingo). So you can do:

      $ git checkout 0998212
      $ git checkout -b 'fix/pkg-name-update'
      

    • Please avoid working directly on the master branch.

  • Make commits of logical units.

    • If you removed pkgs, made some major NixOS changes etc., write about them in nixos/doc/manual/release-notes/rl-unstable.xml.

  • Check for unnecessary whitespace with git diff --check before committing.

  • Format the commit in a following way:

    (pkg-name | service-name): (from -> to | init at version | refactor | etc)
    Additional information.
    
    • Examples:

      • nginx: init at 2.0.1

      • firefox: 3.0 -> 3.1.1

      • hydra service: add bazBaz option

      • nginx service: refactor config generation

  • Test your changes. If you work with

    • nixpkgs:

      • update pkg ->

        • nix-env -i pkg-name -f <path to your local nixpkgs folder>

      • add pkg ->

        • Make sure it's in pkgs/top-level/all-packages.nix

        • nix-env -i pkg-name -f <path to your local nixpkgs folder>

      • If you don't want to install pkg in you profile.

        • nix-build -A pkg-attribute-name <path to your local nixpkgs folder>/default.nix and check results in the folder result. It will appear in the same directory where you did nix-build.

      • If you did nix-env -i pkg-name you can do nix-env -e pkg-name to uninstall it from your system.

    • NixOS and its modules:

      • You can add new module to your NixOS configuration file (usually it's /etc/nixos/configuration.nix). And do sudo nixos-rebuild test -I nixpkgs=<path to your local nixpkgs folder> --fast.

  • If you have commits pkg-name: oh, forgot to insert whitespace: squash commits in this case. Use git rebase -i.

  • Rebase you branch against current master.

14.2. Submitting changes

  • Push your changes to your fork of nixpkgs.

  • Create pull request:

    • Write the title in format (pkg-name | service): improvement.

      • If you update the pkg, write versions from -> to.

    • Write in comment if you have tested your patch. Do not rely much on TravisCI.

    • If you make an improvement, write about your motivation.

    • Notify maintainers of the package. For example add to the message: cc @jagajaga @domenkozar.

14.3. Hotfixing pull requests

  • Make the appropriate changes in you branch.

  • Don't create additional commits, do

    • git rebase -i

    • git push --force to your branch.

14.4. Commit policy

  • Commits must be sufficiently tested before being merged, both for the master and staging branches.

  • Hydra builds for master and staging should not be used as testing platform, it's a build farm for changes that have been already tested.

  • When changing the bootloader installation process, extra care must be taken. Grub installations cannot be rolled back, hence changes may break people's installations forever. For any non-trivial change to the bootloader please file a PR asking for review, especially from @edolstra.

14.4.1. Master branch

  • It should only see non-breaking commits that do not cause mass rebuilds.

14.4.2. Staging branch

  • It's only for non-breaking mass-rebuild commits. That means it's not to be used for testing, and changes must have been well tested already. Read policy here.

  • If the branch is already in a broken state, please refrain from adding extra new breakages. Stabilize it for a few days, merge into master, then resume development on staging. Keep an eye on the staging evaluations here. If any fixes for staging happen to be already in master, then master can be merged into staging.

14.4.3. Stable release branches

  • If you're cherry-picking a commit to a stable release branch, always use git cherry-pick -xe and ensure the message contains a clear description about why this needs to be included in the stable branch.

    An example of a cherry-picked commit would look like this:

    nixos: Refactor the world.
    
    The original commit message describing the reason why the world was torn apart.
    
    (cherry picked from commit abcdef)
    Reason: I just had a gut feeling that this would also be wanted by people from
    the stone age.
          

Chapter 15. Reviewing contributions

Warning

The following section is a draft and reviewing policy is still being discussed.

The nixpkgs projects receives a fairly high number of contributions via GitHub pull-requests. Reviewing and approving these is an important task and a way to contribute to the project.

The high change rate of nixpkgs make any pull request that is open for long enough subject to conflicts that will require extra work from the submitter or the merger. Reviewing pull requests in a timely manner and being responsive to the comments is the key to avoid these. Github provides sort filters that can be used to see the most recently and the least recently updated pull-requests.

When reviewing a pull request, please always be nice and polite. Controversial changes can lead to controversial opinions, but it is important to respect every community members and their work.

GitHub provides reactions, they are a simple and quick way to provide feedback to pull-requests or any comments. The thumb-down reaction should be used with care and if possible accompanied with some explanations so the submitter has directions to improve his contribution.

Pull-requests reviews should include a list of what has been reviewed in a comment, so other reviewers and mergers can know the state of the review.

All the review template samples provided in this section are generic and meant as examples. Their usage is optional and the reviewer is free to adapt them to his liking.

15.1. Package updates

A package update is the most trivial and common type of pull-request. These pull-requests mainly consist in updating the version part of the package name and the source hash.

It can happen that non trivial updates include patches or more complex changes.

Reviewing process:

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: package (update) and any topic label that fit the updated package.

  • Ensure that the package versioning is fitting the guidelines.

  • Ensure that the commit text is fitting the guidelines.

  • Ensure that the package maintainers are notified.

    • mention-bot usually notify GitHub users based on the submitted changes, but it can happen that it misses some of the package maintainers.

  • Ensure that the meta field contains correct information.

    • License can change with version updates, so it should be checked to be fitting upstream license.

    • If the package has no maintainer, a maintainer must be set. This can be the update submitter or a community member that accepts to take maintainership of the package.

  • Ensure that the code contains no typos.

  • Building the package locally.

    • Pull-requests are often targeted to the master or staging branch so building the pull-request locally as it is submitted can trigger a large amount of source builds.

      It is possible to rebase the changes on nixos-unstable or nixpkgs-unstable for easier review by running the following commands from a nixpkgs clone.

      $ git remote add channels https://github.com/NixOS/nixpkgs-channels.git 1
      $ git fetch channels nixos-unstable 2
      $ git fetch origin pull/PRNUMBER/head 3
      $ git rebase --onto nixos-unstable BASEBRANCH FETCH_HEAD 4
      

      1

      This should be done only once to be able to fetch channel branches from the nixpkgs-channels repository.

      2

      Fetching the nixos-unstable branch.

      3

      Fetching the pull-request changes, PRNUMBER is the number at the end of the pull-request title and BASEBRANCH the base branch of the pull-request.

      3

      Rebasing the pull-request changes to the nixos-unstable branch.

    • The nox tool can be used to review a pull-request content in a single command. It doesn't rebase on a channel branch so it might trigger multiple source builds. PRNUMBER should be replaced by the number at the end of the pull-request title.

      $ nix-shell -p nox --run "nox-review -k pr PRNUMBER"
      
  • Running every binary.

Example 15.1. Sample template for a package update review

##### Reviewed points

- [ ] package name fits guidelines
- [ ] package version fits guidelines
- [ ] package build on ARCHITECTURE
- [ ] executables tested on ARCHITECTURE
- [ ] all depending packages build

##### Possible improvements

##### Comments


15.2. New packages

New packages are a common type of pull-requests. These pull requests consists in adding a new nix-expression for a package.

Reviewing process:

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: package (new) and any topic label that fit the new package.

  • Ensure that the package versioning is fitting the guidelines.

  • Ensure that the commit name is fitting the guidelines.

  • Ensure that the meta field contains correct information.

    • License must be checked to be fitting upstream license.

    • Platforms should be set or the package will not get binary substitutes.

    • A maintainer must be set, this can be the package submitter or a community member that accepts to take maintainership of the package.

  • Ensure that the code contains no typos.

  • Ensure the package source.

    • Mirrors urls should be used when available.

    • The most appropriate function should be used (e.g. packages from GitHub should use fetchFromGitHub).

  • Building the package locally.

  • Running every binary.

Example 15.2. Sample template for a new package review

##### Reviewed points

- [ ] package path fits guidelines
- [ ] package name fits guidelines
- [ ] package version fits guidelines
- [ ] package build on ARCHITECTURE
- [ ] executables tested on ARCHITECTURE
- [ ] `meta.description` is set and fits guidelines
- [ ] `meta.license` fits upstream license
- [ ] `meta.platforms` is set
- [ ] `meta.maintainers` is set
- [ ] build time only dependencies are declared in `nativeBuildInputs`
- [ ] source is fetched using the appropriate function
- [ ] phases are respected
- [ ] patches that are remotely available are fetched with `fetchpatch`

##### Possible improvements

##### Comments


15.3. Module updates

Module updates are submissions changing modules in some ways. These often contains changes to the options or introduce new options.

Reviewing process

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: module (update) and any topic label that fit the module.

  • Ensure that the module maintainers are notified.

    • Mention-bot notify GitHub users based on the submitted changes, but it can happen that it miss some of the package maintainers.

  • Ensure that the module tests, if any, are succeeding.

  • Ensure that the introduced options are correct.

    • Type should be appropriate (string related types differs in their merging capabilities, optionSet and string types are deprecated).

    • Description, default and example should be provided.

  • Ensure that option changes are backward compatible.

    • mkRenamedOptionModule and mkAliasOptionModule functions provide way to make option changes backward compatible.

  • Ensure that removed options are declared with mkRemovedOptionModule

  • Ensure that changes that are not backward compatible are mentioned in release notes.

  • Ensure that documentations affected by the change is updated.

Example 15.3. Sample template for a module update review

##### Reviewed points

- [ ] changes are backward compatible
- [ ] removed options are declared with `mkRemovedOptionModule`
- [ ] changes that are not backward compatible are documented in release notes
- [ ] module tests succeed on ARCHITECTURE
- [ ] options types are appropriate
- [ ] options description is set
- [ ] options example is provided
- [ ] documentation affected by the changes is updated

##### Possible improvements

##### Comments


15.4. New modules

New modules submissions introduce a new module to NixOS.

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: module (new) and any topic label that fit the module.

  • Ensure that the module tests, if any, are succeeding.

  • Ensure that the introduced options are correct.

    • Type should be appropriate (string related types differs in their merging capabilities, optionSet and string types are deprecated).

    • Description, default and example should be provided.

  • Ensure that module meta field is present

    • Maintainers should be declared in meta.maintainers.

    • Module documentation should be declared with meta.doc.

  • Ensure that the module respect other modules functionality.

    • For example, enabling a module should not open firewall ports by default.

Example 15.4. Sample template for a new module review

##### Reviewed points

- [ ] module path fits the guidelines
- [ ] module tests succeed on ARCHITECTURE
- [ ] options have appropriate types
- [ ] options have default
- [ ] options have example
- [ ] options have descriptions
- [ ] No unneeded package is added to system.environmentPackages
- [ ] meta.maintainers is set
- [ ] module documentation is declared in meta.doc

##### Possible improvements

##### Comments


15.5. Other submissions

Other type of submissions requires different reviewing steps.

If you consider having enough knowledge and experience in a topic and would like to be a long-term reviewer for related submissions, please contact the current reviewers for that topic. They will give you information about the reviewing process. The main reviewers for a topic can be hard to find as there is no list, but checking past pull-requests to see who reviewed or git-blaming the code to see who committed to that topic can give some hints.

Container system, boot system and library changes are some examples of the pull requests fitting this category.

15.6. Merging pull-requests

It is possible for community members that have enough knowledge and experience on a special topic to contribute by merging pull requests.

TODO: add the procedure to request merging rights.

In a case a contributor leaves definitively the Nix community, he should create an issue or notify the mailing list with references of packages and modules he maintains so the maintainership can be taken over by other contributors.

Chapter 16. Contributing to this documentation

The DocBook sources of the Nixpkgs manual are in the doc subdirectory of the Nixpkgs repository. If you make modifications to the manual, it's important to build it before committing. You can do that as follows:

$ cd /path/to/nixpkgs
$ nix-build doc

If the build succeeds, the manual will be in ./result/share/doc/nixpkgs/manual.html.