Merge #133: Improve modularity, remove dependency on nixops, add modules test

187ff884db add modules test (Erik Arvstedt)
826245484e make secrets dir location configurable (Erik Arvstedt)
b1e13e9415 simplify secrets file format (Erik Arvstedt)
314272a228 lnd, nanopos: move user and group definitions to the bottom (Erik Arvstedt)
766fa4f300 travis: cache all build outputs with cachix (Erik Arvstedt)
b0e759160d travis: set NIX_PATH as early as possible (Erik Arvstedt)
c51bbcf104 travis: move comment (Erik Arvstedt)
7092dce0c7 travis: remove use of deprecated statements (Erik Arvstedt)
190a92507c travis: split up scripts into statements (Erik Arvstedt)
10d6b04ac8 support enabling clightning and lnd simultaneously (Erik Arvstedt)
ad7a519284 bitcoind: wait until RPC port is open (Erik Arvstedt)
5536b64fb3 lnd: wait until wallet is created (Erik Arvstedt)
6f2a55d63c lnd: wait until RPC port is open (Erik Arvstedt)
1868bef462 lnd: add option 'rpcPort' (Erik Arvstedt)
120e3e8cfe lnd postStart: suppress curl response output (Erik Arvstedt)
3e86637327 lnd postStart: poll for REST service availability (Erik Arvstedt)
795c51dc01 lnd postStart: make more idiomatic (Erik Arvstedt)
6e58beae8a lnd: use postStart option for script (Erik Arvstedt)
86167c6e6d clightning: wait until the RPC socket appears (Erik Arvstedt)
60c732a6a1 onion-chef: set RemainAfterExit, fix tor dependency (Erik Arvstedt)
2b9b3ba1c5 systemPackages: improve readability with shorter service references (Erik Arvstedt)
14ecb5511a liquid: add cli option (Erik Arvstedt)
cd5ed39b9c lnd: add cli option (Erik Arvstedt)
1833b15888 clightning: add cli option (Erik Arvstedt)
b90bf6691b add generate-secrets.service (Erik Arvstedt)
6447694214 add generate-secrets pkg (Erik Arvstedt)
e34093a8ac generate_secrets.sh: add opensslConf option (Erik Arvstedt)
9d14d5ba64 generate_secrets.sh: write secrets to working directory (Erik Arvstedt)
51fb054001 generate_secrets.sh: extract makepw command (Erik Arvstedt)
e3b47ce18a add setup-secrets.service (Erik Arvstedt)
437b268433 extract make-secrets.nix (Erik Arvstedt)
f9c29b9318 simplify secret definitions (Erik Arvstedt)
cd0fd6926b don't copy secret files to store during nixops deployment (Erik Arvstedt)
f0a36fe0c7 add 'nix-bitcoin-services' option (Erik Arvstedt)
7aaf30501c nix-bitcoin-services: simplify formatting (Erik Arvstedt)
760da232e0 add nix-bitcoin pkgs namespace (Erik Arvstedt)
6def181dbc add modules.nix (Erik Arvstedt)
3b842e5fe7 add nix-bitcoin-secrets.target (Erik Arvstedt)
bbf2bbc04a network.nix: simplify import of main config (Erik Arvstedt)
7e021a2629 simplify overlay.nix (Erik Arvstedt)
07dc3e04ac move bitcoinrpc group definition to bitcoind (Erik Arvstedt)
d61b185c3a simplify user and group definitions (Erik Arvstedt)

Pull request description:

  The nix-bitcoin modules consist of three fundamental components:
  1. a set of bitcoin-related modules for general use.
  2. an opinionated configuration of these modules (`nix-bitcoin.nix`), to be deployed on a
     dedicated machine.
  3. machinery for nixops deployment.

  This PR removes dependencies that reach from top to bottom in the list.
  This means that 1. is now usable on its own and that 2. can be used without 3.

  Besides improving nix-bitcoin's general usefulness, this
  - simplifies testing. This PR includes a Travis-enabled modules test using the NixOS testing framework.
  - paves the way for krops deployment.
  - unlocks direct deployment in NixOS containers which allows for super fast experimentation.

  ### Details
  Here are the unnecessary inter-component dependencies and how they're resolved by the commits. I'm using the numbering from the list above.

  - `1. -> 3.` The modules (1.) use the nixops-specific (3.) `keys` group.
    Resolved by `add nix-bitcoin-secrets.target`.

  - `1. -> 3.` 1. requires nixops-specific key services.
    Resolved by `add nix-bitcoin-secrets.target`.

  - `1. -> 2.` bitcoind needs the bitcoinrpc group which is defined in `nix-bitcoin.nix` (2.).
    Resolved by `move bitcoinrpc group definition to bitcoind`.

  Further obstacles for standalone usage of 1.:

  - We can't easily import 1. as a standalone module set.
    Resolved by `add modules.nix`.

  - Users of 1. shouldn't be forced to import nix-bitcoin's packages as top-level items in the pkgs namespace.
    Resolved by `add nix-bitcoin pkgs namespace`.

  ### Non-nixops deployments
  Commit `add setup-secrets.service` simplifies non-nixops deployment methods like containers, NixOS VMs or krops.

  Secrets can now deployed as follows:
  1. create local secrets.
  2. transfer secrets to machine.
  3. on the machine, `setup-secrets.service` creates extra secrets from `secrets.nix` and sets owner and
     permissions for all secrets.

  As krops integrates step 2. we now have all ingredients for automatic krops deployment.

  The service is complicated by the creation of secrets like `bitcoin-rpcpassword` that are composed of attrs from `secrets.nix` instead of being simply backed by a file like `lnd_key`. We could simplify this by creating all secret files locally.

  Running nix-bitcoin in NixOS containers gives you faster rebuild cycles when developing. [Here's](https://gist.github.com/5db4fa7dd3f1137920b58e39647116f6) an example.

  ### Test
  The last commits starting with `clightning: add cli option` are testing-related and mostly fix non-critical bugs that were exposed by the test.

  All `STABLE=1` builds from the Travis build matrix are implicit in the modules test.
  Should we remove these individual builds?

  Regarding commit `travis: cache all build outputs with cachix`:
  To replace my cache with a cache that's owned by you (maybe named `nix-bitcoin-ci`), run
  ```
  nix-shell -p travis --run 'travis encrypt CACHIX_SIGNING_KEY=... -r fort-nix/nix-bitcoin'
  ```
  where `...` is the value of `secretKey` in `~/.config/cachix/cachix.dhall`. Let me know the travis secret and I'll fixup the commit.

  ### Docs
  If you like the proposed changes, I'll add another PR with updates to the docs regarding the project layout, non-nixops deployment, and how to use nix-bitcoin within a larger NixOS config.

ACKs for top commit:
  jonasnick:
    ACK 187ff884db

Tree-SHA512: f4be65215c592a4f41bb7fa991a6d8d7c463cf631b88bf53051ca57ba280e7a60b8b09d0d1521345d5b656f844daa2166fff5d00a3105077c9e263465eacfb0a

.gitignore

@@ -1 +1 @@
-secrets/
+/secrets/

.travis.yml

@@ -1,24 +1,25 @@
-language: minimal
+dist: bionic # needed for KVM
+language: shell
 
-# broken:
+install:
-#   - PKG=electrs STABLE=0
+    # Retry installing nix due to nondeterministic error
-
+    #   Fatal error: glibc detected an invalid stdio handle
-# Retry installing nix due to nondeterministic error
+    # see:
-#   Fatal error: glibc detected an invalid stdio handle
+    #   https://github.com/nh2/static-haskell-nix/pull/27#issuecomment-502652181
-# see:
+    #   https://github.com/nixos/nix/issues/2733
-#   https://github.com/nh2/static-haskell-nix/pull/27#issuecomment-502652181
+  - (for i in {1..5}; do bash <(curl https://nixos.org/nix/install) && exit 0; done; exit 1)
-#   https://github.com/nixos/nix/issues/2733
+  - . /home/travis/.nix-profile/etc/profile.d/nix.sh
-install: |
+  - if [[ $STABLE == 1 ]]; then export NIX_PATH="nixpkgs=$(nix eval --raw -f pkgs/nixpkgs-pinned.nix nixpkgs)"; fi
-  (for i in {1..5}; do bash <(curl https://nixos.org/nix/install) && exit 0; done; exit 1)
+  - if [[ $STABLE == 0 ]]; then export NIX_PATH="nixpkgs=$(nix eval --raw -f pkgs/nixpkgs-pinned.nix nixpkgs-unstable)"; fi
-  . /home/travis/.nix-profile/etc/profile.d/nix.sh
+  - nix-env -iA cachix -f https://cachix.org/api/v1/install
-  nix-env -iA cachix -f https://cachix.org/api/v1/install
+  - cachix use nix-bitcoin
-  cachix use nix-bitcoin
+  - VER="$(nix eval nixpkgs.lib.version)"
-  [ $STABLE -eq 1 ] && export NIX_PATH="nixpkgs=$(nix eval --raw -f pkgs/nixpkgs-pinned.nix nixpkgs)"
-  [ $STABLE -eq 0 ] && export NIX_PATH="nixpkgs=$(nix eval --raw -f pkgs/nixpkgs-pinned.nix nixpkgs-unstable)"
-  VER="$(nix eval nixpkgs.lib.version)"
 env:
-  matrix:
+  global:
-  - PKG=nodeinfo STABLE=1
+    # CACHIX_SIGNING_KEY
+  - secure: "xXCFZ7g+k5YmCGm8R8l3bZElVmt+RD1KscG3kGr5w4HyyDPTzFetPo+sT8bUpysDU0u3HWhfVhHtpog2mhNhwVl3tQwKXea3dHKC1i6ypBg3gjDngmJRR5wo++ocYDpK8qPaU7m/jHQTNFnTA4CbmMcc05GcYx/1Ai/ZGkNwWFjdIcVeOUoiol33gykMOXIGDg2qlXudt33wP53FHbX8L4fxzodWfAuxKK4AoGprxy5eSnU7LCaXxxJmu4HwuV+Ux2U1NfE/E33cvhlUvTQCswVSZFG06mg8rwhMG1ozsDvlL2itZlu/BeUQH5y3XMMlnJIUXUazkRBibf1w/ebVjpOF+anqkqmq8tcbFEa7T+RJeVTIsvP+L8rE8fcmuZtdg9hNmgRnLmaeT0vVwD1L2UqW9HdRyujdoS0jPYuoc1W7f1JQWfAPhBPQ1SrtKyNNqcbVJ34aN7b+4vCzRpQL1JTbmjzQIWhkiKN1qMo1v/wbIydW8yka4hc4JOfdQLaAJEPI1eAC1MLotSAegMnwKWE1dzm66MuPSipksYjZrvsB28cV4aCVUffIuRhrSr1i2afRHwTpNbK9U4/576hah15ftUdR79Sfkcoi1ekSQTFGRvkRIPYtkKLYwFa3jVA41qz7+IIZCf4TsApy3XDdFx91cRub7yPq9BeZ83A+qYQ="
+  jobs:
+  - TestModules=1 STABLE=1
   - PKG=hwi STABLE=1
   - PKG=lightning-charge STABLE=1
   - PKG=lightning-charge STABLE=0
@@ -28,7 +29,44 @@ env:
   - PKG=elementsd STABLE=1
   - PKG=elementsd STABLE=0
   - PKG=electrs STABLE=1
+  # broken
+  # - PKG=electrs STABLE=0
   - PKG=liquid-swap STABLE=1
-script: |
+script:
-  printf '%s (%s)\n' "$NIX_PATH" "$VER"
+  - printf '%s (%s)\n' "$NIX_PATH" "$VER"
-  nix-build -A $PKG
+  - |
+    getBuildExpr() {
+        if [[ $TestModules ]]; then
+            if [[ ! -e /dev/kvm ]]; then
+                >&2 echo "No KVM available on VM Host."
+                exit 1
+            fi
+            sudo chmod go+rw /dev/kvm
+            test/run-tests.sh exprForCI
+        else
+            echo "(import ./. {}).$PKG"
+        fi
+    }
+  - buildExpr=$(getBuildExpr)
+  - time nix-instantiate -E "$buildExpr" --add-root ./drv --indirect
+  - outPath=$(nix-store --query ./drv)
+  - |
+    if nix path-info --store https://nix-bitcoin.cachix.org $outPath &>/dev/null; then
+      echo "$outPath" has already been built successfully.
+      travis_terminate 0
+    fi
+    # Travis doesn't expose secrets to pull-request builds,
+    # so skip cache uploading in this case
+  - |
+    if [[ $CACHIX_SIGNING_KEY ]]; then
+      cachix push nix-bitcoin --watch-store &
+      cachixPid=$!
+    fi
+  - nix-build ./drv
+  - |
+    if [[ $CACHIX_SIGNING_KEY ]]; then
+      # Wait until cachix has finished uploading
+      # Run as root because yama/ptrace_scope != 0
+      ruby=$(nix-build '<nixpkgs>' -A ruby)/bin/ruby
+      time sudo $ruby helper/wait-for-network-idle.rb $cachixPid
+    fi

default.nix

@@ -1,16 +1,5 @@
 { pkgs ? import <nixpkgs> {} }:
-{
-  # 'lib', 'modules' and 'overlays' are special, see
-  # https://github.com/nix-community/NUR for more.
-  modules = import ./modules; # NixOS modules
 
-  nodeinfo = pkgs.callPackage ./pkgs/nodeinfo { };
+(import ./pkgs { inherit pkgs; }) // {
-  lightning-charge = pkgs.callPackage ./pkgs/lightning-charge { };
+  modules = import ./modules;
-  nanopos = pkgs.callPackage ./pkgs/nanopos { };
-  spark-wallet = pkgs.callPackage ./pkgs/spark-wallet { };
-  electrs = (pkgs.callPackage ./pkgs/electrs { }).rootCrate.build;
-  elementsd = pkgs.callPackage ./pkgs/elementsd { withGui = false; };
-  hwi = pkgs.callPackage ./pkgs/hwi { };
-  pylightning = pkgs.python3Packages.callPackage ./pkgs/pylightning { };
-  liquid-swap = pkgs.python3Packages.callPackage ./pkgs/liquid-swap { };
 }

helper/wait-for-network-idle.rb

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+require 'open3'
+
+# Wait until the given PID had no network activity for `Timeout` seconds, then exit.
+
+pid = ARGV.first
+Timeout = 2
+
+stdin, out, err, wait_thread = Open3.popen3("strace -f -e trace=network -s 1 -q -p #{pid}")
+while IO.select([err], nil, nil, Timeout)
+  begin
+    out = err.read_nonblock(1 << 10)
+  rescue EOFError
+    status = wait_thread.value
+    if status.success?
+      puts "Monitored process #{pid} exited"
+      exit 0
+    else
+      puts "Strace failed with exit code #{status.to_i}. Last output:\n#{out}"
+      # strace often fails with code 256 which looks like success to shells. fail with 1 instead.
+      exit 1
+    end
+  end
+end
+
+# If we exit without an explicit kill,
+# ptrace can fail on reattachment:  ptrace(PTRACE_SEIZE, $PID): Operation not permitted
+# Only relevant for testing.
+Process.kill("TERM", wait_thread.pid)

modules/bitcoind.nix

@@ -3,8 +3,8 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.bitcoind;
+  inherit (config) nix-bitcoin-services;
   pidFile = "${cfg.dataDir}/bitcoind.pid";
   configFile = pkgs.writeText "bitcoin.conf" ''
     ${optionalString cfg.testnet "testnet=1"}
@@ -19,7 +19,7 @@ let
     listen=${if cfg.listen then "1" else "0"}
 
     # RPC server options
-    ${optionalString (cfg.rpc.port != null) "rpcport=${toString cfg.rpc.port}"}
+    rpcport=${toString cfg.rpc.port}
     ${concatMapStringsSep  "\n"
       (rpcUser: "rpcauth=${rpcUser.name}:${rpcUser.passwordHMAC}")
       (attrValues cfg.rpc.users)
@@ -69,7 +69,7 @@ in {
 
       package = mkOption {
         type = types.package;
-        default = pkgs.blockchains.bitcoind;
+        default = pkgs.nix-bitcoin.bitcoind;
         defaultText = "pkgs.blockchains.bitcoind";
         description = "The package providing bitcoin binaries.";
       };
@@ -108,9 +108,9 @@ in {
 
       rpc = {
         port = mkOption {
-          type = types.nullOr types.ints.u16;
+          type = types.ints.u16;
-          default = null;
+          default = 8332;
-          description = "Override the default port on which to listen for JSON-RPC connections.";
+          description = "Port on which to listen for JSON-RPC connections.";
         };
         users = mkOption {
           default = {};
@@ -225,8 +225,8 @@ in {
     environment.systemPackages = [ cfg.package ];
     systemd.services.bitcoind = {
       description = "Bitcoin daemon";
-      requires = [ "bitcoin-rpcpassword-key.service" ];
+      requires = [ "nix-bitcoin-secrets.target" ];
-      after = [ "network.target" "bitcoin-rpcpassword-key.service" ];
+      after = [ "network.target" "nix-bitcoin-secrets.target" ];
       wantedBy = [ "multi-user.target" ];
       preStart = ''
         if ! test -e ${cfg.dataDir}; then
@@ -238,9 +238,15 @@ in {
         cp '${cfg.configFileOption}' '${cfg.dataDir}/bitcoin.conf'
         chmod o-rw  '${cfg.dataDir}/bitcoin.conf'
         chown -R '${cfg.user}:${cfg.group}' '${cfg.dataDir}'
-        echo "rpcpassword=$(cat /secrets/bitcoin-rpcpassword)" >> '${cfg.dataDir}/bitcoin.conf'
+        echo "rpcpassword=$(cat ${config.nix-bitcoin.secretsDir}/bitcoin-rpcpassword)" >> '${cfg.dataDir}/bitcoin.conf'
         chmod -R g+rX '${cfg.dataDir}/blocks'
       '';
+      # Wait until RPC port is open. This usually takes just a few ms.
+      postStart = ''
+        while ! { exec 3>/dev/tcp/127.0.0.1/${toString cfg.rpc.port}; } &>/dev/null; do
+          sleep 0.05
+        done
+      '';
       serviceConfig = {
         Type = "simple";
         User = "${cfg.user}";
@@ -295,14 +301,16 @@ in {
     };
 
     users.users.${cfg.user} = {
-      name = cfg.user;
       group = cfg.group;
-      extraGroups = [ "keys" ];
       description = "Bitcoin daemon user";
       home = cfg.dataDir;
     };
-    users.groups.${cfg.group} = {
+    users.groups.${cfg.group} = {};
-      name = cfg.group;
+    users.groups.bitcoinrpc = {};
+
+    nix-bitcoin.secrets.bitcoin-rpcpassword = {
+      user = "bitcoin";
+      group = "bitcoinrpc";
     };
   };
 }

modules/clightning.nix

@@ -3,8 +3,8 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.clightning;
+  inherit (config) nix-bitcoin-services;
   configFile = pkgs.writeText "config" ''
     autolisten=${if cfg.autolisten then "true" else "false"}
     network=bitcoin
@@ -57,6 +57,16 @@ in {
       default = "/var/lib/clightning";
       description = "The data directory for clightning.";
     };
+    cli = mkOption {
+      readOnly = true;
+      default = pkgs.writeScriptBin "lightning-cli"
+      # Switch user because c-lightning doesn't allow setting the permissions of the rpc socket
+      # https://github.com/ElementsProject/lightning/issues/1366
+      ''
+        exec sudo -u clightning ${pkgs.nix-bitcoin.clightning}/bin/lightning-cli --lightning-dir='${cfg.dataDir}' "$@"
+      '';
+      description = "Binary to connect with the clightning instance.";
+    };
     enforceTor =  nix-bitcoin-services.enforceTor;
   };
 
@@ -64,16 +74,14 @@ in {
     users.users.clightning = {
         description = "clightning User";
         group = "clightning";
-        extraGroups = [ "bitcoinrpc" "keys" ];
+        extraGroups = [ "bitcoinrpc" ];
         home = cfg.dataDir;
     };
-    users.groups.clightning = {
+    users.groups.clightning = {};
-      name = "clightning";
-    };
 
     systemd.services.clightning = {
       description = "Run clightningd";
-      path  = [ pkgs.blockchains.bitcoind ];
+      path  = [ pkgs.nix-bitcoin.bitcoind ];
       wantedBy = [ "multi-user.target" ];
       requires = [ "bitcoind.service" ];
       after = [ "bitcoind.service" ];
@@ -85,11 +93,11 @@ in {
         chmod u=rw,g=r,o= ${cfg.dataDir}/config
         # The RPC socket has to be removed otherwise we might have stale sockets
         rm -f ${cfg.dataDir}/lightning-rpc
-        echo "bitcoin-rpcpassword=$(cat /secrets/bitcoin-rpcpassword)" >> '${cfg.dataDir}/config'
+        echo "bitcoin-rpcpassword=$(cat ${config.nix-bitcoin.secretsDir}/bitcoin-rpcpassword)" >> '${cfg.dataDir}/config'
         '';
       serviceConfig = {
         PermissionsStartOnly = "true";
-        ExecStart = "${pkgs.clightning}/bin/lightningd --lightning-dir=${cfg.dataDir}";
+        ExecStart = "${pkgs.nix-bitcoin.clightning}/bin/lightningd --lightning-dir=${cfg.dataDir}";
         User = "clightning";
         Restart = "on-failure";
         RestartSec = "10s";
@@ -98,6 +106,11 @@ in {
           then nix-bitcoin-services.allowTor
           else nix-bitcoin-services.allowAnyIP
         );
+      # Wait until the rpc socket appears
+      postStart = ''
+        while read f; do [[ $f == lightning-rpc ]] && break; done \
+          < <(${pkgs.inotifyTools}/bin/inotifywait --quiet --monitor -e create,moved_to --format '%f' '${cfg.dataDir}')
+      '';
     };
   };
 }

modules/default.nix

@@ -1,4 +1,5 @@
 {
+  modules = ./modules.nix;
   bitcoind = ./bitcoind.nix;
   clightning = ./clightning.nix;
   default = ./default.nix;
@@ -7,7 +8,6 @@
   liquid = ./liquid.nix;
   nanopos = ./nanopos.nix;
   nix-bitcoin = ./nix-bitcoin.nix;
-  nix-bitcoin-pkgs = ./nix-bitcoin-pkgs.nix;
   nix-bitcoin-webindex = ./nix-bitcoin-webindex.nix;
   spark-wallet = ./spark-wallet.nix;
   recurring-donations = ./recurring-donations.nix;

modules/electrs.nix

@@ -3,8 +3,9 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.electrs;
+  inherit (config) nix-bitcoin-services;
+  secretsDir = config.nix-bitcoin.secretsDir;
   index-batch-size = "${if cfg.high-memory then "" else "--index-batch-size=10"}";
   jsonrpc-import = "${if cfg.high-memory then "" else "--jsonrpc-import"}";
 in {
@@ -58,15 +59,12 @@ in {
 
   config = mkIf cfg.enable {
     users.users.${cfg.user} = {
-        name = cfg.user;
         description = "electrs User";
         group = cfg.group;
-        extraGroups = [ "bitcoinrpc" "keys" "bitcoin"];
+        extraGroups = [ "bitcoinrpc" "bitcoin"];
         home = cfg.dataDir;
     };
-    users.groups.electrs = {
+    users.groups.${cfg.group} = {};
-      name = cfg.group;
-    };   
 
     systemd.services.electrs = {
       description = "Run electrs";
@@ -77,7 +75,7 @@ in {
       preStart = ''
         mkdir -m 0770 -p ${cfg.dataDir}
         chown -R '${cfg.user}:${cfg.group}' ${cfg.dataDir}
-        echo "${pkgs.electrs}/bin/electrs -vvv ${index-batch-size} ${jsonrpc-import} --timestamp --db-dir ${cfg.dataDir} --daemon-dir /var/lib/bitcoind --cookie=${config.services.bitcoind.rpcuser}:$(cat /secrets/bitcoin-rpcpassword) --electrum-rpc-addr=127.0.0.1:${toString cfg.port}" > /run/electrs/startscript.sh
+        echo "${pkgs.nix-bitcoin.electrs}/bin/electrs -vvv ${index-batch-size} ${jsonrpc-import} --timestamp --db-dir ${cfg.dataDir} --daemon-dir /var/lib/bitcoind --cookie=${config.services.bitcoind.rpcuser}:$(cat ${secretsDir}/bitcoin-rpcpassword) --electrum-rpc-addr=127.0.0.1:${toString cfg.port}" > /run/electrs/startscript.sh
         '';	
       serviceConfig = rec {
         RuntimeDirectory = "electrs";
@@ -106,8 +104,8 @@ in {
             listen ${toString config.services.electrs.nginxport} ssl;
             proxy_pass electrs;
 
-            ssl_certificate /secrets/nginx_cert;
+            ssl_certificate ${secretsDir}/nginx-cert;
-            ssl_certificate_key /secrets/nginx_key;
+            ssl_certificate_key ${secretsDir}/nginx-key;
             ssl_session_cache shared:SSL:1m;
             ssl_session_timeout 4h;
             ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
@@ -116,5 +114,16 @@ in {
         }
       '';
     };
+    systemd.services.nginx = {
+      requires = [ "nix-bitcoin-secrets.target" ];
+      after = [ "nix-bitcoin-secrets.target" ];
+    };
+    nix-bitcoin.secrets = rec {
+      nginx-key = {
+        user = "nginx";
+        group = "root";
+      };
+      nginx-cert = nginx-key;
+    };
   };
 }

modules/lightning-charge.nix

@@ -3,8 +3,8 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.lightning-charge;
+  inherit (config) nix-bitcoin-services;
 in {
   options.services.lightning-charge = {
     enable = mkOption {
@@ -30,8 +30,8 @@ in {
       requires = [ "clightning.service" ];
       after = [ "clightning.service" ];
       serviceConfig = {
-          EnvironmentFile = "/secrets/lightning-charge-api-token";
+          EnvironmentFile = "${config.nix-bitcoin.secretsDir}/lightning-charge-env";
-          ExecStart = "${pkgs.lightning-charge}/bin/charged -l ${config.services.clightning.dataDir} -d ${config.services.clightning.dataDir}/lightning-charge.db";
+          ExecStart = "${pkgs.nix-bitcoin.lightning-charge}/bin/charged -l ${config.services.clightning.dataDir} -d ${config.services.clightning.dataDir}/lightning-charge.db";
           # Unfortunately c-lightning doesn't allow setting the permissions of the rpc socket,
           # so this must run as the clightning user
           # https://github.com/ElementsProject/lightning/issues/1366
@@ -42,5 +42,6 @@ in {
         // nix-bitcoin-services.nodejs
         // nix-bitcoin-services.allowTor;
     };
+    nix-bitcoin.secrets.lightning-charge-env.user = "clightning";
   };
 }

modules/liquid.nix

@@ -3,8 +3,9 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.liquidd;
+  inherit (config) nix-bitcoin-services;
+  secretsDir = config.nix-bitcoin.secretsDir;
   pidFile = "${cfg.dataDir}/liquidd.pid";
   configFile = pkgs.writeText "elements.conf" ''
     chain=liquidv1
@@ -175,16 +176,30 @@ in {
           Validate pegin claims. All functionaries must run this.
         '';
       };
+      cli = mkOption {
+        readOnly = true;
+        default = pkgs.writeScriptBin "elements-cli" ''
+          exec ${pkgs.nix-bitcoin.elementsd}/bin/elements-cli -datadir='${cfg.dataDir}' "$@"
+        '';
+        description = "Binary to connect with the liquidd instance.";
+      };
+      swap-cli = mkOption {
+        readOnly = true;
+        default = pkgs.writeScriptBin "liquidswap-cli" ''
+          exec ${pkgs.nix-bitcoin.liquid-swap}/bin/liquidswap-cli -c '${cfg.dataDir}/elements.conf' "$@"
+        '';
+        description = "Binary for managing liquid swaps.";
+      };
       enforceTor =  nix-bitcoin-services.enforceTor;
     };
   };
 
   config = mkIf cfg.enable {
-    environment.systemPackages = [ pkgs.elementsd ];
+    environment.systemPackages = [ pkgs.nix-bitcoin.elementsd ];
     systemd.services.liquidd = {
       description = "Elements daemon providing access to the Liquid sidechain";
-      requires = [ "liquid-rpcpassword-key.service" ];
+      requires = [ "bitcoind.service" ];
-      after = [ "network.target" "liquid-rpcpassword-key.service" ];
+      after = [ "bitcoind.service" ];
       wantedBy = [ "multi-user.target" ];
       preStart = ''
         if ! test -e ${cfg.dataDir}; then
@@ -193,14 +208,14 @@ in {
         cp '${configFile}' '${cfg.dataDir}/elements.conf'
         chmod o-rw  '${cfg.dataDir}/elements.conf'
         chown -R '${cfg.user}:${cfg.group}' '${cfg.dataDir}'
-        echo "rpcpassword=$(cat /secrets/liquid-rpcpassword)" >> '${cfg.dataDir}/elements.conf'
+        echo "rpcpassword=$(cat ${secretsDir}/liquid-rpcpassword)" >> '${cfg.dataDir}/elements.conf'
-        echo "mainchainrpcpassword=$(cat /secrets/bitcoin-rpcpassword)" >> '${cfg.dataDir}/elements.conf'
+        echo "mainchainrpcpassword=$(cat ${secretsDir}/bitcoin-rpcpassword)" >> '${cfg.dataDir}/elements.conf'
       '';
       serviceConfig = {
         Type = "simple";
         User = "${cfg.user}";
         Group = "${cfg.group}";
-        ExecStart = "${pkgs.elementsd}/bin/elementsd ${cmdlineOptions}";
+        ExecStart = "${pkgs.nix-bitcoin.elementsd}/bin/elementsd ${cmdlineOptions}";
         StateDirectory = "liquidd";
         PIDFile = "${pidFile}";
         Restart = "on-failure";
@@ -214,14 +229,11 @@ in {
         );
     };
     users.users.${cfg.user} = {
-      name = cfg.user;
       group = cfg.group;
-      extraGroups = [ "keys" ];
       description = "Liquid sidechain user";
       home = cfg.dataDir;
     };
-    users.groups.${cfg.group} = {
+    users.groups.${cfg.group} = {};
-      name = cfg.group;
+    nix-bitcoin.secrets.liquid-rpcpassword.user = "liquid";
-    };
   };
 }

modules/lnd.nix

@@ -3,14 +3,17 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.lnd;
+  inherit (config) nix-bitcoin-services;
+  secretsDir = config.nix-bitcoin.secretsDir;
   configFile = pkgs.writeText "lnd.conf" ''
     datadir=${cfg.dataDir}
     logdir=${cfg.dataDir}/logs
     bitcoin.mainnet=1
-    tlscertpath=/secrets/lnd_cert
+    tlscertpath=${secretsDir}/lnd-cert
-    tlskeypath=/secrets/lnd_key
+    tlskeypath=${secretsDir}/lnd-key
+
+    rpclisten=localhost:${toString cfg.rpcPort}
 
     bitcoin.active=1
     bitcoin.node=bitcoind
@@ -26,45 +29,6 @@ let
 
     ${cfg.extraConfig}
   '';
-  init-lnd-wallet-script = pkgs.writeScript "init-lnd-wallet.sh" ''
-#!/bin/sh
-
-set -e
-umask 377
-
-${pkgs.coreutils}/bin/sleep 5
-
-if [ ! -f /secrets/lnd-seed-mnemonic ]
-then
-  ${pkgs.coreutils}/bin/echo Creating lnd seed
-
-  ${pkgs.curl}/bin/curl -s \
-  --cacert /secrets/lnd_cert \
-  -X GET https://127.0.0.1:8080/v1/genseed | ${pkgs.jq}/bin/jq -c '.cipher_seed_mnemonic' > /secrets/lnd-seed-mnemonic
-fi
-
-if [ ! -f ${cfg.dataDir}/chain/bitcoin/mainnet/wallet.db ]
-then
-  ${pkgs.coreutils}/bin/echo Creating lnd wallet
-
-  ${pkgs.curl}/bin/curl -s \
-  --cacert /secrets/lnd_cert \
-  -X POST -d "{\"wallet_password\": \"$(${pkgs.coreutils}/bin/cat /secrets/lnd-wallet-password | ${pkgs.coreutils}/bin/tr -d '\n' | ${pkgs.coreutils}/bin/base64 -w0)\", \
-  \"cipher_seed_mnemonic\": $(${pkgs.coreutils}/bin/cat /secrets/lnd-seed-mnemonic | ${pkgs.coreutils}/bin/tr -d '\n')}" \
-  https://127.0.0.1:8080/v1/initwallet
-else
-  ${pkgs.coreutils}/bin/echo Unlocking lnd wallet
-
-  ${pkgs.curl}/bin/curl -s \
-      -H "Grpc-Metadata-macaroon: $(${pkgs.xxd}/bin/xxd -ps -u -c 99999 ${cfg.dataDir}/chain/bitcoin/mainnet/admin.macaroon)" \
-      --cacert /secrets/lnd_cert \
-      -X POST \
-      -d "{\"wallet_password\": \"$(${pkgs.coreutils}/bin/cat /secrets/lnd-wallet-password | ${pkgs.coreutils}/bin/tr -d '\n' | ${pkgs.coreutils}/bin/base64 -w0)\"}" \
-      https://127.0.0.1:8080/v1/unlockwallet
-fi
-
-exit 0
-'';
 in {
 
   options.services.lnd = {
@@ -80,6 +44,11 @@ in {
       default = "/var/lib/lnd";
       description = "The data directory for LND.";
     };
+    rpcPort = mkOption {
+      type = types.ints.u16;
+      default = 10009;
+      description = "Port on which to listen for gRPC connections.";
+    };
     extraConfig = mkOption {
         type = types.lines;
         default = "";
@@ -88,23 +57,23 @@ in {
         '';
         description = "Additional configurations to be appended to <filename>lnd.conf</filename>.";
       };
+    cli = mkOption {
+      readOnly = true;
+      default = pkgs.writeScriptBin "lncli"
+      # Switch user because lnd makes datadir contents readable by user only
+      ''
+        exec sudo -u lnd ${pkgs.nix-bitcoin.lnd}/bin/lncli --tlscertpath ${secretsDir}/lnd-cert \
+          --macaroonpath '${cfg.dataDir}/chain/bitcoin/mainnet/admin.macaroon' "$@"
+      '';
+      description = "Binary to connect with the lnd instance.";
+    };
     enforceTor =  nix-bitcoin-services.enforceTor;
   };
 
   config = mkIf cfg.enable {
-    users.users.lnd = {
-        description = "LND User";
-        group = "lnd";
-        extraGroups = [ "bitcoinrpc" "keys" ];
-        home = cfg.dataDir;
-    };
-    users.groups.lnd = {
-      name = "lnd";
-    };
-
     systemd.services.lnd = {
       description = "Run LND";
-      path  = [ pkgs.blockchains.bitcoind ];
+      path  = [ pkgs.nix-bitcoin.bitcoind ];
       wantedBy = [ "multi-user.target" ];
       requires = [ "bitcoind.service" ];
       after = [ "bitcoind.service" ];
@@ -113,12 +82,11 @@ in {
         cp ${configFile} ${cfg.dataDir}/lnd.conf
         chown -R 'lnd:lnd' '${cfg.dataDir}'
         chmod u=rw,g=r,o= ${cfg.dataDir}/lnd.conf
-        echo "bitcoind.rpcpass=$(cat /secrets/bitcoin-rpcpassword)" >> '${cfg.dataDir}/lnd.conf'
+        echo "bitcoind.rpcpass=$(cat ${secretsDir}/bitcoin-rpcpassword)" >> '${cfg.dataDir}/lnd.conf'
       '';
       serviceConfig = {
         PermissionsStartOnly = "true";
-        ExecStart = "${pkgs.lnd}/bin/lnd --configfile=${cfg.dataDir}/lnd.conf";
+        ExecStart = "${pkgs.nix-bitcoin.lnd}/bin/lnd --configfile=${cfg.dataDir}/lnd.conf";
-        ExecStartPost = "${pkgs.bash}/bin/bash ${init-lnd-wallet-script}";
         User = "lnd";
         Restart = "on-failure";
         RestartSec = "10s";
@@ -127,6 +95,67 @@ in {
           then nix-bitcoin-services.allowTor
           else nix-bitcoin-services.allowAnyIP
         ) // nix-bitcoin-services.allowAnyProtocol;  # For ZMQ
+      postStart = let
+        mainnetDir = "${cfg.dataDir}/chain/bitcoin/mainnet";
+      in ''
+        umask 377
+
+        attempts=50
+        while ! { exec 3>/dev/tcp/127.0.0.1/8080 && exec 3>&-; } &>/dev/null; do
+              ((attempts-- == 0)) && { echo "lnd REST service unreachable"; exit 1; }
+              sleep 0.1
+        done
+
+        if [[ ! -f ${secretsDir}/lnd-seed-mnemonic ]]; then
+          echo Create lnd seed
+
+          ${pkgs.curl}/bin/curl -s \
+            --cacert ${secretsDir}/lnd-cert \
+            -X GET https://127.0.0.1:8080/v1/genseed | ${pkgs.jq}/bin/jq -c '.cipher_seed_mnemonic' > ${secretsDir}/lnd-seed-mnemonic
+        fi
+
+        if [[ ! -f ${mainnetDir}/wallet.db ]]; then
+          echo Create lnd wallet
+
+          ${pkgs.curl}/bin/curl -s --output /dev/null --show-error \
+            --cacert ${secretsDir}/lnd-cert \
+            -X POST -d "{\"wallet_password\": \"$(cat ${secretsDir}/lnd-wallet-password | tr -d '\n' | base64 -w0)\", \
+            \"cipher_seed_mnemonic\": $(cat ${secretsDir}/lnd-seed-mnemonic | tr -d '\n')}" \
+            https://127.0.0.1:8080/v1/initwallet
+
+          # Guarantees that RPC calls with cfg.cli succeed after the service is started
+          echo Wait until wallet is created
+          while [[ ! -f ${mainnetDir}/admin.macaroon ]]; do
+            sleep 0.1
+          done
+        else
+          echo Unlock lnd wallet
+
+          ${pkgs.curl}/bin/curl -s \
+            -H "Grpc-Metadata-macaroon: $(${pkgs.xxd}/bin/xxd -ps -u -c 99999 '${mainnetDir}/admin.macaroon')" \
+            --cacert ${secretsDir}/lnd-cert \
+            -X POST \
+            -d "{\"wallet_password\": \"$(cat ${secretsDir}/lnd-wallet-password | tr -d '\n' | base64 -w0)\"}" \
+            https://127.0.0.1:8080/v1/unlockwallet
+        fi
+
+        # Wait until the RPC port is open
+        while ! { exec 3>/dev/tcp/127.0.0.1/${toString cfg.rpcPort}; } &>/dev/null; do
+          sleep 0.1
+        done
+      '';
+    };
+    users.users.lnd = {
+      description = "LND User";
+      group = "lnd";
+      extraGroups = [ "bitcoinrpc" ];
+      home = cfg.dataDir;
+    };
+    users.groups.lnd = {};
+    nix-bitcoin.secrets = {
+      lnd-wallet-password.user = "lnd";
+      lnd-key.user = "lnd";
+      lnd-cert.user = "lnd";
     };
   };
 }

modules/modules.nix

@@ -0,0 +1,43 @@
+{ config, pkgs, lib, ... }:
+let
+  nixpkgs-pinned = import ../pkgs/nixpkgs-pinned.nix;
+  unstable = import nixpkgs-pinned.nixpkgs-unstable {};
+
+  allPackages = pkgs: (import ../pkgs { inherit pkgs; }) // {
+    bitcoin = unstable.bitcoin.override { miniupnpc = null; };
+    bitcoind = unstable.bitcoind.override { miniupnpc = null; };
+    clightning = unstable.clightning;
+    lnd = unstable.lnd;
+  };
+in {
+  imports = [
+    ./bitcoind.nix
+    ./clightning.nix
+    ./lightning-charge.nix
+    ./nanopos.nix
+    ./nix-bitcoin-webindex.nix
+    ./liquid.nix
+    ./spark-wallet.nix
+    ./electrs.nix
+    ./onion-chef.nix
+    ./recurring-donations.nix
+    ./hardware-wallets.nix
+    ./lnd.nix
+    ./secrets/secrets.nix
+  ];
+
+  disabledModules = [ "services/networking/bitcoind.nix" ];
+
+  options = {
+    nix-bitcoin-services = lib.mkOption {
+      readOnly = true;
+      default = import ./nix-bitcoin-services.nix lib;
+    };
+  };
+
+  config = {
+    nixpkgs.overlays = [ (self: super: {
+      nix-bitcoin = allPackages super;
+    }) ];
+  };
+}

modules/nanopos.nix

@@ -3,8 +3,8 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.nanopos;
+  inherit (config) nix-bitcoin-services;
   defaultItemsFile = pkgs.writeText "items.yaml" ''
     tea:
       price: 0.02 # denominated in the currency specified by --currency
@@ -52,24 +52,14 @@ in {
   };
 
   config = mkIf cfg.enable {
-    users.users.nanopos =
-      {
-        description = "nanopos User";
-        group = "nanopos";
-        extraGroups = [ "keys" ];
-    };
-    users.groups.nanopos = {
-      name = "nanopos";
-    };
-
     systemd.services.nanopos = {
       description = "Run nanopos";
       wantedBy = [ "multi-user.target" ];
       requires = [ "lightning-charge.service" ];
       after = [ "lightning-charge.service" ];
       serviceConfig = {
-        EnvironmentFile = "/secrets/lightning-charge-api-token-for-nanopos";
+        EnvironmentFile = "${config.nix-bitcoin.secretsDir}/nanopos-env";
-        ExecStart = "${pkgs.nanopos}/bin/nanopos -y ${cfg.itemsFile} -p ${toString cfg.port} --show-bolt11";
+        ExecStart = "${pkgs.nix-bitcoin.nanopos}/bin/nanopos -y ${cfg.itemsFile} -p ${toString cfg.port} --show-bolt11";
 
         User = "nanopos";
         Restart = "on-failure";
@@ -78,5 +68,11 @@ in {
         // nix-bitcoin-services.nodejs
         // nix-bitcoin-services.allowTor;
     };
+    users.users.nanopos = {
+      description = "nanopos User";
+      group = "nanopos";
+    };
+    users.groups.nanopos = {};
+    nix-bitcoin.secrets.nanopos-env.user = "nanopos";
   };
 }

modules/nix-bitcoin-pkgs.nix

@@ -1,17 +0,0 @@
-{ config, pkgs, ... }:
-let
-  nixpkgs-pinned = import ../pkgs/nixpkgs-pinned.nix;
-  nixpkgs-unstable = import nixpkgs-pinned.nixpkgs-unstable { };
-in {
-  disabledModules = [ "services/networking/bitcoind.nix" ];
-
-  nixpkgs.overlays = [ (import ../overlay.nix) ];
-
-  nixpkgs.config.packageOverrides = pkgs: {
-    # Use bitcoin and clightning from unstable
-    bitcoin = nixpkgs-unstable.bitcoin.override { miniupnpc = null; };
-    blockchains.bitcoind = nixpkgs-unstable.bitcoind.override { miniupnpc = null; };
-    clightning = nixpkgs-unstable.clightning.override { };
-    lnd = nixpkgs-unstable.lnd.override { };
-  };
-}

modules/nix-bitcoin-services.nix

@@ -1,11 +1,10 @@
 # See `man systemd.exec` and `man systemd.resource-control` for an explanation
 # of the various systemd options available through this module.
 
-{ config, lib, pkgs, ... }:
+lib:
 
 with lib;
-
+{
-let
   defaultHardening = {
       PrivateTmp = "true";
       ProtectSystem = "full";
@@ -23,9 +22,7 @@ let
       SystemCallFilter= "accept accept4 access adjtimex alarm bind brk capget capset chdir chmod chown chown32 clock_getres clock_gettime clock_nanosleep close connect copy_file_range creat dup dup2 dup3 epoll_create epoll_create1 epoll_ctl epoll_ctl_old epoll_pwait epoll_wait epoll_wait_old eventfd eventfd2 execve execveat exit exit_group faccessat fadvise64 fadvise64_64 fallocate fanotify_mark fchdir fchmod fchmodat fchown fchown32 fchownat fcntl fcntl64 fdatasync fgetxattr flistxattr flock fork fremovexattr fsetxattr fstat fstat64 fstatat64 fstatfs fstatfs64 fsync ftruncate ftruncate64 futex futimesat getcpu getcwd getdents getdents64 getegid getegid32 geteuid geteuid32 getgid getgid32 getgroups getgroups32 getitimer getpeername getpgid getpgrp getpid getppid getpriority getrandom getresgid getresgid32 getresuid getresuid32 getrlimit get_robust_list getrusage getsid getsockname getsockopt get_thread_area gettid gettimeofday getuid getuid32 getxattr inotify_add_watch inotify_init inotify_init1 inotify_rm_watch io_cancel ioctl io_destroy io_getevents io_pgetevents ioprio_get ioprio_set io_setup io_submit ipc kill lchown lchown32 lgetxattr link linkat listen listxattr llistxattr _llseek lremovexattr lseek lsetxattr lstat lstat64 madvise memfd_create mincore mkdir mkdirat mknod mknodat mlock mlock2 mlockall mmap mmap2 mprotect mq_getsetattr mq_notify mq_open mq_timedreceive mq_timedsend mq_unlink mremap msgctl msgget msgrcv msgsnd msync munlock munlockall munmap nanosleep newfstatat _newselect open openat pause pipe pipe2 poll ppoll prctl pread64 preadv preadv2 prlimit64 pselect6 pwrite64 pwritev pwritev2 read readahead readlink readlinkat readv recv recvfrom recvmmsg recvmsg remap_file_pages removexattr rename renameat renameat2 restart_syscall rmdir rt_sigaction rt_sigpending rt_sigprocmask rt_sigqueueinfo rt_sigreturn rt_sigsuspend rt_sigtimedwait rt_tgsigqueueinfo sched_getaffinity sched_getattr sched_getparam sched_get_priority_max sched_get_priority_min sched_getscheduler sched_rr_get_interval sched_setaffinity sched_setattr sched_setparam sched_setscheduler sched_yield seccomp select semctl semget semop semtimedop send sendfile sendfile64 sendmmsg sendmsg sendto setfsgid setfsgid32 setfsuid setfsuid32 setgid setgid32 setgroups setgroups32 setitimer setpgid setpriority setregid setregid32 setresgid setresgid32 setresuid setresuid32 setreuid setreuid32 setrlimit set_robust_list setsid setsockopt set_thread_area set_tid_address setuid setuid32 setxattr shmat shmctl shmdt shmget shutdown sigaltstack signalfd signalfd4 sigreturn socket socketcall socketpair splice stat stat64 statfs statfs64 statx symlink symlinkat sync sync_file_range syncfs sysinfo tee tgkill time timer_create timer_delete timerfd_create timerfd_gettime timerfd_settime timer_getoverrun timer_gettime timer_settime times tkill truncate truncate64 ugetrlimit umask uname unlink unlinkat utime utimensat utimes vfork vmsplice wait4 waitid waitpid write writev arm_fadvise64_64 arm_sync_file_range sync_file_range2 breakpoint cacheflush set_tls arch_prctl modify_ldt clone";
       SystemCallArchitectures= "native";
   };
-in
+
-{
-  inherit defaultHardening;
   # nodejs applications apparently rely on memory write execute
   nodejs = { MemoryDenyWriteExecute = "false"; };
   # Allow tor traffic. Allow takes precedence over Deny.

modules/nix-bitcoin-webindex.nix

@@ -3,8 +3,8 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.nix-bitcoin-webindex;
+  inherit (config) nix-bitcoin-services;
   indexFile = pkgs.writeText "index.html" ''
     <html>
       <body>
@@ -74,7 +74,13 @@ in {
       description = "Get node info";
       wantedBy = [ "multi-user.target" ];
       after = [ "nodeinfo.service" ];
-      path  = [ pkgs.nodeinfo pkgs.clightning pkgs.jq pkgs.sudo ];
+      path  = with pkgs; [
+        nix-bitcoin.nodeinfo
+        config.services.clightning.cli
+        config.services.lnd.cli
+        jq
+        sudo
+      ];
       serviceConfig = {
         ExecStart="${pkgs.bash}/bin/bash ${createWebIndex}";
         User = "root";

modules/nix-bitcoin.nix

@@ -15,21 +15,7 @@ let
     chown -R operator ${config.users.users.operator.home}/.ssh
   '';
 in {
-  imports = [
+  imports = [ ./modules.nix ];
-    ./nix-bitcoin-pkgs.nix
-    ./bitcoind.nix
-    ./clightning.nix
-    ./lightning-charge.nix
-    ./nanopos.nix
-    ./nix-bitcoin-webindex.nix
-    ./liquid.nix
-    ./spark-wallet.nix
-    ./electrs.nix
-    ./onion-chef.nix
-    ./recurring-donations.nix
-    ./hardware-wallets.nix
-    ./lnd.nix
-  ];
 
   options.services.nix-bitcoin = {
     enable = mkOption {
@@ -42,6 +28,8 @@ in {
   };
 
   config = mkIf cfg.enable {
+    nix-bitcoin.secretsDir = mkDefault "/secrets";
+
     networking.firewall.enable = true;
 
     # Tor
@@ -85,9 +73,6 @@ in {
       version = 3;
     };
 
-    # Add bitcoinrpc group
-    users.groups.bitcoinrpc = {};
-
     # clightning
     services.clightning.bitcoin-rpcuser = config.services.bitcoind.rpcuser;
     services.clightning.proxy = config.services.tor.client.socksListenAddress;
@@ -118,29 +103,15 @@ in {
     services.onion-chef.enable = true;
     services.onion-chef.access.operator = [ "bitcoind" "clightning" "nginx" "liquidd" "spark-wallet" "electrs" "sshd" ];
 
-    environment.interactiveShellInit = ''
-      ${optionalString (config.services.clightning.enable) ''
-        alias lightning-cli='sudo -u clightning lightning-cli --lightning-dir=${config.services.clightning.dataDir}'
-      ''}
-      ${optionalString (config.services.lnd.enable) ''
-        alias lncli='sudo -u lnd lncli --tlscertpath /secrets/lnd_cert --macaroonpath ${config.services.lnd.dataDir}/chain/bitcoin/mainnet/admin.macaroon'
-      ''}
-      ${optionalString (config.services.liquidd.enable) ''
-        alias elements-cli='elements-cli -datadir=${config.services.liquidd.dataDir}'
-        alias liquidswap-cli='liquidswap-cli -c ${config.services.liquidd.dataDir}/elements.conf'
-      ''}
-    '';
     # Unfortunately c-lightning doesn't allow setting the permissions of the rpc socket
     # https://github.com/ElementsProject/lightning/issues/1366
-    security.sudo.configFile = (
+    security.sudo.configFile =
-      if config.services.clightning.enable then ''
+     (optionalString config.services.clightning.enable ''
-        operator    ALL=(clightning) NOPASSWD: ALL
+       operator    ALL=(clightning) NOPASSWD: ALL
-      ''
+     '') +
-      else if config.services.lnd.enable then ''
+     (optionalString config.services.lnd.enable ''
-        operator    ALL=(lnd) NOPASSWD: ALL
+       operator    ALL=(lnd) NOPASSWD: ALL
-      ''
+     '');
-      else ""
-    );
 
     # Give root ssh access to the operator account
     systemd.services.copy-root-authorized-keys = {
@@ -184,30 +155,32 @@ in {
       }];
       version = 3;
     };
-    environment.systemPackages = with pkgs; [
+    environment.systemPackages = with pkgs; with nix-bitcoin; let
+      s = config.services;
+    in
+    [
       tor
-      blockchains.bitcoind
+      bitcoind
-      (hiPrio config.services.bitcoind.cli)
+      (hiPrio s.bitcoind.cli)
       nodeinfo
       jq
       qrencode
     ]
-    ++ optionals config.services.clightning.enable [clightning]
+    ++ optionals s.clightning.enable [clightning (hiPrio s.clightning.cli)]
-    ++ optionals config.services.lnd.enable [lnd]
+    ++ optionals s.lnd.enable [lnd (hiPrio s.lnd.cli)]
-    ++ optionals config.services.lightning-charge.enable [lightning-charge]
+    ++ optionals s.lightning-charge.enable [lightning-charge]
-    ++ optionals config.services.nanopos.enable [nanopos]
+    ++ optionals s.nanopos.enable [nanopos]
-    ++ optionals config.services.nix-bitcoin-webindex.enable [nginx]
+    ++ optionals s.nix-bitcoin-webindex.enable [nginx]
-    ++ optionals config.services.liquidd.enable [elementsd liquid-swap]
+    ++ optionals s.liquidd.enable [elementsd (hiPrio s.liquidd.cli) (hiPrio s.liquidd.swap-cli)]
-    ++ optionals config.services.spark-wallet.enable [spark-wallet]
+    ++ optionals s.spark-wallet.enable [spark-wallet]
-    ++ optionals config.services.electrs.enable [electrs]
+    ++ optionals s.electrs.enable [electrs]
-    ++ optionals (config.services.hardware-wallets.ledger || config.services.hardware-wallets.trezor) [
+    ++ optionals (s.hardware-wallets.ledger || s.hardware-wallets.trezor) [
         hwi
-        # To allow debugging issues with lsusb:
+        # To allow debugging issues with lsusb
         usbutils
     ]
-    ++ optionals config.services.hardware-wallets.trezor [
+    ++ optionals s.hardware-wallets.trezor [
         python3.pkgs.trezor
     ];
   };
 }
-

modules/onion-chef.nix

@@ -8,8 +8,8 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.onion-chef;
+  inherit (config) nix-bitcoin-services;
   dataDir = "/var/lib/onion-chef/";
   onion-chef-script = pkgs.writeScript "onion-chef.sh" ''
     # wait until tor is up
@@ -70,14 +70,13 @@ in {
   config = mkIf cfg.enable {
     systemd.services.onion-chef = {
       description = "Run onion-chef";
-      wantedBy = [ "multi-user.target" ];
+      wantedBy = [ "tor.service" ];
-      requires = [ "tor.service" ];
+      bindsTo = [ "tor.service" ];
-      partOf = [ "tor.service" ];
       after = [ "tor.service" ];
       serviceConfig = {
         ExecStart = "${pkgs.bash}/bin/bash ${onion-chef-script}";
-        User = "root";
         Type = "oneshot";
+        RemainAfterExit = true;
       } // nix-bitcoin-services.defaultHardening;
     };
   };

modules/recurring-donations.nix

@@ -3,10 +3,10 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.recurring-donations;
+  inherit (config) nix-bitcoin-services;
   recurring-donations-script = pkgs.writeScript "recurring-donations.sh" ''
-    LNCLI="lightning-cli --lightning-dir=${config.services.clightning.dataDir}"
+    LNCLI="${pkgs.nix-bitcoin.clightning}/bin/lightning-cli --lightning-dir=${config.services.clightning.dataDir}"
     pay_tallycoin() {
       NAME=$1
       AMOUNT=$2
@@ -82,7 +82,7 @@ in {
       description = "Run recurring-donations";
       requires = [ "clightning.service" ];
       after = [ "clightning.service" ];
-      path  = [ pkgs.clightning pkgs.curl pkgs.torsocks pkgs.sudo pkgs.jq ];
+      path = with pkgs; [ nix-bitcoin.clightning curl torsocks sudo jq ];
       serviceConfig = {
         ExecStart = "${pkgs.bash}/bin/bash ${recurring-donations-script}";
         # TODO: would be better if this was operator, but I don't get sudo

modules/secrets/generate-secrets.nix

@@ -0,0 +1,26 @@
+{ config, pkgs, lib, ... }:
+
+# This is mainly for testing.
+# When using this for regular deployments, make sure to create a backup of the
+# generated secrets.
+
+with lib;
+{
+  nix-bitcoin.setup-secrets = true;
+
+  systemd.services.generate-secrets = {
+    requiredBy = [ "setup-secrets.service" ];
+    before = [ "setup-secrets.service" ];
+    serviceConfig = {
+       Type = "oneshot";
+       RemainAfterExit = true;
+    } // config.nix-bitcoin-services.defaultHardening;
+    script = ''
+      mkdir -p "${config.nix-bitcoin.secretsDir}"
+      cd "${config.nix-bitcoin.secretsDir}"
+      chown root: .
+      chmod 0700 .
+      ${pkgs.nix-bitcoin.generate-secrets}
+    '';
+  };
+}

modules/secrets/secrets.nix

@@ -0,0 +1,97 @@
+{ config, pkgs, lib, ... }:
+
+with lib;
+let
+  cfg = config.nix-bitcoin;
+  setupSecrets = concatStrings (mapAttrsToList (n: v: ''
+    setupSecret ${n} ${v.user} ${v.group} ${v.permissions} }
+  '') cfg.secrets);
+in
+{
+  options.nix-bitcoin = {
+    secretsDir = mkOption {
+      type = types.path;
+      default = "/etc/nix-bitcoin-secrets";
+      description = "Directory to store secrets";
+    };
+
+    secrets = mkOption {
+      default = {};
+      type = with types; attrsOf (submodule (
+        { config, ... }: {
+          options = {
+            user = mkOption {
+              type = str;
+              default = "root";
+            };
+            group = mkOption {
+              type = str;
+              default = config.user;
+            };
+            permissions = mkOption {
+              type = str;
+              default = "0440";
+            };
+          };
+        }
+      ));
+    };
+
+    setup-secrets = mkEnableOption "Set permissions for secrets generated by 'generate-secrets.sh'";
+  };
+
+  config = mkIf cfg.setup-secrets {
+    systemd.targets.nix-bitcoin-secrets = {
+      requires = [ "setup-secrets.service" ];
+      after = [ "setup-secrets.service" ];
+    };
+
+    # Operation of this service:
+    #  - Create missing secrets that are composed of attrs from secrets.nix
+    #  - Set owner and permissions for all used secrets
+    #  - Make all other secrets accessible to root only
+    # For all steps make sure that no secrets are copied to the nix store.
+    #
+    systemd.services.setup-secrets = {
+      serviceConfig = {
+         Type = "oneshot";
+         RemainAfterExit = true;
+      } // config.nix-bitcoin-services.defaultHardening;
+      script = ''
+        setupSecret() {
+            file="$1"
+            user="$2"
+            group="$3"
+            permissions="$4"
+            if [[ ! -e $file ]]; then
+              echo "Error: Secret file '$file' is missing"
+              exit 1
+            fi
+            chown "$user:$group" "$file"
+            chmod "$permissions" "$file"
+            processedFiles+=("$file")
+        }
+
+        dir="${cfg.secretsDir}"
+        if [[ ! -e $dir ]]; then
+          echo "Error: Secrets dir '$dir' is missing"
+          exit 1
+        fi
+        chown root: "$dir"
+        cd "$dir"
+
+        processedFiles=()
+        ${setupSecrets}
+
+        # Make all other files accessible to root only
+        unprocessedFiles=$(comm -23 <(printf '%s\n' *) <(printf '%s\n' "''${processedFiles[@]}" | sort))
+        IFS=$'\n'
+        chown root: $unprocessedFiles
+        chmod 0440 $unprocessedFiles
+
+        # Now make the secrets dir accessible to other users
+        chmod 0751 "$dir"
+      '';
+    };
+  };
+}

modules/spark-wallet.nix

@@ -3,12 +3,12 @@
 with lib;
 
 let
-  nix-bitcoin-services = pkgs.callPackage ./nix-bitcoin-services.nix { };
   cfg = config.services.spark-wallet;
+  inherit (config) nix-bitcoin-services;
   dataDir = "/var/lib/spark-wallet/";
   onion-chef-service = (if cfg.onion-service then [ "onion-chef.service" ] else []);
   run-spark-wallet = pkgs.writeScript "run-spark-wallet" ''
-    CMD="${pkgs.spark-wallet}/bin/spark-wallet --ln-path ${cfg.ln-path} -Q -k -c /secrets/spark-wallet-login"
+    CMD="${pkgs.nix-bitcoin.spark-wallet}/bin/spark-wallet --ln-path ${cfg.ln-path} -Q -k -c ${config.nix-bitcoin.secretsDir}/spark-wallet-login"
     ${optionalString cfg.onion-service
       ''
       echo Getting onion hostname
@@ -73,5 +73,6 @@ in {
         // nix-bitcoin-services.nodejs
         // nix-bitcoin-services.allowTor;
     };
+    nix-bitcoin.secrets.spark-wallet-login.user = "clightning";
   };
 }

network/network.nix

@@ -1,92 +1,31 @@
-let
+{
-  secrets = import ../secrets/secrets.nix;
-  bitcoin-rpcpassword = {
-    text = secrets.bitcoinrpcpassword;
-    destDir = "/secrets/";
-    user = "bitcoin";
-    group = "bitcoinrpc";
-    permissions = "0440";
-  };
-  lnd-wallet-password = {
-    text = secrets.lnd-wallet-password;
-    destDir = "/secrets/";
-    user = "lnd";
-    group = "lnd";
-    permissions = "0440";
-  };
-  lightning-charge-api-token = {
-    text = "API_TOKEN=" + secrets.lightning-charge-api-token;
-    destDir = "/secrets/";
-    user = "clightning";
-    group = "clightning";
-    permissions = "0440";
-  };
-  # variable is called CHARGE_TOKEN instead of API_TOKEN
-  lightning-charge-api-token-for-nanopos = {
-    text = "CHARGE_TOKEN=" + secrets.lightning-charge-api-token;
-    destDir = "/secrets/";
-    user = "nanopos";
-    group = "nanopos";
-    permissions = "0440";
-  };
-  liquid-rpcpassword = {
-    text = secrets.liquidrpcpassword;
-    destDir = "/secrets/";
-    user = "liquid";
-    group = "liquid";
-    permissions = "0440";
-  };
-  spark-wallet-login = {
-    text = "login=" + "spark-wallet:" + secrets.spark-wallet-password;
-    destDir = "/secrets/";
-    user = "clightning";
-    group = "clightning";
-    permissions = "0440";
-  };
-  nginx_key = {
-    keyFile = ../secrets/nginx.key;
-    destDir = "/secrets/";
-    user = "nginx";
-    group = "root";
-    permissions = "0440";
-  };
-  nginx_cert = {
-    keyFile = ../secrets/nginx.cert;
-    destDir = "/secrets/";
-    user = "nginx";
-    group = "root";
-    permissions = "0440";
-  };
-  lnd_key = {
-    keyFile = ../secrets/lnd.key;
-    destDir = "/secrets/";
-    user = "lnd";
-    group = "lnd";
-    permissions = "0440";
-  };
-  lnd_cert = {
-    keyFile = ../secrets/lnd.cert;
-    destDir = "/secrets/";
-    user = "lnd";
-    group = "lnd";
-    permissions = "0440";
-  };
-in {
   network.description = "Bitcoin Core node";
 
   bitcoin-node =
-    { config, pkgs, ... }:
+    { config, pkgs, lib, ... }: {
-    let
+      imports = [ ../configuration.nix ];
-      bitcoin-node = import ../configuration.nix;
+
-    in {
+      deployment.keys = builtins.mapAttrs (n: v: {
-      deployment.keys = {
+        keyFile = "${toString ../secrets}/${n}";
-        inherit bitcoin-rpcpassword;
+        destDir = config.nix-bitcoin.secretsDir;
-      }
+        inherit (v) user group permissions;
-      // (if (config.services.lnd.enable) then { inherit lnd-wallet-password lnd_key lnd_cert; } else { })
+      }) config.nix-bitcoin.secrets;
-      // (if (config.services.lightning-charge.enable) then { inherit lightning-charge-api-token; } else { })
+
-      // (if (config.services.nanopos.enable) then { inherit lightning-charge-api-token-for-nanopos; } else { })
+      # nixops makes the secrets directory accessible only for users with group 'key'.
-      // (if (config.services.liquidd.enable) then { inherit liquid-rpcpassword; } else { })
+      # For compatibility with other deployment methods besides nixops, we forego the
-      // (if (config.services.spark-wallet.enable) then { inherit spark-wallet-login; } else { })
+      # use of the 'key' group and make the secrets dir world-readable instead.
-      // (if (config.services.electrs.enable) then { inherit nginx_key nginx_cert; } else { });
+      # This is safe because all containing files have their specific private
-    } // (bitcoin-node { inherit config pkgs; });
+      # permissions set.
+      systemd.services.allowSecretsDirAccess = {
+        requires = [ "keys.target" ];
+        after = [ "keys.target" ];
+        script = "chmod o+x ${config.nix-bitcoin.secretsDir}";
+        serviceConfig.Type = "oneshot";
+      };
+
+      systemd.targets.nix-bitcoin-secrets = {
+        requires = [ "allowSecretsDirAccess.service" ];
+        after = [ "allowSecretsDirAccess.service" ];
+      };
+    };
 }

overlay.nix

@@ -1,21 +1 @@
-# You can use this file as a nixpkgs overlay.
+self: super: import ./pkgs { pkgs = super; }
-# It's useful in the case where you don't want to add the whole NUR namespace
-# to your configuration.
-
-self: super:
-
-let filterSet =
-      (f: g: s: builtins.listToAttrs
-        (map
-          (n: { name = n; value = builtins.getAttr n s; })
-          (builtins.filter
-            (n: f n && g (builtins.getAttr n s))
-            (builtins.attrNames s)
-          )
-        )
-      );
-in filterSet
-     (n: !(n=="lib"||n=="overlays"||n=="modules")) # filter out non-packages
-     (p: true) # all packages are ok
-     (import ./default.nix { pkgs = super; })
-

pkgs/default.nix

@@ -0,0 +1,13 @@
+{ pkgs ? import <nixpkgs> {} }:
+{
+  nodeinfo = pkgs.callPackage ./nodeinfo { };
+  lightning-charge = pkgs.callPackage ./lightning-charge { };
+  nanopos = pkgs.callPackage ./nanopos { };
+  spark-wallet = pkgs.callPackage ./spark-wallet { };
+  electrs = (pkgs.callPackage ./electrs { }).rootCrate.build;
+  elementsd = pkgs.callPackage ./elementsd { withGui = false; };
+  hwi = pkgs.callPackage ./hwi { };
+  pylightning = pkgs.python3Packages.callPackage ./pylightning { };
+  liquid-swap = pkgs.python3Packages.callPackage ./liquid-swap { };
+  generate-secrets = pkgs.callPackage ./generate-secrets { };
+}

pkgs/generate-secrets/default.nix

@@ -0,0 +1,6 @@
+{ pkgs }: with pkgs;
+
+writeScript "generate-secrets" ''
+  export PATH=${lib.makeBinPath [ coreutils apg openssl ]}
+  . ${./generate-secrets.sh} ${./openssl.cnf}
+''

pkgs/generate-secrets/generate-secrets.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+opensslConf=${1:-openssl.cnf}
+
+makePasswordSecret() {
+    [[ -e $1 ]] || apg -m 20 -x 20 -M Ncl -n 1 > "$1"
+}
+
+makePasswordSecret bitcoin-rpcpassword
+makePasswordSecret lnd-wallet-password
+makePasswordSecret liquid-rpcpassword
+makePasswordSecret lightning-charge-token
+makePasswordSecret spark-wallet-password
+
+[[ -e lightning-charge-env ]] || echo "API_TOKEN=$(cat lightning-charge-token)" > lightning-charge-env
+[[ -e nanopos-env          ]] || echo "CHARGE_TOKEN=$(cat lightning-charge-token)" > nanopos-env
+[[ -e spark-wallet-login   ]] || echo "login=spark-wallet:$(cat spark-wallet-password)" > spark-wallet-login
+
+if [[ ! -e nginx-key || ! -e nginx-cert ]]; then
+    openssl genrsa -out nginx-key 2048
+    openssl req -new -key nginx-key -out nginx.csr -subj "/C=KN"
+    openssl x509 -req -days 1825 -in nginx.csr -signkey nginx-key -out nginx-cert
+    rm nginx.csr
+fi
+
+if [[ ! -e lnd-key || ! -e lnd-cert ]]; then
+    openssl ecparam -genkey -name prime256v1 -out lnd-key
+    openssl req -config $opensslConf -new -sha256 -key lnd-key -out lnd.csr -subj '/CN=localhost/O=lnd'
+    openssl req -config $opensslConf -x509 -sha256 -days 1825 -key lnd-key -in lnd.csr -out lnd-cert
+    rm lnd.csr
+fi

pkgs/generate-secrets/update-and-generate.nix

@@ -0,0 +1,10 @@
+{ pkgs }: with pkgs;
+
+let
+  generate-secrets = callPackage ./. {};
+in
+writeScript "make-secrets" ''
+  # Update from old secrets format
+  [[ -e secrets.nix ]] && . ${./update-secrets.sh}
+  ${generate-secrets}
+''

pkgs/generate-secrets/update-secrets.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+set -eo pipefail
+
+# Update secrets from the old format to the current one where each secret
+# has a local source file.
+
+reportError() {
+    echo "Updating secrets failed. (Error in line $1)"
+    echo "The secret files have been moved to secrets/old-secrets"
+}
+trap 'reportError $LINENO' ERR
+
+echo "Updating old secrets to the current format."
+
+mkdir old-secrets
+# move all files into old-secrets
+shopt -s extglob dotglob
+mv !(old-secrets) old-secrets
+shopt -u dotglob
+
+secrets=$(cat old-secrets/secrets.nix)
+
+extractPassword() {
+    pwName="$1"
+    destFile="${2:-$pwName}"
+    echo "$secrets" | sed -nE "s/.*?$pwName = \"(.*?)\".*/\1/p" > "$destFile"
+}
+
+rename() {
+    old="old-secrets/$1"
+    if [[ -e $old ]]; then
+        cp "$old" "$2"
+    fi
+}
+
+extractPassword bitcoinrpcpassword bitcoin-rpcpassword
+extractPassword lnd-wallet-password
+extractPassword liquidrpcpassword liquid-rpcpassword
+extractPassword lightning-charge-api-token lightning-charge-token
+extractPassword spark-wallet-password
+
+rename nginx.key nginx-key
+rename nginx.cert nginx-cert
+rename lnd.key lnd-key
+rename lnd.cert lnd-cert
+
+rm -r old-secrets

pkgs/nixops/release.nix.patch

@@ -32,3 +32,14 @@
 
 
        # For "nix-build --run-env".
+
+--- a/nixops/backends/__init__.py
++++ b/nixops/backends/__init__.py
+@@ -24,6 +24,7 @@ class MachineDefinition(nixops.resources.ResourceDefinition):
+             opts = {}
+             for (key, xmlType) in (('text',        'string'),
+                                    ('keyFile',     'path'),
++                                   ('keyFile',     'string'),
+                                    ('destDir',     'string'),
+                                    ('user',        'string'),
+                                    ('group',       'string'),

pkgs/nodeinfo/nodeinfo.sh

@@ -4,8 +4,8 @@ set -o pipefail
 BITCOIND_ONION="$(cat /var/lib/onion-chef/operator/bitcoind)"
 echo BITCOIND_ONION="$BITCOIND_ONION"
 
-if [ -x "$(command -v lightning-cli)" ]; then
+if systemctl is-active --quiet clightning; then
-    CLIGHTNING_NODEID=$(sudo -u clightning lightning-cli --lightning-dir=/var/lib/clightning getinfo | jq -r '.id')
+    CLIGHTNING_NODEID=$(lightning-cli getinfo | jq -r '.id')
     CLIGHTNING_ONION="$(cat /var/lib/onion-chef/operator/clightning)"
     CLIGHTNING_ID="$CLIGHTNING_NODEID@$CLIGHTNING_ONION:9735"
     echo CLIGHTNING_NODEID="$CLIGHTNING_NODEID"
@@ -13,8 +13,8 @@ if [ -x "$(command -v lightning-cli)" ]; then
     echo CLIGHTNING_ID="$CLIGHTNING_ID"
 fi
 
-if [ -x "$(command -v lncli)" ]; then
+if systemctl is-active --quiet lnd; then
-    LND_NODEID=$(sudo -u lnd lncli --tlscertpath /secrets/lnd_cert --macaroonpath /var/lib/lnd/chain/bitcoin/mainnet/admin.macaroon getinfo | jq -r '.uris[0]')
+    LND_NODEID=$(lncli getinfo | jq -r '.uris[0]')
     echo LND_NODEID="$LND_NODEID"
 fi
 

secrets/generate_secrets.sh

@@ -1,41 +0,0 @@
-#!/bin/sh
-
-SECRETSFILE=secrets/secrets.nix
-
-if [ ! -e "$SECRETSFILE" ]; then
-    echo Write secrets to $SECRETSFILE
-    {
-        echo \{
-        echo "  bitcoinrpcpassword = \"$(apg -m 20 -x 20 -M Ncl -n 1)\";"
-        echo "  lnd-wallet-password = \"$(apg -m 20 -x 20 -M Ncl -n 1)\";"
-        echo "  lightning-charge-api-token = \"$(apg -m 20 -x 20 -M Ncl -n 1)\";"
-        echo "  liquidrpcpassword = \"$(apg -m 20 -x 20 -M Ncl -n 1)\";"
-        echo "  spark-wallet-password = \"$(apg -m 20 -x 20 -M Ncl -n 1)\";"
-        echo \}
-    } >> $SECRETSFILE
-    echo Done
-else
-    echo $SECRETSFILE already exists. Skipping.
-fi
-
-if [ ! -e secrets/nginx.key ] || [ ! -e secrets/nginx.cert ]; then
-    echo Generate Nginx Self-Signed Cert
-    openssl genrsa -out secrets/nginx.key 2048
-    openssl req -new -key secrets/nginx.key -out secrets/nginx.csr -subj "/C=KN"
-    openssl x509 -req -days 1825 -in secrets/nginx.csr -signkey secrets/nginx.key -out secrets/nginx.cert
-    rm secrets/nginx.csr
-    echo Done
-else
-    echo Nginx Cert already exists. Skipping.
-fi
-
-if [ ! -e secrets/lnd.key ] || [ ! -e secrets/lnd.cert ]; then
-    echo Generate LND compatible TLS Cert
-    openssl ecparam -genkey -name prime256v1 -out secrets/lnd.key
-    openssl req -config secrets/openssl.cnf -new -sha256 -key secrets/lnd.key -out secrets/lnd.csr -subj '/CN=localhost/O=lnd'
-    openssl req -config secrets/openssl.cnf -x509 -sha256 -days 1825 -key secrets/lnd.key -in secrets/lnd.csr -out secrets/lnd.cert
-    rm secrets/lnd.csr
-    echo Done
-else
-    echo LND cert already exists. Skipping.
-fi

shell.nix

@@ -7,8 +7,9 @@ stdenv.mkDerivation rec {
   name = "nix-bitcoin-environment";
 
   nixops19_09 = callPackage ./pkgs/nixops {};
+  make-secrets = callPackage ./pkgs/generate-secrets/update-and-generate.nix {};
 
-  buildInputs = with pkgs; [ nixops19_09 figlet apg openssl ];
+  buildInputs = [ nixops19_09 figlet ];
 
   shellHook = ''
     export NIX_PATH="nixpkgs=${nixpkgs}:."
@@ -18,6 +19,6 @@ stdenv.mkDerivation rec {
     # keys already added to my ssh-agent.
     export SSH_AUTH_SOCK=""
     figlet "nix-bitcoin"
-    ./secrets/generate_secrets.sh
+    (mkdir -p secrets; cd secrets; ${make-secrets})
   '';
 }

test/make-test.nix

@@ -0,0 +1,44 @@
+testArgs:
+
+let
+  pkgs = import <nixpkgs> { config = {}; overlays = []; };
+
+  # Stable nixpkgs doesn't yet include the Python testing framework.
+  # Use unstable nixpkgs and patch it so that it uses stable nixpkgs for the VM
+  # machine configuration.
+  testingPkgs = let
+    # unstable as of 2020-01-09
+    rev = "9beb0d1ac2ebd6063efbdc4d3631f8ce137bbf90";
+    src = builtins.fetchTarball {
+        url = "https://github.com/nixos/nixpkgs-channels/archive/${rev}.tar.gz";
+        sha256 = "1v95779di35qhrz70p2v27kmwm09h8pgh74i1wc72v0zp3bdrf50";
+    };
+  in
+    pkgs.runCommand "nixpkgs-testing" {} ''
+    cp -r ${src} $out
+    cd $out
+    chmod +w -R .
+    patch -p1 < ${./use-stable-pkgs.patch}
+  '';
+
+  test = (import "${testingPkgs}/nixos/tests/make-test-python.nix") testArgs;
+
+  # Fix the black Python code formatter that's used in the test to allow the test
+  # script to have longer lines. The default width of 88 chars is too restrictive for
+  # our script.
+  fixedTest = { system ? builtins.currentSystem, ... }@args:
+    let
+       pkgs = (import testingPkgs { inherit system; config = {}; overlays = []; } );
+       pkgsFixed = pkgs // {
+         python3Packages = pkgs.python3Packages // {
+           black = pkgs.writeScriptBin "black" ''
+             fileToCheck=''${@:$#}
+             [[ $fileToCheck = *test-script ]] && extraArgs='--line-length 100'
+             exec ${pkgs.python3Packages.black}/bin/black $extraArgs "$@"
+           '';
+         };
+       };
+    in
+      test (args // { pkgs = pkgsFixed; });
+in
+  fixedTest

test/run-tests.sh

@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+
+# Modules integration test runner.
+# The test (./test.nix) uses the NixOS testing framework and is executed in a VM.
+#
+# Usage:
+#   Run test
+#   ./run-tests.sh
+#
+#   Run test and save result to avoid garbage collection
+#   ./run-tests.sh build --out-link /tmp/nix-bitcoin-test
+#
+#   Run interactive test debugging
+#   ./run-tests.sh debug
+#
+#   This starts the testing VM and drops you into a Python REPL where you can
+#   manually execute the tests from ./test-script.py
+
+set -eo pipefail
+
+numCPUs=${numCPUs:-$(nproc)}
+# Min. 800 MiB needed to avoid 'out of memory' errors
+memoryMiB=${memoryMiB:-2048}
+
+scriptDir=$(cd "${BASH_SOURCE[0]%/*}" && pwd)
+
+getPkgs() {
+    nix eval --raw -f "$scriptDir/../pkgs/nixpkgs-pinned.nix" $1
+}
+export NIX_PATH=nixpkgs=$(getPkgs nixpkgs):nixpkgs-unstable=$(getPkgs nixpkgs-unstable)
+
+# Run the test. No temporary files are left on the host system.
+run() {
+    # TMPDIR is also used by the test driver for VM tmp files
+    export TMPDIR=$(mktemp -d -p /tmp nix-bitcoin-test.XXXXXX)
+    trap "rm -rf $TMPDIR" EXIT
+
+    nix-build --out-link $TMPDIR/driver "$scriptDir/test.nix" -A driver
+
+    # Variable 'tests' contains the Python code that is executed by the driver on startup
+    if [[ $1 == --interactive ]]; then
+        echo "Running interactive testing environment"
+        tests=$(
+            echo 'is_interactive = True'
+            # The test script raises an error when 'is_interactive' is defined so
+            # that it just loads the initial helper functions and stops before
+            # executing the actual tests
+            echo 'try:'
+            echo '    exec(os.environ["testScript"])'
+            echo 'except:'
+            echo '    pass'
+            # Start VM
+            echo 'start_all()'
+            # Start REPL
+            echo 'import code'
+            echo 'code.interact(local=globals())'
+        )
+    else
+        tests='exec(os.environ["testScript"])'
+    fi
+
+    echo "VM stats: CPUs: $numCPUs, memory: $memoryMiB MiB"
+    [[ $NB_TEST_ENABLE_NETWORK ]] || QEMU_NET_OPTS='restrict=on'
+    env -i \
+        NIX_PATH="$NIX_PATH" \
+        TMPDIR="$TMPDIR" \
+        tests="$tests" \
+        QEMU_OPTS="-smp $numCPUs -m $memoryMiB -nographic $QEMU_OPTS"  \
+        QEMU_NET_OPTS="$QEMU_NET_OPTS" \
+        $TMPDIR/driver/bin/nixos-test-driver
+}
+
+debug() {
+    run --interactive
+}
+
+# Run the test by building the test derivation
+build() {
+    vmTestNixExpr | nix-build --no-out-link "$@" -
+}
+
+# On continuous integration nodes there are few other processes running alongside the
+# test, so use more memory here for maximum performance.
+exprForCI() {
+    memoryMiB=3072
+    memTotalKiB=$(awk '/MemTotal/ { print $2 }' /proc/meminfo)
+    memAvailableKiB=$(awk '/MemAvailable/ { print $2 }' /proc/meminfo)
+    # Round down to nearest multiple of 50 MiB for improved test build caching
+    ((memAvailableMiB = memAvailableKiB / (1024 * 50) * 50))
+    ((memAvailableMiB < memoryMiB)) && memoryMiB=$memAvailableMiB
+    >&2 echo "Host memory: total $((memTotalKiB / 1024)) MiB, available $memAvailableMiB MiB, VM $memoryMiB MiB"
+    vmTestNixExpr
+}
+
+vmTestNixExpr() {
+  cat <<EOF
+    (import "$scriptDir/test.nix" {}).overrideAttrs (old: rec {
+      buildCommand = ''
+        export QEMU_OPTS="-smp $numCPUs -m $memoryMiB"
+        echo "VM stats: CPUs: $numCPUs, memory: $memoryMiB MiB"
+      '' + old.buildCommand;
+    })
+EOF
+}
+
+eval "${@:-build}"

test/test-script.py

@@ -0,0 +1,104 @@
+def succeed(*cmds):
+    """Returns the concatenated output of all cmds"""
+    return machine.succeed(*cmds)
+
+
+def assert_matches(cmd, regexp):
+    out = succeed(cmd)
+    if not re.search(regexp, out):
+        raise Exception(f"Pattern '{regexp}' not found in '{out}'")
+
+
+def log_has_string(unit, str):
+    return f"journalctl -b --output=cat -u {unit} --grep='{str}'"
+
+
+def assert_no_failure(unit):
+    """Unit should not have failed since the system is running"""
+    machine.fail(log_has_string(unit, "Failed with result"))
+
+
+def assert_running(unit):
+    machine.wait_for_unit(unit)
+    assert_no_failure(unit)
+
+
+# Don't execute the following test suite when this script is running in interactive mode
+if "is_interactive" in vars():
+    raise Exception()
+
+### Tests
+
+assert_running("setup-secrets")
+# Unused secrets should be inaccessible
+succeed('[[ $(stat -c "%U:%G %a" /secrets/dummy) = "root:root 440" ]]')
+
+assert_running("bitcoind")
+machine.wait_until_succeeds("bitcoin-cli getnetworkinfo")
+assert_matches("su operator -c 'bitcoin-cli getnetworkinfo' | jq", '"version"')
+
+assert_running("electrs")
+machine.wait_for_open_port(4224)  # prometeus metrics provider
+assert_running("nginx")
+# SSL stratum server via nginx. Only check for open port, no content is served here
+# as electrs isn't ready.
+machine.wait_for_open_port(50003)
+# Stop electrs from spamming the test log with 'wait for bitcoind sync' messages
+succeed("systemctl stop electrs")
+
+assert_running("liquidd")
+machine.wait_until_succeeds("elements-cli getnetworkinfo")
+assert_matches("su operator -c 'elements-cli getnetworkinfo' | jq", '"version"')
+succeed("su operator -c 'liquidswap-cli --help'")
+
+assert_running("clightning")
+assert_matches("su operator -c 'lightning-cli getinfo' | jq", '"id"')
+
+assert_running("spark-wallet")
+spark_auth = re.search("login=(.*)", succeed("cat /secrets/spark-wallet-login"))[1]
+machine.wait_for_open_port(9737)
+assert_matches(f"curl -s {spark_auth}@localhost:9737", "Spark")
+
+assert_running("lightning-charge")
+charge_auth = re.search("API_TOKEN=(.*)", succeed("cat /secrets/lightning-charge-env"))[1]
+machine.wait_for_open_port(9112)
+assert_matches(f"curl -s api-token:{charge_auth}@localhost:9112/info | jq", '"id"')
+
+assert_running("nanopos")
+machine.wait_for_open_port(9116)
+assert_matches("curl localhost:9116", "tshirt")
+
+assert_running("onion-chef")
+
+# FIXME: use 'wait_for_unit' because 'create-web-index' always fails during startup due
+# to incomplete unit dependencies.
+# 'create-web-index' implicitly tests 'nodeinfo'.
+machine.wait_for_unit("create-web-index")
+machine.wait_for_open_port(80)
+assert_matches("curl localhost", "nix-bitcoin")
+assert_matches("curl -L localhost/store", "tshirt")
+
+machine.wait_until_succeeds(log_has_string("bitcoind-import-banlist", "Importing node banlist"))
+assert_no_failure("bitcoind-import-banlist")
+
+### Additional tests
+
+# Current time in µs
+pre_restart = succeed("date +%s.%6N").rstrip()
+
+# Sanity-check system by restarting all services
+succeed("systemctl restart bitcoind clightning spark-wallet lightning-charge nanopos liquidd")
+
+# Now that the bitcoind restart triggered a banlist import restart, check that
+# re-importing already banned addresses works
+machine.wait_until_succeeds(
+    log_has_string(f"bitcoind-import-banlist --since=@{pre_restart}", "Importing node banlist")
+)
+assert_no_failure("bitcoind-import-banlist")
+
+### Test lnd
+
+succeed("systemctl stop nanopos lightning-charge spark-wallet clightning")
+succeed("systemctl start lnd")
+assert_matches("su operator -c 'lncli getinfo' | jq", '"version"')
+assert_no_failure("lnd")

test/test.nix

@@ -0,0 +1,52 @@
+# Integration test, can be run without internet access.
+
+import ./make-test.nix rec {
+  name = "nix-bitcoin";
+
+  hardened = {
+    imports = [ <nixpkgs/nixos/modules/profiles/hardened.nix> ];
+    security.allowUserNamespaces = true; # reenable disabled option
+  };
+
+  machine = { pkgs, lib, ... }: with lib; {
+    imports = [
+      ../modules/nix-bitcoin.nix
+      ../modules/secrets/generate-secrets.nix
+      # using the hardened profile increases total test duration by ~50%, so disable it for now
+      # hardened
+    ];
+
+    services.nix-bitcoin.enable = true;
+    services.bitcoind.extraConfig = mkForce "connect=0";
+
+    services.clightning.enable = true;
+    services.spark-wallet.enable = true;
+    services.lightning-charge.enable = true;
+    services.nanopos.enable = true;
+
+    services.lnd.enable = true;
+    systemd.services.lnd.wantedBy = mkForce [];
+
+    services.electrs.enable = true;
+
+    services.liquidd = {
+      enable = true;
+      listen = mkForce false;
+      extraConfig = "noconnect=1";
+    };
+
+    services.nix-bitcoin-webindex.enable = true;
+
+    services.hardware-wallets = {
+      trezor = true;
+      ledger = true;
+    };
+
+    # to test that unused secrets are made inaccessible by 'setup-secrets'
+    systemd.services.generate-secrets.postStart = ''
+      install -o nobody -g nogroup -m777 <(:) /secrets/dummy
+    '';
+  };
+
+  testScript = builtins.readFile ./test-script.py;
+}

test/use-stable-pkgs.patch

@@ -0,0 +1,41 @@
+--- a/nixos/lib/build-vms.nix
++++ b/nixos/lib/build-vms.nix
+@@ -30,10 +30,10 @@ rec {
+   buildVM =
+     nodes: configurations:
+ 
+-    import ./eval-config.nix {
++    import <nixpkgs/nixos/lib/eval-config.nix> {
+       inherit system;
+       modules = configurations ++ extraConfigurations;
+-      baseModules =  (import ../modules/module-list.nix) ++
++      baseModules = (import <nixpkgs/nixos/modules/module-list.nix>) ++
+         [ ../modules/virtualisation/qemu-vm.nix
+           ../modules/testing/test-instrumentation.nix # !!! should only get added for automated test runs
+           { key = "no-manual"; documentation.nixos.enable = false; }
+
+
+services.connman doesn't exist in stable nixpkgs
+--- a/nixos/modules/virtualisation/qemu-vm.nix
++++ b/nixos/modules/virtualisation/qemu-vm.nix
+@@ -620,7 +620,6 @@ in
+ 
+     # Wireless won't work in the VM.
+     networking.wireless.enable = mkVMOverride false;
+-    services.connman.enable = mkVMOverride false;
+ 
+     # Speed up booting by not waiting for ARP.
+     networking.dhcpcd.extraConfig = "noarp";
+
+The test driver assumed coreutils to be in PATH. This fix will be ported to upstream.
+--- a/nixos/lib/testing-python.nix
++++ b/nixos/lib/testing-python.nix
+@@ -155,7 +155,7 @@ in rec {
+             --add-flags "''${vms[*]}" \
+             ${lib.optionalString enableOCR
+               "--prefix PATH : '${ocrProg}/bin:${imagemagick_tiff}/bin'"} \
+-            --run "export testScript=\"\$(cat $out/test-script)\"" \
++            --run "export testScript=\"\$(${coreutils}/bin/cat $out/test-script)\"" \
+             --set VLANS '${toString vlans}'
+           ln -s ${testDriver}/bin/nixos-test-driver $out/bin/nixos-run-vms
+           wrapProgram $out/bin/nixos-run-vms \