Read-only mirror of https://github.com/MeteoSwiss/husk — MeteoSwiss. Issues & pull requests at the source.
  • Rust 68.7%
  • Shell 24.1%
  • C 3.9%
  • Python 3.1%
  • Makefile 0.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Christoph Müller 3edee272ed Feature: compute-node containment, brokered network egress, and the v0.5 review
husk confines an AI agent on a shared HPC system from the outside. v0.4 brokered
job submission; the cage stopped at the login node. v0.5 carries it onto the
compute node and closes the network.

WHAT IS NEW

* Single-node multi-rank MPI runs inside the cage, GPUs included. This is the
  release's substance, and it is real work rather than a smoke test: ICON ran to
  completion with CMA enabled and no `MPICH_SMP_SINGLE_COPY_MODE=NONE`
  workaround, and a production KENDA assimilation experiment ran green — the
  first production workload through husk. Everything such a job needs works
  inside a confined step: `nvidia-smi` sees the node's four GPUs with NVLink
  active, `taskset` and `numactl --cpunodebind/--membind` pin the way ICON
  expects, ranks share /dev/shm and reach each other over CMA, and the session's
  uenv is mounted where the job looks for it.

* A step broker is what makes that possible. `srun` inside a brokered job is
  itself brokered, and every rank runs in its own cage — homes hidden, root
  read-only, its own PID namespace — with /dev/shm and CMA opened only where
  same-node MPI genuinely needs them. Multi-node MPI is REFUSED, not downgraded:
  `--nodes` other than 1 is rejected with a message that says husk brokers
  single-node jobs only and that this is a decision, not an oversight. Shipping
  it as a named weaker profile is on the roadmap; it is not code that exists.

* Network egress through an allowlisted proxy, per session, default-deny, with
  a relay each rank binds for itself. A non-allowlisted host is refused, and the
  refusal says so rather than timing out.

* Submission is default-deny by construction. Options are parsed, re-emitted and
  anything unknown is rejected, so adversary bytes never reach slurmd's parser.
  `--output`/`--error`/`--chdir` accept a checked subset of SLURM's specifiers;
  the rest are refused with a reason the agent can act on.

* A differential harness that measures husk's model of slurmd against the real
  slurmd, case by case, and grades the artefact in Rust. It agrees on every
  measured case on two clusters five SLURM releases apart.

HOW IT WAS CHECKED

Three rounds of multi-agent review, adversarial and independent, each finding
reviewed again before its fix was committed — that second pass found real
defects in most of them, including two live bypasses of controls just shipped.
Test count 328 to 488. Verified end to end on two production clusters,
x86_64 and aarch64: seccomp smoke 16/0, filesystem and network containment
green on both, egress boundary confirmed in both directions.

Two lessons are now principles rather than commit messages. P16: the failure
mode of a fail-closed control is a denial of service aimed at the operator, so
every control must answer what it newly refuses and who can trigger it. P17: a
value meaning "a check passed" must not be constructible where the check is not.

The finding-by-finding record is kept internally — see doc/README.md for which
half of L4 is published and why. Nothing about the design or its defects is
withheld: principles, threat model, controls and rationale are all here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xcrMc3pQ5frnMakM7tLsZ
2026-09-02 10:34:14 +02:00
doc Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
project-config Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
scripts Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
seccomp-wrapper Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
skill Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
slurm-broker Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
user-config Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
.gitignore Feature: SLURM job brokering for the husk sandbox (v0.4.0) 2026-07-28 13:45:36 +02:00
AGENTS.md Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
CLAUDE.md Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
install-husk.sh Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
install-husk.test.sh Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
LICENSE Initial commit providing claude-safe for CSCS supercomputers 2026-05-27 15:01:30 +02:00
make-release.sh Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
make-release.test.sh Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
NOTICE Feature: SLURM job brokering for the husk sandbox (v0.4.0) 2026-07-28 13:45:36 +02:00
README.md Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00
ROADMAP.md Feature: compute-node containment, brokered network egress, and the v0.5 review 2026-09-02 10:34:14 +02:00

husk — sandboxing AI coding agents on HPC supercomputers

husk confines an AI coding agent on a shared supercomputer, from the outside. The agent runs inside a cage it cannot inspect, disable or negotiate with; every SLURM command it issues is validated by a trusted broker running beyond its reach; and its cooperation is never required for any of it.

That is worth doing because the agent runs as you. On a login node an unconfined assistant holds your SSH keys and credentials, your compute allocation, the other home directories on the shared filesystem, and the cluster's internal services — not because it is malicious, but because it inherited your account. husk makes those unreachable rather than discouraged, and tells the agent what it just refused so it stops guessing and works around the boundary.

It wraps Claude Code today; the confinement contract itself is agent-agnostic by design (doc/sandbox-interface.md).

Developed and tested on CSCS supercomputers (Balfrin, Santis).

Getting started

On a Balfrin or Santis login node. Nothing is compiled, nothing needs root.

1. Claude Code, if you don't have it. To check command -v claude.

curl -fsSL https://claude.ai/install.sh | bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc
claude          # log in once, then exit

2. Install husk. One tarball works on both machines — it ships binaries for each and the installer picks the right ones.

curl -fLO https://github.com/MeteoSwiss/husk/releases/download/v0.5/husk-v0.5.tar.gz
tar xzf husk-v0.5.tar.gz && cd husk-v0.5
./install-husk.sh --slurm-partition <partition> --slurm-account <project>

The installer prints exactly what it will add to ~/.claude/settings.json and asks before changing anything. The two flags are recorded in ~/.husk/config.json on a first install — see below if you need to change them later.

3. Use it. Run husk where you would have run claude:

cd ~/my-project
husk         # husk --resume also works

Repeat step 2 on each machine whose $HOME is not shared. Releases are listed at https://github.com/MeteoSwiss/husk/releases — verify the tarball with the published SHA256SUMS if you want to.

Installing somewhere other than Balfrin or Santis, or building from source

husk needs bubblewrap (bwrap) system-wide, wget/python3/tar/sha256sum on PATH, and outbound HTTPS once to fetch Anthropic's apply-seccomp. It also needs socat; if that is missing the installer builds it, which then needs gcc and make. All of this is already in place on CSCS machines.

Building from source is for developing husk, not for using it. It needs a Rust toolchain and must be done separately on each architecture — there is no cross-compilation:

git clone https://github.com/MeteoSwiss/husk && cd husk
(cd seccomp-wrapper && ./build_and_test.sh)   # -> seccomp-wrapper-$(uname -m)
(cd slurm-broker    && ./build-release.sh)    # -> husk-slurm-{broker,wrapper}-$(uname -m)
./install-husk.sh --slurm-partition <partition> --slurm-account <project>

make alone is not a substitute for build_and_test.sh: it writes seccomp-wrapper, and the installer only ever reads seccomp-wrapper-<arch>.

SLURM: record the partition (per machine)

On a machine with SLURM, the broker forces every brokered job onto one partition, recorded at install time. The built-in default is preemptible, so installing bare on a site without that partition leaves the broker forcing one that does not exist — submissions then fail at the scheduler with invalid partition specified.

./install-husk.sh

The partition and account live in ~/.husk/config.json, which is the interface: edit the file, no reinstall. The installer only ever seeds it on a first install and never clobbers an existing one — accounts and partitions change far more often than the installation does.

{ "partitions": ["preemptible"], "accounts": ["<your project>"] }

Balfrin — preemptible (also the built-in default). Santis — has no preemptible partition; use debug (short test jobs) or shared (when nodes are free). On any other site, list what exists with sinfo -s and pick accordingly. A per-system override is ~/.husk/config.<system>.json, which wins outright — $HOME is shared between some machines and their partitions differ.

Prefer a low-priority or preemptible queue where the site has one, so an unattended agent's jobs can be killed and do not consume your allocation's priority. The partition is not auto-detected: which queue an unattended agent submits to is an operator decision, not something to infer from cluster state.

To change it later, edit ~/.husk/config.json. Re-running the installer with a new value does not change it: a config file that names partitions wins over the install-time flags by design, and the installer never clobbers one that exists. It says so when you try — "…but you passed --slurm-account/--slurm-partition, and the config file OVERRIDES them" — and then prints what is in effect. A flag that does nothing must not do it quietly.

One edge, worth knowing because the warning is wrong there. The config wins only where its list is non-empty. On a config that reads {"partitions": [], "accounts": []} the broker falls back to HUSK_SLURM_PARTITION, which the installer does rewrite on every run — so --slurm-partition takes effect while the installer is still printing that the file overrides it. Put the value in the file and the ambiguity goes away.

If ~/.local/bin is not yet on your PATH, add this to ~/.bashrc or ~/.bash_profile:

export PATH="$HOME/.local/bin:$PATH"

Then start Claude Code with husk instead of claude:

husk

The sandbox is now active for all your projects.

To remove everything later — the installed binaries and the settings blocks it added (your other settings are preserved) — run:

./install-husk.sh --uninstall

Optional — per-project restrictions: For tighter control over what Claude can do within a specific project (network access, MCP servers), copy the template into your project and commit it:

mkdir -p .claude
cp <unpacked-release>/project-config/settings.json .claude/settings.json

See Per-project setup for details.

What it restricts

Filesystem: Claude can only read and write files inside the current project directory. All home directories — yours and other users' — are not visible to the agent. Data on shared filesystems outside the home directories (e.g. /scratch/) is readable but not writable.

SLURM: husk handles SLURM automatically. On a cluster it routes job submission through a fail-closed broker — the agent can submit batch jobs and run read-only queries (squeue/sinfo/sacct/…), but the real commands run outside the sandbox, submitted jobs are re-sandboxed on the compute node, and the agent never gets the credentials to submit directly. The broker is spawned only when SLURM is detected; on a laptop there's no broker and no trace, and a sandboxed sbatch has nothing to talk to. See Running SLURM jobs (the broker).

Network access (per-project config): ssh, curl, wget, and similar tools are blocked by default. This is a conservative starting point for shared login nodes; relax it per project if your workflow needs it.

Credential files: .env files, private keys (*.pem, *.key), and credential files are blocked from being read or modified.

MCP servers (per-project config): All MCP tool calls are blocked by default. Enable specific servers you trust in your project config.

Per-project setup

The install script applies machine-wide defaults. For project-specific restrictions, copy the template into your project:

mkdir -p .claude
cp <unpacked-release>/project-config/settings.json .claude/settings.json

Commit .claude/settings.json so the restrictions apply to everyone on the project. See project-config/README.md for what each entry does and which ones you may want to relax.

How it works

The sandbox is built from three layers that stack on top of each other:

Process isolation (seccomp-wrapper): A thin wrapper around the Claude process that installs a syscall deny-list — blocking low-level operations like attaching to other processes, loading kernel modules, and manipulating user IDs. This is the outermost layer and covers the entire Claude process tree.

Filesystem isolation (bubblewrap): A sandboxing tool installed system-wide on CSCS supercomputers. Claude Code uses it to run each agent subprocess in an isolated view of the filesystem, where only the project directory is present. This is what keeps your home directory invisible to the agent.

Additional filter (apply-seccomp): A binary from Anthropic that blocks two specific escape paths: reaching host services via Unix domain sockets, and bypassing filters through io_uring. This runs inside the filesystem sandbox.

Settings are split into two files:

  • ~/.claude/settings.json — written by the install script; applies to all your projects; covers machine-wide security defaults.
  • .claude/settings.json in your project — copied from project-config/; covers workflow decisions that vary per project (network access, MCP servers).

Requirements

All of these are already in place on Balfrin and Santis — this list is for other systems.

  • The claude CLI, installed and authenticated. husk wraps it; it does not install or update it. See Getting started for the one-liner, or the Claude Code docs.
  • x86_64 or aarch64 Linux, kernel ≥ 4.14.
  • bubblewrap (bwrap) installed system-wide. husk cannot sandbox without it.
  • wget, python3, tar, sha256sum on PATH, and outbound HTTPS once, to fetch Anthropic's apply-seccomp.
  • socat. If it is missing the installer builds it from source, which then needs gcc and make. Check with command -v socat.

Running SLURM jobs (the broker)

On a cluster, just run husk — it detects SLURM and starts a small, fail-closed broker that lets an unattended agent submit and monitor jobs — days long, with no human to approve prompts — without ever handing the agent the credentials to do so. On a machine with no SLURM, husk runs the plain sandbox, with no broker and no trace. (There's no separate command — husk does both.)

Activate your uenv (or modules) first. The broker inherits the software environment of the shell that launched it, and the agent cannot mount one from inside the cage. So uenv start <image> (or load your modules) first, then husk.

How it works

Inside the sandbox, sbatch and the read-only query commands are shadowed by a stub. The stub hands each request to the trusted broker running outside the sandbox — the only place MUNGE and the network exist. The broker:

  1. validates the request as hostile input (see policy below);
  2. re-sandboxes the job on the compute node — the job runs inside the same kind of cage as the login session (other users' homes hidden, credential files masked, auto-exec files write-protected, network unshared), so a prompt-injected job cannot read another user's data or escape the node; and
  3. submits it under policy and returns the job id.

Because only the trusted broker holds MUNGE and the network path, the agent cannot bypass it to submit directly — a sandboxed sbatch has nothing to talk to.

What the agent can do

  • Submit batch jobs — sbatch --partition=<site> job.sh. The partition must be the site's configured one — preemptible by default, set per machine in ~/.husk/config.json (Balfrin uses preemptible; Santis has no such partition, so e.g. debug). Any other is rejected with a message telling the agent how to resubmit (so design jobs to checkpoint and tolerate preemption). Risky options (--output/--error/--chdir/--export/--wrap) are forced to safe values, and the script is snapshotted at submit time (no edit-after-validate window).
  • Monitor jobs (read-only) — squeue, sinfo, sacct, sstat, sprio, sreport, sshare. The broker runs them and returns their output; they change no scheduler state.
  • Everything else is rejected — state-changing commands (scancel, scontrol update, …), interactive srun/salloc, and any unknown command.

The submitted job still runs your script with your allocation — but now inside a cage, so an agent that tampers with its own job script is contained to that job's sandbox rather than turned loose unsandboxed on the node. If you want belt-and-suspenders, still keep the job script and its imports read-only to the agent (an absolute path outside the project) and pass the agent's choices as validated data, not as a code path.

Scope in this release (v0.5): single-node batch jobs, brokered srun, and single-node multi-rank MPI including GPU jobs — ICON runs on Balfrin across 4 GPUs inside the cage with CMA enabled, and a production KENDA assimilation experiment has run green through husk. Compute jobs get an allowlisted, CONNECT-only egress proxy where an allowlist is configured, instead of no network at all.

Not in v0.5: multi-node MPI, which husk refuses — --nodes other than 1 is rejected with a message saying so, rather than silently downgraded to one node. Shipping it as a named weaker profile is a decision on the roadmap, not code that exists (see ROADMAP.md Track D). Also not in v0.5: interactive srun/salloc; and read-only scontrol show/sacctmgr list. See ROADMAP.md and slurm-broker/BROKER.md.

Known limitations

  • Network access: the shipped user settings carry an enforced domain allowlist (strictAllowlist: true) rather than a prompt hint — but only where that flag is actually deployed. Check yours: without strictAllowlist, an allowlist is a prompt hint and unlisted hosts auto-approve in auto mode, and at least one CSCS machine has been found running exactly that (measured during the v0.5 review). Brokered compute jobs reach the network only through husk's own CONNECT-only proxy, and get no network at all where no allowlist is configured. The shipped sandbox.network.allowedDomains holds one entry, opendatadocs.meteoswiss.ch:443 — that is an example, not a recommendation. Replace it with the hosts your work needs; an empty list means no network at all. Entries are host, host:port or *.two.labels, and SLURM's own ports are always refused. What is not implemented is confining the agent to Anthropic's servers specifically. curl, wget, ssh and friends stay blocked in the project config template as a second, independent control.
  • SSH to compute nodes: Blocked by default. Remove Bash(ssh *) from your project's deny list if you regularly need Claude to assist on compute nodes.
  • SLURM: on a cluster husk auto-brokers. v0.5 supports single-node batch jobs (sbatch, re-sandboxed on the compute node), brokered srun with single-node multi-rank MPI and GPUs, the compute-job network allowlist, and read-only queries (squeue/sinfo/sacct/…) — see Running SLURM jobs (the broker). All of it has run on both CSCS machines. Not yet: multi-node MPI, interactive srun/salloc, and read-only scontrol show/sacctmgr list (see ROADMAP.md). The compute-node cage is a subset of the login cage — notably its credential auto-scan uses a built-in pattern set rather than your Read() deny globs.
  • Large projects on Lustre: on very large trees (many build directories) on Lustre filesystems, the sandbox can stall for up to ~a minute while it sets up its per-command filesystem rules. This is in the bundled sandbox and not currently configurable; launch the agent from a leaner working directory (not the build-heavy project root) to avoid it.

Acknowledgements

husk builds on Anthropic's open-source sandbox-runtime (Apache-2.0): the login sandbox installs and runs its apply-seccomp helper, and the broker's compute-node filesystem cage is a Rust reimplementation adapted from its Linux model (read/write policy, credential masking, bubblewrap argument construction). See NOTICE for details.

License

BSD 3-Clause — see LICENSE for details.