NanokitDocs

Appearance

Troubleshooting ​

A reference for the most common issues encountered when using Nanokit. Jump to the section that matches your problem.

Docker & Local Environment ​

Docker daemon is not running ​

Symptom: nkapp up fails immediately with Cannot connect to Docker daemon.

Fix:

bash
# macOS / Windows — start Docker Desktop from the Applications menu, then:
nkapp doctor   # confirm Docker is reachable

# Linux — start the service
sudo systemctl start docker

# Confirm your user is in the docker group (avoid sudo)
sudo usermod -aG docker $USER
newgrp docker

Container stuck in a restart loop ​

Symptom: A container shows Restarting in nkapp hub or docker ps.

Diagnosis:

bash
nkapp logs <service-name>           # tail recent crash output
nkapp shell <service-name>          # exec into the container if it stays up long enough

Common causes:

  • Missing environment variable (process.env.X is undefined)
  • Wrong start command (binary not found, wrong path)
  • Port already in use on the host

Port conflict — address already in use ​

Symptom: bind: address already in use on port 80 or 443.

Fix:

bash
# Find what is holding the port
sudo lsof -i :80
sudo lsof -i :443

# Stop the conflicting process, or change the gateway ports in nanokit.yml:
gatewayPort: 8080
gatewayHttpsPort: 8443

HTTPS & Certificates ​

Browser shows "Your connection is not private" on .localhost domains ​

Symptom: After nkapp up -e local, Chrome/Firefox shows a certificate error for *.localhost.

Cause: The Nanokit Root CA has not been trusted on this machine. Nanokit's Caddy gateway generates a local CA stored in .nanokit/gateway/.

Fix — install the Root CA into your system trust store with one command:

bash
nkapp host trust

This handles macOS (System Keychain), Linux (update-ca-certificates plus NSS databases used by Chrome/Firefox), and WSL2 (also propagates to the Windows certificate store). See nkapp host trust.

Manual fallback — if you prefer to do it by hand, the root certificate lives at .nanokit/gateway/data/caddy/pki/authorities/local/root.crt; import it into your OS or browser trust store (for Firefox: about:preferences#privacy → Certificates → View Certificates → Authorities → Import).

Tip. After trusting the CA, you may need to restart your browser completely for the change to take effect.

Let's Encrypt certificate not provisioning on VPS ​

Symptom: Public domain shows a self-signed or expired certificate after deploy.

Checklist:

  1. Port 80 must be open on the VPS firewall (Let's Encrypt HTTP-01 challenge)
  2. The DNS A record must point to the VPS IP before the deploy runs
  3. The gateway.adminEmail must be set to a valid address in nanokit.yml
  4. Check Caddy logs:
    bash
    nkapp logs gateway -e stage

Force certificate reset:

bash
# Nanokit clears stale Caddy PKI state automatically for public domains
nkapp deploy -e stage

If certificate file permissions on the VPS are the problem (a service that mounts the Caddy cert cannot read it), use the dedicated repair command rather than fixing it by hand over SSH:

bash
nkapp host certs-fix stage

Custom service certificate (e.g. Stalwart mail) ​

Symptom: A mail client reports "Cannot verify identity of server".

Cause: Services with their own TLS layer (Stalwart, custom nginx) need the Caddy-issued certificate mounted explicitly.

Fix — mount the Caddy cert into the container via nanokit.yml:

yaml
services:
  mail:
    volumes:
      - /opt/nanokit/projects/<name>/.nanokit/gateway/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<domain>/<domain>.crt:/etc/caddy-certs/tls.crt:ro
      - /opt/nanokit/projects/<name>/.nanokit/gateway/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<domain>/<domain>.key:/etc/caddy-certs/tls.key:ro

Then restart the service and configure it to use /etc/caddy-certs/tls.crt.

Remote / VPS Operations ​

SSH connection refused ​

Symptom: nkapp deploy -e stage fails with Connection refused or Timeout.

Checklist:

bash
# Test SSH manually first
ssh -i ~/.ssh/id_rsa root@<VPS-IP> "echo OK"

# Check firewall on the VPS allows port 22
ufw status

# If you use a non-standard port, set it in nanokit.yml:
deploy:
  sshPort: 2222

up refuses to run against a remote environment ​

Symptom: nkapp up -e stage fails immediately, telling you to run nkapp deploy or pass --infra.

Cause: This is the first-run guard. Since the up/deploy split, up does not provision infrastructure, and the target environment has zero Nanokit-labeled resources — so there is nothing to reconcile onto.

Fix:

bash
nkapp deploy -e stage        # provision + sync + deploy (the correct first run)

# or, to opt the infrastructure phase back into this one `up`:
nkapp up -e stage --infra

See nkapp up.

(HTTP code 404) no such container on restart/stop ​

Symptom: nkapp restart <service> or nkapp stop <service> returns 404 no such container.

Cause: The local state file (.nanokit/services-state.json) contains a stale container ID from a previous run that no longer exists on the remote host.

Fix:

bash
# Option 1: Force a full reconciliation
nkapp up -e <env>

# Option 2: Clear local state and rediscover
rm .nanokit/services-state.json
nkapp up -e <env>

# Option 3: For CI/CD — use teardown instead of destroy/stop
nkapp teardown -p <project> -e <env> -y

Deployment lock — "Shared modules are being initialized" ​

Symptom: nkapp up or nkapp deploy hangs or reports a stale lock.

Cause: A previous deploy was interrupted and left a lock file in place.

Fix:

bash
nkapp infra unlock -e <env>

VPS out of disk space ​

Symptom: Deploy fails with no space left on device.

Do not docker system prune a Nanokit host

A blanket docker system prune -af --volumes over SSH removes declared images and live data volumes — it has no idea what nanokit.yml still needs. Nanokit ships three prune commands with exactly those guards, and every one of them takes an --ssh host override, so you never have to touch the host by hand.

Fix — always preview first:

bash
# 1. See what is accumulating (read-only)
nkapp doctor

# 2. Reclaim image space — dangling layers only, the safe default
nkapp images prune -e stage --dry-run
nkapp images prune -e stage -y

# 3. Where the space usually is: unused tagged images. Review the table first.
nkapp images prune -e stage --unused --dry-run

# 4. Stale containers (exited/created, >24h, not declared, not another project's)
nkapp containers prune -e stage --dry-run

# 5. Orphaned volumes — not referenced by nanokit.yml, not in use
nkapp volumes prune -e stage --dry-run

Every command renders a side-effect-free preview before it removes anything, and refuses to touch images or volumes your configuration still declares. See images prune, containers prune and volumes prune.

If a specific host is stranded (not the environment's configured target), point the same commands at it directly:

bash
nkapp images prune --ssh root@<VPS-IP> -i ~/.ssh/id_rsa --unused --dry-run

Thousands of unnamed containers on a host ​

Symptom: docker ps -a shows hundreds or thousands of auto-named containers (suspicious_jackson, …) that no Nanokit command seems to know about.

Cause: The cron scheduler creates one container per job fire and cannot label them, so when its own reap fails the strays are invisible to every label-scoped command and accumulate indefinitely. Left alone this eventually exhausts the Docker daemon.

Fix:

bash
nkapp doctor                                        # confirms the count
nkapp containers prune --unlabeled --dry-run        # ALWAYS read this table first
nkapp containers prune --unlabeled -y

--unlabeled is opt-in precisely because it is the one mode that reaches containers Nanokit did not demonstrably create.

Configuration & Parsing ​

Environment 'X' not found in configuration ​

Symptom: nkapp up -e myenv fails with environment not found.

Cause: The environment name does not match a key in the environments: block of nanokit.yml.

Fix — check spelling and add the environment if missing:

yaml
environments:
  myenv:           # must match exactly (case-sensitive)
    infra:
      provider: docker

YAML parse error ​

Symptom: nkapp up fails immediately with a YAML syntax error.

Fix:

bash
# Validate your config with the built-in doctor
nkapp doctor

# Or use a YAML linter
npx js-yaml nanokit.yml

Common causes: tabs instead of spaces, missing quotes around values with special characters (:, #, @).

Secret not resolved — env://VAR returns empty ​

Symptom: A service starts but an env:// secret resolves to an empty string.

Cause: The referenced variable is not set in the host shell or .env files.

Fix:

bash
# Check if the variable is exported in the current shell
echo $MY_SECRET_VAR

# Or add it to your .env.local (never commit this file)
echo 'MY_SECRET_VAR=value' >> .env.local

# Verify resolution
nkapp env list -e local

Hub & Bridge ​

Hub shows no services / blank dashboard ​

Symptom: The Hub opens but the dashboard is empty.

Checklist:

  1. Make sure nkapp up -e local has been run first — it starts and manages the bridge container.
  2. Check whether a bridge is running:
    bash
    nkapp hub --status
  3. If it is missing or stale, recycle it:
    bash
    nkapp hub --start      # ensure it is running (idempotent)
nkapp hub --restart    # recreate it, e.g. after rebuilding the CLI

Hub shows "No local bridge detected" ​

Cause: The Hub UI is served over HTTPS from Nanokit's servers, but it talks exclusively to the bridge on your machine at http://127.0.0.1:3006. It never falls back to a remote bridge — that would show server-side data instead of your projects. Opening the Hub from a phone, or with no bridge running, produces this state by design.

Fix: run nkapp up (or nkapp hub --start) on the machine holding your projects, then hit Retry.

Database Operations ​

db fork fails with "volume not found" ​

Symptom: nkapp db fork <name> fails because the source volume does not exist.

Cause: The base database container has never been started, so no Docker volume exists yet.

Fix:

bash
# Start the project first to create the base volume
nkapp up -e local

# Then fork
nkapp db fork <name>

db pull blocked — "remote database is empty" ​

Symptom: nkapp db pull <name> --from stage is blocked with a safety warning.

Cause: The remote database has no data. Nanokit blocks pulls from empty remote DBs to prevent accidental local data loss.

Fix — only if you are sure the remote DB is intentionally empty:

bash
nkapp db pull <name> --from stage --force

Services still use the old password after a rotation ​

Symptom: After nkapp db rotate, a service fails to authenticate.

Cause: The rotation reconciles the database's dependents automatically — unless you passed --no-restart, or the reconcile itself failed (it is reported separately and exits non-zero, but the password was rotated).

Fix:

bash
nkapp up -e <env>     # re-inject the regenerated connection URLs

CLI & General ​

nkapp: command not found ​

Fix:

bash
# Reinstall globally
npm install -g @nkapp/cli

# Or check your PATH
which nkapp
echo $PATH

nkapp is slow to start ​

Symptom: Every CLI command takes several seconds before doing anything.

Cause: Usually a slow DNS lookup or an unreachable SSH host in the config.

Fix:

bash
# Run doctor to identify the bottleneck
nkapp doctor

# If you have an SSH tunnel configured for a remote env,
# run local commands explicitly with -e local to skip remote SSH
nkapp up -e local

Getting more diagnostic output ​

There is no --verbose flag

Nanokit's CLI does not implement a global --verbose / -v verbosity flag (-v on destroy means --volumes). Use these instead:

bash
nkapp doctor                 # full read-only environment diagnosis
nkapp plan -e <env>          # exactly what a reconcile would do, before it runs
nkapp logs gateway -e <env>  # Caddy gateway logs
nkapp logs <service> -f      # follow a service's output
nkapp infra resources        # live cloud inventory, straight from the provider APIs