k3s-homelab/docs/superpowers/specs/2026-07-15-notesnook-k8s-migration-design.md

# Notesnook self-host → k3s migration — design

**Date:** 2026-07-15
**Status:** Approved design, pending spec review → implementation plan

## Goal

Move the self-hosted Notesnook stack from the docker-compose deployment on host
`10.10.0.5` (LXC on Proxmox `pve`) into the k3s cluster (CT 114, node IP
`10.10.0.100`), managed by ArgoCD via the `k3s-homelab` GitOps repo. The existing
account, notes, encryption keys, and attachments must be preserved.

## Current state (source)

- **Host:** `manage@10.10.0.5`, docker-compose stack in `~/notesnook`.
- **Services:** MongoDB 7.0.12 (replica set `rs0`), MinIO (S3), identity/auth,
  sync, sse/events, monograph, plus bootstrap helpers (`validate`, `setup-s3`)
  and `autoheal`.
- **Public hosts** (Traefik on the compose box, cert via Cloudflare resolver):
  `auth`, `sync` (+ apex `notes`), `events`, `monograph`, `attachments`,
  `minio` — all under `notes.aleshym.co`.
- **Data lives in:** Mongo databases `identity` and `notesnook`; MinIO bucket
  `attachments`.
- **Root cause history:** login-code email was failing because SMTP was left at
  `CHANGE_ME`; now fixed to Gmail SMTP. That corrected config carries into the
  migration.

## Target cluster conventions (observed)

- **GitOps:** ArgoCD (no app-of-apps). Monorepo
  `https://git.aleshym.co/funman300/k3s-homelab.git`, branch `main`. App
  manifests under `apps/<name>/` (kustomize); the `Application` CR committed at
  `argocd/<name>.yaml` with `automated {prune, selfHeal}` + `CreateNamespace=true`.
- **Ingress:** Traefik (`ingressClassName: traefik`), entrypoint `websecure`.
- **TLS:** cert-manager clusterissuer `letsencrypt-prod` via **Cloudflare DNS-01**.
  Per-host TLS secret named `<something>-tls`.
- **Storage:** single-node `local-path` (default), `ReadWriteOnce`, `Delete`
  reclaim. **Implication:** deleting a PVC deletes its data — treat Mongo/MinIO
  PVCs as precious.
- **Secrets:** no sealed-secrets / external-secrets controller. Convention is a
  committed `*-secret.yaml.example`; the real `*-secret.yaml` is **gitignored**
  (`apps/**/*-secret.yaml`) and applied once with `kubectl apply`. The
  `Application` CR uses `ignoreDifferences` on the Secret's `/data` so ArgoCD
  won't fight the out-of-band secret.
- **Internal DNS:** resolver `10.10.0.8` holds split-horizon `*.aleshym.co` → LAN
  IP records (`ns.aleshym.co → 10.10.0.100`, `auth.notes.aleshym.co → 10.10.0.5`
  today). This is where the cutover repoint happens. Public authority for
  `aleshym.co` is Cloudflare (used only for the ACME DNS-01 challenge).

## Architecture

Namespace: `notesnook`. Repo path: `apps/notesnook/`. ArgoCD app:
`argocd/notesnook.yaml`.

| Component | Image | k8s workload | Service (in-cluster) | Public ingress host → port |
|---|---|---|---|---|
| MongoDB `rs0` | `mongo:7.0.12` | StatefulSet (1) + headless Service + init Job | `notesnook-db:27017` | none |
| MinIO | `minio/minio:RELEASE.2024-07-29T22-14-52Z` | Deployment + PVC | `notesnook-s3:9000` | `attachments.notes.aleshym.co` → 9000 |
| Identity/auth | `streetwriters/identity:latest` | Deployment | `notesnook-identity:8264` | `auth.notes.aleshym.co` → 8264 |
| Sync | `streetwriters/notesnook-sync:latest` | Deployment | `notesnook-sync:5264` | `sync.notes.aleshym.co` + `notes.aleshym.co` → 5264 |
| SSE/events | `streetwriters/sse:latest` | Deployment | `notesnook-sse:7264` | `events.notes.aleshym.co` → 7264 |
| Monograph | `streetwriters/monograph:latest` | Deployment | `notesnook-monograph:3000` | `monograph.notes.aleshym.co` → 3000 |

**Dropped from compose:** `autoheal` (replaced by k8s liveness/readiness probes),
`validate` (env presence enforced by manifests), `setup-s3` (folded into a MinIO
bucket-create init Job / or `mc` init container).

### MongoDB replica set (the one real deviation from the nightscout template)

Notesnook's connection strings use `?replSet=rs0` and the sync server relies on
change streams/transactions, which require a replica set. Plan:

- **StatefulSet** (1 replica) `notesnook-db`, `command: [mongod, --replSet, rs0, --bind_ip_all]`.
- **Headless Service** `notesnook-db` for stable DNS.
- **One-time init Job** (or `postStart`) runs
  `rs.initiate({_id:"rs0", members:[{_id:0, host:"notesnook-db:27017"}]})`,
  idempotent (skip if already initialized). The RS member host must be the stable
  Service name `notesnook-db:27017` so clients using `?replSet=rs0` resolve the
  primary correctly.
- No Mongo auth (matches current compose: `mongodb://notesnook-db:27017`, no
  credentials). Access is namespace-internal only.

### Config vs secrets

**ConfigMap `notesnook-config`** (committed, non-secret):

```
INSTANCE_NAME=aleshym-notesnook
DISABLE_SIGNUPS=true
NOTESNOOK_APP_PUBLIC_URL=https://app.notesnook.com
AUTH_SERVER_PUBLIC_URL=https://auth.notes.aleshym.co
MONOGRAPH_PUBLIC_URL=https://monograph.notes.aleshym.co
ATTACHMENTS_SERVER_PUBLIC_URL=https://attachments.notes.aleshym.co
NOTESNOOK_CORS_ORIGINS=https://app.notesnook.com,https://notes.aleshym.co
KNOWN_PROXIES=                # may need the Traefik pod CIDR; see open items
# service-discovery (in-cluster names replace compose hostnames)
NOTESNOOK_SERVER_HOST=notesnook-sync
NOTESNOOK_SERVER_PORT=5264
IDENTITY_SERVER_HOST=notesnook-identity
IDENTITY_SERVER_PORT=8264
SSE_SERVER_HOST=notesnook-sse
SSE_SERVER_PORT=7264
IDENTITY_SERVER_URL=https://auth.notes.aleshym.co
NOTESNOOK_APP_HOST=https://app.notesnook.com
# S3/MinIO non-secret
S3_INTERNAL_SERVICE_URL=http://notesnook-s3:9000
S3_INTERNAL_BUCKET_NAME=attachments
S3_SERVICE_URL=https://attachments.notes.aleshym.co
S3_BUCKET_NAME=attachments
S3_REGION=us-east-1
MONGODB_CONNECTION_STRING (identity)=mongodb://notesnook-db:27017/identity?replSet=rs0
MONGODB_CONNECTION_STRING (sync)=mongodb://notesnook-db:27017/?replSet=rs0
```

**Secret `notesnook-secret`** (out-of-band, `*.example` committed, real gitignored):

```
NOTESNOOK_API_SECRET=...        # reuse existing value from old .env
SMTP_USERNAME=funman300@gmail.com
SMTP_PASSWORD=...               # Gmail app password
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
MINIO_ROOT_USER=notesnook
MINIO_ROOT_PASSWORD=...         # reuse existing value
S3_ACCESS_KEY_ID=notesnook          # = MINIO_ROOT_USER
S3_ACCESS_KEY=...                   # = MINIO_ROOT_PASSWORD
# TWILIO_* : present in old .env but SMS 2FA unused; omit unless in use
```

Reusing `NOTESNOOK_API_SECRET`, MinIO creds, and the Mongo data verbatim is what
keeps the migrated account valid — do **not** regenerate the API secret.

## Repo layout to add

```
apps/notesnook/
  kustomization.yaml
  namespace.yaml
  notesnook-config.yaml            # ConfigMap
  notesnook-secret.yaml.example    # secret template (real applied out-of-band)
  db-statefulset.yaml              # mongo rs0
  db-service.yaml                  # headless
  db-init-job.yaml                 # rs.initiate (idempotent)
  minio-pvc.yaml
  minio-deployment.yaml
  minio-service.yaml
  minio-bucket-job.yaml            # create 'attachments' bucket
  identity-deployment.yaml + identity-service.yaml + identity-ingress.yaml
  sync-deployment.yaml     + sync-service.yaml     + sync-ingress.yaml
  sse-deployment.yaml      + sse-service.yaml      + sse-ingress.yaml
  monograph-deployment.yaml+ monograph-service.yaml+ monograph-ingress.yaml
  attachments-ingress.yaml         # attachments.notes.aleshym.co -> notesnook-s3:9000
argocd/notesnook.yaml
```

`.gitignore` already covers `apps/**/*-secret.yaml`.

## Data migration procedure

Runs during the cutover window; both stacks reachable from the migration host.

1. **Mongo:** `mongodump` the `identity` and `notesnook` DBs from the compose
   Mongo (`docker exec notesnook-db mongodump --archive`), stream to the k8s Mongo
   via `kubectl exec` + `mongorestore --archive --nsInclude 'identity.*'
   --nsInclude 'notesnook.*'`. Verify collection counts match on both sides.
2. **MinIO:** `mc mirror` the `attachments` bucket from the compose MinIO to the
   k8s MinIO (mirror via a temporary `mc` pod/job with both aliases, or
   port-forward). Verify object count/size.
3. **Sanity:** confirm identity DB user document and notesnook sync docs present.

## Cutover procedure

1. Deploy the empty k8s stack (commit → ArgoCD sync). Verify all pods healthy,
   ingress created, **TLS certs issued** for all 5 hosts (temporary test hostnames
   or `/etc/hosts` override so we can validate before flipping real DNS).
2. Announce brief downtime. Stop client sync / freeze writes on old stack
   (`docker compose stop` the app services, keep DB+MinIO up for the dump).
3. Run the data migration (above).
4. **DNS cutover on `10.10.0.8`:** repoint
   `auth`/`sync`/`events`/`monograph`/`attachments`.notes.aleshym.co **and** apex
   `notes.aleshym.co` from `10.10.0.5` → `10.10.0.100`. Lower TTL beforehand if
   possible; flush caches after.
5. Verify end-to-end against k8s: request a login code (email arrives), log in,
   sync a note, open an attachment, load a monograph link.
6. `docker compose down` the old stack (data retained) — rollback = flip DNS back
   and `docker compose up -d`.

## Rollback

Until decommission, rollback is: repoint the 6 records on `10.10.0.8` back to
`10.10.0.5` and `docker compose up -d` on the old host. Because writes were frozen
during cutover, no data diverges.

## Decommission (≈1 week after stable cutover)

Remove the compose stack and its data on `10.10.0.5`
(`docker compose down -v`, remove `~/notesnook`), and reclaim the LXC if no longer
needed. Update memory notes to point at the k8s deployment.

## Verification / success criteria

- All Notesnook pods `Running`/healthy; Mongo RS `PRIMARY`; MinIO bucket present.
- All 5 ingress hosts serve valid Let's Encrypt certs.
- Existing account logs in (email code delivered) — **no re-signup**.
- Pre-existing notes + attachments visible in the client after sync.
- Monograph public link resolves.

## Open items / access still needed

- **Git push access** to `git.aleshym.co/funman300/k3s-homelab` (clone was
  anonymous/read-only). Decide branch vs PR workflow.
- **Edit access to internal DNS `10.10.0.8`** for the cutover (mechanism/creds TBD
  — AdGuard/Pi-hole/unbound?).
- **`KNOWN_PROXIES`:** old value empty; behind Traefik the sync/identity servers
  may need the Traefik pod source CIDR here for correct client-IP/proxy handling.
  Confirm during implementation.
- **Public access path:** are the `notes.*` hosts reachable externally (Cloudflare
  tunnel / netbird), or LAN/VPN-only? Affects whether any Cloudflare records also
  need updating, or just internal DNS.
- **Twilio SMS 2FA:** present in old `.env`; assumed unused (email 2FA works). Omit
  unless confirmed in use.
```