k3s-homelab/docs/superpowers/specs/2026-07-15-notesnook-k8s-migration-design.md

Notesnook self-host → k3s migration — design

Date: 2026-07-15 Status: Approved design, pending spec review → implementation plan

Goal

Move the self-hosted Notesnook stack from the docker-compose deployment on host 10.10.0.5 (LXC on Proxmox pve) into the k3s cluster (CT 114, node IP 10.10.0.100), managed by ArgoCD via the k3s-homelab GitOps repo. The existing account, notes, encryption keys, and attachments must be preserved.

Current state (source)

• Host: manage@10.10.0.5, docker-compose stack in ~/notesnook.
• Services: MongoDB 7.0.12 (replica set rs0), MinIO (S3), identity/auth, sync, sse/events, monograph, plus bootstrap helpers (validate, setup-s3) and autoheal.
• Public hosts (Traefik on the compose box, cert via Cloudflare resolver): auth, sync (+ apex notes), events, monograph, attachments, minio — all under notes.aleshym.co.
• Data lives in: Mongo databases identity and notesnook; MinIO bucket attachments.
• Root cause history: login-code email was failing because SMTP was left at CHANGE_ME; now fixed to Gmail SMTP. That corrected config carries into the migration.

Target cluster conventions (observed)

• GitOps: ArgoCD (no app-of-apps). Monorepo https://git.aleshym.co/funman300/k3s-homelab.git, branch main. App manifests under apps/<name>/ (kustomize); the Application CR committed at argocd/<name>.yaml with automated {prune, selfHeal} + CreateNamespace=true.
• Ingress: Traefik (ingressClassName: traefik), entrypoint websecure.
• TLS: cert-manager clusterissuer letsencrypt-prod via Cloudflare DNS-01. Per-host TLS secret named <something>-tls.
• Storage: single-node local-path (default), ReadWriteOnce, Delete reclaim. Implication: deleting a PVC deletes its data — treat Mongo/MinIO PVCs as precious.
• Secrets: no sealed-secrets / external-secrets controller. Convention is a committed *-secret.yaml.example; the real *-secret.yaml is gitignored (apps/**/*-secret.yaml) and applied once with kubectl apply. The Application CR uses ignoreDifferences on the Secret's /data so ArgoCD won't fight the out-of-band secret.
• Internal DNS: resolver 10.10.0.8 holds split-horizon *.aleshym.co → LAN IP records (ns.aleshym.co → 10.10.0.100, auth.notes.aleshym.co → 10.10.0.5 today). This is where the cutover repoint happens. Public authority for aleshym.co is Cloudflare (used only for the ACME DNS-01 challenge).

Architecture

Namespace: notesnook. Repo path: apps/notesnook/. ArgoCD app: argocd/notesnook.yaml.

Component	Image	k8s workload	Service (in-cluster)	Public ingress host → port
MongoDB rs0	mongo:7.0.12	StatefulSet (1) + headless Service + init Job	notesnook-db:27017	none
MinIO	minio/minio:RELEASE.2024-07-29T22-14-52Z	Deployment + PVC	notesnook-s3:9000	attachments.notes.aleshym.co → 9000
Identity/auth	streetwriters/identity:latest	Deployment	notesnook-identity:8264	auth.notes.aleshym.co → 8264
Sync	streetwriters/notesnook-sync:latest	Deployment	notesnook-sync:5264	sync.notes.aleshym.co + notes.aleshym.co → 5264
SSE/events	streetwriters/sse:latest	Deployment	notesnook-sse:7264	events.notes.aleshym.co → 7264
Monograph	streetwriters/monograph:latest	Deployment	notesnook-monograph:3000	monograph.notes.aleshym.co → 3000

Dropped from compose: autoheal (replaced by k8s liveness/readiness probes), validate (env presence enforced by manifests), setup-s3 (folded into a MinIO bucket-create init Job / or mc init container).

MongoDB replica set (the one real deviation from the nightscout template)

Notesnook's connection strings use ?replSet=rs0 and the sync server relies on change streams/transactions, which require a replica set. Plan:

• StatefulSet (1 replica) notesnook-db, command: [mongod, --replSet, rs0, --bind_ip_all].
• Headless Service notesnook-db for stable DNS.
• One-time init Job (or postStart) runs rs.initiate({_id:"rs0", members:[{_id:0, host:"notesnook-db:27017"}]}), idempotent (skip if already initialized). The RS member host must be the stable Service name notesnook-db:27017 so clients using ?replSet=rs0 resolve the primary correctly.
• No Mongo auth (matches current compose: mongodb://notesnook-db:27017, no credentials). Access is namespace-internal only.

Config vs secrets

ConfigMap notesnook-config (committed, non-secret):

INSTANCE_NAME=aleshym-notesnook
DISABLE_SIGNUPS=true
NOTESNOOK_APP_PUBLIC_URL=https://app.notesnook.com
AUTH_SERVER_PUBLIC_URL=https://auth.notes.aleshym.co
MONOGRAPH_PUBLIC_URL=https://monograph.notes.aleshym.co
ATTACHMENTS_SERVER_PUBLIC_URL=https://attachments.notes.aleshym.co
NOTESNOOK_CORS_ORIGINS=https://app.notesnook.com,https://notes.aleshym.co
KNOWN_PROXIES=                # may need the Traefik pod CIDR; see open items
# service-discovery (in-cluster names replace compose hostnames)
NOTESNOOK_SERVER_HOST=notesnook-sync
NOTESNOOK_SERVER_PORT=5264
IDENTITY_SERVER_HOST=notesnook-identity
IDENTITY_SERVER_PORT=8264
SSE_SERVER_HOST=notesnook-sse
SSE_SERVER_PORT=7264
IDENTITY_SERVER_URL=https://auth.notes.aleshym.co
NOTESNOOK_APP_HOST=https://app.notesnook.com
# S3/MinIO non-secret
S3_INTERNAL_SERVICE_URL=http://notesnook-s3:9000
S3_INTERNAL_BUCKET_NAME=attachments
S3_SERVICE_URL=https://attachments.notes.aleshym.co
S3_BUCKET_NAME=attachments
S3_REGION=us-east-1
MONGODB_CONNECTION_STRING (identity)=mongodb://notesnook-db:27017/identity?replSet=rs0
MONGODB_CONNECTION_STRING (sync)=mongodb://notesnook-db:27017/?replSet=rs0

Secret notesnook-secret (out-of-band, *.example committed, real gitignored):

NOTESNOOK_API_SECRET=...        # reuse existing value from old .env
SMTP_USERNAME=funman300@gmail.com
SMTP_PASSWORD=...               # Gmail app password
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
MINIO_ROOT_USER=notesnook
MINIO_ROOT_PASSWORD=...         # reuse existing value
S3_ACCESS_KEY_ID=notesnook          # = MINIO_ROOT_USER
S3_ACCESS_KEY=...                   # = MINIO_ROOT_PASSWORD
# TWILIO_* : present in old .env but SMS 2FA unused; omit unless in use

Reusing NOTESNOOK_API_SECRET, MinIO creds, and the Mongo data verbatim is what keeps the migrated account valid — do not regenerate the API secret.

Repo layout to add

apps/notesnook/
  kustomization.yaml
  namespace.yaml
  notesnook-config.yaml            # ConfigMap
  notesnook-secret.yaml.example    # secret template (real applied out-of-band)
  db-statefulset.yaml              # mongo rs0
  db-service.yaml                  # headless
  db-init-job.yaml                 # rs.initiate (idempotent)
  minio-pvc.yaml
  minio-deployment.yaml
  minio-service.yaml
  minio-bucket-job.yaml            # create 'attachments' bucket
  identity-deployment.yaml + identity-service.yaml + identity-ingress.yaml
  sync-deployment.yaml     + sync-service.yaml     + sync-ingress.yaml
  sse-deployment.yaml      + sse-service.yaml      + sse-ingress.yaml
  monograph-deployment.yaml+ monograph-service.yaml+ monograph-ingress.yaml
  attachments-ingress.yaml         # attachments.notes.aleshym.co -> notesnook-s3:9000
argocd/notesnook.yaml

.gitignore already covers apps/**/*-secret.yaml.

Data migration procedure

Runs during the cutover window; both stacks reachable from the migration host.

1. Mongo: mongodump the identity and notesnook DBs from the compose Mongo (docker exec notesnook-db mongodump --archive), stream to the k8s Mongo via kubectl exec + mongorestore --archive --nsInclude 'identity.*' --nsInclude 'notesnook.*'. Verify collection counts match on both sides.
2. MinIO: mc mirror the attachments bucket from the compose MinIO to the k8s MinIO (mirror via a temporary mc pod/job with both aliases, or port-forward). Verify object count/size.
3. Sanity: confirm identity DB user document and notesnook sync docs present.

Cutover procedure

1. Deploy the empty k8s stack (commit → ArgoCD sync). Verify all pods healthy, ingress created, TLS certs issued for all 5 hosts (temporary test hostnames or /etc/hosts override so we can validate before flipping real DNS).
2. Announce brief downtime. Stop client sync / freeze writes on old stack (docker compose stop the app services, keep DB+MinIO up for the dump).
3. Run the data migration (above).
4. DNS cutover on 10.10.0.8: repoint auth/sync/events/monograph/attachments.notes.aleshym.co and apex notes.aleshym.co from 10.10.0.5 → 10.10.0.100. Lower TTL beforehand if possible; flush caches after.
5. Verify end-to-end against k8s: request a login code (email arrives), log in, sync a note, open an attachment, load a monograph link.
6. docker compose down the old stack (data retained) — rollback = flip DNS back and docker compose up -d.

Rollback

Until decommission, rollback is: repoint the 6 records on 10.10.0.8 back to 10.10.0.5 and docker compose up -d on the old host. Because writes were frozen during cutover, no data diverges.

Decommission (≈1 week after stable cutover)

Remove the compose stack and its data on 10.10.0.5 (docker compose down -v, remove ~/notesnook), and reclaim the LXC if no longer needed. Update memory notes to point at the k8s deployment.

Verification / success criteria

• All Notesnook pods Running/healthy; Mongo RS PRIMARY; MinIO bucket present.
• All 5 ingress hosts serve valid Let's Encrypt certs.
• Existing account logs in (email code delivered) — no re-signup.
• Pre-existing notes + attachments visible in the client after sync.
• Monograph public link resolves.

Open items / access still needed

• Git push access to git.aleshym.co/funman300/k3s-homelab (clone was anonymous/read-only). Decide branch vs PR workflow.
• Edit access to internal DNS 10.10.0.8 for the cutover (mechanism/creds TBD — AdGuard/Pi-hole/unbound?).
• KNOWN_PROXIES: old value empty; behind Traefik the sync/identity servers may need the Traefik pod source CIDR here for correct client-IP/proxy handling. Confirm during implementation.
• Public access path: are the notes.* hosts reachable externally (Cloudflare tunnel / netbird), or LAN/VPN-only? Affects whether any Cloudflare records also need updating, or just internal DNS.
• Twilio SMS 2FA: present in old .env; assumed unused (email 2FA works). Omit unless confirmed in use.