diff --git a/README.md b/README.md index c12eefa..938d0c9 100644 --- a/README.md +++ b/README.md @@ -1,703 +1,233 @@ -# Docker Deployment Manager → Coolify Migration +# Docker Deployment Manager -This project started as a custom Docker deployment tool (CLI + Electron desktop app) for deploying from Windows to Linux servers via SSH/tar transfers. After evaluating the approach, we're migrating to **Coolify** — an open-source self-hosted PaaS that handles the entire deploy pipeline automatically. - -## Why Coolify Instead of a Custom Tool? - -| Custom Tool | Coolify | -|---|---| -| Manual button clicks to deploy | Auto-deploys on `git push` | -| SSH password stored in plaintext `.env` | SSH keys / deploy keys | -| Tar-based image transfer (no registry) | Builds on server, no transfer needed | -| Desktop app required on Windows | Web dashboard from any browser | -| Custom file-diff comparison | Git is the source of truth | -| One developer maintaining it | Thousands of contributors | +Unified REST API + Electron desktop app for managing Docker deployments across two servers via Gitea + Coolify. The API is the single source of truth — the UI, CLI, and LLM agents all use the same HTTP endpoints. ## Architecture ``` -Windows (dev machine) 192.168.69.4 (box-stableapps) -┌──────────────────────┐ ┌──────────────────────────────┐ -│ C:\.bucket\repos.gitea│ │ Coolify (port 2010) │ -│ │ │ ├── Dashboard + DB │ -│ edit code │ │ ├── Traefik proxy │ -│ → git commit │ │ └── Manages deployments │ -│ → git push │ │ │ -│ │ │ │ Gitea (ports 2003/2004) │ -│ ▼ │ │ ├── HTTP: 2003 │ -│ Gitea (webhook) ────────────► │ └── SSH: 2004 │ -│ gitea.clintmasden │ │ │ -│ .duckdns.org │ │ Data: ~/containers/*/data/ │ -└──────────────────────┘ └──────────────────────────────┘ - │ - │ SSH (root) - ▼ - 192.168.69.5 (box-repoapps) - ┌──────────────────────────────┐ - │ Deployed apps: │ - │ ├── learnxiny │ - │ ├── actualbudget.report │ - │ └── ... │ - │ │ - │ Data: ~/containers/*/data/ │ - └──────────────────────────────┘ - -Flow: edit → git push → Gitea webhook → Coolify auto-builds & deploys +idea.llm.gitea.repo.docker.deployment/ +├── api/ # Express API server (:3100) +│ ├── server.js # Entry point, Scalar docs at /reference +│ ├── routes/ +│ │ ├── coolify.js # /api/coolify/* — Coolify app management +│ │ ├── projects.js # /api/projects/* — local scanning & comparison +│ │ ├── servers.js # /api/servers/* — server CRUD, containers, logs +│ │ └── docker.js # /api/docker/* — build, deploy, pull files +│ ├── lib/ +│ │ ├── config.js # Loads Coolify + deployment configs +│ │ ├── coolify-client.js # coolifyFetch() — Coolify API wrapper +│ │ └── ssh.js # SSHService class (exec + SFTP) +│ └── openapi.json # OpenAPI 3.1 spec (24 endpoints) +├── app/ +│ ├── main/ # Electron main process (legacy IPC) +│ ├── renderer/ # React UI (Vite + TanStack Query) +│ │ └── src/ +│ │ ├── lib/api.js # All API calls via HTTP fetch +│ │ ├── hooks/ # React Query hooks +│ │ └── components/ +│ │ ├── coolify/ # Coolify dashboard + deploy dialog +│ │ ├── projects/ # Project management +│ │ ├── servers/ # Server settings +│ │ └── dashboard/ # Overview +│ └── .env # SSH credentials (not committed) +├── cli/ # CLI tool for project scaffolding +├── deployment-config.json # Server list + project root config +└── package.json ``` -## Server Setup - -- **192.168.69.4** (box-stableapps): Coolify host + Gitea host + stable apps -- **192.168.69.5** (box-repoapps): Working/repo apps, managed as remote server by Coolify - ---- - -## Coolify Installation (on 192.168.69.4) - -Coolify runs as Docker containers at `~/containers/coolify/`, consistent with the existing container layout. - -### 1. Create directories +## Quick Start ```bash -mkdir -p ~/containers/coolify/data/{source,ssh,applications,databases,backups,services,proxy,sentinel} -mkdir -p ~/containers/coolify/data/ssh/{keys,mux} -mkdir -p ~/containers/coolify/data/proxy/dynamic +# Install dependencies +npm install +cd app/renderer && npm install && cd ../.. + +# Run everything (API + Vite + Electron) +npm run dev + +# Or without Electron (browser only at http://localhost:5173) +npm run dev:no-electron + +# API only +npm run api:dev ``` -### 2. Create the .env file +- **API server**: http://localhost:3100 +- **Scalar API docs**: http://localhost:3100/reference +- **OpenAPI spec**: http://localhost:3100/openapi.json +- **Vite dev server**: http://localhost:5173 (proxies `/api/*` to :3100) + +## API Endpoints (24 total) + +All endpoints return JSON. The Vite dev server proxies `/api/*` to the API server, so the renderer uses relative paths. + +### Projects + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/projects` | Scan local projects (finds Dockerfiles, coolify.json, etc.) | +| POST | `/api/projects/compare` | Compare local vs remote files (docker-compose, .env, data/) | +| POST | `/api/projects/init` | Initialize a project with deployment config via CLI | + +### Servers + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/servers` | List configured deployment servers | +| POST | `/api/servers` | Add or update a server | +| DELETE | `/api/servers/:id` | Delete a server | +| GET | `/api/servers/:id/scan` | Scan remote ~/containers for deployed projects | +| GET | `/api/servers/:id/containers` | List running Docker containers (docker ps) | +| GET | `/api/servers/:id/logs` | Get container logs (query: containerName, remotePath, lines) | + +### Docker Operations + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/api/docker/build` | Build tar via build-image-tar.ps1 | +| POST | `/api/docker/deploy` | Upload files + docker load + docker compose up | +| POST | `/api/docker/pull` | Pull files/directories from remote server via SFTP | +| POST | `/api/docker/vscode-diff` | Download remote file and open VS Code diff | + +### Coolify + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/coolify/apps` | List all Coolify apps (enriched with HOST_PORT) | +| GET | `/api/coolify/apps/find/:name` | Find app by name | +| POST | `/api/coolify/apps` | Create new Coolify application | +| PATCH | `/api/coolify/apps/:uuid` | Update existing app | +| DELETE | `/api/coolify/apps/:uuid` | Delete app | +| GET | `/api/coolify/apps/:uuid/envs` | List env vars | +| POST | `/api/coolify/apps/:uuid/envs` | Set env var | +| POST | `/api/coolify/apps/:uuid/deploy` | Trigger deployment | +| GET | `/api/coolify/servers` | List Coolify servers | +| GET | `/api/coolify/next-port` | Get next available HOST_PORT | +| POST | `/api/coolify/routes` | Add Traefik route via SSH | +| POST | `/api/coolify/drift` | Check drift between coolify.json and live state | +| POST | `/api/coolify/upsert` | Full pipeline: config → create/update → env → route → deploy | +| GET | `/api/coolify/config?path=...` | Read coolify.json from a project | + +### System + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/health` | Health check | +| GET | `/openapi.json` | OpenAPI 3.1 spec | +| GET | `/reference` | Scalar interactive API docs | + +## Configuration + +### Coolify Config (`gitea.repo.management/config.json`) + +Shared with the gitea.repo.management project. Contains Coolify API credentials, server UUIDs, SSH config: + +```json +{ + "coolify": { + "apiUrl": "http://192.168.69.4:2010", + "apiToken": "", + "serverUuid": "c88okoc4g4css0c4ooo04ko4", + "projectUuid": "bcso4cocokswgo804gogoswo", + "privateKeyUuid": "j0w08woc8s8c0sgok8ccow4w", + "sshHost": "192.168.69.4", + "sshUser": "clint", + "sshPassword": "", + "traefikPath": "/home/clint/containers/coolify/data/proxy/dynamic/custom.yml", + "targetIp": "192.168.69.5" + } +} +``` + +### Deployment Config (`deployment-config.json`) + +Local to this project. Stores server list and project root: + +```json +{ + "servers": [ + { "id": "1771305558920", "name": "Repo Server", "host": "192.168.69.5", "username": "clint", "useSudo": true } + ], + "projectsRoot": "C:\\.bucket\\repos.gitea" +} +``` + +### SSH Credentials (`app/.env`) + +``` +SSH_USERNAME=clint +SSH_PASSWORD= +``` + +## Coolify Upsert Pipeline + +The `POST /api/coolify/upsert` endpoint runs the full deploy pipeline: + +1. **config** — Read coolify.json from the project +2. **server** — Resolve server name to UUID/IP via Coolify API +3. **check** — Find existing app by name +4. **validate** — Compare coolify.json vs live state, report diffs +5. **sync** — Create or update the Coolify application +6. **env** — Set HOST_PORT environment variable +7. **route** — Add Traefik route (remote) or set FQDN (local) +8. **changelog** — Write coolify.changelog.json to repo +9. **deploy** — Trigger Coolify deployment + +## Docker Deploy Pipeline (Legacy/Direct) + +The `POST /api/docker/deploy` endpoint handles tar-based deployment: + +1. Connect to server via SSH +2. Create remote directory (mkdir -p) +3. Upload tar + docker-compose.yml + .env + additional files via SFTP +4. Run `docker load -i .tar` +5. Run `docker compose down && docker compose up -d` +6. Poll health check (10 attempts, 2s apart) + +## Infrastructure + +| Server | IP | Role | +|--------|-----|------| +| box-stableapps | 192.168.69.4 | Coolify host, Gitea, Traefik | +| box-repoapps | 192.168.69.5 | Deployed apps | + +### Port Allocation + +| Port | App | Status | +|------|-----|--------| +| 2001 | Audio Events | Deployed | +| 2002 | Audio Capsule | Deployed | +| 2003 | Audio Sphere | Deployed | +| 2004 | Every Noise at Once | Deployed | +| 2005 | Learn X In Y | Deployed | +| 2006 | Actual Budget Report | Deployed | +| 2007 | dotrepo-timer | Pending | +| 2008 | dotrepo-travel | Pending | +| 2009 | dotrepo-racker | Deployed | + +## LLM / Agent Usage + +This API is designed to be called by LLMs and agents. Key endpoints for automation: ```bash -tee ~/containers/coolify/data/source/.env << 'ENVEOF' -APP_ID=coolify -APP_NAME=Coolify -APP_ENV=production -APP_KEY= -DB_USERNAME=coolify -DB_PASSWORD= -REDIS_PASSWORD= -PUSHER_APP_ID= -PUSHER_APP_KEY= -PUSHER_APP_SECRET= -APP_PORT=2010 -SOKETI_PORT=6001 -REGISTRY_URL=ghcr.io -ENVEOF +# List all Coolify apps with their ports and status +curl http://localhost:3100/api/coolify/apps + +# Deploy a project from its coolify.json +curl -X POST http://localhost:3100/api/coolify/upsert \ + -H "Content-Type: application/json" \ + -d '{"projectPath": "C:/.bucket/repos.gitea/dotrepo.racker"}' + +# Scan local projects +curl http://localhost:3100/api/projects + +# Deploy via tar+SSH to a server +curl -X POST http://localhost:3100/api/docker/deploy \ + -H "Content-Type: application/json" \ + -d '{"projectPath": "C:/.bucket/repos.gitea/myapp", "serverId": "1771305558920", "remotePath": "~/containers/myapp"}' + +# Get container logs +curl "http://localhost:3100/api/servers/1771305558920/logs?remotePath=~/containers/myapp&lines=50" ``` -Generate secrets: - -```bash -sed -i "s|^APP_KEY=.*|APP_KEY=base64:$(openssl rand -base64 32)|" ~/containers/coolify/data/source/.env -sed -i "s|^DB_PASSWORD=.*|DB_PASSWORD=$(openssl rand -base64 32)|" ~/containers/coolify/data/source/.env -sed -i "s|^REDIS_PASSWORD=.*|REDIS_PASSWORD=$(openssl rand -base64 32)|" ~/containers/coolify/data/source/.env -sed -i "s|^PUSHER_APP_ID=.*|PUSHER_APP_ID=$(openssl rand -hex 32)|" ~/containers/coolify/data/source/.env -sed -i "s|^PUSHER_APP_KEY=.*|PUSHER_APP_KEY=$(openssl rand -hex 32)|" ~/containers/coolify/data/source/.env -sed -i "s|^PUSHER_APP_SECRET=.*|PUSHER_APP_SECRET=$(openssl rand -hex 32)|" ~/containers/coolify/data/source/.env -``` - -### 3. Create docker-compose.yml - -Use a single merged compose file at `~/containers/coolify/data/source/docker-compose.yml`. Key customizations from default: - -- All paths use `/home/clint/containers/coolify/data/` instead of `/data/coolify/` -- Named volumes replaced with bind mounts (postgres, redis) -- `APP_PORT=2010` instead of default 8000 -- `host.docker.internal` enabled for accessing other containers on the host -- Gitea known_hosts mounted for SSH git access - -```yaml -services: - coolify: - image: "${REGISTRY_URL:-ghcr.io}/coollabsio/coolify:${LATEST_IMAGE:-latest}" - container_name: coolify - restart: always - working_dir: /var/www/html - extra_hosts: - - host.docker.internal:host-gateway - volumes: - - type: bind - source: /home/clint/containers/coolify/data/source/.env - target: /var/www/html/.env - read_only: true - - /home/clint/containers/coolify/data/ssh:/var/www/html/storage/app/ssh - - /home/clint/containers/coolify/data/applications:/var/www/html/storage/app/applications - - /home/clint/containers/coolify/data/databases:/var/www/html/storage/app/databases - - /home/clint/containers/coolify/data/services:/var/www/html/storage/app/services - - /home/clint/containers/coolify/data/backups:/var/www/html/storage/app/backups - - /home/clint/containers/coolify/data/known_hosts:/root/.ssh/known_hosts:ro - environment: - - APP_ENV=${APP_ENV:-production} - - PHP_MEMORY_LIMIT=${PHP_MEMORY_LIMIT:-256M} - env_file: - - /home/clint/containers/coolify/data/source/.env - ports: - - "${APP_PORT:-8000}:8080" - expose: - - "${APP_PORT:-8000}" - healthcheck: - test: curl --fail http://127.0.0.1:8080/api/health || exit 1 - interval: 5s - retries: 10 - timeout: 2s - depends_on: - postgres: - condition: service_healthy - redis: - condition: service_healthy - soketi: - condition: service_healthy - networks: - - coolify - - postgres: - image: postgres:15-alpine - container_name: coolify-db - restart: always - volumes: - - /home/clint/containers/coolify/data/postgres:/var/lib/postgresql/data - environment: - POSTGRES_USER: "${DB_USERNAME}" - POSTGRES_PASSWORD: "${DB_PASSWORD}" - POSTGRES_DB: "${DB_DATABASE:-coolify}" - healthcheck: - test: ["CMD-SHELL", "pg_isready -U ${DB_USERNAME}", "-d", "${DB_DATABASE:-coolify}"] - interval: 5s - retries: 10 - timeout: 2s - networks: - - coolify - - redis: - image: redis:7-alpine - container_name: coolify-redis - restart: always - command: redis-server --save 20 1 --loglevel warning --requirepass ${REDIS_PASSWORD} - environment: - REDIS_PASSWORD: "${REDIS_PASSWORD}" - volumes: - - /home/clint/containers/coolify/data/redis:/data - healthcheck: - test: redis-cli ping - interval: 5s - retries: 10 - timeout: 2s - networks: - - coolify - - soketi: - image: "${REGISTRY_URL:-ghcr.io}/coollabsio/coolify-realtime:1.0.10" - container_name: coolify-realtime - restart: always - extra_hosts: - - host.docker.internal:host-gateway - ports: - - "${SOKETI_PORT:-6001}:6001" - - "6002:6002" - volumes: - - /home/clint/containers/coolify/data/ssh:/var/www/html/storage/app/ssh - environment: - APP_NAME: "${APP_NAME:-Coolify}" - SOKETI_DEBUG: "${SOKETI_DEBUG:-false}" - SOKETI_DEFAULT_APP_ID: "${PUSHER_APP_ID}" - SOKETI_DEFAULT_APP_KEY: "${PUSHER_APP_KEY}" - SOKETI_DEFAULT_APP_SECRET: "${PUSHER_APP_SECRET}" - healthcheck: - test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:6001/ready && wget -qO- http://127.0.0.1:6002/ready || exit 1"] - interval: 5s - retries: 10 - timeout: 2s - networks: - - coolify - -networks: - coolify: - name: coolify - external: true -``` - -### 4. Set up Traefik proxy - -Coolify uses Traefik as its reverse proxy. The proxy has its own docker-compose at `~/containers/coolify/data/proxy/docker-compose.yml`: - -```yaml -name: coolify-proxy -networks: - coolify: - external: true -services: - traefik: - container_name: coolify-proxy - image: 'traefik:v3.6' - restart: unless-stopped - extra_hosts: - - 'host.docker.internal:host-gateway' - networks: - - coolify - ports: - - '80:80' - - '443:443' - - '443:443/udp' - - '8080:8080' - healthcheck: - test: 'wget -qO- http://localhost:80/ping || exit 1' - interval: 4s - timeout: 2s - retries: 5 - volumes: - - '/var/run/docker.sock:/var/run/docker.sock:ro' - - '/home/clint/containers/coolify/data/proxy/:/traefik' - command: - - '--ping=true' - - '--ping.entrypoint=http' - - '--api.dashboard=true' - - '--entrypoints.http.address=:80' - - '--entrypoints.https.address=:443' - - '--entrypoints.http.http.encodequerysemicolons=true' - - '--entryPoints.http.http2.maxConcurrentStreams=250' - - '--entrypoints.https.http.encodequerysemicolons=true' - - '--entryPoints.https.http2.maxConcurrentStreams=250' - - '--entrypoints.https.http3' - - '--providers.file.directory=/traefik/dynamic/' - - '--providers.file.watch=true' - - '--certificatesresolvers.letsencrypt.acme.httpchallenge=true' - - '--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=http' - - '--certificatesresolvers.letsencrypt.acme.storage=/traefik/acme.json' - - '--api.insecure=false' - - '--providers.docker=true' - - '--providers.docker.exposedbydefault=false' - labels: - - traefik.enable=true - - traefik.http.routers.traefik.entrypoints=http - - traefik.http.routers.traefik.service=api@internal - - traefik.http.services.traefik.loadbalancer.server.port=8080 - - coolify.managed=true - - coolify.proxy=true -``` - -Key: the volume mount points to `/home/clint/containers/coolify/data/proxy/` (not `/data/coolify/proxy/`). - -### 5. Create Docker network and start - -**Run these as separate commands:** - -```bash -sudo docker network create --attachable coolify -``` - -```bash -# Start the proxy first -cd ~/containers/coolify/data/proxy -sudo docker compose up -d - -# Then start Coolify -cd ~/containers/coolify/data/source -sudo docker compose up -d --pull always --remove-orphans --force-recreate -``` - -### 6. Fix data directory permissions - -Each service needs specific ownership. **Do NOT blanket chown the entire data directory** — Postgres and Redis run as different UIDs inside their containers. - -```bash -# Postgres must be UID 70 (postgres user in Alpine) -sudo chown -R 70:70 ~/containers/coolify/data/postgres - -# Redis must be UID 999 (redis user in Alpine) -sudo chown -R 999:999 ~/containers/coolify/data/redis - -# SSH keys — Coolify runs as UID 9999 -sudo chown -R 9999:root ~/containers/coolify/data/ssh -sudo chmod -R 700 ~/containers/coolify/data/ssh - -# Proxy — root-owned, acme.json must be 600 -sudo chown -R root:root ~/containers/coolify/data/proxy -sudo chmod 600 ~/containers/coolify/data/proxy/acme.json - -# Everything else can be clint -sudo chown -R clint:clint ~/containers/coolify/data/source -sudo chown -R clint:clint ~/containers/coolify/data/applications -sudo chown -R clint:clint ~/containers/coolify/data/databases -sudo chown -R clint:clint ~/containers/coolify/data/services -sudo chown -R clint:clint ~/containers/coolify/data/backups -``` - -If you accidentally `chown -R` the whole directory, fix it with the commands above and restart: `docker restart coolify-db coolify-redis coolify`. - -### 7. Open dashboard - -Go to `http://192.168.69.4:2010` and create admin account (first visitor gets admin). - ---- - -## Gitea SSH Fix (Required for Coolify) - -By default, Gitea's Docker image does NOT start its built-in SSH server. This means SSH keys added through Gitea's web UI won't work for git clone over SSH. Coolify needs SSH to clone private repos. - -### The problem - -Gitea's Docker container has OpenSSH running on port 22 internally, but it's not connected to Gitea's user/key database. Keys added in the Gitea web UI are only used by Gitea's **built-in** SSH server, which is disabled by default. - -### The fix - -Add these environment variables to your Gitea docker-compose (`~/containers/gitea/docker-compose.yml`): - -```yaml - gitea: - environment: - # ... existing env vars ... - GITEA__server__START_SSH_SERVER: "true" - GITEA__server__SSH_LISTEN_PORT: "2222" - GITEA__server__ROOT_URL: "https://gitea.clintmasden.duckdns.org/" - GITEA__server__SSH_DOMAIN: "192.168.69.4" - GITEA__webhook__ALLOWED_HOST_LIST: "192.168.69.4,host.docker.internal" - ports: - - "2003:3000" - - "2004:2222" # was 2004:22, now points to Gitea's built-in SSH -``` - -- `START_SSH_SERVER=true` — enables Gitea's built-in SSH server that reads keys from its database -- `SSH_LISTEN_PORT=2222` — avoids conflict with OpenSSH already on port 22 inside the container -- Port mapping changed from `2004:22` to `2004:2222` to match -- `ROOT_URL` — must match how you access Gitea in the browser. Mismatch causes webhooks not to fire on real pushes -- `SSH_DOMAIN` — keeps the IP for SSH clone URLs (Coolify connects via `ssh://git@192.168.69.4:2004/...`) -- `ALLOWED_HOST_LIST` — Gitea blocks webhooks to local/private IPs by default. This allows webhooks to reach Coolify on the same machine - -Restart Gitea: - -```bash -cd ~/containers/gitea && docker compose up -d -``` - -Verify SSH works: - -```bash -ssh -p 2004 git@192.168.69.4 -# Should respond with: "Hi ! You've successfully authenticated..." -``` - ---- - -## Connecting Coolify to Gitea - -### 1. Generate SSH key in Coolify - -In Coolify dashboard → **Keys & Tokens** → create a new private key (e.g., "gitea-key"). - -### 2. Add public key to Gitea (one-time, all repos) - -Copy the public key from Coolify's key detail page. In Gitea → **User Settings → SSH / GPG Keys → Add Key** → paste it. - -This gives Coolify access to ALL your repos under that user. No need to add deploy keys per-repo. - -### 3. Set up Gitea's known_hosts for Coolify - -The Coolify container needs to trust Gitea's SSH host key. Create a persistent known_hosts file: - -```bash -ssh-keyscan -p 2004 192.168.69.4 2>/dev/null | sudo tee /home/clint/containers/coolify/data/known_hosts > /dev/null -``` - -This file is mounted into the Coolify container via the docker-compose volume: - -```yaml -- /home/clint/containers/coolify/data/known_hosts:/root/.ssh/known_hosts:ro -``` - -### 4. Add remote server (192.168.69.5) - -Coolify needs **root SSH access** to remote servers: - -On 192.168.69.5: -1. Add Coolify's SSH public key to `/root/.ssh/authorized_keys` -2. Ensure `/etc/ssh/sshd_config` has `PermitRootLogin prohibit-password` -3. `sudo systemctl restart sshd` - -In Coolify: **Servers → Add Server** → SSH user `root`, IP `192.168.69.5`. - -Do the same for localhost (192.168.69.4) if you want Coolify to manage apps on the same box. - -### 5. Add an application - -1. **Projects → Add Project** → name it -2. Inside the project → **Add Resource → Application** -3. Choose **Private Repository (with deploy key)** -4. Select the **gitea-key** private key -5. Repository URL (must include port via ssh:// format): - ``` - ssh://git@192.168.69.4:2004/clint/your-repo.git - ``` -6. Choose **Docker Compose** as build pack -7. Click **Load Compose File** -8. Set environment variables in Coolify's UI - -### SSH URL format — important - -Because Gitea SSH is on port 2004 (not 22), you **must** use the `ssh://` URL format: - -``` -ssh://git@192.168.69.4:2004/clint/repo-name.git -``` - -The standard `git@host:user/repo.git` format does NOT support custom ports and will fail. - ---- - -## Deployment Repo Structure - -Each service gets a Gitea repo with Docker config: - -``` -dotrepo.learn.x.in.y/ -├── react/ -│ ├── Dockerfile -│ ├── docker-compose.yml ← absolute volume paths -│ ├── server.js -│ ├── package.json -│ └── src/ -``` - -### Volume paths — absolute, not relative - -```yaml -# WRONG — relative path breaks with Coolify -volumes: - - ./data:/app/data - -# CORRECT — absolute path, data stays where it already is -volumes: - - /home/clint/containers/learnxiny/data:/data -``` - -### Environment variables - -`.env` files stay in `.gitignore` — never committed. Production env vars are managed in Coolify's dashboard per-app. Coolify auto-detects `${VARIABLE}` syntax in compose files and creates UI fields for them. - ---- - -## Auto-deploy Webhook - -The webhook URL and secret are **global** — same for every repo. Coolify matches incoming webhooks to the right app based on the repo URL in the payload. - -### One-time: get the URL and secret from Coolify - -1. Open any app in Coolify → **Webhooks** tab -2. Under **Manual Git Webhooks → Gitea**, copy the URL (e.g., `http://192.168.69.4:2010/webhooks/source/gitea/events/manual`) -3. Set a **Gitea Webhook Secret** and save - -### Per-repo: add webhook in Gitea - -For each Gitea repo you want to auto-deploy: - -1. Repo → **Settings → Webhooks → Add Webhook → Gitea** -2. **Target URL:** the Coolify Gitea webhook URL from above -3. **HTTP Method:** POST -4. **Content Type:** application/json -5. **Secret:** the same secret you set in Coolify -6. **Trigger On:** Push Events -7. Save - -Test by clicking **Test Push Event** in the webhook, then check **Recent Deliveries** for a green 200 response. - -### Gitea webhook allow-list - -Gitea blocks webhooks to private/local IPs by default. The `GITEA__webhook__ALLOWED_HOST_LIST` environment variable in Gitea's docker-compose must include the Coolify host IP — see [Gitea SSH Fix](#gitea-ssh-fix-required-for-coolify) section. - -Now every `git push` triggers: clone → build → deploy → health check — automatically. - ---- - -## Traefik — Replacing Caddy - -Coolify manages Traefik as its built-in reverse proxy. Once all services are migrated, Traefik replaces Caddy entirely. - -### Coolify dashboard domain - -In Coolify → **Settings** → set **Instance's Domain (FQDN)** to: - -``` -https://coolify.clintmasden.duckdns.org -``` - -Coolify auto-configures Traefik to route this domain to its dashboard with SSL. - -### Coolify-managed apps - -For apps deployed through Coolify, set the domain in the app's settings. Coolify configures Traefik automatically — no manual config needed. - -### Non-Coolify services (Gitea, Syncthing, etc.) - -For services NOT managed by Coolify, add custom Traefik config at: - -`~/containers/coolify/data/proxy/dynamic/custom.yml` - -```yaml -http: - routers: - gitea-web: - rule: "Host(`gitea.clintmasden.duckdns.org`)" - entryPoints: - - https - service: gitea-web - tls: - certResolver: letsencrypt - - syncthing: - rule: "Host(`syncthing.clintmasden.duckdns.org`)" - entryPoints: - - https - service: syncthing - tls: - certResolver: letsencrypt - - immich: - rule: "Host(`immich.clintmasden.duckdns.org`)" - entryPoints: - - https - service: immich - tls: - certResolver: letsencrypt - - affine: - rule: "Host(`affine.clintmasden.duckdns.org`)" - entryPoints: - - https - service: affine - tls: - certResolver: letsencrypt - - mattermost: - rule: "Host(`mattermost.clintmasden.duckdns.org`)" - entryPoints: - - https - service: mattermost - tls: - certResolver: letsencrypt - - vikunja: - rule: "Host(`vikunja.clintmasden.duckdns.org`)" - entryPoints: - - https - service: vikunja - tls: - certResolver: letsencrypt - - services: - gitea-web: - loadBalancer: - servers: - - url: "http://host.docker.internal:2003" - - syncthing: - loadBalancer: - servers: - - url: "http://host.docker.internal:2000" - - immich: - loadBalancer: - servers: - - url: "http://host.docker.internal:2001" - - affine: - loadBalancer: - servers: - - url: "http://host.docker.internal:2005" - - mattermost: - loadBalancer: - servers: - - url: "http://host.docker.internal:2006" - - vikunja: - loadBalancer: - servers: - - url: "http://host.docker.internal:2009" -``` - -Traefik picks up changes automatically — no restart needed. - -### Switching from Caddy to Traefik - -1. Stop Caddy: `cd ~/containers/caddy && docker compose down` -2. Verify Traefik grabs ports 80/443: `docker ps | grep traefik` -3. If Traefik isn't binding, restart it: `docker restart coolify-proxy` -4. Test: `curl -I https://gitea.clintmasden.duckdns.org` -5. If something breaks, bring Caddy back: `cd ~/containers/caddy && docker compose up -d` - -### Services that still need migration - -These Caddy static sites need to be containerized (simple nginx containers) before Caddy can be fully retired: - -- `dotrepo.com` + subdomains: fitnotes, racker, timer, travel, pianoxml (React/Angular SPAs served from `/sites/`) -- `audiosphere.dotrepo.com`, `audiocapsule.dotrepo.com`, `audioevents.dotrepo.com`, `everynoise.dotrepo.com` (proxied to 192.168.69.5) -- `clintmasden.duckdns.org` (simple "Hi, I'm Clint" response) - ---- - -## Troubleshooting - -### "Could not resolve hostname ssh/http" -Coolify's custom git repo field **only supports SSH**. It strips protocol prefixes. Use the `ssh://` format with port for non-standard SSH ports. - -### "Permission denied (publickey)" on git clone -1. Verify the right private key is selected in the application settings -2. Verify the matching public key is in Gitea (User Settings → SSH Keys, not repo deploy keys) -3. Test from inside Coolify: `docker exec coolify sh -c "ssh -p 2004 -i /var/www/html/storage/app/ssh/keys/ git@192.168.69.4"` -4. Debug with verbose SSH: `docker exec coolify sh -c "ssh -vvv -p 2004 -i /var/www/html/storage/app/ssh/keys/ -o StrictHostKeyChecking=no git@192.168.69.4 2>&1 | tail -30"` - -### "Host key verification failed" -Regenerate known_hosts: `ssh-keyscan -p 2004 192.168.69.4 2>/dev/null | sudo tee /home/clint/containers/coolify/data/known_hosts > /dev/null` - -### SSH key storage not writable -```bash -sudo chown -R 9999:root ~/containers/coolify/data/ssh -sudo chmod -R 700 ~/containers/coolify/data/ssh -``` - -### 500 error / Redis MISCONF / Postgres permission denied -Usually caused by blanket `chown` on the data directory. Fix ownership per-service and restart: -```bash -sudo chown -R 70:70 ~/containers/coolify/data/postgres -sudo chown -R 999:999 ~/containers/coolify/data/redis -sudo chown -R root:root ~/containers/coolify/data/proxy -sudo chmod 600 ~/containers/coolify/data/proxy/acme.json -docker restart coolify-db coolify-redis coolify -``` - -### Gitea SSH rejects all keys -Enable Gitea's built-in SSH server — see [Gitea SSH Fix](#gitea-ssh-fix-required-for-coolify) section above. - -### Webhook "ALLOWED_HOST_LIST" error -Gitea blocks webhooks to private IPs by default. Add `GITEA__webhook__ALLOWED_HOST_LIST: "192.168.69.4,host.docker.internal"` to Gitea's docker-compose environment and restart. - -### Webhooks fire on test but not on real push -Check Gitea's `ROOT_URL` setting. If it doesn't match the URL you use to access Gitea (e.g., set to `http://192.168.69.4:2003` but you access via `gitea.clintmasden.duckdns.org`), webhooks won't fire on actual push events. Fix by setting `GITEA__server__ROOT_URL` in the docker-compose. - -### Useful debug commands - -List Coolify's SSH keys: -```bash -docker exec coolify sh -c "ls /var/www/html/storage/app/ssh/keys/" -``` - -Show public key for each stored key: -```bash -docker exec coolify sh -c "for key in /var/www/html/storage/app/ssh/keys/*; do echo \"--- \$key ---\"; ssh-keygen -y -f \$key 2>&1; done" -``` - -Note: Coolify's container is Alpine-based — use `sh` not `bash` for `docker exec`. - ---- - -## Workflow Summary - -``` -Before (manual): - edit → build tar → SSH upload → docker load → docker compose up → hope it works - -After (Coolify): - edit → git commit → git push → done (Coolify handles the rest) -``` - ---- - -Built for managing 35+ Docker deployments across two servers via Gitea + Coolify. +**Always use this API** to access/change Coolify and Gitea deployment state. Do not call the Coolify API directly — this API wraps it with enrichment (HOST_PORT), safety (drift detection, changelog), and routing (Traefik management). diff --git a/api/routes/coolify.js b/api/routes/coolify.js index 28e07f1..3221053 100644 --- a/api/routes/coolify.js +++ b/api/routes/coolify.js @@ -179,10 +179,19 @@ router.post('/routes', wrap(async (req) => { const routerBlock = [ ` ${routeName}:`, ` rule: "Host(\`${domain}\`)"`, + ` entryPoints:`, + ` - https`, ` service: ${routeName}`, - ` entryPoints: [websecure]`, ` tls:`, - ` certResolver: letsencrypt` + ` certResolver: letsencrypt`, + ``, + ` ${routeName}-http:`, + ` rule: "Host(\`${domain}\`)"`, + ` entryPoints:`, + ` - http`, + ` middlewares:`, + ` - redirect-to-https`, + ` service: ${routeName}`, ].join('\n'); const serviceBlock = [ @@ -196,9 +205,16 @@ router.post('/routes', wrap(async (req) => { return { added: false, reason: 'route_exists' }; } + // Insert router before middlewares section, service before end of services section let newYml; + const middlewaresSplit = current.split(' middlewares:'); const servicesSplit = current.split(' services:'); - if (servicesSplit.length === 2) { + if (middlewaresSplit.length === 2 && servicesSplit.length === 2) { + // Insert router block before middlewares, service block at end of services + newYml = middlewaresSplit[0].trimEnd() + '\n\n' + routerBlock + '\n\n middlewares:' + middlewaresSplit[1]; + // Now add service at the end + newYml = newYml.trimEnd() + '\n\n' + serviceBlock + '\n'; + } else if (servicesSplit.length === 2) { newYml = servicesSplit[0].trimEnd() + '\n' + routerBlock + '\n\n services:' + servicesSplit[1].trimEnd() + '\n' + serviceBlock + '\n'; } else { newYml = current.trimEnd() + '\n' + routerBlock + '\n' + serviceBlock + '\n'; @@ -445,10 +461,19 @@ router.post('/upsert', wrap(async (req) => { const routerBlock = [ ` ${routeName}:`, ` rule: "Host(\`${appConfig.domain}\`)"`, + ` entryPoints:`, + ` - https`, ` service: ${routeName}`, - ` entryPoints: [websecure]`, ` tls:`, ` certResolver: letsencrypt`, + ``, + ` ${routeName}-http:`, + ` rule: "Host(\`${appConfig.domain}\`)"`, + ` entryPoints:`, + ` - http`, + ` middlewares:`, + ` - redirect-to-https`, + ` service: ${routeName}`, ].join('\n'); const serviceBlock = [ ` ${routeName}:`, @@ -457,9 +482,14 @@ router.post('/upsert', wrap(async (req) => { ` - url: "http://${server.ip}:${appConfig.port}"`, ].join('\n'); + // Insert router before middlewares section, service at end let newYml; + const middlewaresSplit = currentYml.split(' middlewares:'); const servicesSplit = currentYml.split(' services:'); - if (servicesSplit.length === 2) { + if (middlewaresSplit.length === 2 && servicesSplit.length === 2) { + newYml = middlewaresSplit[0].trimEnd() + '\n\n' + routerBlock + '\n\n middlewares:' + middlewaresSplit[1]; + newYml = newYml.trimEnd() + '\n\n' + serviceBlock + '\n'; + } else if (servicesSplit.length === 2) { newYml = servicesSplit[0].trimEnd() + '\n' + routerBlock + '\n\n services:' + servicesSplit[1].trimEnd() + '\n' + serviceBlock + '\n'; } else { newYml = currentYml.trimEnd() + '\n' + routerBlock + '\n' + serviceBlock + '\n';