Troubleshooting failed deploys: errors and fixes

When an ox deploy fails, the run's log names the failed step, its cause and the fix, then the last lines of output, and the previous release keeps serving. Find the message you see below: each entry gives the symptom, the cause and the fix. In the log, a failure always has this shape:

<step>: <cause>
fix: <fix>
--- last output ---
<the last lines the command or the app printed>

Open the run in the dashboard, or run ox runs <project> and then ox logs <project> --run <id>. A failed ox deploy --wait exits with status 1.

Health check failed: the app crashed or did not answer within 120s

Symptom

The log shows waiting for … (up to 120s), then a step named health (<unit>) with one of these causes.

the app exited while starting (systemd restarted it 3 times)
the app exited while starting (failed)
… did not answer within 120s: …

Cause

The new release started but never answered. the app exited while starting means the unit crashed: systemd restarts it after every crash, and ox stops waiting at the third restart (within about 15 seconds) instead of the full 120, then shows the output, which ends with the error that stopped it. A worker or a service says the worker exited or the service exited. did not answer within 120s means the unit stayed up and never answered, with one exception: a server whose ox agent is older than this change still waits the full 120 seconds on a crash loop and says did not answer within 120s, so read the output above the message there (ox servers update <server> brings the agent up to date). ox polls every 2 seconds for up to 120 seconds: an HTTP GET on [app] health that must return a 2xx or 3xx status, or, when no health path is set, any answer to GET / but 502, 503 or 504 within 5 seconds (an agent from before 2026-10-14 only connects to the port; ox servers update <server> updates it). did not answer GET / within 5 seconds means the home page is slow: set health in [app] to a path that answers fast. The usual reasons are a start command that does not listen on $PORT, a missing variable that makes the app crash, or a health path that returns 404.

The app may write its home folder while it runs, but a PHP app may not: when a PHP app tried to write in its home folder outside the project's own folders, the cause says so: the app wrote to <path> in its home folder, which is read-only while it runs. The fix line says to point that path under $OX_DATA_DIR with the app's own setting, or to set [app] sandbox = "relaxed". The app may write its own release (a PHP release excepted), but a file it writes there is gone on the next deploy: the deploy's run log says what the app wrote in the live release, so you can add that folder to [storage] keep. A database file (.db, .sqlite, .sqlite3, .db3) written there stops the deploy before anything changes, and the live version keeps serving. The message names the file and gives the [storage] keep line to set in ox.toml. Set it and deploy again: that deploy stops the app for a moment, carries the database into the project's data folder, and starts the new version on the same rows.

When the app refused the host name it was asked for, the cause names the framework and the name: Django refused the host name shop.k7mq2xw4ab.preview.getox.in (status 400): ALLOWED_HOSTS does not list it (Rails' config.hosts and Vite's allowedHosts read the same way). A staging copy or preview that copied production's list as plain text fails like this.

Fix

Nothing changed for your visitors: the fix line ends with the previous release is still serving. Read the app's own log above the failure, then fix it and push.

• For the app exited while starting, the fix line says to read the output above it for the error, and to run the start command on your own computer to see it fail. A missing variable, a wrong path to the built file and a missing module are the usual ones. Two causes have their own fix line: the port is already in use (listen on $PORT, not a fixed number, since another project may hold that one) and killed by the out-of-memory killer as it started (raise its memory in ox.toml, or use a bigger server; for a preview that hit the previews' share, remove another preview; when the whole server ran out, stop a project, staging or preview you do not need; see Not enough memory).
• Bind to 127.0.0.1:$PORT, as in [app] start.
• Point [app] health at a path that answers without a login.
• For a refused host name, read the name from PUBLIC_HOST, which is each environment's own: ALLOWED_HOSTS=${PUBLIC_HOST} on the Variables screen, in production too so its copies inherit the reference rather than production's name.
• For a write in the home folder, follow the fix line. For data the app writes in its release, add the folder to [storage] keep, which links it to a writable folder that survives deploys.

Build failed: exit status 1, or set these before deploying

Symptom

The run stops at an install, build or migrate step, or before anything starts.

build 1/2: npm run build: exit status 1
fix: the command's own output is above; fix it in the repo and push

preflight: 1 problem(s); nothing on the host changed
fix: fix the listed problems and deploy again

Cause

A command from [build] exited non-zero, or preflight found a problem, such as set these before deploying: SECRET_KEY for a variable with no value. A failed build or migrate never touches the live release.

Fix

Run ox check in the repo first: it prints Ready to deploy., or a list of problems with the field each one is about. Then reproduce the failing command locally, since the log shows each one as it ran. Set missing variables on the dashboard or with ox vars set <project> KEY; Variables shows how.

Domain not working: waiting for DNS, points elsewhere, no certificate

Symptom

The domain's row on the project's Settings tab says waiting for DNS, points elsewhere or No certificate instead of HTTPS on.

Cause

The name does not resolve to the server yet, or it resolves somewhere else, or a Cloudflare proxy in front of it has no certificate for it. The row says which one, for example It resolves to 203.0.113.9, not this server. Change the record to: followed by the A record to create.

Fix

Create or change the record the row shows, at your DNS provider (how to point a domain). From a terminal, ox domains <project> lists each domain with its DNS state and prints Point each domain's A record at <ip>. The row turns green by itself once the record resolves.

For No certificate behind Cloudflare, check the certificate settings for the name in Cloudflare, or set its record to DNS only so the server makes its own certificate.

another operation is already running

Symptom

another operation is already running for shop: the daily backup, run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow); wait for it to finish, then retry

Cause

ox runs one thing at a time per project: a deploy, a rollback, a backup, a restore or a delete. A push, the daily backup or another person may have started one a moment before you.

Fix

Wait for it to end, then run your command again. The message names the run that holds the project, and the ox logs command in it prints that run's log until it ends. ox runs shop lists the project's other runs.

interrupted by ox, not the app

Symptom

deploy interrupted: interrupted by ox, not the app: the ox agent stopped mid-run (an ox update or a reboot); ox is deploying it again as run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow)

Cause

ox itself cut the run short: the ox agent on your server restarted (an update or a reboot), or the server did not reconnect within an hour. Your app did nothing wrong, and a step the deploy did not finish leaves the previous release serving.

Fix

Usually nothing: ox deploys an interrupted deploy again once, at once or when the server is back, and the message names that run. Follow it with the ox logs command it gives. When the message ends with deploy again or run it again, ox did not retry (a rollback, a retry that was interrupted too, or a newer run replaced it), so run your command again.

Disk full: a deploy needs at least 2 GB

Symptom

disk: only 1400 MB free on /srv/ox; a deploy needs at least 2 GB
fix: delete old projects or grow the disk

sudo ox doctor on the server says only 4% of the disk is free (…); deploys and backups fail soon once less than 5% is free.

Cause

The server's disk is close to full: releases, backups, logs and your app's own files share it. ox deletes older releases as the disk runs low, and keeps one project's backups under 20% of the disk, but it always keeps the newest release you can roll back to and the newest backup of each service.

Fix

Delete projects you no longer need, or resize the disk at your provider. Backups move off the server when you add an S3 bucket, so fewer stay on the disk. See how many backups ox keeps.

Not enough memory: killed by the out-of-memory killer

Symptom

Preflight refuses the deploy, a build step dies, or an app, worker or service is stopped as it starts:

capacity: the host would need 2304 MB but has 1987 MB: …
capacity: previews on this server share 3968 MB, half of it, so production keeps the rest; they would need 4096 MB: …
killed by the out-of-memory killer (the build needs more memory than the project or host allows)
killed by the out-of-memory killer: the whole server ran out of memory while it ran (the apps and services running on it held the rest)
killed by the out-of-memory killer as it started: it needs more memory than its limit allows
killed by the out-of-memory killer as it started: the previews on this server ran out of the memory they share
killed by the out-of-memory killer as it started: the whole server ran out of memory (what else runs on it held the rest)

Cause

The apps, their services and their staging copies add up to more memory than the server has (a stopped project gives back its [limits] memory, but its services stay reserved), the previews add up to more than their half of it, or one build step needs more than its limit. A build step gets at most three quarters of the server's memory, or the project's [limits] memory when that is lower. The last line means the step stayed under its own limit, but the whole server ran out: the apps and services running on it held the rest. Build and migrate steps run one at a time on a server, so another build is not the cause: a step that waits names the one it waits for (waiting for the build of shop (run ...) to finish), and ox stops a step whose run was cancelled or whose agent restarted. A [models] download is not a build step and may run beside one. An app, worker or service has no memory limit unless ox.toml sets one (its own memory, or the project's [limits] memory; a preview also shares half the server with the other previews), so one stopped as it starts went past that limit, a preview's went past the previews' share, or the whole server ran out.

Fix

For the capacity check, the fix line says it: stop a project whose [limits] memory is listed, delete one you do not need, drop a service or make it shared, lower a [limits] memory, or use a bigger server, or for previews stop a preview whose [limits] memory is listed, remove one you do not need, lower this preview's [limits] memory, or use a bigger server. ox stop frees a project's [limits] memory at once, but not its services' memory: delete the project, or make a service shared, to free that too. A refused ox start gives the same message and leaves the project stopped: stop a project whose [limits] memory is listed or delete one you do not need first, or lower the project's [limits] memory and deploy. A refused ox rollback to a release with a larger [limits] memory says the same and changes nothing. For a build step, raise memory in ox.toml if it sets one, or use a bigger server. For an app, worker or service stopped as it starts, raise its memory in ox.toml (or [limits] memory if the project sets one), or use a bigger server. When the whole server ran out, use a bigger server, or stop a project, staging or preview you do not need. Fewer previews at once also frees memory. A preview that ran out says so: all previews share half the server's memory, so production keeps the rest, and the fix is to remove another preview.

A Node build stops: ran out of memory at its heap limit

Symptom

A build step ends with exit code 134 and the build output shows JavaScript heap out of memory. The failure names the heap Node reached, read from its last trace:

Node ran out of memory at its heap limit, about 2082 MB

Cause

Node limits its heap: to about half the memory it may use, and never past about 2.2 GB. A build step that may use more than 5 GB gets half its memory from ox instead; a running app keeps Node's own limit. A large front-end build (a big Vite or webpack bundle in a monorepo) can need more.

Fix

Set the variable NODE_OPTIONS to the size the failure suggests, half again above the heap it reached and at least --max-old-space-size=4096 (megabytes), on the Variables screen. When the variable already carries other flags, add this one beside them: the variable replaces ox's. When the build command sets --max-old-space-size itself, that flag wins over the variable, so the failure says to raise it in the command instead. The server needs that much free memory while the build runs, and a [limits] memory cap must be higher than the number you set.

A compiled module needs a system library this server does not have

Symptom

A build, a migration or the app's start fails, and the output shows cannot open shared object file. The failure names the library:

a compiled module needs the system library libcairo.so.2, which this server does not have

Cause

A package with compiled code (a Node addon, a Python extension) loads a library from the operating system when it starts. A Docker image often has it already; a fresh Ubuntu server has only what ox and your packages install.

Fix

Add the Ubuntu package that ships the file to packages in ox.toml (for libcairo.so.2, packages = ["libcairo2"]). packages.ubuntu.com finds the package by the file's name. When this Ubuntu ships only another version of the library (Ubuntu 26.04 has libxml2.so.16, not libxml2.so.2), the module needs a build made for it, or a version of the module that does not use it.

The app keeps crashing and restarting

Symptom

The deploy passed, but later an alert says:

web of shop on my-server keeps crashing: it restarted 5 times in the last 10 minutes. The Logs tab shows why.

When the server stopped it for running out of memory, the alert says that instead, with the fix:

web of shop on my-server keeps crashing: it restarted 5 times in the last 10 minutes, and it ran out of memory: it needs more than its limit allows. Raise its memory in ox.toml (the memory of its own entry, or [limits] memory if the project sets one), or use a bigger server.

Cause

The process starts, then exits, and the server restarts it again and again: after 2 seconds, then waiting longer each time up to once a minute, so a broken process never floods the server. It never gives up, so it comes back by itself once what it waits for (a database, a network) is back. A missing variable, a lost database connection, a full disk or too little memory are common reasons. A deploy whose service or HTTP worker does not come up puts back the version that ran before, or stops it, so a failed deploy never leaves one restarting.

Fix

Open the Logs tab, pick Errors, and read the lines just before each restart, or run ox logs shop. See Logs. If the last deploy caused it, roll back while you fix it. An alert that says it ran out of memory carries its fix; Not enough memory says more.

Server offline: the server has not connected yet

Symptom

A deploy or a command is refused with one of these:

server my-server has not connected yet
server my-server went offline 12 minutes ago

Cause

The ox agent on the server is not talking to ox. The server may be off or off the network, or the agent may have stopped. Your sites keep running unless the server itself is down, but ox cannot deploy to it until it reconnects.

Fix

Check that the server is on at your provider. Then, on the server:

sudo systemctl status ox-agent
sudo journalctl -u ox-agent -n 50

sudo systemctl restart ox-agent starts it again. If the server was rebuilt, add it again from the dashboard.

Install refused: this server can't take ox yet

Symptom

The Add server command stops before it changes anything:

this server can't take ox yet; nothing was changed:
  - port 80 is in use by nginx; ox's proxy (Caddy) needs it: stop that program, or use a fresh server
  - the server has 512 MB of RAM and ox needs at least 1 GB: resize it
  - the disk has 6.2 GB free on / and ox needs at least 10 GB: free space or resize the disk

Cause

ox needs ports 80 and 443 free for Caddy, at least 1 GB of memory and at least 10 GB of free disk.

Fix

Do what each line says, then press Add server for a new command and run it: each command works once. sudo ox install --check runs only these checks. A fresh Ubuntu LTS server is the easy way.

repository access removed: GitHub gives ox no access

Symptom

repository access removed: GitHub gives ox no access to acme/shop (Not Found); connect GitHub again and pick the repository

Cause

The ox GitHub App was removed from the repository or the account on GitHub, or the repository was renamed or moved.

Fix

Connect GitHub again from the dashboard and pick the repository, then deploy.

not signed in; run ox login

Symptom

not signed in; run ox login
the plane refused the token (expired or revoked); sign in again with ox login

Cause

This computer has no token yet, or its token expired or was revoked in the dashboard. The command exits with status 3.

Fix

Run ox login and approve the code in the browser. A could not reach … message with status 4 is the network, not the sign-in. See exit codes.