Server operations

Deploy or upgrade#

1. Obtain the exact trusted Docker package; identify its source digest/backend, instance directory and endpoints.
2. Verify a private backup of configuration, credentials, management SQLite and report storage. A SQLite backup must include committed WAL state; do not copy only a live main file.
3. Stop any live operator normally. A second live operator is refused.
4. Start the selected release using –package against the same private directory.
5. The operator validates and removes exact owned runtime/migration orphans, stops writers, verifies the required upgrade backup, runs Knex, then starts the selected application.
6. Check API/native readiness, source and tool provenance, database integrity, retained identities and intended controller/session state.
7. If controller updates are authorized, update explicit targets from verified installed assets, restart that controller and inspect its actual bindings.
8. Verify affected user flows in a real browser.

The operator stops if a phase fails. After migration, check the browser and fleet separately. Updating the website, publishing npm packages and starting agents are separate operations.

Operator ownership#

The Node, C# and Rust operators share the private operator.lock inode and exclusive SQLite locking protocol. The lock spans package verification/import, reconciliation, backup, migration, runtime execution and cleanup. It is not an expiring PID lease; process death releases the lock. Never delete its inode to bypass a live owner.

Runtime and migration container names derive from the exact state directory. Both carry org.crossfire.directory. Before stopping anything, takeover verifies every matching container’s exact ID and ownership label. It then stops/removes those IDs and confirms absence.

A running or exited orphan can be reconciled. Foreign/missing ownership labels, ambiguous inspection, Docker failure or incomplete removal block startup. Do not add --rm, remove a container by a guessed name or treat an inspect error as proof that no writer exists. Concurrent starts cannot overlap migrations.

Persistent data#

Knex is the only SQL migration owner. Schema 17 accepts the inventoried 13–16 forward conversion and current-schema adoption; foreign/future or older unsupported stores are refused. Application startup validates schema identity, version and applied migration names without creating or upgrading tables.

Conduit’s RocksDB, static reports and fleetd/native histories have different owners and are not reset by a SQL migration. Replacing the backend does not import/export management data.

• Migration layout and commands
• Docker package format and site adoption
• Logging retention and protected records
• Public report storage

Diagnose unavailable agents#

fleet test performs model-free checks. For running Codex, attachment prints the exact current terminal command. A stale pointer must be replaced after restart. Lost contact is unknown execution state, not proof that the native process died.

Failure boundaries#

Matrix/provider/Docker outages can prevent work and delay notices. Unexpected failure is contained to the affected instance where possible. Uncertain native input remains durable until receipt recovery or the documented restart-context path settles it. Cancellation does not undo external actions.

Do not discard journals, alter schema metadata, erase histories or relax permissions to clear an error. Preserve the original failure for diagnosis.

• Runtime and decisions
• Fleet session controls
• Operator checks