# Oculo documentation

Oculo is a self-hosted platform for **logs and errors**. Your apps send log events, and Oculo stores them, makes them searchable, groups errors into issues, and alerts you when something breaks. People use the web dashboard. Scripts and LLM agents (Claude, ChatGPT, Cursor…) use the same data through a REST API, a CLI and an MCP server.

These docs are written for both people and LLMs. A running server shows all of them as one page at [`/docs`](https://oculo.funonymous.in/docs), serves the same content as raw Markdown at [`/docs.md`](https://oculo.funonymous.in/docs.md), and has a shorter agent guide at [`/llms.txt`](https://oculo.funonymous.in/llms.txt).

## Quick start

1. **Start the server** (Node 20+):
   ```bash
   npm install
   npm run dev          # https://oculo.funonymous.in
   ```
2. **Create the first account.** Open the dashboard. The first account becomes the instance admin. (An admin API key is also printed on first start and saved to `server/data/admin-key.txt`.)
3. **Create a project** on the Projects page. You get an **ingest key** (to send logs) and a **read key** (to query, which is safe to give to an LLM).
4. **Send a log:**
   ```bash
   curl -X POST https://oculo.funonymous.in/api/v1/ingest \
     -H "Authorization: Bearer oc_ingest_…" -H "Content-Type: application/json" \
     -d '[{"level":"error","message":"Payment failed","service":"api","attributes":{"orderId":8812}}]'
   ```
5. **Look at it.** It shows up within a second on **Logs** and **Live**, and as an **Issue** because it's an error.

For demo data, run `npm run seed` while the server is running. It loads about 24 hours of realistic logs across two releases.

Next: see everything Oculo can do in [Features](#features), run it for real with [Deployment](#deployment), and when volume grows, read [Scaling](#scaling-clickhouse-and-redis).

## Features

A tour of everything Oculo does. Each item links to the page with the details.

### Collect

- **One HTTP endpoint** takes JSON arrays, single objects or NDJSON, optionally gzip/deflate/brotli compressed. Common field aliases (`msg`, `severity`, `@timestamp`, `traceId`…) are understood and unknown fields are kept as attributes. See [Sending logs](#sending-logs).
- **JavaScript/TypeScript SDK** for Node and browsers: batching, retries, flush on exit, uncaught-error capture, Express middleware, per-request breadcrumbs, `console` capture, source-map upload.
- **Python SDK** with no dependencies: stdlib `logging` bridge, ASGI (FastAPI/Starlette) and Django middleware, per-request breadcrumbs.
- **OpenTelemetry**: any OTLP/HTTP log exporter (protobuf or JSON) works, so Go, Java, .NET, Rust and others need no Oculo SDK.
- **Heartbeats** for cron jobs: one `curl` at the end of the job; a missed check-in alerts you.
- **Privacy at ingest**: passwords, tokens, cookies, auth headers, JWTs and card numbers are scrubbed on the server; email masking and extra keys are per project.
- **Sampling and drop rules** per project (for example keep 10% of debug logs, drop health checks). Errors are never sampled away.

### Find

- **One query language** everywhere (dashboard, API, CLI, MCP, alerts): free text, phrases, exclusions, `field:value`, comparisons, wildcards, nested and dotted attributes, `has:`, time windows. See [Query language](#searching-the-query-language).
- **Logs page**: autocomplete, filter chips, facets with *only* / *exclude*, a level-stacked volume chart you can drag to zoom, saved and recent searches, NDJSON/CSV export, shareable URLs.
- **Log detail**: every field with filter/exclude/copy, source-mapped stack trace, attributes, breadcrumbs, surrounding logs, the whole trace, "Copy for LLM".
- **Live tail** with your filter applied, pausable, reconnecting.
- **Log patterns**: millions of similar lines collapse into templates like `Charging order <*>`, with counts and trends. See [Log patterns](#log-patterns).
- **Metrics from logs**: count, avg, sum, min, max, p50/p90/p95/p99 of any numeric attribute, over time, grouped by any field. No separate metrics pipeline. See [Metrics](#metrics).
- **Traces**: every log sharing a `trace_id`, in order, across services.

### Understand errors

- **Issues**: errors are fingerprinted (exception type, message with variables stripped, top in-app frames) so a thousand occurrences are one issue with a count, affected users, releases and a 48-hour/30-day timeline. See [Issues](#issues-and-grouping).
- **Regressions**: a resolved issue that happens again reopens and is flagged.
- **Custom grouping** with a `fingerprint` field.
- **Source maps**: minified JavaScript stacks are rewritten to original files, lines and functions before grouping. See [Releases and source maps](#releases-and-source-maps).
- **Releases**: error rate per release, deploy markers on charts, and "what changed" between two releases (new, increased, gone issues).
- **✨ Explain** (opt-in): sends one issue to Claude for a likely cause and fix, with consent and caching.

### Get told

- **Alert types**: threshold, spike (vs. a 12-window baseline), new issue, regression, missed heartbeat, daily digest. See [Alerts and heartbeats](#alerts-and-heartbeats).
- **Channels**: Slack, Discord, email (SMTP), Telegram, PagerDuty (with auto-resolve), any JSON webhook.
- **Cooldowns, test sends and delivery history**; secrets are masked when read back.

### Work with LLMs and scripts

- **MCP server** with tools such as `project_summary`, `search_logs`, `get_issue`, `query_metrics` and `explain_issue`; read-only unless you opt in. See [Using Oculo from an LLM](#using-oculo-from-an-llm).
- **Agent-friendly REST**: `format=md` for compact tables, `fields=` to trim rows, cursors, hard limits, `/llms.txt`, `/openapi.json`, `/docs.md`.
- **CLI**: `oculo search`, `tail`, `issues`, `summary`, `sourcemaps upload`, `heartbeat`, with `--json` for piping. See [CLI](#cli).

### Run it your way

- **Single process, single file by default**: Node 20 + SQLite (WAL, FTS5). `npm install && npm run dev` and you're running.
- **Scale out when needed**: store logs in **ClickHouse** and put **Redis Streams** in front of ingest by setting two environment variables. Search, issues, alerts and the API behave the same. See [Scaling](#scaling-clickhouse-and-redis).
- **Teams**: email/password accounts and GitHub sign-in, invite links, per-project roles (viewer, member, owner) and instance admins; scoped API keys for apps and agents. See [Concepts](#concepts).
- **Retention** per project, with hourly rollups and issue timelines kept much longer.
- **Docker Compose** file for Oculo + ClickHouse + Redis. See [Deployment](#deployment).

## Concepts

| Thing | What it is |
|---|---|
| **Project** | A container for one app or service's logs, issues, alerts and keys. Most people create one per app. |
| **Log event** | One line your app logged: a timestamp, level, message, and optional service, environment, release, host, trace id, user, attributes and exception. |
| **Issue** | A group of identical errors, such as "TypeError: Cannot read properties of undefined". Oculo fingerprints each error, so a thousand occurrences become one issue with a count, users, releases and a timeline. |
| **Pattern** | A template for similar non-error messages, such as `Charging order <*>`, so a million info logs read as a few dozen lines. |
| **Release** | The version string your app reports. It powers "new in this release" and deploy markers. |
| **Heartbeat** | An expected check-in from a cron job. A missed check-in triggers an alert. |

### Levels

`trace` < `debug` < `info` < `warn` < `error` < `fatal`. Common aliases are understood: `warning`, `err`, `critical`, pino numbers 10–60, and OTel severity numbers. Anything at `error` or above, or carrying an exception, becomes part of an issue.

### Keys, accounts and roles

There are two ways in, and both end up with a **role** on a project.

| Credential | Use it for | Role |
|---|---|---|
| **Ingest key** `oc_ingest_…` | Sending logs from apps, OTel collectors and heartbeats. It's safe to ship in app code. | Can only send |
| **Read key** `oc_read_…` | Scripts, the CLI, LLM agents and MCP | viewer (read-only) |
| **Read/write key** (a read key created with `can_write: true`) | Agents that may also resolve issues, save searches, manage heartbeats or upload source maps | member rights, **except** alerts, keys, members and settings |
| **Admin key** `oc_admin_…` | Bootstrap and automation across all projects | admin |
| **Account** (email/password or GitHub) | People using the dashboard | viewer, member or owner, per project |

- **viewer:** read everything in the project.
- **member:** also resolve, ignore and assign issues, save searches, manage heartbeats and upload source maps. Member *accounts* can also manage alerts; read/write keys can't, because alerts send requests to outside services.
- **owner:** also change project settings, manage keys and people, and delete the project.
- **admin** (instance admin, such as the first account or an admin key): everything, in every project, including creating projects.

Send keys as `Authorization: Bearer <key>`. The `X-Oculo-Key` header and the `?key=` query parameter also work. The dashboard uses an HttpOnly session cookie instead of keys when you sign in with an account.

## Sending logs

Every SDK is a thin wrapper around one endpoint, so a language without an SDK can still send logs over plain HTTP.

### HTTP (any language)

`POST /api/v1/ingest` with an ingest key. The body can be a JSON array of events, a single object, `{"events": […]}`, or NDJSON. It may be gzip, deflate or br encoded. The limits are 5,000 events and 10 MB (decompressed) per request.

```json
{
  "timestamp": "2026-11-14T16:20:01.123Z",
  "level": "error",
  "message": "Payment failed for order 8812",
  "logger": "billing.stripe",
  "service": "api", "environment": "prod", "release": "1.4.2", "host": "web-3",
  "trace_id": "abc123", "span_id": "def456",
  "user": { "id": "u_42", "email": "…" },
  "attributes": { "orderId": 8812, "amount": 499, "route": "/checkout" },
  "exception": { "type": "CardError", "value": "card_declined", "stacktrace": "…" },
  "fingerprint": ["optional", "custom", "grouping"]
}
```

- Only `message` is really needed; everything else is optional.
- **Nothing is lost:** unknown top-level fields move into `attributes`.
- **Aliases work:** `msg`, `severity`, `@timestamp`, `env`, `traceId` and similar names are understood.
- **Timestamps:** ISO strings, or epoch seconds, ms, µs or ns. Timestamps more than 5 minutes in the future are clamped to now.
- **Custom grouping:** `fingerprint` overrides issue grouping. Use `"{{ default }}"` inside it to extend the default fingerprint instead of replacing it.

The response is `202 {"accepted": n, "dropped": m}`. Events are dropped only by the project's sampling and drop rules. With the Redis queue on, the response is `202 {"accepted": n, "dropped": 0, "queued": true}` (rules run later), and a full queue answers `503` with `Retry-After`; see [Scaling](#scaling-clickhouse-and-redis).

### JavaScript / TypeScript

```ts
import { Oculo } from '@gouravsrf/oculo';   // npm install @gouravsrf/oculo  (Node 18+ and browsers, ESM + CommonJS)

const log = Oculo.init({ key: 'oc_ingest_…', url: 'https://oculo.funonymous.in', env: 'prod', release: '1.4.2', service: 'api' });

log.info('User signed in', { userId: 42 });
log.error('Payment failed', { orderId }, err);
log.captureException(err);
const db = log.child({ logger: 'db' });          // child logger with extra context
log.setUser({ id: 'u_42' }); log.setTag('region', 'ap-south');

// Express: request logs, a request id + trace id on every log, and per-request breadcrumbs
app.use(log.expressMiddleware());
app.use(log.expressErrorHandler());

// Any other code path: give a unit of work its own breadcrumbs + context
await log.withScope(() => handleJob(job), { job_id: job.id });
```

- **Batching:** sends every 2 s or every 100 events, gzipped, with exponential backoff that honours `Retry-After`. The buffer is bounded at 1,000 events and drops the oldest when full.
- **Never in your way:** it never throws into your code and never blocks it.
- **Flushes on exit,** and through `sendBeacon` when a browser tab is hidden.
- **Automatic capture:** uncaught exceptions (it still exits with code 1), unhandled rejections, `window.onerror`, and optionally `console.*` (`captureConsole: true | 'breadcrumbs'`).
- **Breadcrumbs:** the last 50 lower-level logs are attached to each error. With the middleware or `withScope` they are **per request**, so concurrent requests never mix.
- **Filtering:** `beforeSend(event)` lets you edit or drop events, and `sampleRate` keeps a fraction of non-error events.
- **Source maps:** `Oculo.uploadSourceMap({url, key, release, file, map})` uploads one; see [Releases and source maps](#releases-and-source-maps).

### Python

```python
import oculo                     # pip install /path/to/oculo/sdk-python  (3.8+, standard library only)

log = oculo.init(key="oc_ingest_…", url="https://oculo.funonymous.in", env="prod", release="1.4.2", service="worker")
log.info("User signed in", user_id=42)
try:
    charge()
except Exception:
    log.exception("Payment failed", order_id=8812)

import logging
logging.getLogger().addHandler(oculo.OculoHandler())        # bridge the stdlib logging module

with oculo.scope(request_id="r1", trace_id="abc"):          # per-request breadcrumbs + context (thread/asyncio safe)
    handle()

# FastAPI / Starlette:  app.add_middleware(oculo.middleware.OculoASGIMiddleware)
# Django:               MIDDLEWARE = ["oculo.middleware.OculoDjangoMiddleware", ...]
```

It behaves like the JS SDK: a background sender thread, batching, gzip, retries, flush at exit, and capture through `sys.excepthook` and `threading.excepthook`.

### OpenTelemetry (Go, Java, .NET, Rust, …)

Point any OTLP log exporter at Oculo. Both **protobuf** (the default) and **JSON** are accepted:

```bash
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://oculo.funonymous.in/v1/logs
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf        # or http/json
OTEL_EXPORTER_OTLP_LOGS_HEADERS=Authorization=Bearer%20oc_ingest_…
```

Resource attributes map to service (`service.name`), release (`service.version`), environment (`deployment.environment.name`) and host (`host.name`). Trace and span ids become hex, and `exception.*` attributes become the exception.

### Heartbeats from cron jobs

```bash
# at the end of the job (creates the heartbeat on first ping when ?period is given)
curl -fsS -X POST "https://oculo.funonymous.in/api/v1/heartbeats/nightly-backup/ping?period=1d" -H "Authorization: Bearer oc_ingest_…"
# or with the CLI
oculo heartbeat nightly-backup
```

## Searching: the query language

One search bar, used everywhere: the dashboard, the API (`q=`), the CLI, MCP tools and alert queries.

```text
payment failed level:>=warn env:prod,staging -healthcheck
@status:>=500 @route:/checkout* release:1.4* trace:abc123 user:u_42
"connection refused" has:exception since:2h until:30m
```

| Syntax | Meaning |
|---|---|
| `payment failed` | Full-text **prefix** match over message, exception and attributes (`pay` matches "payment"). Terms are ANDed. |
| `"exact phrase"` | An exact phrase |
| `-term`, `-field:value` | Excludes matches |
| `field:value` | An exact value. `,` means OR (`env:prod,staging`); `*` is a wildcard (`release:1.4*`) |
| `field:>=v` `>` `<=` `<` `!=` | Comparisons; numeric when the value is a number |
| `@path.to.attr:value` | Attribute filter, nested (`@user.plan:pro`) or flat OTel keys (`@http.status:503`). Unknown field names are treated as attributes. |
| `has:exception` `has:user` `has:trace` `has:release` `has:@attr` | The field is present |
| `since:2h` `until:30m` | A time window: relative (`s m h d w`), ISO or epoch ms |

Built-in fields:
- `level` (supports `>=warn`)
- `env`/`environment`, `service`, `release`, `host`, `logger`
- `trace`, `span`, `user`
- `type` (exception type), `issue` (issue id), `pattern` (pattern id), `id`, `message`

Free text is case- and accent-insensitive (`cafe` finds "Café"). Queries return the same results whether logs are stored in SQLite or ClickHouse (see [Scaling](#scaling-clickhouse-and-redis)).

Every value is a bound SQL parameter or a quoted full-text phrase, so queries can't inject anything. Malformed queries return 400 with a hint.

## The dashboard

- **Overview:** the project's health for 24h or 7d.
  - Headline numbers: logs, errors, error rate and open issues, each compared with the previous window.
  - A volume chart by level with deploy markers.
  - Top issues, issues needing attention (new or regressed), top error messages grouped by template, and the latest releases.
- **Logs:** search with autocomplete. An empty box offers your saved and recent searches.
  - Active filters show as chips. The sidebar facets toggle on and off, with *only* and *exclude* actions.
  - Time presets, a custom range and auto-refresh.
  - The chart shows volume stacked by level: drag across it to zoom, or click the legend to filter.
  - A **Logs / Patterns** switch.
  - Click a row for the detail drawer: fields with filter, exclude and copy actions, the stack trace (source-mapped when possible), attributes, breadcrumbs, surrounding logs, the trace, Copy for LLM and Copy JSON.
  - Export to NDJSON or CSV.
- **Live:** a real-time stream with your filter applied. Pause with `space`; events that arrive while paused are buffered.
- **Issues:** errors grouped by fingerprint, with sparklines and new/regressed/resolved badges. Select several to resolve, ignore or reopen them at once.
  - The issue page shows a timeline, the stack trace, breadcrumbs, attributes, tags, releases, affected users and recent events.
  - It has **✨ Explain** (when enabled), Copy for LLM, resolve (`r`) and assign.
- **Metrics:** charts computed from logs. See [Metrics](#metrics).
- **Releases:** error rate per release, and a comparison of two releases (new, increased and gone issues).
- **Alerts:** alert rules, their history, and **Heartbeats**.
- **Projects:** create projects and get keys and setup snippets.
- **Settings:** the project, retention, privacy (email masking, extra sensitive keys), sampling and drop rules, the **Team** panel (members, roles, invite links), API keys, saved searches and project deletion.

Keyboard shortcuts:
- `Ctrl/⌘ K` opens the command palette.
- `/` focuses search.
- `j` / `k` move between rows; `Enter` opens the selected one.
- `g` then `o`, `l`, `t`, `i`, `m` or `r` jumps to a page.
- `?` shows help.

The query and time range live in the URL, so any view can be shared as a link. There are dark and light themes, relative or absolute times in your local timezone, and the layout works on phones.

## Issues and grouping

Each error gets a **fingerprint** built from:
- the exception type
- the message with variable parts stripped (UUIDs, emails, URLs, IPs, dates, hex ids, quoted strings and numbers, including `3036ms`)
- the top 3 frames from your own code, without line numbers

Library frames (`node_modules`, `site-packages`, the JVM and so on) are skipped. So the same bug on different hosts, for different orders, groups into one issue.

- **Statuses:** unresolved, resolved or ignored. If a **resolved** issue happens again, it reopens automatically and is marked **regressed**.
- **Custom grouping:** send `fingerprint: ["checkout", "{{ default }}"]` to extend the default fingerprint, or `["my-group"]` to replace it.
- **Explain:** if the server sets `ANTHROPIC_API_KEY`, the issue page shows **✨ Explain**, which sends the issue's export to Claude (`claude-opus-5-5`, with server-side refusal fallback on) for a likely cause and fix. It is cached per issue. Issue data leaves your server when you use it, and the dashboard asks once before sending.

## Log patterns

Oculo turns each message into a template by replacing values with `<*>`: numbers, ids, UUIDs, IPs, URLs, quoted strings, path segments containing digits, and the values in `key=value` pairs. So `Charging order 8761` and `Charging order 9283` become one pattern, `Charging order <*>`.

On the Logs page, switch to **Patterns**. It respects your current query and time range. Click a pattern to see its logs (`pattern:<id>`), or exclude it (`-pattern:<id>`). Patterns are capped at 5,000 per project.

## Metrics

Charts computed from your logs, with no separate metrics pipeline:
- **Functions:** `count`, `avg`, `sum`, `min`, `max`, `p50`, `p90`, `p95`, `p99`.
- **Field:** any numeric attribute, such as `@duration_ms`. Not needed for `count`.
- **Grouped by** service, environment, release, host, logger, level, or any attribute such as `@route`. You get the top 8 groups, and the rest fold into "other".
- **Filter** with any query.

Presets include "p95 latency by route" and "errors by service". You can click a series in the legend to hide it, and the table under the chart links to each group's logs.

API: `GET /api/v1/metrics?fn=p95&field=@duration_ms&group_by=@route&q=service:api&since=24h`. Use `GET /api/v1/fields` to discover attribute names and types.

The response:
- `buckets`: the start time of each time bucket. The bucket size is `interval_ms`, chosen to give about 60 buckets unless you pass `interval=5m|1h|1d`.
- `series`: one entry per group, each with:
  - `group`
  - `values`: one number per bucket, `null` where there was no data
  - `total`: the function over the whole window, e.g. the overall p95
  - `count`: matching logs
- `rows_scanned`: how many logs were read.
- `truncated`: `true` if a percentile hit the 500,000-row scan limit, so only the most recent rows were used.

## Alerts and heartbeats

**When** an alert fires:
- **Threshold:** at least N logs matching a query within M minutes.
- **Spike:** at least N matching logs **and** 3× the average of the previous 12 windows.
- **New issue:** a new issue appears.
- **Regression:** a resolved issue happens again.
- **Heartbeat missed:** a heartbeat's check-in is overdue (period + grace). A "recovered" notice follows the next check-in.
- **Daily digest:** a summary of the last 24h at a chosen UTC hour.

**Where** it's sent:
- **Slack** and **Discord:** an incoming-webhook URL.
- **Webhook (JSON):** any URL that accepts JSON. The payload includes `alert`, `project`, `subject`, `message`, `details`, `url` and `fired_at`.
- **Email:** SMTP, configured on the server.
- **Telegram:** a bot token plus a chat id.
- **PagerDuty:** an Events v2 integration key. Missed heartbeats resolve the incident on recovery.

A cooldown stops repeats, **Send test** checks the wiring, and **History** shows every delivery and its result. Secrets (bot tokens, routing keys) are masked whenever an alert is read back.

**Heartbeats** live on the Alerts page:
1. Create one with a slug, an expected period and a grace time.
2. Copy its `curl` command into the end of your job.
3. Add a **Heartbeat missed** alert.

## Releases and source maps

Set `release` in your SDK. The Releases page shows the error rate per release and compares two releases. Deploy markers appear on every chart.

For **minified JavaScript**, upload source maps per release. Stack traces are then rewritten to the original file, line and function before grouping. The original minified stack is kept, and the dashboard can show it.

```bash
oculo sourcemaps upload --release 1.4.2 ./dist          # uploads every *.js.map under ./dist
```

```ts
await Oculo.uploadSourceMap({ url: 'https://oculo.funonymous.in', key: 'oc_read_…(read/write)', release: '1.4.2', file: 'app.3f9a1c.js', map });
```

Uploading needs a read/write key or a member account; ingest keys can't upload. The `release` must match what the app reports.

## Using Oculo from an LLM

Give an agent one link: **`/llms.txt`**. It has everything an agent needs: keys and permissions, how to add the MCP server, the query language, and every endpoint with its request body (generated from the server, so it's always complete). `/openapi.json` has the same in OpenAPI 3.1 form.

- **MCP server** (Claude Code, Claude Desktop, Cursor, …):
  ```bash
  claude mcp add oculo -- npx -y @gouravsrf/oculo-mcp --url https://oculo.funonymous.in --key oc_read_…
  ```
  For Claude Desktop, Cursor and other clients, use the same command in their MCP config: `"command": "npx", "args": ["-y", "@gouravsrf/oculo-mcp", "--url", "…", "--key", "oc_read_…"]`. Give it a **read key** so the LLM can't change anything.
  The tools:
  - `project_summary`: the best first call
  - `search_logs`, `get_log_context`
  - `list_issues`, `get_issue`, `explain_issue`
  - `compare_releases`, `list_patterns`, `query_metrics`, `list_fields`, `list_projects`
  - `resolve_issue`: only with `--allow-write` and a read/write key
- **REST for agents:** every read endpoint takes:
  - `format=json|md|text`: `md` gives compact tables that fit more in context.
  - `fields=a,b.c`: trims each row.
  - `limit`, plus a `next_cursor` for paging. There are hard caps.
- **Discovery:** [`/llms.txt`](https://oculo.funonymous.in/llms.txt), [`/openapi.json`](https://oculo.funonymous.in/openapi.json) and this page ([`/docs.md`](https://oculo.funonymous.in/docs.md)).
- **Copy for LLM:** a button on every log and issue copies a compact Markdown export you can paste into any chat.

A good agent workflow:
1. Call `project_summary`.
2. Call `list_issues sort=freq`.
3. Call `get_issue` on the top issue.
4. Use `search_logs` with `issue:<id>` or `trace:<id>` to see what happened around it.
5. Use `get_log_context` on one event.
6. Use `compare_releases` to check whether it started with a deploy.

## CLI

```bash
npm run build && npm install -g ./cli      # provides the `oculo` command (or: node cli/dist/index.js)
oculo login --url https://oculo.funonymous.in --key oc_read_…  # saved to ~/.config/oculo/config.json (mode 600)
oculo search 'level:>=warn service:api' --since 1h -n 50
oculo tail 'level:error'                    # live; reconnects automatically
oculo issues --sort freq
oculo summary
oculo sourcemaps upload --release 1.4.2 ./dist
oculo heartbeat nightly-backup
```

`--json` prints one JSON object per line, for piping into `jq`. Output is coloured on a terminal and plain when piped. `OCULO_URL`, `OCULO_KEY` and `OCULO_PROJECT` override the saved config.

## API reference

The base URL is `https://oculo.funonymous.in`. Errors look like `{"error": "...", "hint": "..."}` with a normal HTTP status. Read endpoints accept `format`, `fields` and paging. Admin keys and accounts with several projects add `project=<slug>` to **any** endpoint that works inside a project, reads and writes alike; paths with `/projects/:id` take the slug or the numeric id.

The complete list, with every parameter and request body, is generated from the server itself: [`/llms.txt`](https://oculo.funonymous.in/llms.txt) (for LLMs) and [`/openapi.json`](https://oculo.funonymous.in/openapi.json) (OpenAPI 3.1). The table below is a summary.

**Who** means:
- **ingest key:** an ingest key, read/write key, member account or admin key.
- **viewer:** any key or account with access to the project.
- **member:** a read/write key, a member or owner account, or an admin key.
- **owner:** an owner account or an admin key.
- **admin:** an admin key or an instance-admin account.

| Method and path | Who | What |
|---|---|---|
| `POST /api/v1/ingest` | ingest key | Send events (JSON array, object, NDJSON; gzip OK) |
| `POST /v1/logs` | ingest key | OpenTelemetry OTLP/HTTP logs (protobuf or JSON) |
| `GET /api/v1/summary` | viewer | Project health (best first call) |
| `GET /api/v1/logs/search?q&since&until&limit&cursor&order` | viewer | Search logs |
| `GET /api/v1/logs/:id`, `/:id/context?n&scope` | viewer | One log; with surrounding logs and its trace |
| `GET /api/v1/logs/histogram?q&since&until&buckets` | viewer | Counts over time by level, plus deploy markers |
| `GET /api/v1/logs/facets?q&facets` | viewer | Top values per field |
| `GET /api/v1/logs/export?q&as=ndjson\|csv` | viewer | Streamed export |
| `GET /api/v1/traces/:trace_id` | viewer | Every log in a trace |
| `GET /api/v1/tail?q&ticket` | viewer | Live tail (Server-Sent Events) |
| `POST /api/v1/tail/ticket?project` | viewer | One-time, 60 s ticket for the tail URL. The ticket remembers its project, so open `GET /api/v1/tail?ticket=…&q=…` without a key or `project` |
| `GET /api/v1/patterns?q&since&sort` | viewer | Log templates with counts |
| `GET /api/v1/metrics?fn&field&group_by&q&since&interval` | viewer | Metric series |
| `GET /api/v1/fields` | viewer | Attribute keys and types |
| `GET /api/v1/issues?status&sort&q&release&since` | viewer | List issues |
| `GET /api/v1/issues/:id`, `/:id/events` | viewer | Issue detail, events |
| `PATCH /api/v1/issues/:id` · `POST …/resolve`, `…/ignore`, `…/unresolve` | member (incl. read/write keys) | Change status or assignee |
| `POST /api/v1/issues/:id/explain?refresh=1` | viewer | AI explanation (when enabled) |
| `GET /api/v1/releases`, `/releases/compare?a&b` | viewer | Releases; what changed |
| `GET/POST/DELETE /api/v1/saved-searches` | viewer / member | Saved searches |
| `GET/POST/PATCH/DELETE /api/v1/alerts…`, `POST …/:id/test`, `GET …/:id/events` | member **account** or admin key (not project keys) | Alert rules |
| `GET/POST/PATCH/DELETE /api/v1/heartbeats…` | viewer / member | Heartbeats |
| `POST /api/v1/heartbeats/:slug/ping?period&grace` | ingest key or member | Check in |
| `GET/POST/DELETE /api/v1/sourcemaps?release` | viewer / member | Source maps |
| `GET /api/v1/projects`, `/projects/:id` | any | Projects you can see, with your role |
| `POST /api/v1/projects` | admin | Create a project (returns keys once) |
| `PATCH/DELETE /api/v1/projects/:id` | owner | Settings, delete |
| `GET/POST /api/v1/projects/:id/keys`, `DELETE /api/v1/keys/:id` | owner | API keys |
| `GET /api/v1/projects/:id/members`, `PATCH/DELETE …/members/:userId` | viewer / owner | Team |
| `POST/GET/DELETE /api/v1/projects/:id/invites` | owner | Invite links |
| `GET /api/v1/invites/:token`, `POST …/accept` | public / signed in | Join a project |
| `POST /api/v1/auth/signup`, `/login`, `/logout`; `GET /api/v1/auth/config`; `GET /api/v1/auth/github/start` | public | Accounts and GitHub sign-in |
| `GET /api/v1/me` | any | Who am I |
| `GET /healthz` | public | Liveness, storage mode (`sqlite`/`clickhouse`), queue mode, and `outbox_rows` / `queue_backlog` when those are on |

Responses worth knowing:
- `error_rate` in summaries and release comparisons is a **percent** (0–100), e.g. `2.79` means 2.79%.
- Search returns `{data, next_cursor, highlights, took_ms}`; `highlights` are the words your query matched, for highlighting.
- Metrics: see [Metrics](#metrics).

### Request bodies

All bodies are JSON (`Content-Type: application/json`).

**Create a project** (`POST /api/v1/projects`, admin). The reply contains `ingest_key` and `read_key`, shown once.
```json
{ "name": "Shop API", "retention_days": 30 }
```

**Create a key** (`POST /api/v1/projects/:id/keys`, owner). `kind` is `ingest` or `read`. A read key with `"can_write": true` is a read/write key. The reply's `key` is shown once.
```json
{ "kind": "read", "name": "agent", "can_write": false }
```

**Create an alert** (`POST /api/v1/alerts`, member account or admin key):
```json
{ "name": "API errors", "type": "threshold", "query": "level:>=error service:api", "threshold": 10, "window_min": 5,
  "channel": "slack", "target": "https://hooks.slack.com/services/…", "cooldown_min": 30 }
```
| Field | Values |
|---|---|
| `type` | `threshold`, `spike` (≥ threshold and 3× the previous 12 windows), `new_issue`, `regression`, `heartbeat`, `digest` |
| `query`, `threshold`, `window_min` | For `threshold` and `spike` (defaults: none, 10, 5) |
| `channel` | `webhook` (default), `slack`, `discord`, `email`, `telegram`, `pagerduty` |
| `target` | webhook/slack/discord: an http(s) URL · email: comma-separated addresses · telegram: chat id or `@channel` · pagerduty: unused |
| `config` | `bot_token` (telegram), `routing_key` (pagerduty), `hour` (digest, UTC 0–23, default 9), `heartbeat` (heartbeat slug; empty = all) |
| `cooldown_min`, `enabled` | Default 30, true |

`PATCH /api/v1/alerts/:id` takes the same fields, all optional. Secrets come back masked (`••••`); sending the masked value back keeps the stored secret.

**Create a heartbeat** (`POST /api/v1/heartbeats`, member). Durations are `30m`, `6h`, `1d`, `1w` or seconds (1 minute to 90 days):
```json
{ "slug": "nightly-backup", "name": "Nightly backup", "period": "1d", "grace": "15m" }
```

**Invite someone** (`POST /api/v1/projects/:id/invites`, owner). The reply has a `url` to send them, valid once for 7 days:
```json
{ "email": "friend@example.com", "role": "member" }
```

**Other bodies:**
- Change a member's role (`PATCH …/members/:userId`): `{"role": "viewer"}`
- Project settings (`PATCH /api/v1/projects/:id`): `{"name": …, "retention_days": 30, "settings": {"scrubEmails": true, "extraKeys": ["ssn"], "sampling": {"debug": 0.1}, "dropMessages": ["healthcheck"]}}`; `settings` is merged into the existing settings.
- Issue (`PATCH /api/v1/issues/:id`): `{"status": "resolved", "assignee": "alice"}`
- Saved search: `{"name": "API errors", "query": "level:>=error service:api"}`
- Source map (`POST /api/v1/sourcemaps?release=1.4.2`): `{"file": "app.3f9a1c.js", "map": {…}}`

## Configuration

Set these as environment variables on the server.

| Variable | Default | Purpose |
|---|---|---|
| `PORT`, `HOST` | `4318`, `0.0.0.0` | Listen address |
| `OCULO_DB` | `server/data/oculo.db` | SQLite file (WAL mode): metadata always, and log rows unless ClickHouse is on |
| `PUBLIC_URL` | from the Host header | Used in alert links, invite links, OAuth redirect and docs |
| `INGEST_URL` | `PUBLIC_URL` | Where apps should send logs, if different from the dashboard address (for example dashboard on Vercel, API on `oculo-api.example.com`). Used in setup snippets and heartbeat ping URLs. |
| `OCULO_ADMIN_KEY` | generated on first boot | Your own bootstrap admin key |
| `RETENTION_DAYS` / `ROLLUP_RETENTION_DAYS` | `14` / `400` | Default raw-log retention / hourly rollups |
| `INGEST_RATE_LIMIT` | `0` (off) | Events per minute per key |
| `MAX_BODY_BYTES` | 10 MB | Ingest body limit (decompressed) |
| `PRODUCT_NAME` | `Oculo` | Display name |
| `OCULO_OPEN_SIGNUP` | off | `1` lets anyone create an account (otherwise invite only after the first) |
| `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` | – | Enable "Continue with GitHub". The callback is `PUBLIC_URL/api/v1/auth/github/callback`. |
| `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASS`, `SMTP_FROM`, `SMTP_SECURE` | – | Email alerts |
| `ANTHROPIC_API_KEY` (or `OCULO_EXPLAIN=1` with another credential), `OCULO_EXPLAIN_MODEL` | – / `claude-opus-5-5` | Enable ✨ Explain |
| `CLICKHOUSE_URL`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_DATABASE` | off, `default`, empty, `oculo` | Store log rows in ClickHouse. See [Scaling](#scaling-clickhouse-and-redis). |
| `REDIS_URL` | off | Put a Redis Streams queue in front of ingest. See [Scaling](#scaling-clickhouse-and-redis). |
| `OCULO_QUEUE_STREAM`, `OCULO_QUEUE_GROUP`, `OCULO_QUEUE_BATCH`, `OCULO_QUEUE_MAX_BACKLOG`, `OCULO_QUEUE_CLAIM_IDLE_MS` | `oculo:ingest`, `oculo`, `100`, `50000`, `60000` | Queue tuning |
| `TELEGRAM_API_BASE`, `PAGERDUTY_EVENTS_URL`, `GITHUB_OAUTH_BASE`, `GITHUB_API_BASE`, `ANTHROPIC_BASE_URL` | the real services | Override service URLs (proxies, testing) |

Per-project settings (in Settings, or `PATCH /api/v1/projects/:id`):
- retention days
- `scrubEmails`
- `extraKeys` (more sensitive key names to scrub)
- `sampling` (for example `{"debug": 0.1}`)
- `dropMessages` (substrings to drop)

## Scaling: ClickHouse and Redis

Out of the box Oculo keeps everything in one SQLite file. That is fast and needs no setup, and it is the right choice for most teams: on a laptop it ingests about 20,000 events per second.

Two optional pieces take it further. Each one is turned on by a single environment variable, and they can be used separately or together:

| Piece | Turn on with | What it does |
|---|---|---|
| **ClickHouse** | `CLICKHOUSE_URL` | Stores the raw log rows in a ClickHouse table instead of SQLite. Columnar compression and fast scans keep search and metrics quick on tables far larger than SQLite handles comfortably. |
| **Redis Streams** | `REDIS_URL` | Puts a durable queue between the HTTP endpoint and processing. Ingest answers immediately; bursts wait in Redis instead of slowing requests down. |

Everything else (projects, keys, accounts, issues and their counters, rollups, patterns, alerts) always stays in SQLite. The dashboard, the query language, the API, the CLI and MCP behave the same in every mode; the test suite runs in all four combinations and a parity test checks that SQLite and ClickHouse return identical results for the same data and queries.

### When to use them

- **Stay on SQLite** while searches feel fast and the database file is a comfortable size. Most projects never need more.
- **Add ClickHouse** when you keep tens of millions of logs or more, when searches over long time ranges get slow, or when the SQLite file grows uncomfortably large.
- **Add Redis** when traffic is bursty (an outage that floods errors, a batch job), when you want ingest to keep answering quickly no matter what processing is doing, or when you run ClickHouse and want larger, fewer inserts.

For reference, on one laptop with 8 concurrent senders posting batches of 500 events, time until all 100,000 events were searchable:

| Mode | Events per second |
|---|---|
| SQLite | ~21,000 |
| SQLite + Redis | ~23,000 |
| ClickHouse | ~22,000 |
| ClickHouse + Redis | ~32,000 |

These numbers measure ingest only. ClickHouse's main advantage, cheap storage and fast scans over very large tables, shows up at volumes far beyond this test.

### Turning on ClickHouse

1. Run ClickHouse (tested with 24.8). With Docker:
   ```bash
   docker run -d --name oculo-clickhouse -p 127.0.0.1:8123:8123 \
     -e CLICKHOUSE_USER=oculo -e CLICKHOUSE_PASSWORD=change-me \
     -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
     -v oculo-clickhouse:/var/lib/clickhouse clickhouse/clickhouse-server:24.8
   ```
2. Start Oculo with:
   ```bash
   CLICKHOUSE_URL=http://localhost:8123 CLICKHOUSE_USER=oculo CLICKHOUSE_PASSWORD=change-me npm run dev
   ```
   Oculo creates the database (`CLICKHOUSE_DATABASE`, default `oculo`) and the `logs` table on first start. The startup line says `logs: clickhouse`, and `GET /healthz` reports `"storage": "clickhouse"`.

If ClickHouse can't be reached at startup, Oculo exits with a clear error instead of silently falling back, so a typo in the URL is obvious.

#### How ClickHouse mode works

- **Same ids, same API.** Oculo allocates log ids itself (a sequence in SQLite), so cursors, live tail, "surrounding logs" and links from issues to their latest event work exactly as before.
- **Counters and rows can't drift apart.** When a batch arrives, issue counters, rollups and patterns are updated in one SQLite transaction, and the new rows are written to an **outbox** table in that same transaction. The outbox is then flushed to ClickHouse. Each flush carries a deduplication token derived from its log ids, so a retry after a crash never inserts rows twice.
- **ClickHouse outages don't lose logs.** If ClickHouse is down, ingest keeps succeeding; rows wait in the outbox and are inserted when it comes back (with backoff). `GET /healthz` shows `outbox_rows` while anything is waiting.
- **Rows become searchable when they reach ClickHouse.** Without the queue, the ingest request waits for the flush (normally milliseconds), so a log is searchable as soon as `POST /api/v1/ingest` returns.
- **Search.** Free text is matched word by word with the same rules as SQLite (word prefixes, phrases, exclusions): an n-gram skip index finds candidate rows and a word-boundary regex checks them. Attribute filters use ClickHouse's JSON functions with the same semantics (nested or dotted keys, numbers, booleans, missing values). Every value is a bound query parameter.
- **Table layout.** `ORDER BY (project_id, ts, id)`, monthly partitions, bloom-filter indexes on trace, issue, user and fingerprint, and ZSTD compression for the search text.
- **Retention and deletes** use ClickHouse lightweight deletes: rows vanish from queries immediately and disk space is reclaimed by background merges.

#### Existing data

Turning ClickHouse on does not move logs already in SQLite: new logs go to ClickHouse and old ones stay in the SQLite file (they are no longer shown). Issues, counts, timelines, rollups and patterns are kept, because they live in SQLite, and new log ids continue above the old ones. An issue whose latest event is still in SQLite shows no latest event until it happens again. There is no built-in tool to copy old rows into ClickHouse; keep the SQLite file until they would have aged out under retention anyway.

Use one ClickHouse database per Oculo installation (`CLICKHOUSE_DATABASE`). Pointing a brand-new SQLite database at a ClickHouse database that already holds another installation's logs would show those rows under the new installation's project ids.

### Turning on the Redis queue

1. Run Redis (tested with 7.4; 6.2 is the minimum for the stream commands used). Turn on persistence (AOF) so queued batches survive a Redis restart:
   ```bash
   docker run -d --name oculo-redis -p 127.0.0.1:6379:6379 -v oculo-redis:/data redis:7.4 redis-server --appendonly yes
   ```
2. Start Oculo with `REDIS_URL=redis://localhost:6379`. The startup line says `queue: redis`.

#### How the queue works

- **Ingest answers right away.** `POST /api/v1/ingest` checks the key, the size limits and the rate limit, appends the batch to the stream (`OCULO_QUEUE_STREAM`, default `oculo:ingest`) and returns `202 {"accepted": n, "dropped": 0, "queued": true}`. Sampling and drop rules run later in the worker, so `dropped` is always 0 in queue mode.
- **A worker drains it in batches.** The server reads up to `OCULO_QUEUE_BATCH` messages (default 100) at a time through a consumer group and applies them in transactions of about 2,000 events, yielding between them so the HTTP side stays responsive (in tests, a 200,000-event backlog drained at ~40,000 events/s with no pause longer than a quarter of a second), then acknowledges them.
- **Each message counts once.** Message ids are recorded in the same transaction as the counters, so a message delivered twice (for example after a crash between commit and acknowledgement) is skipped.
- **Crashed workers are covered.** Messages a worker read but never acknowledged are reclaimed after `OCULO_QUEUE_CLAIM_IDLE_MS` (default 60 s).
- **Bad messages don't block the queue.** A message that fails three times while other messages are being applied is moved to `<stream>:dead` with the error; the rest of its batch is still applied.
- **Outages don't empty the queue into the dead-letter stream.** If every message fails (disk full, database locked), nothing is dead-lettered: messages stay pending, are retried when reclaimed, and back-pressure starts if the problem lasts.
- **Back-pressure instead of unbounded memory.** When more than `OCULO_QUEUE_MAX_BACKLOG` batches (default 50,000) are waiting, ingest answers `503` with `Retry-After: 5`. The SDKs and OpenTelemetry exporters retry automatically.
- **Redis outages don't refuse logs.** If Redis is unreachable while the server is running, ingest processes the batch in the request instead (and logs a warning once a minute).
- **Acknowledged entries are trimmed** from the stream every 30 seconds.
- `GET /healthz` reports `queue_backlog`, the number of batches not yet applied.

#### Things to know about the queue

- Logs become searchable a few milliseconds after the request returns, not before. Scripts that write and then immediately read should poll, or check `queue_backlog` in `/healthz`.
- Oculo is still one server process: the queue gives burst absorption and durability, not multiple writers. The worker runs inside the server.
- One rare duplicate remains possible: if Redis accepts a batch but the connection drops before it answers, the server treats Redis as down, processes the batch directly, and the worker later applies the queued copy too.

### Both together

With `CLICKHOUSE_URL` and `REDIS_URL` set, a batch goes HTTP → Redis stream → worker (one SQLite transaction for counters + outbox) → ClickHouse. This is the highest-throughput setup and the one used by the Docker Compose file in [Deployment](#deployment).

### Settings

| Variable | Default | Purpose |
|---|---|---|
| `CLICKHOUSE_URL` | (off) | ClickHouse HTTP endpoint, e.g. `http://localhost:8123` |
| `CLICKHOUSE_USER` / `CLICKHOUSE_PASSWORD` | `default` / empty | Credentials |
| `CLICKHOUSE_DATABASE` | `oculo` | Database, created if missing |
| `REDIS_URL` | (off) | e.g. `redis://localhost:6379` or `rediss://user:pass@host:6380` |
| `OCULO_QUEUE_STREAM` | `oculo:ingest` | Stream name (dead letters go to `<stream>:dead`) |
| `OCULO_QUEUE_GROUP` | `oculo` | Consumer group |
| `OCULO_QUEUE_BATCH` | `100` | Messages applied per transaction |
| `OCULO_QUEUE_MAX_BACKLOG` | `50000` | Above this many waiting batches, ingest answers 503 |
| `OCULO_QUEUE_CLAIM_IDLE_MS` | `60000` | Reclaim messages a crashed worker left unacknowledged |

### Checking it

```bash
curl -s https://oculo.funonymous.in/healthz
# {"ok":true,"name":"Oculo","storage":"clickhouse","queue":"redis","outbox_rows":0,"queue_backlog":0}
```

In ClickHouse, `SELECT count() FROM oculo.logs` shows the stored rows. In Redis, `XINFO GROUPS oculo:ingest` shows the consumer group's lag and pending messages, and `XLEN oculo:ingest:dead` shows dead-lettered batches.

## Deployment

Oculo is one Node.js process. Pick the setup that matches your volume; you can move between them later.

| Setup | Good for | Needs |
|---|---|---|
| Node + SQLite | Most teams, single server | Node 20+ |
| Docker, SQLite | Same, as a container | Docker |
| Docker Compose: Oculo + ClickHouse + Redis | High volume, long retention | Docker Compose |

### Node + SQLite

```bash
git clone <your fork of this repository> oculo && cd oculo
npm ci
npm run build -w server
PUBLIC_URL=https://logs.example.com OCULO_DB=/var/lib/oculo/oculo.db node server/dist/index.js
```

- On first start an admin key is printed and saved next to the database (`admin-key.txt`, mode 600). Open the dashboard and create the first account; it becomes the instance admin.
- Keep `server/dist`, `server/public` and `docs/` together: the server serves the dashboard from `server/public` and the documentation from `docs/`.
- Run it under a supervisor so it restarts. A minimal systemd unit:
  ```ini
  [Unit]
  Description=Oculo
  After=network.target

  [Service]
  WorkingDirectory=/opt/oculo
  Environment=PUBLIC_URL=https://logs.example.com
  Environment=OCULO_DB=/var/lib/oculo/oculo.db
  ExecStart=/usr/bin/node server/dist/index.js
  Restart=always
  User=oculo

  [Install]
  WantedBy=multi-user.target
  ```
- `npm run dev` (with `tsx watch`) is for development only.

### Docker (SQLite)

```bash
docker build -t oculo .
docker run -d --name oculo -p 4318:4318 -v oculo-data:/data -e PUBLIC_URL=https://oculo.funonymous.in oculo
docker exec oculo cat /data/admin-key.txt      # the bootstrap admin key
```

The image runs as the unprivileged `node` user, stores the database in the `/data` volume and has a health check on `/healthz`.

### Docker Compose: Oculo + ClickHouse + Redis

The repository's `docker-compose.yml` starts all three, with ClickHouse holding log rows and Redis Streams in front of ingest (see [Scaling](#scaling-clickhouse-and-redis)):

```bash
CLICKHOUSE_PASSWORD=pick-a-strong-password docker compose up -d --build
docker compose logs -f oculo        # look for: logs: clickhouse, queue: redis
```

Then open https://oculo.funonymous.in. Data lives in three named volumes (`oculo-data`, `clickhouse-data`, `redis-data`). ClickHouse and Redis are not published on host ports; only Oculo is.

### Behind a reverse proxy (HTTPS)

Terminate TLS in nginx, Caddy or a load balancer and forward to port 4318.

- Set `PUBLIC_URL` to the public `https://` address. It is used in alert links, invite links, the GitHub OAuth redirect and the docs.
- Forward `X-Forwarded-Proto` so session cookies are marked `Secure`.
- Turn off response buffering for `/api/v1/tail` (Server-Sent Events). nginx: `proxy_buffering off;` (Oculo also sends `X-Accel-Buffering: no`).
- Allow request bodies up to `MAX_BODY_BYTES` (10 MB by default), e.g. nginx `client_max_body_size 10m;`.

Caddy example:

```text
logs.example.com {
  reverse_proxy 127.0.0.1:4318 {
    flush_interval -1
  }
}
```

### Backups

- **SQLite** (always used for projects, keys, accounts, issues, alerts; and for logs unless ClickHouse is on): back up while running with `sqlite3 /data/oculo.db ".backup '/backups/oculo-$(date +%F).db'"`. Copying the file alone while the server writes is not safe; copy the `-wal` file too or use `.backup`.
- **ClickHouse**: `BACKUP DATABASE oculo TO Disk('backups', 'oculo-2026-11-15')` (configure a backup disk), or snapshot the volume. Losing ClickHouse loses log rows but not issues, counts or settings.
- **Redis**: holds only batches not yet processed. With `--appendonly yes` they survive a restart; there is nothing to back up long-term.

### Upgrades

1. Back up the SQLite file.
2. Pull the new version, `npm ci`, `npm run build -w server` (or rebuild the image).
3. Restart. Database migrations run automatically on start (`PRAGMA user_version`); the ClickHouse table is created if missing.

With the Redis queue on, a restart doesn't lose ingest: batches wait in the stream and are applied when the server is back. Without it, SDKs buffer and retry while the server restarts.

### Sizing notes

- Retention is per project (default 14 days). Hourly rollups and issue timelines are kept for `ROLLUP_RETENTION_DAYS` (default 400).
- SQLite: plan disk for roughly the raw size of your logs plus the full-text index. Fast disks (SSD) matter more than CPU.
- If ingest is bursty, add Redis; if storage or long-range searches become the problem, add ClickHouse. See [Scaling](#scaling-clickhouse-and-redis).

## Security and privacy

- **Scrubbing at ingest:** passwords, tokens, API keys, cookies, `Authorization` headers, JWTs and Luhn-valid card numbers are always scrubbed on the server. Masking emails is optional.
- **Keys** are stored as SHA-256 hashes and shown only once.
- **Accounts:** passwords use scrypt. Login is rate-limited and doesn't reveal whether an email exists.
- **Sessions:** HttpOnly, SameSite=Lax cookies (Secure behind HTTPS). They are never accepted on ingest. Cookie-authenticated writes must come from Oculo's own origin.
- **GitHub sign-in** verifies the OAuth `state` and links accounts only through a **verified** GitHub email.
- **Live tail** uses one-time 60-second tickets, so long-lived keys never appear in URLs.
- **Agent keys:** read keys can never write. Project keys can't create alerts, because alerts make outbound requests.
- **Queries** are fully parameterised, and the dashboard escapes all log content (HTML in logs is shown as text).
- **Explain** is off unless configured, and asks before sending issue data to Anthropic.

## Development and testing

### Layout

| Path | What |
|---|---|
| `server/` | The server: HTTP API, ingest pipeline, storage, alerts, dashboard (`server/public`, no build step) |
| `sdk-js/` | JavaScript/TypeScript SDK (ESM + CommonJS builds) |
| `sdk-python/` | Python SDK (standard library only) |
| `mcp/` | MCP server for LLM clients (stdio) |
| `cli/` | The `oculo` command |
| `docs/` | This documentation (served at `/docs` and `/docs.md`) |
| `scripts/` | Seed data, test suites, benchmarks |

### Architecture

```text
apps / SDKs / OTel ──► POST /api/v1/ingest ──► [Redis stream]* ──► worker
                                                                     │  one SQLite transaction:
                                                                     │  normalize, scrub, source maps,
                                                                     │  fingerprint → issues, rollups,
                                                                     │  patterns, log rows (SQLite)
                                                                     │  or outbox (ClickHouse)*
                                                                     ▼
                                                     SQLite (metadata, issues, …)  ──outbox──►  ClickHouse*
dashboard / CLI / MCP / agents ──► REST API ──► query compiler ──► SQLite FTS5 or ClickHouse SQL
                                                                                   * optional
```

- **Server code** (`server/src`):
  - `ingest.ts`: the pipeline
  - `normalize.ts`, `scrub.ts`, `fingerprint.ts`, `patterns.ts`, `sourcemaps.ts`: its steps
  - `logstore.ts`: SQLite and ClickHouse log stores
  - `outbox.ts`: ClickHouse outbox
  - `queue.ts`: Redis Streams
  - `query/`: the query language parser and compiler, with one compiler for both SQL dialects
  - `api/`: routes
  - `alerts.ts`, `users.ts`, `migrations.ts`
- **Queries** are written once with `?` placeholders. For ClickHouse, numbers are inlined and strings become typed parameters (`{p1:String}`), so user input never reaches SQL text.

### Running locally

```bash
npm install
npm run dev            # server on https://oculo.funonymous.in with auto-reload
npm run seed           # (another terminal) ~24h of demo logs
```

### Tests

| Command | What it checks |
|---|---|
| `npm run e2e` | Boots a server on a temp database and runs ~190 API checks: keys and roles, accounts and OAuth (stubbed), query safety, grouping, patterns, metrics, source maps, OTLP (real OpenTelemetry exporter), every alert channel (stubbed), heartbeats, digest, docs |
| `npm run e2e:all` | The same suite in every storage mode: SQLite, SQLite+Redis, ClickHouse, ClickHouse+Redis (the last three when the env vars below are set) |
| `npm run test:parity` | Sends identical data to a SQLite server and a ClickHouse server and checks ~100 queries, metrics, facets, histograms, patterns, issue pages and exports return identical results |
| `npm run test:resilience` | Failure modes against real ClickHouse and Redis: ClickHouse down, crash between insert and acknowledgement, duplicate delivery, crashed consumer, poison message vs. outage, full queue, Redis down, a 200,000-event backlog (event loop must stay responsive), switching an existing SQLite database to ClickHouse, bad startup config |
| `npm run test:sdk` | JS and Python SDK lifecycle: exit flush, uncaught errors, outage recovery, batching |
| `npm run build -w mcp && OCULO_KEY=<read key> npm run mcp:smoke` | The MCP server over stdio |

ClickHouse and Redis tests need running servers:

```bash
docker run -d --name oculo-clickhouse -p 127.0.0.1:8123:8123 -e CLICKHOUSE_USER=oculo -e CLICKHOUSE_PASSWORD=oculo-dev \
  -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 clickhouse/clickhouse-server:24.8
docker run -d --name oculo-redis -p 127.0.0.1:6379:6379 redis:7.4

export E2E_CLICKHOUSE_URL=http://localhost:8123 E2E_CLICKHOUSE_USER=oculo E2E_CLICKHOUSE_PASSWORD=oculo-dev
export E2E_REDIS_URL=redis://localhost:6379
npm run e2e:all && npm run test:parity && npm run test:resilience
```

Each run uses its own ClickHouse database and Redis stream and removes them afterwards.

External services (GitHub, SMTP, Telegram, PagerDuty, Anthropic) are always replaced by local stubs in tests; no test talks to a real third-party service.

### Docs

Each page in `docs/` is plain Markdown. The server stitches them into one page in the order listed in `server/src/docs.ts` (`DOC_ORDER`), so a new page must be added there. Links between pages (`[Metrics](#metrics)`) become in-page anchors, and `https://oculo.funonymous.in` is replaced with the server's own address.

## Troubleshooting

| Symptom | Check |
|---|---|
| `401 Invalid or revoked API key` | The key is wrong or revoked. Sign in, then create a new one in Settings → API keys. |
| `403 Ingest keys can only send logs` | Use a read key (or an account) to query. Ingest keys are write-only by design. |
| `400 Pass ?project=<slug>` | Admin keys and users in several projects must say which project. |
| Nothing shows up | Is the timestamp in the selected time range? Try `30d`. Do sampling or drop rules match? Check the ingest response: `dropped`. |
| Errors grouped too much or too little | Send a custom `fingerprint`. |
| Stack traces still minified | The `release` on the event must equal the release the map was uploaded for, and the frame's file name must match the bundle name. |
| Email alerts fail | Set the `SMTP_*` variables, then use **Send test** and check **History** for the error. |
| GitHub button missing | Set `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET`, and register the callback URL. |
| ✨ Explain missing | Set `ANTHROPIC_API_KEY` on the server and restart. |
| `Can't use ClickHouse at …` on startup | Check `CLICKHOUSE_URL`, user and password; `curl http://host:8123/ping` should answer `Ok.` |
| `Can't use Redis at …` on startup | Check `REDIS_URL`; `redis-cli -u $REDIS_URL ping` should answer `PONG`. |
| `503 Ingest queue is full` | The worker is behind by more than `OCULO_QUEUE_MAX_BACKLOG` batches. Senders retry automatically; check the server log and SQLite disk speed. |
| Just-sent logs not searchable yet (queue or ClickHouse mode) | Ingest is asynchronous there. `GET /healthz` shows `queue_backlog` and `outbox_rows`; both drop to 0 within milliseconds normally. If `outbox_rows` stays up, ClickHouse is unreachable (rows are kept and retried). |
| Messages in `oculo:ingest:dead` | A batch failed on its own (the error is stored with it). Fix the cause, then re-send the events. |