Forensics

Scope: Where each component writes logs, what it records, what it doesn’t, defaults, and the forensic/detection implications for the lab. Basis: Verified against the running lab (Ollama 0.32.1 on the Windows host + remote 3090; Agent Zero container, Kali-based image). Paths/behaviors confirmed live where noted.


1. Ollama

1.1 Log locations

OS / modeLocation
Windows (tray app / desktop)%LOCALAPPDATA%\Ollama\ → server.logapp.log, rotated server-1..N.log / app-1..N.logdb.sqlite*ollama.pid
Windows (ollama serve in a terminal)stdout/stderr of that console — not server.log. Manual serve bypasses the file logger.
Linux (systemd service)journalctl -u ollama (journald)
Linux (manual / ~/.ollama)stdout, or ~/.ollama/logs/server.log depending on launch
macOS~/.ollama/logs/server.log

Lab nuance: early in setup the local server.log was 0 bytes because Ollama was started with a bare ollama serve (logs went to the hidden console). When the tray app manages the server, it writes server.log. If you need reliable local Ollama logs, run it under the tray app or redirect the console yourself.

1.2 What server.log records (the inference server)

  • Startup config dump — the entire environment/config on one line: OLLAMA_HOSTOLLAMA_CONTEXT_LENGTHOLLAMA_KEEP_ALIVEOLLAMA_MODELS (model path), OLLAMA_DEBUG level, origins allow-list, etc. (Config disclosure: this line reveals the full server posture.)
  • GPU / VRAM discovery — detected devices, compute capability, VRAM, and the vram-based default context decision.
  • Model load/unload — which model, blob counts, load timing.
  • HTTP access logs (GIN framework), ON by default — one line per request: [GIN] <ts> | <status> | <latency> | <client-IP> | <METHOD> "<path>" e.g. [GIN] ... | 200 | 0s | 127.0.0.1 | GET "/api/version". Captures method, path, status, client IP, latency for every call — including /api/generate/api/chat/api/pull/api/embed.

1.3 What Ollama does NOT log by default

  • Prompt or response content. Gated behind OLLAMA_DEBUG_LOG_REQUESTS (default false). At default INFO level you get metadata only — which model, which endpoint, when, from which IP, how long — never the text of the prompt or the generated tokens.
  • No auth/identity logging — Ollama has no authentication, so there is no auth event to log (any client that reaches the port is served; only the source IP in the GIN line identifies it).

1.4 Controls & defaults

SettingDefaultEffect
OLLAMA_DEBUGINFO1/DEBUG → verbose (scheduler, memory, layer offload detail)
OLLAMA_DEBUG_LOG_REQUESTSfalsetrue → logs request bodies (prompts) — high sensitivity
GIN HTTP access logonNot separately toggleable; always emits the per-request line
Log rotationonserver.log → server-1..N.log on restart; keeps a handful, no size cap enforced
OLLAMA_MODELS~/.ollama/modelsAlso printed in the startup dump

1.5 Desktop/tray app logs (separate from inference)

  • app.log — the Ollama desktop GUI’s own local server (/api/v1/settings/api/v1/chats/api/v1/inference-compute). Its own GIN-style access lines.
  • db.sqlite (+ -wal/-shm) — the desktop app’s state store: settings and any chats created in the desktop UI. Note: chats from the desktop app live here; chats via the API do not.

1.6 The remote 3090 node

That box runs its own Ollama with its own server.log / journald. Its GIN access log records every inference request from the Agent Zero container’s source IP — i.e. a network-side record of when the agent used the model, independent of anything on the container. Correlate its timestamps with the container’s chat.json to reconstruct activity. (OS of that box unconfirmed — check journalctl -u ollama first, else ~/.ollama/logs/.)


2. Agent Zero

2.1 What it logs and where

SourceLocationPersistenceContent
Chat/context log (helpers/log.py)usr/chats/<id>/chat.json (root, 0600)Bind mount — survives restart/rebuildFull record: user messages, agent thinking + response, tool calls with arguments, tool results, code execution + output, errors
stdout/stderr (all services + print_style)docker logs agent-zeroNo rotation (maxbytes=0), unbounded; survives restartlost on down/rm/recreateThe agent’s live reasoning, tool prints, service output — noisy, ANSI-coloured, unstructured
Time-travel gitusr/.time_travel/.../repo.gitPersistentGit history of files the agent created/modified in its workspace
Scheduler / memoryusr/scheduler/tasks.jsonusr/memoryPersistentRecurring tasks; accumulated state

chat.json is the high-value artifact — the operator’s tasking verbatim plus every tool call and code execution. It is what makes a seized instance intelligible.

2.2 What Agent Zero does NOT log by default

  • HTTP access logs — OFF (uvicorn_access_logs_enabled: false). No request-level record. An unauthenticated endpoint hit (e.g. a planted handler) leaves no HTTP trace.
  • No auth-event logging. A failed login does sleep(1) and re-renders — nothing written. Brute-force attempts are invisible in logs (and unthrottled — no lockout).
  • No dedicated security/audit log, no syslog/SIEM integration.

2.3 Anti-forensics exposure

  • chat.json is writable/deletable by container-root, and the container runs as root. Anyone who lands in the container (e.g. via a dropped endpoint) can edit or wipe the chat history to cover tracks. Docker logs are likewise exposed if the host is reached. On-box logs cannot be trusted after compromise.

2.4 Relevant defaults

SettingDefaultNote
uvicorn_access_logs_enabledfalseEnable for request-level visibility
Auth event loggingnoneNot configurable in-app; add at a proxy
stdout log rotationnone (maxbytes=0)Unbounded growth risk
chat.json perms0600 rootBut container is root → not a real barrier

3. The key contrast (metadata vs content)

Logs request metadata (who/when/which/IP)Logs content (prompt/response/tool)
OllamaYes — GIN access log, on by defaultNo (unless OLLAMA_DEBUG_LOG_REQUESTS=true)
Agent ZeroNo — access logs off by defaultYes — chat.json

They are complementary. Ollama’s server.log tells you when inference happened and from which client; Agent Zero’s chat.json tells you what was asked and done. Correlating the two by timestamp reconstructs an operation end-to-end — and because the Ollama node is a separate box, its record survives tampering on the container.


4. Forensic & detection implications

Collecting from a seized instance (priority order):

  1. usr/chats/*/chat.json — verbatim tasking, tool calls, code exec (assume tamperable)
  2. Remote Ollama server.log / journald — independent, off-box timeline of inference (harder to tamper)
  3. usr/.time_travel/.../repo.git — file-change history
  4. docker logs — live reasoning, if the container is still up
  5. Ollama startup config line — full server posture disclosure

Hardening the lab’s visibility (the gaps that matter):

  • Enable uvicorn_access_logs_enabled for request-level records on Agent Zero.
  • Ship logs off-box in real time — chat.json and Docker logs are attacker-writable; forward to a SIEM/syslog the container can’t reach. The remote Ollama node already provides a partial off-box record; centralise it.
  • Add file-integrity monitoring on api/plugins/usr/scheduler/ — because a webshell/handler drop generates no HTTP or auth log, FIM is the highest-value signal available.
  • Consider OLLAMA_DEBUG_LOG_REQUESTS=true on a contained analysis node only (it logs prompt content — sensitive; do not enable in shared/exposed deployments).

Bottom line: default logging across the stack is oriented to operation, not security. Content lives (attacker-writable) on the container; request metadata lives on the Ollama node. Neither records authentication or is tamper-resistant by default — so off-box shipping + FIM are the two additions that convert this from “operational logging” into a defensible audit trail.


Generated for the agent-zero-lab CTI/hardening effort. Paths verified against Ollama 0.32.1 and the running Agent Zero container; re-verify per-OS locations on the remote inference node.