Brain

What the Brain is

The Brain is Goat's durable knowledge store: a per-workspace graph of markdown documents with typed links, evidence, and a timeline. Every workspace has at least one brain; users can create more and restrict who sees them. Chat, background tasks, and the ingestion agents all read from and write into it.

Three layers make it up:

• packages/goat-brain — the format and toolkit: document parsing, validation, the graph model, retrieval, and a CLI. It works against a plain directory of files and knows nothing about the database.
• packages/db/src/goat-schema.ts + goat-brain-files.ts + goat-brain-ingest.ts — the durable store: goat.* tables that hold brains, documents, derived edges, timeline entries, versions, and the ingestion queue.
• apps/goat + apps/runner — the surfaces: the Brain UI, the chat tools, the durable ingestion agents, and task integration.

The database is the source of truth. Whenever an agent needs to work on the brain, the files are materialized to a directory, the agent works through the CLI, and changed files are synced back with conflict detection.

Document model

A brain document is a markdown file with YAML frontmatter, stored at folder/id.md:

• id — a lowercase slug, unique within the brain (called brainId in document code, distinct from the brain_ref that identifies the brain itself).
• kind — page or evidence. This is the only structural rule in the system: evidence documents must live under the evidence/ zone, pages must live outside it. A database constraint enforces it.
• type — what the record represents, one of eight: person, company, project, meeting, concept, source, analysis, note. External artifacts of any format (articles, videos, email threads, repos) are source; synthesized prose is analysis. Types are pure metadata; they do not dictate where a document lives. Retired v1 names (media, email → source; writing → analysis) still parse via legacy aliases.
• folder — free-form navigation, up to six path segments. There is an opinionated default per type (person → people, note → inbox, and so on in packages/goat-brain/src/schemas.ts), but users and agents can organize folders however they like.
• status — the curation signal: draft (captured, not yet curated), active (curated synthesis; must cite at least one evidence record in its compiled truth — validation enforces it on write), archived (deliberately retired), or merged (tombstone pointing at mergedInto). Ingestion agents promote pages to active once their compiled truth cites evidence. Retrieval (query) excludes merged and archived documents unless --include-merged / --include-archived is passed; the UI shows archived documents in a collapsed sidebar section and hides merged ones entirely.
• aliases, relations, sources — additional frontmatter used for lookup and graph derivation.

The body has two zones: a compiled truth section that writers rewrite in place (never append-only), and a timeline of dated evidence entries, each with a deterministic ev- id, a timestamp, and a source reference. Timeline entries are also denormalized into their own table so the UI can query them efficiently.

Ideas and thoughts do not introduce another kind. kind remains the page/evidence boundary. A saved user-authored idea starts as a draft note in inbox/, then curation either merges it into an existing page, promotes it to a concept, updates or creates a project, files a rationale under decisions/ with the best existing type, moves a durable raw reflection to thoughts/ as a note, or leaves it in inbox/ when it is still uncurated.

Links and the graph

Documents reference each other with typed inline links:

• [[page:some-id|Label]] — link to another page.
• [[evidence:ev-some-id|Label]] — citation of an evidence document.
• [[source:provider:id|Label]] — pointer to an external system, e.g. [[source:jamie:meeting:123|Jamie notes]].

Edges are derived, not authored: deriveGoatBrainEdges in packages/goat-brain/src/edges.ts extracts them from frontmatter relations and inline links, and they are persisted to goat.brain_edges on every write. Edge types are the relation type, wiki_link for page links, and cites for evidence citations. The UI separates page links from evidence links, while retrieval returns link metadata for both and keeps hop expansion focused on pages.

The pointer/copy contract

Every writer into the brain follows one rule, defined once as GOAT_BRAIN_POINTER_COPY_RULE in packages/goat-brain/src/pointer-copy.ts and embedded into each writer's system prompt. The prose version lives in apps/goat/docs/brain/pointer-copy-contract.md.

The short form: prefer pointers over copies. Every claim carries a [[source:provider:id]] pointer. Content is snapshotted into evidence/ only when the source is ephemeral or unfetchable:

• Meeting transcripts, emails, and chat captures → snapshot into evidence/ plus a pointer, because the brain cannot re-fetch them.
• Linear and GitHub items → pointer plus a one-line state summary, never the body, because the tracker is the canonical live home and bodies go stale immediately.
• Everything else → pointer only by default.

Source refs follow a provider:id grammar (lowercase provider slug, provider-native id, max 256 chars). A resolver seam (packages/goat-brain/src/source-resolvers.ts) exists so integrations can later hydrate pointers with live URLs and state at read time; it is interface-only today.

Data model

All tables live in the goat Postgres schema, defined in packages/db/src/goat-schema.ts:

• brains — one row per brain: workspace, name, slug, visibility (workspace or restricted). brain_members holds the allow-list for restricted brains.
• brain_folders — the folder tree per brain (system or custom source).
• brain_documents — the documents: content, body, timeline JSON, kind, entity type, status, relations, sources, aliases, content hash. Format can be markdown, pdf, or docx.
• brain_timeline_entries — denormalized per-entry rows for efficient timeline queries.
• brain_edges — the derived graph.
• brain_document_versions — an audit trail of overwrites and deletes.
• brain_source_items and brain_ingest_jobs — the ingestion pipeline (below).

Every content table carries a brain_ref foreign key to brains.id. That is the isolation boundary: every read and write is scoped to exactly one brain, and access is checked with requireGoatBrainAccess / brainAccessCondition in packages/db/src/goat-workspaces.ts.

Write paths

There are five ways content gets into a brain.

1. The UI editor

The Brain page (/brain/...) renders GoatBrainView with a Tiptap markdown editor that decorates inline links and supports cmd-click navigation. Edits autosave after 1.2s of idle through server actions in apps/goat/lib/brain-actions.ts (updateGoatBrainDocumentAction, move, rename, delete). Updates carry an expectedContentHash so concurrent edits are detected rather than silently overwritten. The UI deliberately has no create affordance: new documents and folders enter through the brain CLI and ingestion agents only.

2. Chat captures (save_to_brain)

When the user asks chat to keep something — "save this reference", "here's an idea" — the chat agent calls save_to_brain. The flow is deliberately capture-first, in two phases:

1. Capture now. captureToGoatBrainInbox (apps/goat/lib/brain-capture.ts) immediately writes the content as a draft page in inbox/ (type note, status draft, sourced to goat-chat:<messageId>) in the same request. Nothing the user saves can be lost, even if everything downstream fails, and the draft appears in the Brain UI instantly.
2. Curate later. The same call normalizes a goat-chat/capture source item, enqueues a brain_agent_ingest job pinned to the active brain, and wakes the runner. The capture-curation agent then reads the draft, checks the brain for an existing home, and either folds the content into an existing page (retiring the draft via merge) or promotes the draft in place: append-evidence to snapshot the raw capture, rewrite compiled truth citing it, set the proper title and type, move it to the right folder, and mark it active.

Duplicate save_to_brain calls within one chat turn dedupe to a single draft; identical webhook re-deliveries dedupe at the source-item layer via content hash.

For ideas and thoughts, the curation agent applies an authorship test before topical filing. A user or team-authored idea with future value belongs in Brain even when rough; an agent behavior preference belongs in Memory; conversation-only scratch stays in chat. Brain-worthy ideas use the existing type and folder model: reusable abstractions become concept pages, concrete initiatives update project pages, choices or rationales go to the natural subject or decisions/, durable raw reflections move to thoughts/ as note pages, and unresolved material remains a draft note in inbox/.

3. The goat_brain chat tool

Foreground chat (apps/goat/lib/chat-agent.ts) also exposes the brain CLI directly, for recall and precise edits the user dictates: get, list, query, timeline, rewrite, set, append-timeline, append-evidence, alias, link, merge, move, delete (dry-run only), folder, doctor. Ordinary saves route through save_to_brain instead; direct create is reserved for records the user spells out explicitly. Mutations are serialized per brain so concurrent chat turns don't clobber each other.

4. The durable ingestion agents

This is the main automated path. Both source kinds run on the same queue and worker; today there are two agent profiles.

Jamie meetings flow end to end:

Jamie webhook (meeting.completed)
  POST /api/webhooks/jamie
    resolve integration from saved x-jamie-api-key hash
    normalize payload -> upsert goat.brain_source_items (idempotent on content hash)
    enqueue goat.brain_ingest_jobs (kind: brain_agent_ingest)
    wake the runner via /internal/goat/brain-ingest/wake

Runner (apps/runner/src/goat-brain-ingest-worker.ts)
  claim job with FOR UPDATE SKIP LOCKED + 5-minute lease, heartbeat every 5s
  materialize the target brain's files to a temp directory
  pre-write a deterministic evidence snapshot (full transcript, evidence/ zone)
  run the ingestion agent (apps/runner/src/goat-brain-agent-ingest.ts):
    model loop with the goat-brain CLI as its only tool
    max 32 steps, 10-minute timeout
    brain-first discipline: query/get before create, rewrite compiled truth,
    link every entity mention, follow the pointer/copy rule, or return SKIP
  sync changed files back to the DB (retry the job if the brain changed underneath)
  record job result, mark source item succeeded/failed

For a meeting, the agent produces a meeting page, person pages for attendees, company pages where relevant, and backlinks between all of them, each citing the pre-written evidence snapshot.

Chat captures run the same loop with a different mission (curate the existing inbox draft, described above), an expanded command surface (merge allowed, no deterministic pre-write), and the instruction that user-saved content is almost never skippable.

Both profiles share runBrainAgentIngestSession: resolve brain → materialize → agent loop → sync back with conflict detection. Failures retry with exponential backoff up to five attempts; expired leases are reclaimed by the next poll. A legacy deterministic job kind (brain_source_item_ingest) remains registered so previously queued jobs drain.

New source providers follow the same template: normalize into brain_source_items (see packages/goat-brain/src/source-items.ts), enqueue a brain_agent_ingest job, and register a handler descriptor in the worker.

5. Task research reports

Background tasks planned with resultMode: "brain_markdown_report" save their final output as an analysis document in the research/ folder instead of a plain text result (createGoatBrainMarkdownReportForTask in apps/runner/src/goat-brain.ts), with a goat-task:<taskId> source ref and a timeline entry. Tasks can also materialize the whole brain into their sandbox and sync edits back the same way the ingestion agents do.

Read paths

• Chat and agents query through the CLI's query command, which runs the hybrid ranker in packages/goat-brain/src/retrieval/: BM25 lexical search, optional embedding + rerank via Vercel AI Gateway, reciprocal-rank fusion, then graph expansion along edges with score decay and a recency blend. Hits report whether they matched by text, graph, or both, and include the hop path.
• The UI does not search this way; it subscribes to Electric-backed live collections of documents, timeline entries, and edges, scoped to the active brain, and re-renders as rows change. Server-rendered props are only the initial snapshot.
• Known limitation: every agent query currently materializes the brain to a temp directory and discards it afterwards, so the embedding cache never persists between queries.

apps/goat/docs/brain/retrieval-planes.md is the design for where this is heading: a deterministic read module over brain_documents rows (no materialization, durable brain_embeddings cache, exposed in-process, over HTTP, and over a per-brain MCP connector) plus a read-only "librarian" agent for synthesized answers with typed citations. That design is not implemented yet; the MCP connector lives on a separate branch.

Multiple brains and access

Brains belong to workspaces. visibility: "workspace" brains are readable by every workspace member; restricted brains only by rows in brain_members. The sidebar's brain switcher (GoatBrainSwitcher) lists accessible brains, lets admins manage access, and switching brains re-creates the live collections scoped to the new brain_ref. When a machine path needs a brain and none was specified (for example an ingest job with a null brain_ref), it resolves the user's default brain via getDefaultGoatBrainForUser.

Validation and health

validateGoatBrainDocument enforces the format on every write: id/folder shape, kind/zone alignment, status values, timestamps, link syntax, and that active pages cite evidence. goat-brain doctor (also exposed to agents) runs checkGoatBrainHealth across the whole brain and reports duplicate ids, parse errors, broken relations and links, broken citations, alias collisions, orphan nodes, weak provenance (active pages without evidence), and nonstandard source refs.

Where to change things

• Document format, links, validation, retrieval, CLI: packages/goat-brain/src/ (after CLI changes, regenerate the bundle with bun run bundle).
• Tables, access control, file sync, ingest queue: packages/db/src/goat-schema.ts, goat-brain-files.ts, goat-brain-ingest.ts, goat-workspaces.ts (schema changes need a Drizzle migration).
• Ingestion worker and agent prompts: apps/runner/src/goat-brain-ingest-worker.ts, apps/runner/src/goat-brain-agent-ingest.ts.
• Chat capture flow: apps/goat/lib/brain-capture.ts and the save_to_brain wiring in apps/goat/lib/chat-agent.ts and apps/goat/app/api/chat/route.ts.
• New source providers: normalize into brain_source_items, enqueue a brain_agent_ingest job, and add a handler — the Jamie webhook route and the chat capture path are the templates.
• Brain UI: apps/goat/components/GoatBrainView.tsx, MarkdownGoatBrainEditor.tsx, GoatBrainSwitcher.tsx, and server actions in apps/goat/lib/brain-actions.ts.
• Design references: apps/goat/docs/brain/pointer-copy-contract.md and apps/goat/docs/brain/retrieval-planes.md.