Notion ingest — v3 (authors, comments, databases, backfill)

​

What it enables

• Author lineage on every doc. owner is the page’s real Notion creator (mapped through ingest/connectors/notion_users.yaml), not a hardcoded @notion-bot. notion-author:<handle> and notion-last-editor:<handle> land as tags so retrieval can filter by who wrote / last touched a page.
• Threaded comments in the body. Each page picks up a ## Comments appendix listing every unresolved discussion (### @handle — TIME (thread: id) + nested replies). One Notion API call per page.
• Database rows are first-class documents. A row in a Notion database emits as its own .md with the row’s properties rendered as a table, derived_from linking back to the database index, and scope = sources/notion/<ws>/<db-slug>. Operational content (OKRs, tickets, runbooks) becomes retrievable per-row.
• Hierarchical scope. A page’s scope segment reflects its top-level Notion parent (sources/notion/<ws>/<top-parent-slug>), and the rel_path mirrors that nesting. Top-level pages stay at sources/notion/<ws>.
• Wikilink graph. Notion @page mentions and notion.so href links resolve to [[kh-id|display-text]] wikilinks; the resolved kh-ids populate the doc’s related frontmatter so graph traversal works at query time.
• One-shot bulk backfill. A new notion_backfill.py enumerates the whole workspace in one pass, with the mention-rewriting step backed by a pre-built id→kh-id map (zero dead links for pages that exist in the corpus).

​

What’s changed (surface)

​

Frontmatter mapping

​

Impact scope

• Re-index of every Notion page on next cron tick. ingest_version bumped to 3 → idempotent rewrite triggered for every existing doc; Qdrant chunks regenerate on the same pod.
• Slack and GitHub are untouched. Their ingest_version stays at 2.
• Path layout changes for nested pages. A page that was at sources/notion/<ws>/page.md may move to sources/notion/<ws>/<top>/page.md on the first v3 tick. The kh-id is stable (derived from source_id), so retrieval keeps working; the old file path becomes a stale orphan and should be git rm’d after the first successful run.
• Database rows that previously emitted as plain “untitled” pages now emit as proper row docs at sources/notion/<ws>/<db-slug>/<row-slug>.md with derived_from = [<db-kh-id>]. Old per-row files emitted under the old layout become orphans (same cleanup story).
• Cron API call count rises. Each page render now calls list_comments once and walks parents via get_page (cached). For a workspace of ~500 pages with ~3-level nesting, expect ~1500 extra calls per full sweep, well under the 3 req/s rate limit but visibly slower per cron tick.
• Backwards-compatible default for unconfigured workspaces. With no notion_users.yaml and no env var, the connector still runs; owners fall back to NOTION_OWNER (@notion-bot), comments still render with fallback handles.

​

Bulk backfill workflow

# from repo root
export NOTION_TOKEN="ntn_..."
/Users/azat/labs/amos/worker/.venv/bin/python -m ingest.connectors.notion_backfill \
  --workspace-slug gdi-labs \
  --hub-dir knowledge-hub \
  --user-map ingest/connectors/notion_users.yaml \
  --dry-run

​

Tests

/Users/azat/labs/amos/worker/.venv/bin/python -m pytest tests/test_connector_notion.py -v

• block rendering (paragraph, heading, lists, code, tables, divider) — unchanged from v1.
• author resolution (mapped UUID → handle; unmapped → fallback slug; missing created_by → connector default).
• hierarchical scope (top-level page → flat; nested page → <top> suffix; rows → <db-slug> scope + derived_from).
• comment rendering (3-message thread groups, block-anchored vs page-anchored heading shape).
• mention rewriting (Notion URL resolves to [[kh-id|text]], populates related; unresolvable URL stays plain).
• backfill discovery (pages + databases + rows enumerated; meta map covers all three; unmapped users captured).

• notion_backfill.py --dry-run against the live workspace.
• Run for real; pick two pages that link to each other in Notion; verify both related lists contain the other’s kh-id after ingest.
• kb_search (MCP) for a Notion page known by title; verify the result payload carries the right owner and updated_at.

​

Known limitations

• Resolved comments are unreachable via the Notion API (reference). Only open discussions appear in the ## Comments appendix. If preservation of resolved threads matters, manually re-open them in Notion before ingest.
• Notion-hosted image URLs still expire after ~1 hour. v3 keeps the v1 behavior of inlining the URL as-is — images render in the markdown but 404 outside the freshness window. Rehost-to-S3 is tracked as a follow-up.
• Formulas, rollups, synced blocks export as computed values only. Source logic is not retrievable from the API.
• Workspace-level access requires a manual share step. A Notion admin must share the integration at the workspace root (or each top-level page) before the backfill enumerates anything. The integration token alone is not enough.
• Mention rewriting on the cron is disabled by default. The cron leaves notion.so URLs as plain links (rewriting to dead wikilinks for un-ingested pages would be worse than not rewriting). The backfill, which sees the full corpus, is where rewriting happens.