> For the complete documentation index, see [llms.txt](https://atomic-blend.gitbook.io/mongo2pg/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://atomic-blend.gitbook.io/mongo2pg/contributing/architecture.md).

# Architecture

For contributors. The [root overview](/mongo2pg/readme.md) covers the data-flow diagram; this page covers package boundaries and the bookkeeping schema behind them.

## Components

```mermaid
flowchart LR
    CFG["Config loader<br/>internal/config"]

    subgraph Read["reading MongoDB"]
        SR["Snapshot reader<br/>internal/mongosrc"]
        CS["Stream watcher<br/>internal/mongosrc"]
    end

    MAP["Mapper<br/>internal/mapper, internal/coerce<br/>pure function of (config, document)"]
    RES["Identity resolver<br/>internal/pk, internal/resolver,<br/>internal/bsonval"]

    subgraph Write["writing PostgreSQL"]
        WR["PG writer<br/>internal/pgstore"]
        QW["Quarantine writer<br/>internal/pgstore"]
    end

    SCH["Schema planner<br/>internal/schema"]
    VER["Verifier<br/>internal/verify"]
    METS["Metrics server<br/>internal/metrics"]

    ORCH["Orchestration + phase machine<br/>internal/pipeline, internal/cli"]

    CFG --> ORCH
    ORCH --> SCH
    ORCH --> SR
    ORCH --> CS
    SR --> MAP
    CS --> MAP
    MAP --> RES
    RES --> WR
    RES -.->|unmappable| QW
    ORCH --> VER
    ORCH --> METS
```

| Component         | Responsibility                                                                                | Package                                                |
| ----------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| Config loader     | Resolve config from arg / env / file / ConfigMap; validate; watch for changes                 | `internal/config`                                      |
| Schema planner    | Diff mapping against the live PostgreSQL catalog; emit DDL; classify additive vs. destructive | `internal/schema`                                      |
| Snapshot reader   | Paginated `_id`-ordered scan of a collection, resumable                                       | `internal/mongosrc`                                    |
| Stream watcher    | Change-stream tail with a durable resume token                                                | `internal/mongosrc`                                    |
| Mapper            | Document → row set: field mapping, coercion, nesting, transforms                              | `internal/mapper`, `internal/coerce`                   |
| Identity resolver | `_id` → PK; every `ref` → parent PK                                                           | `internal/pk`, `internal/resolver`, `internal/bsonval` |
| PG writer         | Batch, upsert, transact; write the checkpoint in the same transaction                         | `internal/pgstore`                                     |
| Quarantine writer | Park unmappable documents with the reason                                                     | `internal/pgstore`                                     |
| Verifier          | Count parity, sampled deep compare, range checksums                                           | `internal/verify`                                      |
| Metrics server    | Prometheus `/metrics`, `/healthz`, `/readyz`                                                  | `internal/metrics`                                     |
| Orchestration     | Wires all of the above into `run`/`snapshot`/`status`/etc, and the phase machine              | `internal/pipeline`, `internal/cli`                    |

The **Mapper** and **Identity resolver** are pure functions of `(config, document)` — no I/O in the derived path — which is what makes them exhaustively unit-testable and safe to run against a document from either the snapshot reader or the change stream with identical results. See [Type coercion](/mongo2pg/concepts/concepts/coercion.md) and [Primary keys and identity](/mongo2pg/concepts/concepts/primary-keys.md).

## The `_migration` schema

Every persistent state mongo2pg keeps lives in the **target PostgreSQL database**, under a `_migration` schema — there is no external state store and no local disk state, so state and data can never diverge independently.

| Table              | Holds                                                                                                                                                          |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collection_state` | Per-collection phase, PK strategy, effective PK namespace, mapping fingerprint, snapshot cursor                                                                |
| `checkpoint`       | The change-stream resume token, committed in the same transaction as the rows it accounts for — see [The phase machine](/mongo2pg/concepts/concepts/phases.md) |
| `id_map`           | Non-derivable primary keys (`uuid_v4`, `bigint`, field-sourced `uuid_v7`) — see [Primary keys and identity](/mongo2pg/concepts/concepts/primary-keys.md)       |
| `pending_fk`       | Deferred foreign keys awaiting a parent that has not landed yet — see [Foreign-key relinking](/mongo2pg/concepts/concepts/foreign-keys.md)                     |
| `quarantine`       | Parked documents, with reason and raw BSON — see [Quarantine](/mongo2pg/concepts/concepts/quarantine.md)                                                       |
| `verify_runs`      | An audit trail of every verification result — see [Verification and checks](/mongo2pg/operating-a-migration/operations/verification-and-checks.md)             |

Every target table additionally carries two tool-owned bookkeeping columns: **`_src_ts`**, the MongoDB cluster time of the change that produced this row version (the write-guard ordering key — see [The phase machine](/mongo2pg/concepts/concepts/phases.md)), and **`legacy_mongo_id`**, the original `_id` when `keepLegacyId: true`, kept for audit and droppable after cutover.

## Schema management — two independent tern migrators

All DDL — mongo2pg's own bookkeeping tables and the generated target tables — is applied through [`jackc/tern/v2`](https://github.com/jackc/tern), never executed ad hoc, via **two** migrators with two independent version tables:

| Migrator | Version table                        | Source                          | Changes when           |
| -------- | ------------------------------------ | ------------------------------- | ---------------------- |
| internal | `_migration.schema_version_internal` | `//go:embed migrations/*.sql`   | the binary is upgraded |
| mapping  | `_migration.schema_version_mapping`  | generated from the mapping YAML | the mapping changes    |

Separating them means a tool upgrade never touches user tables, and a mapping change never touches the tool's own bookkeeping. `migrate.NewMigrator` takes a dedicated `*pgx.Conn` acquired for the duration of a migration run and released afterward — migrations never share a pooled connection with the writer.

Two of tern's own constraints shape the generated SQL: `CREATE INDEX CONCURRENTLY` cannot run inside a transaction, so every index is emitted as its own migration carrying tern's `disable-tx` marker; and tern renders every migration as a Go template, so config validation rejects any mapping string containing `{{` or `}}` before it ever reaches DDL.

## PostgreSQL access

Driven through [`jackc/pgx/v5`](https://github.com/jackc/pgx) natively — never `database/sql` — because backfill and CDC both depend on protocol features `database/sql` cannot express: `pgx.CopyFrom`'s binary `COPY` protocol for backfill throughput, the binary wire format for `timestamptz`/`uuid`/`interval`/`jsonb`/`bytea` and array types (most of this workload's payload, given how timestamp- and UUID-heavy it is), and `pgx.Batch` to pipeline a CDC batch's statements into one round trip.

`COPY` cannot express `ON CONFLICT`, so backfill is `CopyFrom` into a session-scoped `UNLOGGED` staging table followed by one `INSERT … SELECT … ON CONFLICT` — full `COPY` throughput while keeping the idempotent write guard described in [The phase machine](/mongo2pg/concepts/concepts/phases.md).

## Change data capture

MongoDB change streams, watched at the **database** level with a single stream per service rather than one per collection — one cursor, one resume token, one ordering, far less connection overhead than a per-collection stream would carry. `fullDocument: "updateLookup"` means an update event always carries the whole post-image, so the mapper never needs a second read to apply one. `delete` events carry only `documentKey`, which is sufficient: `_id` resolves to a PK through the same resolver, then a real `DELETE` (or a soft-delete, per the collection's `softDelete` mode — see [Soft deletes and normalization](/mongo2pg/concepts/concepts/soft-deletes-and-normalization.md)). A `drop` or `rename` on a mapped collection is a loud alarm that pauses that collection, since it means the source changed shape underneath the migration.

Batches close on whichever comes first — a document count, a byte size, or a time interval — so if PostgreSQL write latency grows, batches close on size rather than time and the MongoDB cursor naturally slows, with no unbounded in-memory queue. `mongo2pg_queue_depth` exports that pressure so it is visible before it turns into replication lag.

## Testing philosophy

Statement coverage (≥90% on every package) is a **floor against neglect**, not evidence of correctness — the project has been burned by fully-covered code three separate times: a compound condition with one subcondition never exercised, an error propagated through a non-branching assignment, and comma-separated `switch` cases Go counts as one statement for coverage purposes. The bar that actually matters, and that no coverage tool can measure, is **mutation testing**: every substantive behaviour is broken deliberately to confirm the matching test fails.

Test documents for the Mapper and coercers are generated by marshalling the **real service model structs** (`models.TaskEntity`, `models.Event`, …) to BSON, so a field added upstream surfaces as a failing unmapped-field assertion rather than silent data loss the next time this tool runs against production. See [Development](/mongo2pg/contributing/development.md) for how to run the suites yourself.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://atomic-blend.gitbook.io/mongo2pg/contributing/architecture.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.