Migrate a small MongoDB collection to Cloudflare D1 with Node.js

Moving a collection from MongoDB to Cloudflare D1 involves decisions that a format converter cannot make for you. MongoDB documents can contain BSON types, nested structures, and inconsistent fields. D1 expects a relational schema and SQLite-compatible values. A successful import needs an explicit agreement between those two models.

We will migrate a small customer collection using a Node.js exporter and Wrangler's local D1 runtime. The exporter streams documents, validates their shape, and writes bounded SQL statements. Replaying those statements updates existing rows by their original identifiers. The sample uses synthetic fixtures; no live MongoDB server or remote D1 database was used during verification.

This is a practical pattern for a modest, controlled migration in a stack like Cloudmerce's. It is not a change-data-capture system, an arbitrary BSON converter, or a promise that moving databases improves application performance. Those are separate engineering questions.

Run the complete example first

The project lives in code/mongo-to-d1. From the articles directory:

cd code/mongo-to-d1
npm ci
npm run export:fixture
npm run check

Verification used Node 22.23.3, MongoDB's Node driver 6.21.0, and Wrangler 4.90.0. The package lockfile fixes the dependency tree. This project is plain JavaScript, so there is no compilation step: running the exporter and the integration check exercises the executable code.

The fixture export writes generated/export/manifest.json and numbered SQL files. It requires an empty output directory. To repeat the export, choose a fresh destination:

OUTPUT_DIR=generated/second-export npm run export:fixture

The check uses a separate temporary directory and its own local D1 state. It creates the schema, imports two batches, imports them again, and compares every stored value with the expected mapping. It also exercises invalid documents and an oversized row. It cleans up its temporary state afterward.

Decide what each field means

Our source contract is intentionally narrow:

Preserving _id keeps existing application references meaningful. Replacing it with an unrelated autoincrement key would require another mapping table and changes wherever those references are used.

We keep arrays as JSON because this example reads an entire customer record. If the application frequently filters by individual tags, consider a separate customer-tags table and suitable indexes. The schema should follow the destination access patterns rather than mechanically reproducing every document shape.

The sample treats a missing profile and an explicit null profile identically. It also projects away fields outside the contract. Both choices are data decisions: review them before exporting real records. An audit should identify any fields or distinctions that would be lost.

Create a schema with useful constraints

The accompanying schema.sql contains:

CREATE TABLE IF NOT EXISTS customers (
  id TEXT PRIMARY KEY NOT NULL,
  email TEXT NOT NULL,
  active INTEGER NOT NULL CHECK (active IN (0, 1)),
  created_at TEXT NOT NULL,
  tags_json TEXT NOT NULL CHECK (json_valid(tags_json)),
  profile_json TEXT CHECK (profile_json IS NULL OR json_valid(profile_json))
);
CREATE INDEX IF NOT EXISTS customers_email ON customers(email);

The email index supports lookups without imposing uniqueness that the source contract never guaranteed. If your product requires unique email addresses, first measure and resolve duplicates in the source, then make that constraint explicit.

SQLite's type system differs from MongoDB's BSON model. Here the exporter checks application types, while SQL constraints provide another boundary for required values, booleans, and valid JSON. The timestamp column's text declaration does not by itself validate a timestamp; that guarantee comes from the mapping function.

CREATE TABLE IF NOT EXISTS makes initial setup repeatable, but does not migrate an existing, incompatible table. For a real application, put schema changes under versioned migrations and test upgrades against a copy of the existing database.

The D1 import guide requires SQLite-compatible SQL rather than a raw SQLite database file. It also warns about transaction wrappers in imported dumps. Our generated files contain insert statements without explicit BEGIN or COMMIT wrappers.

Validate before converting

The full mapCustomer function is in export.mjs. Its most important property is that it rejects unexpected types instead of silently coercing them. For example:

if (typeof doc.active !== 'boolean') {
  throw new Error('Expected active boolean');
}
if (!(doc.createdAt instanceof Date) ||
    !Number.isFinite(doc.createdAt.getTime())) {
  throw new Error('Expected valid createdAt Date');
}

Using Boolean(doc.active) would turn the string "false" into true. Accepting an invalid date would defer the problem to serialization or create inconsistent output. Rejecting either record makes the migration's assumptions visible.

The mapping produces this ordered row:

return [
  doc._id.toHexString(),
  doc.email,
  doc.active ? 1 : 0,
  doc.createdAt.toISOString(),
  JSON.stringify(doc.tags),
  doc.profile == null ? null : JSON.stringify(doc.profile)
];

The executable version also validates ObjectIds, emails, tags, and the flat profile shape. A nested object inside profile is rejected. That prevents an unexpected BSON value from being flattened into a misleading string.

If your source includes Decimal128, binary data, or integers outside JavaScript's safe range, design additional mappings before extending the exporter. MongoDB's Extended JSON documentation explains a representation that preserves BSON type information. Ordinary JSON serialization is not a general solution to BSON fidelity.

Stream from MongoDB

For a real source, the exporter opens a client and obtains a cursor:

const documents = client.db(process.env.MONGO_DB)
  .collection(process.env.MONGO_COLLECTION)
  .find({}, {
    projection: {
      _id: 1, email: 1, active: 1,
      createdAt: 1, tags: 1, profile: 1
    }
  })
  .sort({ _id: 1 })
  .batchSize(100);

The surrounding function consumes it with for await (const doc of documents). This avoids loading the entire collection through toArray(). MongoDB documents asynchronous iteration in its cursor guide.

MongoDB's cursor batch size controls document fetching. It is independent of the SQL batch size used for D1. Keeping those settings separate lets us bound destination statements without pretending that source documents have uniform sizes.

The live-source branch reads MONGO_URI, MONGO_DB, and MONGO_COLLECTION from the process environment. Supply these through your existing secure environment mechanism, using a source account with read-only access. Run node export.mjs without --fixture. The script does not print the URI or raw driver errors, since those can expose connection details.

The live connection path is included for adaptation but was not verified against a running MongoDB instance. The fixture does use real driver ObjectId objects and JavaScript Dates, so the mapping runs against the intended types.

Bound inserts by rows and bytes

Each SQL file contains one multi-row insert followed by an upsert clause. The exporter flushes after at most 50 rows or before a statement would exceed 80,000 UTF-8 bytes. Those are conservative sample settings, not throughput recommendations.

D1's documented limits include a 100,000-byte SQL statement maximum and 100 bound parameters per query at the time of writing. This sample emits literal SQL rather than placeholders, so statement size is the relevant batching constraint. If you switch to prepared statements with six bound values per row, recalculate the row limit against the parameter budget.

Variable-length values make row counts alone unreliable. The exporter includes the insert prefix, separators, and upsert suffix in its byte accounting. It rejects a single row that cannot fit, rather than emitting a statement that the destination will reject later.

To serialize text safely into these generated files, it uses UTF-8 hex literals cast back to text:

function sql(value) {
  if (value === null) return 'NULL';
  if (typeof value === 'number') return String(value);
  return `CAST(X'${Buffer.from(value, 'utf8').toString('hex')}' AS TEXT)`;
}

Only the validated zero-or-one boolean reaches the numeric branch. Column and table names are fixed in code. Hex encoding avoids quote-escaping mistakes and preserves Unicode and embedded control characters; it is encoding, not encryption. Protect generated SQL files as carefully as the source data.

The fixture includes an apostrophe, non-Latin text, a newline, a null character inside a tag, and a string resembling SQL injection. Exact-value assertions confirm that those values remain data after importing.

Import locally and inspect the result

The Wrangler configuration contains a dummy database identifier. With --local, it identifies local development state and does not provision a remote database. Run:

npx wrangler d1 execute migration-demo --local --file schema.sql
for file in generated/export/*.sql; do
  npx wrangler d1 execute migration-demo --local --file "$file" || break
done
npx wrangler d1 execute migration-demo --local \
  --command 'SELECT id, active, created_at FROM customers ORDER BY id'

The loop stops after a failed batch. In an operational migration runner, record the completed filenames and return a failing exit code on error so automation cannot mistake partial progress for completion.

The upsert updates destination fields when id already exists. That makes retrying an immutable export safe for these columns, but it can overwrite edits made in D1 after the first import. It also does not remove destination rows whose source documents were deleted. Define destination ownership and deletion behavior before using this pattern beyond a rehearsal.

Plan the actual cutover

Sorting by _id produces stable ordering; it does not create a consistent snapshot while writers continue changing the collection. For a small migration, a scheduled write pause and a fresh export may be the clearest approach. Larger systems need an explicit snapshot and change-capture strategy.

Export files are individually finalized through a temporary-file rename, and the manifest is written only after iteration succeeds. A failure can still leave completed files from a partial export. Require the manifest, check its counts, and never import a failed export directory as though it were complete. Start the next attempt in a fresh directory.

Before a remote migration, provision the destination through an authorized process, replace the dummy identifier, rehearse against representative data, and define rollback. Compare source and destination counts, selected records, timestamps, and business invariants. A matching count alone cannot detect corrupted field values.

Also validate how the application reads the new representation. A query returns tags_json as text, so the destination data layer must parse it before exposing a string array. The integer active value should be converted deliberately when constructing an API response. Avoid letting a database migration silently change the public response contract.

Keep the original export and its manifest together during reconciliation, with access restricted to the migration operators. Record which export was imported into which destination and when writes resumed. If rollback remains possible, define how destination writes will be recovered first; switching the application back cannot automatically copy new D1 data into MongoDB.

What we learned

We made the migration contract explicit, streamed source documents through strict validation, and generated bounded, replayable SQL. Local D1 checks verified schema compatibility and exact values, including awkward strings. A production cutover still needs a consistent source, destination ownership rules, and independent reconciliation before application traffic moves.