Census

A schema describes structure you already know. Realistic data also needs the statistical shape of the real thing — how values are distributed, how often a field is missing or null, how long strings run, which strings are really enums — and that part is usually guessed. @ghostry/fabricator-census takes a census of real data and reports its shape in fabricator's own vocabulary: distribution variants, whereby ranges, .weighted(...) weights, and string composition.

import { census, report } from "@ghostry/fabricator-census";
 
const users = census.from(rows);
 
console.log(report(users));

rows is any iterable of values already decoded into JS: an array parsed from JSON, rows from a database driver. Nothing has to be declared first — a census reads plain values, never a Schema, and works the structure out as it goes. The report has one line per path, written as the builder call that path suggests:

Census of 5000 values.

$  T.object({ "id", "age", "plan", "referrer", "nickname", "spend" })
$.id  T.number.integer.sequence
    note: Values ran 1, 2, 3, … in input order; T.number.integer.sequence counts the same way.
$.age  T.number.integer.whereby({ min: 18, max: 79, distribution: { kind: "normal", mean: 38.31, spread: 11.95 } })
    fit: normal 0.0049 · triangular 0.0956 · skew 0.1823 · uniform 0.3254 (simpler wins within 0.0192, n = 5000)
$.plan  T.enum.weighted([[3528, "free"], [1237, "pro"], [235, "team"]])
$.referrer  T.nullable(T.string.whereby({ length: 8, composition: { uppercase: 5046, digit: 5018 } })).weighted({ null: 3742, value: 1258 })
    note: composition draws each code point uniformly within its source, so per-character frequencies are not reproduced.
$.nickname  T.omittable(T.string.whereby({ length: { min: 2, max: 10, distribution: { kind: "uniform" } }, composition: { lowercase: 9744 } })).weighted({ omitted: 3367, value: 1633 })
    note: composition draws each code point uniformly within its source, so per-character frequencies are not reproduced.
    fit: normal 0.0053 · skew 0.0078 · uniform 0.0116 · triangular 0.1176 (simpler wins within 0.0337, n = 1633)
$.spend  T.number.whereby({ min: 3.428821919648706e-11, max: 4995.5303206052795, distribution: { kind: "skew", exponent: 2.922 } })
    fit: skew 0.0168 · normal 0.2150 · triangular 0.2773 · uniform 0.3827 · logarithmic 0.6920 (simpler wins within 0.0192, n = 5000)

Not detected: cross-field correlations (a field present only when another holds some value), discriminated unions, positional tuples, and recursive schemas.

Taking a census

census.from(values) reads an iterable in one pass. For any other source — a driver cursor, a stream, several batches — census.begin() returns an accumulator, and the loop is yours:

const tally = census.begin();
 
for await (const row of cursor) tally.add(row);
 
const users = tally.finish();

That is why there is no async variant: the loop is the only asynchronous part. finish() returns the census, and calling add after it throws.

Every value is counted, so counts, type mixes and presence are exact. Quantiles, histograms and fits are exact too while a field has seen no more than reservoir values; past that, they come from a uniform sample of that size.

Those caps are what keep memory flat: data keyed by id cannot grow a census without bound, and neither can a deep tree. A suggestion says so wherever one was reached.

A census is plain data. JSON.stringify it, store it, and suggest from it later; the values JSON cannot carry are tagged, a bigint as { type: "bigint", value: "…" }.

Reading the suggestions

suggest(census) returns the tree that report prints: one node per path, tagged with the kind it suggests and holding that builder's arguments as data.

import { suggest } from "@ghostry/fabricator-census";
 
const suggestion = suggest(users);
 
if (suggestion.kind === "object") {
  suggestion.fields.find((field) => field.key === "plan");
  // { key: "plan", value: { kind: "enum", items: [[3528, "free"], …], notes: [] } }
}

A suggestion is data — not a Schema, and not source text. You read it and write the schema, which keeps it serializable and leaves the decisions it cannot make to you.

Weights are the observed counts, unrounded. fabricator's weights are relative, so { null: 3742, value: 1258 } and { null: 3, value: 1 } describe nearly the same schema. A nested wrapper's weights count only what the outer one let through: in T.omittable(T.nullable(x)), the inner null and value add up to the objects that had the key.

A string's length counts UTF-16 code units, the same budget T.string spends, so "😀".length is 2 here as it is there.

Fits

For every range, a census fits each variant the range could take and scores it by the largest gap between the data's cumulative distribution and the candidate's. The fit: line lists the candidates, best first.

• Scores only rank candidates. Each candidate's parameters were estimated from the same data it is scored against, so a score is not a significance test.
• Simpler wins a near-tie. A candidate with fewer parameters is chosen when its score is within 1.36 / √n of the best — the "simpler wins within" figure. uniform has none, so data with no real shape reads as uniform rather than as a faint normal, as nickname's length does above.
• Whole numbers are scored the way fabricator draws them. T.number.integer and every length draw over [min, max + 1) and floor, so those fields are scored against exactly those buckets rather than against a continuous curve.
• logarithmic is tried only where it applies — a range that starts above zero and spans at least two orders of magnitude.
• Two peaks are not fitted. multi and custom are never suggested. A bimodal field shows up as every candidate scoring poorly; its histogram is on the census, at node.number.summary.histogram.

Paste a fitted distribution into the explorer to see the shape it describes. fit(numeric, options) is exported as well, for fitting any numeric tally in a census yourself.

Notes

Every suggestion carries notes: what the builder call alone would not tell you. That every value was distinct, where fabricator has no uniqueness facility and a fabricated range will collide. That composition draws uniformly within each character class, so letter frequencies are not reproduced. That an object's keys look like data rather than structure.

What a census does not detect

• Cross-field correlations — shippedAt present only when status is "shipped". That is .refine or T.derive.
• Discriminated unions. Objects of different shapes merge into one node with many spurious T.omittable fields. A low-cardinality string field whose values split the other fields' presence cleanly gets a note saying so.
• Positional tuples. An array is one merged element and a length.
• Recursive schemas. A tree is described level by level, down to depth.

Writing the schema

The report above, written out:

const User = T.object({
  id: T.number.integer.sequence,
  age: T.number.integer.whereby({
    min: 18,
    max: 90,
    distribution: { kind: "normal", mean: 38, spread: 12 },
  }),
  plan: T.enum.weighted([
    [14, "free"],
    [5, "pro"],
    [1, "team"],
  ]),
  referrer: T.nullable(
    T.string.whereby({ length: 8, composition: { uppercase: 1, digit: 1 } }),
  ).weighted({ null: 3, value: 1 }),
  nickname: T.omittable(
    T.string.whereby({
      length: { min: 2, max: 10 },
      composition: { lowercase: 1 },
    }),
  ).weighted({ omitted: 2, value: 1 }),
  spend: T.number.whereby({
    min: 0,
    max: 5000,
    distribution: { kind: "skew", exponent: 3 },
  }),
});

What changed on the way is the judgment a census leaves to you:

• Ranges. A sample's extremes sit inside the true range. 79 was the oldest user seen; the domain says 90. 3.4e-11 is zero.
• Weights and parameters. Rounded to what the numbers support. A spread of 11.95 from 5,000 rows is 12.
• Defaults. distribution: { kind: "uniform" } is the default, so it is left out.

How many rows

sampleSize() answers how large a sample a census needs for its statistics to hold to a margin:

import { sampleSize } from "@ghostry/fabricator-census";
 
sampleSize();
// { n: 4612, reasons: [{ criterion: "shape", n: 4612, … }, …] }
 
sampleSize({ population: 20_000, rare: 0.001 });
// { n: 3748, reasons: [{ criterion: "shape", n: 3748, … }, …] }

n is the largest of up to three requirements, and reasons names each:

margin defaults to 0.02 and confidence to 0.95. population applies the finite population correction, and caps n at the population. The number does not grow with the table: 4,612 rows describe two million as well as they describe twenty thousand.

From PostgreSQL

@ghostry/fabricator-census/postgres builds the queries a census is fed from — a sample of the right size, and facts about the whole table to go with it. It never opens a connection: every query runs through your own driver.

import { census, report } from "@ghostry/fabricator-census";
import { collect, type Run } from "@ghostry/fabricator-census/postgres";
 
const run: Run = async (text, values) =>
  (await client.query(text, values)).rows;
 
const { rows, facts } = await collect(run, { table: "users" });
 
console.log(report(census.from(rows, { facts })));

run is the whole integration: query text with $1 placeholders and an array of values in, rows out. The line above is node-postgres and PGlite alike. A table is a name, or { schema, table }.

collect runs these, in order:

1. The planner's row estimate, from pg_class. A table never analyzed has none, and is counted instead.
2. The table's columns and their types.
3. The planner's statistics, from pg_stats, as estimated facts.
4. A sample, sized by sampleSize against the row count.
5. One aggregate query per column, as exact facts.

It returns the sampled rows and the facts, along with the columns, the population, the size it chose and every query it ran as queries.

Facts: what the whole table knows

A sample cannot tell you some things reliably, and the database can answer them exactly with one aggregate each. Facts are those answers, per column:

Passed as census.from(rows, { facts }), they are merged over the sample's own numbers when suggesting: an exact fact first, then an estimated one, then the sample.

• Exact facts come from the aggregates, which read every row.
• Estimated facts come from the planner's statistics. They cost nothing to read, but ANALYZE drew them from a sample of its own, so a suggestion that used one says so in its notes.

Facts set presence weights, ranges, enum members and their weights, and uniqueness. The distribution's shape is still fitted from the sample, as positions inside the table's range, because no aggregate describes a shape. Table extremes only ever widen a range.

Two limits follow from what facts are. They describe top-level columns, so anything nested inside a json column comes from the sample alone. And they describe the whole table: sample.where narrows the sample without narrowing the facts, so do not combine the two.

Facts is plain data with no PostgreSQL in it. They can come from another database, a cache, or your own hand.

What it costs

Every query carries a cost, and collect's queries lists what ran:

On a large table the scans are the expensive part. Run them against a replica, with a statement_timeout, or skip them:

await collect(run, { table: "users", budget: "catalog" });

Under budget: "catalog" every fact is an estimate, and a table that was never analyzed is an error rather than a counting scan. A serial column also stops reading as T.number.integer.sequence: that takes exact extremes, since the planner's sit inside the true range.

An exact distinct count is the one aggregate left out by default, because count(DISTINCT …) is the costliest of them. The planner's estimate stands in; aggregates: { distinct: "exact" } asks for the real one.

Choosing a sampling method

await collect(run, {
  table: "events",
  sample: { method: "hash", key: "id", seed: 1 },
});

system is the fast one on a huge table, and the wrong one for an append-only, time-ordered table, where a page holds rows from the same moment. hash is the one to use when a census should be repeatable against a table that keeps changing.

No method uses LIMIT: truncating a scan favors whichever rows it reaches first. The row count therefore lands near n rather than on it.

The builders on their own

collect is sugar. Each step is a builder returning a Query — text, values, cost, notes, and a read that parses the driver's rows:

import { census, sampleSize } from "@ghostry/fabricator-census";
import {
  aggregates,
  coerce,
  columns,
  count,
  estimate,
  estimatedFacts,
  sample,
  statistics,
  type Query,
} from "@ghostry/fabricator-census/postgres";
 
const execute = async <$Result>(query: Query<$Result>) =>
  query.read(await run(query.text, query.values));
 
const population =
  (await execute(estimate("users"))) ?? (await execute(count("users")));
const described = await execute(columns("users"));
 
const facts = estimatedFacts(
  await execute(statistics("users")),
  described,
  population,
);
 
const { n } = sampleSize({ population });
const rows = coerce(
  await execute(sample("users", { n, population })),
  described,
);
 
for (const query of aggregates("users", described))
  facts.push(await execute(query));
 
const users = census.from(rows, { facts });

Taking them apart is how you sample from one connection and aggregate on another, cache the facts, or leave a column out.

Identifiers are always quoted, and numbers always travel as parameters, never in the query text. sample's where is the one exception to both: it is raw SQL, for trusted text only.

How column types are read

Drivers disagree about the types a census cares most about, so coerce — which collect runs for you — reads each column by its declared type:

A few types come with a caveat, which rides along as a note on the suggestion:

• timestamp without time zone has no offset, so drivers read it in the process's local time zone, and its instants shift with where the census runs.
• date reads as local or UTC midnight depending on the driver.
• Text lengths. PostgreSQL's char_length counts code points where a census counts UTF-16 units. The table's lengths are used when the sample held no character outside the Basic Multilingual Plane; otherwise the sample's stand.

SQL rows are flat, and every column is present in every row. A column is therefore never suggested as T.omittable — a missing value is a null, and reads as T.nullable.