Schema-driven synthetic data for engineering teams

Have a schema but no safe test data?

Great Generator creates realistic, fake, non-production data from your schema for lower environments, pipeline testing, QA, analytics, Pandas, and Spark.

Start in 30 seconds View repository

Your schema in. Realistic test data out.

from great_generator import generate_from_schema

schema = {
    "customer_id": "string",
    "customer_name": "string",
    "email": "string",
    "age": "int",
    "created_at": "datetime",
}

df = generate_from_schema(schema, rows=1000)

Schema-first Start from mappings, Pandas dtypes, compact DDL, or PySpark schemas.

DataFrame-first Return Pandas or Spark DataFrames, then write to any destination your runtime supports.

Semantically realistic Recognize names, IDs, contacts, dates, amounts, statuses, and lifecycle relationships.

Enterprise-ready CDC, anomalies, referential data, dimensional models, Data Vault models, and exports.

Why it exists

Lower environments need useful data without production records.

Teams already have schemas, table contracts, and pipeline definitions, but production records may be unavailable or restricted. Great Generator turns those schemas into realistic synthetic DataFrames for development, QA, analytics, demonstrations, and pipeline testing.

Compared with Faker

Faker creates individual fake values. Great Generator creates complete datasets with relationships, behavior, and outputs that developers can use immediately.

Compared with heavy synthetic data platforms

Great Generator is lightweight and developer-first. It is designed for fast starts, demos, pipelines, tests, and repeatable examples rather than statistical modeling of private source data.

Install

Simple package, optional Spark and Delta extras.

The base package stays local-friendly. Spark and Delta support are optional so users can install only what their environment needs.

pip install great-generator

pip install "great-generator[spark]"

pip install "great-generator[delta]"

Quickstart

Start from the schema you already have.

Use schema generation for real project structures. Domain packs remain available for ready-made demos and learning.

Generate from a Python schema

from great_generator import generate_from_schema

schema = {
    "customer_id": "string",
    "customer_name": "string",
    "email": "string",
    "age": "int",
}

df = generate_from_schema(schema, rows=1000)

Generate related tables

from great_generator import generate_relational

data = generate_relational(
    tables={
        "customers": "customer_id int primary key, customer_name string",
        "orders": "order_id int primary key, customer_id int references customers.customer_id",
    },
    rows={"customers": 1000, "orders": 5000},
)

customers = data["customers"]
orders = data["orders"]

Generate from a schema string

from great_generator import generate_from_schema

df = generate_from_schema(
    "customer_id int, customer_name string, signup_date date, annual_income double",
    rows=100,
)

Generate banking CDC records

from great_generator import generate_cdc

cdc = generate_cdc(
    "banking",
    table="customers",
    rows=1000,
    operations=["insert", "update", "delete"],
    late_arrival_rate=0.02,
)

Spark and lakehouse

Designed for notebooks, Databricks, and Spark clusters.

Use Spark mode when data should be generated as PySpark DataFrames and written with Spark-native writers. This works with DBFS, ADLS, S3, GCS, HDFS, local paths, and catalog tables when those destinations are configured in your Spark environment.

from great_generator import generate_domain

data = generate_domain(
    "banking",
    engine="spark",
    scale="large",
    spark=spark,
)

data["transactions"].write.mode("overwrite").parquet(
    "s3://your-bucket/demo/banking/transactions"
)

Features

The first-release feature set is intentionally practical.

Prebuilt domains

Generate complete industry datasets with related tables and realistic business behavior.

Referential integrity

Parent tables are generated before child tables, with valid primary keys and foreign keys by default.

Semantic schema generation

Name-like, date-like, amount-like, status-like, ID-like, and domain-specific fields are detected from column names.

Pandas and Spark

Return the right DataFrame type for your engine, then write through native DataFrame APIs.

Export helpers

CSV, JSON, Parquet, and Delta convenience exports with table-per-folder output.

Cloud paths

Use local paths with Pandas and DBFS, ADLS, S3, GCS, HDFS, or mounted paths with Spark runtimes.

CDC simulation

Create insert, update, and delete records with event timestamps, ingestion timestamps, late arrival flags, and duplicates when configured.

Anomaly injection

Inject nulls, duplicates, orphan keys, late records, outliers, invalid statuses, and negative amounts for quality testing.

Dimensional models

Create facts and dimensions for analytics engineering, SQL modeling, BI demos, and warehouse examples.

Data Vault models

Create hubs, links, and satellites for enterprise architecture demos and modeling experiments.

Quality checks

Explain generation plans and validate generated schema data for semantic consistency.

Recipes and CLI

Package repeatable data generation scenarios as files or run them from the command line.

Platforms

Use the same library across local notebooks and cloud Spark environments.

Environment	Recommended engine	Typical outputs	Notes
Local Python	Pandas	DataFrame, CSV, JSON, Parquet	Best for unit tests, demos, tutorials, and small to medium local data.
Jupyter or Anaconda	Pandas	DataFrame, local files	Great for exploration and presentation demos.
Databricks	Spark	DataFrame, DBFS, ADLS, S3, Delta	Use paths and credentials already configured in the workspace.
EMR, Glue, Synapse, Fabric, Dataproc	Spark	DataFrame, object storage, Parquet, Delta where available	Use Spark-native writes for cloud storage and catalog targets.
Research and benchmarking	Pandas or Spark	Repeatable DataFrames and files	Use explicit row counts and document your environment for reproducibility.

Scale guidance: Great Generator is designed to support small to large datasets. It can be used to generate datasets ranging from one row to millions of rows, depending on the user environment and memory or compute setup. For very large datasets, chunking or Spark-native generation is recommended.

DataFrame-first output

Return the data first, then write wherever you need it.

Great Generator does not force a storage decision. Functions return a DataFrame or a dictionary of table-name DataFrames. Users can inspect, transform, validate, join, visualize, or write using normal Pandas and Spark APIs.

Pandas write options

data = generate_domain("healthcare")

patients = data["patients"]
patients.to_csv("patients.csv", index=False)
patients.to_json("patients.json", orient="records")
patients.to_parquet("patients.parquet")

Spark write options

data = generate_domain("telecom", engine="spark", spark=spark)

usage = data["usage_events"]
usage.write.mode("overwrite").format("delta").save(
    "abfss://container@account.dfs.core.windows.net/demo/usage"
)

Documentation

Guides for demos, implementation, and release-readiness.

Schema inputsCurrent support matrix, examples, and limitations. QuickstartInstall, import, generate, inspect, and export. API referenceMain public APIs and usage patterns. Realistic valuesSemantic field detection and generation quality. Domain packsSupported industry domains and table layouts. Spark and DeltaCluster usage, large data, and lakehouse outputs. Spark database writesDatabricks and PySpark writes to Snowflake and Azure SQL. Cloud deploymentAWS, Azure, GCP, Databricks, and storage guidance. CDCChange data capture style events. AnomaliesData quality problems for testing tools and pipelines. Demo guideSpeaker-friendly story, examples, and demo flow.

Author

Created and maintained by Ravi Kiran Pagidi.

Great Generator is an open-source Python library for developers, data engineers, QA teams, analytics engineers, and platform teams that need realistic lower-environment data without production records.

PyPI project GitHub repository GitHub Wiki Contact Ravi