Spark and lakehouse

Spark, Databricks, Delta, and lakehouse workflows

Spark support is optional and is designed for environments where PySpark is already available.

Great Generator creates synthetic data. It does not anonymize, mask, de-identify, or transform production records.

Generate Spark DataFrames

Use the Spark engine when running in a Spark-capable notebook or cluster.

df = generate_from_schema(
    schema="id int, customer_name string, created_at timestamp",
    rows=100000,
    engine="spark",
)

Write with Spark APIs

Great Generator returns DataFrames so the user controls the destination and format.

df.write.mode("overwrite").parquet("s3://bucket/demo/customers")
df.write.format("delta").mode("overwrite").save("dbfs:/tmp/customers_delta")

Environment note

Large outputs depend on the user environment, memory, cluster size, and storage configuration. For very large datasets, Spark-native generation or chunking is recommended.

Keep exploring

Installation optionsInstall Great Generator for core Pandas workflows, Spark, Delta Lake, optional AI advisors, or the optional local MCP server. Generate synthetic data from schemasCreate synthetic test data from schema strings, dictionaries, Pandas DataFrames, and Spark schemas using semantic field detection. Relational synthetic dataGenerate related synthetic tables with primary-key and foreign-key behavior for joins, warehouse models, tests, demos, and analytics examples. API reference overviewReview the main Great Generator public APIs for schema generation, SQL DDL parsing, relational generation, CDC, anomalies, advisors, and exports. GitHub repositorySource code, issues, releases, and project history. PyPI packageInstall Great Generator from Python Package Index.