Skip to content

fix: include the version in the default Parquet created_by - #1835

Open
mhatipog wants to merge 1 commit into
apache:mainfrom
mhatipog:parquet-created-by-version
Open

mhatipog wants to merge 1 commit into
apache:mainfrom
mhatipog:parquet-created-by-version

Conversation

@mhatipog

Copy link
Copy Markdown

Which issue does this PR close?

No issue; it is a one-line default change.

Rationale for this change

The default created_by has no version, so parquet-java's PARQUET-246 check cannot parse it and refuses DELTA_BYTE_ARRAY columns (e.g. Spark with the vectorized reader off). This uses the <app> version <x> form the format asks for.

What changes are included in this PR?

  • ParquetWriterOptions now defaults created_by to datafusion-python version <version>.
  • A test that reads the default back from the written files.

Are there any user-facing changes?

Files written with default ParquetWriterOptions carry the version in created_by. Passing created_by explicitly works as before.

🤖 Generated with Claude Code

parquet-java parses created_by as "<application> version <version>"
(PARQUET-246). Without a version it cannot parse the writer and refuses
DELTA_BYTE_ARRAY columns, e.g. in Spark with the vectorized reader off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant