GH-47628: [C++][Parquet] Implement basic parquet file rewriter #47775

HuaHuaY · 2025-10-10T06:58:56Z

This is a draft PR now. I follow Java's implementation but I think it is not a good enough design for C++. Because we must copy lots of code from file_writer.cc or file_reader.cc and it will be troublesome to maintain in the future. I prefer to implement some classes inheriting XXXWriter or XXXReader. I'll think about how to refactor the code. If anyone has any good suggestions, please comment.

Now I have written two kinds of tests. Test the horizontal splicing and vertical splicing of parquet files separately. But only horizontal splicing is implemented now because I don't find an efficient way to merge two parquet files' schema.

Rationale for this change

Allow to rewrite parquet files in binary data formats instead of reading, decoding all values and writing them.

What changes are included in this PR?

Add class ParquetFileRewriter and RewriterProperties.
Add some to_thrift and SetXXX methods to help me copy the metadata.
Add CopyStream methods to call memcpy between ArrowInputStream and ArrowOutputStream.
Add RowGroupMetaDataBuilder::NextColumnChunk(std::unique_ptr<ColumnChunkMetaData> cc_metadata, int64_t shift) which allows to add column metadata without creating ColumnChunkMetaDataBuilder.

Are these changes tested?

Yes

Are there any user-facing changes?

Add some new classes and methods mentioned above.
ReaderProperties::GetStream is changed to a const method. Only the signature has been changed. Its original implementation allows it to be declared as a const method.

GitHub Issue: [C++][Parquet] Provide a rewriter to rewrite parquet files without decoding all the row groups/pages #47628

github-actions · 2025-10-10T06:59:20Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

Pull request overview

Draft implementation of a C++ Parquet “file rewriter” that can rewrite/concatenate/join Parquet files by copying encoded bytes and re-emitting metadata (optionally including page indexes and bloom filters), avoiding full decode/re-encode.

Changes:

Add ParquetFileRewriter + RewriterProperties and implement basic concat (horizontal) + join (vertical) rewriting.
Add metadata copying helpers (to_thrift, new builder entrypoints, new ToThrift overloads) to support fast metadata reconstruction.
Add Arrow-based tests for roundtrip rewriting scenarios (simple, concat, join, concat+join) and wire into CMake.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
cpp/src/parquet/thrift_internal.h	Add `ToThrift` helpers for page-index-related structs.
cpp/src/parquet/properties.h	Make `ReaderProperties::GetStream` const; add `RewriterProperties` + default factory decl.
cpp/src/parquet/properties.cc	Update `ReaderProperties::GetStream` definition to const.
cpp/src/parquet/page_index.h	Extend `PageIndexBuilder` API to allow setting pre-built indexes.
cpp/src/parquet/page_index.cc	Implement new `PageIndexBuilder` setters and mixed builder/prebuilt serialization.
cpp/src/parquet/metadata.h	Add `start_offset()`, expose `to_thrift()`, extend `RowGroupMetaDataBuilder`.
cpp/src/parquet/metadata.cc	Implement new metadata accessors and new row-group column-chunk injection path.
cpp/src/parquet/file_rewriter.h	New public rewriter API (`ParquetFileRewriter`).
cpp/src/parquet/file_rewriter.cc	Core rewriting implementation (copy streams, concat/join logic, index/bloom handling).
cpp/src/parquet/bloom_filter_writer.h	Extend `BloomFilterBuilder` with `InsertBloomFilter`.
cpp/src/parquet/bloom_filter_writer.cc	Implement `InsertBloomFilter`.
cpp/src/parquet/arrow/test_util.h	Add helper to write a table into a Parquet buffer for tests.
cpp/src/parquet/arrow/arrow_rewriter_test.cc	Add Arrow-level rewriter roundtrip tests.
cpp/src/parquet/CMakeLists.txt	Add new source (`file_rewriter.cc`) and new test target.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cpp/src/parquet/bloom_filter_writer.cc

cpp/src/parquet/CMakeLists.txt

cpp/src/parquet/bloom_filter_writer.h

Copilot · 2026-02-09T12:53:55Z

cpp/src/parquet/file_rewriter.cc

+      if (page_index_reader_ != nullptr && page_index_builder != nullptr) {
+        auto column_index = page_index_reader_->GetColumnIndex(column_ordinal_);
+        auto offset_index = page_index_reader_->GetOffsetIndex(column_ordinal_);
+        if (column_index != nullptr) {
+          page_index_builder->SetColumnIndex(column_ordinal_, column_index);
+        }
+        if (offset_index != nullptr) {
+          page_index_builder->SetOffsetIndex(column_ordinal_, offset_index, shift);
+        }
+      }
+
+      if (bloom_filter_reader_ != nullptr && bloom_filter_builder != nullptr) {
+        auto bloom_filter = bloom_filter_reader_->GetColumnBloomFilter(column_ordinal_);
+        if (bloom_filter != nullptr) {
+          bloom_filter_builder->InsertBloomFilter(column_ordinal_,
+                                                  std::move(bloom_filter));
+        }
+      }


The new rewriter code path copies page indexes (SetColumnIndex/SetOffsetIndex) and bloom filters (InsertBloomFilter) when enabled, but the added Arrow-level roundtrip tests only validate the decoded table contents. Consider extending tests to assert that rewritten files actually contain the expected page index / bloom filter structures (e.g., via ParquetFileReader::GetPageIndexReader / GetBloomFilterReader and checking non-null indexes/locations), so regressions in metadata copying/offset shifting are caught.

cpp/src/parquet/metadata.cc

cpp/src/parquet/file_rewriter.cc

Copilot · 2026-02-09T12:53:57Z

cpp/src/parquet/properties.h

  std::shared_ptr<ArrowInputStream> GetStream(std::shared_ptr<ArrowInputFile> source,
-                                              int64_t start, int64_t num_bytes);
+                                              int64_t start, int64_t num_bytes) const;



Changing ReaderProperties::GetStream to be const changes the mangled symbol and will break ABI for downstream code compiled against earlier versions of the library. If ABI compatibility matters here, consider keeping the old non-const overload (forwarding to the const implementation) instead of changing the existing signature in-place.

Suggested change

// Non-const overload kept for ABI compatibility. It forwards to the const

// implementation introduced in a later version.

std::shared_ptr<ArrowInputStream> GetStream(std::shared_ptr<ArrowInputFile> source,

int64_t start, int64_t num_bytes) {

return static_cast<const ReaderProperties*>(this)->GetStream(source, start, num_bytes);

}

Copilot · 2026-02-09T12:53:57Z

cpp/src/parquet/metadata.h

+  const void* to_thrift() const;
+


ColumnChunkMetaData::to_thrift() exposes an untyped pointer to an internal Thrift struct as part of the public metadata API, which couples consumers to internal representation and is easy to misuse/UB (wrong cast / lifetime assumptions). Prefer an internal-only accessor, or return a typed reference/pointer to the concrete thrift type in an internal header, or provide a dedicated cloning/copy helper on the builder to avoid exposing raw thrift at all.

cpp/src/parquet/page_index.h

cpp/src/parquet/metadata.cc

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cpp/src/parquet/file_rewriter.cc

cpp/src/parquet/metadata.cc

cpp/src/parquet/bloom_filter_writer.cc

cpp/src/parquet/file_rewriter.cc

cpp/src/parquet/meson.build

github-actions · 2026-02-10T10:44:56Z

⚠️ GitHub issue #47628 has been automatically assigned in GitHub to PR creator.

github-actions bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Oct 10, 2025

HuaHuaY changed the title ~~[C++][Parquet] Implement basic parquet file rewriter~~ [C++][Parquet] GH-47628: Implement basic parquet file rewriter Oct 10, 2025

HuaHuaY force-pushed the fix_issue_47664 branch from e4de469 to c216849 Compare October 10, 2025 07:36

mapleFU reviewed Oct 13, 2025

View reviewed changes

cpp/src/parquet/properties.h Show resolved Hide resolved

cpp/src/parquet/page_index.cc Outdated Show resolved Hide resolved

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 13, 2025

wgtmac reviewed Oct 13, 2025

View reviewed changes

adamreeve reviewed Oct 15, 2025

View reviewed changes

cpp/src/parquet/file_rewriter.h Show resolved Hide resolved

wgtmac reviewed Oct 16, 2025

View reviewed changes

HuaHuaY force-pushed the fix_issue_47664 branch from e037be7 to 253f281 Compare October 23, 2025 10:49

HuaHuaY force-pushed the fix_issue_47664 branch 2 times, most recently from b70917f to 439103e Compare January 15, 2026 04:36

HuaHuaY force-pushed the fix_issue_47664 branch from 439103e to 641ab8c Compare February 9, 2026 08:15

HuaHuaY marked this pull request as ready for review February 9, 2026 12:36

Copilot AI review requested due to automatic review settings February 9, 2026 12:36

Copilot started reviewing on behalf of HuaHuaY February 9, 2026 12:43 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

HuaHuaY force-pushed the fix_issue_47664 branch from 7be60a0 to 41248c8 Compare February 10, 2026 09:35

HuaHuaY requested a review from Copilot February 10, 2026 09:35

Copilot started reviewing on behalf of HuaHuaY February 10, 2026 09:38 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

Implement basic parquet file rewriter

2140284

HuaHuaY force-pushed the fix_issue_47664 branch from 41248c8 to 2140284 Compare February 10, 2026 10:14

HuaHuaY changed the title ~~[C++][Parquet] GH-47628: Implement basic parquet file rewriter~~ GH-47628: [C++][Parquet] Implement basic parquet file rewriter Feb 10, 2026

+                RowGroupRewriter(std::shared_ptr<ArrowInputFile> source,
+                                 std::shared_ptr<ArrowOutputStream> sink,
+                                 const RewriterProperties* props,
+                                 std::shared_ptr<RowGroupReader> row_group_reader,

+  // Non-const overload kept for ABI compatibility. It forwards to the const
+  // implementation introduced in a later version.
+  std::shared_ptr<ArrowInputStream> GetStream(std::shared_ptr<ArrowInputFile> source,
+                                              int64_t start, int64_t num_bytes) {
+    return static_cast<const ReaderProperties*>(this)->GetStream(source, start, num_bytes);
+  }

GH-47628: [C++][Parquet] Implement basic parquet file rewriter #47775

Are you sure you want to change the base?

GH-47628: [C++][Parquet] Implement basic parquet file rewriter #47775

Conversation

HuaHuaY commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Oct 10, 2025

Uh oh!

HuaHuaY commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wgtmac Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HuaHuaY commented Oct 10, 2025 •

edited

Loading

HuaHuaY commented Oct 13, 2025 •

edited

Loading