fix(deps): update rust crate lance-encoding to v3

[renovate]

This PR contains the following updates:

Package	Type	Update	Change
lance-encoding	dependencies	major	2.0.0 → 3.0.0

---

Release Notes

lance-format/lance (lance-encoding)

v3.0.1

Compare Source

What's Changed

Other Changes

• refactor: rename arrow-scalar to lance-arrow-scalar by @​westonpace in #​6199

Full Changelog: lance-format/lance@v3.0.0...v3.0.1

v3.0.0

Compare Source

What's Changed

Breaking Changes 🛠

• feat!: support index progress reporting via callbacks by @​wkalt in #​5910
• perf!: remove shuffle buffer by @​wkalt in #​5912
• feat!: upgrade DataFusion dependency to 52.1.0 by @​wjones127 in #​6015
• refactor!: refactor java access to file format version by @​jackye1995 in #​6053
• refactor!: remove create_empty_table usage by @​jackye1995 in #​6087
• fix!: bump IVF_RQ version for compatibility check by @​BubbleCal in #​6097

Critical Fixes ‼️

• fix: deduplicate row addresses in take to prevent panic by @​wjones127 in #​5881
• fix: fts flat search drops rows when avg_doc_length < 1.0 by @​wjones127 in #​5897
• fix: invalidate index fragment bitmaps after data replacement and stale merge by @​wjones127 in #​5929

New Features 🎉

• feat: add RLE support for block by @​yingjianwu98 in #​4937
• feat: compress complex all null by @​yingjianwu98 in #​4990
• feat: support cleanup across branches by @​majin1102 in #​5009
• feat: dictionary index always32 bits by @​yingjianwu98 in #​5011
• feat: abort dictionary encode if not useful by @​yingjianwu98 in #​5055
• feat(cdf): cdf support upsert for views by @​zhangyue19921010 in #​5369
• feat(compaction): binary copy capability for compaction by @​zhangyue19921010 in #​5434
• feat: expose use_scalar_index param in Java scanner by @​xloya in #​5487
• feat(python): expose search_filter in scanner by @​wojiaodoubao in #​5506
• feat: add alter column nullable to non-nullable support by @​Xuanwo in #​5589
• feat: evolute all_null_layout to constant layout by @​Xuanwo in #​5641
• feat(java): support creating IVF_RQ index by @​majin1102 in #​5648
• feat(java): support building vector index distributively by @​majin1102 in #​5664
• feat(rust): add datafusion catalog_provider through namespace by @​majin1102 in #​5686
• feat: support List and Struct type for KeyValue in inserted_rows.rs by @​wojiaodoubao in #​5713
• feat: support tencent cos by @​ztorchan in #​5740
• feat: add Lance-HF docs to lance.org/integrations/huggingface/ by @​prrao87 in #​5748
• feat(python): support namespace for tensorflow by @​yuqi1129 in #​5750
• feat: add range to External blob by @​wojiaodoubao in #​5765
• feat(java): support json extraction by scanning by @​majin1102 in #​5770
• feat: introduce RowIdSet and RowIdMask by @​yanghua in #​5771
• feat: expose blob handling APIs to python by @​Xuanwo in #​5790
• feat: add blob handling support for fragment by @​Xuanwo in #​5801
• feat: add plan/execute separation to FilteredReadExec by @​LuQQiu in #​5843
• feat: add LSM scanner with point lookup and vector search support by @​touch-of-grey in #​5850
• feat: add rename table implementations to REST namespaces by @​bryanck in #​5874
• feat(python): expose enable_stable_row_ids in commit() by @​fecet in #​5908
• feat: support aggregate in scanner by @​jackye1995 in #​5911
• feat: spill page metadata to disk during IVF shuffle by @​wkalt in #​5921
• feat: add third party licenses lists by @​jackye1995 in #​5922
• feat(java): support session by @​jackye1995 in #​5931
• feat: make geodatafusion/geoarrow optional via geo feature flag by @​apoc in #​5934
• perf: create local writer for efficient local writes by @​wkalt in #​5939
• feat: add python and java binding for aggregate by @​jackye1995 in #​5951
• feat: add proto serialization for FilteredReadExec by @​LuQQiu in #​5954
• feat: create an arrow-scalar crate utilizing arrow-row and arrow-data by @​westonpace in #​5955
• feat: add progress monitoring via callbacks for inverted indexes by @​vivek-bharathan in #​5958
• feat: add size to object store tracing by @​wjones127 in #​5962
• feat: update minimum supported rust version from 1.88 to 1.91 by @​westonpace in #​5964
• feat: add Dataset::with_object_store for request-scoped store overrides by @​wkalt in #​5966
• feat: support namespace as external manifest store by @​jackye1995 in #​5968
• feat: serialize storage options in table identifier proto by @​LuQQiu in #​5973
• feat(core): add Levenshtein-based suggestions to not-found errors in schema by @​HemantSudarshan in #​5976
• feat: add URI-based commit support to Java SDK by @​hamersaw in #​5978
• fix: concurrent read and write to directory namespace by @​jackye1995 in #​5983
• feat: add ability to pass custom headers to objectstore requests by @​hamersaw in #​5989
• feat: add DeleteResult with num_deleted_rows by @​wkalt in #​6001
• feat: introduce IncompatibleTransaction error by @​wjones127 in #​6003
• feat: surface ambiguous merge insert error as InvalidInput by @​wjones127 in #​6048
• feat(blob): distribute blob sidecar keys with reversed binary ids by @​Xuanwo in #​6060
• feat: handle JSONB literals in Lance SQL planner by @​wkalt in #​6061
• feat(java): expose Dataset.dropIndex method to drop specific index by @​fangbo in #​6065
• feat(blob): map external blob URIs to multi-base base ids by @​Xuanwo in #​6066
• feat: add env toggle for repetition index cache on read by @​Xuanwo in #​6069
• feat(compaction): single reserve_fragment_ids after rewriting files by @​hamersaw in #​6072
• feat: expose compaction binary copy configuration through python and java SDKs by @​hamersaw in #​6074
• feat: mark 2.2 as stable and add 2.3 as the next file format version by @​Xuanwo in #​6088
• feat: support prewarm for IVF-based ANN indices by @​wjones127 in #​6090

Bug Fixes 🐛

• fix: ensure blob encoding work when using file reader directly by @​rahil-c in #​5193
• fix: support system columns in dataset.take* operations by @​hamersaw in #​5722
• fix: skip missing indices in compaction rewrite by @​AndreaBozzo in #​5739
• fix(lance-linalg): check fp16kernels feature before arch-specific code by @​durch in #​5747
• refactor: align blob behavior that write via file format version, read via layout by @​Xuanwo in #​5752
• fix: fix deletion when using file-object-store:// by @​cmccabe in #​5760
• fix: remove unreasonable nullable check for data types in hash_joiner during merge operation by @​zhangyue19921010 in #​5784
• fix: allow unused_unsafe for __cpuid to support both stable and nightly by @​jackye1995 in #​5793
• fix: set JUnit dependency as test scope by @​bryanck in #​5815
• fix(java): transaction fatal bug in java transaction api by @​wojiaodoubao in #​5824
• fix: fix remap so that it handles deletions correctly by @​westonpace in #​5828
• fix: inconsistent transposed pq code and metadata when build ivf_pq index distributedly by @​yanghua in #​5834
• fix: improve error messages in FixedSizeListArrayExt::convert_to_floating_point by @​LuciferYang in #​5836
• fix(java): panic when reading CreateIndex transaction by @​majin1102 in #​5853
• fix: fix mini-block dictionary bitpacking panic by @​Xuanwo in #​5860
• fix: fix boolean inline constant decoding by @​Xuanwo in #​5862
• fix: handle NULL elements in LABEL_LIST index results and explain_plan by @​fenfeng9 in #​5867
• fix: open additional storage options provider related apis in lance dataset by @​jackye1995 in #​5869
• fix: flaky test test_ann_prefilter for HNSW by @​BubbleCal in #​5870
• fix(java): init allocator for new dataset when checkout branch/tag by @​fangbo in #​5876
• fix: avoid panic when repdef serializes empty offsets by @​fenfeng9 in #​5890
• fix(java): align version type from i32 to u64 by @​majin1102 in #​5892
• fix: avoid bitmap range panic on inverted bounds by @​fenfeng9 in #​5893
• fix: split index_statistics to reduce rustc query depth by @​Xuanwo in #​5894
• fix: avoid panic on empty list LABEL_LIST filters by @​fenfeng9 in #​5914
• fix: correct OR null semantics for nullable masks by @​fenfeng9 in #​5919
• fix: respect fragment restrictions in vector and FTS searches when requested fragments by @​yingjianwu98 in #​5924
• fix: don't drop field metadata on merge insert path by @​westonpace in #​5927
• fix: apply SchemaAdapter in Updater by @​westonpace in #​5928
• fix: correct OR null handling for BlockList|BlockList by @​fenfeng9 in #​5944
• fix: remove unnecessary column projection for count aggregate by @​jackye1995 in #​5950
• fix: respect requested indexed fragment in vector and FTS searches by @​yingjianwu98 in #​5953
• fix: spawn part load in fts training by @​westonpace in #​5977
• fix: rest namespace integration with table version apis by @​jackye1995 in #​5980
• fix(encoding): handle empty rows in variable packed struct decode by @​Xuanwo in #​5995
• fix: various bugs to namespace access by @​jackye1995 in #​5996
• fix: set namespace commit handler for LanceDataset.commit by @​jackye1995 in #​6002
• fix: fast_search limits full text search to indexed fragments by @​BubbleCal in #​6006
• fix: fast_search should ignore any unindexed data for vector search by @​BubbleCal in #​6007
• fix: correctly calculate max visible level when a list has no def by @​westonpace in #​6008
• perf: avoid oversized variable buffers in full-zip scan batches by @​Xuanwo in #​6013
• fix: make overwrites retryable instead of compatible by @​jackye1995 in #​6014
• fix(python): avoid interpreter shutdown panic in BackgroundExecutor by @​Xuanwo in #​6023
• fix(btree): include null pages in non-IsNull queries for correct thre… by @​wkalt in #​6043
• fix: allowing headers for static configuration to be consistent by @​hamersaw in #​6045
• fix(build): add Android aarch64 support to lance-linalg by @​dardourimohamed in #​6057
• fix: make blob v2 reads base-aware in multi-base datasets by @​Xuanwo in #​6064
• fix(lance-linalg): fix missing return value in u8x16::bit_and for non-x86_64/aarch64 targets by @​cheungxi in #​6068
• fix: resolve Python lint failure on main by @​Xuanwo in #​6073
• fix: restore main CI by formatting take_blob imports by @​Xuanwo in #​6082
• fix: incorrect deletion masking in DatasetPreFilter by @​cijiugechu in #​6083
• fix: avoid thread pool contention between compression and write operations during FTS indexing by @​BubbleCal in #​6085
• fix: compile error for err_express by @​zhangyue19921010 in #​6094

Documentation 📚

• docs: add docs for branch by @​majin1102 in #​5104
• docs: fix issues in HF integration docs by @​prrao87 in #​5778
• docs: fix MkDocs protobuf reference for ConstantLayout by @​Xuanwo in #​5833
• docs(governance): introduce incubating subproject concept and update subproject list by @​jackye1995 in #​5847
• docs: add lance skills as user guide by @​Xuanwo in #​5877
• docs: add array type support by @​XuQianJin-Stars in #​5884
• docs: add FTS docs by @​prrao87 in #​5888
• docs: add schema data types and field IDs documentation by @​dik654 in #​5925
• docs: add lance-trino doc by @​jackye1995 in #​5943
• docs: add new maintainers by @​prrao87 in #​5959
• docs: expand the FTS index doc explaining the training process and multiple partitions by @​westonpace in #​5988
• docs: clarify v2.2 nested drop rollback risk by @​Xuanwo in #​5999
• docs: require data_storage_version=2.2 in map type example by @​Xuanwo in #​6032
• docs: update file versioning matrix for 2.2 rollout by @​Xuanwo in #​6033
• docs: reorganize blob docs around blob v2 and clarify legacy compatibility by @​Xuanwo in #​6034
• docs: align 2.2 encoding docs and nested add-column notes by @​Xuanwo in #​6038
• docs: clarify how to generate TPCH benchmark dataset locally by @​Xuanwo in #​6063

Performance Improvements 🚀

• perf: add vector throughput benchmark by @​westonpace in #​5644
• perf: add a full text search benchmark by @​westonpace in #​5665
• perf: don't concat the batches for writing posting lists by @​BubbleCal in #​5769
• perf: add a lightweight scheduler implementation by @​westonpace in #​5773
• perf: use cpu pool to process all posting lists by @​BubbleCal in #​5780
• perf: calculate cardinality lazily by @​Xuanwo in #​5783
• perf: replace flatmap in build_distance_table by @​wkalt in #​5898
• perf: change Dataset::sample to sort its random indices by @​wkalt in #​5915
• perf: build fp16kernels with NEON support on iOS by @​chyyran in #​5917
• perf: reduce peak memory in nullable training data sampling by @​wkalt in #​5935
• perf: upgrade roaring to 0.11 and improve bitmap/range conversions by @​LuQQiu in #​5961
• perf: speed up format v2.2 scans by adding shortcut for full page by @​Xuanwo in #​5981
• perf: speed up format 2.2 300% by spawning structural decode batch tasks by @​Xuanwo in #​5982
• perf: improve parallelism of data_stats by @​wkalt in #​5990
• perf: reduce peak memory during cosine IVF-PQ index training by @​wkalt in #​6016
• perf: fast rotation for RQ quantization by @​BubbleCal in #​6024
• perf: avoid re-open shard indices and small reads by @​BubbleCal in #​6026
• perf: disable auto FSST for binary fields by @​Xuanwo in #​6047
• perf: speedup flat fts by @​westonpace in #​6054
• perf: add dict-values compression controls with lz4 default by @​Xuanwo in #​6059
• perf: avoid frequent allocating when computing residual vectors by @​BubbleCal in #​6062
• perf: add take_blob benchmark with cache_repetition_index matrix by @​Xuanwo in #​6067

Other Changes

• refactor: change reader's get_range result to be a static future by @​westonpace in #​5755
• refactor(python): migrate torch.jit.script to torch.compile by @​wjones127 in #​5759
• test: fix tests broken by pandas 3 release by @​westonpace in #​5786
• refactor: use dict entries and encoded size instead of cardinality for dict decision by @​Xuanwo in #​5891
• refactor: correct panic message typos in OrderableScalarValue::cmp by @​fenfeng9 in #​5913
• refactor: deprecate list_indices and migrate tests to describe_indices by @​fenfeng9 in #​5945
• refactor: upgrade to SNAFU 0.9 by @​shepmaster in #​6071

Full Changelog: lance-format/lance@release-root/3.0.0-beta.N...v3.0.0

---

Configuration

📅 Schedule: Branch creation - Between 12:00 AM and 03:59 AM, only on Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[robert3005]

This requires rust 1.91.0

[codspeed-hq]

Merging this PR will improve performance by 18.36%

⚡ 2 improved benchmarks
✅ 1014 untouched benchmarks
⏩ 1522 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	bitwise_not_vortex_buffer_mut[128]	376.1 ns	317.8 ns	+18.36%
⚡	Simulation	bitwise_not_vortex_buffer_mut[1024]	535.6 ns	477.2 ns	+12.22%

---

_{Comparing renovate/lance-encoding-3.x (3e39cce) with develop (7efebc4)}

Footnotes

1. 1522 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[renovate]

Renovate Ignore Notification

Because you closed this PR without merging, Renovate will ignore this update. You will not get PRs for any future 3.x releases. But if you manually upgrade to 3.x then Renovate will re-enable minor and patch updates automatically.

If you accidentally closed this PR, or if you changed your mind: rename this PR to get a fresh replacement PR.