Reduce ExtractLeafExpressions optimizer overhead with fast pre-scan#20341
Merged
adriangb merged 3 commits intoapache:mainfrom Feb 13, 2026
Merged
Reduce ExtractLeafExpressions optimizer overhead with fast pre-scan#20341adriangb merged 3 commits intoapache:mainfrom
adriangb merged 3 commits intoapache:mainfrom
Conversation
github-actions
bot
added
optimizer
Contributor
Author
|
run benchmark sql_planner |
|
🤖 |
github-actions
bot
added
documentation
Add a `has_extractable_expr()` pre-check that short-circuits before any expensive allocations (column HashSets, extractors, expression rewrites) when a plan node has no `MoveTowardsLeafNodes` expressions. Benchmarking shows this reduces overhead on non-struct queries from 5-31% down to 0-3%: - physical_select_aggregates_from_200: +31% → +4% - physical_many_self_joins: +13% → +2% - physical_join_consider_sort: +13% → +1% - physical_plan_tpch_all: +5% → +2% - physical_plan_tpcds_all: +6% → +2% Also adds `enable_leaf_expression_pushdown` config option to allow disabling the optimization entirely. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
adriangb
force-pushed
the
optimize-leaf-expression-prescan
branch
from
be650a4 to
0eb90b6
Compare
|
🤖: Benchmark completed Details
|
alamb
approved these changes
Feb 13, 2026
| /// Default: true | ||
| pub enable_sort_pushdown: bool, default = true | ||
|
|
||
| /// When set to true, the optimizer will extract leaf expressions |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #20117 which added the
ExtractLeafExpressionsandPushDownLeafProjectionsoptimizer rules for get_field pushdown.Benchmarking revealed that these rules added 5-31% overhead on all queries (including those with no struct/get_field expressions) because they unconditionally allocated column HashSets, extractors, and walked every expression tree for every Filter/Sort/Limit/Aggregate/Join node.
This PR adds:
has_extractable_expr()pre-scan: A lightweight check usingExpr::exists()that short-circuits before any expensive allocations when noMoveTowardsLeafNodesexpressions are presentdatafusion.optimizer.enable_leaf_expression_pushdownto disable the rules entirelyBenchmark Results (vs no-rules baseline)
Test plan
extract_leaf_expressionsunit tests passcargo bench -p datafusion --bench sql_planner🤖 Generated with Claude Code