Draft
Conversation
github-actions
bot
removed
the
action/benchmark-sql
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: FineWeb NVMeSummary
Detailed Results Table
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: FineWeb S3Summary
Detailed Results Table
|
Contributor
Benchmarks: Statistical and Population GeneticsSummary
Detailed Results Table
|
Contributor
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Contributor
Benchmarks: Clickbench on NVMESummary
Detailed Results Table
|
github-actions
bot
removed
the
action/benchmark-sql
github-actions
bot
removed
the
action/benchmark-sql
joseph-isaacs
added
the
action/benchmark
github-actions
bot
removed
the
action/benchmark
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Random AccessSummary
|
| if: matrix.remote_storage == null || github.event.pull_request.head.repo.fork == true | ||
| shell: bash | ||
| env: | ||
| VORTEX_USE_SCAN_API: "1" |
Contributor
There was a problem hiding this comment.
this should be 0 or remove the old one?
| pub struct LayoutReaderDataSource { | ||
| reader: LayoutReaderRef, | ||
| session: VortexSession, | ||
| split_size: u64, |
Contributor
Benchmarks: CompressionSummary
Detailed Results Table
|
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Deploying vortex-bench with
|
| Latest commit: |
7aaf8b6
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://b9e83af8.vortex-93b.pages.dev |
| Branch Preview URL: | https://ngates-scan-api.vortex-93b.pages.dev |
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn
added
action/benchmark-sql
github-actions
bot
removed
the
action/benchmark-sql
Comment on lines
+326
to
+381
| loop { | ||
| // Try to pull from the current child's split stream. | ||
| if let Some(ref mut child_stream) = current_stream { | ||
| match child_stream.next().await { | ||
| Some(Ok(split)) => { | ||
| if let Some(ref mut s) = state | ||
| && let Some(ref mut limit) = s.remaining_limit | ||
| { | ||
| let est = split.row_count_estimate(); | ||
| *limit = limit.saturating_sub(est.upper.unwrap_or(est.lower)); | ||
| } | ||
| return Some((Ok(split), (state, current_stream))); | ||
| } | ||
| Some(Err(e)) => { | ||
| return Some((Err(e), (None, None))); | ||
| } | ||
| None => { | ||
| // Current child exhausted, move to next. | ||
| drop(current_stream.take()); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| let s = state.as_mut()?; | ||
|
|
||
| if s.remaining_limit.is_some_and(|l| l == 0) { | ||
| return None; | ||
| } | ||
|
|
||
| // Get the next data source. | ||
| let source = match s.next_source().await { | ||
| Ok(Some(source)) => source, | ||
| Ok(None) => return None, | ||
| Err(e) => return Some((Err(e), (None, None))), | ||
| }; | ||
|
|
||
| if source.dtype() != &s.dtype { | ||
| return Some(( | ||
| Err(vortex_err!( | ||
| "MultiDataSource dtype mismatch: expected {}, got {}", | ||
| s.dtype, | ||
| source.dtype() | ||
| )), | ||
| (None, None), | ||
| )); | ||
| } | ||
|
|
||
| let mut child_request = s.request.clone(); | ||
| child_request.limit = s.remaining_limit; | ||
| let child_scan = match source.scan(child_request) { | ||
| Ok(scan) => scan, | ||
| Err(e) => return Some((Err(e), (None, None))), | ||
| }; | ||
|
|
||
| current_stream = Some(child_scan.splits()); | ||
| } |
Contributor
There was a problem hiding this comment.
This can be extracted into a helper method that takes &mut current_stream, then you can do:
impl DataSourceScan for MultiDataSourceScan {
fn splits(self: Box<Self>) -> SplitStream {
stream::unfold(
(*self, None::<SplitStream>),
|(mut scan, mut current_stream)| async move {
let result = scan.next_split(&mut current_stream).await?;
Some((result, (scan, current_stream)))
},
)
.boxed()
}
}Doesn't seem like you need it to be Some(*self)
| _ => Precision::Absent, | ||
| }; | ||
|
|
||
| let byte_size_est = self.data_source.row_count_estimate(); |
Contributor
There was a problem hiding this comment.
This is probably wrong?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment to use the Scan API from DuckDB and DataFusion integrations.