feat: optimize system table queries with column projection (DRIVER-368)#862
Draft
nikagra wants to merge 4 commits intoscylladb:scylla-3.xfrom
Draft
feat: optimize system table queries with column projection (DRIVER-368)#862nikagra wants to merge 4 commits intoscylladb:scylla-3.xfrom
nikagra wants to merge 4 commits intoscylladb:scylla-3.xfrom
Conversation
On first connection, issue SELECT * to discover which columns the server exposes, then cache the intersection with the driver's column-of-interest sets. Subsequent queries project only those columns instead of SELECT *, reducing network and parsing overhead. Three *_COLUMNS_OF_INTEREST constants enumerate every column the driver reads from system.local, system.peers, and system.peers_v2. Caches are reset to null (the uninitialized sentinel) on every new connection so that reconnecting to a different server re-discovers its schema. Unit tests verify the constants, the intersectWithNeeded() helper, the buildProjectedQuery() helper, and the cache field declarations.
|
@nikagra please check why cicd is failing |
After cluster.init() the column caches are warm, so subsequent fetchNodeInfo() / selectPeersFuture() calls send projected SELECT queries instead of SELECT *. Scassandra matches primes by exact query string, so the projected variants were returning no results, causing the two failing ITs. Fix: add projected-query primes alongside every existing SELECT * prime in ScassandraCluster.primeNode(), primeSystemLocalRow(), and ControlConnectionTest.should_fetch_whole_peers_table_if_broadcast_address_changed().
…eared clearAllPrimes() wipes the projected-query primes set by primeNode(), so any query the driver sends using its warm column cache hits an unprimed (and therefore empty) Scassandra response. The simplest fix is to reset the caches to null just before clearing primes so the driver falls back to SELECT * — which the test already primes.
When fetchNodeInfo sends a WHERE-clause query (SELECT * FROM system.peers WHERE peer='<IP>') and gets an empty result set, do not populate the peersColumns/peersV2Columns cache from that result. If the cache were populated at this point, the subsequent fallback full-scan via selectPeersFuture() would send a projected query (not SELECT *) — which would fail to match SELECT * primes in Scassandra (or any server that only has the broad SELECT * query primed). By deferring cache population to the row-found branch, the full-scan always sends SELECT * when the cache was cold, and selectPeersFuture's own onSuccess callback populates the cache from the full-scan result. Fixes ControlConnectionTest.should_fetch_whole_peers_table_if_broadcast_address_changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes https://scylladb.atlassian.net/browse/DRIVER-368
Backport of the same optimization done for the 4.x driver (DefaultTopologyMonitor), applied here to ControlConnection in the 3.x driver.
On the first connection to each system table (system.local, system.peers, system.peers_v2) the driver issues SELECT * to discover which columns the server exposes. It caches the intersection of those columns with a driver-internal *_COLUMNS_OF_INTEREST set. Subsequent queries project only those columns, reducing bytes on the wire and deserialization work.
Changes:
All 16 unit tests pass. Clean compile.