chore: migrate code from googleapis/python-datastore

[chalmerlowe]

• docs(samples): add samples for IN, NOT_IN, and != operators. ( Complete Python Sample Code for Logging API #312)
• fix: require python 3.7+ (Update compute metadata hanging get example to use "wait_for_change". #332)
• chore(deps): update all dependencies (Updating DNS samples to use newer gcloud features. #340)
• docs: Move the schedule_export samples from python-docs-samples ([Speech] speech_streaming.py code does not worl #344)
• docs(samples): Add an example of using read_time in queries and get() (Adding test for customer-supplied disk encryption key. Fixes #281 #342)
• chore(deps): update dependency google-cloud-datastore to v2.8.1 (TRAVIS_COMMIT_RANGE isn't set on non-PRs #348)
• chore(deps): update dependency pytest to v7.1.3 (Transfer service: http to gs bucket #359)
• chore: detect samples tests in nested directories (update managed_vm naming to flexible everywhere #360)
• samples: Update the read_time snippet. (cloud logging python client docs link 404 #363)
• chore(deps): update dependency google-cloud-datastore to v2.8.2 (Adding error handlers to managed vms hello world samples. #369)
• chore(deps): update dependency backoff to v2.2.1 (Add Bigtable hello world sample. #371)
• chore(deps): update dependency google-cloud-datastore to v2.8.3 (Logging Fixups #375)
• chore(deps): update dependency google-cloud-datastore to v2.9.0 (Add region tags to bigtable/hello sample. #376)
• chore(deps): update dependency pytest to v7.2.0 (Close HappyBase connection in bigtable/hello sample. #377)
• chore(deps): update dependency google-cloud-datastore to v2.10.0 (Added script that manages a set of rolling backups. #381)
• chore(python): drop flake8-import-order in samples noxfile (Update sendgrid samples #387)
• chore(deps): update dependency google-cloud-datastore to v2.11.0 (Don't decrypt on PRs #389)
• samples: Add snippets and samples for Count query (bigtable: Move hello to hello_happybase. #383)
• chore(deps): update dependency google-cloud-datastore to v2.11.1 (Fixing job id parameter, fixes #393 #394)
• chore(python): add support for python 3.11 (Mark tests that require creds #395)
• chore(deps): update dependency google-cloud-datastore to v2.12.0 (Changing cloud speech code samples to work with v1beta1 #399)
• chore(deps): update dependency pytest to v7.2.1 (speech_streaming tests do not pass #403)
• chore(deps): update dependency google-cloud-datastore to v2.13.0 (Adds option to set the file format for export jobs #405)
• chore(deps): update dependency google-cloud-datastore to v2.13.1 (Update requirements #409)
• chore(deps): update dependency google-cloud-datastore to v2.13.2 (Speech beta #411)
• chore(deps): update dependency google-cloud-datastore to v2.14.0 (Readme fix #423)
• chore(deps): update dependency pytest to v7.2.2 (Speech API not working for some files #424)
• chore(deps): update dependency google-cloud-datastore to v2.15.0 (Nl #426)
• chore(deps): update dependency google-cloud-datastore to v2.15.1 (Auth samples for Endpoints. #431)
• chore(deps): update dependency pytest to v7.3.1 (Fix local unit test sample #433)
• chore(deps): update dependency google-cloud-datastore to v2.15.2 (Change the way nox handles installing requirements. #438)
• chore(deps): update dependency pytest to v7.3.2 (Document usage of useLegacySQL parameter in BigQuery samples. #445)
• chore(deps): update all dependencies (Testing hooks #449)
• chore(deps): update dependency google-cloud-datastore to v2.16.1 (Remove output only fields #454)
• chore(deps): update dependency google-cloud-datastore to v2.17.0 (Add workaround for travis commit being wrong. #469)
• chore(deps): update all dependencies (Auto-update dependencies. #473)
• chore(deps): update all dependencies (How to specify quotaUser upon Speech Streaming request #475)
• chore(deps): update all dependencies (add needed exclusion tags for docs. #483)
• samples: Add snippets for sum and avg (Add pubsub iam samples #480)
• chore(deps): update all dependencies (Auto-update dependencies. #493)
• feat: Add support for Python 3.12 (Speech Streaming Issue #498)
• chore(deps): update dependency google-cloud-datastore to v2.19.0 (Stackdriver product rename #508)
• chore(deps): update dependency pytest to v7.4.4 (Auto-update dependencies. #511)
• feat: implement query profiling (Auto-update dependencies. #542)
• chore(deps): update all dependencies (Update shortlinks. #519)
• chore(deps): update all dependencies (Update App Engine Flexible samples for beta 2 #563)
• chore(python): update dependencies in .kokoro/docker/docs (Translate samples #574)
• chore(python): Add support for Python 3.14 (missing sample for ndb in appengine/flexible #644)
• chore(deps): update all dependencies (add grpc bookstore python samples #660)
• chore(deps): update all dependencies ('Issuing an asynchronous request' needs a tip for adding protorpc library #670)

Description

Fixes #

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

• I have followed Sample Guidelines from AUTHORING_GUIDE.MD
• README is updated to include all relevant information
• Tests pass: nox -s py-3.9 (see Test Environment Setup)
• Lint pass: nox -s lint (see Test Environment Setup)
• These samples need a new API enabled in testing projects to pass (let us know which ones)
• These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
• This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample
• This sample adds a new Product API, and I updated the Blunderbuss issue/PR auto-assigner with the codeowners for this sample
• Please merge this PR for me once it is approved

[snippet-bot]

Here is the summary of changes.

You are about to add 19 region tags.

• datastore/samples/snippets/snippets.py:23, tag datastore_size_coloration_query
• datastore/samples/snippets/snippets.py:35, tag datastore_in_query
• datastore/samples/snippets/snippets.py:44, tag datastore_not_equals_query
• datastore/samples/snippets/snippets.py:53, tag datastore_not_in_query
• datastore/samples/snippets/snippets.py:62, tag datastore_stale_read
• datastore/samples/snippets/snippets.py:82, tag datastore_count_in_transaction
• datastore/samples/snippets/snippets.py:114, tag datastore_count_on_kind
• datastore/samples/snippets/snippets.py:131, tag datastore_count_with_limit
• datastore/samples/snippets/snippets.py:149, tag datastore_count_with_property_filter
• datastore/samples/snippets/snippets.py:190, tag datastore_count_query_with_stale_read
• datastore/samples/snippets/snippets.py:234, tag datastore_sum_aggregation_query_on_kind
• datastore/samples/snippets/snippets.py:260, tag datastore_sum_aggregation_query_with_filters
• datastore/samples/snippets/snippets.py:296, tag datastore_avg_aggregation_query_on_kind
• datastore/samples/snippets/snippets.py:322, tag datastore_avg_aggregation_query_with_filters
• datastore/samples/snippets/snippets.py:358, tag datastore_multiple_aggregation_in_structured_query
• datastore/samples/snippets/snippets.py:397, tag datastore_query_explain_analyze_entity
• datastore/samples/snippets/snippets.py:425, tag datastore_query_explain_entity
• datastore/samples/snippets/snippets.py:440, tag datastore_query_explain_analyze_aggregation
• datastore/samples/snippets/snippets.py:469, tag datastore_query_explain_aggregation

---

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

• Refresh this comment

[gemini-code-assist]

Summary of Changes

Hello @chalmerlowe, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates a comprehensive suite of Google Cloud Datastore samples and their supporting development environment. It significantly enhances the project's sample coverage by introducing new query capabilities, aggregation functions, and query profiling tools. The changes also establish a robust and modern testing framework using nox, ensuring better code quality and maintainability, while extending compatibility to recent Python versions and updating key dependencies.

Highlights

• Code Migration: Migrated a significant set of Google Cloud Datastore samples and their associated development infrastructure from the googleapis/python-datastore repository into this project.
• New Development and Testing Setup: Introduced nox for managing development workflows, including linting, code formatting with Black and Isort, and running tests across various Python versions. This includes dedicated noxfile.py and noxfile_config.py files for both the main snippets and the scheduled export sample.
• Expanded Datastore Query Samples: Added new code snippets demonstrating advanced Datastore query operators such as IN, NOT_IN, and !=. Also included examples for using read_time in queries and get() operations for stale reads.
• New Aggregation Query Samples: Implemented samples for various aggregation queries, including Count, Sum, and Avg aggregations, with examples for use within transactions, with limits, and with property filters.
• Query Profiling (Explain/Analyze): Introduced new features and samples for query profiling, allowing users to explain and analyze both entity and aggregation queries to understand their execution plan and statistics.
• Scheduled Datastore Exports Sample: Added a new sample demonstrating how to schedule Datastore exports using a Cloud Function triggered by Cloud Scheduler, complete with its own README.md, main.py implementation, and tests.
• Python Version and Dependency Updates: Updated the supported Python versions to require 3.7+ and added support for Python 3.11, 3.12, and 3.14. All project dependencies, including google-cloud-datastore, pytest, and backoff, were updated to their latest versions.

Changelog

• datastore/samples/snippets/noxfile.py
  • Added a new nox configuration file to define linting, formatting, and testing sessions for the snippets.
• datastore/samples/snippets/noxfile_config.py
  • Added a new configuration override file for nox, allowing customization of test parameters.
• datastore/samples/snippets/requirements-test.txt
  • Added a new file specifying testing dependencies with version constraints for different Python versions.
• datastore/samples/snippets/requirements.txt
  • Added a new file specifying the core google-cloud-datastore dependency.
• datastore/samples/snippets/schedule-export/README.md
  • Added documentation for the scheduled Datastore export sample.
• datastore/samples/snippets/schedule-export/main.py
  • Added the Cloud Function implementation for triggering Datastore exports.
• datastore/samples/snippets/schedule-export/noxfile.py
  • Added a nox configuration file specific to the scheduled export sample.
• datastore/samples/snippets/schedule-export/noxfile_config.py
  • Added a nox configuration override file specific to the scheduled export sample.
• datastore/samples/snippets/schedule-export/requirements-test.txt
  • Added testing dependencies for the scheduled export sample.
• datastore/samples/snippets/schedule-export/requirements.txt
  • Added the core google-cloud-datastore dependency for the scheduled export sample.
• datastore/samples/snippets/schedule-export/schedule_export_test.py
  • Added unit tests for the datastore_export Cloud Function.
• datastore/samples/snippets/snippets.py
  • Added a new file containing various Datastore query, aggregation, and profiling code examples.
• datastore/samples/snippets/snippets_test.py
  • Added tests for the Datastore query, aggregation, and profiling snippets, including index setup.

Activity

• The pull request title indicates a migration of code from googleapis/python-datastore.
• The description lists numerous commits, primarily focusing on dependency updates and the addition of new samples, suggesting a consolidation of recent development efforts.
• New files were added across multiple directories, establishing a new structure for Datastore samples and their testing infrastructure.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request migrates a significant amount of code and samples for python-datastore. The changes are extensive, introducing many new files for snippets and tests. My review focuses on ensuring the new code is correct, maintainable, and follows best practices. I've identified several issues, including potential runtime errors in the noxfile.py and Cloud Function main.py, test-related logic that has leaked into user-facing sample code, and a broken main function in snippets.py. I've provided suggestions to fix these issues.

[gemini-code-assist]

The packages variable is used on line 218 but is only initialized if requirements.txt exists. If requirements.txt is not present but requirements-test.txt is, this will cause an UnboundLocalError. To prevent this, packages should be initialized to an empty string before the conditional blocks.

Suggested change

	concurrent_args = []
	if os.path.exists("requirements.txt"):
	if os.path.exists("constraints.txt"):
	session.install("-r", "requirements.txt", "-c", "constraints.txt")
	else:
	session.install("-r", "requirements.txt")
	with open("requirements.txt") as rfile:
	packages = rfile.read()
	if os.path.exists("requirements-test.txt"):
	if os.path.exists("constraints-test.txt"):
	session.install(
	"-r", "requirements-test.txt", "-c", "constraints-test.txt"
	)
	else:
	session.install("-r", "requirements-test.txt")
	with open("requirements-test.txt") as rtfile:
	packages += rtfile.read()
	concurrent_args = []
	packages = ""
	if os.path.exists("requirements.txt"):
	if os.path.exists("constraints.txt"):
	session.install("-r", "requirements.txt", "-c", "constraints.txt")
	else:
	session.install("-r", "requirements.txt")
	with open("requirements.txt") as rfile:
	packages = rfile.read()
	if os.path.exists("requirements-test.txt"):
	if os.path.exists("constraints-test.txt"):
	session.install(
	"-r", "requirements-test.txt", "-c", "constraints-test.txt"
	)
	else:
	session.install("-r", "requirements-test.txt")
	with open("requirements-test.txt") as rtfile:
	packages += rtfile.read()

[gemini-code-assist]

These lines appear to be test setup logic to ensure the database is empty before the snippet runs. Sample code for users should not contain such setup or cleanup logic. This should be handled by the test code that calls the snippet.

[gemini-code-assist]

Using os.environ.get("GCP_PROJECT") will result in project_id being None if the environment variable is not set, which will cause a failure later in the function. It's better to fail early with a clear error message if this required environment variable is missing. Accessing it directly with os.environ["GCP_PROJECT"] will raise a KeyError if it's not set, which is a more robust way to handle required environment variables.

Suggested change

	project_id = os.environ.get("GCP_PROJECT")
	project_id = os.environ["GCP_PROJECT"]

[gemini-code-assist]

Directly accessing json_data['bucket'] will raise a KeyError if 'bucket' is not in the payload, causing the function to fail with an unhandled exception. It's safer to use .get() and validate that the bucket is provided.

[gemini-code-assist]

The loop over globals().items() will attempt to call imported items like pprint as if they were snippet functions, which will raise a TypeError. The list of names to exclude is incomplete and also contains defaultdict, which is not imported in this file.

Suggested change

	if name in (
	"main",
	"_preamble",
	"defaultdict",
	"datetime",
	"timezone",
	"timedelta",
	) or not callable(function):
	if name in (
	"main",
	"_preamble",
	"pprint",
	"datetime",
	"timezone",
	"timedelta",
	) or not callable(function):

[gemini-code-assist]

Constructing JSON strings with string formatting can be error-prone and hard to read. Using json.dumps() is a safer and more idiomatic way to create JSON strings in Python. You'll need to add import json at the top of the file.

Suggested change

	json_string = '{{ "bucket": "{bucket}", "kinds": "{kinds}", "namespaceIds": "{namespaceIds}" }}'.format(
	bucket=bucket, kinds=kinds, namespaceIds=namespaceIds
	)
	json_string = json.dumps(
	{"bucket": bucket, "kinds": kinds, "namespaceIds": namespaceIds}
	)

[gemini-code-assist]

The out and err variables are captured from capsys but are never used. If you don't intend to make assertions on the captured output, this line can be removed.

[gemini-code-assist]

The docstring for _session_tests is misplaced. It should be located immediately after the function signature on line 188 to follow standard Python conventions.

[gemini-code-assist]

Constructing JSON strings with string formatting can be error-prone and hard to read. Using json.dumps() is a safer and more idiomatic way to create JSON strings in Python. You'll need to add import json at the top of the file.

Suggested change

	json_string = '{{ "bucket": "{bucket}" }}'.format(bucket=bucket)
	json_string = json.dumps({"bucket": bucket})

[gemini-code-assist]

The docstring for _session_tests is misplaced. It should be located immediately after the function signature on line 188 to follow standard Python conventions.

[chalmerlowe]

@iennae

It looks like at least one of the tests is failing because we are performing an exact string match on the repr() of a list containing a dictionary. Since dictionary key order in string representations can be non-deterministic, this makes the assertion brittle even if the data itself is correct.

To fix the flakiness while maintaining clarity, we could consider one of the following approaches:

Option 1: Assert on the data, not the string (Recommended)

We can modify the sample to return the plan_summary object. This allows us to verify the data structure directly in the test. Python’s == operator for dictionaries is order-independent, which makes the test much more robust.

In the sample code:

    # ...
    plan_summary = iterator.explain_metrics.plan_summary
    print(f"Indexes used: {plan_summary.indexes_used}")
    return plan_summary # Return for testing purposes

In the test:

    plan_summary = snippets.explain_aggregation(client)
    expected_indexes = [{'properties': '(__name__ ASC)', 'query_scope': 'Collection group'}]
    assert plan_summary.indexes_used == expected_indexes

Option 2: Canonicalize the output using JSON

If we prefer to keep the assertion strictly against the printed output, we can use json.dumps with sort_keys=True in the sample. This ensures the keys always appear in the same order in stdout, regardless of how the dictionary was initialized.

In the sample code:

    import json
    # ...
    print(f"Indexes used: {json.dumps(plan_summary.indexes_used, sort_keys=True)}")

In the test:

    # Note: JSON uses double quotes
    assert 'Indexes used: [{"properties": "(__name__ ASC)", "query_scope": "Collection group"}]' in captured.out

What do we think? Option 1 feels a bit cleaner as it avoids adding a json import to a simple snippet, but Option 2 keeps the test focused entirely on the capsys output if that is our preferred pattern for snippets.

[chalmerlowe]

@iennae two other failing tests are not familiar to me:

• In the 3.14 CI/CD check we get this error:

404 Project 'python-docs-samples-tests-314' does not exist.

I am not sure on the correct steps for resolving that.

• In the 3.8 CI/CD check we get an error where nox just goes belly up. Not sure if you have seen that before.