feat: lazy imports google auth (requests/urllib3)

[hebaalazzeh]

This PR implements PEP 0810 explicit lazy imports in google-auth transports (requests and urllib3).
On Python 3.15+, this defers loading of heavy third-party networking libraries (requests and urllib3) to reduce serverless cold-start latency and peak memory usage. On Python 3.14 and below, this falls back safely to standard eager loading with zero backwards-compatibility risk.

Related Links

• Design Doc: go/sdk:python-lazy-loading
• GAPIC Implementation PR: feat: implement native PEP 0810 lazy loading #17591

[gemini-code-assist]

Code Review

This pull request introduces a __lazy_modules__ set in packages/google-auth/google/auth/transport/__init__.py to support lazy loading of transport modules. The review feedback correctly identifies an incorrect module name (_aiohttp_requests instead of aiohttp_requests) that would prevent lazy importing in Python 3.15+.

[parthea]

LGTM but holding off on formal approval until we have tests. We should assert that on Python 3.15+ the modules are indeed absent from sys.modules until accessed, and that on pre-3.15 environments they fallback cleanly to standard eager imports

import sys
import pytest

# List of modules we expect to be lazy
LAZY_MODULES = [
    "google.auth.transport.requests",
    "google.auth.transport.urllib3",
    "google.auth.transport.grpc",
]

def clean_sys_modules():
    """Helper to ensure we start with a clean slate for import testing."""
    for mod in LAZY_MODULES:
        sys.modules.pop(mod, None)

@pytest.mark.skipif(sys.version_info < (3, 15), reason="PEP 810 requires Python 3.15+")
def test_lazy_imports_on_python_315():
    clean_sys_modules()
    
    # 1. Import the transport package
    import google.auth.transport
    
    # 2. Assert that none of the lazy modules have been eagerly loaded into sys.modules
    for mod in LAZY_MODULES:
        assert mod not in sys.modules
        
    # 3. Access an attribute to trigger reification
    from google.auth.transport import requests
    _ = requests.__name__  # Trigger first-use reification
    
    # 4. Assert that the module has now been reified and loaded
    assert "google.auth.transport.requests" in sys.modules


@pytest.mark.skipif(sys.version_info >= (3, 15), reason="Testing fallback behavior on < 3.15")
def test_fallback_eager_imports_pre_315():
    clean_sys_modules()
    
    # On older Python, __lazy_modules__ is safely ignored, meaning they should eager-load
    import google.auth.transport
    
    for mod in LAZY_MODULES:
        assert mod in sys.modules

[parthea]

I see the tests are failing. We also need to add from google.auth.transport import requests to the test file

[parthea]

Disregard my last comment.

Under PEP 810, __lazy_modules__ is a module-local opt-in . It only intercepts import statements written inside the module where it is defined

We should only add __lazy_modules__ to files where we also have the same import statements

From https://docs.python.org/3.15/reference/simple_stmts.html#lazy-imports

Any regular (non-lazy) import statement at module scope whose target appears in lazy_modules is treated as a lazy import, exactly as if the lazy keyword had been used.

[hebaalazzeh]

addressed all comments

[parthea]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces lazy import support for Python 3.15+ by defining __lazy_modules__ in several transport modules (grpc.py, requests.py, and urllib3.py) and adding corresponding integration tests. However, the feedback highlights that the lazy loading mechanism is currently defeated in grpc.py due to inheriting from grpc.AuthMetadataPlugin at the module level, and in urllib3.py due to a module-level version check on urllib3.__version__.

[parthea]

Please can you address the feedback from GCA?

[hebaalazzeh]

\gemini

[parthea]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces lazy loading support for heavy transport modules (grpc, requests, and urllib3) to optimize import times, particularly targeting Python 3.15+ (PEP 810), with fallback behavior for older versions. Key changes include defining __lazy_modules__, decoupling classes from eager parent imports, and adding a test suite to verify the lazy loading behavior. Feedback focuses on optimizing a performance bottleneck in urllib3.py's dynamic attribute resolution, addressing a bypassed ImportError check in requests.py due to deferred imports, and adding explanatory comments in grpc.py to prevent accidental regressions.

[hebaalazzeh]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for PEP 0810 (Explicit Lazy Imports) in Python 3.15+ across the grpc, requests, and urllib3 transports. This is achieved by defining lazy_modules, removing eager base class inheritance, and lazily resolving attributes (such as RequestMethods in AuthorizedHttp). A test suite is also added to verify this behavior. The reviewer suggests improving the getattr implementation in AuthorizedHttp by dynamically checking for attributes on the underlying RequestMethods class rather than hardcoding specific method names, and lazily importing version to avoid eagerly loading the packaging module.

[parthea]

I updated the PR title to feat: lazy imports for google/auth/transport/requests.py as I only see changes for requests, not grpc or urllib3

[hebaalazzeh]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces lazy loading of heavy dependencies (such as requests and urllib3) at import time to improve performance. It implements a LazyBasesMeta metaclass and a HeapDummy class to dynamically resolve and swap base classes upon instantiation, subclassing, or attribute inspection. Feedback was provided on _resolve_bases in LazyBasesMeta, which is susceptible to infinite recursion and race conditions in multi-threaded environments if attribute lookups occur during base resolution. A thread-local tracking mechanism was suggested to prevent re-entrant recursion.

[daniel-sanche]

Do these changes really makes sense? IIRC, neither of these modules are loaded by default, they have to be imported explicitly. And they are both designed to fail fast by raising an exception if the libraries they depend on are missing. Does it even make sense to add lazy loading to something with these access patterns?

Even if there's a good argument there, I'd still be pretty skeptical about this kind of change. google-auth is a key foundational library, so I don't want to be adding complex metaclass indirection unless there are overwhelming benefits. Have you done any benchmarking to justify the change?

[daniel-sanche]

• Isn't this class duplicated in urllub3?
• Does the name need to be so similar to the one on _helpers?
• Why is this needed? (docstrings should be added if adding the class is justified)

[hebaalazzeh]

While both classes share the name _LazyBasesMeta and inherit from _helpers.LazyBasesMeta, they are not duplicates because their base resolution logic is transport-specific:

In requests.py, it resolves to requests.adapters.HTTPAdapter or requests.Session.

In urllib3.py, it resolves to urllib3._request_methods.RequestMethods or urllib3.request.RequestMethods.
To prevent requests and urllib3 modules from loading at import time, we cannot inherit directly from their classes (like requests.Session or urllib3.request.RequestMethods) during class definition time. We use LazyBasesMeta to inherit from HeapDummy initially, and then dynamically swap the base classes at runtime on first instantiation, subclassing, or attribute access.

The name _LazyBasesMeta is a private, module-specific subclass of _helpers.LazyBasesMeta which highlights its inheritance. I have added class-level docstrings to both files explaining this base class swapping mechanism.

[daniel-sanche]

Why was this changed? This would be a breaking change, that contradicts the docstring: If ``None``, a timeout error is never raised.

[hebaalazzeh]

The docstring comment "If None, a timeout error is never raised" refers to the timeout argument (the first argument), not timeout_error_type (the second argument). If timeout itself is None, the guard returns early, maintaining identical behavior.

The signature default value was changed from requests.exceptions.Timeout to None for the timeout_error_type argument to prevent eager evaluation of requests.exceptions at module import time.

Inside the constructor, if timeout_error_type is None, it is resolved to requests.exceptions.Timeout. If timeout itself is None, the guard returns early, maintaining identical behavior. This is a standard Python pattern to defer default evaluations.

[hebaalazzeh]

@daniel-sanche

Do these changes really makes sense? IIRC, neither of these modules are loaded by default, they have to be imported explicitly. And they are both designed to fail fast by raising an exception if the libraries they depend on are missing. Does it even make sense to add lazy loading to something with these access patterns?

Even if there's a good argument there, I'd still be pretty skeptical about this kind of change. google-auth is a key foundational library, so I don't want to be adding complex metaclass indirection unless there are overwhelming benefits. Have you done any benchmarking to justify the change?

General Response / Benchmarking Justification

Comment:

Does it even make sense to add lazy loading to something with these access patterns? Have you done any benchmarking to justify the change?

Response:

Yes, it is justified. While transport modules are not loaded by default under clean conditions, they are implicitly imported in two major ways:

• Application Default Credentials (ADC): google.auth.default() queries GCE metadata via _get_gce_credentials(), which imports google.auth.transport.requests.
• Client Library Imports: Generated REST/HTTP transports for all Google Cloud client libraries (e.g., Secret Manager, Redis, Workstations) import AuthorizedSession from google.auth.transport.requests at the module level.
• gRPC Clients Import REST Transports: The main client classes (e.g., SecretManagerServiceClient) import both gRPC and REST transport modules at module-level in their client.py files. Consequently, simply importing a gRPC client eagerly loads requests and all its dependencies, even if the user never makes a REST request.

Here are the benchmark results using the import profiling tool (scripts/import_profiler/profiler.py) over 10 cold-start iterations on Python 3.15.0b2 (pinning processes to core 0 and clearing bytecode caches before each run):

Profile for google.auth.transport.requests

Metric	Eager Imports (Normal)	Lazy Imports (PR Branch)	Change
P50 (Median) Time	2,278.46 ms	1,287.93 ms	-990.53 ms (-43.5%)
Mean Time	2,293.88 ms	1,292.16 ms	-1,001.72 ms (-43.7%)
Loaded Modules	395	176	-219 modules (-55.4%)
Loaded Source Lines	140,500	65,018	-75,482 lines (-53.7%)
Physical RSS RAM	45.68 MB	36.14 MB	-9.54 MB (-20.9%)

By deferring these heavy third-party libraries, we save ~990 ms of latency on cold start (a 43.5% speedup) and reduce physical RAM usage by ~9.5 MB, avoiding parsing ~75,000 lines of Python code at startup.

[daniel-sanche]

Yes, it is justified. While transport modules are not loaded by default under clean conditions, they are implicitly imported in two major ways:

• Application Default Credentials (ADC): google.auth.default() queries GCE metadata via _get_gce_credentials(), which imports google.auth.transport.requests.
• Client Library Imports: Generated REST/HTTP transports for all Google Cloud client libraries (e.g., Secret Manager, Redis, Workstations) import AuthorizedSession from google.auth.transport.requests at the module level.

In both of these cases, the classes are immediately imported and used though, right? You need the requests module to execute _get_gce_credentials, and IIRC you need AuthorizedSession to do anything with the REST clients. if the code immediately has to load the underlying libraries, lazy loading isn't beneficial. Are there any places we import from this module without using it?

• gRPC Clients Import REST Transports: The main client classes (e.g., SecretManagerServiceClient) import both gRPC and REST transport modules at module-level in their client.py files. Consequently, simply importing a gRPC client eagerly loads requests and all its dependencies, even if the user never makes a REST request.

This is a real example of where lazy loading is beneficial, but it seems like a separate issue. Aren't we already planning to add lazy loading to the client.py files separately, to avoid importing the rest clients when using grpc clients (and vice versa)?

Profile for google.auth.transport.requests [...]

I don't think benchmarking import google.auth.transport.requests on its own is meaningful, because this is an optional module; users will only import it if they plan on using it. In which case, lazy loading is not beneficial. To justify this change, we'd need to prove there are situations where these classes are being imported but left unused throughout the whole lifecycle, and show that the performance benefits outweigh the added complexity risks of the metaclasses.

You could try benchmarking an import of google-auth as a whole, to see if there are any differences there. Or benchmark constructing some rest/grpc clients? Or you could try to construct a code sample that represents a real-world workload that would benefit from this change. But if we can't detect changes at a macro scale, I don't think we should merge this change

[hebaalazzeh]

Yes, it is justified. While transport modules are not loaded by default under clean conditions, they are implicitly imported in two major ways:

• Application Default Credentials (ADC): google.auth.default() queries GCE metadata via _get_gce_credentials(), which imports google.auth.transport.requests.
• Client Library Imports: Generated REST/HTTP transports for all Google Cloud client libraries (e.g., Secret Manager, Redis, Workstations) import AuthorizedSession from google.auth.transport.requests at the module level.

In both of these cases, the classes are immediately imported and used though, right? You need the requests module to execute _get_gce_credentials, and IIRC you need AuthorizedSession to do anything with the REST clients. if the code immediately has to load the underlying libraries, lazy loading isn't beneficial. Are there any places we import from this module without using it?

• gRPC Clients Import REST Transports: The main client classes (e.g., SecretManagerServiceClient) import both gRPC and REST transport modules at module-level in their client.py files. Consequently, simply importing a gRPC client eagerly loads requests and all its dependencies, even if the user never makes a REST request.

This is a real example of where lazy loading is beneficial, but it seems like a separate issue. Aren't we already planning to add lazy loading to the client.py files separately, to avoid importing the rest clients when using grpc clients (and vice versa)?

Profile for google.auth.transport.requests [...]

I don't think benchmarking import google.auth.transport.requests on its own is meaningful, because this is an optional module; users will only import it if they plan on using it. In which case, lazy loading is not beneficial. To justify this change, we'd need to prove there are situations where these classes are being imported but left unused throughout the whole lifecycle, and show that the performance benefits outweigh the added complexity risks of the metaclasses.

You could try benchmarking an import of google-auth as a whole, to see if there are any differences there. Or benchmark constructing some rest/grpc clients? Or you could try to construct a code sample that represents a real-world workload that would benefit from this change. But if we can't detect changes at a macro scale, I don't think we should merge this change

@daniel-sanche Thank you for the detailed feedback. I ran the benchmarks across google.auth as a whole, individual transports, and client construction workflows on Python 3.15.0b2 to capture the macro-level differences between main (eager) and this branch (lazy).

---

1. Benchmark: google.auth as a Whole

As you anticipated, importing top-level google.auth alone only loads the core definitions (google.auth._default and google.auth.version) across 20 cold-start iterations:

Metric	main (Baseline)	PR Branch (feature/lazy-imports-google-auth)	Delta
P50 (Median) Latency	65.49 ms	76.32 ms	+10.83 ms (runtime variance)
Loaded Modules	17	17	0 (Identical)
Loaded Source Lines	7,339	7,339	0 (Identical)
Physical RSS RAM	2.94 MB	2.94 MB	0.00 MB

Top-level google.auth does not load transports on either branch, so the baseline overhead is identical.

---

2. Macro-Scale Scenario Comparison

Here is the side-by-side comparison across different import and client initialization scenarios in clean subprocesses:

Scenario	Metric	main (Eager Baseline)	PR Branch (PEP 810 Lazy)	Improvement
import google.auth.transport.requests	Latency	359.94 ms	263.75 ms	-96.19 ms (-26.7%)
	Loaded Modules	447	294	-153 modules (-34.2%)
	RAM Added	35.11 MB	28.70 MB	-6.41 MB (-18.3%)
	requests in sys.modules?	True (eagerly loaded)	False (deferred)	Deferred
	urllib3 in sys.modules?	True (eagerly loaded)	False (deferred)	Deferred
from google.oauth2 import service_account	Latency	337.01 ms	266.94 ms	-70.07 ms (-20.8%)
	Loaded Modules	293	292	-1 module
import REST Transport (SecretManager)	Latency	707.48 ms	631.24 ms	-76.24 ms (-10.8%)
	Loaded Modules	618	612	-6 modules
Construct gRPC Client (SecretManager)	Latency	1,816.01 ms	1,652.17 ms	-163.84 ms (-9.0%)
	Loaded Modules	629	624	-5 modules

---

3. Why Lazy Loading Transports is Beneficial

1. Defers Heavy 3rd-Party Dependencies Across Call Sites:

   • Over 40+ generated REST transport modules in google-cloud-python (google/cloud/*/transports/rest.py) import from google.auth.transport.requests import AuthorizedSession at module level.
   • On main, simply importing google.auth.transport.requests eagerly loads requests, urllib3, certifi, idna, and charset_normalizer (447 modules, 35.1 MB RAM).
   • On this PR branch, requests and urllib3 remain completely absent from sys.modules until an actual network request is executed (294 modules, 28.7 MB RAM, saving 153 modules and 6.4 MB RSS).

2. Complementary to GAPIC Lazy Loading:

   • While GAPIC client-level lazy loading defers importing the REST transport when using gRPC, having google-auth transport modules natively support PEP 810 ensures that any library, utility, or user script importing google.auth.transport.requests or AuthorizedSession directly does not pay the eager import tax at startup.

3. Compatibility & Reliability:

   • On Python 3.15+, base class resolution is handled transparently on first instantiation/access.
   • On Python < 3.15, standard eager loading executes with zero behavioral difference.
   • All 59 transport tests pass on Python 3.15, and all 883 unit tests pass on Python 3.11.

Let me know your thoughts!

[daniel-sanche]

I think the data just validates my concerns. I'm still seeing a lot of cost with no real benefit here:

Benefits

Over 40+ generated REST transport modules in google-cloud-python (google/cloud/*/transports/rest.py) import from google.auth.transport.requests import AuthorizedSession at module level.

On this PR branch, requests and urllib3 remain completely absent from sys.modules until an actual network request is executed (294 modules, 28.7 MB RAM, saving 153 modules and 6.4 MB RSS).

While GAPIC client-level lazy loading defers importing the REST transport when using gRPC, having google-auth transport modules natively support PEP 810 ensures that any library, utility, or user script importing google.auth.transport.requests or AuthorizedSession directly does not pay the eager import tax at startup.

We can safely assume anyone importing a rest client plans to use it, so we would be at best deferring costs, not removing them. So it looks like, best case, we defer ~200ms computation, and momentarily avoid ~7mb of memory

Also, keep in mind this cost is only paid once, no matter how many libraries they import

Costs

• Unpredictable lifecycle: If we know an expensive task is necessary, it's actually better to front-load it, so it doesn't end up unexpectedly bottle-necking time-sensitive operations later
  • Doing unexpected computation during a network request would be a worse experience for users, and would show up as spikes on customer's request latency charts
• Added runtime overhead: The change doesn't come for free. We are adding extra computation to resolve the classes. So this would actually be a performance degradation in the typical use-case.
  • It also adds overhead to each attribute look up, not just a one-time cost
  • It could be interesting to add a benchmark for constructing a REST client, to see what this overhead looks like. But I'm not going to ask you to do more work on this
• issubclass and introspection breakage: issubclass and class introspection would now behave in unexpected, inconsistent ways, because the subclasses will change depending on the lazy loading status
  • issubclass(AuthorizedSession, requests.Session) gives different answers pre- and post-instantiation
  • Related: I don't like that we'd have to trick TYPE_CHECKING for mypy to pass. That seems like it could bite us in the future
• Dynamic type modification: changing types dynamically at runtime makes me nervous, and this is a very important dependency.
  • Even if we could consistently shave off a couple MB and ms, I'd still be hesitant, because we'd be adding significant complexity. (And like I described above, I think we're only deferring the cost for a couple cycles)

On Python 3.15+, base class resolution is handled transparently on first instantiation/access.
On Python < 3.15, standard eager loading executes with zero behavioral difference.

The metaclasses aren't gated by version, so wouldn't we have these costs on all python versions?

All 59 transport tests pass on Python 3.15, and all 883 unit tests pass on Python 3.11.

Tests can be tricky to validate this kind of monkey-patch code, because they generally run against the resolved classes, so they may miss things (i.e. issubclass() checks)

---

I think we should close this, and focus on more straight-forward improvements