Skip to content

Execution.create() fails with 503 due to credentials annotation typo #6610

Description

@javabrett

Bug

google.cloud.aiplatform.metadata.execution.Execution.create() fails with a 503 error when called without explicitly passing credentials=:

503 Getting metadata from plugin failed with error: before_request

Root cause

Two issues in google/cloud/aiplatform/metadata/execution.py:

1. credentials parameter annotation typo (high severity)

Both create() and _create() use = instead of : for the credentials type annotation:

# Current (broken) -- execution.py lines 103, 181:
credentials=Optional[auth_credentials.Credentials],

# Should be (matches artifact.py, context.py):
credentials: Optional[auth_credentials.Credentials] = None,

Because = is used instead of :, the default value is the typing.Optional[google.auth.credentials.Credentials] type object itself (a _UnionGenericAlias), not None. This non-None type object is passed down to the gRPC auth stack, which attempts to call .before_request() on it, producing the 503.

Introduced in PR #1410 (June 2022). artifact.py and context.py both have the correct : ... = None syntax.

2. Missing ensure_default_metadata_store_exists() call (lower severity)

Artifact.create() calls metadata_store._MetadataStore.ensure_default_metadata_store_exists() before delegating to _create(). Execution.create() skips this call entirely. This means standalone Execution.create() will also fail if the default metadata store hasn't been implicitly created by a prior Artifact.create() or aiplatform.init(experiment=...) call. Context.create() has the same gap but is not addressed here.

Reproduction

from google.cloud import aiplatform
from google.cloud.aiplatform.metadata import execution as metadata_execution
from google.cloud.aiplatform.compat.types import execution as gca_execution

aiplatform.init(project="my-project", location="us-central1")

# This fails with 503:
exec_obj = metadata_execution.Execution.create(
    schema_title="system.CustomJob",
    resource_id="test-execution",
    display_name="test",
    state=gca_execution.Execution.State.COMPLETE,
    metadata={"component_type": "custom_job"},
)

Workaround

Call ensure_default_metadata_store_exists() manually, which also initializes credentials properly:

from google.cloud.aiplatform.metadata import metadata_store
metadata_store._MetadataStore.ensure_default_metadata_store_exists(
    project="my-project", location="us-central1",
)
# Now Execution.create() works

Fix

Two changes to execution.py:

  1. Fix credentials=Optional[auth_credentials.Credentials] to credentials: Optional[auth_credentials.Credentials] = None on both create() and _create()
  2. Add the ensure_default_metadata_store_exists() call in create() (matching Artifact.create())

Environment

  • google-cloud-aiplatform 1.140.0
  • Python 3.13.2
  • macOS / Vertex AI training containers (both affected)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: vertex-aiIssues related to the googleapis/python-aiplatform API.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions