Skip to content

Flyte 2 WIP#6583

Draft
EngHabu wants to merge 205 commits intomainfrom
v2
Draft

Flyte 2 WIP#6583
EngHabu wants to merge 205 commits intomainfrom
v2

Conversation

@EngHabu
Copy link
Copy Markdown
Contributor

@EngHabu EngHabu commented Aug 22, 2025

  • Add flyteidl and minimal publishing workflows

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

  • added: For new features.
  • changed: For changes in existing functionality.
  • deprecated: For soon-to-be-removed features.
  • removed: For features being removed.
  • fixed: For any bug fixed.
  • security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

* Add flyteidl and minimal publishing workflows

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* pretend version for buf python

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* python version fix and use OIDC to authenticate crates

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Fix python versioning

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Set version before setting up env

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* echo version

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Write to GITHUB_ENV

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Use env:

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf generate

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Mark go package as /v2 to be able to import both at the same time

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Upgrade to go 1.24

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Rename python, rust and npm to flyteidl2

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

…6610)

* trying idl2

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update trace def

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor: rename idl2 to flyteidl2

Signed-off-by: machichima <nary12321@gmail.com>

* generate -> buf

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor: rename idl2 to flyteidl2

Signed-off-by: machichima <nary12321@gmail.com>

* build: re-gen

Signed-off-by: machichima <nary12321@gmail.com>

* fix: exclude python venv

Signed-off-by: machichima <nary12321@gmail.com>

* Add translator service

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Regen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf gen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* remove node execution service proto

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Breakup task service into a separate package

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Help strings for make file targets

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update TaskLog with read bool

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Trying the updated protovalidate plugin

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* protovalidate with semver

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* try claude :shrug

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* add buf

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update deps

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* build: remove generate validate for python

Signed-off-by: machichima <nary12321@gmail.com>

* build: use managed mode on buf.gen.yaml

Signed-off-by: machichima <nary12321@gmail.com>

* Update validate go plugin

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update imagebuild

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Fix oneof buf lint

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Add deps to generated python code

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Be explicit about the index

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* updated idl

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Catch up to main

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Generate mocks for workflow package

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* generate mocks for other packages

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf improvements

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* updates

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* feat: generate mock for common

Signed-off-by: machichima <nary12321@gmail.com>

* use make gen in GH Actions

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Do not setup python env in generation CI

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* setup uv

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* set working directory

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Try to reference files by path

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Explicitly move .venv to root

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* try exclude in buf.yaml

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* set UV_PROJECT_ENVIRONMENT

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* download tooling

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* add which mockery

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: machichima <nary12321@gmail.com>
Co-authored-by: machichima <nary12321@gmail.com>
@flyte-bot
Copy link
Copy Markdown
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Add Trigger APIs

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update triggers

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Fix cargo build and add CI check

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* lint lib.rs

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update Readme and contributing guides

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* signed?

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build-crate target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix included ts files in npm

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Add device class to `GpuAccelerator`

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* make gen

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add HABANA_GAUDI as supported device class

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Clean up triggers api

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

* make gen

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

---------

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link
Copy Markdown

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

@github-actions github-actions bot mentioned this pull request Apr 1, 2026
3 tasks
* feat: add k8s event watcher

Signed-off-by: machichima <nary12321@gmail.com>

* fix: let WatchClusterEvents return cluster event

Signed-off-by: machichima <nary12321@gmail.com>

* feat: use last updated at to track sent event

Signed-off-by: machichima <nary12321@gmail.com>

* fix: initialize watcher when register plugin

Signed-off-by: machichima <nary12321@gmail.com>

* docs: add comments

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add per event object lock

Signed-off-by: machichima <nary12321@gmail.com>

* fix: watch before get

Signed-off-by: machichima <nary12321@gmail.com>

* docs: add todo comment

Signed-off-by: machichima <nary12321@gmail.com>

* test: fix test failure

Signed-off-by: machichima <nary12321@gmail.com>

* refactor: WatchActionUpdates rename to WatchAllActionUpdates

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add WatchActionUpdates to watch single action

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add event reason to event watcher

Signed-off-by: machichima <nary12321@gmail.com>

* feat: object cache use sync map

Signed-off-by: machichima <nary12321@gmail.com>

* fix: also use RecordedAt as cursor

Signed-off-by: machichima <nary12321@gmail.com>

* fix: SQLite support for WatchActionUpdates

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: machichima <nary12321@gmail.com>
@github-actions github-actions bot mentioned this pull request Apr 1, 2026
3 tasks
machichima and others added 3 commits April 1, 2026 01:22
* feat: executor store manager cache to sc.K8sCache

Signed-off-by: machichima <nary12321@gmail.com>

* feat: action k8s client watch through informer cache

Signed-off-by: machichima <nary12321@gmail.com>

* refactor: remove unused struct

Signed-off-by: machichima <nary12321@gmail.com>

* refactor: default watch workers to 10

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: machichima <nary12321@gmail.com>
…nt (#7110)

* change: change the return of the app and trigger service endpoint

Change return to service is not implemented.

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com>

* fix: re-define the RootActionName to rootActionName

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com>

* Revert "fix: re-define the RootActionName to rootActionName"

This reverts commit 1eaa77d.

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com>

---------

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com>
pmahindrakar-oss and others added 3 commits April 1, 2026 11:03
* Add provider based email templates for notifications

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>

* make gen

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>

* relax backward compat requirement

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>

* added comments

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>

* added atleast one required

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>

---------

Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>
…ible (#7130)

The generatedName template variable can contain underscores when
the task execution name is derived from inputs with underscores
(e.g. Python task names). This causes issues when the generated
name is used in log URLs that expect DNS-compatible names.

Apply ConvertToDNS1123SubdomainCompatibleString to the generatedName
value at template substitution time, ensuring it is always DNS-safe
without affecting the raw output prefix or other downstream usage.

Signed-off-by: Kevin Su <kevin@union.ai>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
pingsutw and others added 2 commits April 1, 2026 14:26
Signed-off-by: Kevin Su <pingsutw@apache.org>
…ased outputs (#7135)

When a connector task succeeds but returns nil outputs (relying on
file-based outputs written to remote storage), writeOutput was skipped
entirely, leaving the BufferedOutputWriter without a reader. This caused
ValidateOutput to fail with "Outputs not generated by task execution".

Always call writeOutput on SUCCEEDED so the RemoteFileOutputReader
fallback is used when outputs are nil, matching v1 behavior.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: machichima <nary12321@gmail.com>
machichima and others added 4 commits April 1, 2026 22:36
Signed-off-by: machichima <nary12321@gmail.com>
…server stub (#7137)

* Add dataplane_domain to GetPublicClientConfigResponse

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* Implement auth metadata service stub too

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

---------

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
…7141)

* add: group field to action metadata

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

---------

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
* fix(actions): remove minimum length validation from state field

- remove min_len=2 constraint from UpdateRequest.state field in
  actions_service.proto
- state field no longer enforces minimum 2-character requirement

* fix(actions): remove minimum length validation from state field

- remove min_len=2 constraint from UpdateRequest.state field in
  actions_service.proto
- state field no longer enforces minimum 2-character requirement

Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* make gen

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
@github-actions github-actions bot mentioned this pull request Apr 2, 2026
3 tasks
Signed-off-by: Kevin Su <pingsutw@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.