Add static and runtime dag info, API to fetch ancestor and successor tasks #2124

talsperre · 2024-10-31T09:07:57Z

Add runtime DAG info so that we can query the ancestor and successor tasks for a given task easily.

Usage

from metaflow import Task, namespace
namespace(None)
task = Task('RuntimeDAGFlow/18/step_c/32076012', attempt=0)

To get ancestors, progenies, and siblings, use the following API:

ancestors = task.ancestors
successors = task.successors

The output would be a list of metaflow Task objects.

metaflow/metadata/metadata.py

romain-intel

A few comments. I think it's pretty close though. I haven't looked at hte metadata service changes. We may also want to raise a better error message if the service is not new enough?

metaflow/client/core.py

metaflow/metaflow_config.py

metaflow/plugins/metadata_providers/local.py

metaflow/task.py

metaflow/plugins/metadata_providers/service.py

metaflow/client/core.py

savingoyal

quick UX feedback - let me know if in the new proposed UX we miss out on any use cases. i am reviewing the rest of the PR meanwhile.

metaflow/client/core.py

metaflow/task.py

metaflow/client/core.py

metaflow/task.py

metaflow/client/core.py

test/core/tests/client_ancestors.py

metaflow/client/core.py

romain-intel

not a review -- just some comments.

metaflow/client/core.py

…a services

metaflow/client/core.py

savingoyal · 2025-02-12T20:35:28Z

metaflow/client/core.py

+                yield Task(pathspec=task_pathspec, _namespace_check=False)
+
+    @property
+    def parent_tasks(self) -> List["Task"]:


would be good to handle the case where the user is inspecting a task that ran using an old version of metaflow or is using an old version of the service...

metaflow/client/core.py

metaflow/metadata_provider/metadata.py

metaflow/plugins/metadata_providers/local.py

metaflow/task.py

test/core/tests/client_child_tasks.py

savingoyal · 2025-02-20T17:25:07Z

metaflow/client/core.py

+                task_pathspecs = self._metaflow.metadata.filter_tasks_by_metadata(
+                    flow_id, run_id, step.id, metadata_key, metadata_pattern
+                )
+            except Exception as e:
+                if e.http_code == 404:
+                    # filter_tasks_by_metadata endpoint does not exist in the version of metadata service
+                    # deployed currently. Raise a more informative error message.
+                    raise MetaflowInternalError(
+                        "The version of metadata service deployed currently does not support filtering tasks by metadata. "
+                        "Upgrade to a newer version of Metadata service to use this feature."
+                    ) from e
+


this code block can be within the service implementation and not in the client. the client shouldn't have to bother about the actual implementation semantics of metadata.

If the endpoint does not exist in the metadata service then how do you even return that error message?

this try-except block can be in service.py rather than in the client.

Okay, that sounds reasonable.

metaflow/client/core.py

savingoyal · 2025-02-20T17:30:27Z

metaflow/exception.py

@@ -160,6 +160,15 @@ def __init__(self, msg, unhandled):
        self.artifact_names = unhandled


+class ServiceException(MetaflowException):


why move this here? - this is tied to the service implementation and ideally shouldn't be pulled into the core

metaflow/task.py

test/core/contexts.json

test/core/tests/runtime_dag_tests.py

metaflow/client/core.py

savingoyal · 2025-02-20T19:24:13Z

metaflow/plugins/metadata_providers/service.py

+                # deployed currently. Raise a more informative error message.
+                raise MetaflowInternalError(
+                    "The version of metadata service deployed currently does not support filtering tasks by metadata. "
+                    "Upgrade to a newer version of Metadata service to use this feature."


can you add the metaflow service version that they need to upgrade to? basically if the latest is 2.3.4, then, have a message that says upgrade to >=2.4.0

cc @saikonen to bump the version of metaflow-service when the endpoint is released.

talsperre force-pushed the dev/add-runtime-dag-info branch from 48c771d to ec43f14 Compare November 1, 2024 18:34

saikonen reviewed Nov 11, 2024

View reviewed changes

metaflow/metadata/metadata.py Outdated Show resolved Hide resolved

metaflow/metadata/metadata.py Outdated Show resolved Hide resolved

saikonen mentioned this pull request Nov 22, 2024

feature: add task filtering based on metadata Netflix/metaflow-service#449

Open

talsperre force-pushed the dev/add-runtime-dag-info branch from ffbf68a to c6fb9ac Compare January 2, 2025 23:25

talsperre changed the title ~~Add static and runtime dag info, API to fetch ancestor tasks~~ Add static and runtime dag info, API to fetch ancestor and successor tasks Jan 7, 2025

talsperre force-pushed the dev/add-runtime-dag-info branch 2 times, most recently from d66d32b to 7644058 Compare January 12, 2025 03:12

talsperre requested review from savingoyal and romain-intel January 12, 2025 05:53

romain-intel reviewed Jan 13, 2025

View reviewed changes

saikonen reviewed Jan 14, 2025

View reviewed changes

metaflow/plugins/metadata_providers/service.py Outdated Show resolved Hide resolved

metaflow/plugins/metadata_providers/service.py Outdated Show resolved Hide resolved

talsperre force-pushed the dev/add-runtime-dag-info branch from 17a4489 to 7cdfb41 Compare January 15, 2025 00:53

romain-intel reviewed Jan 16, 2025

View reviewed changes

metaflow/client/core.py Outdated Show resolved Hide resolved

saikonen requested changes Jan 16, 2025

View reviewed changes

metaflow/client/core.py Outdated Show resolved Hide resolved

talsperre force-pushed the dev/add-runtime-dag-info branch from a8df33d to 7833e40 Compare January 22, 2025 08:30

savingoyal requested changes Jan 28, 2025

View reviewed changes

metaflow/client/core.py Outdated Show resolved Hide resolved

metaflow/client/core.py Outdated Show resolved Hide resolved

metaflow/client/core.py Outdated Show resolved Hide resolved

savingoyal reviewed Jan 28, 2025

View reviewed changes

metaflow/task.py Outdated Show resolved Hide resolved

savingoyal requested changes Feb 4, 2025

View reviewed changes

romain-intel reviewed Feb 7, 2025

View reviewed changes

metaflow/client/core.py Outdated Show resolved Hide resolved

metaflow/client/core.py Outdated Show resolved Hide resolved

talsperre added 11 commits February 11, 2025 23:12

Add static and runtime dag info, API to fetch ancestor tasks

255485b

Add API to get immediate successors

bd377a9

Add API for getting closest siblings

0214a61

Update metadata API params

480d360

Refactor ancestor and successor client code

0131f43

Remove unneccessary prints

d70cf98

Support querying ancestors and successors in local metadata provider

70863e5

Refactor and simplify client code

b01fc7d

Make query logic more descriptive

f08be5a

Add core tests for ancestor task API

cf49ace

Add core test for immediate successor API

6218ffe

talsperre and others added 8 commits February 11, 2025 23:15

Update logic for siblings, make it work for static splits as well

bec68f7

update service url for filter task requests. update query param names.

e5f1ed7

Fix bug in parsing steps due to different data formats across metadat…

959e6a3

…a services

json serialize the ancestry metadata

7c827b5

Address comments

f4936fa

Update docstrings

32a889e

Remove duplicate code

7b1d717

Address comments

bc9e456

talsperre force-pushed the dev/add-runtime-dag-info branch from 75c1301 to bc9e456 Compare February 12, 2025 11:49

talsperre added 7 commits February 12, 2025 03:53

Remove commented out code

b03f0c4

Update docstrings

67ee87e

Remove spurious import in core

ff27d87

Update OSS metadata service API call

adc5bf7

Remove commented code from parent task tests

6aae908

Remove spurious function

035e3a3

Remove spurious comment

f821d1e

savingoyal requested changes Feb 12, 2025

View reviewed changes

talsperre added 2 commits February 19, 2025 02:46

Address comments

ffffee4

Update tests

9b93945

savingoyal requested changes Feb 20, 2025

View reviewed changes

talsperre added 4 commits February 20, 2025 11:08

Address comments

81bac83

Update docstrings

997b8fd

Address black comments

ea66248

Address black comments

618e23b

savingoyal reviewed Feb 20, 2025

View reviewed changes

metaflow/client/core.py Show resolved Hide resolved

savingoyal reviewed Feb 20, 2025

View reviewed changes

talsperre added 2 commits February 20, 2025 11:26

Update docstrings, remove duplicate property

f3b63d5

Return metadata service version needed for runtime dag apis

9541757

talsperre merged commit 333eeeb into master Feb 20, 2025
29 checks passed

talsperre deleted the dev/add-runtime-dag-info branch February 20, 2025 21:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add static and runtime dag info, API to fetch ancestor and successor tasks #2124

Add static and runtime dag info, API to fetch ancestor and successor tasks #2124

talsperre commented Oct 31, 2024 •

edited

Loading

romain-intel left a comment

savingoyal left a comment

romain-intel left a comment

savingoyal Feb 12, 2025

savingoyal Feb 20, 2025

talsperre Feb 20, 2025

savingoyal Feb 20, 2025

talsperre Feb 20, 2025

savingoyal Feb 20, 2025

savingoyal Feb 20, 2025

savingoyal Feb 20, 2025

		@@ -160,6 +160,15 @@ def __init__(self, msg, unhandled):
		self.artifact_names = unhandled


		class ServiceException(MetaflowException):

Add static and runtime dag info, API to fetch ancestor and successor tasks #2124

Add static and runtime dag info, API to fetch ancestor and successor tasks #2124

Conversation

talsperre commented Oct 31, 2024 • edited Loading

Usage

romain-intel left a comment

Choose a reason for hiding this comment

savingoyal left a comment

Choose a reason for hiding this comment

romain-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

talsperre commented Oct 31, 2024 •

edited

Loading