KEP-672: Implement the DependsOn API #740

andreyvelich · 2024-12-27T13:17:09Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

I added implementation for the DependsOn API. The majority of validations are implemented using CEL.

TODO list:

Add unit tests for the depends_on.go
Add controller integration tests to verify CEL validation.
Add controller integration tests to verify Job creation.

/cc @ahg-g @kannon92 @danielvegamyhre @tenzen-y @vsoch

Which issue(s) this PR fixes:

Fixes: #672

Does this PR introduce a user-facing change?

Add the DependsOn API to support serial execution of ReplicatedJobs.

k8s-ci-robot · 2024-12-27T13:17:15Z

@andreyvelich: GitHub didn't allow me to request PR reviews from the following users: vsoch.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

I added implementation for the DependsOn API. The majority of validations are implemented using CEL.

TODO list:

Add unit tests for the depends_on.go

Add controller integration tests to verify CEL validation.

Add controller integration tests to verify Job creation.

/cc @ahg-g @kannon92 @danielvegamyhre @tenzen-y @vsoch

Which issue(s) this PR fixes:

Fixes: #672

Does this PR introduce a user-facing change?
Add the DependsOn API to support serial execution of ReplicatedJobs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2024-12-27T13:17:19Z

Hi @andreyvelich. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2024-12-27T13:17:30Z

✅ Deploy Preview for kubernetes-sigs-jobset canceled.

Name	Link
🔨 Latest commit	`d76b286`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-jobset/deploys/677c5b97ebcb1b000823f38e

andreyvelich · 2024-12-27T13:20:07Z

api/jobset/v1alpha2/jobset_types.go

+
+	// Complete status means the Succeeded counter equals the number of child Jobs.
+	// .spec.replicatedJobs["name==<JOB_NAME>"].replicas == .status.replicatedJobsStatus.name["name==<JOB_NAME>"].succeeded
+	CompleteStatus DependsOnStatus = "Complete"


What do we think about Complete status here, given the discussion in: #723 ?

We probably want to match the Kubernetes Job and use Complete, so keep as is.

Yea I think that is fine.

kannon92 · 2024-12-27T14:43:04Z

/ok-to-test

andreyvelich · 2025-01-03T01:12:40Z

I've added integration and unit tests. Let me know if that looks good to you!
/assign @ahg-g @kannon92 @danielvegamyhre @tenzen-y @vsoch @shravan-achar @akshaychitneni

k8s-ci-robot · 2025-01-03T01:12:44Z

@andreyvelich: GitHub didn't allow me to assign the following users: vsoch, shravan-achar, akshaychitneni.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

I've added integration and unit tests. Let me know if that looks good to you!
/assign @ahg-g @kannon92 @danielvegamyhre @tenzen-y @vsoch @shravan-achar @akshaychitneni

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

vsoch · 2025-01-03T03:49:08Z

api/jobset/v1alpha2/jobset_types.go

+
+	// Complete status means the Succeeded counter equals the number of child Jobs.
+	// .spec.replicatedJobs["name==<JOB_NAME>"].replicas == .status.replicatedJobsStatus.name["name==<JOB_NAME>"].succeeded
+	CompleteStatus DependsOnStatus = "Complete"


We probably want to match the Kubernetes Job and use Complete, so keep as is.

kannon92 · 2025-01-03T16:27:30Z

test/integration/webhook/jobset_webhook_test.go

@@ -417,5 +417,126 @@ var _ = ginkgo.Describe("jobset webhook defaulting", func() {
 			},
 			updateShouldFail: true,
 		}),
+		ginkgo.Entry("DependsOn and StartupPolicy can't be set together", &testCase{


Please add a test case for StartupPolicy set to AnyOrder and allowing DependsOn

kannon92 · 2025-01-03T16:28:34Z

api/jobset/v1alpha2/jobset_types.go

+	// only after the referenced ReplicatedJobs reach their desired state.
+	// The Order of ReplicatedJobs is defined by their enumeration in the slice.
+	// Note, that the first ReplicatedJob in the slice cannot use the DependsOn API.
+	// TODO (andreyvelich): Currently, only a single item is supported in the DependsOn list.


I don't think we should have TODO in user facing APIs.

Sure, I will remove it.

kannon92 · 2025-01-03T16:28:50Z

api/jobset/v1alpha2/jobset_types.go

+	// only after the referenced ReplicatedJobs reach their desired state.
+	// The Order of ReplicatedJobs is defined by their enumeration in the slice.
+	// Note, that the first ReplicatedJob in the slice cannot use the DependsOn API.
+	// TODO (andreyvelich): Currently, only a single item is supported in the DependsOn list.


Suggested change

// TODO (andreyvelich): Currently, only a single item is supported in the DependsOn list.

// Currently, only a single item is supported in the DependsOn list.

kannon92 · 2025-01-03T16:30:26Z

api/jobset/v1alpha2/jobset_types.go

+	// .spec.replicatedJobs["name==<JOB_NAME>"].replicas == .status.replicatedJobsStatus.name["name==<JOB_NAME>"].ready
+	ReadyStatus DependsOnStatus = "Ready"
+
+	// Complete status means the Succeeded counter equals the number of child Jobs.


I'm not sure if this is actually true. A ReplicatedJob can be mark as succeeded if the success policy passes which doesn't necessarly mean that all the replicated jobs finished.

A ReplicatedJob can be mark as succeeded if the success policy passes which doesn't necessarly mean that all the replicated jobs finished.

Can you give me an example of this use-case? From my understanding, ReplicatedJob completion is based on ReplicatedJob's SuccessPolicy: https://kubernetes.io/docs/concepts/workloads/controllers/job/#success-policy.
Which means, ReplicatedJob will be in Complete status when this policy is met.

SuccessPolicy on the JobSet level just sets whether we should mark JobSet as Completed, when All or Any ReplicatedJobs reach Complete status.

Ah I see. You are correct.

kannon92 · 2025-01-03T16:31:02Z

api/jobset/v1alpha2/jobset_types.go

+
+const (
+	// Ready status means the Ready counter equals the number of child Jobs.
+	// .spec.replicatedJobs["name==<JOB_NAME>"].replicas == .status.replicatedJobsStatus.name["name==<JOB_NAME>"].ready


I don't think the boolean expression is necessary.

Why ?

Should we modify this as follows:

.spec.replicatedJobs["name==<JOB_NAME>"].replicas == .status.replicatedJobsStatus.name["name==<JOB_NAME>"].ready + .status.replicatedJobsStatus.name["name==<JOB_NAME>"].succeeded + .status.replicatedJobsStatus.name["name==<JOB_NAME>"].failed

#740 (comment)

Yes.

kannon92 · 2025-01-03T16:31:07Z

api/jobset/v1alpha2/jobset_types.go

+	ReadyStatus DependsOnStatus = "Ready"
+
+	// Complete status means the Succeeded counter equals the number of child Jobs.
+	// .spec.replicatedJobs["name==<JOB_NAME>"].replicas == .status.replicatedJobsStatus.name["name==<JOB_NAME>"].succeeded


I don't think the boolean expression is necessary.

Oh, you mean for user-facing docs. Sure, I think we can remove it.

kannon92 · 2025-01-03T16:34:02Z

pkg/webhooks/jobset_webhook.go

+		replicatedJobNames[rJob.Name] = rIdx
+		// Check that DependsOn references the previous ReplicatedJob.
+		if rIdx > 0 && rJob.DependsOn != nil {
+			dependsOnIdx, ok := replicatedJobNames[rJob.DependsOn[0].Name]


Suggested change

dependsOnIdx, ok := replicatedJobNames[rJob.DependsOn[0].Name]

dependsOnIdx, ok := replicatedJobNames[rJob.DependsOn[rIdx - 1].Name]

I think you want to look at the previous job and not just the 0th element.

Actually, DependsOn is an array which always has a single element, as you can see here:

jobset/api/jobset/v1alpha2/jobset_types.go

Line 245 in d751c13

// +kubebuilder:validation:MaxItems=1

Does it make sense @kannon92 ?

Ah yes. I got my streams crossed.

kannon92 · 2025-01-03T16:36:54Z

pkg/controllers/jobset_controller.go

+		rJobsReplicas[replicatedJob.Name] = replicatedJob.Replicas
+
+		// For depends on, the Job is created only after the previous replicatedJob reached the status.
+		if replicatedJob.DependsOn != nil && !isDependsOnJobReachedStatus(replicatedJob.DependsOn[0], rJobsReplicas[replicatedJob.DependsOn[0].Name], replicatedJobStatuses) {


I think this is a bug. You probably want the idx - 1 rather than 0.

kannon92

Given how important this is for kubeflow, I am wondering if a e2e test would be useful.

kannon92 · 2025-01-03T16:40:11Z

pkg/webhooks/jobset_webhook_test.go

@@ -1487,10 +1487,152 @@ func TestValidateCreate(t *testing.T) {
 		},
 	}

+	dependsOnTests := []validationTestCase{
+		{
+			name: "dependsOn is valid",


Can we add a case where job-2 waits for job-1 and job-3 waits for job-2?

Improve API docs

k8s-ci-robot · 2025-01-06T15:21:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: andreyvelich, vsoch
Once this PR has been reviewed and has the lgtm label, please ask for approval from ahg-g. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich · 2025-01-06T15:43:22Z

Given how important this is for kubeflow, I am wondering if a e2e test would be useful.

Yeah, maybe we could add one e2e test as part of JobSet: https://github.com/kubernetes-sigs/jobset/blob/main/test/e2e/e2e_test.go.
Will add it soon.

andreyvelich · 2025-01-07T14:50:52Z

@kannon92 @tenzen-y I've added the simple E2E test for DependsOn API.

andreyvelich added 2 commits December 20, 2024 16:24

Add DependsOn API

370b4fe

Add JobSet controller changes

39f712a

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 27, 2024

k8s-ci-robot requested a review from ahg-g December 27, 2024 13:17

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Dec 27, 2024

k8s-ci-robot requested review from kannon92 and danielvegamyhre December 27, 2024 13:17

k8s-ci-robot requested a review from tenzen-y December 27, 2024 13:17

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 27, 2024

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 27, 2024

andreyvelich commented Dec 27, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 27, 2024

andreyvelich added 2 commits January 2, 2025 19:16

Add integration tests for DependsOn CEL validation

34c0593

Add unit tests for DependsOn

22a408b

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 2, 2025

Add controller integration tests for DependsOn

600f0a6

k8s-ci-robot assigned ahg-g, danielvegamyhre, kannon92 and tenzen-y Jan 3, 2025

andreyvelich changed the title ~~[WIP] KEP-672: Implement the DependsOn API~~ KEP-672: Implement the DependsOn API Jan 3, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 3, 2025

Fix go lint

d751c13

vsoch approved these changes Jan 3, 2025

View reviewed changes

kannon92 reviewed Jan 3, 2025

View reviewed changes

Add test case with DependsOn and StartupPolicy: AnyOrder

725e7d9

Improve API docs

Test case when job-2 depends on job-1 and job-3 depends on job-2

bf743bb

andreyvelich force-pushed the issue-672-implement-depends-on branch from 76d53e5 to bf743bb Compare January 6, 2025 15:27

andreyvelich added 2 commits January 6, 2025 15:46

Add manifests to the make generate

c8e052c

Add E2E test for the DependsOn API

d76b286

andreyvelich force-pushed the issue-672-implement-depends-on branch from c0b8760 to d76b286 Compare January 6, 2025 22:39

	// TODO (andreyvelich): Currently, only a single item is supported in the DependsOn list.
	// Currently, only a single item is supported in the DependsOn list.

	dependsOnIdx, ok := replicatedJobNames[rJob.DependsOn[0].Name]
	dependsOnIdx, ok := replicatedJobNames[rJob.DependsOn[rIdx - 1].Name]

KEP-672: Implement the DependsOn API #740

Are you sure you want to change the base?

KEP-672: Implement the DependsOn API #740

Conversation

andreyvelich commented Dec 27, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Dec 27, 2024

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Dec 27, 2024

netlify bot commented Dec 27, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-jobset canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kannon92 commented Dec 27, 2024

andreyvelich commented Jan 3, 2025

k8s-ci-robot commented Jan 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kannon92 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 6, 2025

andreyvelich commented Jan 6, 2025

andreyvelich commented Jan 7, 2025

andreyvelich commented Dec 27, 2024 •

edited

Loading

netlify bot commented Dec 27, 2024 •

edited

Loading