Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MINOR - Hybrid Ingestion Agent - Link ingestion agents to ingestion pipelines #19751

Merged
merged 5 commits into from
Feb 12, 2025
Merged

MINOR - Hybrid Ingestion Agent - Link ingestion agents to ingestion pipelines #19751

merged 5 commits into from
Feb 12, 2025

Conversation

gyohub
Copy link
Contributor

@gyohub gyohub commented Feb 11, 2025

Describe your changes:

Adds the IngestionAgent field to the IngestionPipeline schema. The field is a simple entity reference.
There's no db migrations since the field can be null.
This is required for the Collate's Hybrid ingestion agent feature.

Fixes GEN-2117

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@gyohub gyohub requested review from a team as code owners February 11, 2025 13:26
@notion-workspace Notion Workspace
Copy link

Link Agents to Ingestion Pipelines

@gyohub gyohub added the Collate label Feb 11, 2025
@github-actions GitHub Actions
Copy link
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@gyohub gyohub added the safe to test Add this label to run secure Github workflows on PRs label Feb 11, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 11, 2025 •
edited
Loading

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 64%
 64.63% (41371/64009) 41% (16709/40757) 44.22% (5040/11398)

@gyohub gyohub requested a review from pmbrull February 11, 2025 14:53
@github-actions GitHub Actions
Copy link
Contributor

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

harshach
harshach reviewed Feb 11, 2025
View reviewed changes
...pec/src/main/resources/json/schema/entity/services/ingestionPipelines/ingestionPipeline.json
@@ -221,6 +221,10 @@
"applicationType": {
"description": "Type of the application when pipelineType is 'application'.",
"type": "string"
},
"ingestionAgent" : {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires migration right cc @pmbrull

Copy link
Contributor Author

@gyohub gyohub Feb 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harshach db migration is not required since this field is nullable so any ingestion pipeline that is not hybrid won't have an agent reference.

@gyohub gyohub requested a review from harshach February 11, 2025 17:07
@gyohub gyohub force-pushed the add-ingestion-agent-reference-to-ingestion-pipeline branch from f5a8118 to d073f36 Compare February 11, 2025 17:18
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@gyohub gyohub enabled auto-merge (squash) February 12, 2025 12:33
@pmbrull pmbrull disabled auto-merge February 12, 2025 13:31
@pmbrull pmbrull merged commit 2559c79 into open-metadata:main Feb 12, 2025
26 of 28 checks passed
@gyohub gyohub mentioned this pull request Feb 13, 2025
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pmbrull pmbrull pmbrull approved these changes

@harshach harshach Awaiting requested review from harshach

Assignees
No one assigned
Labels
Collate safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@gyohub @harshach @pmbrull