[E2E Databricks]: feat: improving deployment experience and functionality #970
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
updating lilem branch
to delete the pipeline if already exist before creating a new one.
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
fixing as feedbacks
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
deploy_azdo_pipelines -> removed the functions Common -> has the functions from deploy_azdo_pipelines README -> some fixes around Dev Container consistency
deploy_azdo_pipelines -> removed the functions Common -> has the functions from deploy_azdo_pipelines README -> some fixes around Dev Container consistency
deploy_azdo_pipelines -> removed the functions Common -> has the functions from deploy_azdo_pipelines README -> some fixes around Dev Container consistency
tested and it worked: Pipeline 23 was deleted successfully. Deleted existing pipeline: mdwdops-ci-qa-python (Pipeline ID: 23)
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
Co-authored-by: Elena Terenzi <[email protected]>
… when applicable (#915) * add soft delete and other fixes * added purge protection * moving variable initialization in init_environment script * rebase from feature branch * Update e2e_samples/parking_sensors/.devcontainer/.envtemplate Co-authored-by: Yennifer Santos <[email protected]> * initial commit * fix merge errors * merge errors * add delete by deployment id * handle key vault purge protection * small fix in condition * fix if condition * nit * code refactoring --------- Co-authored-by: Yennifer Santos <[email protected]>
fix the path ./scripts/
move images from root folder to isolate sample changes
removed debug flag
adding comment ## when returning a pipeline that does not exist. This is an unhandled result for the variable. So I made the validation direct in the IF
the command list handles the pipeline does not exist results, as command show does not handle. hence, changing to list
name change
anem change
to delete the pipeline if already exist before creating a new one.
tests and fixes: upon consolidating the create and delete pipelines in the same function. It does not work for this scenario.... .."cd_release_pipeline_id=$(createPipeline..." this one is waiting for an ID to proceed in create a variable. If I add everything in the same function the results kind of get confuse and it fails: like + az pipelines delete --id 59 --yes + cd_release_pipeline_id='Pipeline 59 was deleted successfully.' + az pipelines variable create --name devAdfName --pipeline-id 'Pipeline 59 was deleted successfully.' --value mdwdops-adf-dev-dnqom invalid literal for int() with base 10: 'Pipeline 59 was deleted successfully.' .....I tried multiple ways to fix that, removed all the echos, and lect only the return pipeline_id. But still failed. Because the return pipeline 59 was deleted successfully... is from AZdo itself. So I am keeping separated, 2) I was testing 2 scenarios.... First creation of pipelines if there is nothing, second round of deployment if the pipeline is there, delete first. the scenario that delete first was fine. the first round of deployment was failing because az pipelines show --name "$full_pipeline_name" --output json in fact returns something like: No pipeline found with name if using the json 2>/dev/null, it returns nothing. Nut validating with [[ -z "$pipeline_output" || "$pipeline_output" == "null" ] did not work. Bash just did not validated any if....I guess because is not empty or null, it is kind of unhandled result . logs would be like this: ++ az pipelines show --name mdwdops-ci-qa-python --output json + pipeline_id= For this what I did, instead of using a variable I added the instruction in the IF. So in that way, if does not exist, this validation is just skipt,if exists it works and delete.
tests and fixes Upon consolidating the create and delete pipelines in the same function. It does not work for this scenario.... .."cd_release_pipeline_id=$(createPipeline..." this one is waiting for an ID to proceed in create a variable. If I add everything in the same function the results kind of get confuse and it fails: like + az pipelines delete --id 59 --yes + cd_release_pipeline_id='Pipeline 59 was deleted successfully.' + az pipelines variable create --name devAdfName --pipeline-id 'Pipeline 59 was deleted successfully.' --value mdwdops-adf-dev-dnqom invalid literal for int() with base 10: 'Pipeline 59 was deleted successfully.' .....I tried multiple ways to fix that, removed all the echos, and lect only the return pipeline_id. But still failed. Because the return pipeline 59 was deleted successfully... is from AZdo itself. So I am keeping separated, 2) I was testing 2 scenarios.... First creation of pipelines if there is nothing, second round of deployment if the pipeline is there, delete first. the scenario that delete first was fine. the first round of deployment was failing because az pipelines show --name "$full_pipeline_name" --output json in fact returns something like: No pipeline found with name if using the json 2>/dev/null, it returns nothing. Nut validating with [[ -z "$pipeline_output" || "$pipeline_output" == "null" ] did not work. Bash just did not validated any if....I guess because is not empty or null, it is kind of unhandled result . logs would be like this: ++ az pipelines show --name mdwdops-ci-qa-python --output json + pipeline_id= For this what I did, instead of using a variable I added the instruction in the IF. So in that way, if does not exist, this validation is just skipt,if exists it works and delete.
removed debug flag
adding comment
the command list handles the pipeline does not exist results, as command show does not handle. hence, changing to list
anem change
2) Remove Azdo configuration from the post_create 3)Adding the init_enviromnent to run before the prequirements 4)change Readme to reflect those changes in Login
2) Remove Azdo configuration from the post_create 3)Adding the init_enviromnent to run before the prequirements 4)change Readme to reflect those changes in Login
adjusting identantion
aligment incorrect
Co-authored-by: Yennifer Santos <[email protected]>
Co-authored-by: Yennifer Santos <[email protected]>
Co-authored-by: Yennifer Santos <[email protected]>
Co-authored-by: Yennifer Santos <[email protected]>
…m/Azure-Samples/modern-data-warehouse-dataops into e2e/databricks/parking-sensors-V1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Type of PR
Purpose
Does this introduce a breaking change? If yes, details on what can break
Yes, all the changes were already tested before merging the feature branch, however the changes can break the following:
Author pre-publish checklist
Validation steps
Issues Closed or Referenced