JobDefinition env parameter to specify pip index url and host

[ChristianZaccaria]

Issue link

Resolves #408

What changes have been made

• Created a function that checks for env variables in the DDPJobDefinition, and looks for the PIP_INDEX_URL and/or PIP_TRUSTED_HOST keys. If these keys are found, pip will leverage the dedicated index location provided with the mentioned env variables.

This allows the user to submit a job with the provided env variables for pip.

Verification steps

• Checkout this PR
• Install codeflare-sdk in your kubernetes cluster. Probably easiest way is to run poetry build, then install the generated wheel with pip install codeflare_sdk-0.0.0.dev0-py3-none-any.whl.
• Clone the codeflare-sdk repository.
• Run through the guided-demo notebook 2_basic_jobs.ipynb.
• Add the env vars to the JobDefinition such as this example:

jobdef = DDPJobDefinition(
    name="mnisttest",
    script="mnist.py",
    env={"PIP_INDEX_URL": "https://pypi.org/simple", 
        "PIP_TRUSTED_HOST": "pypi.org"},
    # script="mnist_disconnected.py", # training script for disconnected environment
    scheduler_args={"requirements": "requirements.txt"}
)
job = jobdef.submit(cluster)

• Go to your dashboard, look for your submitted job, and look at the Runtime environment used:
  
• Dependencies are installed from the trusted source provided.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign michaelclifford for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[VanillaSpoon]

Hey Christian,
I followed your verification steps and it works as expected :) Good job.

One thing is, if this is also being used to add --trusted-host and --index-url if they are not yet set, should we also create a requirements.txt file if it does not exist?

[KPostOffice]

closing as jobs.py file no longer exists