bug: Unable to containerize the bentoml build

[Mayurji]

Describe the bug

Unable to deploy the bentoml service for the given example on kubeflow.

To reproduce

Steps to reproduce

We are following the sample code for deploying the BentoML service.

• Example

We were able to execute all steps till "Serve Bento", after which we tried to deploy the same using both Bento's custom resources.

1. Deploy with BentoRequest CRD
2. Deploy with Bento CRD

Error:

In the first deployment method, we exported the bento build as a bento file into an s3 bucket, and tried executing "deployment_from_bentorequest.yaml", updated the s3 bucket path in yaml file and bento:tag. We didn't receive any error while applying the yaml file, but we also didn't see any pod spinning up. We looked at the kubernetes events

In the second deployment method, we tried containerizing the build using bentoml, but it is throwing an error as follows:

Error: [bento-cli] containerize failed: Backend Docker is not healthy

Expected behavior

Should create pods in the kubernetes cluster:

kubectl -n kubeflow get pods -l yatai.ai/bento-deployment=fraud-detection

NAME                                        READY   STATUS    RESTARTS   AGE
fraud-detection-67f84686c4-9zzdz            4/4     Running   0          10s
fraud-detection-runner-0-86dc8b5c57-q4c9f   3/3     Running   0          10s
fraud-detection-runner-1-846bdfcf56-c5g6m   3/3     Running   0          10s
fraud-detection-runner-2-6d48794b7-xws4j    3/3     Running   0          10s

Environment

Kubernetes: 1.25
Kubeflow: 1.7
Yatai-image-builder: 1.1.3
bentoml: 1.1.6

[parano]

@Mayurji could you confirm the status of docker registry used?