How are you distributing the projects built with Kubuilder?

[camilamacedo86]

Would be great to know how the Kubebuilder community has been distributing their solutions built with the tool. Therefore, could you let us know:

• a) How are you distributing the projects built with Kubuilder?
• b) Would you would like to propose any suggestion that could help improve the project?

[camilamacedo86]

Related topic (if you are doing that with helm-charts let us know:): #3074

[Kavinjsir]

For developing or testing, I use IMG=... make deploy, which is handy.

For production, I used to observe the diff before apply:

kustomize build config/default | kubectl diff -f - | diff-so-fancy

[wmuizelaar]

For local development, we use the make run option, together with a local kind cluster. Deployment to is now done by an out-of-repo kustomization yaml that currently looks like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - [email protected]/config/default?ref=v0.1.4 # the tag here needs to match the image

namespace: random-operator-system

generatorOptions:
  disableNameSuffixHash: true

configMapGenerator:
  - files:
      - controller_manager_config.yaml
    name: manager-config
    behavior: replace
  - files:
      - path-to-local-file.yaml
    name: local

images:
  - name: controller
    newName: eu.gcr.io/bolcom-stg-hammertime-1d7/kb-applicationproject-operator
    newTag: v0.1.4

labels:
  - pairs:
      key: value

We're not using helm-charts to deploy, but pure kustomize. But the rest of our ecosystem is moving towards usage of kpt now, so if kubebuilder operator would support kpt as well (i.e. maybe just craft a blueprint-package from the generated yamls?) that could be helpful, but it's not a must from my perspective.

[camilamacedo86]

Can you please add a link for this kpt solution?
Could you please let me know more why use kpt?
Also, what is the diff of the idea bellow #3081 (comment) vs supporting kpt?

[LCaparelli]

We use helm charts, but we're looking to move over to kustomize for better integration with kubebuilder/operator-sdk (we currently create our helm charts based on a template and manually maintain it, with a few hacks here and there to copy stuff over from config/) or (hopefully) a more robust package system.

So yeah, basically publish a chart from within the operator's repo and consume this chart from a separate repo that manages deployment configuration for different environments, etc.

[robbie-demuth]

Our team did away with the Kustomize scaffolds and maintains a custom Helm chart based on the scaffolds. We updated our Makefile such that manifests, etc are generated into the proper directories and patched such that they're valid Helm templates. We then package up the Helm chart and publish it to an internal repository.

I'd love to see a community maintained Kubebuilder plugin for outputting a Helm chart instead of Kustomize scaffolds.

As an aside, I've been meaning to look into OLM

[Jay-Madden]

A supported helm chart solution would work wonders for large teams that are already integrating heavily with existing helm charts for deployments like argocd and rancher. Allowing custom operators to easily fit into the existing pattern.

[camilamacedo86]

I added the same content in the other thread, so I will add here:

Hi @robbie-demuth, @Jay-Madden, @LCaparelli,

Firstly, I appreciate the insights you've shared regarding the use of Helm charts for distributing Operators. I'm aiming to understand the core need for this approach.

Looking at successful projects like prometheus-operator, they've opted for a different distribution method. Here's a potential alternative solution:

1. Publish the Operator Image: The same way it's done when building the Helm chart. This image can reside in a registry.
2. Distribute Configuration YAMLs: The main distribution requirement seems to be the YAML files for configuration. (same result of run make install and make deploy IMG=)
3. Single Command Deployment: Create a consolidated YAML file, much like what make install or make deploy IMG=myregistry/myoperator:tag accomplishes. This can allow users to deploy the operator with a straightforward kubectl apply -f <url_to_raw_yaml>.

My question is: wouldn't the above approach meet your distribution needs? If Kubebuilder could generate such a consolidated YAML and host it on GitHub, users could simply apply it as:

kubectl apply -f https://raw.githubusercontent.com/myorg/myproject/master/deploy/install.yaml

So, if kubebuilder project the content for the install.yaml with all config for you just have it your repo would not the need to be addressed?

It seems a majority of the solutions revolve around converting Kustomize manifests to Helm charts. I'm wondering if the complexity they introduce is a necessity. Especially if the alternative method mentioned above can achieve the same outcome.

I'm keen to understand any specific advantages Helm offers in this context, or if there are limitations or challenges with the proposed approach.

Your feedback will be invaluable!
Thanks!

[Jay-Madden]

Left a response here #3074 (reply in thread)

[LCaparelli]

Same: #3074 (reply in thread)

[aerfio]

How are you distributing the projects built with Kububuilder?

As an OCI artifact using helm

Would you would like to propose any suggestion that could help improve the project?

Helm plugin for kubebuilder?

To add more context: in my $work we only use annotations for CRD generation/validation, we do not use kubebuilder rbac markers that add permissions to clusterrole, because we found it's easier to just add it manually to that 1 file than to learn the syntax of those markers

[camilamacedo86]

I think we can close this one.
It seems that is well desired to use Helm to distribute the projects and we have been working on a proposal to help users with: #3632

Also, we can try to provide an option where all YAML files will be generated, and users could only run kubectl apply -f to get the solution. This work started to be done in the PR: #3626

[Kavinjsir]

Hey @camilamacedo86 I'm curious about the status about #3626, the idea and the code in general looks good. Is there any further additions expected?

[camilamacedo86]

We need to rebase this PR with the master and continue the work.
it has not passed the e2e tests yet.
I had no time to do so, but if someone would like to move forward with it, help is always wanted.

[camilamacedo86]

@Kavinjsir it is done now and in good shape to fly, see: #3626

[Kavinjsir]

That is awesome @camilamacedo86 ! I love this great feature and appreciate your prompt update! Just bring my review there.