Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[helm] Error on workload-launcher after migrating to 1.4.0 version #52034

Open
dantonbertuol opened this issue Jan 20, 2025 · 10 comments
Open

[helm] Error on workload-launcher after migrating to 1.4.0 version #52034

dantonbertuol opened this issue Jan 20, 2025 · 10 comments
Labels
area/platform issues related to the platform community helm team/deployments type/bug Something isn't working

Comments

@dantonbertuol
Copy link

dantonbertuol commented Jan 20, 2025 •
edited
Loading

Helm Chart Version

1.4.0

What step the error happened?

On deploy

Relevant information

I am facing an issue after upgrading to Airbyte version 1.4.0.

The problem occurs in the airbyte-workload-launcher pod and is an error when trying to create new pods, such as checking a data source. The problem that occurs is:

failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"]. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).

My values.yaml contains the configuration:

global:
    imagePullSecrets:
        - name: seniorsacred
    jobs:
        resources:
            requests:
                cpu: "500m"
                memory: "1Gi"
            limits:
                memory: "1Gi"
        kube:
             main_container_image_pull_secret: seniorsacred

Note: removing the main_container_image_pull_secret property works, but it no longer uses the custom pullSecret, and I need to use it.

Complete exception 
io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to create pod rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm.
	at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:46)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:38)
	at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.$$access$$apply(Unknown Source)
	at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456)
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:134)
	at io.airbyte.metrics.interceptors.InstrumentInterceptorBase.doIntercept(InstrumentInterceptorBase.kt:61)
	at io.airbyte.metrics.interceptors.InstrumentInterceptorBase.intercept(InstrumentInterceptorBase.kt:44)
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:143)
	at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.apply(Unknown Source)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:24)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:193)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
	at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)
	at reactor.core.scheduler.ImmediateScheduler$ImmediateSchedulerWorker.schedule(ImmediateScheduler.java:84)
	at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(MonoSubscribeOn.java:55)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
	at reactor.core.publisher.Mono.subscribeWith(Mono.java:4642)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4403)
	at io.airbyte.workload.launcher.pipeline.LaunchPipeline.accept(LaunchPipeline.kt:50)
	at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:28)
	at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:12)
	at io.airbyte.commons.temporal.queue.QueueActivityImpl.consume(Internal.kt:87)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
	at io.temporal.common.interceptors.ActivityInboundCallsInterceptorBase.execute(ActivityInboundCallsInterceptorBase.java:39)
	at io.temporal.opentracing.internal.OpenTracingActivityInboundCallsInterceptor.execute(OpenTracingActivityInboundCallsInterceptor.java:78)
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.airbyte.workers.exception.KubeClientException: Failed to create pod rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm.
	at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:244)
	at io.airbyte.workload.launcher.pods.KubePodClient.launchCheck(KubePodClient.kt:180)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:49)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:24)
	at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:42)
	... 53 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/airbyte/pods/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm?fieldManager=fabric8. Message: failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"]. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:205)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:251)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:1179)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:98)
	at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:56)
	at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:51)
	at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand$lambda$2(KubePodLauncher.kt:322)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand(KubePodLauncher.kt:322)
	at io.airbyte.workload.launcher.pods.KubePodLauncher.create(KubePodLauncher.kt:51)
	at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:241)
	... 57 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/airbyte/pods/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm?fieldManager=fabric8. Message: failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"]. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:134)
	... 3 common frames omitted

Relevant log output

io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to create pod rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm.
	at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:46)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:38)
	at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.$$access$$apply(Unknown Source)
	at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456)
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:134)
	at io.airbyte.metrics.interceptors.InstrumentInterceptorBase.doIntercept(InstrumentInterceptorBase.kt:61)
	at io.airbyte.metrics.interceptors.InstrumentInterceptorBase.intercept(InstrumentInterceptorBase.kt:44)
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:143)
	at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.apply(Unknown Source)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:24)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:193)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
	at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)
	at reactor.core.scheduler.ImmediateScheduler$ImmediateSchedulerWorker.schedule(ImmediateScheduler.java:84)
	at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(MonoSubscribeOn.java:55)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
	at reactor.core.publisher.Mono.subscribeWith(Mono.java:4642)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4403)
	at io.airbyte.workload.launcher.pipeline.LaunchPipeline.accept(LaunchPipeline.kt:50)
	at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:28)
	at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:12)
	at io.airbyte.commons.temporal.queue.QueueActivityImpl.consume(Internal.kt:87)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
	at io.temporal.common.interceptors.ActivityInboundCallsInterceptorBase.execute(ActivityInboundCallsInterceptorBase.java:39)
	at io.temporal.opentracing.internal.OpenTracingActivityInboundCallsInterceptor.execute(OpenTracingActivityInboundCallsInterceptor.java:78)
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.airbyte.workers.exception.KubeClientException: Failed to create pod rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm.
	at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:244)
	at io.airbyte.workload.launcher.pods.KubePodClient.launchCheck(KubePodClient.kt:180)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:49)
	at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:24)
	at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:42)
	... 53 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/airbyte/pods/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm?fieldManager=fabric8. Message: failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"]. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:205)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:251)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:1179)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:98)
	at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:56)
	at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:51)
	at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand$lambda$2(KubePodLauncher.kt:322)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand(KubePodLauncher.kt:322)
	at io.airbyte.workload.launcher.pods.KubePodLauncher.create(KubePodLauncher.kt:51)
	at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:241)
	... 57 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.30.0.1:443/api/v1/namespaces/airbyte/pods/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm?fieldManager=fabric8. Message: failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"]. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=failed to create typed patch object (/rce-bigquery-check-90026f61-b2bd-40bf-92f8-d7f4820d35e6-0-ytyqm; /v1, Kind=Pod): .spec.imagePullSecrets: duplicate entries for key [name="seniorsacred"], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:134)
	... 3 common frames omitted
@dantonbertuol dantonbertuol added area/platform issues related to the platform needs-triage type/bug Something isn't working labels Jan 20, 2025
@Hesperide
Copy link
Contributor

Hi @dantonbertuol! What previous Airbyte version were you on prior to upgrading to 1.4?

@dantonbertuol
Copy link
Author

Hi @dantonbertuol! What previous Airbyte version were you on prior to upgrading to 1.4?

Hi. Version 1.3.1

@abuchanan-airbyte
Copy link
Contributor

@dantonbertuol Can you try removing global.jobs.kube.main_container_image_pull_secret in the config on 1.4.0 please?

We made a change that uses global.imagePullSecrets for that value, but looks like we merge the two in the charts in 1.4.0, which causes duplicates if both are used. You can see that change (that I made) here. Sorry for the trouble.

I tested this locally and it seemed to work for me. All the job pods still had the image pull secret. Let me know if that works, I hope I didn't miss anything. Thanks.

@dantonbertuol
Copy link
Author

dantonbertuol commented Jan 21, 2025 •
edited
Loading

Hello @abuchanan-airbyte, thanks for the answer. As I mentioned in the issue, removing the main_container_image_pull_secret property works.

Even if I leave it out, if I apply global.imagePullSecrets, will it be used in the job containers?

If that's the case, everything is fine, because removing it worked.

@abuchanan-airbyte
Copy link
Contributor

Right. I expect that in 1.4.0 you should only need global.imagePullSecrets and those secrets should be included in the job pods.

@dantonbertuol
Copy link
Author

I checked the Airbyte configmap, and the JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET variable is set with the value from global.imagePullSecrets, even after removing the global.jobs.kube.main_container_image_pull_secret configuration.

So it should work.

@jamezrin jamezrin mentioned this issue Jan 22, 2025
Open
2 tasks
@jamezrin
Copy link

@marcosmarxm @abuchanan-airbyte I submitted a PR to airbyte-platform with a fix, please check airbytehq/airbyte-platform#383

@HaJunYoo
Copy link

HaJunYoo commented Jan 22, 2025 •
edited
Loading

The error only occurs in version 1.4.0. I have checked versions 1.2.0 and 1.3.2, but the issue does not arise in those versions.
I hope the following error is resolved as well

Error: couldn't find key JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET in ConfigMap airbyte/airbyte-airbyte-env

@rowanmoul
Copy link

The error only occurs in version 1.4.0. I have checked versions 1.2.0 and 1.3.2, but the issue does not arise in those versions. I hope the following error is resolved as well

Error: couldn't find key JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET in ConfigMap airbyte/airbyte-airbyte-env

I'm getting the same error. I have never set the image pull secret value in my helm values and no default is being set in the configmap, so the pod fails to start due to the missing key. I have manually removed the mapping from the deployment template for now.

@abuchanan-airbyte
Copy link
Contributor

Error: couldn't find key JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET in ConfigMap airbyte/airbyte-airbyte-env

For some reason, I can't reproduce this problem. We can probably fix the charts to be more defensive, but I'm still curious where this is coming from.

Can people share their helm version please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/platform issues related to the platform community helm team/deployments type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@marcosmarxm @jamezrin @Hesperide @rowanmoul @dantonbertuol @HaJunYoo @octavia-squidington-iii @abuchanan-airbyte