Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] "Not enough non-faulty bookies available error code: -6" persisting after outage while all bookies are reported as healthy #23807

Open
2 of 3 tasks
mrtworo opened this issue Jan 3, 2025 · 0 comments
Open
2 of 3 tasks

[Bug] "Not enough non-faulty bookies available error code: -6" persisting after outage while all bookies are reported as healthy #23807

mrtworo opened this issue Jan 3, 2025 · 0 comments
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@mrtworo
Copy link

mrtworo commented Jan 3, 2025

Search before asking

  • I searched in the issues and found nothing similar.

Read release policy

  • I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

Apache pulsar-3.8.0 chart with pulsar 4.0.1 deployed in EKS.

Minimal reproduce step

Pulsar consisting of:
4 bookies
3 brokers
3 zookeepers

two persistent topics configured with E=3, Qw=3, Qa=2

Two bookies and zookeeper restart and are unavailable for minutes.
Both topics owned by the same brooker.

What did you expect to see?

  1. Both topics affected by 2/4 bookies unavailable.
  2. Broker not being able to form new ensembles to handle writes.
  3. Both recovered after bookies are up.

What did you see instead?

Both were rightly complaining on "org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available", however when bookies were back, only first topic a7u_transactions was able to recover, second topic a7u_transactions_sanitized was stuck in endless loop of un-fencing and failed writes until broker was restarted an it opened normally on the new owner.

Before the restart in following morning, we've double checked the bookies status:

pulsar-toolset-0:/pulsar$ bin/bookkeeper shell listbookies -rw -h
...
2025-01-02T09:28:46,406+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies :
2025-01-02T09:28:46,422+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.219.28, Port:3181, Hostname:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local
2025-01-02T09:28:46,423+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.219.36, Port:3181, Hostname:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local
2025-01-02T09:28:46,424+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.225.176, Port:3181, Hostname:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local
2025-01-02T09:28:46,426+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181, IP:10.11.232.43, Port:3181, Hostname:pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local

all were up, there were no other topics were reporting any issues with bookies.

We don't understand why a7u_transactions recovered while a7u_transactions_sanitized remained impacted for hours while all bookies were up. It seemed like it's an edge case where there was no attempt to form new ensemble for the affected topic.

Detailed logs attached.

Anything else?

bookie-0 returns at:

2025-01-01T20:53:09,319+0000 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
...
2025-01-01T20:53:09,624+0000 [main] INFO  org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server.

bookie-1 returns at:

2025-01-01T20:53:34,979+0000 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181
...
2025-01-01T20:53:35,305+0000 [main] INFO  org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server.

broker-1 (owner) logs when topics seemed to start behaving differently:

2025-01-01T20:53:09,314+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running
2025-01-01T20:53:09,314+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 1) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 2) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, bookie does not exist or it is not running
2025-01-01T20:53:09,315+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (535058, 3) to bookie (0, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181): Bookie handle is not available
2025-01-01T20:53:09,314+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:535058, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:850) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:770) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:225) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:619) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:246) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:542) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:139) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$11(LedgerHandle.java:2222) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:2241) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:2189) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.BookieClientImpl.completeAdd(BookieClientImpl.java:287) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.BookieClientImpl.access$000(BookieClientImpl.java:79) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:414) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.BookieClientImpl$ChannelReadyForAddEntryCallback.operationComplete(BookieClientImpl.java:373) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2687) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.PerChannelBookieClient$ConnectionFutureListener.operationComplete(PerChannelBookieClient.java:2592) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.PerChannelBookieClient.processBookieNotResolvedError(PerChannelBookieClient.java:336) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:545) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:679) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.obtain(DefaultPerChannelBookieClientPool.java:115) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:312) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:150) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.PendingAddOp.initiate(PendingAddOp.java:258) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.LedgerHandle.doAsyncAddEntry(LedgerHandle.java:1640) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.client.LedgerHandle.asyncAddEntry(LedgerHandle.java:1338) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.mledger.impl.OpAddEntry.initiate(OpAddEntry.java:153) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.updateLedgersIdsComplete(ManagedLedgerImpl.java:1736) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$14.operationComplete(ManagedLedgerImpl.java:1606) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$14.operationComplete(ManagedLedgerImpl.java:1592) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
	at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncUpdateLedgerIds$7(MetaStoreImpl.java:187) ~[org.apache.pulsar-managed-ledger-4.0.1.jar:4.0.1]
	at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?]
	at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
	at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 535058 due to NotEnoughBookiesException: Not enough non-faulty bookies available
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E0
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E1
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E2
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L535058 E3
2025-01-01T20:53:09,315+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Bookie pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 created. path: /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,315+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Bookie pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 created. path: /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,315+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.pulsar.broker.transaction.buffer.impl.TopicTransactionBuffer - Failed to append buffer to txn (11,2578)
org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,317+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0xd93fe9b7, L:/10.11.217.164:6650 - R:/10.11.219.27:58438] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,318+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5053 (v. Optional.empty)
2025-01-01T20:53:09,318+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5052 (v. Optional.empty)
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,319+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,320+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,321+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic...
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions_sanitized] Managed ledger is now ready to accept writes again
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Creating a new ledger after closed 535058
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions_sanitized] Delete complete for empty ledger 535053. rc=0
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Delete complete for empty ledger 535052. rc=0
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>], allBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>].
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>], allBookies [<Bookie:pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181>, <Bookie:pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181>].
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger with ensembleSize=3, writeQuorumSize=3 and ackQuorumSize=2
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Error creating ledger rc=-6 Not enough non-faulty bookies available
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0xe3c4b8be, L:/10.11.217.164:6650 - R:/10.11.233.64:37306] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-191, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x320c7da6, L:/10.11.217.164:6650 - R:/10.11.219.27:57008] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-193, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x96ac0dc3, L:/10.11.217.164:6650 - R:/10.11.233.64:37318] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-192, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x89f930ab, L:/10.11.217.164:6650 - R:/10.11.219.27:57010] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-194, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,322+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,323+0000 [pulsar-io-5-2] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions] Un-fencing topic...
2025-01-01T20:53:09,323+0000 [pulsar-io-5-2] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Managed ledger is now ready to accept writes again
2025-01-01T20:53:09,324+0000 [pulsar-io-5-1] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.233.64:37318
2025-01-01T20:53:09,325+0000 [pulsar-io-5-2] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57008
2025-01-01T20:53:09,325+0000 [pulsar-io-5-3] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57010
2025-01-01T20:53:09,325+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2025-01-01T20:53:09,326+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-38-1] WARN  org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack.
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-19-1] WARN  org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local, using default rack for it : /default-rack.
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-38-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,326+0000 [pulsar-registration-client-19-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2025-01-01T20:53:09,328+0000 [pulsar-io-5-2] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:58438
2025-01-01T20:53:09,463+0000 [pulsar-io-5-3] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57018] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=system:serviceaccount:a7u:ddbss-records-processor using authMethod=token, clientVersion=Pulsar-Java-v3.0.0, clientProtocolVersion=21, proxyVersion=4.0.1
2025-01-01T20:53:09,466+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57018] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x3133e9f2, L:/10.11.217.164:6650 - R:/10.11.219.27:57018] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, role: system:serviceaccount:a7u:ddbss-records-processor
2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] INFO  org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x3133e9f2, L:/10.11.217.164:6650 - R:/10.11.219.27:57018] [SR:10.11.219.27, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,469+0000 [pulsar-io-5-3] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic...
2025-01-01T20:53:09,474+0000 [pulsar-io-5-3] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.219.27:57018
2025-01-01T20:53:09,482+0000 [pulsar-io-5-4] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.230.148:60912] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=system:serviceaccount:a7u:ddbss-records-processor using authMethod=token, clientVersion=Pulsar-Java-v3.0.0, clientProtocolVersion=21, proxyVersion=4.0.1
2025-01-01T20:53:09,550+0000 [pulsar-io-5-3] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57036] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1
2025-01-01T20:53:09,553+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57036] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x90f74dd6, L:/10.11.217.164:6650 - R:/10.11.219.27:57036] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-193, producerId=0}, role: a7u-ddbss
2025-01-01T20:53:09,554+0000 [pulsar-io-5-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57030] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1
2025-01-01T20:53:09,555+0000 [pulsar-io-5-2] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.233.64:37332] connected role=system:serviceaccount:pulsar:pulsar-proxy and originalAuthRole=a7u-ddbss using authMethod=token, clientVersion=Pulsar-CPP-v3.5.1, clientProtocolVersion=20, proxyVersion=4.0.1
2025-01-01T20:53:09,555+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Creating a new ledger
2025-01-01T20:53:09,555+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering to Placement Policy. quarantinedBookies: []
2025-01-01T20:53:09,557+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.233.64:37332] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x563b128e, L:/10.11.217.164:6650 - R:/10.11.233.64:37332] [SR:10.11.233.64, state:Connected], producerName=pulsar-313-192, producerId=0}, role: a7u-ddbss
2025-01-01T20:53:09,566+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.219.27:57030] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions}, client=[id: 0x7c18fd74, L:/10.11.217.164:6650 - R:/10.11.219.27:57030] [SR:10.11.219.27, state:Connected], producerName=pulsar-313-194, producerId=0}, role: a7u-ddbss
2025-01-01T20:53:09,571+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-0000535064 (v. Optional.empty)
2025-01-01T20:53:09,575+0000 [broker-topic-workers-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.11.230.148:60912] Created new producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x6e4d4bd8, L:/10.11.217.164:6650 - R:/10.11.230.148:60912] [SR:10.11.230.148, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, role: system:serviceaccount:a7u:ddbss-records-processor
2025-01-01T20:53:09,578+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-0000535064 (v. Optional.empty)
2025-01-01T20:53:09,578+0000 [metadata-store-12-1] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 535064
2025-01-01T20:53:09,578+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Created new ledger 535064
2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] INFO  org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://a7u/ddbs/a7u_transactions_sanitized}, client=[id: 0x6e4d4bd8, L:/10.11.217.164:6650 - R:/10.11.230.148:60912] [SR:10.11.230.148, state:Connected], producerName=DynamoDBStreamProcessor-a7u_transactions-a7u_transactions_sanitized - 0d7bfcbc-724e-4674-aa90-8a48ec8d861a, producerId=0}, assignedBrokerLookupData: Optional.empty
2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Failed to persist msg in store: Not enough non-faulty bookies available error code: -6
2025-01-01T20:53:09,580+0000 [pulsar-io-5-4] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://a7u/ddbs/a7u_transactions_sanitized] Un-fencing topic...
2025-01-01T20:53:09,585+0000 [pulsar-io-5-4] INFO  org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.11.230.148:60912
2025-01-01T20:53:09,589+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/00/0053/L5058 (v. Optional.empty)
2025-01-01T20:53:09,602+0000 [pulsar-io-5-4] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181 [id: 0x3c1af6f7, L:/10.11.217.164:49698 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.11.225.176:3181]
2025-01-01T20:53:09,603+0000 [pulsar-io-5-4] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x3c1af6f7, L:/10.11.217.164:49698 - R:pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local/10.11.225.176:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2025-01-01T20:53:09,609+0000 [metadata-store-12-1] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/00/0053/L5058 (v. Optional.empty)
2025-01-01T20:53:09,609+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [a7u/ddbs/persistent/a7u_transactions] Delete complete for empty ledger 535058. rc=0

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@mrtworo mrtworo added the type/bug The PR fixed a bug or issue reported a bug label Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mrtworo