forked from OpenAtomFoundation/pika
-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
map/set/zset/list data_cf distributed compaction #15
Comments
rockeet
added a commit
to rockeet/blackwidow
that referenced
this issue
Sep 12, 2023
rockeet
added a commit
that referenced
this issue
Sep 12, 2023
rockeet
added a commit
to rockeet/blackwidow
that referenced
this issue
Sep 12, 2023
rockeet
added a commit
that referenced
this issue
Sep 12, 2023
rockeet
added a commit
to rockeet/blackwidow
that referenced
this issue
Sep 13, 2023
rockeet
added a commit
that referenced
this issue
Sep 13, 2023
rockeet
added a commit
to rockeet/blackwidow
that referenced
this issue
Sep 13, 2023
rockeet
added a commit
that referenced
this issue
Sep 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
data_cf distributed compaction needs
DB::Get
fromdefault
cf inCompactionFilter
, we bulk load all hash keys fromdefault
cf and save to shared filesystem for reading in dcompact workers.When compaction input of
dafa_cf
is small but hash keys indefault
cf is large, this is a very big waste, --- when compact upper levels ofdata_cf
, this may be likely happens.So we should check data size in
default
cf before starting remote compaction, if it reaches a threshold of a percent of compaction input ofdafa_cf
, it should fallback to local compact.Compaction::column_family_data()
to getdefault
cf handle and DB ptr, this needs a globalstd::map
DB::GetApproximateSizes()
to get size indefault
cf.Thus a customized
CompactionExecutorFactory
should be defined -- it should referencesDcompactEtcd
factory and forward methods, the key point is to overrideShouldRunLocal(compaction)
.The text was updated successfully, but these errors were encountered: