Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

forknet: Drop unneeded columns in the fork-network command #12921

Merged
merged 2 commits into from
Feb 14, 2025

Conversation

marcelo-gonzalez
Copy link
Contributor

fork-network finalize deletes data in every column except the ones we need to start the network, but it does it with a delete DB transaction. This only logically deletes the values, but does not free up the disk space, which will only happen after a compaction. So here we remove the columns by calling drop_cf() on the unnecessary columns, which actually removes the sst files and frees up the space. This will reduce the size of images used for forknet by quite a bit.

@marcelo-gonzalez
Copy link
Contributor Author

I tested it with this branch. That adds a script at pytest/tests/sanity/forknet_bug.py to generate state for fork-network, and then a bash script at pytest/tests/sanity/forknet_run.sh to run the fork network commands. It also adds some code that will print what's in the db at different points of the fork-network commands

To test it I ran:

$ python3 tests/sanity/forknet_bug.py
$ bash pytest/tests/sanity/forknet_run.sh ~/nearcore/target/debug/neard ~/.near/test0_finished/ /tmp/fork-after

Then checkout HEAD~ for the commit without this PR, and again run:

$ bash pytest/tests/sanity/forknet_run.sh ~/nearcore/target/debug/neard ~/.near/test0_finished/ /tmp/fork-before

Then check there is no diff between these files:

$ diff /tmp/fork-before/init-db-values.txt /tmp/fork-after/init-db-values.txt
$ diff /tmp/fork-before/finalize-post-db-values.txt /tmp/fork-after/finalize-post-db-values.txt

Copy link

codecov bot commented Feb 12, 2025

Codecov Report

Attention: Patch coverage is 13.33333% with 52 lines in your changes missing coverage. Please review.

Project coverage is 70.50%. Comparing base (ecc1373) to head (0fd1a69).
Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
tools/fork-network/src/cli.rs 0.00% 27 Missing ⚠️
core/store/src/opener.rs 30.76% 18 Missing ⚠️
core/store/src/db/rocksdb.rs 0.00% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #12921      +/-   ##
==========================================
+ Coverage   70.48%   70.50%   +0.02%     
==========================================
  Files         851      851              
  Lines      174917   175032     +115     
  Branches   174917   175032     +115     
==========================================
+ Hits       123285   123415     +130     
+ Misses      46518    46482      -36     
- Partials     5114     5135      +21     
Flag Coverage Δ
backward-compatibility 0.36% <0.00%> (-0.01%) ⬇️
db-migration 0.36% <0.00%> (-0.01%) ⬇️
genesis-check 1.42% <0.00%> (?)
linux 70.33% <13.33%> (-0.01%) ⬇️
linux-nightly 70.14% <13.33%> (-0.05%) ⬇️
pytests 1.73% <0.00%> (+1.37%) ⬆️
sanity-checks 1.54% <0.00%> (?)
unittests 70.34% <13.33%> (-0.04%) ⬇️
upgradability 0.36% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@Longarithm Longarithm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment on lines 671 to 672
let (_hot_db, _hot_snapshot, _cold_db, _cold_snapshot) =
opener.open_dbs(Mode::ReadWriteExisting)?;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let (_hot_db, _hot_snapshot, _cold_db, _cold_snapshot) =
opener.open_dbs(Mode::ReadWriteExisting)?;
let _ = opener.open_dbs(Mode::ReadWriteExisting)?;

nit

@marcelo-gonzalez marcelo-gonzalez added this pull request to the merge queue Feb 14, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Feb 14, 2025
@marcelo-gonzalez marcelo-gonzalez added this pull request to the merge queue Feb 14, 2025
Merged via the queue into near:master with commit 489303d Feb 14, 2025
26 of 29 checks passed
@marcelo-gonzalez marcelo-gonzalez deleted the drop-columns branch February 14, 2025 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants