STR-985: Make ArbitraryGenerator generate more entropy on the fly

[storopoli]

Description

The default was 65kb, which was even called for u64, e.g. BitcoinAmount.
So let's make ArbitraryGenerator reasonable where the default is 1kb and calling new_with_size() for bigger sizes.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactor
• New or updated tests
• Dependency Update

Checklist

• I have performed a self-review of my code.
• I have commented my code where necessary.
• I have updated the documentation if needed.
• My changes do not introduce new warnings.
• I have added (where necessary) tests that prove my changes are effective or that my feature works.
• New and existing tests pass with my changes.

Related Issues

STR-985

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 54.69%. Comparing base (b7d69ea) to head (9c89ee3).

@@            Coverage Diff             @@
##             main     #674      +/-   ##
==========================================
+ Coverage   54.68%   54.69%   +0.01%     
==========================================
  Files         318      318              
  Lines       34047    34055       +8     
==========================================
+ Hits        18619    18627       +8     
  Misses      15428    15428              

Files with missing lines	Coverage Δ
bin/prover-client/src/task_tracker.rs	76.61% <100.00%> (ø)
bin/strata-client/src/extractor.rs	94.12% <100.00%> (-0.21%)	⬇️
crates/btcio/src/reader/query.rs	47.57% <100.00%> (ø)
crates/btcio/src/reader/tx_indexer.rs	91.66% <100.00%> (+0.03%)	⬆️
crates/btcio/src/test_utils.rs	61.45% <100.00%> (ø)
crates/btcio/src/writer/task.rs	31.10% <100.00%> (ø)
crates/chaintsn/src/transition.rs	82.86% <100.00%> (ø)
crates/evmexec/src/engine.rs	65.74% <100.00%> (ø)
crates/l1tx/src/filter/mod.rs	96.26% <100.00%> (+0.02%)	⬆️
crates/primitives/src/l1/btc.rs	82.44% <100.00%> (ø)
... and 14 more

... and 1 file with indirect coverage changes

[github-actions]

Commit: 60a10d1

SP1 Performance Test Results

program	cycles	success
BTC_BLOCKSPACE	30,370,506	✅
EL_BLOCK	97,860	✅
CL_BLOCK	89,672	✅
L1_BATCH	30,419,728	✅
L2_BATCH	3,432	✅
CHECKPOINT	25,784	✅

[delbonis]

The way the ticket was written, we can keep the comfortable buffer sizes (or even increase the default), and it would simply only generate the precise amount of randomness requested when asked for it.

We should keep track of how much entropy is being asked for and generate it on the fly into a preallocated buffer. This buffer won’t be expanded (as that could break lifetimes), but we’ll only generate the randomness as needed.

^This is really the right way to do it, which I should have done when I reworked it last. Sure we can keep new_with_size, but that would just change the size of the backing buffer.

[bewakes]

• One thing that crossed my mind is that if we want dynamic size estimation, we could probably use std::mem::size_of::<T>(). But this wouldn't work for structs that contain heap allocated items like Vec. I really wish there was some way around this, because otherwise we have to try hard to estimate the size of the resulting object.
• Another thing is that why are we opting for exact sizes instead of bit operations(like 1 << 12) while using new_with_size(). The latter seems easier to reason about just by looking at it and we can also easily control how many bits/bytes we are allocating.

[storopoli]

@delbonis so the solution is doing the reverse of what I did?
Keep the big default and move everything to new() and not new_with_size()?

[delbonis]

@bewakes I agree it would be better if it worked like that. But the Arbitrary interface demands borrowed slices so you can generate types like struct Foo<'a>(&'a [u8]);. Also yeah it might have been slightly simpler to define the default size with 1 << n.

@storopoli It's still good practice to constrain it where it's known to be safe so we use less memory. But this would be a more complete solution.