Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust quorum intersection checker integration #4629

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Rust quorum intersection checker integration #4629

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Builds/VisualStudio/stellar-core.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -521,6 +521,7 @@ exit /b 0
<ClCompile Include="..\..\src\herder\PendingEnvelopes.cpp" />
<ClCompile Include="..\..\src\herder\QuorumIntersectionCheckerImpl.cpp" />
<ClCompile Include="..\..\src\herder\QuorumTracker.cpp" />
<ClCompile Include="..\..\src\herder\RustQuorumCheckerAdaptor.cpp" />
<ClCompile Include="..\..\src\herder\SurgePricingUtils.cpp" />
<ClCompile Include="..\..\src\herder\test\HerderTests.cpp" />
<ClCompile Include="..\..\src\herder\test\PendingEnvelopesTests.cpp" />
Expand Down Expand Up @@ -976,6 +977,7 @@ exit /b 0
<ClInclude Include="..\..\src\herder\QuorumIntersectionChecker.h" />
<ClInclude Include="..\..\src\herder\QuorumIntersectionCheckerImpl.h" />
<ClInclude Include="..\..\src\herder\QuorumTracker.h" />
<ClInclude Include="..\..\src\herder\RustQuorumCheckerAdaptor.h" />
<ClInclude Include="..\..\src\herder\SurgePricingUtils.h" />
<ClInclude Include="..\..\src\herder\test\TestTxSetUtils.h" />
<ClInclude Include="..\..\src\herder\TransactionQueue.h" />
Expand Down
6 changes: 6 additions & 0 deletions Builds/VisualStudio/stellar-core.vcxproj.filters
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -696,6 +696,9 @@
<ClCompile Include="..\..\src\herder\QuorumTracker.cpp">
<Filter>herder</Filter>
</ClCompile>
<ClCompile Include="..\..\src\herder\RustQuorumCheckerAdaptor.cpp">
<Filter>herder</Filter>
</ClCompile>
<ClCompile Include="..\..\src\herder\SurgePricingUtils.cpp">
<Filter>herder</Filter>
</ClCompile>
Expand Down Expand Up @@ -1892,6 +1895,9 @@
<ClInclude Include="..\..\src\herder\QuorumTracker.h">
<Filter>herder</Filter>
</ClInclude>
<ClInclude Include="..\..\src\herder\RustQuorumCheckerAdaptor.h">
<Filter>herder</Filter>
</ClInclude>
<ClInclude Include="..\..\src\herder\SurgePricingUtils.h">
<Filter>herder</Filter>
</ClInclude>
Expand Down
45 changes: 45 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions docs/versioning-soroban.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ We are leveraging Rust's support for linking together multiple copies of "the
same" library (soroban) with different versions, but we are doing so somewhat
against the grain of how cargo normally wants to do it.

Do do this "the normal way", we would just list the different versions of the
To do this "the normal way", we would just list the different versions of the
soroban crate in `Cargo.toml`, and then when we built it cargo would attempt to
resolve all the dependencies and transitive-dependencies of all those soroban
versions into a hopefully-minimal set of crates and download, compile and link
Expand All @@ -165,7 +165,7 @@ This has one minor and one major problem:
p22 module on foo 0.1, cargo will bump _both_ to foo 0.2, which _changes_
the semantics of the p22 module.

- We initially though a way out of this is to add redundant exact-version
- We initially thought a way out of this is to add redundant exact-version
dependencies (like `foo = "=0.2"`) to `Cargo.toml` for
`soroban-env-host` but there turn out to be both a minor and a major
problem with that too.
Expand Down
89 changes: 78 additions & 11 deletions src/herder/HerderImpl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
#include "herder/HerderUtils.h"
#include "herder/LedgerCloseData.h"
#include "herder/QuorumIntersectionChecker.h"
#include "herder/RustQuorumCheckerAdaptor.h"
#include "herder/TxSetFrame.h"
#include "herder/TxSetUtils.h"
#include "ledger/LedgerManager.h"
Expand Down Expand Up @@ -261,6 +262,13 @@ HerderImpl::newSlotExternalized(bool synchronous, StellarValue const& value)
safelyProcessSCPQueue(synchronous);
}

void
HerderImpl::interrupt_quorum_checker()
{
mLastQuorumMapIntersectionState.mInterruptFlag = true;
mLastQuorumMapIntersectionState.mInterrupt->fire();
}

void
HerderImpl::shutdown()
{
Expand All @@ -273,7 +281,7 @@ HerderImpl::shutdown()
// avoid a long pause joining worker threads.
CLOG_DEBUG(Herder,
"Shutdown interrupting quorum transitive closure analysis.");
mLastQuorumMapIntersectionState.mInterruptFlag = true;
interrupt_quorum_checker();
}
mTransactionQueue.shutdown();
if (mSorobanTransactionQueue)
Expand Down Expand Up @@ -1883,7 +1891,7 @@ HerderImpl::checkAndMaybeReanalyzeQuorumMap()
CLOG_DEBUG(Herder, "Transitive closure of quorum has "
"changed, interrupting existing "
"analysis.");
mLastQuorumMapIntersectionState.mInterruptFlag = true;
interrupt_quorum_checker();
}
}
else
Expand All @@ -1897,33 +1905,76 @@ HerderImpl::checkAndMaybeReanalyzeQuorumMap()
auto& cfg = mApp.getConfig();
releaseAssert(threadIsMain());
auto seed = gRandomEngine();
auto qic = QuorumIntersectionChecker::create(
qmap, cfg, mLastQuorumMapIntersectionState.mInterruptFlag, seed);

auto ledger = trackingConsensusLedgerIndex();
auto nNodes = qmap.size();
auto& hState = mLastQuorumMapIntersectionState;
auto& app = mApp;
auto worker = [curr, ledger, nNodes, qic, qmap, cfg, seed, &app,
&hState] {
auto worker = [curr, ledger, nNodes, qmap, cfg, seed, &app, &hState] {
try
{
ZoneScoped;
bool ok = qic->networkEnjoysQuorumIntersection();
auto split = qic->getPotentialSplit();
bool useV2 = app.getConfig().USE_QUORUM_INTERSECTION_CHECKER_V2;
bool ok = false;
std::pair<std::vector<PublicKey>, std::vector<PublicKey>> split;
if (useV2)
{
ok = RustQuorumCheckerAdaptor::
networkEnjoysQuorumIntersection(
qmap, cfg, *hState.mInterrupt, split);
}
else
{
auto qic = QuorumIntersectionChecker::create(
qmap, cfg, hState.mInterruptFlag, seed);
ok = qic->networkEnjoysQuorumIntersection();
split = qic->getPotentialSplit();
}
std::set<std::set<PublicKey>> critical;
if (ok)
{
// Only bother calculating the _critical_ groups if we're
// intersecting; if not intersecting we should finish ASAP
// and raise an alarm.
critical = QuorumIntersectionChecker::
getIntersectionCriticalGroups(
qmap, cfg, hState.mInterruptFlag, seed);
if (useV2)
{
auto cb =
[&hState](
QuorumIntersectionChecker::QuorumSetMap const&
qSetMap,
std::optional<Config> const& config) -> bool {
std::pair<std::vector<PublicKey>,
std::vector<PublicKey>>
potential_split;
return RustQuorumCheckerAdaptor::
networkEnjoysQuorumIntersection(
qSetMap, config, *hState.mInterrupt,
potential_split);
};
critical = QuorumIntersectionChecker::
getIntersectionCriticalGroups(qmap, cfg, cb);
}
else
{
auto cb =
[&hState, seed](
QuorumIntersectionChecker::QuorumSetMap const&
qSetMap,
std::optional<Config> const& config) -> bool {
auto checker = QuorumIntersectionChecker::create(
qSetMap, config, hState.mInterruptFlag, seed,
/*quiet=*/true);
return checker->networkEnjoysQuorumIntersection();
};
critical = QuorumIntersectionChecker::
getIntersectionCriticalGroups(qmap, cfg, cb);
}
}
app.postOnMainThread(
[ok, curr, ledger, nNodes, split, critical, &hState] {
hState.mRecalculating = false;
hState.mInterruptFlag = false;
hState.mInterrupt->reset();
hState.mNumNodes = nNodes;
hState.mLastCheckLedger = ledger;
hState.mLastCheckQuorumMapHash = curr;
Expand All @@ -1945,10 +1996,26 @@ HerderImpl::checkAndMaybeReanalyzeQuorumMap()
[&hState] {
hState.mRecalculating = false;
hState.mInterruptFlag = false;
hState.mInterrupt->reset();
hState.mCheckingQuorumMapHash = Hash{};
},
"QuorumIntersectionChecker interrupted");
}
catch (const RustQuorumCheckerError& e)
{
CLOG_DEBUG(Herder,
"Quorum transitive closure analysis failed due to "
"Rust solver error: {}",
e.what());
app.postOnMainThread(
[&hState] {
hState.mRecalculating = false;
hState.mInterruptFlag = false;
hState.mInterrupt->reset();
hState.mCheckingQuorumMapHash = Hash{};
},
"QuorumIntersectionChecker rust error");
}
};
mApp.postOnBackgroundThread(worker, "QuorumIntersectionChecker");
}
Expand Down
11 changes: 10 additions & 1 deletion src/herder/HerderImpl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
#include "herder/PendingEnvelopes.h"
#include "herder/TransactionQueue.h"
#include "herder/Upgrades.h"
#include "rust/RustBridge.h"
#include "util/Timer.h"
#include "util/UnorderedMap.h"
#include "util/XDROperators.h"
Expand All @@ -30,7 +31,7 @@ constexpr uint32 const SOROBAN_TRANSACTION_QUEUE_SIZE_MULTIPLIER = 2;
class Application;
class LedgerManager;
class HerderSCPDriver;

class InterruptGuard;
/*
* Is in charge of receiving transactions from the network.
*/
Expand Down Expand Up @@ -338,7 +339,13 @@ class HerderImpl : public Herder
Hash mLastCheckQuorumMapHash{};
Hash mCheckingQuorumMapHash{};
bool mRecalculating{false};

// for v1 (QuorumIntersectionChecker)
std::atomic<bool> mInterruptFlag{false};
// for v2 (rust quorum checker)
rust::Box<rust_bridge::quorum_checker::Interrupt> mInterrupt{
rust_bridge::quorum_checker::new_interrupt()};

std::pair<std::vector<PublicKey>, std::vector<PublicKey>>
mPotentialSplit{};
std::set<std::set<PublicKey>> mIntersectionCriticalNodes{};
Expand All @@ -357,6 +364,8 @@ class HerderImpl : public Herder
};
QuorumMapIntersectionState mLastQuorumMapIntersectionState;

void interrupt_quorum_checker();

State mState;
void setState(State st);

Expand Down
50 changes: 49 additions & 1 deletion src/herder/HerderUtils.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@
// of this distribution or at http://www.apache.org/licenses/LICENSE-2.0

#include "herder/HerderUtils.h"
#include "crypto/KeyUtils.h"
#include "main/Config.h"
#include "rust/RustVecXdrMarshal.h"
#include "scp/Slot.h"
#include "xdr/Stellar-ledger.h"
#include <algorithm>
#include <xdrpp/marshal.h>

namespace stellar
{
Expand Down Expand Up @@ -40,4 +42,50 @@ getStellarValues(SCPStatement const& statement)

return result;
}

// Render `id` as a short, human readable string. If `cfg` has a value, this
// function uses `cfg` to render the string. Otherwise, it returns the first 5
// hex values `id`.
std::string
toShortString(std::optional<Config> const& cfg, NodeID const& id)
{
if (cfg)
{
return cfg->toShortString(id);
}
else
{
return KeyUtils::toShortString(id).substr(0, 5);
}
}

QuorumIntersectionChecker::QuorumSetMap
toQuorumIntersectionMap(QuorumTracker::QuorumMap const& qmap)
{
QuorumIntersectionChecker::QuorumSetMap ret;
for (auto const& elem : qmap)
{
ret[elem.first] = elem.second.mQuorumSet;
}
return ret;
}

std::pair<std::vector<PublicKey>, std::vector<PublicKey>>
toQuorumSplitNodeIDs(QuorumSplit& split)
{
std::vector<NodeID> leftNodes;
leftNodes.reserve(split.left.size());
for (const auto& str : split.left)
{
leftNodes.push_back(KeyUtils::fromStrKey<NodeID>(std::string(str)));
}
std::vector<NodeID> rightNodes;
rightNodes.reserve(split.right.size());
for (const auto& str : split.right)
{
rightNodes.push_back(KeyUtils::fromStrKey<NodeID>(std::string(str)));
}
return std::make_pair(std::move(leftNodes), std::move(rightNodes));
}

}
Loading
Loading