[ProjectTracking]: New bridge indexer

[akhi3030]

Description

For more than a year already, there were two separate components to provide the observability of Rainbow Bridge operations: Bridge Monitoring and Bridge Indexer. The decoupling of these components in terms of development, along with some erroneous design decisions, made the system not reliable enough and hard to support.

As a result of this, the decision was made to join two components and update the technological stack to meet the following goals.

Goals

• Improved Stability: Reduce potential points of failure and improve overall system reliability.
• Cost Efficiency: Utilize cost-effective tools and potentially eliminate unnecessary infrastructure.
• Real-time Data Processing: Process and enrich Bridge transactions in real-time to provide up-to-date information.
• On-demand Reprocessing: Enable reprocessing and enrichment of historical data as needed.
• High Data Reliability: Maintain 100% data collection and validity.
• Availability: Maintain 99.9% system availability (existing requirement).

Links to external documentation and discussions

• New Bridge Indexer project (includes board and roadmap).
• Project wiki

Estimated effort

• Estimated project duration: Six months, which could be extended due to the new Omni-bridge.
• Target date: end of October.
• Roadmap

Assumptions

• There are cost-effective and reliable data sources for Ethereum and NEAR Protocol blockchain data.

Pre-requisites

Initial pre-requisites:

• MongoDB cluster for data storage (satisfied).
• Web server (satisfied).

Out of scope

• Major changes and the support of the API.

[kisialiou]

During the last 2 weeks, we worked on the following issue.

Its main goals were:

• introduce the first level of processing for the indexed data;
• add various architecture-related things, like duplicate handling and graceful processing restart.

The work resulted in the PR which is currently in the process of fixing several minor comments.

Immediate plans include:

• the launch of the processing;
• migration of the current API for the Bridge frontend to the new database (produced by the new indexer).

See more details in the corresponding project roadmap.

[kisialiou]

March 22nd update

Completed:

• Finalized the implementation the first-level processors for indexed bridge data, along with architectural improvements.
• Updated one endpoint of NEAR Minimal API to work with a new indexer database => not deployed yet due to the lack of testnet data.
• Launched processing of the indexed NEAR data.
• Prepared indexer and processor to work with testnet data.

Immediate plans:

1. Deployment of updated API.
2. Establishing monitoring for indexer/processor hangs.

See more details in the corresponding project roadmap.

[kisialiou]

April 5th update

Completed:

• Fixed issues preventing the deployment of new bridge indexer version with testnet and added more stability features.
• Deployed new solutions: updated indexer for mainnet; indexer for testnet; mainnet and testnet processors.
• Finalized, tested and deployed the update of Bridge NEAR API to work with a new database.
• Investigated and fixed bugs, causing missed transactions.
• Updated Fast Bridge liquidity report generator to use database produced by the new indexer.

Immediate plans:

1. Establishing monitoring for indexer/processor hangs.
2. Integration of Ethereum indexer.

Shift in dealines: due to appeared bugs and unexpected development, the expected deadline is shifted from end of May to the beginning of June (~5-10th of June).

See more details in the corresponding project roadmap.

[kisialiou]

April 19th update

Completed:

• Added CI to the project and refactored the code.
• Added automatic build and deployment of the solution with Makefile.
• Establishing monitoring for indexer/processor hangs through Grafana dashboards and alerts.
• Started the implementation of the enricher.

Immediate plans:

1. Finalize the enricher implementation and migrate all existing tools to the new data.

See more details in the corresponding project roadmap.

[kisialiou]

May 5th update

Worked on the enricher implementation. Namely:

• Implemented tokens data fetching from Coingecko + caching + handling of missing tokens.
• Implemented batch price fetching from Coingecko to reduce the number of requests.
• Defined and implemented enrichment process tracking.
• Refactored and cleaned the code (error handling, logical grouping etc.).

Immediate plans:

1. Test the current solution.
2. Implement transaction "matching" functionality.

Unfortunately, due to underestimated feature complexity and national holidays, the development of the enricher appeared to be longer than expected, thus shifting overall project delivery date by ~10 days, i.e. to ~20th of June.

Nevertheless, I plan to finish the project still as expected ~10th of June.

[kisialiou]

May 17th update

Due to my short vacation, national holidays and work on other tasks, there is little to no progress on the indexer at the moment. I plan to return to it next week after accomplishing more prioritized tasks.

[kisialiou]

May 31st update

During the previous week, I was mostly concentrated on eth-connector migration monitoring tools which again postponed the work on the indexer. Still, in connection with the emergence of a new NEP-141 connector, support for it was added to the indexer.

[kisialiou]

June 14th update

Completed:

1. Finished the implementation of enricher and transactions linking.
2. Deployed the enricher.
3. Completely migrated Bridge API to the new indexer data.
4. Created new panel and alerts to monitor enrichment process.
5. Wrote README + documentation for all components of new bridge indexer.

After 2 week vacation (back at work on 07/01):

1. Start the implementation of Ethereum indexer.

[kisialiou]

July 12th update

Completed:

1. Refined the code for processing of Aurora-related transactions.
2. Updated enricher to work with Aurora-related transactions.
3. Migrated and redeployed indexer to the new address of EthConnector.

Immediate plans:

1. Finalize the support of Aurora-related transactions processing and deploy.
2. Create a new Bridge dashboard with enhanced capabilities (e.g. search by tx hash), relying on the new indexer.

[kisialiou]

July 26th update

Completed:

1. Finalized the support of Aurora-related transactions processing and deployed.
2. Added support for linking Fast Bridge transactions.
3. Added minor improvements and bug fixes to current indexer and processor implementations.
4. Added minor changes, ensuring correct processes restart after server crashes.

Immediate plans:

1. Add Fast Bridge alerts, relying on the new indexer.
2. Create a new Bridge dashboard with enhanced capabilities (e.g. search by tx hash), relying on the new indexer.

[kisialiou]

August 9th update

Completed:

1. Set up monitoring for RAM and CPU on the server.
2. Deployed new version and resolved issues for Fast Bridge testnet indexing. Made Fast Bridge mainnet reprocessing with new data.
3. Wrote, tested and deployed alerts for Fast Bridge with a new indexer.
4. Explored alternatives to Grafana for dashboards: Metabase, PowerBI.
5. Created a "Rainbow Bridge Explorer" dashboard in Grafana, providing search and filtering capabilities over Rainbow Bridge transactions. The dashboards rely on new indexer data.

Immediate plans:

1. Optimize speed for the "Rainbow Bridge Explorer" dashboard and refine the interface.
2. Create a dashboard with metrics for the connector's statistics.

[kisialiou]

August 22nd update

Completed:

1. Finalized query optimization and formatting for the "Rainbow Bridge Explorer" dashboard. Achieved nearly instant result loads for most queries.
2. Investigated and reported a bug in Coingecko which caused incorrect data in the indexer.
3. Added alerting for BTC light client on testnet.
4. Detected and investigated problems with missing data.

Immediate plans:

1. Add support for new methods to cover the missing data.
2. Create a dashboard with metrics for the Ethereum connector's statistics.