Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Plugins Support with WebAssembly #2056

Closed
wants to merge 128 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
ce31fe2
Plugins System: Multiple squashed commits
AmmarAbouZor Jun 14, 2024
65be020
Plugins: Parser Client Defs & Macro (in progress)
AmmarAbouZor Jul 10, 2024
94c0ffe
Plugins: Parser Client Exports & Macro (in progress)
AmmarAbouZor Jul 11, 2024
ab0f282
Plugin: Fixes & Adjustments to Parser Plugin Client
AmmarAbouZor Jul 12, 2024
ec5c19a
Parser Plugin Macro: Insure Parser trait in scope
AmmarAbouZor Jul 15, 2024
1390d65
Integration tests for parser export macro
AmmarAbouZor Jul 15, 2024
4d365c4
Plugins API: Add build check for enabled features
AmmarAbouZor Jul 15, 2024
80c1da8
Integration tests for parser export macro
AmmarAbouZor Jul 15, 2024
39ee2d3
Plugins API: Add warning for internal module + renaming.
AmmarAbouZor Jul 15, 2024
092a2fb
Plugins API: Adding logging functionality
AmmarAbouZor Jul 15, 2024
c19a15e
Plugins API: Adding logging functionality
AmmarAbouZor Jul 16, 2024
080435c
Plugins API: Adding logging functionality
AmmarAbouZor Jul 16, 2024
a8346f5
Plugins API: Adding logging functionality
AmmarAbouZor Jul 16, 2024
266aa2a
Plugins API: Adding logging functionality
AmmarAbouZor Jul 16, 2024
1d3a905
Plugins API: Adding logging functionality
AmmarAbouZor Jul 17, 2024
3b30d1a
Parser Plugin: Send collection of results as default
AmmarAbouZor Jul 17, 2024
3963c2d
Plugin API Docs: Configurations & Improvements
AmmarAbouZor Jul 17, 2024
8a4715e
Plugin API: Docs & Example for export parser macro
AmmarAbouZor Jul 18, 2024
b496298
Plugin API Parser: Small fixes for export macro
AmmarAbouZor Jul 18, 2024
6fa5ee1
Plugins: Use Add method on host & Clippy Fixes
AmmarAbouZor Jul 26, 2024
330e99a
Plugins: Rename host library to `Plugins_host`
AmmarAbouZor Jul 30, 2024
1efd2c4
Plugins: Improve documentation for Parser trait
AmmarAbouZor Jul 31, 2024
f7218dd
Plugins API: Bytesource WIT & Bindings & Macro
AmmarAbouZor Jul 31, 2024
48dd637
Plugins Parser: Integration tests fixes
AmmarAbouZor Jul 31, 2024
4dace50
Plugin Bytesource: Add integration tests
AmmarAbouZor Jul 31, 2024
12219ec
Plugins API: Insure tests run with all features
AmmarAbouZor Jul 31, 2024
e763db4
Plugins: Provide 'File Source' Plugin as example
AmmarAbouZor Jul 31, 2024
99a0c4b
Plugins: Refactoring & Cleaning up
AmmarAbouZor Aug 1, 2024
696d506
Plugins Host: Implementing Bytesource & Refactoring
AmmarAbouZor Aug 1, 2024
4f06e25
Plugins: Remove Seek form trait bound of ByteSource
AmmarAbouZor Aug 1, 2024
c94da3f
Plugins: Basic integration of ByteSource in host
AmmarAbouZor Aug 1, 2024
8e2d593
Plugins API: Add Input Source definitions
AmmarAbouZor Aug 2, 2024
8b3d30a
Plugins Host: Create Bytesource & Parser Adjustments
AmmarAbouZor Aug 2, 2024
c0695cd
Plugins: Implement Read method on Bytesource Host
AmmarAbouZor Aug 2, 2024
36dcc3f
Plugins: Remove Sync bound from ByteSource Trait
AmmarAbouZor Aug 2, 2024
7387d4e
Plugins Source Guest: Add missing export macro
AmmarAbouZor Aug 5, 2024
6ffc42a
Plugins Source Guest: Use uninitialized buffer
AmmarAbouZor Aug 5, 2024
ce25f63
Plugin: Add missing indexer CLI command
AmmarAbouZor Nov 7, 2024
82271eb
Plugins Parser: Adjustment according to master
AmmarAbouZor Nov 7, 2024
488f3aa
Plugins: Use dependencies from workspace
AmmarAbouZor Nov 7, 2024
3034403
Plugins: Add temporary basic benchmarks
AmmarAbouZor Nov 8, 2024
b58deb5
Plugins: Benchmark for parser plugin
AmmarAbouZor Nov 8, 2024
0254285
Plugin: Adjustments and comments due to benchmarks
AmmarAbouZor Nov 8, 2024
cd9fde8
Plugins: Add benchmark for compiling plugins
AmmarAbouZor Nov 11, 2024
ea4a3ef
Plugins Parser: Improve handling plugins' errors
AmmarAbouZor Nov 11, 2024
1da6d69
Plugins Parser: Remove method `parsing_with_add`
AmmarAbouZor Nov 11, 2024
a646a6a
Plugin Parser: Add invariant & security tests
AmmarAbouZor Nov 13, 2024
e6f55f3
Plugin Bytesource: Add invariant & security tests
AmmarAbouZor Nov 13, 2024
b6fe883
Plugins: String parser example & Improvement on Source
AmmarAbouZor Nov 15, 2024
4714179
Plugins: String parser errors on invalid UTF-8
AmmarAbouZor Nov 15, 2024
ef8502d
Plugins Parser: Multiple columns in parsed message
AmmarAbouZor Nov 25, 2024
ea0c6fb
Plugins API: Add Custom Configurations Schema
AmmarAbouZor Nov 28, 2024
1dbdc3d
Plugins: Custom configs for string parser plugin
AmmarAbouZor Nov 28, 2024
49541b8
Plugins custom configurations on host
AmmarAbouZor Nov 28, 2024
fb14b4e
Plugins: Deliver config schemas on host & Refactor
AmmarAbouZor Nov 29, 2024
534d878
Plugins: Custom configurations for byte-source API
AmmarAbouZor Nov 29, 2024
27e0a6e
Plugins: Custom configs for file source example
AmmarAbouZor Nov 29, 2024
45c3985
Plugins: Custom configuration on byte-source host
AmmarAbouZor Nov 29, 2024
23c4a60
Plugins API: Add missing documentation warning
AmmarAbouZor Dec 2, 2024
3553a4a
Create Plugins Manger + Integration in unbound session
AmmarAbouZor Dec 2, 2024
9aece0e
Plugins Manger Inside unbound session temporally
AmmarAbouZor Dec 3, 2024
036abdc
Plugins: Add version to parser API
AmmarAbouZor Dec 3, 2024
f43606c
Plugins: Add version to byte-source API
AmmarAbouZor Dec 3, 2024
c53f22c
Plugins host: Plugins version support
AmmarAbouZor Dec 3, 2024
e493564
Plugins Parser: Render options on plugins API
AmmarAbouZor Dec 3, 2024
9e75dcc
Plugins Parser: Support for render options on host
AmmarAbouZor Dec 3, 2024
eb29efc
Plugins: Version & Render options for examples
AmmarAbouZor Dec 4, 2024
01eed59
Plugin Manger: Loading ... (in progress)
AmmarAbouZor Dec 4, 2024
8448a28
Plugin Manager: Loading ... (in progress)
AmmarAbouZor Dec 4, 2024
9b72324
Plugin Manager: Loading ... (in progress)
AmmarAbouZor Dec 5, 2024
6b94045
Plugin Manager: Loading ... (in progress)
AmmarAbouZor Dec 5, 2024
34861b4
Plugin Manager: Loading for bytesource
AmmarAbouZor Dec 5, 2024
dd7c2aa
Plugins Manger: Loading method in rust bindings
AmmarAbouZor Dec 5, 2024
7c9c632
Plugins Integrations in ts bindings
AmmarAbouZor Dec 6, 2024
30d28c9
Plugins Manager: Integrations front end
AmmarAbouZor Dec 12, 2024
2ea1f83
Plugins Service in client & Wire up with core
AmmarAbouZor Dec 13, 2024
598a838
Plugins Manager: Basic UI implementation.
AmmarAbouZor Dec 13, 2024
fa45a79
Plugins: Use types on typescript for plugins
AmmarAbouZor Dec 16, 2024
fd0f94e
Plugins: Create Byte-Source structure in platform
AmmarAbouZor Dec 16, 2024
973fe00
Plugin: Build & Integrate plugin parser in platform
AmmarAbouZor Dec 17, 2024
5c0911c
Plugins: Integration into sessions (in progress)
AmmarAbouZor Dec 17, 2024
bcfc88d
Plugins UI: Placeholder for parser configurations
AmmarAbouZor Dec 18, 2024
fda8ce8
Plugins UI: Parser with streams (in progress)
AmmarAbouZor Dec 18, 2024
e7d4221
Plugins UI: Configuration Schemas (in progress)
AmmarAbouZor Dec 18, 2024
1139df9
Plugins UI: Integrating config schema in UI
AmmarAbouZor Dec 19, 2024
67ed968
Plugins UI: Save config value to their entry
AmmarAbouZor Dec 19, 2024
e641021
Plugin UI: Add basic renderer for plugins.
AmmarAbouZor Dec 19, 2024
e121414
Plugins: Finish integrating plugin in system (prototyping)
AmmarAbouZor Dec 19, 2024
59e68b1
Plugins: UI Fix binding & Extend Example Plugins
AmmarAbouZor Dec 20, 2024
28945f8
Merge branch 'master' into plugins_support
AmmarAbouZor Jan 7, 2025
4f85206
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 7, 2025
a71ea6a
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 8, 2025
a971c5e
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 8, 2025
57a8927
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 8, 2025
fab4a8a
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 8, 2025
b54aca1
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 9, 2025
8532801
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 9, 2025
212003b
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 9, 2025
8e1b4b2
Plugins: Adjustments after merging BinCode...
AmmarAbouZor Jan 10, 2025
28a8953
Plugins: Update wasmtime & wit-bindgen.
AmmarAbouZor Jan 10, 2025
f8af289
Plugins Parser: Temp changes to run benchmarks.
AmmarAbouZor Jan 13, 2025
3f6611a
Plugins: Extend config items in string parser
AmmarAbouZor Jan 13, 2025
b48bcca
Plugins UI: Add config renderer for numbers.
AmmarAbouZor Jan 13, 2025
d929421
Plugins UI: Add config renderer for dropdown.
AmmarAbouZor Jan 13, 2025
d826adb
Plugins Configs: Add paths UI & Adjustments.
AmmarAbouZor Jan 13, 2025
477c946
Plugins Configs: Separate files and dirs & Tooltips
AmmarAbouZor Jan 14, 2025
d441a15
Plugins: Extend render options & Their UI Implementation
AmmarAbouZor Jan 15, 2025
1546966
Plugins: Add render options to string parser.
AmmarAbouZor Jan 15, 2025
4939419
Plugins Parser: UI Integration with files (in progress)
AmmarAbouZor Jan 16, 2025
02b5c9c
Plugins Parser: UI Integration with files (done)
AmmarAbouZor Jan 17, 2025
1b85cdb
Plugins: Parser on all stream formats.
AmmarAbouZor Jan 17, 2025
0f268de
Plugins: Search results & Todos & Cleanup
AmmarAbouZor Jan 17, 2025
67f5058
Plugins: Fix warnings.
AmmarAbouZor Jan 17, 2025
f7cdab4
Plugins: Split WIT into three packages & Fixes
AmmarAbouZor Jan 21, 2025
2eac9b0
Plugins: API Documentations & Cleanup
AmmarAbouZor Jan 23, 2025
f2ac717
Plugins: Apply changes for shared types in host.
AmmarAbouZor Jan 23, 2025
955723d
Plugins: Functions for retrieving configuration values
AmmarAbouZor Jan 23, 2025
2434293
Plugins: Owning variant for some config getter functions.
AmmarAbouZor Jan 24, 2025
43e56c2
Plugins: Move example to root directory.
AmmarAbouZor Jan 24, 2025
026b72f
Plugins: Export only needed logging functions.
AmmarAbouZor Jan 24, 2025
e399436
Plugins: Adding DLT parser plugin.
AmmarAbouZor Jan 27, 2025
7ee8572
Merge branch 'master' into plugins_support
AmmarAbouZor Feb 5, 2025
1c947d0
Plugins: Fixes after merging master.
AmmarAbouZor Feb 5, 2025
799ddc1
Plugins: Configuration default values on plugins side
AmmarAbouZor Feb 5, 2025
a463d19
Plugins: Configs default values on host.
AmmarAbouZor Feb 6, 2025
3dc3202
Plugins: README for plugin development.
AmmarAbouZor Feb 7, 2025
833cf90
Plugins: Resolve TODOs & Refactoring & Documentation.
AmmarAbouZor Feb 7, 2025
7991de9
Merge branch 'master' into plugins_support
AmmarAbouZor Feb 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Plugins: Parser Client Exports & Macro (in progress)
- Define which types from bindings will be exported to the plugins
  implementers
- Export types needed in export macros as hidden
- Modify generate! Macro attributes to make its export macro public and
  set the bindings types for its needed types
- Set all types withing export macros with their full path.
AmmarAbouZor committed Dec 11, 2024
commit 94c0ffec9e4500cbe87c4303e9f2ef96e8d6aec4
3 changes: 0 additions & 3 deletions application/apps/indexer/plugins_api/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,2 @@
//TODO: public export the needed types only here instead the broad export.
pub mod bytesource;
pub mod parser;

pub use parser::Parser;
59 changes: 39 additions & 20 deletions application/apps/indexer/plugins_api/src/parser/mod.rs
Original file line number Diff line number Diff line change
@@ -3,17 +3,34 @@

use std::path::{Path, PathBuf};

use wit_bindgen::generate;
use self::bindings::exports::chipmunk::plugin::parser::Guest;

mod bindings {
use super::*;

wit_bindgen::generate!({
path: "wit/v_0.1.0",
world: "parse-plugin",
export_macro_name: "export_intern",
// Export macro is used withing the exported `parser_export!` macro and must be public
pub_export_macro: true,
// Bindings for export macro must be set, because it won't be called from withing the
// same module where `generate!` is called
default_bindings_module: "crate::parser::bindings",
});
}

use self::{
chipmunk::plugin::{parse_types::*, shared_types::*},
exports::chipmunk::plugin::parser::Guest,
// External exports for users
pub use bindings::chipmunk::plugin::{
parse_types::{Attachment, ParseError, ParseReturn, ParseYield, ParserConfig},
shared_types::InitError,
};
// Exports needed in macro but not needed by users
#[doc(hidden)]
pub use bindings::{
add as add_res_intern, export_intern,
exports::chipmunk::plugin::parser::Guest as PluginInternalGuest,
};

generate!({
path: "wit/v_0.1.0",
world: "parse-plugin",
});

/// Chipmunk Parser Plugin
pub trait Parser {
@@ -57,18 +74,18 @@ impl Parser for Dummy {
#[macro_export]
macro_rules! parser_export {
($par:ty) => {
//TODO AAZ: have all the types with the full path after defining the exports from wit file
static mut PARSER: ::std::option::Option<$par> = ::std::option::Option::None;

struct PluginParser;
// Name intentionally lengthened to avoid conflict with user's own types
struct InternalPluginParserGuest;

impl Guest for PluginParser {
impl $crate::parser::PluginInternalGuest for InternalPluginParserGuest {
/// Initialize the parser with the given configurations
fn init(
general_configs: ParserConfig,
general_configs: $crate::parser::ParserConfig,
plugin_configs: ::std::option::Option<::std::string::String>,
) -> ::std::result::Result<(), InitError> {
let parser = <$par as $crate::Parser>::create(
) -> ::std::result::Result<(), $crate::parser::InitError> {
let parser = <$par as $crate::parser::Parser>::create(
general_configs,
plugin_configs.map(|path| path.into()),
)?;
@@ -81,27 +98,29 @@ macro_rules! parser_export {

/// Parse the given bytes returning a list of plugins results
fn parse(
data: _rt::Vec<u8>,
data: ::std::vec::Vec<u8>,
timestamp: ::std::option::Option<u64>,
) -> _rt::Vec<Result<ParseReturn, ParseError>> {
) -> ::std::vec::Vec<
::std::result::Result<$crate::parser::ParseReturn, $crate::parser::ParseError>,
> {
// SAFETY: Parse method has mutable reference to self and can't be called more than
// once on the same time on host
let parser = unsafe { PARSER.as_mut().expect("parser already initialized") };
parser.parse(&data, timestamp).into_iter().collect()
}

/// Parse the given bytes returning the results to the host one by one using the function `add` provided by the host.
fn parse_with_add(data: _rt::Vec<u8>, timestamp: Option<u64>) {
fn parse_with_add(data: ::std::vec::Vec<u8>, timestamp: ::std::option::Option<u64>) {
// SAFETY: Parse method has mutable reference to self and can't be called more than
// once on the same time on host
let parser = unsafe { PARSER.as_mut().expect("parser already initialized") };
for item in parser.parse(&data, timestamp) {
add(item.as_ref());
$crate::parser::add_res_intern(item.as_ref());
}
}
}

export!(PluginParser);
$crate::parser::export_intern!(InternalPluginParserGuest);
};
}