Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove buf redux #2179

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
439 changes: 211 additions & 228 deletions application/apps/indexer/Cargo.lock

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion application/apps/indexer/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ members = [
"addons/someip-tools",
"addons/file-tools",
"addons/text_grep",
"addons/bufread",
"indexer_base",
"indexer_cli",
"merging",
Expand All @@ -28,7 +29,6 @@ dlt-core = "0.18.0"
crossbeam-channel = "0.5"
futures = "0.3"
tokio-util = "0.7"
buf_redux = "0.8"
regex = "1"
grep-regex = "0.1"
rand = "0.8"
Expand All @@ -44,6 +44,7 @@ envvars = "0.1"
# Support for `html_reports` needs running the benchmarks via `cargo-criterion` tool.
criterion = { version = "0.5", features = ["html_reports"] }
insta = { version = "1.41", features = ["yaml"] }
proptest = "1.6"

# `insta` crate and its dependency `similar` will be compiled once and run many times,
# therefore it's suggested to compile them with more optimizations for faster runs.
Expand Down
19 changes: 19 additions & 0 deletions application/apps/indexer/addons/bufread/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[package]
name = "bufread"
version = "0.1.1"
edition = "2021"

[dependencies]

[dev-dependencies]
criterion.workspace = true
proptest.workspace = true
rand.workspace = true

[[bench]]
name = "buffer_benches"
harness = false

[[bench]]
name = "reader_benches"
harness = false
2 changes: 2 additions & 0 deletions application/apps/indexer/addons/bufread/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# bufread
A buffered reader implementation in Rust with special behavior.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to provide more infos in the readme?

23 changes: 23 additions & 0 deletions application/apps/indexer/addons/bufread/benches/bench_config.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
// Copyright (c) 2025 ESR Labs GmbH. All rights reserved.
//
// NOTICE: All information contained herein is, and remains
// the property of E.S.R.Labs and its suppliers, if any.
// The intellectual and technical concepts contained herein are
// proprietary to E.S.R.Labs and its suppliers and may be covered
// by German and Foreign Patents, patents in process, and are protected
// by trade secret or copyright law.
// Dissemination of this information or reproduction of this material
// is strictly forbidden unless prior written permission is obtained
// from E.S.R.Labs.

use criterion::Criterion;
use std::time::Duration;

pub fn bench_config(sample_size: usize) -> Criterion {
Criterion::default()
.warm_up_time(Duration::from_secs(5))
.measurement_time(Duration::from_secs(10))
.sample_size(sample_size)
.significance_level(0.01)
.noise_threshold(0.05)
}
62 changes: 62 additions & 0 deletions application/apps/indexer/addons/bufread/benches/buffer_benches.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
// Copyright (c) 2025 ESR Labs GmbH. All rights reserved.
//
// NOTICE: All information contained herein is, and remains
// the property of E.S.R.Labs and its suppliers, if any.
// The intellectual and technical concepts contained herein are
// proprietary to E.S.R.Labs and its suppliers and may be covered
// by German and Foreign Patents, patents in process, and are protected
// by trade secret or copyright law.
// Dissemination of this information or reproduction of this material
// is strictly forbidden unless prior written permission is obtained
// from E.S.R.Labs.

use criterion::{criterion_group, criterion_main, Criterion};
use std::hint::black_box;

#[path = "./bench_config.rs"]
mod bench_config;
use bench_config::bench_config;

use bufread::DeqBuffer;

fn write_buffer(buffer: &mut DeqBuffer, src: &[u8]) {
while buffer.write_available() >= src.len() {
buffer.write_from(src);
}
}

fn write_buffer_benchmark(c: &mut Criterion) {
let mut buffer = DeqBuffer::new(50 * 1000);
let src: [u8; 100] = [1; 100];

c.bench_function("write_buffer", |b| {
b.iter(|| write_buffer(black_box(&mut buffer), black_box(&src)))
});
}

fn write_read_buffer(buffer: &mut DeqBuffer, src: &[u8], dst: &mut [u8]) {
while buffer.write_available() >= src.len() {
buffer.write_from(src);
}
while buffer.read_available() != 0 {
buffer.read_to(dst);
}
}

fn write_read_buffer_benchmark(c: &mut Criterion) {
let mut buffer = DeqBuffer::new(50 * 1000);
let src: [u8; 100] = [1; 100];
let mut dst: [u8; 100] = [1; 100];

c.bench_function("write_read_buffer", |b| {
b.iter(|| write_read_buffer(black_box(&mut buffer), black_box(&src), black_box(&mut dst)))
});
}

criterion_group! {
name = benches;
config = bench_config(1000);
targets = write_buffer_benchmark, write_read_buffer_benchmark
}

criterion_main!(benches);
46 changes: 46 additions & 0 deletions application/apps/indexer/addons/bufread/benches/reader_benches.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#![allow(dead_code)]

// Copyright (c) 2025 ESR Labs GmbH. All rights reserved.
//
// NOTICE: All information contained herein is, and remains
// the property of E.S.R.Labs and its suppliers, if any.
// The intellectual and technical concepts contained herein are
// proprietary to E.S.R.Labs and its suppliers and may be covered
// by German and Foreign Patents, patents in process, and are protected
// by trade secret or copyright law.
// Dissemination of this information or reproduction of this material
// is strictly forbidden unless prior written permission is obtained
// from E.S.R.Labs.

use criterion::{criterion_group, criterion_main, Criterion};
use std::hint::black_box;

#[path = "./bench_config.rs"]
mod bench_config;
use bench_config::bench_config;

#[path = "../tests/proto.rs"]
mod proto;

use bufread::BufReader;
use proto::{Parser, Source, MAX_PACKET_LEN};

fn proto_benchmark(c: &mut Criterion) {
let source_min_size = 100 * MAX_PACKET_LEN;
let buffer_max_size = 3 * MAX_PACKET_LEN;
let buffer_min_size = MAX_PACKET_LEN;
Comment on lines +29 to +31
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if having black_box() on the parser only is enough here. Can you try to have the values here black-boxed as well? I think we may get different result with them black-boxed

let source_min_size = black_box(100 * MAX_PACKET_LEN);
...


let source = Source::fixed(source_min_size);
let reader = BufReader::new(buffer_max_size, buffer_min_size, source.data());
let mut parser = Parser::new(reader);

c.bench_function("proto", |b| b.iter(|| Parser::run(black_box(&mut parser))));
}

criterion_group! {
name = benches;
config = bench_config(1000);
targets = proto_benchmark
}

criterion_main!(benches);
Loading
Loading