Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize repo open pipeline #5

Open
amitsr4 opened this issue Dec 9, 2024 · 1 comment
Open

Optimize repo open pipeline #5

amitsr4 opened this issue Dec 9, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@amitsr4
Copy link
Collaborator

amitsr4 commented Dec 9, 2024

The current implementation uses limited prefetching between the main thread and the worker thread. To improve performance, I've modified the architecture to use a ring buffer shared between the main and worker threads. It employs a shared buffer pool with three sections (30MB each) that rotate between states (EMPTY → WORKER_PROCESSING → READY_FOR_MAIN → MAIN_PROCESSING → EMPTY), allowing simultaneous reading and processing of data.

@amitsr4
Copy link
Collaborator Author

amitsr4 commented Dec 9, 2024

On my machine, MacBook Pro M1 with 16GB RAM, the worker thread reads and identifies JSON object positions in the file (~35ms per section), while the main thread handles parsing and data persistence (~250ms per section). Each section processes approximately 39,500 objects. The parallel architecture ensures continuous data flow by having the worker prepare upcoming sections while the main thread processes the current one. However, performance testing revealed that this architectural change did not yield the expected improvements. Testing with JSON log files containing 1 million commits still takes approximately 7 seconds to process, similar to the original implementation. Further investigation and optimization are needed to achieve meaningful performance improvements.

@ofriw ofriw added the enhancement New feature or request label Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants