Develop Chunking Strategies Module

[datagero]

Objective: Create a module for managing document chunking strategies, offering multiple chunking options.
[Note, we may do this leveraging LlamaIndex]

Details:
• Implement at least two chunking strategies (e.g., sentence-based and paragraph-based).
• The module should allow switching between strategies dynamically.
• Ensure compatibility with the embedding system.

Dependencies: Integration with embedding storage and retrieval.

Acceptance Criteria:
• A reusable chunking module with multiple strategy options.
• Embedding workflow compatible with selected chunking strategies.

Priority: Medium
Estimated Effort: