InformationIngestion.PRD

# Research Platform PRD - v0.1

## 1. Product Overview

### Vision Statement
A comprehensive research and news monitoring platform that empowers journalists, investigators, and private citizens to conduct deep research while staying informed about relevant current events through automated tracking and analysis.

### Target Users
- Investigative journalists
- Private investigators
- Concerned citizens
- Amateur researchers

### Core Value Proposition
- Unified platform for document research and news monitoring
- Powerful search capabilities across multiple content types
- Automated entity tracking and alerting
- Research organization and synthesis tools

## 2. Feature Requirements

### 2.1 Document Processing System
```python
class DocumentProcessor:
    """
    MVP Requirements:
    - Support for PDF, TXT, URL ingestion
    - Automatic metadata extraction/creation
    - Content classification
    - Entity recognition
    - Source credibility scoring
    """
```

### 2.2 Search System
```python
class SearchSystem:
    """
    Required Capabilities:
    - Keyword search with boolean operators
    - Semantic search
    - Hybrid search (combining both)
    - Cross-document reference detection
    - Search within specific time periods/sources
    """
```

### 2.3 Alert System
```python
class AlertSystem:
    """
    Required Features:
    - Entity tracking across all sources
    - Configurable alert thresholds
    - Multiple notification channels
    - Alert digest customization
    - False positive filtering
    """
```

### 2.4 Project Management System
```python
class ProjectManagementSystem:
    """
    Required Features:
    - Project creation and organization
    - Folder hierarchy management
    - Document organization
    - Access control and sharing
    - Version tracking
    """
```

## 3. Technical Requirements

### 3.1 Performance Metrics
- Search response time: < 2 seconds
- Document processing time: < 30 seconds
- System uptime: 99.9%
- News ingestion delay: < 5 minutes
- Concurrent users: 1000+

### 3.2 Security Requirements
- End-to-end encryption for sensitive data
- Role-based access control
- Audit logging
- Data retention policies
- Compliance with GDPR/CCPA

## 4. User Interface Requirements

### 4.1 Research Workspace
- Document library management
- Search interface with advanced filters
- Entity tracking dashboard
- Alert configuration panel
- Report generation tools

### 4.2 News Monitoring Interface
- Real-time news feed
- Geographic filtering
- Entity highlight view
- Timeline visualization
- Source credibility indicators

### 4.3 Project Management Interface
- **Project Creation and Organization**
  - Create new research projects
  - Set project metadata (name, description, settings)
  - Define project scope and objectives
  - Configure project-specific settings

- **Folder Structure**
  - Hierarchical folder organization
  - Drag-and-drop file management
  - Folder path breadcrumb navigation
  - Bulk file operations

- **Document Management**
  - Upload documents to specific folders
  - Move/copy documents between folders
  - Document status tracking
  - Processing status indicators
  - Version history

### Project Workflow Example:

1. **Project Initialization**
```
User: Creates new research project
System: 
- Generates unique project ID
- Creates root folder
- Initializes project settings
- Sets up document tracking
```

2. **Folder Organization**
```
User: Creates folder structure
System:
- Maintains folder hierarchy
- Updates path arrays
- Validates folder names
- Ensures unique paths
```

3. **Document Upload Flow**
```
User: Uploads document to folder
System:
- Validates project/folder existence
- Processes document
- Updates folder contents
- Triggers necessary analysis
```

4. **Project Navigation**
```
User: Browses project structure
System:
- Displays folder hierarchy
- Shows document counts
- Indicates processing status
- Provides search within project
```

### Implementation Priority Timeline Update:

1. **Month 1: Core Project Management**
```python
class Phase1ProjectPriorities:
    """
    Key Deliverables:
    - Project CRUD operations
    - Basic folder management
    - Document upload to folders
    - Project settings management
    """
```

2. **Month 2: Enhanced Organization**
```python
class Phase2ProjectPriorities:
    """
    Key Deliverables:
    - Advanced folder operations
    - Bulk document management
    - Project search capabilities
    - Document status tracking
    """
```

### Technical Requirements Update:

```python
class ProjectManagementRequirements:
    """
    Technical Specifications:
    - Nested folder depth limit: 10
    - Max documents per folder: 1000
    - Supported document types: PDF, TXT, DOCX, URL
    - Max file size: 100MB
    - Project name length: 3-255 characters
    - Folder name length: 1-255 characters
    """
```

### User Interface Components:

1. **Project Dashboard**
   - Project overview
   - Recent activity
   - Document statistics
   - Processing status

2. **Folder Navigator**
   - Tree view of folders
   - Drag-and-drop interface
   - Context menus
   - Bulk operations

3. **Document List**
   - File details
   - Processing status
   - Quick actions
   - Sort/filter options

4. **Project Settings**
   - Project configuration
   - Access management
   - Integration settings
   - Notification preferences

## 5. MVP Scope

### Phase 1 (Core Features)
````python
class MVPFeatures:
    """
    Priority Order:
    1. Document Processing & Storage
    2. Search Functionality
    3. Research Assistant Interface
    4. Entity Tracking
    5. Basic Alerts
    6. User Authentication
    """
````

### 5.1 Research Assistant Interface
````python
class ResearchAssistant:
    """
    Core Capabilities:
    - Context-aware conversation about uploaded documents
    - Query refinement assistance
    - Information synthesis
    - Citation support
    - Explanation generation
    """
    
    def __init__(self):
        self.conversation_modes = {
            'explore': 'Open-ended research exploration',
            'analyze': 'Deep analysis of specific documents',
            'synthesize': 'Combine insights across sources',
            'fact_check': 'Verify claims against sources'
        }
        
        self.context_window = {
            'active_documents': [],  # Currently discussed documents
            'conversation_history': [],  # Recent interaction context
            'key_findings': []  # Important insights noted
        }
````

### Implementation Priority Timeline:

1. **Month 1-2: Core Infrastructure**
- Document ingestion pipeline
- Basic storage and retrieval
- Initial LLM integration
- Basic user authentication

2. **Month 2-3: Search & Research Assistant**
````python
class Phase2Priorities:
    """
    Key Deliverables:
    - Keyword search implementation
    - Semantic search capability
    - Basic research assistant conversations
    - Document context management
    """
````

3. **Month 3-4: Entity Tracking & Alerts**
````python
class Phase3Priorities:
    """
    Key Deliverables:
    - Entity extraction
    - Basic alert system
    - Enhanced research assistant features
    - Initial UI/UX implementation
    """
````

4. **Month 4-5: Integration & Enhancement**
````python
class Phase4Priorities:
    """
    Key Deliverables:
    - System integration testing
    - Performance optimization
    - User feedback incorporation
    - Documentation completion
    """
````

### Research Assistant Interaction Examples:

1. **Document Exploration**
```
User: "What are the main themes across these documents about renewable energy?"
Assistant: *analyzes documents in context and provides synthesis*
```

2. **Deep Analysis**
```
User: "Can you help me understand how the conclusions in Document A compare to the findings in Document B?"
Assistant: *provides comparative analysis with specific citations*
```

3. **Research Planning**
```
User: "I'm investigating local corruption cases. What should I look for in these documents?"
Assistant: *suggests research approach and key entities to track*
```

### Phase 2 (Enhanced Features)
- Advanced search capabilities
- Report generation
- Timeline visualization
- Source credibility scoring
- API access
- Enhanced research assistant capabilities
  - Multi-document reasoning
  - Automated research suggestions
  - Custom research frameworks


## 6. Success Metrics

### Technical Metrics
- Document processing accuracy > 95%
- Search relevance score > 90%
- System availability > 99.9%
- Alert precision > 90%

### User Metrics
- Daily active users
- Document processing volume
- Search query volume
- Alert engagement rate

## 7. Risk Assessment

### Technical Risks
- LLM reliability
- Data accuracy
- System scalability
- Processing pipeline stability

### Mitigation Strategies
- Multiple LLM provider fallback
- Source verification system
- Cloud-native architecture
- Robust error handling

## 8. Future Considerations

### API Marketplace
- Public API documentation
- Rate limiting
- Usage monitoring
- Integration examples

### Analytics Enhancement
- Pattern detection
- Trend analysis
- Network visualization
- Predictive analytics
----

1. Detail specific conversation flows for the research assistant
2. Expand on the technical implementation of the context management system
3. Outline the integration points between the research assistant and other system components



I'll break down each of these aspects in detail:

1. **Research Assistant Conversation Flows**

`````python
class ResearchAssistantFlows:
    """
    Defines core conversation patterns and their implementations
    """
    def __init__(self):
        self.conversation_types = {
            'initial_research': {
                'trigger': 'New research project or document upload',
                'flow': [
                    'Document overview generation',
                    'Key entity identification',
                    'Research question elicitation',
                    'Suggested investigation paths'
                ]
            },
            'deep_analysis': {
                'trigger': 'Specific document/topic analysis request',
                'flow': [
                    'Document relationship mapping',
                    'Contradiction identification',
                    'Evidence assessment',
                    'Gap analysis'
                ]
            },
            'synthesis': {
                'trigger': 'Multi-document analysis request',
                'flow': [
                    'Cross-document theme extraction',
                    'Timeline construction',
                    'Network analysis',
                    'Conclusion generation'
                ]
            }
        }

    def generate_research_plan(self, user_query, context):
        """
        Creates structured research approach based on user needs
        Example:
        User: "I'm investigating environmental violations by local companies"
        Assistant: *Generates plan*
        - Phase 1: Document gathering (permits, reports, news)
        - Phase 2: Entity identification (companies, locations)
        - Phase 3: Timeline construction
        - Phase 4: Pattern analysis
        """
        pass
`````

2. **Context Management System**

`````python
class ContextManager:
    """
    Manages conversation context and document relationships
    """
    def __init__(self):
        self.active_context = {
            'documents': {
                'current_focus': [],
                'related_docs': [],
                'citation_history': []
            },
            'conversation': {
                'current_thread': [],
                'key_findings': [],
                'pending_questions': []
            },
            'research_state': {
                'verified_facts': [],
                'hypotheses': [],
                'contradictions': []
            }
        }

    def update_context(self, new_information):
        """
        Updates context based on new information or user interaction
        - Maintains relevance window
        - Updates document relationships
        - Tracks citation usage
        """
        pass

    def manage_memory(self):
        """
        Handles context window management
        - Prunes irrelevant information
        - Maintains important context
        - Summarizes historical context
        """
        pass
`````

3. **System Integration Points**

`````python
class ResearchSystemIntegration:
    """
    Manages integration between research assistant and other components
    """
    def __init__(self):
        self.integration_points = {
            'document_processor': {
                'methods': [
                    'extract_entities',
                    'generate_embeddings',
                    'classify_content'
                ],
                'events': [
                    'new_document_processed',
                    'entity_detected',
                    'classification_updated'
                ]
            },
            'search_system': {
                'methods': [
                    'semantic_search',
                    'keyword_search',
                    'citation_lookup'
                ],
                'events': [
                    'search_performed',
                    'relevant_document_found',
                    'citation_verified'
                ]
            },
            'alert_system': {
                'methods': [
                    'create_entity_alert',
                    'update_tracking',
                    'notify_user'
                ],
                'events': [
                    'entity_mentioned',
                    'alert_triggered',
                    'context_updated'
                ]
            }
        }

    async def handle_research_query(self, query, context):
        """
        Orchestrates system components for research queries
        
        Example flow:
        1. Process user query
        2. Retrieve relevant documents
        3. Extract entities and relationships
        4. Generate response with citations
        5. Update research context
        6. Suggest next steps
        """
        pass

    async def manage_research_session(self):
        """
        Maintains coherent research session
        - Tracks conversation state
        - Manages document context
        - Updates alert configurations
        - Generates research summaries
        """
        pass
`````

Example Interaction Flow:
```
User: "I'm researching environmental violations in California"

System Flow:
1. Context Initialization:
   - Create research session
   - Load relevant document types
   - Initialize entity tracking

2. Initial Response:
   - Suggest document sources
   - Identify key entities
   - Create tracking alerts

3. Ongoing Interaction:
   - Process new documents
   - Update context
   - Generate insights
   - Suggest connections

4. Research Synthesis:
   - Generate summaries
   - Create visualizations
   - Identify patterns
   - Suggest conclusions
```

1. Provide more specific implementation details for any of these components?
2. Expand on the error handling and edge cases?
3. Detail the specific LLM prompting strategies for different research scenarios?

I'll break down each of these aspects. Due to length, I'll split this into multiple responses.

First, let's look at specific implementation details:

1. **Detailed Component Implementation**

``````python
class ResearchAssistant:
    """
    Core research assistant implementation with error handling and state management
    """
    def __init__(self, llm_service, vector_store, document_processor):
        self.llm = llm_service
        self.vector_store = vector_store
        self.doc_processor = document_processor
        
        # Conversation state management
        self.state = {
            'active_session': None,
            'context_depth': 10,  # Number of recent exchanges to maintain
            'max_documents': 5    # Max documents to consider in single analysis
        }
        
        # Error tracking
        self.error_log = []
        
        # Performance metrics
        self.metrics = {
            'response_times': [],
            'context_switches': 0,
            'citation_accuracy': []
        }

    async def start_research_session(self, initial_query: str) -> ResearchSession:
        """
        Initialize new research session with error handling
        """
        try:
            session = ResearchSession(
                query=initial_query,
                timestamp=time.now(),
                context=self.initialize_context()
            )
            
            # Validate initial query
            refined_query = await self.validate_and_refine_query(initial_query)
            
            # Set up document tracking
            await self.setup_document_tracking(session)
            
            return session
            
        except Exception as e:
            self.error_log.append({
                'timestamp': time.now(),
                'error': str(e),
                'context': 'session_initialization'
            })
            raise ResearchSessionError(f"Failed to initialize session: {str(e)}")

    async def process_query(self, query: str, session: ResearchSession) -> QueryResponse:
        """
        Process user query with comprehensive error handling and fallbacks
        """
        try:
            # Validate and enhance query
            enhanced_query = await self.enhance_query(query, session.context)
            
            # Retrieve relevant documents
            relevant_docs = await self.retrieve_documents(enhanced_query)
            
            # Generate response with citations
            response = await self.generate_response(
                query=enhanced_query,
                documents=relevant_docs,
                context=session.context
            )
            
            # Validate response
            validated_response = await self.validate_response(response)
            
            # Update session context
            await self.update_session_context(session, query, response)
            
            return validated_response
            
        except DocumentRetrievalError as e:
            # Fallback to simpler document retrieval
            return await self.fallback_retrieval(query, session)
            
        except LLMError as e:
            # Fallback to simpler response generation
            return await self.fallback_response(query, session)
            
        except Exception as e:
            self.error_log.append({
                'timestamp': time.now(),
                'error': str(e),
                'context': 'query_processing'
            })
            raise QueryProcessingError(f"Failed to process query: {str(e)}")

    async def validate_and_refine_query(self, query: str) -> str:
        """
        Validate and improve query quality
        """
        try:
            # Check query length and content
            if len(query.strip()) < 3:
                raise QueryValidationError("Query too short")
                
            # Remove harmful content
            cleaned_query = self.sanitize_query(query)
            
            # Enhance query with context
            enhanced_query = await self.llm.enhance_query(cleaned_query)
            
            return enhanced_query
            
        except Exception as e:
            self.error_log.append({
                'timestamp': time.now(),
                'error': str(e),
                'context': 'query_validation'
            })
            return query  # Fallback to original query
``````

This first part focuses on the core implementation with robust error handling and state management. Would you like me to continue with the error handling and LLM prompting strategies next?

YES. 



2. **Error Handling and Edge Cases**

```````python
class ResearchErrorHandler:
    """
    Comprehensive error handling for research system components
    """
    def __init__(self):
        self.error_types = {
            'llm': {
                'timeout': LLMTimeoutStrategy(),
                'content_filter': ContentFilterStrategy(),
                'context_overflow': ContextOverflowStrategy(),
                'hallucination': HallucinationDetectionStrategy()
            },
            'document': {
                'parsing': DocumentParsingStrategy(),
                'embedding': EmbeddingStrategy(),
                'retrieval': RetrievalStrategy()
            },
            'context': {
                'memory_overflow': MemoryOverflowStrategy(),
                'context_loss': ContextRecoveryStrategy(),
                'citation_error': CitationValidationStrategy()
            }
        }

    async def handle_llm_error(self, error: LLMError, context: dict):
        """
        Handle LLM-specific errors with fallback strategies
        """
        try:
            if isinstance(error, LLMTimeout):
                return await self.error_types['llm']['timeout'].handle(context)
            elif isinstance(error, ContentFilterError):
                return await self.error_types['llm']['content_filter'].handle(context)
            elif isinstance(error, ContextOverflowError):
                return await self.error_types['llm']['context_overflow'].handle(context)
            
        except Exception as e:
            return await self.emergency_fallback(error, context)

class HallucinationDetectionStrategy:
    """
    Detect and handle potential LLM hallucinations
    """
    def __init__(self):
        self.confidence_threshold = 0.85
        self.citation_required = True

    async def validate_response(self, response: str, context: dict) -> bool:
        """
        Validate response against source documents
        """
        citations = self.extract_citations(response)
        return await self.verify_citations(citations, context['documents'])

    async def handle_hallucination(self, response: str, context: dict) -> str:
        """
        Handle detected hallucination
        """
        corrected_response = await self.regenerate_with_strict_citations(response, context)
        return corrected_response

class ContextRecoveryStrategy:
    """
    Recover from context loss or corruption
    """
    def __init__(self):
        self.backup_window = 5  # Keep last 5 states
        self.context_checkpoints = []

    async def create_checkpoint(self, context: dict):
        """
        Create context checkpoint for recovery
        """
        checkpoint = {
            'timestamp': time.now(),
            'context_state': deepcopy(context),
            'active_references': self.get_active_references(context)
        }
        self.context_checkpoints.append(checkpoint)
        self.prune_old_checkpoints()

    async def recover_context(self, error_point: dict) -> dict:
        """
        Attempt to recover context from last valid state
        """
        last_valid = self.find_last_valid_checkpoint(error_point)
        if last_valid:
            return await self.rebuild_context(last_valid)
        return await self.initialize_new_context()
```````

3. **LLM Prompting Strategies**

```````python
class ResearchPromptStrategy:
    """
    Manages different prompting strategies for research scenarios
    """
    def __init__(self):
        self.prompt_templates = {
            'initial_analysis': """
            Analyze the following research materials:
            Context: {context}
            Documents: {documents}
            
            Provide:
            1. Key themes and patterns
            2. Important entities and relationships
            3. Potential areas for deeper investigation
            4. Any contradictions or inconsistencies
            
            Base all observations on the provided documents only.
            Include specific citations for each point.
            """,
            
            'fact_verification': """
            Verify the following claim against our sources:
            Claim: {claim}
            
            Available documents: {documents}
            
            Provide:
            1. Verification status (Confirmed/Partially Confirmed/Unconfirmed)
            2. Supporting evidence with specific citations
            3. Any contradicting evidence
            4. Confidence assessment
            """,
            
            'synthesis_generation': """
            Generate a synthesis of the following research:
            Research question: {question}
            Available documents: {documents}
            Previous findings: {findings}
            
            Create:
            1. Comprehensive summary
            2. Evidence-based conclusions
            3. Identified gaps
            4. Suggested next steps
            
            Ensure all statements are supported by specific citations.
            """
        }

    async def generate_research_prompt(self, 
                                     prompt_type: str, 
                                     context: dict,
                                     parameters: dict) -> str:
        """
        Generate specific research prompt based on context
        """
        base_prompt = self.prompt_templates[prompt_type]
        
        # Add context management
        context_window = self.format_context_window(context)
        
        # Add citation requirements
        citation_guide = self.generate_citation_guide(context['documents'])
        
        # Add confidence scoring requirements
        confidence_guide = self.generate_confidence_guide()
        
        return self.combine_prompt_elements(
            base_prompt,
            context_window,
            citation_guide,
            confidence_guide,
            parameters
        )

    def format_context_window(self, context: dict) -> str:
        """
        Format context window for optimal LLM understanding
        """
        return f"""
        Current research context:
        1. Active documents: {self.format_active_documents(context)}
        2. Recent findings: {self.format_recent_findings(context)}
        3. Established facts: {self.format_verified_facts(context)}
        4. Open questions: {self.format_pending_questions(context)}
        """
```````

Would you like me to:
1. Provide implementation details for any of the specific strategies mentioned?
2. Add more error handling scenarios?
3. Expand on the prompt templates for other research scenarios?

I think we've done enough for the PRD. Now we move onto project implementation. Do you have any input on what web frameworks we should use? I'd like to stick with Python as I am most familiar with it. 


For a Python-based web application with these requirements, here are my recommendations:

1. **Backend Framework Options**:

- **FastAPI** (Recommended)
  - Modern, fast, async-native
  - Great for building APIs
  - Built-in OpenAPI documentation
  - Excellent type hinting support
  - Works well with async database operations
  - Easy integration with WebSockets for real-time features

- **Django** (Alternative)
  - More full-featured, batteries included
  - Robust admin interface
  - Strong ORM
  - Great for user management
  - Larger learning curve
  - Less performant for async operations

2. **Key Components**:

````````python
# Example FastAPI Structure
from fastapi import FastAPI, WebSocket
from pydantic import BaseModel

app = FastAPI()

class ResearchPlatform:
    def __init__(self):
        self.document_processor = DocumentProcessor()
        self.search_engine = SearchEngine()
        self.llm_service = LLMService()
        self.alert_system = AlertSystem()

    async def setup(self):
        # Initialize services
        await self.document_processor.initialize()
        await self.search_engine.initialize()
        # etc...

# API Routes
@app.post("/research/session")
async def create_research_session():
    pass

@app.websocket("/research/chat")
async def research_chat(websocket: WebSocket):
    # Handle real-time chat with research assistant
    pass
````````

3. **Supporting Technologies**:

- **Database**:
  - PostgreSQL (primary database)
  - Qdrant/Milvus/Weaviate (vector store)
  - Redis (caching, real-time features)

- **Task Queue**:
  - Celery (document processing)
  - Redis Queue (simpler alternative)

- **Search**:
  - Elasticsearch (full-text search)
  - Qdrant (vector search)

4. **Project Structure**:

````````
.
├── app
│   ├── api
│   │   └── v1
│   │       ├── auth
│   │       │   ├── __init__.py
│   │       │   ├── __pycache__
│   │       │   │   ├── __init__.cpython-313.pyc
│   │       │   │   └── routes.cpython-313.pyc
│   │       │   ├── routes.py
│   │       │   └── schemas.py
│   │       ├── projects
│   │       │   ├── __init__.py
│   │       │   ├── __pycache__
│   │       │   │   ├── __init__.cpython-313.pyc
│   │       │   │   └── routes.cpython-313.pyc
│   │       │   ├── routes.py
│   │       │   └── schemas.py
│   │       └── websocket
│   │           ├── __init__.py
│   │           ├── __pycache__
│   │           │   ├── __init__.cpython-313.pyc
│   │           │   └── routes.cpython-313.pyc
│   │           ├── routes.py
│   │           └── schemas.py
│   ├── core
│   │   ├── config.py
│   │   ├── dependencies.py
│   │   ├── exceptions.py
│   │   ├── __init__.py
│   │   └── __pycache__
│   │       ├── config.cpython-313.pyc
│   │       ├── dependencies.cpython-313.pyc
│   │       ├── exceptions.cpython-313.pyc
│   │       └── __init__.cpython-313.pyc
│   ├── database.py
│   ├── __init__.py
│   ├── main.py
│   ├── models
│   │   ├── __init__.py
│   │   ├── projectexplanation
│   │   ├── project.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-313.pyc
│   │   │   ├── project.cpython-313.pyc
│   │   │   └── user.cpython-313.pyc
│   │   └── user.py
│   ├── __pycache__
│   │   ├── database.cpython-313.pyc
│   │   ├── __init__.cpython-313.pyc
│   │   └── main.cpython-313.pyc
│   ├── scripts
│   │   ├── add_raw_content_column.py
│   │   ├── add_updated_at_column.py
│   │   ├── insert_test_api_key.py
│   │   └── test_update_owner_id.py
│   └── services
│       ├── auth
│       │   ├── __init__.py
│       │   ├── __pycache__
│       │   │   ├── __init__.cpython-313.pyc
│       │   │   └── security_service.cpython-313.pyc
│       │   └── security_service.py
│       ├── document_processorexplanation
│       ├── document_processor.py
│       ├── project_serviceexplanation
│       ├── project_service.py
│       └── __pycache__
│           ├── document_processor.cpython-313.pyc
│           └── project_service.cpython-313.pyc
├── db_cmds
├── extract.py
├── implement_news_entity_feed_conversation.txt
├── InformationIngestion.PRD
├── proj_structure
├── README.md
├── requirements.txt
├── run_appcmd
├── startqdrant
├── static
│   ├── css
│   │   └── styles.css
│   ├── images
│   └── js
│       ├── auth.js
│       ├── chat.js
│       ├── dashboard.js
│       ├── document_upload.js
│       ├── main.js
│       ├── projects.js
│       ├── research_sidebar.js
│       ├── websocket.js
│       └── workspace.js
├── templates
│   └── index.html
├── test_key
└── tests

5. **Example Dependencies**:

````````python
# requirements.txt
fastapi==0.104.1
uvicorn[standard]==0.24.0
sqlalchemy==2.0.23
asyncpg==0.29.0
python-multipart==0.0.6
qdrant-client==1.6.4
redis==5.0.1
celery==5.3.4
elasticsearch==8.11.0
pydantic==2.4.2
````````

6. **Development Setup**:

````````python
# docker-compose.yml
version: '3.8'
services:
  web:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      - postgres
      - redis
      - qdrant
      
  postgres:
    image: postgres:15
    
  redis:
    image: redis:7
    
  qdrant:
    image: qdrant/qdrant:latest
    
  elasticsearch:
    image: elasticsearch:8.11.0
````````

Would you like me to:
1. Elaborate on any of these components?
2. Provide more detailed setup instructions?
3. Show example implementations of specific features?

The choice between FastAPI and Django might depend on your specific needs, but I recommend FastAPI for this project because:
- Better async support (important for LLM interactions)
- More lightweight and flexible
- Better performance for API-heavy applications
- Easier integration with modern async Python libraries
- Great for real-time features (WebSocket support)

I think we've done enough for the PRD. Now we move onto project implementation. Do you have any input on what web frameworks we should use? I'd like to stick with Python as I am most familiar with it. 




Yes, FastAPI would still be excellent for this use case. Let me break down how we can handle multiple concurrent systems:

1. **Real-time System Architecture**:

````````python
from fastapi import FastAPI, WebSocket
from fastapi.middleware.cors import CORSMiddleware
from typing import List

app = FastAPI()

# Enable CORS for frontend
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],  # Configure appropriately for production
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

class ConnectionManager:
    """Manage active WebSocket connections"""
    def __init__(self):
        self.active_connections: List[WebSocket] = []
        
    async def connect(self, websocket: WebSocket):
        await websocket.accept()
        self.active_connections.append(websocket)
        
    async def broadcast(self, message: dict):
        """Send updates to all connected clients"""
        for connection in self.active_connections:
            await connection.send_json(message)

# Separate managers for different real-time features
chat_manager = ConnectionManager()
news_manager = ConnectionManager()
alerts_manager = ConnectionManager()

# WebSocket endpoints for different features
@app.websocket("/ws/chat/{client_id}")
async def chat_websocket(websocket: WebSocket, client_id: str):
    await chat_manager.connect(websocket)
    try:
        while True:
            message = await websocket.receive_text()
            # Process chat message
            response = await research_assistant.process_message(message)
            await websocket.send_json(response)
    except WebSocketDisconnect:
        chat_manager.disconnect(websocket)

@app.websocket("/ws/news/{client_id}")
async def news_websocket(websocket: WebSocket, client_id: str):
    await news_manager.connect(websocket)
    try:
        while True:
            # Send real-time news updates
            news_updates = await news_service.get_updates()
            await websocket.send_json(news_updates)
    except WebSocketDisconnect:
        news_manager.disconnect(websocket)
````````

2. **Background Tasks System**:

````````python
from fastapi import BackgroundTasks
from celery import Celery

# Configure Celery for background tasks
celery_app = Celery('research_platform', broker='redis://localhost:6379/0')

class BackgroundServices:
    """Manage long-running background services"""
    def __init__(self):
        self.news_monitor = NewsMonitorService()
        self.document_processor = DocumentProcessor()
        self.alert_system = AlertSystem()
    
    async def start_services(self):
        """Start all background services"""
        await self.news_monitor.start()
        await self.document_processor.start()
        await self.alert_system.start()

# Background task endpoints
@app.post("/documents/process")
async def process_document(
    document: UploadFile,
    background_tasks: BackgroundTasks
):
    """Handle document upload and processing"""
    # Save document
    document_id = await save_document(document)
    
    # Add to processing queue
    background_tasks.add_task(process_document_task, document_id)
    
    return {"status": "processing", "document_id": document_id}

@celery_app.task
def process_document_task(document_id: str):
    """Long-running document processing task"""
    # Process document
    # Update vector store
    # Send notification when complete
````````

3. **Service Integration**:

````````python
class ResearchPlatformServices:
    """Coordinate all platform services"""
    def __init__(self):
        # Real-time services
        self.chat_service = ChatService()
        self.news_service = NewsService()
        self.alert_service = AlertService()
        
        # Processing services
        self.document_service = DocumentService()
        self.search_service = SearchService()
        self.llm_service = LLMService()
        
        # Background tasks
        self.background_tasks = BackgroundTasks()
        
    async def initialize(self):
        """Initialize all services"""
        await self.start_real_time_services()
        await self.start_background_services()
        
    async def start_real_time_services(self):
        """Start WebSocket-based services"""
        await self.chat_service.start()
        await self.news_service.start()
        await self.alert_service.start()
        
    async def start_background_services(self):
        """Start background processing services"""
        await self.document_service.start()
        await self.search_service.start()
````````

4. **Frontend Integration**:

````````javascript
// Example frontend WebSocket connections
class ResearchPlatformClient {
    constructor() {
        this.chatSocket = new WebSocket('ws://localhost:8000/ws/chat');
        this.newsSocket = new WebSocket('ws://localhost:8000/ws/news');
        this.alertsSocket = new WebSocket('ws://localhost:8000/ws/alerts');
        
        this.setupEventListeners();
    }
    
    setupEventListeners() {
        // Chat events
        this.chatSocket.onmessage = (event) => {
            const message = JSON.parse(event.data);
            this.updateChatInterface(message);
        };
        
        // News events
        this.newsSocket.onmessage = (event) => {
            const news = JSON.parse(event.data);
            this.updateNewsDashboard(news);
        };
        
        // Alert events
        this.alertsSocket.onmessage = (event) => {
            const alert = JSON.parse(event.data);
            this.showAlert(alert);
        };
    }
}
````````

FastAPI's async support makes it ideal for handling:
1. WebSocket connections for real-time chat
2. Server-Sent Events for news updates
3. Background tasks for document processing
4. API endpoints for user interactions
5. Integration with async database operations


### Qdrant Docker Instance Command
sudo docker run -p 6333:6333 \
    -v $(pwd)/qdrant:/qdrant/storage \
    qdrant/qdrant

 - Ran in the root directory of the project