Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement initial OWASP Snapshots support #800

Open
arkid15r opened this issue Feb 11, 2025 · 2 comments · May be fixed by #809
Open

Implement initial OWASP Snapshots support #800

arkid15r opened this issue Feb 11, 2025 · 2 comments · May be fixed by #809
Assignees
Labels

Comments

@arkid15r
Copy link
Collaborator

Describe the solution you'd like

In owasp application:

Snapshot Model

Create a Snapshot model that includes the following fields:

  • start_at: A datetime field indicating when the snapshot period begins.
  • end_at: A datetime field indicating when the snapshot period ends.
  • status: One of "pending", "processing", "completed", or "error".
  • new_chapters: A many-to-many relationship with the OWASP Chapter model.
  • new_projects: A many-to-many relationship with the OWASP Project model.
  • new_issues: A many-to-many relationship with the GitHub Issue model.
  • new_releases: A many-to-many relationship with the GitHub Release model.
  • new_users: A many-to-many relationship with the GitHub User model.

This model will serve as a record for each snapshot period, capturing the new entities within that timeframe.

Management Command Design

The management command will perform the following steps:

  1. Fetch Unprocessed Snapshots:
    Query the database for all Snapshot records where the status is "pending". These are the snapshots that have not yet been populated with new data.

  2. Update Status to Processing:
    For each pending snapshot, update the status to "processing" before starting the data population. This ensures that multiple runs of the command do not process the same snapshot concurrently.

  3. Populate Snapshot Data:

    • Using the start_at and end_at values of the snapshot, query each of the associated models (Chapters, Projects, GitHub Issues, GitHub Releases, and GitHub Users) for objects created or updated within that time window.
    • Populate the respective many-to-many relationships of the snapshot with the newly retrieved data.
  4. Update Snapshot Status:

    • Once the snapshot has been successfully populated with data, update the status to "completed".
    • If an error occurs during processing, mark the snapshot with a status of "error" for further review.
  5. Logging and Reporting:
    The command should log its progress, including which snapshots were processed successfully and which encountered issues. This information will be valuable for debugging and for ensuring that all unprocessed snapshots are handled appropriately.

@arkid15r
Copy link
Collaborator Author

@yashgoyal0110 please comment to be assigned.

@arkid15r arkid15r moved this from Backlog to Todo in Project Nest Feb 11, 2025
@yashgoyal0110
Copy link
Contributor

Assign

@arkid15r arkid15r moved this from Todo to In progress in Project Nest Feb 11, 2025
@yashgoyal0110 yashgoyal0110 linked a pull request Feb 11, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In progress
Development

Successfully merging a pull request may close this issue.

2 participants