Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Windborne Ingestor to store as GeoParquet #400

Closed
danangmassandy opened this issue Jan 28, 2025 · 1 comment
Closed

Update Windborne Ingestor to store as GeoParquet #400

danangmassandy opened this issue Jan 28, 2025 · 1 comment
Assignees
@danangmassandy
Milestone
V2 - DCAS decisio...

Comments

@danangmassandy
Copy link
Collaborator

danangmassandy commented Jan 28, 2025 •
edited
Loading

Using same logic from #397, we can update the ingestor logic to store the data as GeoParquet and remove the ingested data from measurement table.
@danangmassandy danangmassandy self-assigned this Jan 30, 2025
@danangmassandy
Copy link
Collaborator Author

The PR #414 can also process windborne dataset, but it used high memory because the windborne dataset has 15 seconds data. It seems that it's better to partition into year and month for Windborne dataset. I will need more time to work on this ticket.

@danangmassandy danangmassandy mentioned this issue Feb 4, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danangmassandy danangmassandy

Labels
None yet
Projects
None yet
Milestone
V2 - DCAS decision tree module operat...
Development

No branches or pull requests
2 participants
@danangmassandy @mvmaltitz