Introduce data loading utility for reading from local cache or downloading from external URL #3282
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Jan 29, 2025
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. Differential Revision: D68790695
ltiao
force-pushed
the
export-D68790695
branch
from
37eb6ee to
1ec00fb
Compare
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
force-pushed
the
export-D68790695
branch
from
1ec00fb to
0f9d6d3
Compare
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Jan 30, 2025
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. ## WIP/TODO 1. Add new unit tests 2. Address OSS coverage requirements Differential Revision: D68790695
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Jan 30, 2025
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. ## WIP/TODO 1. Add new unit tests 2. Address OSS coverage requirements Differential Revision: D68790695
ltiao
force-pushed
the
export-D68790695
branch
from
0f9d6d3 to
468bb5a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Jan 30, 2025
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. ## WIP/TODO 1. Add new unit tests 2. Address OSS coverage requirements Differential Revision: D68790695
ltiao
force-pushed
the
export-D68790695
branch
from
468bb5a to
05cece7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Jan 30, 2025
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. ## WIP/TODO 1. Add new unit tests 2. Address OSS coverage requirements Differential Revision: D68790695
ltiao
force-pushed
the
export-D68790695
branch
from
05cece7 to
ef84f84
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3282 +/- ##
==========================================
- Coverage 96.04% 95.74% -0.30%
==========================================
Files 518 525 +7
Lines 52162 52468 +306
==========================================
+ Hits 50098 50238 +140
- Misses 2064 2230 +166 ☔ View full report in Codecov by Sentry. |
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Jan 31, 2025
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. ## WIP/TODO 1. Add new unit tests 2. Address OSS coverage requirements Differential Revision: D68790695
ltiao
force-pushed
the
export-D68790695
branch
from
ef84f84 to
b2d2497
Compare
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
force-pushed
the
export-D68790695
branch
from
b2d2497 to
981c50b
Compare
ltiao
added a commit
to ltiao/Ax
that referenced
this pull request
Feb 4, 2025
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. ## WIP/TODO 1. Add new unit tests 2. Address OSS coverage requirements Reviewed By: esantorella Differential Revision: D68790695
…ading from external URL (facebook#3282) Summary: Pull Request resolved: facebook#3282 ## Context Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold. ## Changes This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include: - Introduced AbstractParquetDataLoader class, providing a way to load parquet data from a cache on local disk or download from an external URL. - Implemented methods for: * Getting the cache path * Checking if the data is cached * Reading the data from the cache * Downloading from an external URL and caching the data - Added abstract properties for getting the directory name and URL of the cached file, allowing easy specialization for other benchmark datasets. With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from `ax.fb` to `ax`. ## WIP/TODO 1. Add new unit tests 2. Address OSS coverage requirements Reviewed By: esantorella Differential Revision: D68790695
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
ltiao
force-pushed
the
export-D68790695
branch
from
981c50b to
61be9d1
Compare
|
This pull request was exported from Phabricator. Differential Revision: D68790695 |
|
This pull request has been merged in 38916d1. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Context
Our preprocessed and compressed derivatives of open-source benchmarking datasets (e.g., LCBench) are currently hosted in Manifold blob storage, which limits their accessibility in our open-source software (OSS). To address this, we need to remove the dependency on Manifold.
Changes
This diff introduces a data download utility that enables loading Pandas DataFrames (stored in a compressed parquet format) from local disk or downloading it from an external URL source if not found. The key changes include:
With these changes, we can now make our LCBench surrograte benchmark problems accessible in OSS and move from
ax.fbtoax.Differential Revision: D68790695