Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify HTAN Ingest API #1

Open
dgutman opened this issue Jan 15, 2021 · 0 comments
Open

Modify HTAN Ingest API #1

dgutman opened this issue Jan 15, 2021 · 0 comments
Assignees

Comments

@dgutman
Copy link
Contributor

dgutman commented Jan 15, 2021

Want to extend ingestion endpoint to support multiple buckets/endpoints

An ingestion endpoint should consist of the following parameters when being registered with the DSA.

importProcedureName : This is a short handle to describe the process ( e.g. dcc-htan-duke or dcc-htan-duke-images)
importProcedureNameDescrtiption: Text description that's more verbose than the short name
importAssetstore --- Girder Internal MongoID that maps to internal assestore name
importPath--- This is the AWS path to use to begin the ingestion process; this can also be overridden during an actual POST
Import Folder: Girder Folder where the data is ingested to
includeRegex: As per the current ingestion endpoint
excludeRegex: As part current ingestion endpoint
onImportEnd: For now this will do nothing, but if we want to call some process on completion, this would be where the function could be stored; I'm not sure yet how we want to create/generate these

image

EndPoint for actual ingestion should require the importProcedureName to be specified, other parameters from above can be included if desired, but otherwise use the default that was specified during in the HTAN configuration

SECOND endpoint: This would dump the list of currently registered importProcedures and their metadata.. basically just dumps the JSON from above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants