Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Describe handling of paged resources, add helper functions #74

Open
rafalkrupinski opened this issue Aug 30, 2024 · 0 comments
Open

Describe handling of paged resources, add helper functions #74

rafalkrupinski opened this issue Aug 30, 2024 · 0 comments

Comments

@rafalkrupinski
Copy link
Contributor

Paging, even though has many different implementations, boils down to a couple of elements

  1. extracting paging parameters from response or generating them

    Paging parameters may be returned in header or headers, body or not returned at all.

    In case of parameters not being returned, the client keeps track of either page offset or page number and optionally page size.

  2. extracting parameters from links

    Some servers return links instead of parameters. Typically these links lead to the same endpoint and only add paging parameters.

    As long as server returns paging parameters in some form, their semantic (cursor, offset or page number) doesn't matter. They are simply passed from the previous response to the following request.
    Only when server doesn't return any parameters, the client is expected calculate offsets or page numbers and stop iteration when it receives response with no items.

  3. supplying the paging parameters to the next requests.

    The paging parameters may need to be supplied to query, path parameters or headers, but Lapidary treats in the same way, as function parameters.

Paging will be handled as function external to ApiClient:

def loop_pages(fn: Callable[..., [R]], param_extractor: Callable[[Any], dict])->Callable[[...],AsyncIterator[R]]:
  """Accept an operation method and an extractor function, and return AsyncGenerator that
  1. calls `fn` with provided parameters,
  5. yields the result and passes it to `param_extractor` to generate additional parameters for the next call
  6. call fn with additional params
  7. as long as param_extractor returns params, goto 3
  """

def extract_params(url: str, params: Sequence[str])->dict:
"""
extract query parameters from a URL
"""

async def explode_list(iter: AsyncIterator, list_extractor: Callable = lambda x:x)-> AsyncIterator:
  """If the operation method returns a list or an object that wraps a list, turn the async iterator returned by loop_pages to a single continuous iterator of items contained in those lists.
Optionally accept `prefetch: int` parameter
"""

Add helper function to extract query parameters from links

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant