Regression "TypeError: can't concat coroutine to bytes"

[cvjjm]

Describe the bug

I previously filed this under boto/botocore#3341 (comment) against boto as I am unsure which library component is responsible for the problem.

Describe the bug
I am using boto in conjuction with s3fs.

A piece of code that looks like

s3 = s3fs.S3FileSystem(anon=False)
with s3.open(f"{s3_bucket}/{taget_path}/{file_name}", "w") as f:
    ....

and works perfectly with boto3<1.36 now fails with boto3==1.36 with

../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/fsspec/spec.py:2033: in close
    self.flush(force=True)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/fsspec/spec.py:1897: in flush
    if self._upload_chunk(final=force) is not False:
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/s3fs/core.py:2387: in _upload_chunk
    self.commit()
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/s3fs/core.py:2412: in commit
    write_result = self._call_s3("put_object", **kw)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/s3fs/core.py:2249: in _call_s3
    return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/fsspec/asyn.py:118: in wrapper
    return sync(self.loop, func, *args, **kwargs)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/fsspec/asyn.py:103: in sync
    raise return_result
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/fsspec/asyn.py:56: in _runner
    result[0] = await coro
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/s3fs/core.py:371: in _call_s3
    return await _error_wrapper(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/s3fs/core.py:146: in _error_wrapper
    raise err
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/s3fs/core.py:114: in _error_wrapper
    return await func(*args, **kwargs)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/client.py:394: in _make_api_call
    http, parsed_response = await self._make_request(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/client.py:420: in _make_request
    return await self._endpoint.make_request(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/endpoint.py:100: in _send_request
    while await self._needs_retry(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/endpoint.py:271: in _needs_retry
    responses = await self._event_emitter.emit(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/hooks.py:68: in _emit
    response = await resolve_awaitable(handler(**kwargs))
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/_helpers.py:6: in resolve_awaitable
    return await obj
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/retryhandler.py:107: in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/_helpers.py:6: in resolve_awaitable
    return await obj
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/retryhandler.py:126: in _call
    should_retry = await self._should_retry(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/retryhandler.py:152: in _should_retry
    return await resolve_awaitable(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/_helpers.py:6: in resolve_awaitable
    return await obj
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/retryhandler.py:174: in _call
    checker(attempt_number, response, caught_exception)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/botocore/retryhandler.py:247: in __call__
    return self._check_caught_exception(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/botocore/retryhandler.py:416: in _check_caught_exception
    raise caught_exception
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/endpoint.py:176: in _do_get_response
    responses = await self._event_emitter.emit(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/aiobotocore/hooks.py:68: in _emit
    response = await resolve_awaitable(handler(**kwargs))
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/moto/core/botocore_stubber.py:38: in __call__
    response = self.process_request(request)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/moto/core/botocore_stubber.py:88: in process_request
    status, headers, body = method_to_execute(
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/moto/core/responses.py:307: in _inner
    response = getattr(cls(), to_call.__name__)(request, full_url, headers)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/moto/s3/responses.py:1369: in key_response
    self.setup_class(request, full_url, headers)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/moto/s3/responses.py:175: in setup_class
    super().setup_class(request, full_url, headers, use_raw_body=True)
../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/moto/core/responses.py:366: in setup_class
    self.body = self.body.read()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aiobotocore.httpchecksum.AioAwsChunkedWrapper object at 0x7efcffb81350>, size = None

    def read(self, size=None):
        # Normalize "read all" size values to None
        if size is not None and size <= 0:
            size = None
    
        # If the underlying body is done and we have nothing left then
        # end the stream
        if self._complete and not self._remaining:
            return b""
    
        # While we're not done and want more bytes
        want_more_bytes = size is None or size > len(self._remaining)
        while not self._complete and want_more_bytes:
>           self._remaining += self._make_chunk()
E           TypeError: can't concat coroutine to bytes

../../../miniforge3/envs/data_layer_uploader/lib/python3.11/site-packages/botocore/httpchecksum.py:194: TypeError

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue (don't know how to do that)
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue (don't know how to do that)
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore==2.18.0
aiohappyeyeballs==2.4.3
aiohttp==3.10.10
aioitertools==0.12.0
aiosignal==1.3.1
annotated-types @ file:///home/conda/feedstock_root/build_artifacts/annotated-types_1716290248287/work
attrs==24.2.0
awscli @ file:///home/conda/feedstock_root/build_artifacts/awscli_1730970415844/work
awscrt @ file:///home/conda/feedstock_root/build_artifacts/awscrt_1729557262365/work
babel==2.16.0
black==24.10.0
boto3==1.36.0
botocore==1.36.1
Brotli @ file:///home/conda/feedstock_root/build_artifacts/brotli-split_1725267488082/work
certifi==2024.8.30
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1725560564262/work
cfgv==3.4.0
charset-normalizer==3.4.0
click==8.1.7
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
coverage @ file:///home/conda/feedstock_root/build_artifacts/coverage_1729610035939/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography-split_1725443043614/work
-e git+ssh://[email protected]/VRX/data_layer_uploader.git@75d123cf695614dbf477d68f8d48abb41034a526#egg=data_layer_uploader
decopatch==1.4.10
distlib==0.3.9
distro @ file:///home/conda/feedstock_root/build_artifacts/distro_1675116244235/work
dnspython==2.7.0
docstring_parser @ file:///home/conda/feedstock_root/build_artifacts/docstring_parser_1710507861579/work
docutils @ file:///home/conda/feedstock_root/build_artifacts/docutils_1666754865005/work
email_validator==2.2.0
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1720869315914/work
filelock==3.16.1
frozendict @ file:///home/conda/feedstock_root/build_artifacts/frozendict_1728841334936/work
frozenlist==1.5.0
fsspec==2024.10.0
ghp-import==2.1.0
griffe==1.5.1
griffe-typingdoc==0.2.7
httplib2==0.22.0
identify==2.6.1
idna==3.10
iniconfig @ file:///home/conda/feedstock_root/build_artifacts/iniconfig_1673103042956/work
Jinja2==3.0.3
jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1655568249366/work
makefun==1.15.6
Markdown==3.3.7
markdown-inline-graphviz-extension==1.1.2
markdown-it-py==2.2.0
MarkupSafe==3.0.2
mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work
mdx-truly-sane-lists==1.3
mergedeep==1.3.4
mkdocs==1.6.1
mkdocs-autorefs==1.2.0
mkdocs-get-deps==0.2.0
mkdocs-material==9.5.43
mkdocs-material-extensions==1.3.1
mkdocs-monorepo-plugin==1.1.0
mkdocs-redirects==1.2.1
mkdocs-techdocs-core==1.5.0
mkdocstrings==0.25.2
mkdocstrings-python==1.10.9
moto==5.0.26
multidict==6.1.0
mypy-extensions==1.0.0
nodeenv==1.9.1
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work
paginate==0.5.7
pathspec==0.12.1
plantuml==0.3.0
plantuml-markdown==3.10.4
platformdirs==4.3.6
pluggy @ file:///home/conda/feedstock_root/build_artifacts/pluggy_1713667077545/work
pre_commit==4.0.1
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1677600924538/work
propcache==0.2.0
py-partiql-parser==0.6.1
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1711811537435/work
pydantic @ file:///home/conda/feedstock_root/build_artifacts/pydantic_1697789541754/work
pydantic_core @ file:///home/conda/feedstock_root/build_artifacts/pydantic-core_1695733019300/work
Pygments==2.17.2
pymdown-extensions==10.3.1
pyparsing==2.4.7
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
pytest @ file:///home/conda/feedstock_root/build_artifacts/pytest_1725977232342/work
pytest-cases==3.8.6
pytest-cov @ file:///home/conda/feedstock_root/build_artifacts/pytest-cov_1730284278208/work
pytest-mock==3.14.0
python-dateutil==2.9.0.post0
python-slugify==8.0.4
PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1725456139051/work
pyyaml_env_tag==0.1
regex==2024.11.6
requests==2.32.3
responses==0.25.3
rich==13.9.4
ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1678272991830/work
ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1728724459810/work
ruff==0.8.5
s3fs==2024.10.0
s3transfer==0.11.1
shtab @ file:///home/conda/feedstock_root/build_artifacts/shtab_1709842379479/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
text-unidecode==1.3
toml @ file:///home/conda/feedstock_root/build_artifacts/toml_1604308577558/work
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1727974628237/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1717802530399/work
tyro @ file:///home/conda/feedstock_root/build_artifacts/tyro_1729690340723/work
urllib3==2.2.3
virtualenv==20.27.1
watchdog==6.0.0
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work
Werkzeug==3.1.3
wrapt==1.16.0
xmltodict==0.14.2
yarl==1.17.1

Environment:

• Python Version: Python 3.11.10
• OS name and version: ubuntu:20.04

Additional context
Add any other context about the problem here.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

s3.open() should work as it did with previous versions.

Current Behavior

See failure above.

Reproduction Steps

See code above.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

not using SDK

Environment details (OS name and version, etc.)

ubuntu:20.04

[jakob-keller]

Hello and thanks for reporting. I'm confused about the moto calls being part of the stacktrace. Could that be related to the issue?

[Apakottur]

I'm also getting this error in a moto+aiobotocore setup.

Shouldn't AioAwsChunkedWrapper override read() as well?
The boto function calls _make_chunk synchronously, but the latter is async?

[jakob-keller]

Shouldn't AioAwsChunkedWrapper override read() as well? The boto function calls _make_chunk synchronously, but the latter is async?

Looking at the stacktrace and moto source, I believe overriding read() would not do much good. The issue appears to arise from the attempt to integrate moto and aiobotocore (indirectly via s3fs it seems). Moto just simply does not expect an instance of AioAwsChunkedWrapper, but its synchronuous parent AwsChunkedWrapper from botocore. Am I missing something?

@thehesiod: Can you comment on why AioAwsChunkedWrapper looks the way it does? Do we gain anything by overriding read()?

[brentropic]

Looks like there's a new botocore feature which automatically validates checksums, I don't think this has been patched in aiobotocore yet:

boto/botocore@1.35.93...1.36.1#diff-2c623f3c6a917be56c59d43279244996836262cb1e12d9d0786c9c49eef6b43cR25

[thehesiod]

@jakob-keller AwsChunkedWrapper is a sync iterator, the Aio variant exposes an async variant. This must be a new workflow where .read needs to be called directly. So most likely we would need to expose it as well.

[thehesiod]

I'd like to understand though why moto is calling read on this fairly internal class from botocore

[jakob-keller]

My guess is that moto or s3fs uses the hooks mechanism of botocore to insert custom logic. That custom logic is not equipped to work with AwsChunkedWrapper. Could that be the case?

But again, why is moto involved in the first place? My hunch is that there's an incompatibility between s3fs and moto. We could override AwsChunkedWrapper.read(), but that alone would not fix the issue, as far as I can tell.

[thehesiod]

ok, yea they're using moto w/ s3fs, moto has a botocore_stubber triggered from before-send, note, not aiobotocore_stubber. This will not work is it calls a sync method and returns an AWSResponse and not AioAWSResponse @cvjjm you, or someone in moto needs to implement an aiobotocore stubber. So someone needs to write that. Closing as it's not due to this product. If when writing a aiobotocore stubber for moto you actually need an async read method you can then open an issue, however I doubt it as we haven't needed it in all this time.

[jakob-keller]

Turns out, we need to provide AioAwsChunkedWrapper.read after all.