Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Implement Progress Details in podman pull API as occurs in docker pull API #24887

Open
D3vil0p3r opened this issue Dec 21, 2024 · 10 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@D3vil0p3r
Copy link

Feature request description

Currently, by invoking podman pull API:

curl -XPOST --unix-socket /run/user/1000/podman/podman.sock -H content-type:application/json http://d/v4.0.0/libpod/images/pull?reference=docker.io/athenaos/base

produces an output in JSON containing only stream field with no progress details, i.e.:

{'stream': 'Trying to pull docker.io/athenaos/base:latest...\n'}
{'stream': 'Getting image source signatures\n'}
{'stream': 'Copying blob sha256:30babffc8f090f7c78c263b9a1b683b34ca5f73da74911ffe942d4d8225dca57\n'}
{'stream': 'Copying config sha256:420f00cb47debe421a8b70b1a2e9ccf16a942cce2f70d1724eff6be764e95fea\n'}
{'stream': 'Writing manifest to image destination\n'}
{'images': ['420f00cb47debe421a8b70b1a2e9ccf16a942cce2f70d1724eff6be764e95fea'], 'id': '420f00cb47debe421a8b70b1a2e9ccf16a942cce2f70d1724eff6be764e95fea'}

This output is also different by the one produced by podman pull command that actually shows a progress bar:

Trying to pull docker.io/athenaos/base:latest...
Getting image source signatures
Copying blob 30babffc8f09 [>-------------------------------] 5.5MiB / 145.1MiB | 3.9 MiB/s
...

It would be very nice if this pull API could be improved in order to output progress details as occurs by docker pull API:

{'status': 'Pulling from athenaos/base', 'id': 'latest'}
{'status': 'Pulling fs layer', 'progressDetail': {}, 'id': '30babffc8f09'}
{'status': 'Downloading', 'progressDetail': {'current': 527607, 'total': 152137719}, 'progress': '[>                                    
]  527.6kB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Downloading', 'progressDetail': {'current': 1055991, 'total': 152137719}, 'progress': '[>                                   
]  1.056MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Downloading', 'progressDetail': {'current': 1596663, 'total': 152137719}, 'progress': '[>                                   
]  1.597MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Downloading', 'progressDetail': {'current': 2137335, 'total': 152137719}, 'progress': '[>                                   
]  2.137MB/152.1MB', 'id': '30babffc8f09'}
...
...
{'status': 'Downloading', 'progressDetail': {'current': 103782662, 'total': 152137719}, 'progress': 
'[==================================>                ]  103.8MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Downloading', 'progressDetail': {'current': 104319238, 'total': 152137719}, 'progress': 
'[==================================>                ]  104.3MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Downloading', 'progressDetail': {'current': 104851718, 'total': 152137719}, 'progress': 
'[==================================>                ]  104.9MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Downloading', 'progressDetail': {'current': 105392390, 'total': 152137719}, 'progress': 
'[==================================>                ]  105.4MB/152.1MB', 'id': '30babffc8f09'}
...
...
{'status': 'Verifying Checksum', 'progressDetail': {}, 'id': '30babffc8f09'}
{'status': 'Download complete', 'progressDetail': {}, 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 557056, 'total': 152137719}, 'progress': '[>                                     
]  557.1kB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 2228224, 'total': 152137719}, 'progress': '[>                                    
]  2.228MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 3899392, 'total': 152137719}, 'progress': '[=>                                   
]  3.899MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 5570560, 'total': 152137719}, 'progress': '[=>                                   
]  5.571MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 7241728, 'total': 152137719}, 'progress': '[==>                                  
]  7.242MB/152.1MB', 'id': '30babffc8f09'}
...
...
{'status': 'Extracting', 'progressDetail': {'current': 107511808, 'total': 152137719}, 'progress': 
'[===================================>               ]  107.5MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 108068864, 'total': 152137719}, 'progress': 
'[===================================>               ]  108.1MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 109182976, 'total': 152137719}, 'progress': 
'[===================================>               ]  109.2MB/152.1MB', 'id': '30babffc8f09'}
{'status': 'Extracting', 'progressDetail': {'current': 110297088, 'total': 152137719}, 'progress': 
'[====================================>              ]  110.3MB/152.1MB', 'id': '30babffc8f09'}
...
...
{'status': 'Pull complete', 'progressDetail': {}, 'id': '30babffc8f09'}
{'status': 'Digest: sha256:694ae0bf54f96475b2dec18b1dd2c313405a7e73cda8467c71e1a222df15ffc6'}
{'status': 'Status: Downloaded newer image for athenaos/base:latest'}

It can be useful for third-party programs using Podman SDK to manage or show the progress process in a flexible manner.

Suggest potential solution

My proposal is to replace stream element and use the same ones used in docker pull API, that are:

  • status
  • progressDetails
    • current
    • total
  • progress
  • id

Have you considered any alternatives?

No response

Additional context

No response

@D3vil0p3r D3vil0p3r added the kind/feature Categorizes issue or PR as related to a new feature. label Dec 21, 2024
@D3vil0p3r D3vil0p3r changed the title [FEATURE] Implement Progress Details in podman pull API as docker [FEATURE] Implement Progress Details in podman pull API as occurs in docker pull API Dec 21, 2024
@afbjorklund
Copy link
Contributor

@D3vil0p3r
Copy link
Author

@rhatdan I would like to have your feedback here

@rhatdan
Copy link
Member

rhatdan commented Dec 23, 2024

@mheon @Luap99 Ideas?

@Luap99
Copy link
Member

Luap99 commented Jan 6, 2025

The API already supports the same progress details as the docker compat API endpoint. Just need to add compatMode=true to the query. I did not check if this actually contains all the details that docker offers.

@D3vil0p3r
Copy link
Author

D3vil0p3r commented Jan 6, 2025

I just run:

curl -XPOST --unix-socket /run/user/1000/podman/podman.sock -H content-type:application/json "http://d/v4.0.0/libpod/images/pull?reference=docker.io/athenaos/base&compatMode=true"

This is the output:

{"status":"Pulling fs layer","progressDetail":{},"id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":3373319,"total":152137719},"progress":"[=\u003e                                                 ]  3.373MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":6650119,"total":152137719},"progress":"[==\u003e                                                ]   6.65MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":9386247,"total":152137719},"progress":"[===\u003e                                               ]  9.386MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":12159239,"total":152137719},"progress":"[===\u003e                                               ]  12.16MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":14817543,"total":152137719},"progress":"[====\u003e                                              ]  14.82MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":18155783,"total":152137719},"progress":"[=====\u003e                                             ]  18.16MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":20814087,"total":152137719},"progress":"[======\u003e                                            ]  20.81MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":24279303,"total":152137719},"progress":"[=======\u003e                                           ]  24.28MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":27633927,"total":152137719},"progress":"[=========\u003e                                         ]  27.63MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":30951687,"total":152137719},"progress":"[==========\u003e                                        ]  30.95MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":33462535,"total":152137719},"progress":"[==========\u003e                                        ]  33.46MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":36743431,"total":152137719},"progress":"[============\u003e                                      ]  36.74MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":39774471,"total":152137719},"progress":"[=============\u003e                                     ]  39.77MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":42486023,"total":152137719},"progress":"[=============\u003e                                     ]  42.49MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":44923143,"total":152137719},"progress":"[==============\u003e                                    ]  44.92MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":47851783,"total":152137719},"progress":"[===============\u003e                                   ]  47.85MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":51038471,"total":152137719},"progress":"[================\u003e                                  ]  51.04MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":54405383,"total":152137719},"progress":"[=================\u003e                                 ]  54.41MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":57743623,"total":152137719},"progress":"[==================\u003e                                ]  57.74MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":61061383,"total":152137719},"progress":"[====================\u003e                              ]  61.06MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":64420103,"total":152137719},"progress":"[=====================\u003e                             ]  64.42MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":67594503,"total":152137719},"progress":"[======================\u003e                            ]  67.59MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":70068487,"total":152137719},"progress":"[=======================\u003e                           ]  70.07MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":72882439,"total":152137719},"progress":"[=======================\u003e                           ]  72.88MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":76253447,"total":152137719},"progress":"[=========================\u003e                         ]  76.25MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":77478151,"total":152137719},"progress":"[=========================\u003e                         ]  77.48MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":85764359,"total":152137719},"progress":"[============================\u003e                      ]  85.76MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":88848647,"total":152137719},"progress":"[=============================\u003e                     ]  88.85MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":91859207,"total":152137719},"progress":"[==============================\u003e                    ]  91.86MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":94701831,"total":152137719},"progress":"[===============================\u003e                   ]   94.7MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":98101511,"total":152137719},"progress":"[================================\u003e                  ]   98.1MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":101460231,"total":152137719},"progress":"[=================================\u003e                 ]  101.5MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":104397063,"total":152137719},"progress":"[==================================\u003e                ]  104.4MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":107026695,"total":152137719},"progress":"[===================================\u003e               ]    107MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":110004487,"total":152137719},"progress":"[====================================\u003e              ]    110MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":113355015,"total":152137719},"progress":"[=====================================\u003e             ]  113.4MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":115730695,"total":152137719},"progress":"[======================================\u003e            ]  115.7MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":119113991,"total":152137719},"progress":"[=======================================\u003e           ]  119.1MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":122460423,"total":152137719},"progress":"[========================================\u003e          ]  122.5MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":125597959,"total":152137719},"progress":"[=========================================\u003e         ]  125.6MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":128628999,"total":152137719},"progress":"[==========================================\u003e        ]  128.6MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":131938567,"total":152137719},"progress":"[===========================================\u003e       ]  131.9MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":135301383,"total":152137719},"progress":"[============================================\u003e      ]  135.3MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":138361095,"total":152137719},"progress":"[=============================================\u003e     ]  138.4MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":141383943,"total":152137719},"progress":"[==============================================\u003e    ]  141.4MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":144742663,"total":152137719},"progress":"[===============================================\u003e   ]  144.7MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":148068615,"total":152137719},"progress":"[================================================\u003e  ]  148.1MB/152.1MB","id":"30babffc8f09"}
{"status":"Downloading","progressDetail":{"current":150935815,"total":152137719},"progress":"[=================================================\u003e ]  150.9MB/152.1MB","id":"30babffc8f09"}
{"status":"Download complete","progressDetail":{},"id":"30babffc8f09"}
{"status":"Pulling fs layer","progressDetail":{},"id":"420f00cb47de"}
{"status":"Download complete","progressDetail":{},"id":"420f00cb47de"}
{"status":"Download complete","progressDetail":{},"id":"420f00cb47de"}

What I note is that, unlike docker API, the \u003e is not decoded as >. Furthermore, there is not the checksum verification message, as in docker API, before the Download complete string, for example:

{'status': 'Verifying Checksum', 'progressDetail': {}, 'id': '30babffc8f09'}

and there is not the Extracting status details, as in docker API, for example:

{'status': 'Extracting', 'progressDetail': {'current': 557056, 'total': 152137719}, 'progress': '[>                                     
]  557.1kB/152.1MB', 'id': '30babffc8f09'}

and there are not the pull complete message after the extraction, as in docker, for example:

{'status': 'Pull complete', 'progressDetail': {}, 'id': '30babffc8f09'}
{'status': 'Digest: sha256:694ae0bf54f96475b2dec18b1dd2c313405a7e73cda8467c71e1a222df15ffc6'}
{'status': 'Status: Downloaded newer image for athenaos/base:latest'}

Is it possible to implement these missing details?

@Luap99
Copy link
Member

Luap99 commented Jan 6, 2025

The encoding thing should be an easy fix. The other messages at the end should be possible to add with the current info I think. PRs welcome.

I don't think our backend makes any difference for "Extracting" so that info simply does not exists. No idea what "Verifying Checksum" means either.
cc @mtrmac

@mtrmac
Copy link
Collaborator

mtrmac commented Jan 6, 2025

c/image and related repos do track a bunch of RFEs in this general area (e.g. containers/skopeo#658 , containers/image#2585 ), but short-term no work is happening in this area.

(Historically, we were limited by a frozen API of the transport abstraction, making it impossible for individual transports to contribute progress information. That’s no longer the case, we do have an option to transfer more data between the generic copy code and transports. But none of that exists.)


Furthermore, there is not the checksum verification message

(and the like…)

I don’t consider that a valid way to consume Docker’s progress API. status is English text for users to read, not a constant for any software to expect.

E.g. whether checksum verification is a separate phase or something happening concurrently with the transfer is an internal implementation detail, and depends on the source/destination.

What do you need these details for?

If the goal is to communicate progress to humans, there are various ways to structure a generic progress API to allow that (just a percentage, percentage + text (and localization!), some hierarchical structure — https://developer.apple.com/documentation/foundation/nsprogress is one of the more complex designs). Similarly, logs used for after-the-fact debugging need records of what happened, but not a commitment to any specific structure or order.

I’m fundamentally opposed to making any promises on the structure of the copy process on the requested level of detail, constraining our ability to change the implementation — at least without any identified benefit to tying out hands that way. Keeping stable APIs is hard enough when talking about the actions, inputs, and outputs; making promises about logs would be way harder.

@D3vil0p3r
Copy link
Author

Thanks for the detailed answer. And ok, I agree. The only thing to fix should be the \u003e, but I guess, as mentioned by @Luap99 , it should be a quick fix.

@mtrmac
Copy link
Collaborator

mtrmac commented Jan 6, 2025

What I note is that, unlike docker API, the \u003e is not decoded as >.

\u003e is a valid JSON encoding. If a consumer has difficulties with that, that consumer needs to be fixed to parse JSON correctly, either way.

The producer can produce either one, and I wouldn’t object to a PR that changes this just to save some trivial amount of bandwidth, but either way the parser needs to be able to parse JSON as specified ( https://github.com/moby/moby/blob/0342576056e80459c18cd78a2cdbf6f29961fe20/pkg/jsonmessage/jsonmessage.go#L145 allows for arbitrary JSON fields with arbitrary types to be included); we are not going to be adding special commitments to avoid valid JSON values for the benefit of incorrect parsers.

@D3vil0p3r
Copy link
Author

Thank you for the answers. On my use case, that was to show a human readable download and extraction phase:
Image

the download progress works well, thanking your suggestions about compatMode=True.

The only missing information is the Extraction, that will stuck at that value, even when download progress bar ended, because, of course, podman API does not provide that info. But I also understand that touching APIs and keeping them stable is hard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests

5 participants