Skip to content

introduce ImagePullResponse with helper method to manage JSONMessage stream decoding#50935

Merged
robmry merged 1 commit intomoby:masterfrom
ndeloof:decode-JSONMessage
Oct 9, 2025
Merged

introduce ImagePullResponse with helper method to manage JSONMessage stream decoding#50935
robmry merged 1 commit intomoby:masterfrom
ndeloof:decode-JSONMessage

Conversation

@ndeloof
Copy link
Contributor

@ndeloof ndeloof commented Sep 9, 2025

- What I did

Introduce ImagePullResponse with helper method to manage JSONMessage stream decoding. Still can be consumed as a raw io.Reader for backward compatibility

- How I did it

- How to verify it

- Human readable description for the release notes

`ImagePull` now returns an object with `JSONMessages` method returning iterator over the message objects.

- A picture of a cute animal (not mandatory but encouraged)
image

@thaJeztah
Copy link
Member

Thanks! This is something I definitely want to have included as part of the planned rewrite of the client; it want to have a closer look at some of the handling;

Some of the streaming responses require special handling to detect errors during the stream; for example pulling an image was successfully started, but an error-occurred while pulling. I think those require checking specific fields that are returned in the stream. We should make sure that those errors are returned as error of the ImagePull function. Ultimately, I'd want these streaming functions to default to being synchronous;

<possibly some return type>, err := client.ImagePull()
if err != nil {
	// pull didn't complete successfully
}

Users of the function could pass a discard logger to omit output;

result, err := client.ImagePull(WithDiscardOutput)

// or
result, err := client.ImagePull(WithProgressWriter(DiscardLogger))

Or pass multiple loggers / stream processors;

result, err := client.ImagePull(
	WithProgressWriter(MyPrettyPrinter),
	WithProgressWriter(WriteToLogFile)
)

Some things to look into (without having it an in-depth thought);

cc @vvoland @austinvazquez

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 9, 2025

@thaJeztah thanks for this promising plan toward a revisited client SDK!
My goal here is not so ambitious, just a baby step without signature update to make usage easier for general purpose usage

}

// Messages decodes the response stream as a sequence of JSONMessages
func (r *ImagePullResponse) Messages() iter.Seq2[*jsonmessage.JSONMessage, error] {
Copy link
Contributor

@vvoland vvoland Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should make it more explicit -

Suggested change
func (r *ImagePullResponse) Messages() iter.Seq2[*jsonmessage.JSONMessage, error] {
func (r *ImagePullResponse) JSONMessages() iter.Seq2[*jsonmessage.JSONMessage, error] {

In future, I think we should have a better public interface than JSONMessage and we wouldn't need to change the signaure of Messages then.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, perhaps worth passing a ctx so there's a way to cancel this without consuming all messages?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added context as parameter in JSONMessages so stream processing is stopped and reader closed if context is cancelled + test case

@ndeloof ndeloof force-pushed the decode-JSONMessage branch 2 times, most recently from 3595df1 to 7fdc424 Compare October 3, 2025 10:00
@@ -204,18 +208,29 @@ func (jm *JSONMessage) Display(out io.Writer, isTerminal bool) error {
// called if a JSONMessage contains an Aux field, in which case
// DisplayJSONMessagesStream does not present the JSONMessage.
func DisplayJSONMessagesStream(in io.Reader, out io.Writer, terminalFd uintptr, isTerminal bool, auxCallback func(JSONMessage)) error {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left this func for backward compatibility, maybe could ne removed ?

@@ -204,18 +208,29 @@ func (jm *JSONMessage) Display(out io.Writer, isTerminal bool) error {
// called if a JSONMessage contains an Aux field, in which case
// DisplayJSONMessagesStream does not present the JSONMessage.
func DisplayJSONMessagesStream(in io.Reader, out io.Writer, terminalFd uintptr, isTerminal bool, auxCallback func(JSONMessage)) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a "note to self", as @vvoland 's comment reminded me of that.

I recall when we moved this package that we also considered cleaning up the API; there was a change in the CLI to improve some bits (passing context etc), but also to properly take advantage of the package name, so changing these to jsonmessage.Stream and jsonmessage.Display (or jsonmessage.DisplayStream).

https://github.com/docker/cli/blob/v28.5.0/internal/jsonstream/display.go

NOT a blocker for this PR IMO, as it's some bike-shedding, but we should look at that before we release.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering if this is safe to change DisplayJSONMessagesStream signature to drop support for io.Reader. If this is fine, then I can also adopt a shorted name

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👇 copy/paste from Slack

strictly speaking, moby/client is a completely new module; we have no requirement to carry anything from the old docker/docker/client module, as long as we can provide a replacement for users of it, which sometimes requires a bit of digging (I tend to use https://grep.app to get a quick "guess" what's used, and where, and if it's relevant) to see if it can reasonably be swapped.

The whole json-message was a massacre; it really had to be reverse-engineered to understand "what was the meaning here in the first place?"; fun things I discovered was that (IIRC) there were no daemon-side types corresponding with it! It just happened to be a type that could unmarshal some random other types produced by the daemon because some of the fields matched (basically it "cherry-picked" some fields produced by build, some other fields for events, and yet some other fields for image pull / push streams).

I jotted down some things in #50575 - but even with changes I made in the API, it's possible that definitions are in the wrong place, or ... shouldn't be there;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines 21 to 22
// Messages decodes the response stream as a sequence of JSONMessages
func (r *ImagePullResponse) JSONMessages() iter.Seq2[jsonmessage.JSONMessage, error] {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also to be clear w.r.t. my other comment; this SGTM, because we're not in the jsonmessage package here, so it's ok to be explicit.

@ndeloof ndeloof force-pushed the decode-JSONMessage branch from 7fdc424 to c77feb6 Compare October 3, 2025 10:11
)

type ImagePullResponse struct {
io.ReadCloser
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
io.ReadCloser
rd io.ReadCloser

We should probably make it a concrete field and then implementing the io.ReadCloser for the ImagePullResponse.

This will also allow us to wrap Close method so we don't close the underlying reader twice in cases like:

out, err := ImagePull(...)
...

defer out.Close()

for i in out.JSONMessages() {
...
}
...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@ndeloof ndeloof force-pushed the decode-JSONMessage branch 3 times, most recently from ea3735a to e098d39 Compare October 3, 2025 12:13
Comment on lines 26 to 33
// Close implements io.ReadCloser
func (r ImagePullResponse) Close() (err error) {
if r.rc != nil {
err = r.rc.Close()
r.rc = nil
}
return
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Close implements io.ReadCloser
func (r ImagePullResponse) Close() (err error) {
if r.rc != nil {
err = r.rc.Close()
r.rc = nil
}
return
}
// Close implements io.ReadCloser
func (r *ImagePullResponse) Close() (err error) {
if r.rc != nil {
err = r.rc.Close()
r.rc = nil
}
return
}

This needs a pointer receiver, otherwise the r.rc = nil will only be applied to the local copy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed. But then ImagePullResponse can't be used as an io.ReadCloser (need to return *ImagePullResponse). Let me test another approach

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, do we have a strict requirement for ImagePull to not return a pointer?

cc @thaJeztah

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked in depth at the problem, but w.r.t. pointer vs not; I don't think we have a strict requirement; In fact for some cases, I like pointers, because they allow return nil, err, and callers should never consume returned values on error 😄. See #51076 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a quick comment; I'd like to avoid named error output variables; at least if they're named err - in this case it's a very small function, but we've had cases where errors were not handled correctly, so;

// Close implements io.ReadCloser
func (r *ImagePullResponse) Close() error {
	if r.rc != nil {
		err := r.rc.Close()
		r.rc = nil
		return err
	}
	return nil
}

Or perhaps an early return;

// Close implements io.ReadCloser
func (r *ImagePullResponse) Close() error {
	if r.rc == nil {
		return nil
	}
	err := r.rc.Close()
	r.rc = nil
	return err
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at that; could there be a race happening in the above? (can close be called by anything concurrently and panic if one wins and sets r.rc to nil?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's indeed risks for a race condition. Use of sync.Once prevents this as well

@thaJeztah thaJeztah added impact/changelog kind/refactor PR's that refactor, or clean-up code area/go-sdk labels Oct 3, 2025
@ndeloof ndeloof force-pushed the decode-JSONMessage branch from e098d39 to 4ee43d4 Compare October 3, 2025 12:45
@ndeloof ndeloof marked this pull request as ready for review October 3, 2025 12:45
@thaJeztah
Copy link
Member

minor issue with an import @ndeloof (haven't looked at the code, but should be easy fix)

# github.com/moby/moby/client [github.com/moby/moby/client.test]
client/image_pull.go:7:2: "fmt" imported and not used

@ndeloof ndeloof force-pushed the decode-JSONMessage branch 2 times, most recently from 27d19b0 to 24651b2 Compare October 3, 2025 15:21
@ndeloof ndeloof changed the title introduce ProgressFunc so ImagePull manages stream decoding introduce ImagePullResponse with helper method to manages JSONMessages stream decoding Oct 4, 2025
@ndeloof ndeloof changed the title introduce ImagePullResponse with helper method to manages JSONMessages stream decoding introduce ImagePullResponse with helper method to manage JSONMessage stream decoding Oct 4, 2025
@thaJeztah thaJeztah added this to the 29.0.0 milestone Oct 6, 2025
@thaJeztah thaJeztah added the release-blocker PRs we want to block a release on label Oct 6, 2025
"github.com/moby/moby/client/pkg/jsonmessage"
)

func NewImagePullResponse(rc io.ReadCloser) ImagePullResponse {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick comment before I forget (havent reviewed the latest iteration in depth);

do we want this one exported, or consider it internal to the client?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me make this one internal, we could export in the future if there's some usages for it - better safe than sorry

@ndeloof ndeloof force-pushed the decode-JSONMessage branch 5 times, most recently from 9e8cdee to bf5adeb Compare October 8, 2025 09:33
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@robmry want to have a peek as well?

@ndeloof ndeloof force-pushed the decode-JSONMessage branch from bf5adeb to f1e6b53 Compare October 8, 2025 16:03
Copy link
Contributor

@vvoland vvoland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@robmry
Copy link
Contributor

robmry commented Oct 9, 2025

@robmry want to have a peek as well?

Missed that - looking now ...

Copy link
Contributor

@robmry robmry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just nits / comments-on-comments ...

return DisplayJSONMessages(f, out, terminalFd, isTerminal, auxCallback)
}

func DisplayJSONMessages(messages JSONMessagesStream, out io.Writer, terminalFd uintptr, isTerminal bool, auxCallback func(JSONMessage)) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a docstring?

return
}

// Messages decodes the response stream as a sequence of JSONMessages
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be worth saying it closes the ImagePullResponse when the context is cancelled or the stream ends?

// It's up to the caller to handle the [io.ReadCloser] and close it.
func (cli *Client) ImagePull(ctx context.Context, refStr string, options ImagePullOptions) (io.ReadCloser, error) {
// Caller can rely on ImagePullResponse#Messages() to monitor pull progress as a sequence of JSONMessages
// otherwise it's up to the caller to handle the [io.ReadCloser] and close it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// otherwise it's up to the caller to handle the [io.ReadCloser] and close it.
// Callers can use [ImagePullResponse.JSONMessages] to monitor pull progress as a sequence of JSONMessages, `[ImagePullResponse.Close]` does not need to be called in this case. Or, use the
// `[io.Reader]` interface and call `[ImagePullResponse.Close]`.

Comment on lines 36 to 46
func (r ImagePullResponse) Close() (err error) {
if r.close == nil {
return nil
}
r.close.Do(func() {
if r.rc != nil {
err = r.rc.Close()
}
})
return
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably best not to use a named result & naked return, and just var err error in the body.

@ndeloof ndeloof force-pushed the decode-JSONMessage branch 2 times, most recently from cf23b40 to ccc580a Compare October 9, 2025 13:22
Comment on lines 49 to 51
// Messages decodes the response stream as a sequence of JSONMessages.
// if stream ends ir context is cancelled, the underlying [io.Reader] is closed.
func (r ImagePullResponse) JSONMessages(ctx context.Context) iter.Seq2[jsonmessage.JSONMessage, error] {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh - this was wrong before, but I don't think the comment will be picked up by godoc as the function name doesn't match. Maybe worth fixing, now it has more info?

Suggested change
// Messages decodes the response stream as a sequence of JSONMessages.
// if stream ends ir context is cancelled, the underlying [io.Reader] is closed.
func (r ImagePullResponse) JSONMessages(ctx context.Context) iter.Seq2[jsonmessage.JSONMessage, error] {
// JSONMessages decodes the response stream as a sequence of
// [jsonmessage.JSONMessage].
// If the stream ends or context is cancelled, the underlying [io.Reader] is closed.
func (r ImagePullResponse) JSONMessages(ctx context.Context) iter.Seq2[jsonmessage.JSONMessage, error] {

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
@ndeloof ndeloof force-pushed the decode-JSONMessage branch from ccc580a to e6bac89 Compare October 9, 2025 13:36
Copy link
Contributor

@robmry robmry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - thank you!

@robmry robmry merged commit 802142d into moby:master Oct 9, 2025
181 checks passed
@ndeloof ndeloof deleted the decode-JSONMessage branch October 9, 2025 15:31
@ndeloof
Copy link
Contributor Author

ndeloof commented Oct 9, 2025

follow-up: #51148

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/go-sdk impact/changelog kind/refactor PR's that refactor, or clean-up code module/client release-blocker PRs we want to block a release on

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants