introduce ImagePullResponse with helper method to manage JSONMessage stream decoding#50935
introduce ImagePullResponse with helper method to manage JSONMessage stream decoding#50935robmry merged 1 commit intomoby:masterfrom
Conversation
|
Thanks! This is something I definitely want to have included as part of the planned rewrite of the client; it want to have a closer look at some of the handling; Some of the streaming responses require special handling to detect errors during the stream; for example pulling an image was successfully started, but an error-occurred while pulling. I think those require checking specific fields that are returned in the stream. We should make sure that those errors are returned as error of the <possibly some return type>, err := client.ImagePull()
if err != nil {
// pull didn't complete successfully
}Users of the function could pass a result, err := client.ImagePull(WithDiscardOutput)
// or
result, err := client.ImagePull(WithProgressWriter(DiscardLogger))Or pass multiple loggers / stream processors; result, err := client.ImagePull(
WithProgressWriter(MyPrettyPrinter),
WithProgressWriter(WriteToLogFile)
)Some things to look into (without having it an in-depth thought);
|
|
@thaJeztah thanks for this promising plan toward a revisited client SDK! |
370409c to
ce153ab
Compare
ce153ab to
66f0a39
Compare
9bc8b81 to
f2fe184
Compare
client/image_pull.go
Outdated
| } | ||
|
|
||
| // Messages decodes the response stream as a sequence of JSONMessages | ||
| func (r *ImagePullResponse) Messages() iter.Seq2[*jsonmessage.JSONMessage, error] { |
There was a problem hiding this comment.
I think we should make it more explicit -
| func (r *ImagePullResponse) Messages() iter.Seq2[*jsonmessage.JSONMessage, error] { | |
| func (r *ImagePullResponse) JSONMessages() iter.Seq2[*jsonmessage.JSONMessage, error] { |
In future, I think we should have a better public interface than JSONMessage and we wouldn't need to change the signaure of Messages then.
There was a problem hiding this comment.
Also, perhaps worth passing a ctx so there's a way to cancel this without consuming all messages?
There was a problem hiding this comment.
added context as parameter in JSONMessages so stream processing is stopped and reader closed if context is cancelled + test case
3595df1 to
7fdc424
Compare
| @@ -204,18 +208,29 @@ func (jm *JSONMessage) Display(out io.Writer, isTerminal bool) error { | |||
| // called if a JSONMessage contains an Aux field, in which case | |||
| // DisplayJSONMessagesStream does not present the JSONMessage. | |||
| func DisplayJSONMessagesStream(in io.Reader, out io.Writer, terminalFd uintptr, isTerminal bool, auxCallback func(JSONMessage)) error { | |||
There was a problem hiding this comment.
left this func for backward compatibility, maybe could ne removed ?
| @@ -204,18 +208,29 @@ func (jm *JSONMessage) Display(out io.Writer, isTerminal bool) error { | |||
| // called if a JSONMessage contains an Aux field, in which case | |||
| // DisplayJSONMessagesStream does not present the JSONMessage. | |||
| func DisplayJSONMessagesStream(in io.Reader, out io.Writer, terminalFd uintptr, isTerminal bool, auxCallback func(JSONMessage)) error { | |||
There was a problem hiding this comment.
Just a "note to self", as @vvoland 's comment reminded me of that.
I recall when we moved this package that we also considered cleaning up the API; there was a change in the CLI to improve some bits (passing context etc), but also to properly take advantage of the package name, so changing these to jsonmessage.Stream and jsonmessage.Display (or jsonmessage.DisplayStream).
https://github.com/docker/cli/blob/v28.5.0/internal/jsonstream/display.go
NOT a blocker for this PR IMO, as it's some bike-shedding, but we should look at that before we release.
There was a problem hiding this comment.
I was wondering if this is safe to change DisplayJSONMessagesStream signature to drop support for io.Reader. If this is fine, then I can also adopt a shorted name
There was a problem hiding this comment.
👇 copy/paste from Slack
strictly speaking, moby/client is a completely new module; we have no requirement to carry anything from the old docker/docker/client module, as long as we can provide a replacement for users of it, which sometimes requires a bit of digging (I tend to use https://grep.app to get a quick "guess" what's used, and where, and if it's relevant) to see if it can reasonably be swapped.
The whole json-message was a massacre; it really had to be reverse-engineered to understand "what was the meaning here in the first place?"; fun things I discovered was that (IIRC) there were no daemon-side types corresponding with it! It just happened to be a type that could unmarshal some random other types produced by the daemon because some of the fields matched (basically it "cherry-picked" some fields produced by build, some other fields for events, and yet some other fields for image pull / push streams).
I jotted down some things in #50575 - but even with changes I made in the API, it's possible that definitions are in the wrong place, or ... shouldn't be there;
There was a problem hiding this comment.
- opened a quick ticket to look at these functions after this PR is merged; client: look if we need DisplayJSONMessages and DisplayJSONMessagesStream #51145
client/image_pull.go
Outdated
| // Messages decodes the response stream as a sequence of JSONMessages | ||
| func (r *ImagePullResponse) JSONMessages() iter.Seq2[jsonmessage.JSONMessage, error] { |
There was a problem hiding this comment.
Also to be clear w.r.t. my other comment; this SGTM, because we're not in the jsonmessage package here, so it's ok to be explicit.
7fdc424 to
c77feb6
Compare
client/image_pull.go
Outdated
| ) | ||
|
|
||
| type ImagePullResponse struct { | ||
| io.ReadCloser |
There was a problem hiding this comment.
| io.ReadCloser | |
| rd io.ReadCloser |
We should probably make it a concrete field and then implementing the io.ReadCloser for the ImagePullResponse.
This will also allow us to wrap Close method so we don't close the underlying reader twice in cases like:
out, err := ImagePull(...)
...
defer out.Close()
for i in out.JSONMessages() {
...
}
...ea3735a to
e098d39
Compare
| // Close implements io.ReadCloser | ||
| func (r ImagePullResponse) Close() (err error) { | ||
| if r.rc != nil { | ||
| err = r.rc.Close() | ||
| r.rc = nil | ||
| } | ||
| return | ||
| } |
There was a problem hiding this comment.
| // Close implements io.ReadCloser | |
| func (r ImagePullResponse) Close() (err error) { | |
| if r.rc != nil { | |
| err = r.rc.Close() | |
| r.rc = nil | |
| } | |
| return | |
| } | |
| // Close implements io.ReadCloser | |
| func (r *ImagePullResponse) Close() (err error) { | |
| if r.rc != nil { | |
| err = r.rc.Close() | |
| r.rc = nil | |
| } | |
| return | |
| } |
This needs a pointer receiver, otherwise the r.rc = nil will only be applied to the local copy
There was a problem hiding this comment.
indeed. But then ImagePullResponse can't be used as an io.ReadCloser (need to return *ImagePullResponse). Let me test another approach
There was a problem hiding this comment.
Hmm, do we have a strict requirement for ImagePull to not return a pointer?
cc @thaJeztah
There was a problem hiding this comment.
I haven't looked in depth at the problem, but w.r.t. pointer vs not; I don't think we have a strict requirement; In fact for some cases, I like pointers, because they allow return nil, err, and callers should never consume returned values on error 😄. See #51076 (comment)
There was a problem hiding this comment.
Just a quick comment; I'd like to avoid named error output variables; at least if they're named err - in this case it's a very small function, but we've had cases where errors were not handled correctly, so;
// Close implements io.ReadCloser
func (r *ImagePullResponse) Close() error {
if r.rc != nil {
err := r.rc.Close()
r.rc = nil
return err
}
return nil
}Or perhaps an early return;
// Close implements io.ReadCloser
func (r *ImagePullResponse) Close() error {
if r.rc == nil {
return nil
}
err := r.rc.Close()
r.rc = nil
return err
}There was a problem hiding this comment.
Looking at that; could there be a race happening in the above? (can close be called by anything concurrently and panic if one wins and sets r.rc to nil?)
There was a problem hiding this comment.
there's indeed risks for a race condition. Use of sync.Once prevents this as well
e098d39 to
4ee43d4
Compare
|
minor issue with an import @ndeloof (haven't looked at the code, but should be easy fix) |
27d19b0 to
24651b2
Compare
client/image_pull.go
Outdated
| "github.com/moby/moby/client/pkg/jsonmessage" | ||
| ) | ||
|
|
||
| func NewImagePullResponse(rc io.ReadCloser) ImagePullResponse { |
There was a problem hiding this comment.
Quick comment before I forget (havent reviewed the latest iteration in depth);
do we want this one exported, or consider it internal to the client?
There was a problem hiding this comment.
let me make this one internal, we could export in the future if there's some usages for it - better safe than sorry
9e8cdee to
bf5adeb
Compare
bf5adeb to
f1e6b53
Compare
Missed that - looking now ... |
robmry
left a comment
There was a problem hiding this comment.
Just nits / comments-on-comments ...
| return DisplayJSONMessages(f, out, terminalFd, isTerminal, auxCallback) | ||
| } | ||
|
|
||
| func DisplayJSONMessages(messages JSONMessagesStream, out io.Writer, terminalFd uintptr, isTerminal bool, auxCallback func(JSONMessage)) error { |
client/image_pull.go
Outdated
| return | ||
| } | ||
|
|
||
| // Messages decodes the response stream as a sequence of JSONMessages |
There was a problem hiding this comment.
It'd be worth saying it closes the ImagePullResponse when the context is cancelled or the stream ends?
client/image_pull.go
Outdated
| // It's up to the caller to handle the [io.ReadCloser] and close it. | ||
| func (cli *Client) ImagePull(ctx context.Context, refStr string, options ImagePullOptions) (io.ReadCloser, error) { | ||
| // Caller can rely on ImagePullResponse#Messages() to monitor pull progress as a sequence of JSONMessages | ||
| // otherwise it's up to the caller to handle the [io.ReadCloser] and close it. |
There was a problem hiding this comment.
| // otherwise it's up to the caller to handle the [io.ReadCloser] and close it. | |
| // Callers can use [ImagePullResponse.JSONMessages] to monitor pull progress as a sequence of JSONMessages, `[ImagePullResponse.Close]` does not need to be called in this case. Or, use the | |
| // `[io.Reader]` interface and call `[ImagePullResponse.Close]`. |
client/image_pull.go
Outdated
| func (r ImagePullResponse) Close() (err error) { | ||
| if r.close == nil { | ||
| return nil | ||
| } | ||
| r.close.Do(func() { | ||
| if r.rc != nil { | ||
| err = r.rc.Close() | ||
| } | ||
| }) | ||
| return | ||
| } |
There was a problem hiding this comment.
Probably best not to use a named result & naked return, and just var err error in the body.
cf23b40 to
ccc580a
Compare
client/image_pull.go
Outdated
| // Messages decodes the response stream as a sequence of JSONMessages. | ||
| // if stream ends ir context is cancelled, the underlying [io.Reader] is closed. | ||
| func (r ImagePullResponse) JSONMessages(ctx context.Context) iter.Seq2[jsonmessage.JSONMessage, error] { |
There was a problem hiding this comment.
Oh - this was wrong before, but I don't think the comment will be picked up by godoc as the function name doesn't match. Maybe worth fixing, now it has more info?
| // Messages decodes the response stream as a sequence of JSONMessages. | |
| // if stream ends ir context is cancelled, the underlying [io.Reader] is closed. | |
| func (r ImagePullResponse) JSONMessages(ctx context.Context) iter.Seq2[jsonmessage.JSONMessage, error] { | |
| // JSONMessages decodes the response stream as a sequence of | |
| // [jsonmessage.JSONMessage]. | |
| // If the stream ends or context is cancelled, the underlying [io.Reader] is closed. | |
| func (r ImagePullResponse) JSONMessages(ctx context.Context) iter.Seq2[jsonmessage.JSONMessage, error] { |
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ccc580a to
e6bac89
Compare
|
follow-up: #51148 |
- What I did
Introduce
ImagePullResponsewith helper method to manageJSONMessagestream decoding. Still can be consumed as a rawio.Readerfor backward compatibility- How I did it
- How to verify it
- Human readable description for the release notes
- A picture of a cute animal (not mandatory but encouraged)
