Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

solver: mark history and graph concistency errors as internal #5163

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions errdefs/errdefs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package errdefs

import "errors"

type internalErr struct {
error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add a pointer in the error structs in the containerd package?

The other option would be have these as a sibling in some sort of causal wrapper.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not fully understand the question, but this error type will return correctly in the current containerd IsInternal function.

Iiuc there is a issue that this customMessage https://github.com/containerd/errdefs/blob/124d0dc9f0cd27578004a3f14e01ef8a3456d147/errors.go#L396 does not have a Unwrap method but otherwise the custom IsInterface there looks equivalent to errors.As() in here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, all of the errors in errdefs don't have an internal pointer to a cause: https://github.com/containerd/errdefs/blob/124d0dc9f0cd27578004a3f14e01ef8a3456d147/errors.go#L40. I think we are good with the matching interface, but classically, errdefs represent the end of the line for the error chain.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but classically, errdefs represent the end of the line for the error chain.

I don't think that's a common case for internal actually. End error is likely a syscall error/parse error etc. (or if containerd loses the syscall error then the containerd API error). If containerd, for example, says blob not found, it could be a user error if the blob is wrong in manifest or wrong user input, but internal if it is during history addition, for example. Eg. in this PR, all the errors that come wrong writing history are automatically internal errors (they could be errors from blobstore, bolt, leases etc) because buildkit guarantees that it can save history on any build, no matter if the user asks for something broken. Same for syscall, lstat(user-copy-path) is user error, but lstat(boltdb) is internal.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed. Most internally errors will be unhandled errors that bubble up.

}

func (internalErr) System() {}

func (err internalErr) Unwrap() error {
return err.error
}

type system interface {
System()
}

var _ system = internalErr{}

func Internal(err error) error {
Copy link
Member Author

@tonistiigi tonistiigi Jul 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The separate function is mostly to avoid direct grpcerrors dependency in packages that don't interact with grpc. Otherwise WrapCode() is the generic version where any code can be passed directly.

That's also why this isn't part of solver/errdefs for example.

if err == nil {
return nil
}
return internalErr{err}
}

func IsInternal(err error) bool {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches the interface definition and behavior of https://github.com/containerd/errdefs/blob/124d0dc9f0cd27578004a3f14e01ef8a3456d147/errors.go#L304 (unreleased, not part of the v1).

In the future containerd IsInternal() function can be used instead in the caller side for equivalent behavior.

var s system
return errors.As(err, &s)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this just be errors.Is? Also, does system need to implement err for this to work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, Is would do a direct comparison of any error in chain.

If you mean adding func (e internalErr) Is(target) bool { ... interface check } and then calling errors.Is(err, InternalErr{}) instead, then this will only work if some error in the chain is type internalErr and not when there is any error that just satisfies system interface as otherwise the custom Is() method would not get called.

Also, does system need to implement err for this to work?

No. Err needs to be assignable to system. System does not need to be assignable to error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I see.

Typically, I would expect the following to be true for an internal error:

err := fallableWithInternalError()
errors.Is(err, errdefs.ErrInternal)

We'd likely need to have Unwrap() []error to make this work with a causal chain but that is why I was asking about having the pointer in the internalErr type above.

I guess it would have to be something like this:

type causal struct {
	err   error
	cause error
}

func (c causal) Error() string {
	return fmt.Sprintf("%v: %v", c.err, c.cause)
}

func (c causal) Unwrap() []error {
	errs := [2]error{c.err, c.cause}
	return errs[:]
}

type errInternal struct{}

var ErrInternal = errInternal{}

func (errInternal) Error() string { return "internal error" }

func WithCause(err, cause error) error {
	return causal{
		err:   err,
		cause: cause,
	}
}

func (errInternal) System() {}

type system interface {
	System()
	Error() string
}

func main() {
	var s system
	err := WithCause(ErrInternal, fmt.Errorf("the disk is done or something"))
	fmt.Println(errors.As(err, &s))
	fmt.Println(errors.Is(err, ErrInternal))
}

I don't think we need to block the PR on this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main difference is still that for errors.Is(err, ErrInternal) this would only work if errdefs.Internal() (or some form of Wrap(errdefs.ErrInternal) is called on the error and not if the system interface is implemented or not. So would not automatically pick up containerd and moby system errors.

}
10 changes: 6 additions & 4 deletions solver/llbsolver/solver.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"github.com/moby/buildkit/cache/remotecache"
"github.com/moby/buildkit/client"
controlgateway "github.com/moby/buildkit/control/gateway"
"github.com/moby/buildkit/errdefs"
"github.com/moby/buildkit/executor/resources"
resourcestypes "github.com/moby/buildkit/executor/resources/types"
"github.com/moby/buildkit/exporter"
Expand Down Expand Up @@ -158,7 +159,7 @@ func (s *Solver) Bridge(b solver.Builder) frontend.FrontendLLBBridge {
func (s *Solver) recordBuildHistory(ctx context.Context, id string, req frontend.SolveRequest, exp ExporterRequest, j *solver.Job, usage *resources.SysSampler) (func(context.Context, *Result, []exporter.DescriptorReference, error) error, error) {
stopTrace, err := detect.Recorder.Record(ctx)
if err != nil {
return nil, err
return nil, errdefs.Internal(err)
}

st := time.Now()
Expand All @@ -183,7 +184,7 @@ func (s *Solver) recordBuildHistory(ctx context.Context, id string, req frontend
if stopTrace != nil {
stopTrace()
}
return nil, err
return nil, errdefs.Internal(err)
}

return func(ctx context.Context, res *Result, descrefs []exporter.DescriptorReference, err error) error {
Expand Down Expand Up @@ -370,7 +371,8 @@ func (s *Solver) recordBuildHistory(ctx context.Context, id string, req frontend
})
}
if err1 := eg.Wait(); err == nil {
err = err1
// any error from exporting history record is internal
err = errdefs.Internal(err1)
}

defer func() {
Expand All @@ -397,7 +399,7 @@ func (s *Solver) recordBuildHistory(ctx context.Context, id string, req frontend
Record: rec,
}); err1 != nil {
if err == nil {
err = err1
err = errdefs.Internal(err1)
}
}

Expand Down
3 changes: 2 additions & 1 deletion solver/scheduler.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"os"
"sync"

"github.com/moby/buildkit/errdefs"
"github.com/moby/buildkit/solver/internal/pipe"
"github.com/moby/buildkit/util/bklog"
"github.com/moby/buildkit/util/cond"
Expand Down Expand Up @@ -403,7 +404,7 @@ func (pf *pipeFactory) NewInputRequest(ee Edge, req *edgeRequest) pipe.Receiver
WithField("edge_index", ee.Index).
Error("failed to get edge: inconsistent graph state")
return pf.NewFuncRequest(func(_ context.Context) (interface{}, error) {
return nil, errors.Errorf("failed to get edge: inconsistent graph state in edge %s %s %d", ee.Vertex.Name(), ee.Vertex.Digest(), ee.Index)
return nil, errdefs.Internal(errors.Errorf("failed to get edge: inconsistent graph state in edge %s %s %d", ee.Vertex.Name(), ee.Vertex.Digest(), ee.Index))
})
}
p := pf.s.newPipe(target, pf.e, pipe.Request{Payload: req})
Expand Down
5 changes: 5 additions & 0 deletions util/grpcerrors/grpcerrors.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
gogotypes "github.com/gogo/protobuf/types"
"github.com/golang/protobuf/proto" //nolint:staticcheck
"github.com/golang/protobuf/ptypes/any"
"github.com/moby/buildkit/errdefs"
"github.com/moby/buildkit/util/bklog"
"github.com/moby/buildkit/util/stack"
spb "google.golang.org/genproto/googleapis/rpc/status"
Expand Down Expand Up @@ -94,6 +95,10 @@ func withDetails(ctx context.Context, s *status.Status, details ...proto.Message
}

func Code(err error) codes.Code {
if errdefs.IsInternal(err) {
return codes.Internal
}

if se, ok := err.(interface {
Code() codes.Code
}); ok {
Expand Down
Loading