Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Handle panics in the rpc server #1330

Merged
merged 2 commits into from
Apr 11, 2023

Conversation

AndrewSisley
Copy link
Contributor

Relevant issue(s)

Resolves #1298

Description

Adds middleware to handle panics in the rpc server.

Currently panics via the server will result in the full defradb instance crashing, this PR adds middleware that converts the panics into errors.

Tested by hacking in a panic and calling into the server via the CLI to see if it results in an error or a panic/crash.

@AndrewSisley AndrewSisley added area/p2p Related to the p2p networking system code quality Related to improving code quality action/no-benchmark Skips the action that runs the benchmark. labels Apr 11, 2023
@AndrewSisley AndrewSisley added this to the DefraDB v0.5 milestone Apr 11, 2023
@AndrewSisley AndrewSisley requested a review from a team April 11, 2023 19:52
@AndrewSisley AndrewSisley self-assigned this Apr 11, 2023
go.mod Outdated
@@ -87,6 +87,7 @@ require (
github.com/google/gopacket v1.1.19 // indirect
github.com/google/pprof v0.0.0-20221203041831-ce31453925ec // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Don't think this should be indirect. Try running make tidy.

Copy link
Contributor Author

@AndrewSisley AndrewSisley Apr 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good spot - thanks Shahzad - I think I'm guilty of skimming over this file :)

  • make tidy

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right Shahzad. This probably needs to change.

@codecov
Copy link

codecov bot commented Apr 11, 2023

Codecov Report

Merging #1330 (9b30b7a) into develop (0c57c9b) will increase coverage by 0.08%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1330      +/-   ##
===========================================
+ Coverage    70.52%   70.60%   +0.08%     
===========================================
  Files          184      184              
  Lines        17792    17801       +9     
===========================================
+ Hits         12547    12568      +21     
+ Misses        4286     4278       -8     
+ Partials       959      955       -4     
Impacted Files Coverage Δ
cli/start.go 46.59% <100.00%> (+1.78%) ⬆️

... and 6 files with indirect coverage changes

MaxConnectionIdle: rpcTimeoutDuration,
}))
server := grpc.NewServer(
grpc.UnaryInterceptor(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: What was the reason for Unary over Stream?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it would be because we don't use rpc streaming.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was easy to test that - I had no idea which one we used and added the streaming one first - to no effect on the CLI. If we end up using it at some point we can add in the middleware for it then IMO

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation :)

Copy link
Member

@shahzadlone shahzadlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for doing make tidy. Got one question for you.

@@ -20,6 +20,8 @@ import (
"strings"

badger "github.com/dgraph-io/badger/v3"
grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: If you don't mind, there is v2 that is now available. We should probably use that instead.

Copy link
Contributor Author

@AndrewSisley AndrewSisley Apr 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

V2 is not released yet (pre-release build only)

MaxConnectionIdle: rpcTimeoutDuration,
}))
server := grpc.NewServer(
grpc.UnaryInterceptor(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it would be because we don't use rpc streaming.

@AndrewSisley AndrewSisley dismissed fredcarle’s stale review April 11, 2023 21:48

requested change is not possible, and I'd like to merge this and another PR before logging off today before the release

@AndrewSisley AndrewSisley merged commit fe304d6 into develop Apr 11, 2023
@AndrewSisley AndrewSisley deleted the sisley/fix/I1298-handle-rpc-panics branch April 11, 2023 21:49
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
action/no-benchmark Skips the action that runs the benchmark. area/p2p Related to the p2p networking system code quality Related to improving code quality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Panics in net/api package crash the server taking it offline
3 participants