Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: 更新过程中服务间调用失败导致无损更新失效 #3242

Closed
jsonwan opened this issue Oct 9, 2024 · 0 comments
Closed

fix: 更新过程中服务间调用失败导致无损更新失效 #3242

jsonwan opened this issue Oct 9, 2024 · 0 comments
Assignees
@jsonwan
Labels
done 已上线到正式环境并验收通过 kind/enhancement 功能改进特性

Comments

@jsonwan
Copy link
Collaborator

jsonwan commented Oct 9, 2024

在业务量较大的环境中发现无损更新失效，更新过程中出现3类报错，各类报错起止时间点分别如下：
Connection refused：
2024-10-09 11:20:47.709
2024-10-09 11:23:02.262
connect timed out：
2024-10-09 11:20:54.853
2024-10-09 11:24:19.918
No route to host：
2024-10-09 11:21:22.883
2024-10-09 11:24:41.076
需要排查解决。
@jsonwan jsonwan added kind/enhancement 功能改进特性 backlog 需求初始状态，等待产品进行评估 labels Oct 9, 2024
@jsonwan jsonwan self-assigned this Oct 9, 2024
@bkjob-bot bkjob-bot added todo 进入开发排期的状态，纳入了最近的迭代 and removed backlog 需求初始状态，等待产品进行评估 labels Oct 9, 2024
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
3299c23 
调整GracefulShutdown等待时间，观察负载均衡缓存刷新情况。
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
e1ac6ae 
打印负载均衡实例获取记录，分析缓存未刷新原因。
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
5cbcc51 
打印负载均衡实例获取记录，分析缓存未刷新原因。
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
13a34b5 
1.打印FeignClient调用失败时的真实URL信息用于排查问题；
2.打印负载均衡实例获取记录日志。
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
4a27be5 
1.调整GracefulShutdown等待时间为40s，避免边界误差导致负载均衡缓存未刷新。
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
6d82452 
1.抽取配置至公共模块，应用于所有服务。
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
03e9fcf 
1.修复非k8s模式编译失败问题。
jsonwan added a commit to jsonwan/bk-job that referenced this issue Oct 11, 2024
@jsonwan
fix: 更新过程中服务间调用失败导致无损更新失效 TencentBlueKing#3242
c390d9e 
1.处理review意见。
jsonwan added a commit that referenced this issue Oct 11, 2024
@jsonwan
Merge pull request #3244 from jsonwan/github_fix/graceful_update
e5816c4 
fix: 更新过程中服务间调用失败导致无损更新失效 #3242
@bkjob-bot bkjob-bot added doing 已经接受处理，正在开发实施 for gray 可以在灰度环境/预发布环境验收 done 已上线到正式环境并验收通过 and removed todo 进入开发排期的状态，纳入了最近的迭代 doing 已经接受处理，正在开发实施 for gray 可以在灰度环境/预发布环境验收 labels Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsonwan jsonwan

Labels
done 已上线到正式环境并验收通过 kind/enhancement 功能改进特性
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jsonwan @bkjob-bot