expression: `StaticExprContext/EvalContext` support to load state from system variables #55800

lcwangchao · 2024-09-02T09:32:27Z

What problem does this PR solve?

Issue Number: close #55799

Problem Summary:

Sometimes we need to load state of StaticExprContext/EvalContext from system variables, just like in lightning:

tidb/pkg/lightning/backend/kv/session.go

Lines 327 to 357 in df02db0

    
           if options.SysVars != nil { 
        
           	for k, v := range options.SysVars { 
        
           		// since 6.3(current master) tidb checks whether we can set a system variable 
        
           		// lc_time_names is a read-only variable for now, but might be implemented later, 
        
           		// so we not remove it from defaultImportantVariables and check it in below way. 
        
           		if sv := variable.GetSysVar(k); sv == nil { 
        
           			logger.DPanic("unknown system var", zap.String("key", k)) 
        
           			continue 
        
           		} else if sv.ReadOnly { 
        
           			logger.Debug("skip read-only variable", zap.String("key", k)) 
        
           			continue 
        
           		} 
        
           		if err := vars.SetSystemVar(k, v); err != nil { 
        
           			logger.DPanic("new session: failed to set system var", 
        
           				log.ShortError(err), 
        
           				zap.String("key", k)) 
        
           		} 
        
           	} 
        
           } 
        
           vars.StmtCtx.SetTimeZone(vars.Location()) 
        
           if err := vars.SetSystemVar("timestamp", strconv.FormatInt(options.Timestamp, 10)); err != nil { 
        
           	logger.Warn("new session: failed to set timestamp", 
        
           		log.ShortError(err)) 
        
           } 
        
           vars.TxnCtx = nil 
        
           s.Vars = vars 
        
           s.exprCtx = exprctximpl.NewSessionExprContext(s) 
        
           s.tblctx = tbctximpl.NewTableContextImpl(s) 
        
           s.txn.kvPairs = &Pairs{} 
        
           return s

What changed and how does it work?

This PR introduces a new method LoadSystemVars to StaticExprContext and StaticEvalContext to load state from system variables.
Also do some modifies in deeptest to make it easier to use.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

…m system variables

tiprow · 2024-09-02T09:32:58Z

Hi @lcwangchao. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-02T09:33:44Z

pkg/util/deeptest/statictesthelper.go

@@ -157,6 +157,9 @@ func (h *staticTestHelper) assertDeepClonedEqual(t require.TestingT, valA, valB
 			h.assertDeepClonedEqual(t, valA.Elem(), valB.Elem(), path)
 		}
 	case reflect.Slice:
+		if valA.IsNil() && valB.IsNil() {


For slice, map, and function, they can just be recognized as deep cloned when comparing nil

codecov · 2024-09-02T09:43:57Z

Codecov Report

Attention: Patch coverage is 85.71429% with 12 lines in your changes missing coverage. Please review.

Project coverage is 57.9122%. Comparing base (df02db0) to head (874260f).
Report is 29 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #55800         +/-   ##
=================================================
- Coverage   72.9210%   57.9122%   -15.0088%     
=================================================
  Files          1585       1729        +144     
  Lines        443451     633971     +190520     
=================================================
+ Hits         323369     367147      +43778     
- Misses       100237     242952     +142715     
- Partials      19845      23872       +4027

Flag	Coverage Δ
integration	`39.2420% <0.0000%> (?)`
unit	`73.5542% <85.7142%> (+1.5039%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`54.5253% <ø> (+1.5686%)`	⬆️
parser	`∅ <ø> (∅)`
br	`54.2881% <ø> (+8.8366%)`	⬆️

YangKeao

LGTM

pkg/expression/contextstatic/evalctx.go

pkg/expression/contextstatic/exprctx.go

ti-chi-bot · 2024-09-04T09:32:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XuHuaiyu, YangKeao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [XuHuaiyu,YangKeao]
~~pkg/expression/OWNERS~~ [XuHuaiyu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-09-04T09:32:39Z

[LGTM Timeline notifier]

Timeline:

2024-09-03 09:22:13.154194392 +0000 UTC m=+349857.672247329: ☑️ agreed by YangKeao.
2024-09-04 09:32:38.148167469 +0000 UTC m=+436882.666220406: ☑️ agreed by XuHuaiyu.

lcwangchao · 2024-09-04T11:00:15Z

/retest

tiprow · 2024-09-04T11:00:37Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-04T11:36:33Z

/retest

tiprow · 2024-09-04T11:36:54Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-04T12:27:03Z

/retest

tiprow · 2024-09-04T12:27:28Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-04T12:53:55Z

/retest

tiprow · 2024-09-04T12:54:20Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-04T14:44:45Z

/retest

tiprow · 2024-09-04T14:45:07Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

expression: StaticExprContext/EvalContext support to load state fro…

874260f

…m system variables

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 2, 2024

lcwangchao commented Sep 2, 2024

View reviewed changes

lcwangchao requested review from YangKeao and XuHuaiyu September 2, 2024 09:36

YangKeao approved these changes Sep 3, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 3, 2024

XuHuaiyu reviewed Sep 4, 2024

View reviewed changes

pkg/expression/contextstatic/evalctx.go Show resolved Hide resolved

pkg/expression/contextstatic/exprctx.go Show resolved Hide resolved

XuHuaiyu approved these changes Sep 4, 2024

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 4, 2024

ti-chi-bot bot merged commit e77d4a1 into pingcap:master Sep 4, 2024
23 checks passed

lcwangchao deleted the exprctxloadvars branch September 4, 2024 15:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

expression: `StaticExprContext/EvalContext` support to load state from system variables #55800

expression: `StaticExprContext/EvalContext` support to load state from system variables #55800

lcwangchao commented Sep 2, 2024 •

edited

Loading

tiprow bot commented Sep 2, 2024

lcwangchao Sep 2, 2024

codecov bot commented Sep 2, 2024 •

edited

Loading

YangKeao left a comment

ti-chi-bot bot commented Sep 4, 2024

ti-chi-bot bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

	if options.SysVars != nil {
	for k, v := range options.SysVars {
	// since 6.3(current master) tidb checks whether we can set a system variable
	// lc_time_names is a read-only variable for now, but might be implemented later,
	// so we not remove it from defaultImportantVariables and check it in below way.
	if sv := variable.GetSysVar(k); sv == nil {
	logger.DPanic("unknown system var", zap.String("key", k))
	continue
	} else if sv.ReadOnly {
	logger.Debug("skip read-only variable", zap.String("key", k))
	continue
	}
	if err := vars.SetSystemVar(k, v); err != nil {
	logger.DPanic("new session: failed to set system var",
	log.ShortError(err),
	zap.String("key", k))
	}
	}
	}
	vars.StmtCtx.SetTimeZone(vars.Location())
	if err := vars.SetSystemVar("timestamp", strconv.FormatInt(options.Timestamp, 10)); err != nil {
	logger.Warn("new session: failed to set timestamp",
	log.ShortError(err))
	}
	vars.TxnCtx = nil
	s.Vars = vars
	s.exprCtx = exprctximpl.NewSessionExprContext(s)
	s.tblctx = tbctximpl.NewTableContextImpl(s)
	s.txn.kvPairs = &Pairs{}

	return s

expression: StaticExprContext/EvalContext support to load state from system variables #55800

expression: StaticExprContext/EvalContext support to load state from system variables #55800

Conversation

lcwangchao commented Sep 2, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Sep 2, 2024

lcwangchao Sep 2, 2024

Choose a reason for hiding this comment

codecov bot commented Sep 2, 2024 • edited Loading

Codecov Report

YangKeao left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 4, 2024

ti-chi-bot bot commented Sep 4, 2024

[LGTM Timeline notifier]

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

lcwangchao commented Sep 4, 2024

tiprow bot commented Sep 4, 2024

expression: `StaticExprContext/EvalContext` support to load state from system variables #55800

expression: `StaticExprContext/EvalContext` support to load state from system variables #55800

lcwangchao commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 2, 2024 •

edited

Loading