You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Rate limiting can be applied to all Get, List and column Hydrate calls.
For each call, multiple rate limiters may apply. When more than one limiter applies to a call,
the rate limiter with the largest required wait is respected.
The rate limiters which apply to a call are resolved using scopes. Each rate limiter definition specifies
scopes which apply to it, for example region, connection. Then for each call, the values of these scopes are
determined and used to identify which limiters apply.
Defining Rate Limiters
Rate limiters may be defined in the plugin definition (by the plugin author), or in HCL config (by the user)
Plugin Definition
A rate limiters is defined using the Definition struct:
typeDefinitionstruct {
// the limiter nameNamestring// the actual limiter configFillRate rate.LimitBucketSizeint// the scopes which identify this limiter instance// one limiter instance will be created for each combination of scopes which is encounteredScopes []string// filter used to target the limiterWherestring
}
Scopes is list of all the scopes which apply to the rate limiter.
For example, if you want a rate limiter that applies to a single account, region and service, you could use the scopes:
[connection, region,service].
(See below for details of predefined vs custom scope names)
Where is a SQL compatible where clause which allows a rate limiter to be targeted to spcific set of scope values,
for example to specify a rate limiter for a specific service only, the filter "service"="s3 be used.
Rate limiters may be define in HCL in an .spc file in the config folder.
If a limiter has the same name as one defined in the plugin it will override it, if not, a new limiter is defined.
When executing a hydrate call the following steps are followed:
Build the set of rate limiter definitions which may apply to the hydrate call
Build the set of scope values which apply to the hydrate call
Determine which limiter defintions are satisfied by the scope values (looking at both required scopes and the scope filters)
Build a MultiLimiter from the resultant limiter defintions
Resolving Scope Values
Scope values are popuylated from 3 sources:
implicit scope values populated automatically
tabe, connection
matrix scope values populated from matrix quals (e.g. region)
custom scope values (tags?) which may be defined in Table defintions, HydrateConfig, GetConfig and ListConfig
Paged List Calls
If the list call uses paging, the SDK provides a hook, WaitForListRateLimit, which can be called before paging to apply rate limiting to the list call:
// List callforpaginator.HasMorePages() {
// apply rate limitingd.WaitForListRateLimit(ctx)
output, err:=paginator.NextPage(ctx)
iferr!=nil {
plugin.Logger(ctx).Error("aws_codepipeline_pipeline.listCodepipelinePipelines", "api_error", err)
returnnil, err
}
for_, items:=rangeoutput.Pipelines {
d.StreamListItem(ctx, items)
// Context can be cancelled due to manual cancellation or the limit has been hitifd.RowsRemaining(ctx) ==0 {
returnnil, nil
}
}
}
NOTE: region must be defined as a matrix qual in order to use the matrix scope value,
and service must be defined as a custom scope value for tables or hydrate calls which this limiter targets.
2a. Table definition which defines a "region" key column and sets the "service" scope value for all hydrate calls
functableAwsS3AccessPoint(_ context.Context) *plugin.Table {
return&plugin.Table{
Name: "aws_s3_access_point",
List: &plugin.ListConfig{
Hydrate: listS3AccessPoints,
KeyColumns: plugin.SingleColumn("region"),
},
Get: &plugin.GetConfig{
KeyColumns: plugin.AllColumns([]string{"name", "region"}),
Hydrate: getS3AccessPoint,
},
// set "service" scope to "s3" for all hydrate callsTags: map[string]string{
"service": "s3",
},
Columns: awsRegionalColumns([]*plugin.Column{...}),
}
}
2b. Hydrate call definition which specifies the "service" scope value
functableAwsS3AccountSettings(_ context.Context) *plugin.Table {
return&plugin.Table{
Name: "aws_s3_account_settings",
List: &plugin.ListConfig{...},
HydrateConfig: []plugin.HydrateConfig{
{
Func: getAccountBucketPublicAccessBlock,
// set the "service" scope value for this hydrate callTags: map[string]string{
"service": "s3",
},
},
},
Columns: awsGlobalRegionColumns([]*plugin.Column{...}),
}
}
3. Plugin defines rate limiters for "s3" and "ec2" services and one for all other services
NOTE: also scoped by "connection" and "region"
// scopes used for all rate limitersvarrateLimiterScopes=[]string{"connection","service","region",}
funcPlugin(_ context.Context) *plugin.Plugin {
p:=&plugin.Plugin{
Name: pluginName,
TableMap: map[string]*plugin.Table{ ... },
RateLimiters: []*rate_limiter.Definition{
// rate limiter for s3 service
{
Limit: 20,
BurstSize: 5,
Scopes: rateLimiterScopes,
Where: "service='s3'",
},
},
// rate limiter for ec2 service
{
Limit: 40,
BurstSize: 5,
Scopes: rateLimiterScopes,
Where: "service='ec2'",
},
// rate limiter for all other services
{
Limit: 75,
BurstSize: 10,
Where: "service not in ('s3,'ec2')",
},
},
...
}
returnp
}
The text was updated successfully, but these errors were encountered:
Rate Limiting
Overview
Rate limiting can be applied to all
Get
,List
and columnHydrate
calls.For each call, multiple rate limiters may apply. When more than one limiter applies to a call,
the rate limiter with the largest required wait is respected.
The rate limiters which apply to a call are resolved using
scopes
. Each rate limiter definition specifiesscopes which apply to it, for example
region
,connection
. Then for each call, the values of these scopes aredetermined and used to identify which limiters apply.
Defining Rate Limiters
Rate limiters may be defined in the plugin definition (by the plugin author), or in HCL config (by the user)
Plugin Definition
A rate limiters is defined using the
Definition
struct:Scopes
is list of all the scopes which apply to the rate limiter.For example, if you want a rate limiter that applies to a single account, region and service, you could use the scopes:
[
connection
,region
,service
].(See below for details of predefined vs custom scope names)
Where
is a SQL compatible where clause which allows a rate limiter to be targeted to spcific set of scope values,for example to specify a rate limiter for a specific service only, the filter
"service"="s3
be used.For example:
HCL Definition
Rate limiters may be define in HCL in an
.spc
file in the config folder.If a limiter has the same name as one defined in the plugin it will override it, if not, a new limiter is defined.
Resolving Rate Limiters
When executing a hydrate call the following steps are followed:
Resolving Scope Values
Scope values are popuylated from 3 sources:
tabe
,connection
region
)Table
defintions,HydrateConfig
,GetConfig
andListConfig
Paged List Calls
If the list call uses paging, the SDK provides a hook,
WaitForListRateLimit
, which can be called before paging to apply rate limiting to the list call:Scenarios
1. Plugin defines a single unscoped rate limiter
2. Plugin defines a rate limiter scoped by implicit scope "connection", custom scope "service" and matrix scope "region"
Plugin definition
NOTE:
region
must be defined as a matrix qual in order to use the matrix scope value,and
service
must be defined as a custom scope value for tables or hydrate calls which this limiter targets.2a. Table definition which defines a "region" key column and sets the "service" scope value for all hydrate calls
2b. Hydrate call definition which specifies the "service" scope value
3. Plugin defines rate limiters for "s3" and "ec2" services and one for all other services
NOTE: also scoped by "connection" and "region"
The text was updated successfully, but these errors were encountered: