Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding option to randomize Kinesis partition key #2705

Merged
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Godeps
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ github.com/go-sql-driver/mysql 2e00b5cd70399450106cec6431c2e2ce3cae5034
github.com/gobwas/glob bea32b9cd2d6f55753d94a28e959b13f0244797a
github.com/golang/protobuf 8ee79997227bf9b34611aee7946ae64735e6fd93
github.com/golang/snappy 7db9049039a047d955fe8c19b83c8ff5abd765c7
github.com/google/uuid 6a5e28554805e78ea6141142aba763936c4761c0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to use https://github.com/satori/go.uuid for uuids.

github.com/gorilla/mux 392c28fe23e1c45ddba891b0320b3b5df220beea
github.com/hailocab/go-hostpool e80d13ce29ede4452c43dea11e79b9bc8a15b478
github.com/hashicorp/consul 63d2fc68239b996096a1c55a0d4b400ea4c2583f
Expand Down
26 changes: 21 additions & 5 deletions plugins/outputs/kinesis/kinesis.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (

"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/kinesis"
"github.com/google/uuid"

"github.com/influxdata/telegraf"
internalaws "github.com/influxdata/telegraf/internal/config/aws"
Expand All @@ -23,10 +24,11 @@ type KinesisOutput struct {
Filename string `toml:"shared_credential_file"`
Token string `toml:"token"`

StreamName string `toml:"streamname"`
PartitionKey string `toml:"partitionkey"`
Debug bool `toml:"debug"`
svc *kinesis.Kinesis
StreamName string `toml:"streamname"`
PartitionKey string `toml:"partitionkey"`
RandomPartitionKey bool `toml:"randomize_partitionKey"`
Debug bool `toml:"debug"`
svc *kinesis.Kinesis

serializer serializers.Serializer
}
Expand Down Expand Up @@ -54,6 +56,11 @@ var sampleConfig = `
streamname = "StreamName"
## PartitionKey as used for sharding data.
partitionkey = "PartitionKey"
## If set the paritionKey will be a random UUID on every put.
## This allows for scaling across multiple shards in a stream.
## This will cause issues with ordering.
randomize_partitionKey = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to use_random_partitionkey. Mention that if this is set then partitionkey is ignored. Can you explain a little about the ordering issues (perhaps in the README)



## Data format to output.
## Each data format has it's own unique set of configuration options, read
Expand Down Expand Up @@ -173,9 +180,18 @@ func (k *KinesisOutput) Write(metrics []telegraf.Metric) error {
return err
}

partitionKey := k.PartitionKey
if k.RandomPartitionKey {
u, err := uuid.NewRandom()
if err != nil {
return err
}
partitionKey = u.String()
}

d := kinesis.PutRecordsRequestEntry{
Data: values,
PartitionKey: aws.String(k.PartitionKey),
PartitionKey: aws.String(partitionKey),
}

r = append(r, &d)
Expand Down