Skip to content

Commit

Permalink
Merge pull request kube-logging#1643 from kube-logging/fix-openobserv…
Browse files Browse the repository at this point in the history
…e-port-arg

fix(syslog-ng): add port to OpenObserve output
  • Loading branch information
pepov authored Jan 19, 2024
2 parents ec3f380 + 9a8e948 commit 68f0e88
Show file tree
Hide file tree
Showing 9 changed files with 120 additions and 15 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1329,6 +1329,8 @@ spec:
type: object
persist_name:
type: string
port:
type: integer
record:
type: string
response-action:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1325,6 +1325,8 @@ spec:
type: object
persist_name:
type: string
port:
type: integer
record:
type: string
response-action:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1329,6 +1329,8 @@ spec:
type: object
persist_name:
type: string
port:
type: integer
record:
type: string
response-action:
Expand Down
2 changes: 2 additions & 0 deletions config/crd/bases/logging.banzaicloud.io_syslogngoutputs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1325,6 +1325,8 @@ spec:
type: object
persist_name:
type: string
port:
type: integer
record:
type: string
response-action:
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/plugins/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ For more information please click on the plugin name
| **[Loki](syslogng-outputs/loki/)** | syslogng-outputs | Sending messages to Loki over gRPC | Testing | [](https://axoflow.com/docs/axosyslog-core/chapter-destinations/destination-loki/) |
| **[MongoDB Destination](syslogng-outputs/mongodb/)** | syslogng-outputs | Sending messages into MongoDB Server | Testing | [](https://axoflow.com/docs/axosyslog-core/chapter-destinations/configuring-destinations-mongodb/) |
| **[MQTT Destination](syslogng-outputs/mqtt/)** | syslogng-outputs | Sending messages over MQTT Protocol | Testing | [](https://axoflow.com/docs/axosyslog-core/chapter-destinations/destination-mqtt-intro/) |
| **[Openobserve](syslogng-outputs/openobserve/)** | syslogng-outputs | Sending messages over Openobserve | Testing | [](https://axoflow.com/docs/axosyslog-core/chapter-destinations/openobserve/) |
| **[OpenObserve](syslogng-outputs/openobserve/)** | syslogng-outputs | Sending messages over OpenObserve | Testing | [](https://axoflow.com/docs/axosyslog-core/chapter-destinations/openobserve/) |
| **[Redis Server Destination](syslogng-outputs/redis/)** | syslogng-outputs | Sending messages from local network to the Redis server | Testing | [](https://axoflow.com/docs/axosyslog-core/chapter-destinations/configuring-destinations-redis/) |
| **[S3 Destination](syslogng-outputs/s3/)** | syslogng-outputs | Sending messages from a local network to a S3 (compatible) server | Testing | [](https://axoflow.com/docs/axosyslog-core/chapter-destinations/destination-s3/) |
| **[SplunkHEC](syslogng-outputs/splunk_hec/)** | syslogng-outputs | Sending messages over Splunk HEC | Testing | []() |
Expand Down
16 changes: 11 additions & 5 deletions docs/configuration/plugins/syslogng-outputs/openobserve.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: Openobserve
title: OpenObserve
weight: 200
generated_file: true
---

# Sending messages over Openobserve
# Sending messages over OpenObserve
## Overview

## Example
Expand All @@ -17,7 +17,7 @@ metadata:
spec:
openobserve:
url: "https://some-openobserve-endpoint"
port: 5040
port: 5080
organization: "default"
stream: "default"
user: "username"
Expand All @@ -38,17 +38,23 @@ More information at https://axoflow.com/docs/axosyslog-core/chapter-destinations

### organization (string, optional) {#openobserveoutput-organization}

Name of the organization in Openobserve.
Name of the organization in OpenObserve.


### port (int, optional) {#openobserveoutput-port}

The port number of the OpenObserve server. Specify it here instead of appending it to the URL.

Default: 5080

### record (string, optional) {#openobserveoutput-record}

Arguments to the `$format-json()` template function. Default: --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}"


### stream (string, optional) {#openobserveoutput-stream}

Name of the stream in Openobserve.
Name of the stream in OpenObserve.



4 changes: 4 additions & 0 deletions go.work.sum
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ github.com/moby/sys/sequential v0.5.0/go.mod h1:tH2cOOs5V9MlPiXcQzRC+eEyab644PWK
github.com/moby/sys/signal v0.7.0/go.mod h1:GQ6ObYZfqacOwTtlXvcmh9A26dVRul/hbOZn88Kg8Tg=
github.com/moby/sys/symlink v0.2.0/go.mod h1:7uZVF2dqJjG/NsClqul95CqKOBRQyYSNnJ6BMgR/gFs=
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus=
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
github.com/nelsam/hel/v2 v2.3.3/go.mod h1:1ZTGfU2PFTOd5mx22i5O0Lc2GY933lQ2wb/ggy+rL3w=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
Expand All @@ -108,6 +109,7 @@ github.com/posener/complete v1.2.3/go.mod h1:WZIdtGGp+qx0sLrYKtIRAruyNpv6hFCicSg
github.com/pquerna/cachecontrol v0.1.0/go.mod h1:NrUG3Z7Rdu85UNR3vm7SOsl1nFIeSiQnrHV5K9mBcUI=
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/russross/blackfriday v1.6.0 h1:KqfZb0pUVN2lYqZUYRddxF4OR8ZMURnJIG5Y3VRLtww=
github.com/russross/blackfriday v1.6.0/go.mod h1:ti0ldHuxg49ri4ksnFxlkCfN+hvslNlmVHqNRXXJNAY=
github.com/soheilhy/cmux v0.1.5/go.mod h1:T7TcVDs9LWfQgPlPsdngu6I6QIoyIFZDDC6sNE1GqG0=
github.com/stefanberger/go-pkcs11uri v0.0.0-20201008174630-78d3cae3a980/go.mod h1:AO3tvPzVZ/ayst6UlUKUv6rcPQInYe3IknH3jYhAKu8=
Expand Down Expand Up @@ -149,6 +151,7 @@ golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/sync v0.5.0 h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE=
golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
Expand All @@ -162,6 +165,7 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
google.golang.org/genproto v0.0.0-20230526161137-0005af68ea54 h1:9NWlQfY2ePejTmfwUH1OWwmznFa+0kKcHGPDvcPza9M=
google.golang.org/genproto v0.0.0-20230526161137-0005af68ea54/go.mod h1:zqTuNwFlFRsw5zIts5VnzLQxSRqh+CGOTVMlYbY0Eyk=
google.golang.org/genproto/googleapis/api v0.0.0-20230525234035-dd9d682886f9/go.mod h1:vHYtlOoi6TsQ3Uk2yxR7NI5z8uoV+3pZtR4jmHIkRig=
google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ source "main_input" {
};
destination "output_default_test-openobserve-out" {
openobserve-log(url("http://localhost:5080/") user("[email protected]") password("V2tsn88GhdNTKxaS") persist_name("output_default_test-openobserve-out") organization("default") stream("default"));
openobserve-log(url("http://localhost") user("[email protected]") password("V2tsn88GhdNTKxaS") persist_name("output_default_test-openobserve-out") port(5080) organization("default") stream("default"));
};
`)

Expand Down Expand Up @@ -81,7 +81,87 @@ destination "output_default_test-openobserve-out" {
Spec: v1beta1.SyslogNGOutputSpec{
Openobserve: &output.OpenobserveOutput{
HTTPOutput: output.HTTPOutput{
URL: "http://localhost:5080/",
URL: "http://localhost",
User: "[email protected]",
Password: secret.Secret{
ValueFrom: &secret.ValueFrom{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{
Name: "openobserve",
},
Key: "password",
},
}},
},
},
},
},
},
}

var buf strings.Builder
err := config.RenderConfigInto(testCaseInput, &buf)
config.CheckError(t, false, err)
require.Equal(t, expectedConfig, buf.String())

}

func TestOpenobserveOutputWithOtherPort(t *testing.T) {
expectedConfig := config.Untab(`@version: current
@include "scl.conf"
source "main_input" {
channel {
source {
network(flags("no-parse") port(601) transport("tcp"));
};
parser {
json-parser(prefix("json."));
};
};
};
destination "output_default_test-openobserve-out" {
openobserve-log(url("http://localhost") user("[email protected]") password("V2tsn88GhdNTKxaS") persist_name("output_default_test-openobserve-out") port(5081) organization("default") stream("default"));
};
`)

testCaseInput := config.Input{
Namespace: "config-test",
Name: "test",
SyslogNGSpec: &v1beta1.SyslogNGSpec{},
ClusterOutputs: []v1beta1.SyslogNGClusterOutput{},
ClusterFlows: []v1beta1.SyslogNGClusterFlow{},
Flows: []v1beta1.SyslogNGFlow{},
SourcePort: 601,
SecretLoaderFactory: &config.TestSecretLoaderFactory{
Reader: config.SecretReader{
Secrets: []corev1.Secret{
{
ObjectMeta: metav1.ObjectMeta{
Namespace: "default",
Name: "openobserve",
},
Data: map[string][]byte{
"password": []byte("V2tsn88GhdNTKxaS"),
},
},
},
},
MountPath: "/etc/syslog-ng/secret",
},
Outputs: []v1beta1.SyslogNGOutput{
{
ObjectMeta: metav1.ObjectMeta{
Namespace: "default",
Name: "test-openobserve-out",
},
Spec: v1beta1.SyslogNGOutputSpec{
Openobserve: &output.OpenobserveOutput{
Port: 5081,
HTTPOutput: output.HTTPOutput{
URL: "http://localhost",
User: "[email protected]",
Password: secret.Secret{
ValueFrom: &secret.ValueFrom{
Expand Down
21 changes: 14 additions & 7 deletions pkg/sdk/logging/model/syslogng/output/openobserve.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@

package output

// +name:"Openobserve"
// +name:"OpenObserve"
// +weight:"200"
type _hugoOpenobserve interface{} //nolint:deadcode,unused

// +docName:"Sending messages over Openobserve"
// +docName:"Sending messages over OpenObserve"
/*
## Example
Expand All @@ -30,7 +30,7 @@ metadata:
spec:
openobserve:
url: "https://some-openobserve-endpoint"
port: 5040
port: 5080
organization: "default"
stream: "default"
user: "username"
Expand All @@ -44,25 +44,32 @@ More information at https://axoflow.com/docs/axosyslog-core/chapter-destinations
*/
type _docOpenobserve interface{} //nolint:deadcode,unused

// +name:"Openobserve"
// +name:"OpenObserve"
// +url:"https://axoflow.com/docs/axosyslog-core/chapter-destinations/openobserve/"
// +description:"Sending messages over Openobserve"
// +description:"Sending messages over OpenObserve"
// +status:"Testing"
type _metaOpenobserve interface{} //nolint:deadcode,unused

// +kubebuilder:object:generate=true
type OpenobserveOutput struct {
HTTPOutput `json:",inline"`
// Name of the organization in Openobserve.
// The port number of the OpenObserve server. (default: 5080)
// Specify it here instead of appending it to the URL.
Port int `json:"port,omitempty"`
// Name of the organization in OpenObserve.
Organization string `json:"organization,omitempty"`
// Name of the stream in Openobserve.
// Name of the stream in OpenObserve.
Stream string `json:"stream,omitempty"`
// Arguments to the `$format-json()` template function.
// Default: --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}"
Record string `json:"record,omitempty"`
}

func (o *OpenobserveOutput) BeforeRender() {
if o.Port == 0 {
o.Port = 5080
}

if o.Organization == "" {
o.Organization = "default"
}
Expand Down

0 comments on commit 68f0e88

Please sign in to comment.