Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[modules] Add metrics-scraper module #11

Merged
merged 5 commits into from
Dec 7, 2023
Merged

[modules] Add metrics-scraper module #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
34e358f
[modules] Add metrics-scraper module
fpapon Oct 19, 2023
a7b2a1c
[modules] Add metrics-scraper documentation
fpapon Dec 6, 2023
f745b13
[dependencies] Upgrade to Fusion 1.0.12
fpapon Dec 7, 2023
18845d5
[modules] Use completion stage in metrics relay
fpapon Dec 7, 2023
d130deb
[modules] Close bufferReader
fpapon Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions documentation/src/main/minisite/content/metrics-relay.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
= Getting Started
:minisite-index: 300
:minisite-index-title: Metrics Relay
:minisite-index-description: Collect batch execution metrics for observability polling.
:minisite-index-icon: heartbeat

[abstract]
Yupiik Batch provides metrics relay to help components like batch or script to store metrics that will be polled periodically
like Prometheus.
For now the metrics are stored in memory, so it's better to use the dropOnPull to true to avoid too much
memory usage.

== Stack

The metrics relay is based on link:https://www.yupiik.io/fusion[Yupiik Fusion].

== Usage

The metrics relay provide an endpoint with 2 methods:

* POST: `/relay?id=test` to store metrics in OpenMetric format.

[source,bash]
----
curl -X POST http://localhost:8080/relay?id=test -H 'Content-Type: text/plain' -d '
# TYPE foo1 gauge
# HELP foo1 doc
foo1 1234
# TYPE foo2 gauge
# HELP foo2 doc
foo2 1235'
----

The response is a http code 201 with body `OK`.

You can set the query param `dropOnPull` to `true` (default is `false`) to remove the metric after a polling (GET):

[source,bash]
----
curl -X POST http://localhost:8080/relay?id=test&dropOnPull=true -H 'Content-Type: text/plain' -d '
# TYPE foo1 gauge
# HELP foo1 doc
foo1 1234
# TYPE foo2 gauge
# HELP foo2 doc
foo2 1235'
----

* GET: `/relay?id=test` to fetch the latest metrics.

[source,bash]
----
curl http://localhost:8080/relay?id=test
----

You can set the query param `ignoreDrop` to `true` (default is `false`) to disable the drop policy define on the entry:

[source,bash]
----
curl http://localhost:8080/relay?id=test&ignoreDrop=true
----
12 changes: 11 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,15 @@
<owb.version>2.0.27</owb.version>
<uship.version>1.0.17</uship.version>
<yupiik-logging.version>1.0.7</yupiik-logging.version>
<fusion.version>1.0.12</fusion.version>
<tomcat.version>10.1.16</tomcat.version>
</properties>

<modules>
<module>dependencies</module>
<module>yupiik-batch-runtime</module>
<module>yupiik-batch-ui</module>
<module>yupiik-batch-metrics-relay</module>
<module>documentation</module>
<module>iterators</module>
<module>simple-configuration</module>
Expand All @@ -51,6 +54,11 @@
<version>2.2.220</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-catalina</artifactId>
<version>${tomcat.version}</version>
</dependency>
</dependencies>
</dependencyManagement>

Expand Down Expand Up @@ -123,7 +131,7 @@
<plugin><!-- mvn ossindex:audit -->
<groupId>org.sonatype.ossindex.maven</groupId>
<artifactId>ossindex-maven-plugin</artifactId>
<version>3.1.0</version>
<version>3.2.0</version>
<executions>
<execution>
<id>audit-dependencies</id>
Expand All @@ -135,6 +143,8 @@
</executions>
<configuration>
<scope>compile,runtime</scope>
<!-- // exclude false positive for Tomcat 10.1.x as it's raised because of Tomcat 9.x -->
<excludeVulnerabilityIds>CVE-2023-42794</excludeVulnerabilityIds>
</configuration>
</plugin>
<!--
Expand Down
211 changes: 211 additions & 0 deletions yupiik-batch-metrics-relay/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--

Copyright (c) 2021-2023 - Yupiik SAS - https://www.yupiik.com
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>yupiik-batch</artifactId>
<groupId>io.yupiik.batch</groupId>
<version>1.0.4-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>yupiik-batch-metrics-relay</artifactId>
<name>Yupiik Batch :: Metrics Scraper</name>

<properties>
<image.java.base>ossyupiik/java:17.0.9@sha256:88d42b5b803e74fe2149efb942c70d4eb6ecdadb8902c87ac030a01419b2268e</image.java.base>
<image.java.workdir>/opt/yupiik</image.java.workdir>
<image.native.base>scratch</image.native.base>
<image.registry>yupiik</image.registry>
<main.class>io.yupiik.fusion.framework.api.main.Launcher</main.class>
</properties>

<dependencies>
<dependency>
<groupId>io.yupiik.fusion</groupId>
<artifactId>fusion-build-api</artifactId>
<version>${fusion.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.yupiik.fusion</groupId>
<artifactId>fusion-processor</artifactId>
<version>${fusion.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.yupiik.fusion</groupId>
<artifactId>fusion-api</artifactId>
<version>${fusion.version}</version>
</dependency>
<dependency>
<groupId>io.yupiik.fusion</groupId>
<artifactId>fusion-http-server</artifactId>
<version>${fusion.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-catalina</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-catalina</artifactId>
</dependency>
<dependency>
<groupId>io.yupiik.fusion</groupId>
<artifactId>fusion-json</artifactId>
<version>${fusion.version}</version>
</dependency>
<dependency>
fpapon marked this conversation as resolved.
Show resolved Hide resolved
<groupId>io.yupiik.fusion</groupId>
<artifactId>fusion-observability</artifactId>
<version>${fusion.version}</version>
</dependency>
<dependency>
<groupId>io.yupiik.logging</groupId>
<artifactId>yupiik-logging-jul</artifactId>
<version>${yupiik-logging.version}</version>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>io.yupiik.fusion</groupId>
<artifactId>fusion-testing</artifactId>
<version>${fusion.version}</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<java.util.logging.manager>io.yupiik.logging.jul.YupiikLogManager</java.util.logging.manager>
<java.net.preferIPv4Stack>true</java.net.preferIPv4Stack>
<fusion.http-server.port>0</fusion.http-server.port>
<fusion.observability.server.port>0</fusion.observability.server.port>
</systemPropertyVariables>
</configuration>
</plugin>

<plugin><!-- mvn exec:java -->
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<mainClass>${main.class}</mainClass>
</configuration>
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>native</id>
<build>
<plugins>
<plugin>
<!--
mvn arthur:native-image -Pnative for binaries
mvn arthur:native-image arthur:docker -Pnative for build docker image with binaries
-->
<groupId>org.apache.geronimo.arthur</groupId>
<artifactId>arthur-maven-plugin</artifactId>
<version>1.0.7</version>
<configuration>
<graalVersion>17.0.8-graalce</graalVersion>
<main>${main.class}</main>
<to>${image.registry}/yupiik-batch/metrics-scraper:${project.version}</to>
<creationTimestamp>-1</creationTimestamp>
<labels>
<org.opencontainers.image.created>${maven.build.timestamp}</org.opencontainers.image.created>
<org.opencontainers.image.authors>Yupiik</org.opencontainers.image.authors>
<org.opencontainers.image.vendor>Yupiik</org.opencontainers.image.vendor>
<org.opencontainers.image.title>${project.artifactId}</org.opencontainers.image.title>
<org.opencontainers.image.description>${project.description}</org.opencontainers.image.description>
<org.opencontainers.image.version>${project.version}</org.opencontainers.image.version>
</labels>
<customOptions>
<customOption>-Djava.util.logging.manager=io.yupiik.logging.jul.YupiikLogManager</customOption>
<customOption>-Djava.net.preferIPv4Stack=true</customOption>
<customOption>-Duser.language=en</customOption>
<customOption>-Duser.country=US</customOption>
<customOption>-Dfile.encoding=UTF-8</customOption>
<customOption>-H:+UnlockExperimentalVMOptions</customOption>
<customOption>--static</customOption>
</customOptions>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>docker-jvm</id>
<build>
<plugins>
<!-- mvn jib:docker -Pdocker-jvm -->
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<configuration>
<containerizingMode>packaged</containerizingMode>
<from>
<image>${image.java.base}</image>
</from>
<to>
<image>${image.registry}/yupiik-batch/metrics-scraper:${project.version}</image>
</to>
<container>
<appRoot>${image.java.workdir}</appRoot>
<workingDirectory>${image.java.workdir}</workingDirectory>
<extraClasspath>${image.java.workdir}/yupiik-batch-metrics-relay-${project.version}/lib/*</extraClasspath>
<creationTime>${maven.build.timestamp}</creationTime>
<filesModificationTime>${maven.build.timestamp}</filesModificationTime>
<mainClass>${main.class}</mainClass>
<jvmFlags>
<jvmFlag>-Djava.util.logging.manager=io.yupiik.logging.jul.YupiikLogManager</jvmFlag>
<jvmFlag>-Dio.yupiik.logging.jul.handler.StandardHandler.formatter=json</jvmFlag>
<jvmFlag>-Djava.security.egd=file:/dev/./urandom</jvmFlag>
<jvmFlag>-Djdk.serialFilter=!*</jvmFlag>
<jvmFlag>-Dyupiik.build.timestamp=${maven.build.timestamp}</jvmFlag>
</jvmFlags>
<labels>
<org.opencontainers.image.created>${maven.build.timestamp}</org.opencontainers.image.created>
<org.opencontainers.image.authors>Yupiik</org.opencontainers.image.authors>
<org.opencontainers.image.vendor>Yupiik</org.opencontainers.image.vendor>
<org.opencontainers.image.title>${project.artifactId}</org.opencontainers.image.title>
<org.opencontainers.image.description>${project.description}</org.opencontainers.image.description>
<org.opencontainers.image.version>${project.version}</org.opencontainers.image.version>
</labels>
</container>
<outputPaths>
<imageJson>${project.build.directory}/jib-image.json</imageJson>
</outputPaths>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
Loading
Loading