[Doc] add memory suggestion #454

kaisecheng · 2022-08-11T16:09:25Z

Many users come across java.lang.OutOfMemoryError: Direct buffer memory
This commit adds a memory session to set the expectation of how much memory the plugin could use.

karenzone · 2022-08-11T17:20:54Z

docs/index.asciidoc

+This plugin uses direct memory which is not included in the heap memory.
+By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.
+Make sure heap and direct memory does not exceed the total size of the server. You can set the amount of direct memory with -XX:MaxDirectMemorySize.


Suggested change

This plugin uses direct memory which is not included in the heap memory.

By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.

Make sure heap and direct memory does not exceed the total size of the server. You can set the amount of direct memory with -XX:MaxDirectMemorySize.

This plugin uses direct memory (bypassing the CPU) to improve performance.

Direct memory and heap size (Xmx) allocations are separate, but are the same by default.

For example, setting 10GB Xmx sets direct memory to 10GB.

You can change the amount of direct memory with -XX:MaxDirectMemorySize.

NOTE: Be sure that heap and direct memory total allocation does not exceed the total size of the server.

karenzone

Left some suggestions inline for consideration.

yaauie

I would like for this doc (and/or for the suggested doc addition in LS-core) to cover the consequences of having direct memory limit too low or too high. From my understanding, setting it too high will cause a runtime failure to allocate the direct memory, but if we set it too low do we merely not buffer as much from the inbound connections?

yaauie · 2022-08-11T17:15:07Z

docs/index.asciidoc

+===== Memory usage
+
+This plugin uses direct memory which is not included in the heap memory.
+By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.


Suggested change

By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.

By default, a JVM's off-heap direct memory limit is the same as the heap size.

For example, setting a heap size of 10GB without setting the direct memory limit will result in the direct limit _also_ being 10GB for a total of 20GB.

yaauie · 2022-08-11T17:16:07Z

docs/index.asciidoc

+
+This plugin uses direct memory which is not included in the heap memory.
+By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.
+Make sure heap and direct memory does not exceed the total size of the server. You can set the amount of direct memory with -XX:MaxDirectMemorySize.


Suggested change

Make sure heap and direct memory does not exceed the total size of the server. You can set the amount of direct memory with -XX:MaxDirectMemorySize.

Make sure heap and direct memory combined does not exceed the total memory available on the server.

You can set the amount of direct memory with `-XX:MaxDirectMemorySize` in {logstash-ref}/jvm-settings.html[Logstash's JVM Settings].

We should also add an off-heap section to Logstash's JVM Settings.

yaauie · 2022-08-11T17:27:07Z

docs/index.asciidoc

+[id="plugins-{type}s-{plugin}-memory"]
+===== Memory usage
+
+This plugin uses direct memory which is not included in the heap memory.


since direct memory has historically also been called "off-heap", including that phrasing will help people find and/or understand this doc.

Suggested change

This plugin uses direct memory which is not included in the heap memory.

This plugin uses "off-heap" direct memory _in addition to_ heap memory.

robbavey

Couple of suggestions, but final approval should go to @karenzone or @kilfoyle

robbavey · 2022-08-11T17:31:22Z

docs/index.asciidoc

+
+This plugin uses direct memory which is not included in the heap memory.
+By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.
+Make sure heap and direct memory does not exceed the total size of the server. You can set the amount of direct memory with -XX:MaxDirectMemorySize.


Maybe add a link to the jvm settings doc in logstash core?

We should also update that document too, along the lines of this issue

Adding a link sounds good. Users mostly come across direct memory with beats-input. I will update the JVM settings page. Thanks for the pointer

robbavey · 2022-08-11T17:42:22Z

docs/index.asciidoc

+This plugin uses direct memory which is not included in the heap memory.
+By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.


Suggested change

This plugin uses direct memory which is not included in the heap memory.

By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.

This plugin uses direct memory which is allocated in addition to that allocated to the heap.

By default, this is set to the same value as allocated to the heap via the `Xmx` flags. For example, setting `-Xmx10G` will allocate `10GB` for heap and an additional `10GB` for direct memory, for a total of `20GB` allocated.

kaisecheng · 2022-08-11T21:09:53Z

I would like for this doc (and/or for the suggested doc addition in LS-core) to cover the consequences of having direct memory limit too low or too high. From my understanding, setting it too high will cause a runtime failure to allocate the direct memory, but if we set it too low do we merely not buffer as much from the inbound connections?

@yaauie Logstash can start with -XX:MaxDirectMemorySize=100G in my mac, probably will throw OutOfMemoryError later when it takes more than the total size of the machine. Also, tried to ingest 10G data from filebeat to logstash with -XX:MaxDirectMemorySize=64M without exception. I will add a note that set too low direct memory will lower the performance.

Co-authored-by: Karen Metts <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Ry Biesemeyer <[email protected]>

kaisecheng · 2022-08-12T12:32:39Z

@karenzone Thanks for the suggestion. I have combined everyone's ideas into one. Also, I update the JVM session in logstash core elastic/logstash#14431 Please review both PRs.

docs/index.asciidoc

karenzone

Left some minor suggestions inline. Otherwise, LGTM

karenzone · 2022-08-12T23:42:52Z

docs/index.asciidoc

+For example, setting `-Xmx10G` without setting the direct memory limit will allocate `10GB` for heap and an additional `10GB` for direct memory, for a total of `20GB` allocated. 
+You can set the amount of direct memory with `-XX:MaxDirectMemorySize` in {logstash-ref}/jvm-settings.html[Logstash's JVM Settings]. Consider setting it to half of the heap size. Setting too low direct memory decreases the performance of ingestion.
+
+NOTE: Be sure that heap and direct memory combined does not exceed the total memory available on the server to avoid OutOfDirectMemoryError


What about "... to avoid an OutOfDirectMemoryError." (singular) or "... to avoid errors." (plural)

Co-authored-by: Karen Metts <[email protected]>

kaisecheng added 3 commits August 11, 2022 17:03

[Doc] add memory suggestion

8569ded

bump version

394509a

move up

11871b1

roaksoax added the status:needs-review label Aug 11, 2022

karenzone reviewed Aug 11, 2022

View reviewed changes

yaauie reviewed Aug 11, 2022

View reviewed changes

robbavey reviewed Aug 11, 2022

View reviewed changes

roaksoax assigned kaisecheng Aug 11, 2022

kaisecheng added 3 commits August 11, 2022 22:15

rephase

f660ed5

Co-authored-by: Karen Metts <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Ry Biesemeyer <[email protected]>

add consequences of high/low mem limit

2e75e8a

recommendation

3578ce4

kaisecheng mentioned this pull request Aug 12, 2022

[Doc] add off-heap memory session elastic/logstash#14431

Merged

kaisecheng requested a review from karenzone August 12, 2022 12:27

karenzone reviewed Aug 12, 2022

View reviewed changes

docs/index.asciidoc Outdated Show resolved Hide resolved

karenzone approved these changes Aug 12, 2022

View reviewed changes

roaksoax added status:approved and removed status:needs-review labels Aug 13, 2022

kaisecheng and others added 2 commits August 13, 2022 11:30

Update docs/index.asciidoc

5e911c7

Co-authored-by: Karen Metts <[email protected]>

wording

0d4f0e5

kaisecheng merged commit 02ff81f into logstash-plugins:main Aug 15, 2022

roaksoax removed the status:approved label Aug 15, 2022

This was referenced Aug 24, 2022

Backport PR #14431 to 8.4: [Doc] add off-heap memory session elastic/logstash#14470

Closed

Backport PR #14431 to 8.4: [Doc] add off-heap memory session elastic/logstash#14475

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Doc] add memory suggestion #454

[Doc] add memory suggestion #454

kaisecheng commented Aug 11, 2022

karenzone Aug 11, 2022

karenzone left a comment

yaauie left a comment

yaauie Aug 11, 2022

yaauie Aug 11, 2022

yaauie Aug 11, 2022 •

edited

Loading

robbavey left a comment

robbavey Aug 11, 2022

kaisecheng Aug 11, 2022

robbavey Aug 11, 2022

kaisecheng commented Aug 11, 2022

kaisecheng commented Aug 12, 2022

karenzone left a comment

karenzone Aug 12, 2022

-This plugin uses direct memory which is not included in the heap memory.
-By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.
-Make sure heap and direct memory does not exceed the total size of the server. You can set the amount of direct memory with -XX:MaxDirectMemorySize.
+This plugin uses direct memory (bypassing the CPU) to improve performance.
+Direct memory and heap size (Xmx) allocations are separate, but are the same by default.
+For example, setting 10GB Xmx sets direct memory to 10GB.
+You can change the amount of direct memory with -XX:MaxDirectMemorySize.
+NOTE: Be sure that heap and direct memory total allocation does not exceed the total size of the server.

	By default, it is the same as the heap size (Xmx), for example, setting 10GB Xmx means setting 10GB of direct memory.
	By default, a JVM's off-heap direct memory limit is the same as the heap size.
	For example, setting a heap size of 10GB without setting the direct memory limit will result in the direct limit _also_ being 10GB for a total of 20GB.

	Make sure heap and direct memory does not exceed the total size of the server. You can set the amount of direct memory with -XX:MaxDirectMemorySize.
	Make sure heap and direct memory combined does not exceed the total memory available on the server.
	You can set the amount of direct memory with `-XX:MaxDirectMemorySize` in {logstash-ref}/jvm-settings.html[Logstash's JVM Settings].

	This plugin uses direct memory which is not included in the heap memory.
	This plugin uses "off-heap" direct memory _in addition to_ heap memory.

[Doc] add memory suggestion #454

[Doc] add memory suggestion #454

Conversation

kaisecheng commented Aug 11, 2022

karenzone Aug 11, 2022

Choose a reason for hiding this comment

karenzone left a comment

Choose a reason for hiding this comment

yaauie left a comment

Choose a reason for hiding this comment

yaauie Aug 11, 2022

Choose a reason for hiding this comment

yaauie Aug 11, 2022

Choose a reason for hiding this comment

yaauie Aug 11, 2022 • edited Loading

Choose a reason for hiding this comment

robbavey left a comment

Choose a reason for hiding this comment

robbavey Aug 11, 2022

Choose a reason for hiding this comment

kaisecheng Aug 11, 2022

Choose a reason for hiding this comment

robbavey Aug 11, 2022

Choose a reason for hiding this comment

kaisecheng commented Aug 11, 2022

kaisecheng commented Aug 12, 2022

karenzone left a comment

Choose a reason for hiding this comment

karenzone Aug 12, 2022

Choose a reason for hiding this comment

yaauie Aug 11, 2022 •

edited

Loading