Tracing flattenTags memory leak with api stream

[intech]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I'm reporting the issue to the correct repository

Current Behavior

Use moleculer-web with stream and tracing exporters.

Expected Behavior

Check types obj and ignore binary stream.

Failure Information

Memory leak in source: https://github.com/moleculerjs/moleculer/blob/master/src/tracing/exporters/base.js#L88

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

1. moleculer-web route with stream alias
2. enable tracing exporters
3. send binary data on route

Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

• Moleculer version: v0.14.7 (JaegerTraceExporter)
• Moleculer-web version: v0.9.1
• NodeJS version: v12.18.1
• Operating System: docker alpine linux

Failure Logs

[2020-06-23T16:33:19.244Z] INFO  6c019c92a454-17/API: => PUT /file/64efa182-7d12-4152-a812-42c40232619
[2020-06-23T16:33:19.246Z] INFO  6c019c92a454-17/API:    Call 'v1.upload.file' action

<--- Last few GCs --->

[17:0x5568baa7dee0]    34788 ms: Mark-sweep 1982.9 (2054.8) -> 1982.9 (2051.8) MB, 28.7 / 0.0 ms  (average mu = 0.646, current mu = 0.279) last resort GC in old space requested
[17:0x5568baa7dee0]    34830 ms: Mark-sweep 1982.9 (2051.8) -> 1982.9 (2051.8) MB, 42.1 / 0.0 ms  (average mu = 0.434, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x5568b888fd59]
Security context: 0x0664e27c08d1 <JSObject>
    1: flattenTags [0x21165e78b399] [/app/node_modules/moleculer/src/tracing/exporters/base.js:~87] [pc=0x3728c00e6203](this=0x1ca5cad8f801 <BaseTraceExporter map = 0x34590a45489>,0x21168fe05fc1 <Socket map = 0x21a745820aa9>,0x33a1c54006e9 <false>,0x3fd13943f091 <Very long string[24653]>)
    2: flattenTags [0x21165e78b399] [/app/node_modules/moleculer/src/t...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[intech]

Config:

	tracing: {
		enabled: true,
		events: true,
		stackTrace: true,
		exporter: [
			//"Console",
			{
				type: "Jaeger",
				options: {
					endpoint: process.env.JAEGER,
					sampler: {
						type: "Const",
						options: {}
					},
					tracerOptions: {},
					defaultTags: registry => ({
						namespace: registry.broker.namespace,
						nodeID: registry.broker.nodeID,
						version: registry.broker.MOLECULER_VERSION
					})
				}
			}
		]
	},

[intech]

Temporary fix for action with stream:

tracing: {
	tags(ctx, response) {
		return {
			params: ctx.params.$params,
			meta: ctx.meta,
			response
		};
	}
},

[intech]

related: https://github.com/moleculerjs/moleculer-web/issues/188

[abhisheksinha1601]

Same issue happening with me when I added Datadog tracing causing frequent pod restarts. I am not using API stream though.

• Moleculer version: v0.14.6
• DdTrace version: v0.21.1
• NodeJS version: v10.15.3
• Operating System: docker alpine linux

[abhisheksinha1601]

Hey @icebob any luck on what actually might be causing the issue? I'm out of options here and relying on just the logs for debugging. I wasn't having this issue with Newrelic but when I migrated to Datadog, I've started facing this issue.

[icebob]

@abhisheksinha1601 are you also using streams?

[abhisheksinha1601]

@icebob not stream, just the datadog tracing exporter (ddtrace-js-v0.21.1). Can this be because of the flattenTags function? I'm using the same version of dd-trace on my monolithic api server and it seems to run fine.

[icebob]

I don't know, but the issue opener has a problem with streams. The flattenTags make a deep copy of the params. So if you have big or circular object in the ctx.params it can cause a problem. In this case, you should filter out these params from tags.

[VitaliiHrozynskyi]

I've encountered the same issue using Jaeger and moleculer-io simultaneously. moleculer-io passing socket instance to action, and flattenTags runs into endless recursion as socket instance is a circular object.

In this case, you should filter out these params from tags.

So i've just extended Jaeger tracer, and replaced socket object with socket.id. I had to prospect sources for a bit, so i believe this could be helpful for anyone who have similar issues.

moleculer.config.js

import { TracerExporters } from 'moleculer';
class FilteredJaegerTracer extends TracerExporters.Jaeger {
  spanFinished(span) {
    // span.tags.params contain params that is passed to action/event
    if (span.tags.params.socket) {
      this.generateJaegerSpan({
        ...span,
        tags: {
          ...span.tags,
          params: {
            ...span.tags.params,
            socket: span.tags.params.socket.id,
          },
        },
      });
    } else {
      this.generateJaegerSpan(span);
    }
  }
}

const brokerConfig = {
...
  tracing: {
    enabled: true,
    events: false,
    sampling:{ rate: 0},
    exporter: [
      new FilteredJaegerTracer(
          // jaeger options here
      ),
    ],
  },
...
}

Hope this can save few hours for someone :)