Log4j dynamic threshold filter with redirection to specific file.

[sahilkamboj334]

Hi All,

Recently i analysed dynamic threshold filter feature of log4j. It is working for the configured trace level. But my problem is that i am enabling level TRACE if request has header x-trace-enabled as true and logs are going to all appenders configured under xml file. i am using springboot framework with log4j2 and below is the configuration

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">

    <Properties>
        <Property name="LOG_PATTERN">[%-5level] %d{DEFAULT} [%t] %c{1.}%notEmpty{ - %X{requestId}} - %msg %n</Property>
        <Property name="log-path">/var/log/path</Property>
    </Properties>

    <DynamicThresholdFilter key="x-trace-enabled" onMatch="ACCEPT" onMismatch="NEUTRAL">
                            <KeyValuePair key="true" value="TRACE"/>
    </DynamicThresholdFilter>
            <Appenders>
                <Console name="console-log" target="SYSTEM_OUT">
                    <PatternLayout>
                        <Pattern>${LOG_PATTERN}</Pattern>
                    </PatternLayout>
                </Console>

                <RollingRandomAccessFile append="true" name="RollingFile" fileName="${log-path}/info.log"
                             filePattern="${log-path}/info-%d{yyyy-MM-dd-HH}.%i.log.zst">
                    <PatternLayout>
                        <Pattern>${LOG_PATTERN}</Pattern>
                    </PatternLayout>
                    <Policies>
                        <OnStartupTriggeringPolicy/>
                        <TimeBasedTriggeringPolicy interval="1"/>
                        <SizeBasedTriggeringPolicy size="1GB"/>
                    </Policies>

                    <DefaultRolloverStrategy fileIndex="nomax">
                        <Delete basePath="${log-path}">
                            <IfFileName glob="info-*.log.zst"/>
                            <IfAccumulatedFileCount exceeds="50"/>
                        </Delete>
                        <Delete basePath="${log-path}">
                            <IfFileName glob="access_log*.log"/>
                            <IfAccumulatedFileCount exceeds="2"/>
                        </Delete>
                        <Delete basePath="${log-path}">
                            <IfFileName glob="management_access_log*.log"/>
                            <IfAccumulatedFileCount exceeds="2"/>
                        </Delete>

                    </DefaultRolloverStrategy>
                </RollingRandomAccessFile>

                <RollingRandomAccessFile name="error-log"
                             fileName="${log-path}/error.log"
                             filePattern="${log-path}/error-%d{yyyy-MM-dd}.%i.log.zst">
                    <JsonLayout compact="true" eventEol="true">
                    </JsonLayout>

                    <Policies>
                        <TimeBasedTriggeringPolicy interval="1"/>
                        <SizeBasedTriggeringPolicy size="500MB"/>
                    </Policies>

                    <DefaultRolloverStrategy fileIndex="nomax">
                        <Delete basePath="${log-path}">
                            <IfFileName glob="error-*.log.zst"/>
                            <IfAccumulatedFileCount exceeds="2"/>
                        </Delete>
                    </DefaultRolloverStrategy>
                </RollingRandomAccessFile>
            </Appenders>

            <Loggers>
                <Root level="WARN" additivity="false">
                    <appender-ref ref="RollingFile"/>
                    <AppenderRef ref="error-log"/>
                </Root>

        <Logger name="com.example.package" level="DEBUG" additivity="false">
            <appender-ref ref="RollingFile"/>
            <AppenderRef ref="error-log" level="WARN" />
        </Logger>
    </Loggers>
</Configuration>

Answers i am looking for : -

1. These trace logs should go only under RollingFile appender.
2. Can i control the package for these trace logs? currently everything logs on trace which is not required. I want to track only classes that belongs to my package.

[ppkarwasz]

Answers i am looking for : -

1. These trace logs should go only under RollingFile appender.

2. Can i control the package for these trace logs? currently everything logs on trace which is not required. I want to track only classes that belongs to my package.

Log4j has a highly configurable filtering pipeline. I assume that you want to:

• log to the error-log appender all events at least as severe as WARN, coming from any logger of your system.
• log to the RollingFile appender only events from com.example.package or its descendants that match one of these conditions:
  1. either x-trace-enabled is set
  2. or the event is at least as severe as INFO.

In the case above you can use:

<!-- if `x-trace-enabled` is set, ignore the configured level -->
<DynamicThresholdFilter key="x-trace-enabled" onMatch="ACCEPT" onMismatch="NEUTRAL">
  <KeyValuePair key="true" value="TRACE"/>
</DynamicThresholdFilter>
<Loggers>
  <Root level="WARN">
    <!-- Apply the level check again, in case:
          * it was skipped, since "x-trace-enabled" is set.
          * or the event comes from the `com.example.package` logger,
            which has a lower (INFO) threshold. -->
    <AppenderRef ref="error-log" level="WARN"/>
  </Root>
  <Logger name="com.example.package" level="INFO">
    <AppenderRef ref="RollingFile/>
  </Logger>
</Loggers>

[sahilkamboj334]

@ppkarwasz thanks for reply. Could you please point to the correct configuration to use. i tried some filter chains but no luck. One issue was with log level trace which is superior to other configured levels so everything starts logging.

[ppkarwasz]

One issue was with log level trace which is superior to other configured levels so everything starts logging.

I guess, I should explain more the example above.

x-trace-enabled is true

If the context element x-trace-enabled is true, then the effective logger configuration is:

<Loggers>
  <Root level="TRACE">
    <!-- (1) -->
    <AppenderRef ref="error-log" level="WARN"/>
  </Root>
  <Logger name="com.example.package" level="TRACE" additivity="true"> <!-- (2) -->
    <AppenderRef ref="RollingFile/>
  </Logger>
</Loggers>

Since the <AppenderRef> element has level="WARN" (1), all the log events with level WARN or more severe will go to error-log. This is only true for log events from com.example.package, since additivity is true by default (2).

On the other hand the RollingFile appender will get all the log events produced by com.example.package.

x-trace-enabled is false

If the context element x-trace-enabled is false, then the effective logger configuration is:

<Loggers>
  <Root level="WARN">
    <AppenderRef ref="error-log" level="WARN"/>
  </Root>
  <Logger name="com.example.package" level="INFO" additivity="true"> <!-- (1) -->
    <AppenderRef ref="RollingFile/>
  </Logger>
</Loggers>

The error-log appender will get the same log events as before.

The RollingFile appender on the other hand will only print the events from the com.example.package logger (or its descendants) with a level at least as severe as INFO.

[sahilkamboj334]

 <Loggers>
        <Logger name="com.example.package" level="TRACE" additivity="true">
            <appender-ref ref="RollingFile"/>
            <ThreadContextMapFilter onMatch="ACCEPT" onMismatch="DENY">
                <KeyValuePair key="x-trace-enabled" value="true"/>
            </ThreadContextMapFilter>
        </Logger>
    </Loggers>

    <Loggers>
        <Root level="WARN" additivity="false">
            <appender-ref ref="RollingFile"/>
            <AppenderRef ref="error-log"/>
            <ThreadContextMapFilter onMatch="ACCEPT" onMismatch="NEUTRAL" operator="or">
                <KeyValuePair key="x-trace-enabled" value="false"/>
                <KeyValuePair key="x-trace-enabled" value="null"/>
            </ThreadContextMapFilter>
        </Root>

        <Logger name="com.example.package" level="DEBUG" additivity="true">
            <appender-ref ref="RollingFile"/>
            <AppenderRef ref="error-log" level="WARN" />
            <ThreadContextMapFilter onMatch="ACCEPT" onMismatch="NEUTRAL" operator="or">
                <KeyValuePair key="x-trace-enabled" value="false"/>
                <KeyValuePair key="x-trace-enabled" value="null"/>
            </ThreadContextMapFilter>
        </Logger>
    </Loggers>

like this ? This config is also not working

[sahilkamboj334]

for above config as well if you have x-trace-enabled as true then console appender will have all trace logs from root packages. And similar goes with

<Logger name="com.example.package" level="INFO"> <AppenderRef ref="FILE"/> <AppenderRef ref="CONSOLE" level="WARN"/> </Logger>
It will log both data in both the files. But my initial question is somewhat different. can i use some kind of composite filters to have this setting work accordingly if flag is true or not present accordigly.

[ppkarwasz]

for above config as well if you have x-trace-enabled as true then console appender will have all trace logs from root packages.

The level attribute on the AppenderRef element should prevent that. What version of Log4j Core are you using? I tested the above config against 2.23.1, but it should work with most versions.

Note: the console receives output from a log of logging backends. Maybe replace the Console appender with a file appender with a unique name to check if the trace output comes really from Log4j Core.

[sahilkamboj334]

@ppkarwasz i am using 2.23.1 itself but it is not working as expected. I have same config for root level logger where i have given level as warn but it is logging everything from root pacakges. As highlighted issue is with level trace.
Order - ALL < TRACE < DEBUG < INFO < WARN < ERROR < FATAL < OFF
so seems everything will go under configured appenders.

[ppkarwasz]

The level attribute on Root is ignored if the global filter returns ACCEPT. However the level attribute of AppenderRef is not ignored.

See Filtering Process for an in-depth description on how filters are applied and in which order.

[sahilkamboj334]

@ppkarwasz thanks for helping me out. The doc link shared by you is really helped me and now i got to know why it was not working when filter was returning ACCEPT.
thanks again !!!

[ppkarwasz]

I am happy to hear that our new heavily reworked documentation is useful.

The documentation will be published on our production logging.apache.org site soon, after the 2.24.0 release.