Easiest was to add regularization (weight decay) to loss?

[hmf]

I am trying to add a L1 loss to a single bottleneck layer of an Autoencoder. There seems to be 2 ways one can encode the train and evaluation loop:

1. Explicitly (as shown in 4.5.3.3. Defining the Train and Test Functions)
2. Concisely (as shown in 4.3. Concise Implementation of Multilayer Perceptron) using EasyTrain

I am using Block, Model and Predictor as shown here in combination with the DefaultTrainingConfig and EasyTrain as shown here. I am assuming that I should use the SimpleCompositeLoss to combine a Loss.l2Loss() with this kind of penalty and use that in the DefaultTrainingConfig.

I looked for but did not find the weight penalties L1 or L2 for ready use. Looking at the loss functions, I see they do not include this weight decay penalty (regularization), nor do they seem to be parametrized for this. Am I correct in this or did I miss something?

Admitting that DJL does not provide pre-baked weight penalty (regularization) on a single layer or on all layers (may have missed this), would I be correct in saying that I should use the SimpleCompositeLoss to combine a Loss.l2Loss() and my own L1 loss using a specific set of parameters by extending the Loss class with something like:

  def l1Penalty(lambda: Float, w: NDArray): NDArray = {
    ((w.abs()).sum()).mul(lambda)
  }

or is there some other way one should go about it.

TIA

[zachgk]

That's exactly how I would go about it. I might even have the l1Penalty take an NDList so it can sum the penalties across multiple weights.

This seems like a really good improvement. Would you be interested in adding a PR to DJL with your weight penalty loss?

[hmf]

Would you be interested in adding a PR to DJL with your weight penalty loss?

Cannot promise anything.

In terms of implementation, that would be a class L1Penalty that extends Loss? Maybe a better name would be L1WeightDecay or L1Regularization? And in terms of constructor it would take in an NDList of NDArray where each array are the weights to penalize.

Haven't looked at the source but you also need a test for this, correct? Something like:

https://github.com/awslabs/djl/blob/d879306909ab8a3e2a52ad52a4ff6508d8143ed6/integration/src/main/java/ai/djl/integration/tests/training/LossTest.java

is Ok?

EDIT: I have checkout the repo and am trying to execute a test. The following command works fine:

./gradlew integration:test --tests *LossTest*

But these fail:

./gradlew integration:test --tests "ai.djl.integration.tests.training.LossTest*"
./gradlew integration:test --tests "ai.djl.integration.tests.training.LossTest.*"

I get the error:

:~/VSCodeProjects/djl$ ./gradlew integration:test --tests "ai.djl.integration.tests.training.LossTest.*"
> Task :integration:test FAILED
> Task :integration:jacocoTestReport FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':integration:test'.
> No tests found for given includes: [ai.djl.integration.tests.training.LossTest.*](--tests filter)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':integration:jacocoTestReport'.
> Unable to read execution data file /home/hmf/VSCodeProjects/djl/integration/build/jacoco/test.exec

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1s
41 actionable tasks: 3 executed, 38 up-to-date

What should I be doing to run a single test unit/case? Sorry but I don't usually work in Java and Gradle. 8-(

[zachgk]

That all looks right, and feel free to choose your favorite of those names.

Our integration tests are a bit different because we actually have a single test in IntegrationTests.java which runs the integration test suite. The suite is designed to be applied for all engines and handle engines that don't fully implement everything. The easiest trick to run it is to get an IDE (I use IntelliJ) which works well for this. You can also follow the instructions at http://docs.djl.ai/master/docs/development/development_guideline.html#test

[hmf]

PR at: #788