[WIP] ULMFiT for Text Classification #168

ComputerMaestro · 2019-08-17T11:39:16Z

Universal Language Model Fine-tuning for Text Classification. Here the model will be used for sentiment analysis as of now.

src/ULMFiT/custom_layers.jl

ComputerMaestro · 2019-08-21T08:13:21Z

@aviks did you upload the weights for which I gave the link?? if not I have different weights for Language model

aviks · 2019-08-21T08:58:43Z

I added the weights that were in the source in your PR

…

On Wed, Aug 21, 2019 at 9:40 AM Yash Patel ***@***.***> wrote: @aviks <https://github.com/aviks> did you upload the weights for which I gave the link?? if not I have different weights for Language model — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#168?email_source=notifications&email_token=AAC4QJQU2Q76PC5724LHEH3QFT2KFA5CNFSM4IMPEZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Y2VRY#issuecomment-523348679>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC4QJRIYW5KE4BDTXUOKZDQFT2KFANCNFSM4IMPEZZA> .

ComputerMaestro · 2019-08-21T12:46:13Z

Can you please give me that link??

aviks · 2019-08-23T00:23:33Z

No actually, sorry, I was confused. I did not upload the weights for ULMFiT. Let me know when they are ready, and I will upload.

ComputerMaestro · 2019-08-25T17:26:13Z

https://drive.google.com/open?id=1Ki8XH_hkJc8qlqUBqMN8KyFYHcHcFo_N
These are pretrained weights for ULMFiT Language model.

ComputerMaestro · 2019-09-05T11:12:02Z

https://drive.google.com/open?id=1lE3DiVs7RvesGVnu2LqqNEVmJ8QET3Tq
@aviks , These are the weights for bin sentiment classifier with about 91% accuracy. Please upload these weights.

ComputerMaestro · 2019-09-27T08:44:41Z

@aviks I will test the model for the AG news dataset as well, and will let you know the results soon.
Please let me know if there is something to change in the PR code.

DhairyaLGandhi · 2019-09-27T08:16:48Z

src/ULMFiT/custom_layers.jl

+    PooledDense
+"""
+
+import Flux: gate, _testmode!, _dropout_kernel


testmode! will be deprecated, on Flux#master and in the upcoming releases.

DhairyaLGandhi · 2019-09-27T08:16:54Z

src/ULMFiT/custom_layers.jl

+    y = similar(x, size(x))
+    Flux.rand!(y)
+    y .= Flux._dropout_kernel.(y, p, 1 - p)
+    return y


Why not use a normal dropout layer here? Internal APIs could change without notice.

That is to generate the mask for the variational dropout VarDrop . I will add a custom function instead of using _dropout_kernel their.

DhairyaLGandhi · 2019-09-27T08:16:59Z

src/ULMFiT/custom_layers.jl

+    @assert 0 ≤ p ≤ 1
+    cell = WeightDroppedLSTMCell(
+        param(init(out*4, in)),
+        param(init(out*4, out)),


param is not needed anymore with Zygote

DhairyaLGandhi · 2019-09-27T08:17:02Z

src/ULMFiT/custom_layers.jl

+
+Flux.@treelike WeightDroppedLSTMCell
+
+_testmode!(m::WeightDroppedLSTMCell, test) = (m.active = !test)


istraining should handle it. Maybe look at https://github.com/FluxML/Flux.jl/blob/acb6a8924551a146ad757c706e9a659a7efd92e2/src/layers/normalise.jl#L1

and
https://github.com/FluxML/Flux.jl/blob/acb6a8924551a146ad757c706e9a659a7efd92e2/src/layers/normalise.jl#L58

DhairyaLGandhi · 2019-09-27T08:17:06Z

src/ULMFiT/custom_layers.jl

+        if avg_fact != 1
+            layer.accum = layer.accum .+ Tracker.data.(p)
+            for (ps, accum) in zip(p, layer.accum)
+                Tracker.data(ps) .= avg_fact*accum


data is deprecated

DhairyaLGandhi · 2019-09-27T08:20:25Z

src/ULMFiT/fine_tune_lm.jl

+function discriminative_step!(layers, ηL::Float64, l, opts::Vector)
+    @assert length(opts) == length(layers)
+    # Gradient calculation
+    grads = Tracker.gradient(() -> l, get_trainable_params(layers))


Consider using Zygote, maybe?

I have less experience with zygote, I will look into it and will make changes

DhairyaLGandhi · 2019-09-27T08:22:22Z

src/ULMFiT/fine_tune_lm.jl

+    for (layer, opt) in zip(layers, opts)
+        opt.eta = ηl
+        for ps in get_trainable_params([layer])
+            Tracker.update!(opt, ps, grads[ps])


The update! from Flux.Optimise will be better suited, since Tracker is decoupled

aviks · 2019-11-25T23:59:24Z

@ComputerMaestro sorry for not looking at this for a while. Shall we get this finished up?

The pretrained weights are at https://github.com/JuliaText/TextAnalysis.jl/releases/download/v0.6.0/bin_sentiment_classifier_weights.bson.xz

Are all of Dhairya's feedback incorporated? (Except Zygote, we'll move to it later)

aviks · 2019-12-09T00:48:51Z

src/TextAnalysis.jl

@@ -112,9 +112,19 @@ module TextAnalysis
    include("sequence/pos.jl")
    include("sequence/sequence_models.jl")

+    # ULMFiT
+    include("ULMFiT/utils.jl")
+    include("ULMFiT/WikiText103_DataDeps.jl.jl")


This file does not exist

aviks · 2019-12-09T00:54:25Z

src/ULMFiT/train_text_classifier.jl

+    Flux.testmode!(classifier)
+    loss = 0
+    iters = take!(gen)
+    (num_of_batches != : & num_of_batches < iters) && (iters = num_of_batches)


seems like a typo here. this line does not compile!

aviks · 2019-12-09T01:26:24Z

test/ulmfit.jl

+        @test length(params(awd)) == 5
+    end
+
+    @test "VarDrop" begin


should be @testset

aviks · 2019-12-09T01:26:44Z

test/ulmfit.jl

@@ -0,0 +1,104 @@
+@testset "Custom layers" begin


this test file is not included in runtests.jl

ComputerMaestro · 2019-12-10T07:34:49Z

@aviks , there seems to be a problem with implementing this now if we are using Flux 0.10.0 (latest). Since it supports Zygote . And the model I have made is based on Tracker . So it might break if we only put some changes to make it fit for current flux version. Can I make Flux less than or equal to 0.9.0 as a requirement??? For now till there is a zygote implementation

aviks · 2019-12-12T02:56:19Z

Updated version in #179

Main ULMFiT files added

bf8c62a

aviks reviewed Aug 18, 2019

View reviewed changes

src/ULMFiT/custom_layers.jl Outdated Show resolved Hide resolved

ComputerMaestro added 2 commits August 19, 2019 18:03

Merge branch 'master' of https://github.com/JuliaText/TextAnalysis.jl

7421c41

Docs added

fcc5e16

Some basic tests added

247d2fe

ComputerMaestro force-pushed the master branch from cafe592 to 247d2fe Compare August 22, 2019 18:16

ComputerMaestro added 3 commits August 24, 2019 01:18

bug fixes

19bcd5f

Vocabulary for binary sentiment classifier

8c9fe95

bug fixes for sentiment classifier

d776ea2

ComputerMaestro added 3 commits August 29, 2019 21:30

generic data loader

4df6ba0

Merge branch 'master' of https://github.com/JuliaText/TextAnalysis.jl

4c0ba12

Merge branch 'master' of https://github.com/JuliaText/TextAnalysis.jl

8cedd58

ComputerMaestro force-pushed the master branch from cf093bf to 8cedd58 Compare August 29, 2019 16:03

ComputerMaestro added 3 commits August 31, 2019 03:33

Docs for custom layers

09c3e1d

bug fix for pooled dense layer

c79f53b

no need of asgd in fine-tuning

d01dd50

ComputerMaestro force-pushed the master branch from ed10529 to d01dd50 Compare September 5, 2019 10:38

ComputerMaestro added 4 commits September 5, 2019 16:56

link to original ULMFiT language models

6fecd4d

data loader

64f3eb5

bug fixes

9613d7c

bug fix

be9501d

DhairyaLGandhi reviewed Sep 30, 2019

View reviewed changes

Merge branch 'master' of https://github.com/JuliaText/TextAnalysis.jl

8fd8a5f

aviks reviewed Dec 9, 2019

View reviewed changes

ComputerMaestro added 3 commits December 9, 2019 17:23

Merge branch 'master' of https://github.com/JuliaText/TextAnalysis.jl

73aeffb

issues fixes

48b985f

removed testmode dependency

2c860c8

ComputerMaestro closed this Dec 10, 2019

ComputerMaestro reopened this Dec 10, 2019

aviks mentioned this pull request Dec 12, 2019

ULMFiT #179

Merged

ComputerMaestro closed this Jan 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] ULMFiT for Text Classification #168

[WIP] ULMFiT for Text Classification #168

ComputerMaestro commented Aug 17, 2019 •

edited

Loading

ComputerMaestro commented Aug 21, 2019

aviks commented Aug 21, 2019 via email

ComputerMaestro commented Aug 21, 2019

aviks commented Aug 23, 2019

ComputerMaestro commented Aug 25, 2019

ComputerMaestro commented Sep 5, 2019

ComputerMaestro commented Sep 27, 2019

DhairyaLGandhi Sep 27, 2019

DhairyaLGandhi Sep 27, 2019

ComputerMaestro Sep 30, 2019

DhairyaLGandhi Sep 27, 2019

DhairyaLGandhi Sep 27, 2019

DhairyaLGandhi Sep 27, 2019

DhairyaLGandhi Sep 27, 2019

ComputerMaestro Sep 30, 2019

DhairyaLGandhi Sep 27, 2019

aviks commented Nov 25, 2019

aviks Dec 9, 2019

aviks Dec 9, 2019

ComputerMaestro Dec 9, 2019

aviks Dec 9, 2019

aviks Dec 9, 2019

ComputerMaestro commented Dec 10, 2019 •

edited

Loading

aviks commented Dec 12, 2019


		Flux.@treelike WeightDroppedLSTMCell

		_testmode!(m::WeightDroppedLSTMCell, test) = (m.active = !test)

[WIP] ULMFiT for Text Classification #168

[WIP] ULMFiT for Text Classification #168

Conversation

ComputerMaestro commented Aug 17, 2019 • edited Loading

ComputerMaestro commented Aug 21, 2019

aviks commented Aug 21, 2019 via email

ComputerMaestro commented Aug 21, 2019

aviks commented Aug 23, 2019

ComputerMaestro commented Aug 25, 2019

ComputerMaestro commented Sep 5, 2019

ComputerMaestro commented Sep 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aviks commented Nov 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ComputerMaestro commented Dec 10, 2019 • edited Loading

aviks commented Dec 12, 2019

ComputerMaestro commented Aug 17, 2019 •

edited

Loading

ComputerMaestro commented Dec 10, 2019 •

edited

Loading