Add the ClarkWestTest for time series

[guilhermebodin]

This closes #202.

[guilhermebodin]

@PaulSoderlind we must complete the ClarkWestTest function and add the unit tests.

[PaulSoderlind]

OK, expect something early next week.

[PaulSoderlind]

I am now (23 May) trying Master to run the dm test. It seems to work, but when I do
show(dm) I get

Diebold-Mariano test                                          
--------------------                                          
Population details:                                           
    parameter of interest:   mean                             
    value under h_0:         0.0                              
    point estimate:          5.788390364544851                
ERROR: type DieboldMarianoTest has no field xbar  

The reason for asking is that I (of course) get a similar error message when I try to use the skeleton files for the CW test.

[guilhermebodin]

Indeed it had an error, I fixed it in #205 and corrected the ClarkWestTest struct here

[PaulSoderlind]

I now have files for src and test for this (see below for cut and paste). Two questions:

1. Should I make a new PR or try to change the current one? If the latter, what is the first step to do that?
2. In contrast to the skeleton files, my src file uses a ZTest type. After all, the Clark-West test has an asymptotic motivation/foundation so it makes little sense to try a dof correction. Do you agree with that?

src

# clark_west.jl
# Clark-West
#
# Copyright (C) 2020   Guilherme Bodin, Paul Söderlind
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

export ClarkWestTest

struct ClarkWestTest <: ZTest
    n::Int       # number of observations
    xbar::Real   # estimated mean
    stderr::Real # population standard error
    z::Real      # t-statistic
    μ0::Real     # mean under h_0
end

"""
    ClarkWestTest(e1::AbstractVector{<:Real}, e2::AbstractVector{<:Real};lookahead::Integer=1)

Perform the Clark-West test of equal performance of two nested predition models, in terms of the
out-of-sample mean squared prediction errors.

`e1` is a vector of forecasts from the smaller (nested) model, `e2` is a vector of forecast
errors from the larger model, and `lookahead` is the number of steps ahead of the forecast.
Typically, the null hypothesis is that the two models perform equally well (a two-sided test),
but sometimes we test whether the larger model performs better, which is indicated by a
positive test statistic, for instance, above 1.645 for the 5% significance level (right tail test).

Implements: [`pvalue`](@ref)
# References
 * Clark, T. E., West, K. D. 2006, Using out-of-sample mean squared prediction errors to test
   the martingale difference hypothesis. Journal of Econometrics, 135(1): 155–186.
 * Clark, T. E., West, K. D. 2007, Approximately normal tests for equal predictive accuracy
   in nested models. Journal of Econometrics, 138(1): 291–311.

"""
function ClarkWestTest(e1::AbstractVector{<:Real}, e2::AbstractVector{<:Real};
                       lookahead::Integer=1)
    length(e1) == length(e2) || throw(DimensionMismatch("inputs must have the same length"))
    n            = length(e1)
    d            = 2*e1.*(e1 - e2)
    cw_cov       = HypothesisTests.autocov(d, collect(0:lookahead-1))
    cw_var       = (cw_cov[1] + 2 * sum(cw_cov[2:end]))/n
    statistic_cw = mean(d)/sqrt(cw_var)
    return ClarkWestTest(n,mean(d),sqrt(cw_var),statistic_cw,0.0)
end


testname(::ClarkWestTest) = "Clark West test"
population_param_of_interest(x::ClarkWestTest) = ("Mean",x.μ0, x.xbar)
default_tail(test::ClarkWestTest) = :both

function show_params(io::IO, x::ClarkWestTest, ident)
    println(io, ident, "number of observations:    $(x.n)")
    println(io, ident, "CW statistic:              $(x.z)")
    println(io, ident, "population standard error: $(x.stderr)")
end

test

using HypothesisTests, Test

@testset "Clark-West tests" begin

    e1 = [ 9.78432,  12.73500,   8.67224,   2.62740,   5.60947,
           7.57809,   4.53774,   2.51084,   0.49290,   3.48394,
           9.46152,   9.41220,   2.36289,   0.34495,   3.33599,
           5.31357,   4.28219,  -3.74471,  -5.73575,  -3.71781 ]

    e2 =  [ 2.82053,   4.39754,  -1.78647,  -4.30662,   3.69526,
            3.37259,  -1.09002,  -0.50984,  -0.78973,   3.89077,
            7.52849,   2.82373,  -3.95838,  -0.13606,   4.54444,
            4.18216,   1.67993,  -5.38077,  -0.85686,   2.70479 ]

    atol = 1e-4
    cw_test = ClarkWestTest(e1, e2)
    @test pvalue(cw_test) ≈ 0.0002 atol=atol
    @test pvalue(cw_test, tail=:right) ≈ 0.0001 atol=atol

    cw_test = ClarkWestTest(e1, e2; lookahead=3)
    @test pvalue(cw_test) ≈ 0.0157 atol=atol
    @test pvalue(cw_test, tail=:right) ≈ 0.0079 atol=atol

    show(IOBuffer(), cw_test)

    @test_throws DimensionMismatch ClarkWestTest(rand(3), rand(4))

end

[guilhermebodin]

1 - This PR is ready, if you want I can close it and you open a new one with the exact same changes so GitHub understands that you made the contribution. I have no problem with that, it is up to you :)

2 - I agree it should be a ZTest

[PaulSoderlind]

I am happy to add/change the current PR, but I am no good at this Github machinery. What is the easiest way of doing it? Edit the files here (copy and paste will probably work)?

[guilhermebodin]

you can git clone https://github.com/guilhermebodin/HypothesisTests.jl.git and edit the branch clark-west then commit and push.

[PaulSoderlind]

@guilhermebodin I just checked the src and test files on your clark-west branch - and they look fine (thanks for the changes). What else is needed? doc? If so, can you do that?

[guilhermebodin]

I think we are good, I am going to request for some reviews.

[guilhermebodin]

@mschauer I am not able to click in the reviewers button, could you review this PR?

[azev77]

Thanks guys!
Would it make sense to create a separate category in the Docs for forecast accuracy tests?

[guilhermebodin]

@mschauer is it good?

[PaulSoderlind]

Please let me know if there is anything I can do to make this move forward.

[azev77]

Hey guys while we're talking about tests for predictive accuracy, have you seen @colintbowers' ForecastEval.jl?
He has:

• Diebold-Mariano Test (DM): already implemented
• Reality Check (RC):
• Superior Predictive Ability (SPA) Test
• Model Confidence Set (MCS)

[colintbowers]

Hi all. Yes please feel free to integrate anything from ForecastEval.jl that is of use. I'm happy to help as much as I'm able. One point probably worth noting also is that if you check out ForecastEval.jl, you'll note it depends on DependentBootstrap.jl, which is another registered package of mine. DependentBootstrap.jl implements the iid, stationary, circular, and moving block bootstraps, along with associated block-length selection procedures and bandwidth selection routines. This might be useful if you ever want to offer variants of some of the hypothesis tests here that use bootstrapping to estimate statistical distributions, such as I have done for my implementation of Diebold-Mariano in ForecastEval.jl. And obviously for some stats like the SPA test and Model Confidence Set, they can only be implemented in practice using a bootstrap.

[mschauer]

@nalimilan Want to have a look before I merge?

[nalimilan]

Int is better as a neutral zero as it doesn't force promotion of e.g. Float32:

Suggested change

	return ClarkWestTest(n, xbar, stderr, statistic_cw, 0.0)
	return ClarkWestTest(n, xbar, stderr, statistic_cw, 0)

But is it really useful to have this field? Is it required by the ZTest interface (if it exists at all)?

[PaulSoderlind]

only used in population_param_of_interest. Probably not needed

[guilhermebodin]

It is on the ZTest interface

struct OneSampleZTest <: ZTest
    n::Int       # number of observations
    xbar::Real   # estimated mean
    stderr::Real # population standard error
    z::Real      # z-statistic
    μ0::Real     # mean under h_0
end

[nalimilan]

This isn't an interface, it's the definition of a concrete struct inheriting from ZTest.

[nalimilan]

IIUC this is a Z statistic?

[PaulSoderlind]

yes

[nalimilan]

I mean, should it be mentioned as such? Also related: is there a point in inheriting from ZTest?

[mschauer]

Thank you!

[PaulSoderlind]

@guilhermebodin would you like to go ahead and make the suggested changes?

In case you don't have time to do it soon, I can give it a go. If so, would I just fork your repo to get the current files and send a PR to you? Is that how it works?

[guilhermebodin]

@mschauer Can we merge it?

[mschauer]

Comments, @nalimilan?