Add Generalised Linear Models to FSharpStats #334

LibraChris · 2024-06-10T12:21:41Z

Changes introduced in this PR

Implemented QR-based solver for GLMs
Added multiple tests for various GLM link and variance functions
Added functions for GLM statistics and documentation

Description

This PR introduces the GLM (Generalized Linear Models) module in FSharp.Stats. The key features include the implementation of an Iteratively Reweighted Least Squares (IRLS) solver on a QR-based solver for GLMs, enhancing the accuracy and performance of statistical modeling. Extensive tests have been added and updated to ensure robustness, covering various link functions and variance calculations. Documentation has been thoroughly updated to assist users in leveraging these new features effectively.

[Required] please make sure you checked that

The project builds without problems on your machine

[Optional]

Added unit tests regarding the added features

LibraChris · 2024-06-10T12:44:06Z

Hey @bvenn, would you mind checking this out? Please squish all the commits if possible to avoid confusion. If there is anything I should add or change please let me know

kMutagene · 2024-06-11T09:09:39Z

src/FSharp.Stats/Fitting/GeneralisedLinearModel.fs

+            }
+        )
+
+module QR =


Is this the correct location for a function like this? Are these functions so specialized that nobody ever needs to use them in another context? if not, i'd suggest putting them in the managed LinearAlgebra module, otherwise you can also set the whole QR module here to internal

While some of the functions in QR are highly specialised, some, qrAlternative and solveLinearQR, aren't. The module is needed for future development when I will add a solver based on singular value decomposition, but at the moment it is redundant.

I will move the the specified functions to LinearAlgebra but keep the rest there if that´s ok.

Allrighty. You can then mark the whole specialized module as internal to avoid leaking any of that stuff in the public API.

Doing that would also remove it from the testing I on the stepwise function. You can have a look and tell me if it is ok now.

You can expose internals to other projects in the repo by adding this attribute in the project file:

<ItemGroup> <InternalsVisibleTo Include="TestProject1" /> </ItemGroup>

@kMutagene hey, I changed the code accordingly. Could you have another look if its better now?

src/FSharp.Stats/Fitting/GeneralisedLinearModel.fs

kMutagene · 2024-06-19T10:58:44Z

tests/FSharp.Stats.Tests/GeneralisedLinearModels.fs

+            for i=0 to testingArray.Length-1 do
+                let expected    = linkExpected.[i]
+                let actual      = linkFActual.[i] 
+                if isInf actual then


I do not understand this test setup. I think you should base the first conditional on the expected value and not on the actual value, e.g. "If the expected value is infinity then actual should be infinity", same for the tests below

kMutagene

LGTM from a maintenance standpoint - i cannot assess the implementation itself or the Linear Algebra though, so as long as your tests pass i think this is fine, test setup is a little weird but maybe i just do not understand the specifics

LibraChris · 2024-06-19T11:50:05Z

Hey @bvenn, anything you would change before the merge?

bvenn · 2024-06-22T19:19:56Z

Hey @bvenn, anything you would change before the merge?

Sorry, I was busy during the last weeks. I'll check out your PR this week.

bvenn

GLM + documentation looks good, but I have a general thought: Why is the GLM module assigned to the Fitting-namespace rather than Testing? I know that GLMs are directly linked to regression analysis, but I personally would search at Testing when looking for GLMs. But maybe I'm wrong here, lets discuss.

bvenn · 2024-06-23T17:39:22Z

src/FSharp.Stats/Algebra/LinearAlgebra.fs

+            let qN = setqOfA i 
+            let qLength = getVectorLength qN
+            let rValue = sqrt(qLength)
+            r[i,i] <- rValue
+            qLengths[i] <- qLength


I'm not sure if I get whats happening here:

qLengths is a zero-vector that gets filled iteratively for each i in line 262

in line 256 setqOfA is called with the current i. However in l 247 qLength is called at position i which always should be 0, because it was initialized with 0. Just after the determination of qLength the value at this specific index is replaced in line 262.

Obviously I miss a step, but is there a possibility to create the qLength by Vector.init (...) rather than keeping it mutable and accessing/mutating it at multiple positions? If not, I'm happy to merge it, but I got confused during my attempt to understand whats happening here..

Could you additionally set into -brackets a reference to this implementation and a short description what the benefit, drawback is when using this implementation over others.

I do not have a direct reference since I had a look at the mathematical explanation for QR via Gram Schmidt and tried implementing it myself. The only real upside using this method instead of the in FSharp.Stats established one is the difference in the output Dimensions in R. An actual reference would be the numpy implantation of the reduced qr https://numpy.org/doc/stable/reference/generated/numpy.linalg.qr.html .

I also think that this implemtation needs to be optimised in the future to perfect the GLM.

I'm not sure if I get whats happening here:

qLengths is a zero-vector that gets filled iteratively for each i in line 262

in line 256 setqOfA is called with the current i. However in l 247 qLength is called at position i which always should be 0, because it was initialized with 0. Just after the determination of qLength the value at this specific index is replaced in line 262.

Obviously I miss a step, but is there a possibility to create the qLength by Vector.init (...) rather than keeping it mutable and accessing/mutating it at multiple positions? If not, I'm happy to merge it, but I got confused during my attempt to understand whats happening here..

The mutation was done to simplify the function to match the mathematical explanation better. This is one of the things that could be updated when I get to it.

src/FSharp.Stats/Fitting/GeneralisedLinearModel.fs

bvenn · 2024-06-23T18:23:04Z

src/FSharp.Stats/Fitting/GeneralisedLinearModel.fs

+            // Computes the link function value for a given parameter.
+            getLink = fun b -> Math.Pow(b,-2.)//1.0 / b
+            // Computes the inverse link function value for a given parameter.
+            getInvLink = fun a -> Math.Pow(a,(1./ -2.))//1.0 / a


I assume the 1./2. and later -1.-1. are left on purpose to make the implementation easier to understand 👍

bvenn · 2024-06-23T18:24:37Z

src/FSharp.Stats/Fitting/GeneralisedLinearModel.fs

+    let internal signFunction x =
+        if x > 0. then 1.
+        elif x = 0. then 0.
+        else -1.


You could use the build in sign function

bvenn · 2024-06-23T18:26:31Z

src/FSharp.Stats/Fitting/GeneralisedLinearModel.fs

+                    let b = System.Math.Log(a)
+                    let c = endV - muV
+                    let d = c / muV
+                    let e = -b + d


e isn't used, is it correct?

bvenn · 2024-06-23T18:40:31Z

When watching the docs dotnet fsdocs watch --eval I noticed the API-Reference is broken. Nothing is displayed. I'm not sure if the cause lies within this PR or some error happened earlier. If you find time, you can check if it worked prior to your commits.

bvenn · 2024-07-03T05:59:34Z

When watching the docs dotnet fsdocs watch --eval I noticed the API-Reference is broken. Nothing is displayed. I'm not sure if the cause lies within this PR or some error happened earlier. If you find time, you can check if it worked prior to your commits.

The issue seems to be older than this PR or it is due to my setup. Could you check the docs in your machine?

kMutagene · 2024-07-03T06:52:44Z

When watching the docs dotnet fsdocs watch --eval I noticed the API-Reference is broken. Nothing is displayed. I'm not sure if the cause lies within this PR or some error happened earlier. If you find time, you can check if it worked prior to your commits.

The issue seems to be older than this PR or it is due to my setup. Could you check the docs in your machine?

Try using the build target (build.cmd watchdocs), which should make sure that the necessary binaries are built before creating the docs

bvenn · 2024-07-03T09:40:45Z

I still get the following error during build watchdocs:

"System.Xml.XmlException: The ',' character, hexadecimal value 0x2C, cannot be included in a name. Line 15278, position 99"
I think I fixed an issue similar to this within the last months.

@LibraChris your branch is 42 commits behind, if you could rebase, the issue may be gone

Implement iterative reweighted least squares (IRLS) solver for generalised linear models

add qr based GLM and Statistics for it

- Update glm QR Solver by substituting a Cholesky decomposition with another QR decomposition. - Update InverseLinkFunction to be identical to python/r linker function

- Rename Poisson -> LogLinkFunction - Rename Gamme -> InverseLinkFunction

Rename PersonOfZ to PearsonOfZScore

bvenn · 2024-07-18T15:51:44Z

tests/FSharp.Stats.Tests/FSharp.Stats.Tests.fsproj

+	<Compile Include="GeneralisedLinearModels.fs" />
+	  <Compile Include="Quantile.fs" />
    <Compile Include="Rank.fs" />
    <Compile Include="Main.fs" />
  </ItemGroup>
  <ItemGroup>
+	<EmbeddedResource Include="data/glm_test_lungcap.csv" CopyToOutputDirectory="PreserveNewest" />
+	<EmbeddedResource Include="data/glm_test_energy.csv" CopyToOutputDirectory="PreserveNewest" />
+	<EmbeddedResource Include="data/glm_test_cheese.csv" CopyToOutputDirectory="PreserveNewest" />


Could you please fix these indentations

bvenn · 2024-09-02T15:58:38Z

src/FSharp.Stats/Algebra/LinearAlgebra.fs

+    /// Solves a linear system of equations using QR decomposition.
+    /// 
+    /// Parameters:
+    ///   - A: The coefficient matrix of the linear system.
+    ///   - t: The target vector of the linear system.
+    /// 
+    /// Returns:
+    ///   - mX: The solution vector of the linear system.
+    ///   - r: The upper triangular matrix obtained from QR decomposition.


Thanks for the updates!
I'm going to merge this PR if you just could add proper XML docs like seen in:

FSharp.Stats/src/FSharp.Stats/Fitting/LinearRegression.fs

Lines 128 to 145 in c5444cc

/// <summary>

/// Calculates the slope of a regression line through the origin

/// </summary>

/// <param name="xData">vector of x values</param>

/// <param name="yData">vector of y values</param>

/// <returns>Slope of the regression line through the origin</returns>

/// <example>

/// <code>

/// // e.g. days since a certain event

/// let xData = vector [|0.;1.;2.;3.;4.;5.;|]

/// // some measured feature, that theoretically is zero at day 0

/// let yData = vector [|1.;5.;9.;13.;17.;18.;|]

///

/// // Estimate the slope of the regression line.

/// let coefficients =

/// LinearRegression.OLS.Linear.RTO.fit xData yData

/// </code>

/// </example>

The same for line 222 f.

Updated XML documentation for qrAlternative and solveLinearQR

kMutagene reviewed Jun 11, 2024

View reviewed changes

LibraChris added a commit to LibraChris/FSharp.Stats that referenced this pull request Jun 18, 2024

Adress changes requested in fslaborg#334

fb32a4a

kMutagene requested changes Jun 19, 2024

View reviewed changes

src/FSharp.Stats/Fitting/GeneralisedLinearModel.fs Show resolved Hide resolved

kMutagene reviewed Jun 19, 2024

View reviewed changes

kMutagene approved these changes Jun 19, 2024

View reviewed changes

bvenn requested changes Jun 23, 2024

View reviewed changes

LibraChris added 17 commits July 3, 2024 15:20

Implement IRLS solver for GLMs

6288d55

Implement iterative reweighted least squares (IRLS) solver for generalised linear models

Rename variables

425d12b

add qr based GLM

d1c9c41

add qr based GLM and Statistics for it

add inital tests for the glm

d637be7

Update glm QR Solver

d0fc5ee

- Update glm QR Solver by substituting a Cholesky decomposition with another QR decomposition. - Update InverseLinkFunction to be identical to python/r linker function

Add new Test for GLMs using Gamma Distribution

d24955a

Add tests for the Poisson linker functions

ddcf09c

Add tests for the Gamma linker functions

60c3ec1

Rename testcases to Reflect their log function

0f0661c

- Rename Poisson -> LogLinkFunction - Rename Gamme -> InverseLinkFunction

Add tests for the LogitLinkFunction

fe83ba6

Add tests for the InverseSquaredLinkFunction

ab44068

Add tests by example for glm IrLS solver

c00980e

Add tests for the IdentityLinkFunction

1e6a524

Add tests groudwork for the BinomialLinkFunction

ac416bf

Add tests for the variance of Binominal Family

8f88c1e

Add tests for the variance of Poisson Family

148a933

Add tests for the variance of Gaussian/Normal Family

a8b5f00

LibraChris and others added 21 commits July 3, 2024 15:20

Rework GLM QR Solver

43cea23

Modify tests

3e83833

Add tests prototype for QR-Stepwise iteration

2816155

Fix QR based solver for GLMs

c5ced84

Modify Variance tests

5029c3a

Update statistics

c3dddcb

Update GeneralisedLinearModel.fs

a7c5c1b

Update GeneralisedLinearModel.fs

d8877b7

Update GeneralisedLinearModel.fs

3cd68a8

Rework GLMStatistics

253ac91

Remove deprecated GLM.Irls

19cad0f

Fix minor testing issue

1b3336f

add getFamilyReisualDeviance for more families

a1d0ee4

Write code comments and documentation

37d03e0

add Documentation for GLM Usage

5e9a1b6

Update formating for documentation

72bfb83

added data for Documentation

a8a0004

remote tests for binominal family variance

f694340

Adress changes requested in #344

8dcd8ab

Adress changes requested in fslaborg#334

170519e

Update xml comments

ba5ae9c

LibraChris force-pushed the glm branch from 469422c to ba5ae9c Compare July 3, 2024 13:20

LibraChris added 2 commits July 3, 2024 15:37

fix building error

6c3a235

Fix Typo

2e80081

Rename PersonOfZ to PearsonOfZScore

bvenn reviewed Jul 18, 2024

View reviewed changes

Fix indentations

13b3de9

bvenn requested changes Sep 2, 2024

View reviewed changes

Updated XML documentation

df24c3f

Updated XML documentation for qrAlternative and solveLinearQR

bvenn approved these changes Oct 24, 2024

View reviewed changes

bvenn merged commit 57f8939 into fslaborg:developer Oct 24, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Generalised Linear Models to FSharpStats #334

Add Generalised Linear Models to FSharpStats #334

LibraChris commented Jun 10, 2024

LibraChris commented Jun 10, 2024

kMutagene Jun 11, 2024

LibraChris Jun 11, 2024

kMutagene Jun 11, 2024

LibraChris Jun 12, 2024

kMutagene Jun 13, 2024

LibraChris Jun 18, 2024

kMutagene Jun 19, 2024

kMutagene left a comment

LibraChris commented Jun 19, 2024

bvenn commented Jun 22, 2024

bvenn left a comment

bvenn Jun 23, 2024 •

edited

Loading

bvenn Jun 23, 2024

LibraChris Jul 3, 2024

LibraChris Jul 3, 2024

bvenn Jun 23, 2024

bvenn Jun 23, 2024

bvenn Sep 2, 2024

bvenn Jun 23, 2024

bvenn Sep 2, 2024

bvenn commented Jun 23, 2024

bvenn commented Jul 3, 2024

kMutagene commented Jul 3, 2024

bvenn commented Jul 3, 2024

bvenn Jul 18, 2024

bvenn Sep 2, 2024

	/// <summary>
	/// Calculates the slope of a regression line through the origin
	/// </summary>
	/// <param name="xData">vector of x values</param>
	/// <param name="yData">vector of y values</param>
	/// <returns>Slope of the regression line through the origin</returns>
	/// <example>
	/// <code>
	/// // e.g. days since a certain event
	/// let xData = vector [\|0.;1.;2.;3.;4.;5.;\|]
	/// // some measured feature, that theoretically is zero at day 0
	/// let yData = vector [\|1.;5.;9.;13.;17.;18.;\|]
	///
	/// // Estimate the slope of the regression line.
	/// let coefficients =
	/// LinearRegression.OLS.Linear.RTO.fit xData yData
	/// </code>
	/// </example>

Add Generalised Linear Models to FSharpStats #334

Add Generalised Linear Models to FSharpStats #334

Conversation

LibraChris commented Jun 10, 2024

LibraChris commented Jun 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kMutagene left a comment

Choose a reason for hiding this comment

LibraChris commented Jun 19, 2024

bvenn commented Jun 22, 2024

bvenn left a comment

Choose a reason for hiding this comment

bvenn Jun 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bvenn commented Jun 23, 2024

bvenn commented Jul 3, 2024

kMutagene commented Jul 3, 2024

bvenn commented Jul 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bvenn Jun 23, 2024 •

edited

Loading