Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: Add feature contributions as attribute and tutorial on how to use #448

Merged
merged 41 commits into from
Sep 3, 2024

Conversation

marcopeix
Copy link
Contributor

Adds the attribute feature_contributions to the client when exogenous features are used.
Tutorial on how to access feature contributions and use shap to make different plots.

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

fern/openapi/timegpt.json Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Show resolved Hide resolved
nixtla/nixtla_client.py Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Outdated Show resolved Hide resolved
nixtla/types/forecast_output.py Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Outdated Show resolved Hide resolved
nixtla/nixtla_client.py Show resolved Hide resolved
@jmoralez
Copy link
Member

Thanks! Please add a test with num_partitions on the cell where the test for the contributions is. It can be something like the following:

# this part already exists
fcst_hist_df = nixtla_client.forecast(df=df_date_features, h=h, X_df=future_df, add_history=True, feature_contributions=True)
shap_values_hist = nixtla_client.feature_contributions.copy()  # <- add the copy here
assert len(shap_values_hist) == len(fcst_hist_df)
np.testing.assert_allclose(fcst_hist_df["TimeGPT"].values, shap_values_hist.iloc[:, 3:].sum(axis=1).values)

# the following is new

# test num partitions
_ = nixtla_client.forecast(df=df_date_features, h=h, X_df=future_df, add_history=True, feature_contributions=True, num_partitions=2)
pd.testing.assert_frame_equal(nixtla_client.feature_contributions, shap_values_hist)

Also please add this on the forecast method:

self.__dict__.pop('feature_contributions', None)

Just below the one for weights_x

@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
Copy link
Contributor

Experiment Results

Experiment 1: air-passengers

Description:

variable experiment
h 12
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 12.6793 11.0623 47.8333 76
mape 0.027 0.0232 0.0999 0.1425
mse 213.936 199.132 2571.33 10604.2
total_time 19.0151 7.2955 0.0064 0.0041

Plot:

Experiment 2: air-passengers

Description:

variable experiment
h 24
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 58.1031 58.4587 71.25 115.25
mape 0.1257 0.1267 0.1552 0.2358
mse 4040.22 4110.79 5928.17 18859.2
total_time 1.6552 0.9024 0.0048 0.0042

Plot:

Experiment 3: electricity-multiple-series

Description:

variable experiment
h 24
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 178.293 268.121 269.23 1331.02
mape 0.0234 0.0311 0.0304 0.1692
mse 121588 219457 213677 4.68961e+06
total_time 1.6161 3.6996 0.0054 0.0049

Plot:

Experiment 4: electricity-multiple-series

Description:

variable experiment
h 168
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 465.532 346.984 398.956 1119.26
mape 0.062 0.0437 0.0512 0.1583
mse 835120 403787 656723 3.17316e+06
total_time 0.9678 2.0231 0.006 0.0055

Plot:

Experiment 5: electricity-multiple-series

Description:

variable experiment
h 336
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 558.649 459.769 602.926 1340.95
mape 0.0697 0.0566 0.0787 0.17
mse 1.22721e+06 739135 1.61572e+06 6.04619e+06
total_time 2.2391 1.3987 0.0058 0.0054

Plot:

Copy link
Member

@jmoralez jmoralez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Please don't merge this until we release v2, otherwise it'll show up in the docs and we'll get issues about it.

@Nixtla Nixtla deleted a comment from github-actions bot Aug 26, 2024
Copy link
Contributor

Experiment Results

Experiment 1: air-passengers

Description:

variable experiment
h 12
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 12.6793 11.0623 47.8333 76
mape 0.027 0.0232 0.0999 0.1425
mse 213.936 199.132 2571.33 10604.2
total_time 1.9381 1.9746 0.0061 0.0039

Plot:

Experiment 2: air-passengers

Description:

variable experiment
h 24
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 58.1031 58.4587 71.25 115.25
mape 0.1257 0.1267 0.1552 0.2358
mse 4040.21 4110.79 5928.17 18859.2
total_time 0.7434 0.775 0.0043 0.0039

Plot:

Experiment 3: electricity-multiple-series

Description:

variable experiment
h 24
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 178.293 268.121 269.23 1331.02
mape 0.0234 0.0311 0.0304 0.1692
mse 121588 219457 213677 4.68961e+06
total_time 3.0712 3.034 0.0052 0.0048

Plot:

Experiment 4: electricity-multiple-series

Description:

variable experiment
h 168
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 465.532 346.984 398.956 1119.26
mape 0.062 0.0437 0.0512 0.1583
mse 835121 403787 656723 3.17316e+06
total_time 2.0271 1.4137 0.0055 0.005

Plot:

Experiment 5: electricity-multiple-series

Description:

variable experiment
h 336
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 558.649 459.769 602.926 1340.95
mape 0.0697 0.0566 0.0787 0.17
mse 1.22721e+06 739135 1.61572e+06 6.04619e+06
total_time 1.4194 1.1337 0.0058 0.0052

Plot:

Copy link
Contributor

Experiment Results

Experiment 1: air-passengers

Description:

variable experiment
h 12
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 12.6793 11.0623 47.8333 76
mape 0.027 0.0232 0.0999 0.1425
mse 213.936 199.132 2571.33 10604.2
total_time 2.0895 0.7611 0.0066 0.004

Plot:

Experiment 2: air-passengers

Description:

variable experiment
h 24
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 58.1031 58.4587 71.25 115.25
mape 0.1257 0.1267 0.1552 0.2358
mse 4040.21 4110.79 5928.17 18859.2
total_time 0.5304 0.7691 0.0048 0.0041

Plot:

Experiment 3: electricity-multiple-series

Description:

variable experiment
h 24
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 178.293 268.121 269.23 1331.02
mape 0.0234 0.0311 0.0304 0.1692
mse 121588 219457 213677 4.68961e+06
total_time 0.995 1.0451 0.0053 0.0052

Plot:

Experiment 4: electricity-multiple-series

Description:

variable experiment
h 168
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 465.532 346.984 398.956 1119.26
mape 0.062 0.0437 0.0512 0.1583
mse 835120 403787 656723 3.17316e+06
total_time 0.6201 0.568 0.0062 0.0054

Plot:

Experiment 5: electricity-multiple-series

Description:

variable experiment
h 336
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 558.649 459.769 602.926 1340.95
mape 0.0697 0.0566 0.0787 0.17
mse 1.22721e+06 739135 1.61572e+06 6.04619e+06
total_time 1.0763 0.6128 0.0059 0.0056

Plot:

Copy link
Contributor

github-actions bot commented Sep 3, 2024

Experiment Results

Experiment 1: air-passengers

Description:

variable experiment
h 12
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 12.6793 11.0623 47.8333 76
mape 0.027 0.0232 0.0999 0.1425
mse 213.936 199.132 2571.33 10604.2
total_time 0.9519 1.1893 0.0064 0.004

Plot:

Experiment 2: air-passengers

Description:

variable experiment
h 24
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 58.1031 58.4587 71.25 115.25
mape 0.1257 0.1267 0.1552 0.2358
mse 4040.21 4110.79 5928.17 18859.2
total_time 0.736 1.2682 0.0044 0.004

Plot:

Experiment 3: electricity-multiple-series

Description:

variable experiment
h 24
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 178.293 268.121 269.23 1331.02
mape 0.0234 0.0311 0.0304 0.1692
mse 121588 219457 213677 4.68961e+06
total_time 1.049 1.2738 0.0053 0.0051

Plot:

Experiment 4: electricity-multiple-series

Description:

variable experiment
h 168
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 465.532 346.984 398.956 1119.26
mape 0.062 0.0437 0.0512 0.1583
mse 835120 403787 656723 3.17316e+06
total_time 0.9634 2.4906 0.0057 0.0051

Plot:

Experiment 5: electricity-multiple-series

Description:

variable experiment
h 336
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 558.649 459.769 602.926 1340.95
mape 0.0697 0.0566 0.0787 0.17
mse 1.22721e+06 739135 1.61572e+06 6.04619e+06
total_time 1.1737 2.2624 0.0058 0.0053

Plot:

Copy link
Contributor

github-actions bot commented Sep 3, 2024

Experiment Results

Experiment 1: air-passengers

Description:

variable experiment
h 12
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 12.6793 11.0623 47.8333 76
mape 0.027 0.0232 0.0999 0.1425
mse 213.936 199.132 2571.33 10604.2
total_time 10.241 1.7504 0.0073 0.0041

Plot:

Experiment 2: air-passengers

Description:

variable experiment
h 24
season_length 12
freq MS
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 58.1031 58.4587 71.25 115.25
mape 0.1257 0.1267 0.1552 0.2358
mse 4040.21 4110.79 5928.17 18859.2
total_time 0.4633 0.5913 0.0048 0.0045

Plot:

Experiment 3: electricity-multiple-series

Description:

variable experiment
h 24
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 178.293 268.121 269.23 1331.02
mape 0.0234 0.0311 0.0304 0.1692
mse 121588 219457 213677 4.68961e+06
total_time 1.4791 1.9427 0.0053 0.0049

Plot:

Experiment 4: electricity-multiple-series

Description:

variable experiment
h 168
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 465.532 346.984 398.956 1119.26
mape 0.062 0.0437 0.0512 0.1583
mse 835120 403787 656723 3.17316e+06
total_time 0.7866 0.4723 0.0056 0.0053

Plot:

Experiment 5: electricity-multiple-series

Description:

variable experiment
h 336
season_length 24
freq H
level None
n_windows 1

Results:

metric timegpt-1 timegpt-1-long-horizon SeasonalNaive Naive
mae 558.649 459.769 602.926 1340.95
mape 0.0697 0.0566 0.0787 0.17
mse 1.22721e+06 739135 1.61572e+06 6.04619e+06
total_time 0.9021 0.4284 0.0057 0.0054

Plot:

@marcopeix marcopeix merged commit e26e434 into main Sep 3, 2024
13 checks passed
@marcopeix marcopeix deleted the feature/shap branch September 3, 2024 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants