Skip to content

Fix plot_backtest and plot_backtest_interactive on one-step forecast #1260

Merged
merged 3 commits into from
May 12, 2023

Conversation

Mr-Geekman
Copy link
Contributor

@Mr-Geekman Mr-Geekman commented May 5, 2023

Before submitting (must do checklist)

  • Did you read the contribution guide?
  • Did you update the docs? We use Numpy format for all the methods and classes.
  • Did you write any new necessary tests?
  • Did you update the CHANGELOG?

Proposed Changes

  • Process case of horizon=1 and 1 fold separately
  • Fix finding next point using pd.date_range

Closing issues

Closes #1122.

@Mr-Geekman Mr-Geekman self-assigned this May 5, 2023
@Mr-Geekman
Copy link
Contributor Author

Script for plotting:

import matplotlib.pyplot as plt
import pandas as pd

from etna.analysis import plot_backtest, plot_backtest_interactive
from etna.datasets import TSDataset
from etna.metrics import MAE
from etna.models import ProphetModel
from etna.pipeline import FoldMask
from etna.pipeline import Pipeline


def main():
    df = pd.read_csv("examples/data/example_dataset.csv")
    df = TSDataset.to_dataset(df)
    ts = TSDataset(df, freq="D")

    # plot with horizon=1, n_folds=1
    pipeline = Pipeline(model=ProphetModel(), horizon=1)
    metrics_df, forecast_df, fold_info_df = pipeline.backtest(ts=ts, metrics=[MAE()], n_folds=1)
    plot_backtest(forecast_df=forecast_df, ts=ts, history_len=10)
    plt.savefig("plot_horizon_1_folds_1.png")
    fig = plot_backtest_interactive(forecast_df=forecast_df, ts=ts, history_len=10)
    fig.write_image("plot_int_horizon_1_folds_1.png")

    # plot with horizon=1, n_folds=3
    pipeline = Pipeline(model=ProphetModel(), horizon=1)
    metrics_df, forecast_df, fold_info_df = pipeline.backtest(ts=ts, metrics=[MAE()], n_folds=3)
    plot_backtest(forecast_df=forecast_df, ts=ts, history_len=10)
    plt.savefig("plot_horizon_1_folds_3.png")
    fig = plot_backtest_interactive(forecast_df=forecast_df, ts=ts, history_len=10)
    fig.write_image("plot_int_horizon_1_folds_3.png")

    # plot with horizon=7, n_folds=3
    pipeline = Pipeline(model=ProphetModel(), horizon=7)
    metrics_df, forecast_df, fold_info_df = pipeline.backtest(ts=ts, metrics=[MAE()], n_folds=3)
    plot_backtest(forecast_df=forecast_df, ts=ts, history_len=10)
    plt.savefig("plot_horizon_7_folds_3.png")
    fig = plot_backtest_interactive(forecast_df=forecast_df, ts=ts, history_len=10)
    fig.write_image("plot_int_horizon_7_folds_3.png")

    # fold mask example
    pipeline = Pipeline(model=ProphetModel(), horizon=14)
    folds = [
        FoldMask(
            first_train_timestamp=None,
            last_train_timestamp=pd.Timestamp("2019-10-19"),
            target_timestamps=[pd.Timestamp("2019-10-26"), pd.Timestamp("2019-10-27"), pd.Timestamp("2019-11-01")],
        ),
        FoldMask(
            first_train_timestamp=None,
            last_train_timestamp=pd.Timestamp("2019-11-02"),
            target_timestamps=[pd.Timestamp("2019-11-06"), pd.Timestamp("2019-11-08"), pd.Timestamp("2019-11-10")],
        ),
        FoldMask(
            first_train_timestamp=None,
            last_train_timestamp=pd.Timestamp("2019-11-16"),
            target_timestamps=[pd.Timestamp("2019-11-20"), pd.Timestamp("2019-11-22"), pd.Timestamp("2019-11-24")],
        ),
    ]
    metrics_df, forecast_df, fold_info_df = pipeline.backtest(ts=ts, metrics=[MAE()], n_folds=folds)

    plot_backtest(forecast_df=forecast_df, ts=ts, history_len=50)
    plt.savefig("plot_masks.png")
    fig = plot_backtest_interactive(forecast_df=forecast_df, ts=ts, history_len=50)
    fig.write_image("plot_int_masks.png")


if __name__ == "__main__":
    main()

plot_horizon_1_folds_1

plot_int_horizon_1_folds_1

plot_horizon_1_folds_3

plot_int_horizon_1_folds_3

plot_horizon_7_folds_3

plot_int_horizon_7_folds_3

plot_masks

plot_int_masks

@github-actions
Copy link

github-actions bot commented May 5, 2023

@github-actions github-actions bot temporarily deployed to pull request May 5, 2023 13:23 Inactive
@codecov-commenter
Copy link

codecov-commenter commented May 5, 2023

Codecov Report

Merging #1260 (aa4d99d) into master (e1f642f) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #1260      +/-   ##
==========================================
- Coverage   87.99%   87.97%   -0.03%     
==========================================
  Files         186      186              
  Lines       10749    10747       -2     
==========================================
- Hits         9459     9455       -4     
- Misses       1290     1292       +2     
Impacted Files Coverage Δ
etna/analysis/forecast/plots.py 18.83% <0.00%> (-0.11%) ⬇️

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions github-actions bot temporarily deployed to pull request May 12, 2023 12:12 Inactive
@Mr-Geekman Mr-Geekman merged commit 405764c into master May 12, 2023
@Mr-Geekman Mr-Geekman deleted the issue-1122 branch May 12, 2023 13:20
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] etna.analysis.backtest does't visualize the forecast for a horizon equal to one
3 participants