[FEATURE] Add a "debug mode" to help detailed analysis

[sarthakpati]

Is your feature request related to a problem? Please describe.

Currently, when a model fails to train, there is nothing a user can do other than to get into the code and start putting print statements everywhere, which is cumbersome.

Describe the solution you'd like

It would be good to have this as a parameter so that a few things can be recorded under a new ${output_dir}/debug_info directory.

Describe alternatives you've considered

N.A.

Additional context

The following things can be added (per epoch):

• Related to [FEATURE] Add tensorboard support #834
• Input and output tensors along with the target.
• Model weights
• Medcam outputs (regardless of whether it was enabled or not)

Notes from SP

• wandb would be great for this, but it needs server deployment [ref], and this is a major bar for a regular user. If we ever have enterprise support, this can be explored as a cloud-hosted solution.
• From what I have seen, for local deployment, torchexplorer provides the majority of the items we need.

[github-actions]

Stale issue message