Fanova, Pairs of Variable Plots, AttributeError: 'CategoricalHyperparameter' object has no attribute 'lower' #25
Comments
|
That is definitely in the fANOVA itself. I think that is in part Christina's (@Krxsy) HiWi task for next week. |
|
Yes, it makes sense that there should be a "plot_pairwise_categorical_marginal" function. Right now it only works for a single categorical param. |
|
Hm, however plot is not created and variables are both not categorical, but float and integer: |
|
Ok.. that's weird. Could you give me the configspace info e.g print(cs.get_hyperparameters()) , please? I would like to compare it to the pcs-file for checking if the order of the parameters is correct. |
|
Oki, but how to get the |
|
import ConfigSpace |
|
Hm ok I still don't understand, I assume you want me to create the configspace from the pcs file and then print it right? But how do I read in the pcs file to the config space? Or do you want me to do something else? |
|
I think she want's you to do the following: from ConfigSpace.io import pcs
with open(file_name) as fp:
pcs_str = fp.readlines()
cs = pcs.read(pcs_str)
cs.seed(42)
print(cs.get_hyperparameters())if ' |
|
ok in that case file_name = 'deep4.pcs'
from ConfigSpace.io import pcs
with open(file_name) as fp:
pcs_str = fp.readlines()
cs = pcs.read(pcs_str)
cs.seed(42)
cs.get_hyperparameters()leads to [allow_crash, Type: Categorical, Choices: {true}, Default: true,
data_folder, Type: Categorical, Choices: {/home/schirrmr/data/epilepsy/earlyseiz/manuel_elecs/}, Default: /home/schirrmr/data/epilepsy/earlyseiz/manuel_elecs/,
do_batch_norm, Type: Categorical, Choices: {true, false}, Default: true,
double_time_convs, Type: Categorical, Choices: {true, false}, Default: true,
drop_prob, Type: UniformFloat, Range: [0.0, 0.9], Default: 0.5,
exponential_demean_factor, Type: Categorical, Choices: {0.001}, Default: 0.001,
exponential_standardize_factor, Type: Categorical, Choices: {null}, Default: null,
filter_length_2, Type: UniformInteger, Range: [3, 14], Default: 10,
filter_length_3, Type: UniformInteger, Range: [3, 14], Default: 10,
filter_length_4, Type: UniformInteger, Range: [3, 14], Default: 10,
filter_time_length, Type: UniformInteger, Range: [3, 25], Default: 10,
final_dense_length, Type: UniformInteger, Range: [3, 14], Default: 10,
first_nonlin, Type: Categorical, Choices: {elu, identity, relu, square}, Default: elu,
first_pool_mode, Type: Categorical, Choices: {average_exc_pad, max}, Default: max,
last_seizure_train_ms, Type: UniformFloat, Range: [10000.0, 60000.0], Default: 30000.0, on log-scale,
later_nonlin, Type: Categorical, Choices: {elu, relu, square}, Default: elu,
later_pool_mode, Type: Categorical, Choices: {average_exc_pad, max}, Default: max,
n_chans, Type: Categorical, Choices: {null}, Default: null,
n_epochs, Type: UniformInteger, Range: [15, 16], Default: 15,
non_seiz_to_seiz_factor, Type: UniformFloat, Range: [2.0, 9.0], Default: 5.0,
num_filters_2, Type: UniformInteger, Range: [15, 100], Default: 25,
num_filters_3, Type: UniformInteger, Range: [15, 100], Default: 25,
num_filters_4, Type: UniformInteger, Range: [15, 100], Default: 25,
num_filters_spat, Type: UniformInteger, Range: [15, 100], Default: 25,
num_filters_time, Type: UniformInteger, Range: [15, 100], Default: 25,
only_return_exp, Type: Categorical, Choices: {false}, Default: false,
pool_time_length, Type: UniformInteger, Range: [1, 8], Default: 3,
pool_time_stride, Type: UniformInteger, Range: [1, 8], Default: 3,
resample_mean_pred_ms, Type: UniformFloat, Range: [2000.0, 20000.0], Default: 2000.0, on log-scale,
run_file, Type: Categorical, Choices: {/home/schirrmr/code/earlyseiz/configs/deep4_net.py}, Default: /home/schirrmr/code/earlyseiz/configs/deep4_net.py,
sample_by_preds, Type: Categorical, Choices: {true, false}, Default: true,
sampling_rate, Type: UniformFloat, Range: [32.0, 256.0], Default: 128.0,
save_folder, Type: Categorical, Choices: {/home/schirrmr/data/epilepsy/earlyseiz/models/deep4-pscmac-dirs/}, Default: /home/schirrmr/data/epilepsy/earlyseiz/models/deep4-pscmac-dirs/,
split_first_layer, Type: Categorical, Choices: {true, false}, Default: true,
threshold, Type: UniformFloat, Range: [0.2, 0.9], Default: 0.59999999999999998] |
|
So the order is ok. |
|
Ok I put all runhistories here: https://gist.github.com/robintibor/6a378e38cb033c58dd72da1e4c5f6e90 |
|
Ok, thanx. It might take some time until the problem is found/fixed. |
|
Can you try it again with the latest changes and let me know if the error still shows up? |
|
Hm, there seems to be a new problem: >>> python3.5 /home/schirrmr/programs/ParameterImportance/scripts/evaluate.py --scenario_file scenario.txt --history 'runhistories-shared/runhistory*.json' --modus fanova
INFO:Importance:Reading Scenario file and files specified in the scenario
INFO:smac.scenario.scenario.Scenario:Reading scenario file: scenario.txt
INFO:smac.scenario.scenario.Scenario:Output to PIMP_fanova_2017_06_10_15:08:02
INFO:Importance:Reading Runhistory
INFO:Importance:#RunHistories found: 24
INFO:Importance:Combined number of Runhistory data points: 1100
INFO:Importance:Number of Configurations: 274
INFO:Importance:Converting Data and constructing Model
INFO:Importance:Size of training X: (1100, 35)
INFO:Importance:Size of training y: (1100, 1)
INFO:Importance:Data was not imputed
INFO:Importance:Thus the size of X might be smaller than the datapoints in the RunHistory
Traceback (most recent call last):
File "/home/schirrmr/programs/ParameterImportance/scripts/evaluate.py", line 38, in <module>
impute_censored=args.impute) # create importance object
File "/home/schirrmr/programs/ParameterImportance/pimp/importance/importance.py", line 88, in __init__
self.incumbent = incumbents[0]
IndexError: list index out of rangeAnyways thanks for your help until this point I am now on vacation until 19th June :) |
|
That's my fault. Apparently it does not find any traj_aclib2.json files so it can also not determine an incumbent because I assumed that traj_aclib2.json files would always be located in a smac3-output... folder. I'll quickly fix that |
|
Okay with the latest commit (3dfbfda) the assumption is that the trajectory file(s) will be located somewhere in the same directory. Glob is used recursively to go through all subfolders to find all traj files |
|
I am encountering the same problem as OP. Stacktrace: the pcs (new) file: Is there any incumbent of a solution? |
|
OK, this time there are some categoricals in the pcs file and the "most important pairwise marginals" plotting function doesn't take care of that case properly. I will fix that today. |
|
@Krxsy after fixing the/any fanova bug/s could you please update pimps requirements file to use the latest fanova version? |
|
@Krxsy did you fix the error with the latest commit? |
|
Yes, right now it only throws a warning but I'll take care of being able to plot pairwise categorical marginals in the near future. |
|
Does it now generate the other pairwise plots? I probably won't care about
pairwise categoricals but the pairwise numeric plots seem very useful to
me.
(Not at my workspace so can not check if it works)
2017-07-13 11:46 GMT+02:00 Christina Hernández Wunsch <
[email protected]>:
… Yes, right now it only throws a warning but I'll take care of being able
to plot pairwise categorical marginals in the near future.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACL7-l-z9vb_IvHeVaKVRpfmMazOGam5ks5sNedugaJpZM4Ny6Qb>
.
|
|
Yes, it should generate all pairwise marginals plots in which are no categoricals involved. |
|
I reinstalled fanova package (from git) and updated Pimp to master, but still the same error messages. Any clue on how to fix this? |
|
Contrarily to our off line discussion, I actually still do have the problem as described above. Apparently, this problem is dependent on the config space. Attached is the pcs file that causes trouble: (Hereby the pcs of this morning, that worked fine): I do not see a pattern |
|
Ok, that's really weird. The only thing that crosses my mind is that categoricals are somehow mistaken as floats or ints ? But that can't be possible because I assume that the configspace was created correctly... |
Hi,
If I want to create the fanova plots, the single variable plots work, but pairs fail:
If you need more context: http://nbviewer.jupyter.org/gist/robintibor/6dad6629987cd66b060503bc43e4e2bc/Hyperparameter_June_Report.ipynb
The text was updated successfully, but these errors were encountered: