Baseline model notebook and embeddings trainer notebook

[cocochrane]

No description provided.

[review-notebook-app]

Check out this pull request on ReviewNB: https://app.reviewnb.com/microsoft/nlp/pull/47

Visit www.reviewnb.com to know how we simplify your Jupyter Notebook workflows.

[cocochrane]

@AbhiramE @caseyhong @irshaffe @irshaffe @catherine667 Opened this PR to the staging branch. You all have already reviewed this work, but tagging you in case you want to make any more edits!

[saidbleik]

We might want to keep the utils in preprocess.py general-purpose
For example:

• pass the columns names that need to be lower-cased in to_lowercase()
• to_nltk_tokens() and rm_nltk_stopwords() take in an arbitrary number of columns but only 2 are operated on.

[cocochrane]

We might want to keep the utils in preprocess.py general-purpose
For example:

pass the columns names that need to be lower-cased in to_lowercase()
to_nltk_tokens() and rm_nltk_stopwords() take in an arbitrary number of columns but only 2 are operated on.

@caseyhong @AbhiramE @janhavi13 Just keeping you updated on the comments that correspond to changes to the utils

[saidbleik]

@caseyhong @AbhiramE @janhavi13 Just keeping you updated on the comments that correspond to changes to the utils

We can also have 2 versions like to_lowercase_all(df) and to_lowecase(df, col_names)

[jisooghd]

@saidbleik makes sense for the lowercase. for your second point, do you mean we should explicitly enforce the 2 column limit since that is what is actually happening under the hood? so

to_spacy_tokens(df, sent_col_1, sent_col_2, tok_col_1, tok_col_2)
instead of
to_spacy_tokens(df, sentence_cols=["sentence1", "sentence2"], token_cols=["sentence1_tokens", "sentence2_tokens"])
?

[saidbleik]

@saidbleik makes sense for the lowercase. for your second point, do you mean we should explicitly enforce the 2 column limit since that is what is actually happening under the hood? so

to_spacy_tokens(df, sent_col_1, sent_col_2, tok_col_1, tok_col_2)
instead of
to_spacy_tokens(df, sentence_cols=["sentence1", "sentence2"], token_cols=["sentence1_tokens", "sentence2_tokens"])
?

No, I meant the current implementation only applies to token_cols[0] and token_cols[1]. It should allow an arbitrary number of columns.

[janhavi13]

@saidbleik makes sense for the lowercase. for your second point, do you mean we should explicitly enforce the 2 column limit since that is what is actually happening under the hood? so
to_spacy_tokens(df, sent_col_1, sent_col_2, tok_col_1, tok_col_2)
instead of
to_spacy_tokens(df, sentence_cols=["sentence1", "sentence2"], token_cols=["sentence1_tokens", "sentence2_tokens"])
?

No, I meant the current implementation only applies to token_cols[0] and token_cols[1]. It should allow an arbitrary number of columns.

@saidbleik - so like loop through the token_cols[] list passed and do the function for each column. Is that correct ? That way it's not restricted to [0] and [1] ?

[saidbleik]

@saidbleik makes sense for the lowercase. for your second point, do you mean we should explicitly enforce the 2 column limit since that is what is actually happening under the hood? so
to_spacy_tokens(df, sent_col_1, sent_col_2, tok_col_1, tok_col_2)
instead of
to_spacy_tokens(df, sentence_cols=["sentence1", "sentence2"], token_cols=["sentence1_tokens", "sentence2_tokens"])
?
No, I meant the current implementation only applies to token_cols[0] and token_cols[1]. It should allow an arbitrary number of columns.

@saidbleik - so like loop through the token_cols[] list passed and do the function for each column. Is that correct ? That way it's not restricted to [0] and [1] ?

yes, the argument is a list (not restricted to 2)

[miguelgfierro]

you guys are doing an amazing job. Sorry I broke your folder structure. If you have any problem, please let me know and I will help :-)

I did a pass to the notebooks and will do another pass later

[janhavi13]

@saidbleik makes sense for the lowercase. for your second point, do you mean we should explicitly enforce the 2 column limit since that is what is actually happening under the hood? so
to_spacy_tokens(df, sent_col_1, sent_col_2, tok_col_1, tok_col_2)
instead of
to_spacy_tokens(df, sentence_cols=["sentence1", "sentence2"], token_cols=["sentence1_tokens", "sentence2_tokens"])
?
No, I meant the current implementation only applies to token_cols[0] and token_cols[1]. It should allow an arbitrary number of columns.

@saidbleik - so like loop through the token_cols[] list passed and do the function for each column. Is that correct ? That way it's not restricted to [0] and [1] ?

yes, the argument is a list (not restricted to 2)

@saidbleik makes sense for the lowercase. for your second point, do you mean we should explicitly enforce the 2 column limit since that is what is actually happening under the hood? so
to_spacy_tokens(df, sent_col_1, sent_col_2, tok_col_1, tok_col_2)
instead of
to_spacy_tokens(df, sentence_cols=["sentence1", "sentence2"], token_cols=["sentence1_tokens", "sentence2_tokens"])
?
No, I meant the current implementation only applies to token_cols[0] and token_cols[1]. It should allow an arbitrary number of columns.

@saidbleik - so like loop through the token_cols[] list passed and do the function for each column. Is that correct ? That way it's not restricted to [0] and [1] ?

yes, the argument is a list (not restricted to 2)

@saidbleik - Take a look at the fixed nltk utils and I also added to_lowercase_all and to_lowercase variation in preprocess.py. Let know if it's good enough.

[saidbleik]

Thanks.