Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chain functions broken due to oa prefix in new batch-machine #18

Open
andrewharvey opened this issue Jul 11, 2021 · 0 comments
Open

chain functions broken due to oa prefix in new batch-machine #18

andrewharvey opened this issue Jul 11, 2021 · 0 comments

Comments

@andrewharvey
Copy link
Contributor

https://github.com/openaddresses/openaddresses/blob/master/ATTRIBUTE_FUNCTIONS.md#chain documents use of chain functions with a _wip variable, the example given is:

"street": {
    "function": "chain",
    "variable": "street_wip",
    "functions": [
        {
            "function": "postfixed_street",
            "field": "Prop_Addr"
        },
        {
            "function": "remove_postfix",
            "field": "street_wip",
            "field_to_remove": "Prop_Addr_Unit"
        }
    ]
}

This used to work fine with https://github.com/openaddresses/machine but once that we deprecated in favour of https://github.com/openaddresses/batch-machine a breaking change was introduced which meant this example would now fail as evidenced by the errors in openaddresses/openaddresses#5496

After a few months of not understanding the issue, I took another look comparing the old conform https://github.com/openaddresses/machine/blob/05db17d8492b3d8f4064f0f5b0ca9c68041c535a/openaddr/conform.py with the new one https://github.com/openaddresses/batch-machine/blob/8bb6ecfb20beec56913ae7ae267cf5489c9c35ad/openaddr/conform.py this revealed an issue at

def row_fxn_chain(sc, row, key, fxn):
functions = fxn["functions"]
var = fxn.get("variable")
original_key = key
if var and var.upper().lstrip('OA:') not in sc.SCHEMA and var not in row:
row['oa:' + var] = u''
key = var
for func in functions:
row = row_function(sc, row, key, func)
if row.get('oa:' + key):
row[key] = row['oa:' + key]
row['oa:{}'.format(original_key.lower())] = row['oa:{}'.format(key)]
return row

The code here now creates these intermediate variables with the oa: prefix, once I added this prefix to the _wip variable fields at https://github.com/openaddresses/openaddresses/pull/5828/files#diff-8441ea2e82820b7decef018f2d606c8a543ad86411a2c49aedab254a0f72f848 it worked.

So the question for you @ingalls is, is this either

a) a big in batch-machine that we can fix, or
b) a change in how these conform files are processed and we should update the documentation and existing sources which broke

I'm happy to help out, but I need to first understand some of the background and motivation of this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant