Custom coerce types/functions

[dmikam]

Would be great to be able to do something like:
{ "date": { "format": "date", "coerce": function(value){ return moment(value,'DD/MM/YYYY') } } }

There would be infinite possibilities and usage cases.

Anyway, thanks - great module!

[epoberezkin]

You can do all that and more with custom keywords.

[epoberezkin]

See #147 for an example

[ksmithut]

@epoberezkin I know this has been closed for a while, but looking through the example in that issue, I couldn't find how to do custom coercion in that issue or the custom keywords docs. I was trying to figure it out using the "inline" option, but everything seemed to only have the use case of validating data, and not adding custom coercions.

My use case is that I'm trying to get ajv to handle parsing/validating/coercing data from a swagger schema. One of the pieces of swagger that I'm trying to support is the collectionFormat keyword. So when the value comes in, it's a string with some kind of delimiter. I'd like to take that string, and parse it into an array before validation. Is this supported with custom keywords? From your statement:

You can do all that and more with custom keywords.

I suspect that it can, but I can't find any documentation on how to do this.

[epoberezkin]

Maybe example would help in docs, but the idea here is explained in docs you mention - the validation function in all keyword types but "macro" (i.e. "validate", "compile" and "inline") receives "parent data object" and "the property name in the parent data object" as parameters. That allows custom keywords to modify the data being validated in any way they want, e.g.

const collectionFormat = {
  csv: (data, dataPath, parentData, parentDataProperty, rootData) => {
    var arr = [];
    // process data to populate array
    if (success) parentData[parentDataProperty] = arr;
    return success; // true or false - validation result
  },
  // ... other list types
};

ajv.addKeyword('collectionFormat', {
  type: 'string',
  compile: (schema) => collectionFormat[schema],
  metaSchema: {
    enum: ['csv' /* , ... */]
  }
});

The only caveat at the moment is that custom keywords will be validated AFTER standard keywords in the same schema object, so if you expect to define validation rules for the CHANGED data these rules cannot be in the same object, they should be in the next subschema of "allOf" keyword:

{
  allOf: [
    {
      // modify data
      collectionFormat: 'csv'
    },
    {
      //validate data
      type: 'array'
      // etc.
    }
  ]
}

I thought several times about allowing to define custom keywords that execute BEFORE standard, but I decided against it because it implies the need to manage order implicitly and there is no way of telling the execution order by looking at the schema. I think it is better that in cases the order matters you MUST use allOf keyword, otherwise the keywords should be independent of execution order.

What may make sense though is to add some extra requirement to the keyword definition that it MUST NOT be used in the same schema with other validation keywords to enforce that it is used in its own schema object, which can be useful in cases like above.

In any case, at the moment it is something you just have to remember.

[ksmithut]

Thank you! That makes much more sense now. I'm fine with doing it that way, with the allOf. I'm transforming the swagger schema into an ajv schema, so transforming it in this way would work just fine.

[epoberezkin]

Ok. Actually if you define type in keyword definition as above, there it no need to check type in validation function - it won't be called for anything but a string. Also if the keyword value is valid according to metaSchema the schema compilation will simply fail, so no need to check it as well.

[ksmithut]

@epoberezkin Aplogies, and if this belongs in a different place (like stack overflow or whatever), let me know. I can't get the following to do what I'd like it to:

'use strict'

const Ajv = require('ajv')

const ajv = new Ajv({
  allErrors: true,
  removeAdditional: true,
  useDefaults: true,
  coerceTypes: 'array',
  async: true,
})
ajv.addKeyword('collectionFormat', {
  type: 'string',
  async: true,
  compile() {
    return (data, dataPath, parentData, parentDataProperty) => {
      parentData[parentDataProperty] = data.split(',')
      console.log('split')
      return Promise.resolve(true)
    }
  },
  metaSchema: {
    enum: [
      'csv',
    ],
  },
})

const validate = ajv.compile({
  $async: true,
  type: 'object',
  properties: {
    foo: {
      allOf: [
        { collectionFormat: 'csv' },
        {
          type: 'array',
          items: { type: 'string' },
        },
      ],
    },
  },
  additionalProperties: false,
})

const obj = {
  foo: 'foo,bar,hello',
}

validate(obj)
  .then(() => {
    console.log('final', obj)
  })
  .catch(console.log)

The output of that is:

final { foo: [ 'foo,bar,hello' ] }

The expected output is:

final { foo: [ 'foo', 'bar', 'hello' ] }

It seems like the change I make to parentData doesn't actually mutate the original data. If I change the coerceTypes to true, it fails validation because when it gets to the type: 'array' validation, it's still the original string. Am I doing something wrong? Is it because of the async stuff?

Again, if there is a better place for these questions, let me know.

[epoberezkin]

@ksmithut That's a really interesting discovery. It worked for me in similar cases, but I think I understand the difference now. So thank you.

In the beginning I thought that async is messing things around, but it seems like while you have the same data to validate all keywords re-use the data variable for validation (that's assigned in properties keyword) rather than using parent object on every iteration. So while underlying object gets updated the next allOf subschema doesn't see it yet.

It is something that would be good to improve, although it may be tricky... Probably there should be a property in custom keywords definition to define them as "mutating" - this property would cause the variable to be re-assigned AFTER keyword function is executed (as there is no way you can update the variable in the calling function from keyword function). Another improvement to custom keywords I keep thinking about that keywords that only do mutation or some logging and always return true (or false) as a validation result can have "valid: true/false" property in keyword definition that would simplify generated code because the result is known in advance (like my log keyword in examples below).

At the moment the only workarounds I can come up with are:

• to split into a different properties keywords, but it's quite ugly - see a working example here https://runkit.com/esp/586eba75290f430013c4b6fd . It works because the second properties/foo re-assigns the variable from the parent object
• use inline keyword and update both the parent object and the variable - https://runkit.com/esp/586ec1210b602700145eb23f . It allows for your schema that I thought would work in all cases, but has more complex keyword definition.

I also added "log" keyword just to understand what's going on.

Thank you for the patience.

[ksmithut]

No, thank you for your patience. You owe me nothing, and you're work has already provided me with so much functionality. Do you have some sort of gittips or other kind of support "tip jar"? I'd love to contribute

[epoberezkin]

Lol. Thank you. Contributions to this issue are welcome: #100 (comment) There is a pledge from @crkepler to match them :)

[ksmithut]

Sweet, just pledged :) BTW, this pledge isn't for this mutation data stuff. As far as I'm concerned, you've already earned it, so don't feel beholden to improve upon what you've already done. I'm happy with the solutions you provided

[epoberezkin]

It's very generous, thank you :) This bounty is for a particular big issue: to support custom error message generation.

Mutation I'm going to improve anyway, it's quite simple actually. I just always assumed it works in all cases, but it turns out I just somehow managed to use it in the way it works...

[epoberezkin]

A bit cleaner inline keyword with doT template: https://runkit.com/esp/586ec1210b602700145eb23f

[ksmithut]

Awesome! Thanks!

[epoberezkin]

@ksmithut now with option modifying: true in custom keyword definition the keywords that change data work as expected.

[ksmithut]

@epoberezkin Nice!

[mathieucaroff]

I drop this here.

import Ajv from "ajv";

describe("ajv keywords can modify the data", () => {
  const applyStore = {
    Date: ({ data }) => new Date(data),
    logParent: ({ data, parentData }) => {
      console.log({ parentData });
      return data;
    },
    log: ({ data }) => {
      console.log({ data });
      return data;
    }
  };

  const ajv = new Ajv();
  ajv.addKeyword("apply", {
    modifying: true,
    compile: (schema, parentSchema, it) => {
      return (data, dataPath, parentData, parentKey) => {
        parentData[parentKey] = applyStore[schema]({
          schema,
          parentSchema,
          it,

          data,
          dataPath,
          parentData,
          parentKey
        });
        return true;
      };
    },
    metaSchema: {
      enum: Object.keys(applyStore)
    }
  });

  const validate = ajv.compile({
    type: "object",
    properties: {
      time: {
        allOf: [
          { type: "string" },
          { format: "date-time" },
          { apply: "log" },
          { apply: "Date" },
          { apply: "log" }
        ]
      }
    }
  });

  const obj = {
    time: "2019-05-24T07:02:28.678Z"
  };

  validate(obj);

  test("modification", () => {
    expect(obj.time).toBeInstanceOf(Date);
  });

  test("validity", () => {
    expect(validate.errors).toBe(null);
  });

  console.log({ obj });
});

https://codesandbox.io/s/hopeful-dust-5t2jr

[wmthor]

I think this can be documented somewhere as there is plenty of use-case for this (e.g. buffer, date).

There doesn't seem to be a clear and simple documented way to custom coerce data types. The custom keywords examples show how to perform validation but the steps to modify the data isn't so obvious. Above examples seem outdated, as it looks like the return value format has changed.

However, I've managed to figure it out and get it to work with the following:

const Ajv = require('ajv');

const ajv = new Ajv({ coerceTypes: true });

ajv.addKeyword({
  keyword: 'buffer',
  compile: (schema) => (value, obj) => {
    if (schema === 'hex') {
      obj.parentData[obj.parentDataProperty] = Buffer.from(value, 'hex');
    } else if (schema === 'base64') {
      // Decode here
    }

    // Pass validation
    return true;
  },
});

const data = {
  myData: '00ffffff',
};

const schema = {
  type: 'object',
  properties: {
    myData: { type: 'string', buffer: 'hex' },
  },
};

const validate = ajv.compile(schema);

validate(data);

console.log(data); // { myData: <Buffer 00 ff ff ff> }

[epoberezkin]

yes, you need some custom keyword for all non-standard types. You can also use typeof or instanceof keywords defined in ajv-keywords.