JS API to type-cast and handle all the match types supported in the DSL

[ptrthomas]

from discussion here: https://stackoverflow.com/a/62638847/143475

currently type conversion is possible only via the companions to def e.g. json xml etc: https://github.com/intuit/karate#type-conversion

sometimes there is a need to type-cast when within a JS block. also - a feature to auto-detect a string and type-cast it would be nice. proposals below (JS block, not Gherkin)

// not 100% sure that data1 can "remain" as XML after making it back to a feature
var data1 = karate.fromString(someString);
// this will follow the same rules as type-conversion keywords (see link above)
var data2 = karate.toType('json', anyObject);

[ptrthomas]

while we're at it - I've felt the need for a karate.typeOf(object) function as well, so maybe add that as well

[ptrthomas]

and karate.isList(object) plus karate.isMap(object)

[KostasKgr]

Thank you for raising this @ptrthomas !
I was expanding a bit my testing in a new branch, maybe you will find some of it of use.
https://github.com/KostasKgr/karate-issues/blob/java_json_interop_v2/src/test/java/examples/consumption/consumption.feature

I managed to get the minimum behavior I was looking for by parsing json in java, so that karate understood that to be a json. Not sure i can make any such workaround for xml based messages though. But if you see this coming to 0.9.6 it would be awesome.

By the way I noticed that match contains, and match with jsonpath cannot be used in javascript, I have 3 features exploring that.

I would love to contribute but I am still wrapping my head around how this all works :D

p.s. I am working on adding something close to a kafka dsl to karate, so that people that are not very technical could write end to end tests that involve both kafka and api calls, not sure if that will end up working, it has become more complicated than i hoped for :D

[ptrthomas]

I looked at the consumption.feature it is a little hard to understand, let me try later

match contains, and match with jsonpath

yes, this is somewhat deliberate, to prevent teams doing "too much" in javascript - but I'm, open to revising that as well. btw you can use the "short-cuts" so this will work

* def valid = match(foo, '#(^bar)');

working on adding something close to a kafka dsl to karate

you may get some ideas from this thread: #1191 (comment)

[ptrthomas]

we now have a way to do all the DSL match types in JS

if a variable needs to be set from JS, that is easy using karate.set() that can even take a JSON as the arg to set multiple key-values in one shot

EDIT: in the rare case that you need to set a variable as XML into the context from JS you can use setXml(name, value)

note that we can conditionally fail a test using karate.fail('some message')

[ptrthomas]

and implemented karate.fromString() and karate.fromObject() the twist here is they return a ScriptValue - which is packed with a whole bunch of helper methods and properties that evolved over time. anyway this API is intended for advanced users

@KostasKgr I think you should have all you need !

[KostasKgr]

Thank you @ptrthomas ! Sorry for not being responsive, looks to be everything that was discussed! I will try it out!

[KostasKgr]

Hello @ptrthomas , I tried develop commit 2f8765b, I think i built it correctly (I have testBigDecimalsInJson test failing both there and on 0.9.5, so i am assuming its something weird with being on windows)

I tried the karate.fromString() but I couldn't get it to work, also I am getting some very confusing behavior while trying to log things.

You can hopefully reproduce this at the first scenario here (used version 2.0.0 which was in develop branch): https://github.com/KostasKgr/karate-issues/blob/java_json_interop_v3/src/test/java/examples/consumption/consumption.feature#L5

So we start with this in java, and return it with another one in a List

        jsonTopic.add("{\n" +
                "    \"correlationId\": \"100\",\n" +
                "    \"text\": \"Hello World\",\n" +
                "    \"data\": {\n" +
                "        \"deeply\": {\n" +
                "            \"nested\": \"yes\",\n" +
                "            \"id\": \"1000\"\n" +
                "        }\n" +
                "    }\n" +
                "}");

The iterate the list in consume.js and covert using karate.fromString().
karate.log() of the result of karate.fromString() in consume.js logs:

21:38:50.085 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - Native object: [type: JSON, value: com.jayway.jsonpath.internal.JsonContext@5dbd0edf] 

Keep the value of this and print it, it still says jayway:

21:38:50.101 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - Evaluating: com.jayway.jsonpath.internal.JsonContext@5dbd0edf 

Pass the value to the filtering function, but now it appears like the json I was expecting to see if I karate.log it from within the filter:

21:38:50.108 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - In filter, message: {
  "correlationId": "100",
  "text": "Hello World",
  "data": {
    "deeply": {
      "nested": "yes",
      "id": "1000"
    }
  }
}

However if I print the correlation id, it is undefined, so the test fails, as I cant actually filter on correlation id:

21:38:50.118 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - Correlation id: undefined

Am I using this wrong?

Cheers

p.s. I didnt get time to test the new karate.match features

[ptrthomas]

@KostasKgr I think you missed that karate.fromString() does not return the final form, you need to do one more step to un-pack it. it returns a ScriptValue see my link above. the idea is that you can check what kind of object it is before getting it "as some type" - do re-try and let me know

[KostasKgr]

Hello @ptrthomas , I am already using .value on the scriptvalue object, is that what you mean?
https://github.com/KostasKgr/karate-issues/blob/java_json_interop_v3/src/test/java/consume.js#L29

21:38:50.085 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - Native object: [type: JSON, value: com.jayway.jsonpath.internal.JsonContext@5dbd0edf] 
...
21:38:50.101 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - Evaluating: com.jayway.jsonpath.internal.JsonContext@5dbd0edf 

However this did not work in a javascript function.

Could it be that match autoconverts left value to a native type on the fly in your examples in #1202 (comment) ?

I can't seem to get that behaviour inside of a javascript function,

[ptrthomas]

@KostasKgr use ScriptValue.asMap for JSON and asList for JSON arrays when within a JS block. let me know:

Scenario: parsing json, xml or string within a js block    
    * eval
    """
    var temp = karate.fromString('{ "foo": "bar" }');
    if (!temp.json) karate.fail('expected json');
    var val = temp.asMap;
    var res = karate.match(val, { foo: 'bar' });
    if (!res.pass) karate.fail(res.message);
    """

[KostasKgr]

Hello @ptrthomas , I will provide feedback as soon as I try it.

Not sure I understood when to use asMap and asList, as the code will not have knowledge of if the json is a json whose root element is a list, or if it's root element is a map.

[KostasKgr]

I added both elements with a list root and a map root in my topic. asMap worked before I added list elements in the topic, afterwards I get an exception such as this:

I have updated the branch mentioned above to have this code.

15:00:05.264 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - Original record string:  [
    {"order_id": 5, "name": "bicycle"},
    {"order_id": 6, "name": "car"}
] 
15:00:05.265 [ForkJoinPool-1-worker-3] INFO  com.intuit.karate - Native object: [type: JSON, value: com.jayway.jsonpath.internal.JsonContext@6a3da357] 
15:00:05.266 [ForkJoinPool-1-worker-3] ERROR com.intuit.karate - feature call failed: classpath:consume.feature
arg: {"topic":"json.topic","filter":"#function"}
consume.feature:6 - evaluation (js) failed: consume_function(topic, filter, kafka_message_format), net.minidev.json.JSONArray incompatible with java.util.Map

Is there no uniform api that determines the correct method to call? It seems to work in feature files from your examples :(

[ptrthomas]

@KostasKgr can you please paste a self-contained snippet here with the problem, my comment above would have given you an example

[KostasKgr]

Ofcourse! I will try to make one such as the above. My main question though was that I do not know what jsons will be arriving, so I cannot decide to use asMap or asList

[ptrthomas]

@KostasKgr the answer is you should use mapLike or listLike to make that decision, see my code examples above

feel free to suggest any better approaches you see

[KostasKgr]

Oh, ok! I will try it, i only saw the fromString example and missed the mapLike and listLike mentioned in the fromObject example!

Cheers!

[KostasKgr]

Managed to get things working! Thank you for your explanations!
Seems that if I use match then no asMap or asList is needed, but they are needed if I want to invoke a custom javascript filter.

All of the below pass with latest develop, many thanks, I'll have to think how to incorporate it with the rest of our code once it is in a release candidate

Feature: Matching and filtering in javascript

  Scenario Outline: Match native types in js: <type>
    * eval
    """
    var temp = karate.fromString(<data>);
    karate.log(temp);
    karate.log(temp.value);
    var val = temp.value;
    // Match handles type conversions, so no asMap or asList needed
    var res = karate.match(<match>);
    if (!res.pass) karate.fail(res.message);
    """

    Examples:
      | type      | data                                                        | match                                             |
      | json map  | '{ "foo": "bar", "nested": { "deeply": "here", "id": 5 } }' | "val contains deep { nested: { deeply: 'here' }}" |
      | json list | '[{"type": "car"}, {"type": "bicycle"}]'                    | "val contains [{type: 'car'}]"                    |
      | string    | 'hello world'                                               | "val contains 'hello'"                            |
      | xml       | '<greeting language="en">Hello World</greeting>'            | "val /greeting/@language == 'en'"                 |


  Scenario Outline: Filter native types in js: <type>
    * eval
    """
    var temp = karate.fromString(<data>);
    karate.log(temp);
    karate.log(temp.value);
    var val = temp.value;
    if(temp.json) {
      val = temp.mapLike ? temp.asMap : temp.asList;
    }
    var filter = function(data) { <filter> }
    var res = filter(val)
    if (!res) karate.fail("Filter did not match");
    """

    Examples:
      | type      | data                                                        | filter                                                    |
      | json map  | '{ "foo": "bar", "nested": { "deeply": "here", "id": 5 } }' | return data.nested.id == 5                                |
      | json list | '[{"type": "car"}, {"type": "bicycle"}]'                    | return data[0].type == "car"                              |
      | string    | 'hello world'                                               | return data.contains("hello")                             |
      | xml       | '<greeting language="en">Hello World</greeting>'            | return karate.xmlPath(data, '/greeting/@language') == 'en' |

[Sdaas]

Hi Kostas

I had put together something simple a while ago to test Kafka from Karate. Pls see if https://github.com/Sdaas/karate-kafka helps

Daas

p.s. I am working on adding something close to a kafka dsl to karate, so that people that are not very technical could write end to end tests that involve both kafka and api calls, not sure if that will end up working, it has become more complicated than i hoped for :D

[ptrthomas]

0.9.6 released