v2.1.19

This release adds two new primitives to the core:

• join
  • This allows converting an array to a string, using a specified deliminator.
• replace
  • This allows replacing the contents of a string with a regular-expression based search and a literal replacement value.

Both these primitives have full test-coverage, and seem useful. Beyond the addition of these primitives minor changes were made to resolve warnings/failures detected by an updated release of the golangci-lint tool.

v2.1.18

This release mostly features only internal cleanups:

• We've changed to using the fuzz-testing which is available in the golang 1.18 release, rather than using the external go-fuzz utility.
  • This was implemented in #178.
• The linter we test our pull-requests with was updated to use golangci-lint, rather than the previous selection of tools.
  • This was implemented in #179

There were two new featuresadded:

• The implementation of a JSON interface to our objects, for those who embed our library
  • Reported in #173, and implemented in #174.
• Support for chained/nested if and else if statements.
  • Reported in #171, and implemented in #172.

v2.1.17

v2.1.17

This release fixes a panic when reflection was used against a JSON object which contained a null value for a key. This was reported in #165, and resolved in #170.

After resolving this panic I was reminded we have the setup for running fuzz-testing so I ran the fuzzer for several hours and resolved the few parser issues it caught. (These new issues were all related to the recently added support for hash-access via the . operator.)

Although fuzzing is always somewhat random (by design!) I've been running it for several hours without any additional problems reported which is a reassuring thing:

2021/05/15 12:20:19 workers: 1, corpus: 1885 (1m49s ago), crashers: 0, restarts: 1/9999, execs: 60544462 (4680/sec), cover: 1898, uptime: 3h35m
2021/05/15 12:20:22 workers: 1, corpus: 1885 (1m52s ago), crashers: 0, restarts: 1/10000, execs: 60560974 (4680/sec), cover: 1898, uptime: 3h35m
2021/05/15 12:20:25 workers: 1, corpus: 1885 (1m55s ago), crashers: 0, restarts: 1/9999, execs: 60577329 (4681/sec), cover: 1898, uptime: 3h35m

v2.1.16

v2.1.16

This release concentrates on stability and error-recovery:

• The same instance of the evalfilter object can now be used across goroutines
  • Reported in #166, implemented in #169.
• Any code, be it internal to the evaluator, or external in your built-in functions, which calls panic will now have that reported.
  • This was reported in #167, and resolved in #168.

These changes, combined, should improve our robustness. This release also features a new built-in function panic which can be used to test the recovery, or terminate your scripts with a specific error-value.

v2.1.15

v2.1.15

This release contains a small number of changes, as well as the usual internal cleanups and reorganizations.

The following new primitives were added, and documented:

• between
• max
• min

We can now access nested hash-values, which is particularly useful when dealing with JSON objects. This can be achieved either via foo.bar.baz, or via the long-form which was previously supported (foo["bar"]["baz"]).

Finally it is now possible to use the underscore (_) character in identifiers, which is necessary if you're working upon fields with names containing that character.

Our test-coverage remains high (100% for the newly added features), and our reflection support was unified to ensure that the different kinds of input (struct vs. object) are both processed in the same manner. Some of our longer tests were broken down into smaller functions, to remove complexity but there were no real functional changes.

v2.1.14

v2.1.14

This release features several new features and improvements, as well as the usual collection of bugfixes, and improvements to our internal codebase.

Once more testing has been a big focus, we now have ~99% test-coverage of our virtual-machine, and significantly improved coverage of other areas in our package.

New features include:

• Support for hash-objects, in addition to the existing datatypes we support.
  • Hashes are defined as you would expect:
    • a = { "Name": "Steve", "Alive": true }
  • Hashes can be indexed, and iterated over
    • printf("%s\n", a["Name"]);
    • foreach key, value in a { printf("Key %s Value %s\n", key, value ); }
  • Example script available at _examples/scripts/hashes.script
• Regular expressions were promoted to first-class objects.
  • Which means when dumping bytecode they are identified as regular-expressions rather than strings.
  • There are no actual changes to functionality though.
• It is no longer a fatal error if a script doesn't end with a return statement.
  • The caller will receive &object.Void{} in that case.
  • If you're running as a filter you can decide what you want that to mean.
• Errors from the parser will now contain more accurate line/column numbers.
  • Several new parser-errors will be identified more cleanly.
    • Only the initial error will be reported by default, because later errors are often bogus.
• We gained support for case/switch statements.
  • These are similar to C-like switch expressions, but we use blocks to avoid any potential bugs with fall-through behaviour.
  • Example script available at _examples/scripts/switch.script

v2.1.13

v2.1.13

The major new feature in this release is the ability to create functions inside the scripts that the engine executes. For example it is now possible to write:

   // Sum the values in the provided array
   function sum(array) {
      local total;
      total = 0;

      foreach val in array {
         total += val;
      }

      return( total );
   }

   printf("Sum of 1-100 is %d\n", sum(1..100));
   return sum(1..100) == 5050;

In this example you can see several new things demonstrated:

• The definition of a new function, complete with a named argument, via the function .. block.
  • Note that functions don't need to be defined before they're used.
• The use of the local keyword. This binds the variable to the local scope.
  • Without that you'd have a new global variable named total.
• The use of the new mutator operation +=.

The addition of functions required many small changes throughout the codebase, and as a result of that our test-coverage was improved significantly to give confidence nothing was broken in the process. (There is still room for additional test-coverage, and that will be worked upon going forward.)

As minor features mutator-operations were added (+=, -=, *= and /=), as demonstrated above, and the keyword for was added as a synonym for while.

A bugfix was made to the way that evalfilter-arrays are exported to Golang values, bounds-checking was added to the runtime virtual machine to guard against panics when executing untrusted bytecode. (Executing user-provided scripts is safe, as a result of the sanity-checks and error-detection in the lexer, parser, and evaluator. It is never expected that you'll process untrusted bytecode - but adding sanity-checks is still a useful thing to do.)

Finally the new built-in function getenv was added, allowing scripts to read the contents of environmental variables.

As a consequence of supporting user-defined functions the output of the bytecode subcommand in the standalone evalfilter binary was altered - The standalone executable also gained support for integrated TAB-completion (for bash).

v2.1.12

v2.1.12

This release makes some minor changes and improvements to our library, but nothing hugely significant:

• We support the use of a context.Context to implement timeouts when executing scripts.
  • Reported in #127, implemented in #128.
• The handling of numbers as conditionals was improved; negative numbers are not "true".
  • Reported in #125, implemented in #126.
• Handling of variable-scoping was improved for our foreach iteration support.
  • Reported in #123, implemented in #124.

Finally we've included a simple webassembly demo, which involves compiling our library to WASM and executing it in a browser. Find it beneath _examples/wasm.

v2.1.11

v2.1.11

This release contains a number of changes to the scripting-language available to consumers of the library, largely as a result of my use of the evalfilter-library in a simple application for scripting the manipulation of Google Gmail labels.

The language as a whole benefits from increasing usage, as it points out shortcomings that might otherwise not be apparent. In my case this largely revolved around the pain of iterating over arrays manually with a for-loop.

A brief summary of changes in this release:

• Implement the ability to iterate over arrays via the new foreach primitive.
  • Reported in #103, implemented in #105.
• Allow creating arrays of integers via 1..10
  • Reported in #107, implemented in #108
• Allow external functions added to the scripting environment to return "void".
  • This means that no return value will be available to the caller.
  • This fixes #86 and #106 - and was implemented in #109.
• Implemented some new built-in functions:
  • sort() - reported in #104, implemented in #110.
  • reverse() - reported in #104, implemented in #110.
  • split() - reported in #111, implemented in #122.
• Add support for the postfix ++ and -- operations.
  • Reported in #114, implemented in #115.
• Allow iterating over the characters in a string, just like array iteration.
  • Reported in #113, implemented in #117.
• Fixed bug: The optimizer could cause infinite loops when scripts didn't finish with a return statement
  • Reported in #119, fixed in #120.
• Fixed bug: String-indexing was broken for multibyte characters
  • Reported in #118, fixed in #121
• Fixed bug: Our parser was updated to avoid crashing on bogus input.
  • These were discovered as a result of fuzz-testing
    • Sample input which caused crashes included 0+foreach+0(), and !foreach%0().

v2.1.10

v2.1.10

Once again this release features on minor improvements to the code structure, layout, and internal organization.

There have been new features in the release though, most notably:

• The introduction of the ternary statement #99:
  • e.g. "field = Subject ? Subject : Title"
• The reorganisation of the examples, beneath _examples/ handled in #102.
  • This now contains a couple of example scripts, as well as the examples showing how the evaluation-engine / scripting language can be embedded in your host application.
• The regular expression support was improved in #98
  • It is now possible to escape characters via \, and only valid flags are accepted
  • (We use /i for ignoring-case, and /m for multi-line matches.)
• Our opcode implementation was improved a little, via #100
• We gained two new built-in functions sprintf and printf, via #101.