Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve readability of errors returned by definition validator #124

Open
ppeble opened this issue Feb 11, 2019 · 0 comments
Open

Improve readability of errors returned by definition validator #124

ppeble opened this issue Feb 11, 2019 · 0 comments

Comments

@ppeble
Copy link
Member

ppeble commented Feb 11, 2019

Brief Synopsis of the feature request / desired behavior change

The make validate command should always validate all YAML files rather than immediately raising an error on the first problem found and stopping execution.

Additionally the error output displayed to the user should be updated to display the following:

  • a summary of the total definition YAML files checked and the number that passed or failed
  • an easily readable list of the errors that are grouped by definition file and clearly specify each problem found and what must be done to fix it

Description of current behavior

A user can run make validate in the repository root and it will iterate over every definition yaml file and ensure that each one contains no syntax issues.

The current behavior is that the entire process halts once the first error is found. The message displayed to the user provides an explanation and usually also the record in question rather than the specific value that caused the error.

As an example I added an invalid value to the ar.yaml on my local machine:

~/code/ppeble/definitions(master ✗) make validate
bundle exec ruby lib/validation/run.rb
Failed on file '/Users/phil/Code/ppeble/definitions/lib/validation/../../ar.yaml', error: All months must be an integer, received: {0=>[{"name"=>"Viernes Santo", "regions"=>["ar"], "function"=>"easter(year)", "function_modifier"=>-2}, {"name"=>"Carnaval Lunes", "regions"=>["ar"], "function"=>"easter(year)", "function_modifier"=>-48}, {"name"=>"Carnaval Martes", "regions"=>["ar"], "function"=>"easter(year)", "function_modifier"=>-47}], 1=>[{"name"=>"Año Nuevo", "regions"=>["ar"], "mday"=>1}], 3=>[{"name"=>"Día Nacional de la Memoria por la Verdad y la Justicia", "regions"=>["ar"], "mday"=>24}], 4=>[{"name"=>"Día del Veterano y de los Caídos en la Guerra de Malvinas", "regions"=>["ar"], "mday"=>2}], 5=>[{"name"=>"Día del Trabajador", "regions"=>["ar"], "mday"=>1}, {"name"=>"Día de la Revolución de Mayo", "regions"=>["ar"], "mday"=>25}], 6=>[{"name"=>"Día de la Bandera", "regions"=>["ar"], "mday"=>20}], 7=>[{"name"=>"Feriado puente turístico", "regions"=>["ar"], "mday"=>8}, {"name"=>"Día de la Independencia", "regions"=>["ar"], "mday"=>9}], "xyz"=>[{"name"=>"Paso a la Inmortalidad del General José de San Martín", "regions"=>["ar"], "mday"=>15}], 10=>[{"name"=>"Día del Respeto a la Diversidad Cultural", "regions"=>["ar"], "mday"=>12}], 11=>[{"name"=>"Día de la Soberanía Nacional", "regions"=>["ar"], "mday"=>20}], 12=>[{"name"=>"Inmaculada Concepción de María", "regions"=>["ar"], "mday"=>8}, {"name"=>"Feriado puente turístico", "regions"=>["ar"], "mday"=>9}, {"name"=>"Navidad", "regions"=>["ar"], "mday"=>25}]}

The above error was caused because I changed month 8 to xyz. This is very difficult to spot without digging through each part of the part of the message labeled 'received', which contains the entirety of the months keyword.

In addition, since the process halts on the first error that it finds this means that no other parts of the ar.yaml file are checked. If there are any validation errors present in methods or tests then a user will only see them after they have fixed the initial error and rerun make validate.

This also means that since execution is immediately stopped it is possible that other definition files are not checked. If there are errors present in other files a user will only find them after fixing all of ar.yaml and rerunning make validate.

Description of desired behavior

The make validate command should always check all found definition YAML files on each execution. Any validation errors that are found should be gathered and saved until the end of the run. Upon completing the validation of all definition files the process should display a summary and a grouped list of human readable errors with actionable information for the user.

Here is a potential example of the desired output based on the ar.yaml scenario from the section above:

~/code/ppeble/definitions(master ✗) make validate
bundle exec ruby lib/validation/run.rb

Found 70 definition YAML files to check! Starting validation checks ...
Validations complete!

Summary
---------
Total files checked: 70
Passed: 69
Failed: 1
Files containing validation errors: 'ar.yaml'

Error details by region definition:

----
Region: ar
File: ar.yaml
Total errors found: 1
Details:
  Parent key: 'months'
  Message: Expected month 'xyz' to be an integer
  Value: {'xyz' => [{'name' => 'Paso a la Inmortalidad del General José de San Martín', 'regions' => ['ar'], 'mday' => 15}]
----

(the above is only a suggestion, the final product can be tweaked for readability as necessary)

This would allow users to see all problems at once in a clear format rather than forcing them to rerun the validation process over and over.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant