Evaluation Schema + a necessary revamp #118

Udayraj123 · 2023-01-03T19:48:35Z

Fixes #89
This PR attempts to provide a generic solution for evaluation.

This is PR#1 of a chained PR.
Next

Udayraj123 · 2023-01-03T19:49:09Z

@Rohan-G you can start having a look in the main files. Starting with the schema. I'll push working changes soon.

samples/community/Sandeep-1507/template.json

fix: refactor; add sample evaluation.json fix: update gitignore and rename ignored folder fix: setup skeleton; refactor

…; wip pass instance

fix: refactor; connect evaluation schema fix: checked validation using evaluation schema

…ther; move constants/configs to better places; fix: refactor; move instance ops to core; minor changes refactor: extract tuning config from instance fix: add screeninfo for simple window info

fix: evaluation schema fixes

Udayraj123 · 2023-01-07T10:24:36Z

The changes include(wip list):

working evaluation code
support for 4+ types of marking schemes: default(+1, 0), negative, multi-weighted, streak-based
support for answer_key_omr.csv
(foundation) support for answer_key_omr.jpg
add range operator support
add validations for marking scheme and omr response
picking up window_width (using screeninfo)
pickup evaluation.json recursively;
added schema for all 3 jsons
use format strings in all logs
improved error messages
reduce hardcoded configs
support config.json per instance
pickup config.json recursively;
test threshold jump value
upgrade pip packages
load json error handling
separate out utils into ImageInstanceOps, ImageUtils(static methods) and InteractionUtils(static methods)
add upsc mock; refactor; create evaluation-schema

…nd omr response

…refactor

fix: additional warning

fix: compress images

ayushjain2809 · 2023-02-11T17:57:07Z

src/logger.py

 FORMAT = "%(message)s"

+# TODO: set logging level from config.json dynamically


Env variable would be better.

will add in a later PR

ayushjain2809 · 2023-02-11T18:50:14Z

src/evaluation.py

+import ast
+import os
+import re
+from copy import deepcopy
+from fractions import Fraction
+
+import cv2
+import pandas as pd
+from rich.table import Table
+
+from src.logger import console, logger
+from src.schemas.evaluation_schema import (
+    BONUS_SECTION_PREFIX,
+    DEFAULT_SECTION_KEY,
+    MARKING_VERDICT_TYPES,
+    QUESTION_STRING_REGEX_GROUPS,
+)
+from src.utils.parsing import get_concatenated_response, open_evaluation_with_validation


Re-org imports

It seems like isort hook and black standards is keeping the same order as earlier.
It should sort it alphabetically, but oddly it's not.
Moving import cv2 to line 2 again moves it back to line 7

ayushjain2809 · 2023-02-11T18:52:17Z

src/evaluation.py

+def parse_float_or_fraction(result):
+    if type(result) == str and "/" in result:
+        result = float(Fraction(result))
+    else:
+        result = float(result)
+    return result


move to appropriate place (maybe utils?)

addressing this in PR#3 now

ayushjain2809 · 2023-02-11T19:04:02Z

src/evaluation.py

@@ -0,0 +1,598 @@
+import ast


A couple of articles that might be useful for refactoring this file:
https://dzone.com/articles/the-stepdown-rule
https://dzone.com/articles/slap-your-methods-and-dont-make-me-think

Will do via PR#3

* fix: load answer key from image with working samples * fix: pytests set up with snapshots for all samples * fix: run sample1 on pre-commit; run all tests on pre-push fix: add default_install_hook_types fix: add explicit stages * fix: update snapshots * fix: bug fixes * [Feature] Simplify template jsons (#127) * fix: update snapshots feat: simplify template schema and block logic; consume it; use fields terminology; update all samples fix: bug fixes * fix: refactor template.py; minor template json cleanup * fix: changes after updating wiki * feat: setup tests for template validations; fixed few bugs fix: renaming fix: test * fix: refactor fix: refactor tests structure and add utils * fix: minor fixes * fix: reorder imports; rename question -> field * fix: review changes * fix: remove streak logic

Udayraj123 added enhancement New feature or request Discussion advanced refactor labels Jan 3, 2023

Udayraj123 requested a review from deepakgouda January 3, 2023 19:48

Udayraj123 force-pushed the feature/evalution-refactor branch from 8632f70 to df4462b Compare January 3, 2023 20:11

deepakgouda reviewed Jan 3, 2023

View reviewed changes

samples/community/Sandeep-1507/template.json Show resolved Hide resolved

Udayraj123 force-pushed the feature/evalution-refactor branch 4 times, most recently from 0bee168 to 603a4d3 Compare January 7, 2023 09:56

Udayraj123 changed the title ~~Evaluation Schema + a small revamp~~ Evaluation Schema + a necessary revamp Jan 7, 2023

Udayraj123 added 8 commits January 7, 2023 15:53

fix: add upsc mock; refactor; create evaluation-schema

8471190

fix: refactor; add sample evaluation.json fix: update gitignore and rename ignored folder fix: setup skeleton; refactor

fix: refactor; change defaults to .py; pickup config.json recursively…

a796fe6

…; wip pass instance

fix: update evaluation-schema and sample json wip

f41d128

fix: refactor; connect evaluation schema fix: checked validation using evaluation schema

refactor: separate static vs instance ImageUtils; breakfown files fur…

e74394e

…ther; move constants/configs to better places; fix: refactor; move instance ops to core; minor changes refactor: extract tuning config from instance fix: add screeninfo for simple window info

fix: load json error handling; dynamic config json working for sample5

c69b978

fix: use formatted strings at all places necessary

91fc506

fix: correct config schema

3e7450f

fix: working evaluation code

453680b

fix: evaluation schema fixes

Udayraj123 force-pushed the feature/evalution-refactor branch from 603a4d3 to 16c3c5c Compare January 7, 2023 10:23

Udayraj123 force-pushed the feature/evalution-refactor branch from 16c3c5c to 4070878 Compare January 7, 2023 18:55

Udayraj123 added 6 commits January 8, 2023 00:47

fix: add range operator support; add validations for marking scheme a…

f8b97c3

…nd omr response

fix: add a sample with answer_key.csv

aa69b50

fix: minor fixes

d16b647

fix: foundation for answer_key_image_path; refactoring; minor changes

9bd65dd

fix: remove unnecessary comments; fix minor comments

cb6817d

fix: further on foundation for csv_from_image feature; refactoring

f8a7d91

fix: stronger validations; conditionality in streak scheme; joinpath …

8169a81

…refactor

Udayraj123 force-pushed the feature/evalution-refactor branch from 4070878 to 8169a81 Compare January 12, 2023 19:48

Udayraj123 added 3 commits January 13, 2023 16:56

fix: support multi-correct and basic multi-weighted; update explanations

54383b3

fix: csv load custom key support

b715c8f

fix: additional warning

fix: add evaluation configs in some samples

ea75858

Udayraj123 force-pushed the feature/evalution-refactor branch 3 times, most recently from 6558b86 to 278d482 Compare January 23, 2023 22:42

Udayraj123 mentioned this pull request Jan 24, 2023

[Feature] made a classifier based on information retrieved from QR on the omr sheets #116

Merged

fix: load answer key from image with working samples

3692500

fix: compress images

Udayraj123 force-pushed the feature/evalution-refactor branch from 278d482 to 3692500 Compare January 25, 2023 23:35

Udayraj123 and others added 3 commits January 26, 2023 05:34

fix: compress images

fb15654

fix: update samples; add readme for template shift sample

d308609

Merge branch 'master' into feature/evalution-refactor

1f54a6e

Udayraj123 marked this pull request as ready for review February 1, 2023 13:49

fix: reduce images; minor changes

b2f394b

Udayraj123 mentioned this pull request Feb 6, 2023

[Feature] setup pytest on existing samples #124

Merged

deepakgouda approved these changes Feb 10, 2023

View reviewed changes

ayushjain2809 reviewed Feb 11, 2023

View reviewed changes

Udayraj123 merged commit 080b360 into master Feb 12, 2023

Udayraj123 deleted the feature/evalution-refactor branch February 12, 2023 06:58

Udayraj123 mentioned this pull request Feb 19, 2023

[Enhancement] Extra steps and error messages in schema validation #65

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Evaluation Schema + a necessary revamp #118

Evaluation Schema + a necessary revamp #118

Udayraj123 commented Jan 3, 2023 •

edited

Loading

Udayraj123 commented Jan 3, 2023

Udayraj123 commented Jan 7, 2023 •

edited

Loading

ayushjain2809 Feb 11, 2023

Udayraj123 Feb 11, 2023

ayushjain2809 Feb 11, 2023

Udayraj123 Feb 12, 2023 •

edited

Loading

ayushjain2809 Feb 11, 2023

Udayraj123 Feb 11, 2023

ayushjain2809 Feb 11, 2023

Udayraj123 Feb 11, 2023

		FORMAT = "%(message)s"

		# TODO: set logging level from config.json dynamically

Evaluation Schema + a necessary revamp #118

Evaluation Schema + a necessary revamp #118

Conversation

Udayraj123 commented Jan 3, 2023 • edited Loading

Udayraj123 commented Jan 3, 2023

Udayraj123 commented Jan 7, 2023 • edited Loading

ayushjain2809 Feb 11, 2023

Choose a reason for hiding this comment

Udayraj123 Feb 11, 2023

Choose a reason for hiding this comment

ayushjain2809 Feb 11, 2023

Choose a reason for hiding this comment

Udayraj123 Feb 12, 2023 • edited Loading

Choose a reason for hiding this comment

ayushjain2809 Feb 11, 2023

Choose a reason for hiding this comment

Udayraj123 Feb 11, 2023

Choose a reason for hiding this comment

ayushjain2809 Feb 11, 2023

Choose a reason for hiding this comment

Udayraj123 Feb 11, 2023

Choose a reason for hiding this comment

Udayraj123 commented Jan 3, 2023 •

edited

Loading

Udayraj123 commented Jan 7, 2023 •

edited

Loading

Udayraj123 Feb 12, 2023 •

edited

Loading