Skip to content

Implementing new Judge Agent

Jump to bottom
Jacek Spólnik edited this page Dec 30, 2016 · 1 revision

Introduction

JAlgoArena is easily extensible thanks to it's design. You can easily develop new judge agent, supporting new programming language or just doing the judgement in a different way.

API

There is couple of requirements which has to be met - so the judge agent can be easily plugged-in into existing architecture:

  • it has to expose REST API which has enabled CORS
  • they all should return JSON objects
  • they have to follow pre-specified schema for JSON objects

/problems/{id}/submit

  • POST
  • Accepts - text/plain
  • Produces - application/json
  • this is main endpoint which serves UI to pass user source code for judgement. It assumes that you can recognize languages you support and handle them
  • it produces JudgeResult json object, which is result of judgement. See below the schema

JudgeResult Schema

JSON object with below fields:

  • status_code - (string) status of the judgement result. It's enumeration of possible states, for details see below. It's required field which has to be always filled.
  • error_message - (string) error message in case of runtime or compile error. Value is ignored if no error.
  • elapsed_time - (double) elapsed time, calculated for processing all test cases in milliseconds. Value is ignored if error appears.
  • consumed_memory - (int) consumed memory, calculated for processing all test cases in bytes. Value is ignored if error appears.
  • testcase_results -(bool[]) array of Boolean values, indicating by true the passed test case and by false the failure. If there is error - the field can be skept.

Example of accepted result:

{
   "status_code": "ACCEPTED",
   "error_message": "",
   "elapsed_time": 12.24235,
   "consumed_memory": 0,
   "testcase_results": [true, true, true, true, true, true]
}

Example of compile error result:

{
   "status_code": "COMPILE_ERROR",
   "error_message": "Line:11: error: missing return statement\n    }\n    ^\n",
   "elapsed_time": 0.0,
   "consumed_memory": 0
}

Example of runtime error result:

{
   "status_code": "RUNTIME_ERROR",
   "error_message": "java.lang.RuntimeException: Example error",
   "elapsed_time": -1.0,
   "consumed_memory": -1
}

Example of wrong answer result:

{
   "status_code": "WRONG_ANSWER",
   "elapsed_time": -1.0,
   "consumed_memory": -1,
   "testcase_results": [false, false, false, false, true, false, true]
}

Example of time limit exceeded result:

{
   "status_code": "TIME_LIMIT_EXCEEDED",
   "elapsed_time": -1.0,
   "consumed_memory": -1
}

Example of memory limit exceeded result:

{
   "status_code": "MEMORY_LIMIT_EXCEEDED",
   "elapsed_time": -1.0,
   "consumed_memory": -1
}

StatusCode Schema

Enumeration of below values:

  • ACCEPTED - everything is fine and submission is accepted
  • WRONG_ANSWER - one of the test cases didn't succeed
  • COMPILE_ERROR - there is compilation/syntax error
  • RUNTIME_ERROR - there is runtime error
  • TIME_LIMIT_EXCEEDED - time limit is exceeded, solutions takes too long to execute
  • MEMORY_LIMIT_EXCEEDED - memory limit is exceeded, solutions takes too much memory to execute

/problems/{id}

  • GET
  • Produces - application/json
  • this endpoint returns problem data enriched with skeleton code for supported languages
  • it produces Problem json object, which is result of enrichment of original problem queried from Problems Service. The problem itself is enriched with skeleton codes for all supported programming languages. See below the Problem schema

/problems

  • GET
  • Produces - application/json
  • this endpoint returns all problems data enriched with skeleton code for supported languages
  • it produces array of Problems json object, which is result of enrichment of original problems queried from Problems Service. Every problem itself is enriched with skeleton codes for all supported programming languages. See below the Problem schema

Problem Schema

JSON object with below fields:

  • id - (string) problem id
  • title - (string) problem title
  • description - (string) problem description written in Markdown - so it can be later displayed with formatting - check Remarkable for more details
  • timeLimit - (int) time limit in seconds
  • memoryLimit - (int) memory limit in kilobytes
  • function - (Function) function specification - check Function schema below for more details. This is something you receive from Problems Service, but cleanup before sending to UI.
  • testCases - (meta-data) test cases specification - check TestCases schema below for more details. This is something you receive from Problems Service, but cleanup before sending to UI.
  • skeletonCode - (string) skeleton code which will be used by UI for pasting it before user will start implementing solution for a problem. It's agent responsibility to generate it based on function meta-data
  • level - (int) difficulty level, 1 means easy, 2 means medium, 3 means hard

From all above - id, title, description, timeLimit, memoryLimit and level you pass as it is when receiving from Problems Service. Additionally - you have to clean values of function and testCases and generating skeletonCode based on function value.

You can find instance of Problems Service on heroku: jalgoarena-problems.herokuapp.com

Function Schema

  • name - (string) function name - usually used for generated method name
  • return - (Return) meta-data describing return value of the generated method. See Return schema below for more details.
  • parameters - (Parameters[]) meta-data describing parameters of the generated method. See Parameters schema below for more details.

Function Return Schema

  • type - (string) if you are using dynamic language, you can skip it. In other case it defines java type name, which you should map into your language types
  • comment - (string) defines comment which should be added next to generated method for return type

Function Parameter Schema

  • name - (string) parameter name used in signature of generated method
  • type - (string) if you are using dynamic language, you can skip it. In other case it defines java type name, which you should map into your language types
  • comment - (string) efines comment which should be added next to generated method for particular parameter type

TestCases Schema

  • input - (string) json array with json objects inserted as parameters for generated function
  • output - (string) json object expected as result - checked during judgement

Example problem for Fibonacci problem:

{
  "id": "fib",
  "title": "Fibonacci",
  "description": "Write the `fib` function to return the N'th term.\r\nWe start counting from:\r\n* fib(0) = 0\r\n* fib(1) = 1.\r\n\r\n### Examples\r\n\r\n* `0` -> `0`\r\n* `6` -> `8`",
  "timeLimit": 1,
  "memoryLimit": 32,
  "function": {
    "name": "fib",
    "return": {
      "type": "java.lang.Long",
      "comment": " N'th term of Fibonacci sequence"
    },
    "parameters": [
      {
        "name": "n",
        "type": "java.lang.Integer",
        "comment": "id of fibonacci term to be returned"
      }
    ]
  },
  "testCases": [
    {
      "input": [
        "0"
      ],
      "output": 0
    },
    {
      "input": [
        "1"
      ],
      "output": 1
    },
    {
      "input": [
        "2"
      ],
      "output": 1
    }
  ],
  "level": 1
}

Component Diagram

Clone this wiki locally