The Design of the Active-Grader system.
The Active-Grader system (previously known as OpenGrader, but due name conflict with existing open-sourced project I can’t register that name on Google Code) is a system designed to be a unified grading system for programming contest.
From my last post, Aj. Jittat (more commonly known as Aj. Manow) gave me this valuable feedback:
I think that the problem is not about open-sourcing the grading system. It’s more about configuration and documentation. Different contests have different rules. Different task types need different grading methods and submission interfaces: think about output-only tasks or interactive tasks. These clearly make the grading systems some crazy beasts.
Very thanks to him, I finally come out with final design that suite all needs I can think of programming contests.
- Configuration and documentation. Yes, this is clearly a problem in past designs of contest environment. As I stated before in my last post, many of the contest environments was designed for private and or single use. So far, only little or no documentations were made. Even in the Moe Contest Environment, which is supposed to be a unified one, has really little documentation. When I studied it, I still need to read a shell scripts, which I don’t have knowledge for
- Different contests have different rules. Yes, but the task interface is mostly the same. The rules is to put in the UI, not the grading system.
- Different task types need different grading methods and submission interfaces. Those can be handle via my task grading model. About the submission UI, I think with a clear documented sets of utilities scripts and an API is enough so that it can be built into any system.
Well, now you must be questioning: What the heck is your task grading model?
In simple definition, it is model for grading tasks of any types and languages.
In my model, and from my experience with a few programming contests and online judges, I can split the process into three phrases:
- Compiling. Some task types (open-data/output-only) or languages (PHP, Perl) may not need this step. But most programming contest which involve automatic grading with an industrial-standard languages like C will require this part. The compiler can be configure on per-tasks basis.
- Executing. Some tasks type (open-data/output-only) may not require this step, but for automatic grading tasks this step the most important. By running user-submitted program a security must be ensured that user’s programs couldn’t harm the server. This is done by sandboxing. On Linux there is a ptrace interface. But I am sill seeking ways on windows[1].
- Grading/Checking. This is core of the testing. Basically, it just checks the correctness of the output.
Here is an example of the configuration. It is just for show off, not for any specific task! ==> grader-format.xml
Any comments or suggestions about my task grading model?
<?xml version=”1.0″ ?>
<test-request>
<!– sandboxes setup –>
<sandboxes>
<sandbox name=”executing”>
<timelimit value=”1″ />
<memlimit value=”64M” />
<fileaccess value=”restricted” />
<allowtime value=”false” />
<stdio in=”stdin” out=”stdout” err=”stderr” />
<!– … –>
</sandbox>
</sandboxes>
<!– custom compiler setup –>
<compilers>
<compiler name=”default”>
<compile mode=”extended” name=”%name%” compiler=”detect” additional=”%data%/testlib.c” maxsize=”200k” />
</compiler>
</compilers>
<!– Compiling part –>
<compile single-source=”true” compiler=”detect” />
<iflang mode=”detect”>
<case lang=”c”>
<gcc args=”-E” input=”abc.c xyz.c” output=”source.prep” />
<compile compiler=”default” name=”source.ext” />
</case>
<case lang=”pascal”>
<!– … –>
</case>
</iflang>
<testcases from=”1″ to=”10″ zerofill=”false”>
<!– evaluating part –>
<copy from=”%data%/%L.in” to=”stdin” />
<execute path=”prog” sandbox=”executing” />
<execute path=”prog” crossed-io=”judge” sandbox=”executing” crossed-io-sandbox=”judge” main-sandbox=”deadlock-avoider”
crossed-io-args=”%L” />
<!– checking part –>
<check checker=”internal|internal-float-mode|checker-path” solution=”%data%/%L.sol” output=”stdout” />
</testcases>
</text-request>
[1]: Here is a few links on my list that may help to get this working.
- WinAPI Override 32: http://jacquelin.potier.free.fr/winapioverride32/
- NTTrace (like strace, but with source so maybe we can emulate ptrace): http://www.howzatt.demon.co.uk/NtTrace/
- Logexts.dll: http://msdn.microsoft.com/en-us/library/ff560170.aspx
- Windows’ Syscalls Table (since we are working directly with system calls, this might be useful): http://www.metasploit.com/users/opcode/syscalls.html
- StraceNT: http://www.intellectualheaven.com/default.asp?BH=projects&H=strace.htm (if we can get the source code…)
- API Monitor: http://www.rohitab.com/apimonitor (again if we can get the source code)
- I also remember that there is Windows API in Windows Vista that works just like ptrace (for intercepting syscalls made by other process), but I can’t find them like now. Ahh.. Found’it: http://msdn.microsoft.com/en-us/library/aa964809(VS.85).aspx , http://msdn.microsoft.com/en-us/library/aa363691(VS.85).aspx
I found all above from http://stackoverflow.com/questions/864839/monitoring-certain-system-calls-done-by-a-process and http://stackoverflow.com/questions/865106/is-there-something-like-linux-ptrace-syscall-in-windows.
