Technical Note - Assessment System

Summary

The ECS Assessment System allows staff and tutors to mark and provide comments for submissions that students made using the Submission System. This document illustrates how the assessment system works. Please report any problems with this system to bugs@ecs.vuw.ac.nz

Details

The Assessment System is available at http://apps.ecs.vuw.ac.nz/cgi-bin/marking.

This application provides several functions for staff and tutors to support online assessment submission. Some of the functions provided for staff:

  • Add and configure online assessments to make them available on the online Submission System.
  • Display a list of all submissions as well as summary submission information for the assessment.
  • Display students that did not submit.
  • Approve late submissions for students.
  • Show marks for the assessment.
  • Import and export marks (available from the assessment marks page)
  • Allocate tutors to mark a submission.
  • Display late submission information.
  • Approve tutor comments and marks so that students can view it on studentmarks.
  • View a calendar of all assessments. (There is also a full Assignment Calendar, updated automatically from the Assessment System.)

The following functions are mainly for tutors:

  • Take an submission to mark.
  • View the submission files in a browser or download it (individually or as a combined zip file).
  • Upload a file to the submission's marking directory.
  • Add and edit marks and comments.
  • View the marking guide for the assessment.
  • Flag a problem with a submission and add administrative notes.

How to add a new assessment

Staff can set up an online assessment by going to http://apps.ecs.vuw.ac.nz/cgi-bin/marking and following these steps:

  1. Select the course from the list by clicking on the course name.
  2. On the course page the current assessments will be listed. Click on the "Add new assessment" link at the top.
  3. Fill in the assessment name and optionally add a due date, due time and submission message. Click on save to complete the setup.

The new assessment will immediately be available in the submission system. If the course you want to configure is not on the initial course list, please submit a job to jobs@ecs.vuw.ac.nz.

Please Note: (only for ECS courses) If you change an assessment's due date 2 weeks after the start of the trimester an email containing detail of this change will be send to all academic staff. This will let other lecturers know about potential changes that can have an impact on their courses. It will also happen if you add a new assessment 2 weeks after the start of the trimester.

Other assessment types

Above steps is all that is needed to setup a basic assessment. There are also other assessment types that can be used. To configure them click on the 'Additional settings' button and then select one of the following assessment types near the bottom of the page:
  1. Marking only - this is used when there is no submission required from students, but you want to mark and give feedback using the assessment system. An example of this is to show students marks for a written test.
  2. Group assessment - this is used so that students can work in teams and submit to the same area. It also gives the option to distribute marks from one marked student in a team to the other students in a team. Integrates with the Team Signup system which needs to be setup first before it can be used here. See How to mark Group Assessments for more info.

How to add marking items (questions) to assessments

From Trimester 2 2016 marking items need to be setup before tutors can mark an assessment. This is done by following these steps:

  1. Go to the assessment's Config page and then click on the Marking tab at the top.
  2. Click on 'Add new marking item'
  3. Add the item's name (eg. 'Question 1', or 'Core', etc.)
  4. Optionally add a maximum mark. The tutor's mark will be tested to not exceed this value. By default the allowed values are A-F with an optional plus or minus sign or numbers ranging from 0 to 100.
  5. Optionally add a minimum mark. If this is not set it will default to 0 if the maximum mark is set. This allows you to also use negative marks as penalties - for example: set the maximum to 0 and the minimum to -5.
  6. Set the Mark type. Currently there are Unspecified, Numeric and Grade. If set to Numeric or Grade it will restrict the marks that are allowed.
  7. Optionally set an item as an admin mark. These items are used to import marks and are not displayed to tutors. They can also be used as marks that you do not want the tutor to enter, eg Penalty. Students will see these marks on their studentmarks page.

Complete these steps for each marking item. The tutors will see a list of these items on their marking page and can only complete an assessment's marking when all marking items have a value.

How to add a final mark calculation to an assessment

Final marks can be automatically calculated and displayed to students. To configure this do the following:

  1. First setup the marking items as explained in the previous section.
  2. Next enter a simple arithmetic formula in the 'Final mark calculation' field.

Please note that only approved marks are used to calculate final marks. The minimum calculated mark is automatically limited to 0 so the formula does not need to compensate for this. The following functions can be used in the formula: floor, ceil, round, min, max, piecewise_linear_scale and piecewise_linear_scale2. (Please note: these functions must be lower-case)

After a formula has been added, the maximum mark for each marking item will be used to calculate the maximum final mark. You can test this before saving by using the 'Test new formula' button. If any part of your formula is optional, for instance a penalty or a bonus mark, don't set a maximum mark for these items and also set them as admin marks. The final mark calculation is based on the maximum values for each marking item and will otherwise be incorrect in these cases. Penalty marks that need to be added by tutors (not admin marks) can be handled by setting the maximum to 0 and asking the tutors to add a negative mark.

Example formulas

  • Average of Item 1 and Item 2: (M1 + M2)/2
  • Turn a single mark out of 30 into a percentage: M1 / 30 * 100
  • Limit an assessment with extra marks to 100 maximum: min(M1+M2+M3,100)
  • Add Item 1 (out of 15) as 70% and Item 2 (out of 12) as 30% of the total: (M1/15)*70+(M2/12)*30
  • Deduct 20% absolute for each day late from three marks (out of 100 each): (M1+M2+M3)/3 - ceil(M0)*20
  • Deduct 10% of total for each day late from three marks (that sum to 100): (M1+M2+M3)*(1-ceil(M0)*.1)
  • Deduct 10% as above, but if later than 2 days final mark is 0: (M1+M2+M3)*(1-ceil(M0)*.1) - 100*max(0, ceil(M0)-2)
  • Use piecewise linear scaling to scale marks:
    • The basic formula is: piecewise_linear_scale(current_formula, raw_mark, scaled_mark)
    • For example scale from 50 to 60: piecewise_linear_scale((M1+M2)/2, 50, 60)
    • This transform has little effect on the lowest and highest marks, but increases the marks around the raw mark configured to the scaled mark.
    • There is also a 2 point piecewise linear scaling function: piecewise_linear_scale2(current_formula, raw_mark1, scaled_mark1, raw_mark2, scaled_mark2)

Configuring a marking guideline

The marking guideline is displayed to tutors on each submission's marking page. To configure it enter the marking guide using plain text.

Moderation

Previously all assessments allowed more than one tutor to add marks and comments for the same submission. From Trimester 1 in 2020 a new moderation setting was added (found on the Assessment config Marking tab). By default assessments are now set to not allow moderation and you have to turn it on to use this feature. A second setting to hide moderation marks was also added. This allows tutors to see other tutors' marks or not. It is set to show all marks by default. Note that if moderation is enabled for an assessment it can't be disabled again.

How to edit an existing assessment

Go to http://apps.ecs.vuw.ac.nz/cgi-bin/marking and following these steps:

  1. Select the course from the list by clicking on the course name.
  2. On the course page the current assessments will be listed. Click on the wrench icon in the Config column of the assessment you want to change.
  3. Change any of the editable fields and click save to complete. (To change the assessment name requires a job to jobs@ecs.vuw.ac.nz)

How to copy assessments from the previous year

You can also copy your assessments from a previous year's instance of the course (or even another course). This should only be done if you are sure you will be using the exact same setup as the previous time since all settings are copied (except for some explained on the copy page). If you plan to change your assessments copying it might make more work for you.

  1. Go to http://apps.ecs.vuw.ac.nz/cgi-bin/marking and following these steps:
  2. Select the course from the list by clicking on the course name.
  3. On the course page the current assessments will be listed. Click on the "Copy assessments" link at the top.
  4. Select the course instance you want to copy from. By default the previous instance of the course is selected.
  5. Select the assessments to copy (or All if you are copying everything) and then click on Copy assessments.
  6. If successful the copied assessments will now show in the list on the right.
  7. You now need to configure the year specific information on the normal assessment config page. (Due date, Marking due date, and if a group assessment, the Team signup project to use).

Submission scripts

An assessment can be configured to execute a submission script when a student completes the submission. The script can do many things - for example test if a file compiles correctly, test if certain functions are present, or do unit tests on the compiled code. See /vol/submit/scripts/ and specifically /vol/submit/scripts/examples for ideas.

New from 2021 The submission script is called with the following arguments:
  • -a [ASSESSMENT_NAME]
  • -u [USERNAME]
  • -r [RESULT_DIRECTORY]
  • PATH

These arguments needs to be handled by your script in order to work correctly. The path will not be in the normal submission area and everything in there will be deleted after the script completes. If you want to copy any result files back to the student's submission directory (for example java .class files) you need to place them in the RESULT_DIRECTORY.

Anything written to stdout by the script is captured into a file that will be created in the student's submission directory.

A link to the script output file is presented to the student on the submission page (so that they can always read the output) and the output is also displayed as a one-time notice on the submission page immediately after completion.

The return code from the submission script is used as follows (values are for historic reasons):

  • 0 is success with no script output displayed
  • 1 - 10 is failure with red coloured script output displayed
  • 11 - 20 is success with green coloured script output displayed
  • anything else is failure with no script output displayed

Scripts must be be placed in the /vol/submit/scripts/course_name directory (contact jobs@ecs.vuw.ac.nz if the course_name directory does not exist).

The script is configured in the 'Submission script' field (expand Additional settings to see it) and a custom name for the script output can be configured in the 'Script output file' field (it defaults to 'script_output.txt' if not configured).

Automatic Late Penalties

The Assessment System can automatically calculate late penalties for submissions. This is useful for courses that allow a grace period (a number of allowed late days before penalties will be applied) for late submissions during the whole trimester.

When activated the number of late days will be calculated for each submission and if late a 'Late Penalty Days' mark will be added to the student's marks (this happens automatically the first time Marks are approved). This mark will also be displayed to students on studentmarks and can be used in the final mark calculation of assessments.

How to enable Automatic Late Penalties

  • To enable Automatic Late Penalties click on the course's config link. You can find this on the Assessment System home page http://apps.ecs.vuw.ac.nz/cgi-bin/marking or after you have selected a course by clicking on the 'Course config' link in the Menu under the course name (Menu->CourseCode->Course config).
  • Next click on the Marking tab. The first listed item is 'Enable automatic late penalties' - check this to enable. The second item to configure is 'Days allowed before late penalties applies'. Enter the number of days for your course. Note that these days are a total for the course and not for each assessment.
  • Click on Save config.

After this has been done you need to do two steps:
  • Adjust the final mark calculation formula for each assessment. Note that you can use M0 (the 'Late Penalty Days') in the formula even if it does not show yet. M0 will be automatically added after the first late calculation is done.
  • Configure the weight of each assessment.

Setting the weight higher means some assessments are worth more than others and are prioritised for applying late submission days. Setting the weight to 0 means the assignment will be the last one late days are applied to. If you don't want an assignment to use late days at all (for example a Lab that must be handed in at lab time) uncheck the 'Use late days' setting found beneath the weight setting.

Detail of late penalty calculation

Assessments are ordered by weight and then due date. Then for each assessment:
  • Determine if and how late an submission was by using the submitted files' dates compared to the assessment due date and factor in any extensions given.
  • If the submission is late, calculate if any allowed late days are left for the student (considering previous late submissions ordered by the same method) and subtract that from the late time.
  • If the remaining result is positive a penalty mark called 'Late Penalty Days' is added to the student submission. The format of this mark is a decimal number - for example 1.75 means the assessment was late by one day and 18 hours. This mark is automatically approved.

The late penalties for the course's assessments are calculated each time all marks are approved for an assessment. Since the order of assessments depend on weight and the later due assessments can potentially weight more than earlier due ones, late date marks for an assessment can change over the duration of the course. This will benefit students because the higher weighting assessment can use the allowed late days first.

Final mark calculation using late days

To use the 'Late Penalty Days' in the final mark calculation use the variable M0. For example to deduct 10% from the total for each full day late use the following formula:

(M1 + M2) * (10-floor(M0))/10

In this example the mark will be 0 if the submission is later than 10 days. If M0 is not used in the formula late days will have no influence on the calculated mark.

Example of a more complicated formula:

(M1+M2+M3+M4+M5)*(0.9**ceil(M0))

which gives an exponentially decaying 10% penalty for each late day.

See Example formulas above for more examples.

Other notes for automatic late penalties

  • If you approve a late submission for a student all late penalties for the student will be recalculated, so this is safe at any stage.
  • On the course marks page all the marks that contain a late penalty will be flagged with a light red background.
  • The assessment weight is configured on the Marking tab of the Assessment Config page.
  • If you uncheck the 'Use late days' setting on an assignment it won't use any of the late days allowed. The system will however still add a M0 mark for a student if they submit late. It is up to the lecturer to decide whether they want to use this in the final mark formula or not.
  • You can view the late penalty status for all students in the course on the 'Course late submission info' page. This is found in the Menu: Menu->CourseCode->Course late submission info

Mark approval

Marks need to be approved as the first step before students are able to view them (the final step is to release the marks). This allows lectures to resolve issues and also checking the marks in general. After tutors have marked assessments the marks are in an unapproved state. After all issues have been resolved and the marks checked, marks can be approved by clicking on the 'Approve all' button on the submissions page. There are also options to approve only marks or only comments. After this process has completed all marks (and comments) should be approved and any submissions with further issues will have a red flag in the Approved column (the number in brackets is the unapproved mark count). Marks and comments can also be approved individually on the student's submission page.

The criteria for the above process to approve marks for a submission are as follow:
  • All questions must be marked (except for admin marks which are optional).
  • All questions must have only one mark or comment. For instance if two tutors marked a student, and the same question was marked twice it cannot approve both.
  • There can't be general comments from more than one tutor or staff member - these comments can conflict and need to be individually resolved.
  • Unless specified in the assessment config at least one comment is required.

If you are not sure why a student's flag won't go green on approval, go to the student's submission page and click on 'Approve all' there. When done here the page will show a detailed report on why something did not approve.

Please note that imported marks are now also approved, but the same rules as above apply.

Mark release

From Trimester 1 in 2020 marks also needs to be released before students will be able to see them. Some of the reasons for this extra step:
  • accidental incorrect import settings might release marks before they are completely ready
  • the Approve all function fails on some submissions because not everything is complete or there are conflicting marks.
This feature will ensure that all students get their marks at the same time, and will also show clearly for which assessments students have their marks (and when it was released).

The Release button is found on the top of the submissions page for each assessment and acts as a toggle. It will show the status as 'Released' or 'Not Released'. After marks are released it will show the date it was released on the course's assessments list page in the Marks Released column.

If a problem is discovered with marks after release, clicking the button again will cause them to be "withdrawn" and not visible to students any more. Since some students may have seen their marks before this happens an explanation to the class may be necessary.

How to mark Group Assessments

Groups Assessments are marked differently from normal assessments:

  1. Mark only one team member from each team.
  2. (Optional) Upload feedback files for the team to the marked team member, after the next steps are done all team members will be able to view it.
  3. After marking is done approve marks as normal and fix any issues resulting from this.
  4. Next on the Admin functions panel there will be an additional section called "Distribute group marks" - press the Distribute group marks button. This will copy the marks and comments from the marked and approved team member in each team to all the unmarked team members (and approve them).
  5. (Optional) If there was an individual component as well, or a team member needs to have different marks for some reason you can now make these changes. It is save to run Distribute group marks again since this will only copy marks and comments if there are not an approved mark for a student yet.

Submission location

Submissions can be saved to any configured location, but generally they go into /vol/submit/[coursecode]/[assessment]/. For security reasons, this area is not available for writing to from a workstation.

Marking directories

When students submit code you will often want to be able to change or compile their code. To do this, you will need to copy their submissions to their marking directories. This is a sub-directory of a student's submission that has write permission for staff. This way we keep the original submission intact without any changes. To do this click on the Admin functions button on the submissions page and then on the Copy to Marking button.

Extensions and Exemptions

Both of these functions are done from a student's submission page for an assignment.

  • Extensions allow students to submit specific assignments late without it showing as such and it also remove automatic penalties (if a date is supplied for the extension it is used as the new due date for this calculation).
  • Exemptions are used to add a 'X' symbol in the assignment marks page. It should be used in cases where a student did not submit but there is an acceptable reason and the marks should be handled differently from normal. If a student submits after being given an exemption, the exemption flag should be removed and an appropriate extension added (if needed).

How to add marks and comments

  • Go to the Submissions list page by selecting a course and then an assessment's marking option.
  • On this page you can go to a specific student's submission page by clicking on their username, or if you want to mark it click on 'Start marking' in the status column. (If you are already assigned by a staff member this column will say 'Assigned (username)' with your username.)
  • The submission page is divided into 6 sections:
    1. At the top is the summary containing information such as student name, number of submitted files, late submission information, whether late submission was approved, exemptions and problem status.
    2. The next section list all submitted files. You can view/download files here and also download all of them as a zip file.
    3. The Marks section is next. In this area will be a list of all the items to mark along with a field to add the mark and comment. This field is updated immediately when you press tab or click outside it. Also displayed is the approval status and a delete button (only available if not yet approved). If more than one tutor are marking an submission (only for some courses) any marks added by other tutors will also be displayed here, they can be distinguished from yours by the username in the tutor column.
    4. The next section is for General comments. Existing comments are listed first and underneath that there are blue areas where new comments can be added. Existing comments can be edited (only if not yet approved) by clicking on the pencil icon at the bottom. The comment will update immediately when you press tab or click outside this area.
    5. The Administrative notes section is next. Here you can add notes meant for other markers or the lecturer. You can tick options at the bottom to send an email of the note to lectures. These notes are administrative and won't be displayed to students.
    6. The last section will display the Marking guide if available. It can be displayed as a popup dialog to keep open next to the Marks or Comments section.
  • After you have marked all items and added comments required by the Marking guide, complete the marking by clicking on the 'Complete marking' button in the Summary section. Complete marking will only be successful if you have marked all the items and added the required comments. Next to this button is also a 'Give back' button. This will 'unassign' you from the submission so that another tutor can complete the marking.

How to import marks

An alternative method to add marks is to import it directly using a .csv file. This can be done on each assessment's Marks page. The Marks page can be found by going to https://apps.ecs.vuw.ac.nz/cgi-bin/marking, then click on your course and lastly click on the icon in the Marks column next to the Assessment you want to upload marks for.

The import file format is documented in the Import marks panel on this page. A quick way to get a example for your assessment is to download from this page using the Export marks button. This file can be edited and imported again using a spreadsheet program (Excel or LibreOffice Calc) or text editor.

Import/export marks for tutors

Tutors can also import and export marks of submissions assigned to them. This is done from the Tutor Marking page by clicking on the Import/Export button. There are further instructions that displays after the button is clicked. Note that this can only be done if marks and comments are still unapproved.

FAQ

What to do if students can't see their marks?

The following 3 things need to be correct before students will be able to see their marks:
  • Is the assessment visible to students? Check the top setting on the Assessment Config page.
  • Was the marks released to students? The marks release status can be seen on the Course's Assessment listing page in the Release column or on the Assessment's Submissions page.
  • Does each student have approved marks? Check for green flags next to each student in the Assessment's Submissions page.

Why does an assignment not show on the Course Marks page?

On the course marks page an assignment shows as a single mark for each student. For the system to be able to do this one of the following three options needs to be in place:
  • A configured final mark formula.
  • A question/marking item called Final or Total.
  • Only a single question/marking item setup. If automatic penalties are enabled this will never be the case since this automatically adds a marking item (called 'Late Penalty Days').