Technical Note - Assessment System
The ECS Assessment System allows staff and tutors to mark and provide comments for submissions that students made using the Submission System
. This document illustrates how the assessment system works. Please report any problems with this system to firstname.lastname@example.org
The Assessment System is available at http://apps.ecs.vuw.ac.nz/cgi-bin/marking
This application provides several functions for staff and tutors to support online assessment submission. Some of the functions provided for staff:
- Add and configure online assessments to make them available on the online Submission System.
- Display a list of all submissions as well as summary submission information for the assessment.
- Display students that did not submit.
- Approve late submissions for students.
- Show marks for the assessment.
- Import and export marks (available from the assessment marks page)
- Allocate tutors to mark a submission.
- Display late submission information.
- Approve tutor comments and marks so that students can view it on studentmarks.
- View a calendar of all assessments. (There is also a full Assignment Calendar, updated automatically from the Assessment System.)
The following functions are mainly for tutors:
- Take an submission to mark.
- View the submission files in a browser or download it (individually or as a combined zip file).
- Upload a file to the submission's marking directory.
- Add and edit marks and comments.
- View the marking guide for the assessment.
- Flag a problem with a submission and add administrative notes.
How to add a new assessment
Staff can set up an online assessment by going to http://apps.ecs.vuw.ac.nz/cgi-bin/marking
and following these steps:
- Select the course from the list by clicking on the course name.
- On the course page the current assessments will be listed. Click on the "Add new assessment" link at the top.
- Fill in the assessment name and optionally add a due date, due time, weight and submission message. Click on save to complete the setup.
The new assessment will immediately be available in the submission system. If the course you want to configure is not on the initial course list, please submit a job to email@example.com
(only for ECS courses) If you change an assessment's due date 2 weeks after the start of the trimester an email containing detail of this change will be send to all academic staff. This will let other lecturers know about potential changes that can have an impact on their courses. It will also happen if you add a new assessment 2 weeks after the start of the trimester.
Other assessment types
Above steps are all that is needed to setup a basic assessment. There are also other assessment types that can be used. To configure them scroll down to the "Assessment type settings" and then select one of the following assessment types near the bottom of the page.
This is used when there is no submission required from students, but you want to mark and give feedback using the assessment system. An example of this is to show students marks for a written test.
This is used so that students can work in teams and submit to the same area. It also gives the option to distribute marks from one marked student in a team to the other students in a team. Integrates with the Team Signup system which needs to be setup first before it can be used here. See How to mark Group Assessments
for more info.
This is used so that students can directly submit from a project in GitLab into the Assessment System.
- An existing GitLab group will have to be associated with the assessment. Only groups setup on
https://gitlab.ecs.vuw.ac.nz/course-work/COMP103/2022) are available for this. Since GitLab uses enrolments to create the needed student groups it works best to request this after the trimester has started.
- Students will be able to select either a project directly (the last commit from the default branch is submitted) or optionally a tag from their project to submit.
- When a student submits their GitLab project using the 'GitLab Submit' button an 'engr-submit-[datetime]' tag will be added to the last commit on the default branch of the selected project. The repository is then zipped and placed in the normal submission area.
- If only the code is required and not the full Git repository, select the "Delete Git repository if not needed" option to delete the repository on the Assessment System to save disk space (The GitLab repository is not changed in any way).
- If you plan to mark the assessement on GitLab and don't need the code on the Assessment System you can select the 'Only add submission tag to GitLab project' option. This option will also add a gitlab-receipt.txt file to the student's submission directory that contains the GitLab project url, the name of the submission tag and the commit sha hash. Note: Selecting this option means the GitLab project is not added to the student's submission directory.
This is used so that students can add information about pair work.
This indicates that it is an alternative to another assessment. If enabled it will not count towards to total course weight but the assessment it is alternative to will be added.
How to add marking items (questions) to assessments
From Trimester 2 2016 marking items need to be setup before tutors can mark an assessment. This is done by following these steps:
- Go to the assessment's Config page and then click on the Marking tab at the top.
- Click on 'Add new marking item'
- Add the item's name (eg. 'Question 1', or 'Core', etc.)
- Optionally add a maximum mark. The tutor's mark will be tested to not exceed this value. By default the allowed values are A-F with an optional plus or minus sign or numbers ranging from 0 to 100.
- Optionally add a minimum mark. If this is not set it will default to 0 if the maximum mark is set. This allows you to also use negative marks as penalties - for example: set the maximum to 0 and the minimum to -5.
- Set the Mark type. Currently there are Unspecified, Numeric and Grade. If set to Numeric or Grade it will restrict the marks that are allowed.
- Optionally set an item as an admin mark. These items are used to import marks and are not displayed to tutors. They can also be used as marks that you do not want the tutor to enter, eg Penalty. Students will see these marks on their studentmarks page.
Complete these steps for each marking item. The tutors will see a list of these items on their marking page and can only complete an assessment's marking when all marking items have a value.
How to add a final mark calculation to an assessment
Final marks can be automatically calculated and displayed to students. To configure this do the following:
- First setup the marking items as explained in the previous section.
- Next enter a simple arithmetic formula in the 'Final mark calculation' field.
Please note that only approved marks are used to calculate final marks. The minimum calculated mark is automatically limited to 0 so the formula does not need to compensate for this. The following functions can be used in the formula:
. (Please note: these functions must be lower-case)
After a formula has been added, the maximum mark for each marking item will be used to calculate the maximum final mark. You can test this before saving by using the 'Test new formula' button. If any part of your formula is optional, for instance a penalty or a bonus mark, don't set a maximum mark for these items and also set them as admin marks. The final mark calculation is based on the maximum values for each marking item and will otherwise be incorrect in these cases. Penalty marks that need to be added by tutors (not admin marks) can be handled by setting the maximum to 0 and asking the tutors to add a negative mark.
- Average of Item 1 and Item 2:
(M1 + M2)/2
- Turn a single mark out of 30 into a percentage:
M1 / 30 * 100
- Limit an assessment with extra marks to 100 maximum:
- Add Item 1 (out of 15) as 70% and Item 2 (out of 12) as 30% of the total:
- Deduct 20% absolute for each day late from three marks (out of 100 each):
(M1+M2+M3)/3 - ceil(M0)*20
- Deduct 10% of total for each day late from three marks (that sum to 100):
- Deduct 10% as above, but if later than 2 days final mark is 0:
(M1+M2+M3)*(1-ceil(M0)*.1) - 100*max(0, ceil(M0)-2)
- Use piecewise linear scaling to scale percentages (marks must be out of 100):
- The basic formula is:
piecewise_linear_scale(current_formula, raw_mark, scaled_mark)
- For example scale from 50 to 60:
piecewise_linear_scale((M1+M2)/2, 50, 60)
- This transform has little effect on the lowest and highest marks, but increases the marks around the raw mark configured to the scaled mark.
- There is also a 2 point piecewise linear scaling function:
piecewise_linear_scale2(current_formula, raw_mark1, scaled_mark1, raw_mark2, scaled_mark2)
Configuring a marking guideline
The marking guideline is displayed to tutors on each submission's marking page. To configure it enter the marking guide using plain text.
Previously all assessments allowed more than one tutor to add marks and comments for the same submission. From Trimester 1 in 2020 a new moderation setting was added (found on the Assessment config Marking tab). By default assessments are now set to not allow moderation and you have to turn it on to use this feature. A second setting to hide moderation marks was also added. This allows tutors to see other tutors' marks or not. It is set to show all marks by default. Note that if moderation is enabled for an assessment it can't be disabled again.
How to edit an existing assessment
Go to http://apps.ecs.vuw.ac.nz/cgi-bin/marking
and following these steps:
New from 2022
- Select the course from the list by clicking on the course name.
- On the course page the current assessments will be listed. Click on the wrench icon in the Config column of the assessment you want to change.
- Change any of the editable fields and click save to complete. (To change the assessment name requires a job to firstname.lastname@example.org)
Some courses are now configured to lock their scheduling at the start of the trimester. This has the following effects after the lock has been enabled:
- The assessment weight can't be changed any more.
- The due date can be changed as normal except if the assessment you are changing has a weight of 10% or more and you are moving the due date by more than 3 days. In such a case you will get a popup notification warning you that after saving your change will be treated as a request until is has been approved by the Associate Dean (Students). The change request are automatically send for approval, you don't need to do anything except changing the date and saving the assessment.
How to copy assessments from the previous year
You can also copy your assessments from a previous year's instance of the course (or even another course). This should only be done if you are sure you will be using the exact same setup as the previous time since all settings are copied (except for some explained on the copy page). If you plan to change your assessments copying it might make more work for you.
- Go to http://apps.ecs.vuw.ac.nz/cgi-bin/marking and following these steps:
- Select the course from the list by clicking on the course name.
- On the course page the current assessments will be listed. Click on the "Copy assessments" link at the top.
- Select the course instance you want to copy from. By default the previous instance of the course is selected.
- Select the assessments to copy (or All if you are copying everything) and then click on Copy assessments.
- If successful the copied assessments will now show in the list on the right.
- You now need to configure the year specific information on the normal assessment config page. (Due date, Marking due date, and if a group assessment, the Team signup project to use).
An assessment can be configured to execute a submission script when a student completes the submission. The script can do many things - for example test if a file compiles correctly, test if certain functions are present, or do unit tests on the compiled code. See
New from 2021
The submission script is called with the following arguments:
- -a [ASSESSMENT_NAME]
- -u [USERNAME]
- -r [RESULT_DIRECTORY]
- -d [DUE_DATE_TIME]
These arguments needs to be handled by your script in order to work correctly. The path will not be in the normal submission area and everything in there will be deleted after the script completes. If you want to copy any result files back to the student's submission directory (for example java .class files) you need to place them in the RESULT_DIRECTORY.
The DUE_DATE_TIME will be a string with the following format: '2022-03-22T12:30:00'. It can be 'None' if the due date is not set for the assessment, and will be adjusted if an extension was given to the submitting student.
Anything written to stdout by the script is captured into a file that will be created in the student's submission directory.
A link to the script output file is presented to the student on the submission page (so that they can always
read the output) and the output is also displayed as a one-time notice on the submission page immediately
The return code from the submission script is used as follows (values are for historic reasons):
- 0 is success with no script output displayed
- 1 - 10 is failure with red coloured script output displayed
- 11 - 20 is success with green coloured script output displayed
- anything else is failure with no script output displayed
Scripts must be be placed in the
directory (contact email@example.com
if the course_name directory does not exist).
The script is configured in the 'Submission script' field (scroll to the "Submission script settings" on the Assessment config page) and a custom name for the script output can be configured in the 'Script output file' field (it defaults to 'script_output.txt' if not configured).
Automatic Late Penalties
The Assessment System can automatically calculate late penalties for submissions. This is useful for courses that allow a grace period (a number of allowed late days before penalties will be applied) for late submissions during the whole trimester.
When activated the number of late days will be calculated for each submission and if late a 'Late Penalty Days' mark will be added to the student's marks (this happens automatically the first time Marks are approved). This mark will also be displayed to students on studentmarks and can be used in the final mark calculation of assessments.
How to enable Automatic Late Penalties
- To enable Automatic Late Penalties click on the course's config link. You can find this on the Assessment System home page http://apps.ecs.vuw.ac.nz/cgi-bin/marking or after you have selected a course by clicking on the 'Course config' link in the Menu under the course name (Menu->CourseCode->Course config).
- Look for the 'Late submission settings' heading. The first listed item is 'Enable automatic late penalties' - check this to enable. The second item to configure is 'Days allowed before late penalties applies'. Enter the number of days for your course. Note that these days are a total for the course and not for each assessment.
- Click on Save config.
After this has been done you need to do two steps:
- Adjust the final mark calculation formula for each assessment. Note that you can use M0 (the 'Late Penalty Days') in the formula even if it does not show yet. M0 will be automatically added after the first late calculation is done.
- Configure the weight of each assessment.
Setting the weight higher means some assessments are worth more than others and are prioritised for applying late submission days. Setting the weight to 0 means the assessment will be the last one late days are applied to. If you don't want an assessment to use late days at all (for example a Lab that must be handed in at lab time) uncheck the 'Use late days' setting found beneath the weight setting.
Detail of late penalty calculation
Assessments are ordered by weight and then due date. Then for each assessment:
- Determine if and how late an submission was by using the submitted files' dates compared to the assessment due date and factor in any extensions given.
- If the submission is late, calculate if any allowed late days are left for the student (considering previous late submissions ordered by the same method) and subtract that from the late time.
- If the remaining result is positive a penalty mark called 'Late Penalty Days' is added to the student submission. The format of this mark is a decimal number - for example 1.75 means the assessment was late by one day and 18 hours. This mark is automatically approved.
The late penalties for the course's assessments are calculated each time all marks are approved for an assessment. Since the order of assessments depend on weight and the later due assessments can potentially weight more than earlier due ones, late date marks for an assessment can change over the duration of the course. This will benefit students because the higher weighting assessment can use the allowed late days first.
Final mark calculation using late days
To use the 'Late Penalty Days' in the final mark calculation use the variable M0. For example to deduct 10% from the total for each full day late use the following formula:
(M1 + M2) * (10-floor(M0))/10
In this example the mark will be 0 if the submission is later than 10 days. If M0 is not used in the formula late days will have no influence on the calculated mark.
Example of a more complicated formula:
which gives an exponentially decaying 10% penalty for each late day.
See Example formulas
above for more examples.
Other notes for automatic late penalties
- If you approve a late submission for a student all late penalties for the student will be recalculated, so this is safe at any stage.
- On the course marks page all the marks that contain a late penalty will be flagged with a light red background.
- The assessment weight is configured under the "Main settings" heading on the Assessment Config page.
- If you uncheck the 'Use late days' setting on an assessment it won't use any of the late days allowed. The system will however still add a M0 mark for a student if they submit late. It is up to the lecturer to decide whether they want to use this in the final mark formula or not.
- You can view the late penalty status for all students in the course on the 'Course late submission info' page. This is found in the Menu: Menu->CourseCode->Course late submission info
Marks need to be approved as the first step before students are able to view them (the final step is to release the marks). This allows lectures to resolve issues and also checking the marks in general. After tutors have marked assessments the marks are in an unapproved state. After all issues have been resolved and the marks checked, marks can be approved by clicking on the 'Approve all' button on the submissions page. There are also options to approve only marks or only comments. After this process has completed all marks (and comments) should be approved and any submissions with further issues will have a red flag in the Approved column (the number in brackets is the unapproved mark count). Marks and comments can also be approved individually on the student's submission page.
The criteria for the above process to approve marks for a submission are as follow:
- All questions must be marked (except for admin marks which are optional).
- All questions must have only one mark or comment. For instance if two tutors marked a student, and the same question was marked twice it cannot approve both.
- There can't be general comments from more than one tutor or staff member - these comments can conflict and need to be individually resolved.
- Unless specified in the assessment config at least one comment is required (uncheck the 'Comments required' setting).
If you are not sure why a student's flag won't go green on approval, go to the student's submission page and click on 'Approve all' there. When done here the page will show a detailed report on why something did not approve.
Please note that imported marks are now also approved, but the same rules as above apply.
From Trimester 1 in 2020 marks also needs to be released before students will be able to see them. Some of the reasons for this extra step:
- incorrect import settings might release marks before they are completely ready
- the Approve all function fails on some submissions because not everything is complete or there are conflicting marks.
This feature will ensure that all students get their marks at the same time, and will also show clearly for which assessments students have their marks (and when it was released).
The Release button is found on the top of the submissions page for each assessment and acts as a toggle. It will show the status as 'Released' or 'Not Released'. After marks are released it will show the date it was released on the course's assessments list page in the Marks Released column.
If a problem is discovered with marks after release, clicking the button again will cause them to be "withdrawn" and not visible to students any more. Since some students may have seen their marks before this happens an explanation to the class may be necessary.
How to mark Group Assessments
Groups Assessments are marked differently from normal assessments:
- Mark only one team member from each team.
- (Optional) Upload feedback files for the team to the marked team member, after the next steps are done all team members will be able to view it.
- After marking is done approve marks as normal and fix any issues resulting from this.
- Next on the Admin functions panel there will be an additional section called "Distribute group marks" - press the Distribute group marks button. This will copy the marks and comments from the marked and approved team member in each team to all the unmarked team members (and approve them).
- (Optional) If there was an individual component as well, or a team member needs to have different marks for some reason you can now make these changes. It is save to run Distribute group marks again since this will only copy marks and comments if there are not an approved mark for a student yet.
Submissions can be saved to any configured location, but generally they go into
. For security reasons, this area is not available for writing to from a workstation.
When students submit code you will often want to be able to change or compile their code. To do this, you will need to copy their submissions to their marking directories. This is a sub-directory of a student's submission that has write permission for staff. This way we keep the original submission intact without any changes. To do this click on the Admin functions
button on the submissions page and then on the Copy to Marking
Extensions and Exemptions
Both of these functions are done from a student's submission page for an assessment.
New from 2022
- Extensions allow students to submit specific assessments late without it showing as such and it also remove automatic penalties (if a date is supplied for the extension it is used as the new due date for this calculation).
- Exemptions are used to add a 'X' symbol in the assessment marks page. It should be used in cases where a student did not submit but there is an acceptable reason and the marks should be handled differently from normal. If a student submits after being given an exemption, the exemption flag should be removed and an appropriate extension added (if needed).
Extension requests are now possible from the submission system.
Enable Extension Requests
Course coordinators can enable extension requests for their courses by doing the following:
- Go to the course's Config page by either clicking on the wrench icon in the Config column on https://apps.ecs.vuw.ac.nz/cgi-bin/marking (select the correct trimester using the Next trimester button if needed at the top) or by clicking on the course and then selecting Course->Course Config from the top right drop down menu.
- Next find the Extension request settings section on the config page and then enable the "Allow extension requests" checkbox and click on 'Save config'.
After this is checked students can use the Submission system to send extension requests. They will see a page allowing them to select the assessment, add a message and attach supporting documents.
After a student submits a request, the course coordinator and configured delegates (* see below) will receive an email with the detail and a link to the Extension Request page for the course.
On the Extension Request page you can see a list of all requests for the course. You can order it by any of the columns in the table. The open requests will be at the top of the list by default. To approve/deny a request click on the View button for it and then fill in the optional date, message (send to the student) and note (not send to the student). After clicking on Approve/Deny the students will receive an email with the outcome.
Extensions handled this way is backwards compatible with the old method of giving students extensions on the submission page and will show there as well. On the Extension request page there is also a second tab called "Approved Extensions". This will list all approved extensions for the course and includes both extensions made using a request from a student or directly on the submissions page using the old method.
(*) Delegates can be configured to approve/deny requests and to receive email. On the course's Config page there is a "Extension request delegates" field below the "Allow extension requests" setting. Here the course coordinator can set additional email addresses (used to only notify) and ECS usernames (to notify and to handle requests).
- Go to the Submissions list page by selecting a course and then an assessment's marking option.
- On this page you can go to a specific student's submission page by clicking on their username, or if you want to mark it click on 'Start marking' in the status column. (If you are already assigned by a staff member this column will say 'Assigned (username)' with your username.)
- The submission page is divided into 6 sections:
- At the top is the summary containing information such as student name, number of submitted files, late submission information, whether late submission was approved, exemptions and problem status.
- The next section list all submitted files. You can view/download files here and also download all of them as a zip file.
- The Marks section is next. In this area will be a list of all the items to mark along with a field to add the mark and comment. This field is updated immediately when you press tab or click outside it. Also displayed is the approval status and a delete button (only available if not yet approved). If more than one tutor are marking an submission (only for some courses) any marks added by other tutors will also be displayed here, they can be distinguished from yours by the username in the tutor column.
- The next section is for General comments. Existing comments are listed first and underneath that there are blue areas where new comments can be added. Existing comments can be edited (only if not yet approved) by clicking on the pencil icon at the bottom. The comment will update immediately when you press tab or click outside this area.
- The Administrative notes section is next. Here you can add notes meant for other markers or the lecturer. You can tick options at the bottom to send an email of the note to lectures. These notes are administrative and won't be displayed to students.
- The last section will display the Marking guide if available. It can be displayed as a popup dialog to keep open next to the Marks or Comments section.
- After you have marked all items and added comments required by the Marking guide, complete the marking by clicking on the 'Complete marking' button in the Summary section. Complete marking will only be successful if you have marked all the items and added the required comments. Next to this button is also a 'Give back' button. This will 'unassign' you from the submission so that another tutor can complete the marking.
How to import marks
An alternative method to add marks is to import it directly using a .csv file. This can be done on each assessment's Marks page. The Marks page can be found by going to https://apps.ecs.vuw.ac.nz/cgi-bin/marking
, then click on your course and lastly click on the icon in the Marks column next to the Assessment you want to upload marks for.
When you import marks you can decide to directly approve the marks by checking 'Approve marks while importing' on the form. If this is done you don't need to later use 'Approve all' to do so. This is mostly intended for 'Marking only' type assessments.
Notes about the import file:
- The import file format is a standard .CSV file. CSV files exported by Excel, LibreOffice Calc and other Spreadsheet programs should all work. You can also create the file using a text editor, but make sure it is a valid CSV file.
- The file must have a header line as follows:
Username,Question1,Question2,... where you need to replace Question1, etc. with the real question names.
- You can use
StudentID in place of
- Comments can also optionally be imported using columns named
Question1_comment,Question2_comment,... (again replacing Question1, etc. with the real question names) and
general_comment for general comments.
- You can also have an optional
Tutor column. This needs to contain the tutor usernames and will then allocate the tutor as the marker so that the marking statistics can be used. If this is not needed the marker is set to "import"
- A quick way to get a example CSV file for your assessment is to use the 'Export marks' button on the same page. This file can be edited and imported again using a spreadsheet program (Excel or LibreOffice Calc) or text editor.
- "Quoted" and unquoted headers and data are accepted as long as it is valid CSV.
Please note that the import process will unapprove/replace marks that were added by tutors if they already marked submissions.
Import/export marks for tutors
Tutors can also import and export marks of submissions assigned to them. This is done from the Tutor Marking page by clicking on the Import/Export button. There are further instructions that displays after the button is clicked. Note that this can only be done if marks and comments are still unapproved.
Assessment schedule locking
New from 2022
Courses can be configured to use schedule locking. At the moment this is an "opt in" feature that will be agreed between the AD Students and the course coordinator. When enabled the following will happen:
- The initial assessment setup is the same as before but you will notice 'Total weight' and 'All due dates set' headings with check marks next to them on the Assessments list page.
- Schedule locking requires that all assessments have due dates and weights set. Additionally the weights need to sum to 100. This might require adding all marking only assessments even if you don't plan to use the submission/marking system for them. There is also a new 'Alternative assessment' setting that you can use to indicate that the assessment's weight should not be added to the total.
- When all assessments have due dates and the weights add to 100 the course auditor needs to make sure everything is correct and then lock the schedule.
- After it has been locked:
- You won't be able to change the assessment's weight.
- You can only change the due date for the assessment yourself if the weight is less than 10% or the new date differs only 3 days from the locked due date. If you move a date for an assessment with a weight equal to or larger than 10% for more than 3 days it will be send as a request to the AD Students for approval. (The page will warn you when this will happen.)
- After the AD students approve of deny the request you will receive an email response with optional comments. If the request was approved the change will already be applied and you don't need to do anything further.
- You can still add new assessments, but they will be added with weight set to zero.
What to do if students can't see their marks?
The following 3 things need to be correct before students will be able to see their marks:
- Is the assessment visible to students? Check the top setting on the Assessment Config page.
- Was the marks released to students? The marks release status can be seen on the Course's Assessment listing page in the Release column or on the Assessment's Submissions page.
- Does each student have approved marks? Check for green flags next to each student in the Assessment's Submissions page.
- Note that if you imported marks and approved them while importing, but did not uncheck the "Comments required" option the flags will still show red. In this case the students will however be able to see their marks.
Why does an assessment not show on the Course Marks page (or without statistics on the Assessment Marks page)?
On the course marks page an assessment shows as a single mark for each student. For the system to be able to do this one of the following three options needs to be in place:
- A configured final mark formula.
- A question/marking item called Final or Total.
- Only a single question/marking item setup. If automatic penalties are enabled this will never be the case since this automatically adds a marking item (called 'Late Penalty Days').