Support instructions and feedback for individual checks
Saul Costa
Currently, instructions and positive/negative feedback is only supported on the task level. By adding to the check level as well, more precise feedback can be provided to the user.
If there is both feedback and the check results (e.g. unit test output) are not hidden, the check results should be hidden by default under an "Advanced" link or something like that.
Log In
This post was marked as
open
Saul Costa
planned
Troy Dundas
"My suggestion would be to introduce dynamic Codey feedback. I would think this could be accomplished by having Codey read his feedback from a file (.md or .txt), which would be created by the test itself. Since the checks would be hidden, they could be more thorough and complex without concern for end user readability, it would also enable all tests to be kept and managed in GitHub. File paths could be provided for each task within the task settings tab."
-- My comment from the NT support ticket
Saul Costa
Troy Dundas thanks for the feedback! I imagine this will be configured via the creator interface, just like the task feedback is. Is there a particular reason that you're a fan of reading it from a file?
Troy Dundas
Saul Costa:
I think if the feedback box or Codey's message box was reading from a file generated during runtime for each task it would give you all the flexibility you would ever need.
If each task could be configured to read from a file configured in the task configuration pane,
/workspace/.feedback1.md
, I believe it would be much more dynamic, even give the impression that Codey is more of an AI and actively grading work. Let's take the following check as an example:
with open('/home/nt-user/workspace/main.py', 'r') as f:
count = 0
for line in f.readlines():
count += 1
if "17.37951650185092" in line:
raise AssertionError("The value for knots was found hard coded in the program. Please remove this value.")
elif "20.0" in line:
raise AssertionError("The value for mph was found hard coded in the program. Please remove this value.")
elif "29.333333333333332" in line:
raise AssertionError("The value for feet was found hard coded in the program. Please remove this value.")
if count < 3:
raise AssertionError("Not enough progress has been made to pass this test. Write more code.")
f.close()
Currently, a check like the one above when failed prints out the assertion messages with a lot of other content, most of which can be difficult for a student to understand and depending on the error it can really clutter the output box.
If instead, the test could be print those custom messages to a file, and the check hidden, I think it could provide the following benefits:
* Clear up the check's feedback window by removing the potential need to display the test contents. Especially REGEX and checks focused on cheating.
* Condense the number of checks needed to grade. Currently, checks are more focused on grading what their name indicates.
def add(x, y):
return x + y
In the above it would make sense to have a check with the title "Testing
add
with 2 positive operands" and another check "Testing add
with 2 negative operands" and a third "Testing add
with 1 positive and 1 negative operands". This is done mainly so that when a check fails there is more context as to what failed. However, if reading from a file these 3 and more could be condensed into 1 check, which I would also imagine cuts down on task's overall runtime. * Better impression of Codey and Task feedback. Gives the impression that Codey is actively grading tasks. Task feedback is currently static. If a student fails it once or 30 times, regardless of what changes they make, it will show only that it is wrong. With a file, and a single check, Codey could provide feedback on multiple different scenarios which I think could actively instruct a student what is wrong with their code and how to fix it: "Your function does not add positive numbers correctly", "Your function does not add two negative numbers correctly", "Your function does not add a positive and negative numbers properly."
Saul Costa
Troy Dundas thanks for the detailed answer! It's really helpful to see the thought process behind how this would be used.
What we are currently envisioning is something very similar to the existing "positive" and "negative" feedback that is available on tasks, but added on a check level. That way the "advanced" results from a check (e.g. the test output) could be hidden, but the feedback would still be available for the students.
Does that sound like it would work for what you are trying to accomplish?
Troy Dundas
Saul Costa:
It sounds helpful and an improvement. There are advantages and disadvantages which might be from differences in implementation approaches. If you have a lot of checks per task having positive / negative feedback per check is helpful, whereas I think reading from a file is a means to reduce the number of checks per task and simplify the interface from the UX perspective.
I am personally biased towards having it read from a file. Having the positive and negative feedback per check is taking the current system which has its limitations and moving it down a level where they are not as apparent but may still exist. If a student is printing out "Hello, John!" the negative feedback would be the same for if the student printed out "Hi, World". Using the file method it could be more specific "We did not find the word 'Hello' in your greeting. Make sure to include it!".
Troy Dundas
I am also not blind to some of the limitations to using the file approach. I would imagine implementation would be more difficult. It is a different methodology when compared to the current Next Tech UI, and it might be possible for someone to go overboard with it but that should be chalked up to the course design and not the platform.
Saul Costa
Troy Dundas thanks for the continued details! I'm tracking most of what you're suggesting, but not all of it. If you're open to it, I'd like to set up a time to chat with you as we prepare to start development on this feature so I understand completed, which will be in a few weeks. Would you be open to that?
Troy Dundas
Saul Costa:
Sure! I am free rest of this week. Next week is a little bit more difficult.