PyCharm Edu runs student code and answer code when student gets answer wrong

Answered

I have written a course for PyCharm Edu which works fine when student gets answer correct and clicks Check Task. However if they put in a wrong answer, their solution is run, and then the correct answer is run, and this repeats itself several times. I am hampered by not being able to see output of print statements when in "learners" mode as per https://youtrack.jetbrains.com/issue/EDU-979 . I am noticing problem because the code is sending messages to an IRC channel and when the student gets it wrong I am getting the incorrect message then the correct message sent to IRC several times. I am happy for the student's answer to be run once but I don't want the correct answer run at all.

 

Here is a simplified task.py file which shows the same behaviour talking to a socket:

import socket

sock = socket.create_connection(('localhost', 10000))

def set_nodes(x1, y1, z1, x2, y2, z2, item):
s = "x1=" + str(x1) + " y1=" + str(y1) + " z1=" + str(z1) + " x2=" + str(x2) + " y2=" + str(y2) + " z2=" + str(z2) + " " + item
sock.sendall(s.encode('utf-8'))
print(sock.recv(2048).decode('utf-8'))

y = 14
z = 20
try:
sock.sendall(b'Student attempt')
print(sock.recv(2048).decode('utf-8'))
set_nodes(93, y-1, z-1, 70, y+5, z+1, "default:glass")
finally:
sock.close()

# I have answer placeholders on Student attempt; y-1; z-1; y+5; z+1
# Correct answers are: Correct message; y; z-2; y+6; z+2

####### Here is my tests.py file - Nothing special
from test_helper import run_common_tests, failed, passed, get_answer_placeholders

def test_answer_placeholders():
placeholders = get_answer_placeholders()
placeholder = placeholders[0]
if placeholder == "Correct message":
passed()
else:
failed("Message should be 'Correct message' not '" + placeholder + "'")

if __name__ == '__main__':
run_common_tests()
test_answer_placeholders()

######## Expected output at my python socket server
Student attempt
x1=93 y1=13 z1=19 x2=70 y2=19 z2=21 default:glass
waiting for a connection
######## Actual output at my python socket server
Student attempt
x1=93 y1=13 z1=19 x2=70 y2=19 z2=21 default:glass
waiting for a connection
Student attempt
x1=93 y1=14 z1=18 x2=70 y2=20 z2=22 default:glass
waiting for a connection
Correct message
x1=93 y1=13 z1=18 x2=70 y2=20 z2=22 default:glass
waiting for a connection
Correct message
x1=93 y1=14 z1=19 x2=70 y2=20 z2=22 default:glass
waiting for a connection
Correct message
x1=93 y1=14 z1=18 x2=70 y2=19 z2=22 default:glass
waiting for a connection
Correct message
x1=93 y1=14 z1=18 x2=70 y2=20 z2=21 default:glass
waiting for a connection

######### Code for my python socket server

# Simple python socket server which accepts data and sends it
# back in reverse order

import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('', 10000)
sock.bind(server_address)
sock.listen(3)

while True:
    print('waiting for a connection')
    connection, client_address = sock.accept()
    try:
        while True:
            data = connection.recv(2048)
            if data:
                s = data.decode('utf-8')
                print(s)
                connection.sendall((s[::-1]).encode('utf-8'))
            else:
                break
    finally:
        connection.close()

3 comments

Hi Dtimcummings!

Sorry for inconvenience, but also thank you for sharing your interesting use case with us.

PyCharm Edu has a feature called "smart check". If a student has failed a task with several answer placeholders in it, we try to identify which ones are incorrect. We do it the following way:

  1. Get correct solution
  2. For every placeholder replace it with student's version and run tests
  3. If this versions is recognized as correct, then we mark this placeholder as green. Otherwise red

That's why you can see that tests are being run several times. As solution I can suggest you the following: PyCharm Edu passes name of student's file as an argument, so you can check if sys.argv[1] contains "_window."(special marker that we add to our artificial files) part and in this case not to send info to IRC.

Please let me know, if it works for you.

 

 

0

Thanks for this suggestion. I used it to get my tests working. Is there any documentation that explains this test methodology?

0

We don't have documentation describing test process now, but we'll consider adding this information.

0

Please sign in to leave a comment.