unittest specify order

Answered

Hi,

I have a bunch of unit tests in a file using the unittest format.  When i right click on the file in the left pane of pycharm i see the option to "Run Unittests in test_filename,py".  That runs all the unit tests and im a happy camper.  The device im testing however, sometimes needs some of the unit tests to be run before others.  Before the inevitable "you shouldn't do that", "you're a moron" comments i was hoping to get some help.  The unittest.TestLoader class has a property called sortTestMethodsUsing.  If that is set to None then the test loader will load up the tests in the order it finds them in the unittest.TestCase class containing all the unit tests in the test_filename.py script.  You can also set that property to whatever sort order you want.  Its quite useful. 

I understand the right click > "Run 'Unittests in test_filename.py'" operation uses unittest.TestLoader in the background to load up all the tests to execute.  Is there any way to set the sortTestMethodsUsing property of the TestLoader pycharm is using?  Ideally i'd like to default it to None for all the unit tests i run.

At this point my 2 options i can see are:

1) name the unit test functions under my unittest.TestCase class to be something like test_0010_function_name, test_0020_function_name2...etc.  It works but is a pain.

2) create another script that instantiates a TestLoader allowing me to set the sortTestMethodsUsing property to whatever i want.  Then run the test suite the loader returns.  But then i wont be able to use the right click run option in pycharm which is incredibly convenient.  And ill have to create one of these scripts for each of my unittest files.

 

Thanks for your time,

Bryan

12 comments
Official comment

Hi.

You can monkeypatch this function. Try 

TestLoader.sortTestMethodsUsing = lambda self, a, b: cmp(a, b) * -1

 

Be sure to disable "sort alphabetically" option in PyCharm.

Thanks,

 

Where is the "sort alphabetically" option in pycharm?

0

 

It does not actually affects test order, but sorts results.

0

Thanks,

Im not seeing any difference in the order unit tests are executed.  They're still executing alphabetically.  I shortened one of my unit test files so i could post it:

import unittest
from amsv4_interface.amsv4 import AMSv4
from common.socket_comms import SocketComms
from amsv4_interface.devices.device_to_test import DeviceToTest


class GetAfterDefaults1(unittest.TestCase):

@classmethod
def setUpClass(self):
self.sc = SocketComms(DeviceToTest.ip_address, DeviceToTest.port)
self.ams = AMSv4(self.sc)
unittest.TestLoader.sortTestMethodsUsing = lambda self, a, b: (a > b) - (a < b) * -1

@classmethod
def tearDownClass(self):
self.sc.close_connection()

# outputs
def test_get_output_stereo(self):
for out_id in range(0, DeviceToTest.num_zones):
out_id_hex = self.ams.outputs.int_to_hex_string(out_id)
with self.subTest(out_id=out_id + 1):
self.assertEqual('Get Out[{0}] Stereo Mono status :{1}' .format(out_id + 1, 'stereo')
, self.ams.outputs.odsc(out_id_hex, self.ams.outputs.out1.get_stereo(send_it=False)))

def test_get_output_source(self):
for out_id in range(0, DeviceToTest.num_zones):
out_id_hex = self.ams.outputs.int_to_hex_string(out_id)
with self.subTest(out_id=out_id + 1):
self.assertEqual('Get Out[{0}] Input Source : input {1}' .format(out_id + 1, out_id + 1)
, self.ams.outputs.odsc(out_id_hex, self.ams.outputs.out1.get_source(send_it=False)))

def test_get_output_volumes(self):
for out_id in range(0, DeviceToTest.num_zones):
out_id_hex = self.ams.outputs.int_to_hex_string(out_id)
with self.subTest(out_id=out_id + 1):
self.assertEqual('Get Out[{0}] Volume : {1}' .format(out_id + 1, '-8.5')
, self.ams.outputs.odsc(out_id_hex, self.ams.outputs.out1.get_volume(send_it=False)))

def test_get_output_max_volumes(self):
for out_id in range(0, DeviceToTest.num_zones):
out_id_hex = self.ams.outputs.int_to_hex_string(out_id)
with self.subTest(out_id=out_id + 1):
self.assertEqual('Get Out[{0}] Max Volume : {1}' .format(out_id + 1, '0')
, self.ams.outputs.odsc(out_id_hex, self.ams.outputs.out1.get_max_volume(send_it=False)))

def test_get_output_turn_on_volumes(self):
for out_id in range(0, DeviceToTest.num_zones):
out_id_hex = self.ams.outputs.int_to_hex_string(out_id)
with self.subTest(out_id=out_id + 1):
self.assertEqual('Get Out[{0}] Turn on Vol : {1}' .format(out_id + 1, '-16.5')
, self.ams.outputs.odsc(out_id_hex, self.ams.outputs.out1.get_turn_on_volume(send_it=False)))


if __name__ == '__main__':
unittest.TestLoader.sortTestMethodsUsing = lambda self, a, b: (a > b) - (a < b) * -1

 

It looks like paste removed tabs so im sorry the formatting looks poor.  As you can see i tried monkey patching in the setUpClass section as well as the if __name__ == '__main__' section.  I'm running this in python 3.4.3 so the cmp function is gone.  I googled and found that (a > b) - (a < b) is a suitable replacement.  Should the monkey patching be done somewhere else?

Bryan

 

0

Try to monkeypatch it on the top level of your file, just before test class.

 

Following code runs B before A, for example

TestLoader.sortTestMethodsUsing = lambda self, a, b: (a < b) - (a > b)

class SampleTest(unittest.TestCase):
def test_a_true(self):
print("A")

def test_b_true(self):
print("B")
0

Thanks, monkeypatch at the top before the class definition changed the order of execution.  It's still alphanumeric though. see the example below:

Executing this runs D, C, B, A.  with no monkeypatch it runs A, B, C, D.  monkeypatching to None runs A, B, C, D.  

python documentation states:

sortTestMethodsUsing

Function to be used to compare method names when sorting them in getTestCaseNames() and all the loadTestsFrom*() methods. The default value is the built-in cmp() function; the attribute can also be set to None to disable the sort.

Any ideas how to get the unit tests to run in the order they're defined in the class?  Above it would be B, A, D, C.

Thanks,

Bryan

 

0

You could probably patch ``loadTestsFrom*`` or ``dir()`` your test case to know real order, but these solutions do not look elegant. I'd suggest to ask this question on StackOverflow since Jetbrains Support forum is more about PyCharm issues and not general Python questions. Defenetely there are much more Python developers on SO than on this forum.

https://stackoverflow.com/questions/tagged/python

0

ok, thanks for the help =)

 

Bryan

0

the problem here is : when we have hundreds of methods and updating/renaming them takes hell lot of time

do not have any other option

is there any option like : execute the way we have written?

 

Radha

0

Klaxmi361, PyCharm does not affect test order. Tests are executed by test runner (pytest, nose or unittest). You can check test framework documentation to modify test order.

 

Here is example for pytest

http://pytest-ordering.readthedocs.io/en/develop/

0

when i tried to install the package gettign below error

 

Collecting pytest-ordering
  Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x109fbb250>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /artifactory/api/pypi/python/simple/pytest-ordering/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x109fbb2d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /artifactory/api/pypi/python/simple/pytest-ordering/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x109fc1290>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /artifactory/api/pypi/python/simple/pytest-ordering/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x109fc1a90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /artifactory/api/pypi/python/simple/pytest-ordering/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x109fc1d90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /artifactory/api/pypi/python/simple/pytest-ordering/
  Could not find a version that satisfies the requirement pytest-ordering (from versions: )
No matching distribution found for pytest-ordering

0

Klaxmi361 please make sure you have internet connection working, and then try to install this package from terminal/command line. 

0

Please sign in to leave a comment.