Be careful to Import-time Side Effects in pytest

This is a note about what I found when fixing the existing test codes in pytest.

Photo by Steve Harvey on Unsplash

Background

I worked on Python software last week. What I did was fix the existing code and tests. I usually write Python for a service from scratch, which means I know all about the software and I can come up with a solution very quickly when I get a problem. However, I needed to work on the existing project which I didn’t know much and fix a small bug in the test code last week. So I needed to investigate step by step. The test codes were written with pytest and the whole codes were made without any frameworks like Django so the software was built of many Python scripts.

What I needed to fix was something weird about executing tests. Specifically, when I executed whole tests with the pytest tests/ command, it failed. However, when I executed the test which failed before with pytest tests/path/to/failed_test.py , it passed. Also, there were two markers, unitand integration, and the weird test was in the integrationmarker. Then I executed the tests separately with pytest -m unitand pytest -m intergrationThen both passed. So I thought there was something in the unit tests to block the integration tests from functioning as expected and started to investigate.

What I did

First of all, I won’t explain the details of the test and software. I’ll give you a quick rundown: the test failed because the HTTP server didn’t work due to the duplicated port number. So I assumed there were code lines to start an HTTP server with the same port in the unit test. Then I found one code block to start an HTTP server like below.

from src.server import API

API.start()

@pytest.mark.unit
def test_a():
    ...
    assert status_code == 200

@pytest.mark.unit
def test_b():
    ...

At this time, I didn’t understand why the server started. So I put some logs in the file like below.

from src.server import API

print("---------- Will start server -----------")
API.start()
print("---------- Have started server -----------")

@pytest.mark.unit
def test_a():
    ...
    assert status_code == 200

@pytest.mark.unit
def test_b():
    ...

Then I understood what was happening to us. The root cause was the import-time side effect. I should have remembered it when I saw the file. I don’t encounter it when I write codes with Ruby on Rails and React because I don’t have to care about the side effects. Instead of me, the framework and library prevent me from having it. So I just follow the Ruby on Rails/React rules. However, in this case, the software was made of many Python scripts so the import-time side effect could happen more.

Even more surprising to me was the pytestloading mechanism. When I executed the test with pytest tests/ , it loaded the target tests like below.

# pytest -m integration
====================== test session starts ====================================
platform linux -- Python 3.9.2, pytest-7.0.1, pluggy-1.5.0
rootdir: /usr/src/app, configfile: pytest.ini
collected 248 items / 18 deselected / 230 selected

Then I added -s option to the command to see the printed logs. It looked like below.

# pytest -m integration
====================== test session starts ====================================
platform linux -- Python 3.9.2, pytest-7.0.1, pluggy-1.5.0
rootdir: /usr/src/app, configfile: pytest.ini
---------- Will start server -----------
---------- Have started server -----------
collected 248 items / 18 deselected / 230 selected

So while collecting the target tests, the files were also imported and the side effects happened. Then everything made sense. When executing the pytest , it collected the target tests as importing the files and the import-time side effects happened at the same time. So I fixed the file below to avoid the side effects.

from src.server import API

@pytest.fixture(scope='class')
def start_api():
    API.start()

@pytest.mark.unit
@pytest.mark.usefixtures("start_api")
class TestApi:

    @staticmethod
    def test_a():
        ...
        assert status_code == 200
  
    @staticmethod
    def test_b():
        ...

I totally forgot about the import-time side effects because I usually don’t write at the top level unconsciously. It was a good opportunity that I understood its importance once again although it took time.

That’s it!