Python FastAPI app runs in debugger, but crashes when run with docker compose

Hi everyone, 
 
I wanted to share some upfront information; despite my years as a developer, I'm a relative noob when it comes to Docker.
 
At work, I'm building a new FastAPI application. I'm developing this to be deployed as a Docker container and am using Docker to develop locally on my M1 Macbook Pro. I'm using Python 3.10.8 and the latest FastAPI, etc., and the latest PyCharm. 
 
Here's my simplified project directory structure:
 
payment
    +-- < general project stuff, including my docker-compose.yml file >
    +-- project
           +-- < Dockerfile, pyproject.toml, etc >   
           +-- app
                                +-- < directories containing application code >
 
I've got a remote interpreter setup using the docker-compose.yml file, and it starts when I click the run or debug button in PyCharm. However, the application fails to start because the Python path is screwy. If open a bash shell into the container, my project is now in /opt/project/project. In the configuration file in PyCharm, to run the project, I need to set an environment variable PYTHONPATH=/opt/project/project to get the application to start correctly and allow me to debug it.
 
This works as I can debug the application in the Docker container in the PyCharm debugger, but it seems weird to me and took me some time to figure it out.
 
When I run docker compose -f docker-compose.yml up --build -d from the command line, that application crashes. If I open a bash shell in the container, the project is located at /project right at the root of the container file system.
 
Here is the output on the terminal when I run "docker compose...."
 
$ docker logs 36e4b6a84816             
ENVIRONMENT: development
Starting development server
INFO:     Will watch for changes in these directories: ['/project']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [1] using StatReload
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/config.py", line 474, in load
    self.loaded_app = import_from_string(self.app)
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/importer.py", line 24, in import_from_string
    raise exc from None
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/project/./app/main.py", line 20, in <module>
    from .server import app
  File "/project/./app/server.py", line 9, in <module>
    from auth.routes import router as auth_router
ModuleNotFoundError: No module named 'auth'
 
Would you happen to have any insights about why this is happening?
 
I can send you my Dockerfile and docker-compose.yml file if that would be useful.
 
Thanks very much for taking a look at this,
Doug Farrell 
7 comments
Comment actions Permalink

Hello,

Dockerfile and docker-compose.yml are needed to understand better what might cause an issue. 

1
Comment actions Permalink

Hi Daniil,

I'll upload the Dockerfile and docker-compose.yml files using a link I've gotten from a direct email from Sergey.

Here is the upload ID from the above link: 2022_11_15_KgVJjhgp41r8Dh98LZisXF

Thanks!

Doug

0
Comment actions Permalink

Hello Doug,

I've built an image with a dummy fastapi project and your Dockerfile and .yml files, and it's expected that /project will contain project files but /opt contains only venv, as it states in Dockerfile.

The interpreter was also added without any issues and ran test files without extra environment variables.

I'm a little bit confused, the only thing that bothers me is that there is an extra step between the project and app dirs in your log: /project/./app/main.py. But I can't find anything related to that "dot folder" in Dockerfile or docker-compose; those seem fine. Could it be a misconfiguration in other project files?

0
Comment actions Permalink

Daniil,

Thanks for getting back to me. I'm also seeing that strange /project/./app/main.py behavior, and don't understand it. 

Right now, I can run the app from the debugger in PyCharm and the terminal command line. If I run this command:

docker compose -f docker-compose.yml up -d --build

I get a running application inside the docker container. If open a bash shell into the container, I get this information with some command line stuff:

appuser@20bb48f9148a:/project$ ls -la /
total 80
drwxr-xr-x   1 root    root    4096 Nov 18 18:41 .
drwxr-xr-x   1 root    root    4096 Nov 18 18:41 ..
-rwxr-xr-x   1 root    root       0 Nov 18 18:41 .dockerenv
drwxr-xr-x   1 root    root    4096 Nov 15 09:32 bin
drwxr-xr-x   2 root    root    4096 Sep  3 12:10 boot
drwxr-xr-x   5 root    root     360 Nov 18 18:42 dev
drwxr-xr-x   1 root    root    4096 Nov 18 18:41 etc
drwxr-xr-x   2 root    root    4096 Sep  3 12:10 home
drwxr-xr-x   1 root    root    4096 Nov 15 09:32 lib
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 media
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 mnt
drwxr-xr-x   1 root    root    4096 Nov 16 22:03 opt
dr-xr-xr-x 217 root    root       0 Nov 18 18:42 proc
drwxr-xr-x  15 appuser appuser  480 Nov 14 18:43 project
drwx------   1 root    root    4096 Nov 18 18:41 root
drwxr-xr-x   3 root    root    4096 Nov 14 00:00 run
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 sbin
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 srv
dr-xr-xr-x  13 root    root       0 Nov 18 18:42 sys
drwxr-xr-x   9 root    root     288 Nov 10 18:15 tests
drwxrwxrwt   1 root    root    4096 Nov 16 22:04 tmp
drwxr-xr-x   1 root    root    4096 Nov 14 00:00 usr
drwxr-xr-x   1 root    root    4096 Nov 14 00:00 var
appuser@20bb48f9148a:/project$ pwd
/project
appuser@20bb48f9148a:/project$ ls -la
total 84
drwxr-xr-x 15 appuser appuser   480 Nov 14 18:43 .
drwxr-xr-x  1 root    root     4096 Nov 18 18:41 ..
-rw-r--r--  1 appuser appuser   385 Oct 21 15:05 .dockerignore
drwxr-xr-x  6 appuser appuser   192 Nov 10 17:59 .pytest_cache
drwxr-xr-x  6 appuser appuser   192 Nov 14 16:05 .venv
-rw-r--r--  1 appuser appuser   232 Nov 16 17:52 BUILD
-rw-r--r--  1 appuser appuser  1739 Nov 14 18:43 Dockerfile
-rw-r--r--  1 appuser appuser     0 Oct 31 14:28 __init__.py
drwxr-xr-x  3 appuser appuser    96 Oct 31 14:34 __pycache__
drwxr-xr-x  2 appuser appuser    64 Sep 29 21:12 alembic
-rw-r--r--  1 appuser appuser  1591 Oct  6 19:58 alembic.ini
drwxr-xr-x 18 appuser appuser   576 Nov 18 18:42 app
-rw-r--r--  1 appuser appuser 54975 Nov 16 22:01 poetry.lock
-rw-r--r--  1 appuser appuser  1344 Nov 16 22:01 pyproject.toml
-rwxr-xr-x  1 appuser appuser   381 Nov 11 22:44 startup.sh
appuser@20bb48f9148a:/project$ 

If I run the application from the PyCharm debugger and open a bash shell in the container and run the same set of command line tools, I get this:

appuser@be88621182fa:/opt/project/project/app$ ls -la /
total 76
drwxr-xr-x   1 root    root    4096 Nov 18 18:48 .
drwxr-xr-x   1 root    root    4096 Nov 18 18:48 ..
-rwxr-xr-x   1 root    root       0 Nov 18 18:48 .dockerenv
drwxr-xr-x   1 root    root    4096 Nov 15 09:32 bin
drwxr-xr-x   2 root    root    4096 Sep  3 12:10 boot
drwxr-xr-x   5 root    root     360 Nov 18 18:48 dev
drwxr-xr-x   1 root    root    4096 Nov 18 18:48 etc
drwxr-xr-x   2 root    root    4096 Sep  3 12:10 home
drwxr-xr-x   1 root    root    4096 Nov 15 09:32 lib
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 media
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 mnt
drwxr-xr-x   1 root    root    4096 Nov 18 18:48 opt
dr-xr-xr-x 214 root    root       0 Nov 18 18:48 proc
drwxr-xr-x  15 appuser appuser  480 Nov 14 18:43 project
drwx------   1 root    root    4096 Nov 18 18:48 root
drwxr-xr-x   3 root    root    4096 Nov 14 00:00 run
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 sbin
drwxr-xr-x   2 root    root    4096 Nov 14 00:00 srv
dr-xr-xr-x  13 root    root       0 Nov 18 18:48 sys
drwxr-xr-x   9 root    root     288 Nov 10 18:15 tests
drwxrwxrwt   1 root    root    4096 Nov 16 22:04 tmp
drwxr-xr-x   1 root    root    4096 Nov 14 00:00 usr
drwxr-xr-x   1 root    root    4096 Nov 14 00:00 var
appuser@be88621182fa:/opt/project/project/app$ pwd
/opt/project/project/app
appuser@be88621182fa:/opt/project/project/app$ ls -la
total 32
drwxr-xr-x 18 appuser appuser  576 Nov 18 18:42 .
drwxr-xr-x 15 appuser appuser  480 Nov 14 18:43 ..
-rw-r--r--  1 appuser appuser  341 Nov 16 21:58 .env
-rw-r--r--  1 appuser appuser  233 Nov 10 21:37 .env.template
drwxr-xr-x  5 appuser appuser  160 Nov  9 13:38 .mypy_cache
-rw-r--r--  1 appuser appuser   17 Nov  9 20:17 BUILD
-rw-r--r--  1 appuser appuser    0 Nov 16 18:22 __init__.py
drwxr-xr-x  7 appuser appuser  224 Nov 10 17:22 __pycache__
drwxr-xr-x  7 appuser appuser  224 Nov 18 18:36 auth
drwxr-xr-x  7 appuser appuser  224 Nov 16 19:07 common
-rw-r--r--  1 appuser appuser 3723 Nov 16 17:50 config.py
-rw-r--r--  1 appuser appuser  858 Nov 16 17:50 main.py
-rw-r--r--  1 appuser appuser    0 Nov 16 17:50 models.py
drwxr-xr-x  6 appuser appuser  192 Nov 16 17:50 patients
drwxr-xr-x  6 appuser appuser  192 Nov 16 16:10 schedules
-rw-r--r--  1 appuser appuser 2280 Nov 16 22:04 server.py
-rw-r--r--  1 appuser appuser  925 Nov 18 18:42 settings.toml
-rw-r--r--  1 appuser appuser  377 Nov 16 16:01 tags_metadata.json

So things are working, but I don't understand why they're set up this way. Like why does my application end up in /opt/project/project when run from the PyCharm debugger? I still need to add this:

PYTHONPATH=/opt/project/project

To the Run/Debug environment in order to run the app from PyCharm.

Doug

0
Comment actions Permalink

This is indeed very odd. 

Could you please try to edit docker-compose.yml and change the following:

dockerfile: ./Dockerfile 

to 

dockerfile: Dockerfile

Also, I mocked the last part of the docker building process with a dummy startup.sh with only one echo command. Could it be an issue inside that script? 

0
Comment actions Permalink

Hi Daniil,

I changed the docker-compose.yml file as you suggested:

dockerfile: ./Dockerfile   =>   dockerfile: Dockerfile

And everything runs as it did before. The application runs from the command line, and the project is in (from the root) /project. 

When run from the PyCharm debugger, it runs, and the project is in (from the root) /opt/project/project.

So effectively no change in behavior.

Doug

0
Comment actions Permalink

Hi Doug, 

I'm very sorry; I was able to reproduce the issue after deleting all previously created images and found where the problem is. 

At the moment, PyCharm creates a docker-compose.override.yml file which overrides your docker-compose.yml and makes an unexpected /opt/project dir with your project. 

To mitigate this issue, please turn off Help | Find Action | Registry | python.use.targets.api (as I remember from our other conversation, it should be turned off already) and re-create the interpreter. There should be a new option called Path mappings in the Project interpreter window, and the paths in the container should be correct.

0

Please sign in to leave a comment.