Script executed multiple times Follow

Hi,

This is an odd one for me, after a catastrophic blow up of my OS and a reinstall, I've got eveyrthing up and runing again and started retraining some of my models / testing them out.

However for some reason one of the projects / models has an odd behaviour in that when ever I use the Pycharm IDE Run / Debug button, it decides to run it multiple times.

Running Script
==> training model with cspn and unet on cables
==> Preparing data..
==> Prepare results folder and files...
==> Building model..
{'step': 24, 'kernel': 3, 'norm_type': '8sum'}
==> Resuming from best model..
output/nyu_pretrain_cspn_1_net_cp500_bs8_adlr_ep40_8norm_cables\best_model.pth
==> model dict with addtional module, remove it...
Running Main
  0%|          | 0/109 [00:00<?, ?it/s]
Running Script
==> training model with cspn and unet on cables
==> Preparing data..
==> Prepare results folder and files...
==> Building model..
{'step': 24, 'kernel': 3, 'norm_type': '8sum'}
==> Resuming from best model..
output/nyu_pretrain_cspn_1_net_cp500_bs8_adlr_ep40_8norm_cables\best_model.pth
==> model dict with addtional module, remove it...

It is hard to see, but the "Running Script" is the top init part, whilst the "Running Model" is in the 'main' function call and as you can see Pycharm just continiously runs the script multiple times until there is a GPU out of memory error

Traceback (most recent call last):
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\momiji\Documents\Github\CSPN\cspn_pytorch\train.py", line 194, in <module>
    net.cuda()
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 491, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
    module._apply(fn)
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
    module._apply(fn)
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 409, in _apply
    param_applied = fn(param)
  File "C:\Users\momiji\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 491, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA out of memory. Tried to allocate 200.00 MiB (GPU 0; 11.00 GiB total capacity; 1.33 GiB already allocated; 120.17 MiB free; 1.36 GiB reserved in total by PyTorch)

This is running on a CUDA compiled Pytorch.

I honestly can't understand how this behaviour is occuring :). The GPU is a RTX2080ti eGPU, but it ran fine last time, though I may have been using an Ubuntu WSL2 environment in that instance, not sure why that would matter though.

Any ideas on why the run commandn would execute multiple instances of the same script without telling it to?

Regards

Cdr