'ImportError: No module named <project_name>.settings' in smoke test - scrapy

Question

scrapy crawl <spider_name> works fine in my project.
In a smoke test I crawl a private page and assert the items are as expected.
Things were fine until I finished a serious project refactor, which for the smoke test ends up setting an env var in setUp(), which in effect sets os.environ['SCRAPY_SETTINGS_MODULE'] = '<project_name>.settings'
After setting this env var, scrapy crawl complains that 'ImportError: No module named <project_name>.settings'
scrapy.cfg settings are correct, every thing else except this smoke test works fine.
The problem goes away simply by os.environ.pop('SCRAPY_SETTINGS_MODULE') before os.sys('scrapy crawl <spider_name>'), but this doesn't feel right.
So, please someone let me know whtat caused the problem and what is the proper way to fix it, thanks!

Answer 1

I found out the reason:
Before scrapy crawl scrapy executes scrapy.utils.project.get_project_settings, where scrapy.utils.conf.init_env is called if os.environ['SCRAPY_SETTINGS_MODULE'] is not set, where project main dir is added to sys.path.
By setting this env var, the copy of sys.path scrapy gets does not have the path for project main dir, so the import fails.
Popping out os.environ['SCRAPY_SETTINGS_MODULE'] will cause init_env to be called, thus scrapy crawl works as supposed. Another way to get around would be to set os.environ['PYTHONPATH'] to project main dir path, which is where scrapy looks later for the path.
The approach I finally take is to modify scrapy, disable the check for os.environ['SCRAPY_SETTINGS_MODULE'], and always execute init_env. The worst cost for this is that every time scrapy crawl executes, scrapy will look for the project main dir path once, which does not seem like much work to me. And the path will not be added multiple times to sys.path because it is checked before appending.
That's about it, please let me know if I am mistaken somewhere, thanks!