Scrapy picks up settings from settings.py (there are default settings, project settings, per-spider settings as well). What I'm looking for is being able to have more than one file with settings and being able to switch between them as I launch my spiders quickly. If there is some inheritance between files that would be awesome too.
If you know Spring Boot from Java world there is an idea of profile. You have application.settings file with your base settings. And then you can have application-dev.settings and application-prod.settings. If you run your application with option -Dspring.profiles.active=dev then it will pick up application.settings and add application-dev.settings on top of it. This way you can maintain multiple configurations in parallel and rapidly switch between them.
I've found an approach for Scrapy with no supporting code required. The approach is to use SCRAPY_SETTINGS_MODULE and import base settings file in my dev and prod modules. Are there any other approaches that you use?
Launch line in my case would look like:
export SCRAPY_SETTINGS_MODULE=projectname.profiles.dev && scrapy crawl myspider
Firstly, if you're only going to change one or two values, then it would be simpler to use a single dynamic settings.py (as mentioned in Gallaecio's answer).
However, if you really need separate settings, there is an even shorter way by defining separate "projects" in scrapy.cfg (docs):
[settings]
default = myproject.settings.dev
dev = myproject.settings.dev
prod = myproject.settings.prod
Then to run a specific one:
SCRAPY_PROJECT=prod scrapy crawl myspider
SCRAPY_PROJECT=dev scrapy crawl myspider
If you don't specify SCRAPY_PROJECT it will use default.
And yes, you can inherit from settings files. Replace your settings.py file with a module instead:
myproject/settings/__init__.py
myproject/settings/base.py
myproject/settings/dev.py
myproject/settings/prod.py
In base.py you can have exactly what you have in settings.py. Then at the top of each override file you add:
from .base import *
# Override settings in the same way as if they were declared in settings.py
That wildcard import is usually a bad practice, but in this case since it's just a plain Python file so the end result is just having all the variables available. This is a trick we often use in Django (example).
I believe SCRAPY_SETTINGS_MODULE is the best approach.
Alternatively, since a settings module is a Python script, you could change settings dynamically from within settings.py. I’ve seen this done, for example, to detect automatically whether a spider is running in a local machine or on a Scrapyd server, and adjust the settings accordingly at run time.
Related
When I am editing a standalone Python file in VSCode (not part of a workspace) I will often need to alter the value of python.pythonPath to reflect a specific virtualenv I am using to run that code.
As the setting is just for the one file, I don't want to change my persistent global settings, and I don't have workspace settings. Is there a way to change a setting just for this session? (Ideally, just for this one file, but I don't expect that to be possible, so I'd be happy with "just for the session"). If there isn't a built in way to do this, is there an extension which allows this? Or even an extension API that I could use to write my own extension for this?
As an alternative, is there a way to use an environment variable in a setting, and then set that environment variable for the current VSCode process? That would have the same effect, it would just require me to set up my user settings specifically to allow this usage.
If you launch vscode from the terminal after having activated your virtualenv, vscode will automatically use the aforementioned virtualenv (with no modification to your settings):
Exemple:
source venv/bin/activate
code .
Note: if vscode is already opened, use code -n . in order to open the file/folder in a new window.
I have a run configuration in PyCharm with script parameters that take in a directory. Then, I have several directories (a changing number) in my project which I would like to easily be able to run this configuration on. Is there anyway to add an option to the right-click menu of directories to run the configuration passing that directory to the configuration? Or some other method which provides similar accessibility to running the configuration on a directory?
For my specific problem, I have many log directories for TensorBoard (from TensorFlow) and I would like to selectively and easily be able to start up an instance of TensorBoard running on a given directory.
Basically if you would run it in pycharm You can write your own plugin to add some functionality to the IDE. There is a great documentation on how to create plugins in idea/pycharm:
http://www.jetbrains.org/intellij/sdk/docs/basics/getting_started.html
We have done it before and it was a successful plugin, it really sped up the development processs. :)
Is there mechanism to import local settings to codeception.yml configuration file?
Our developers have their own databases and maybe some other environment specific settings for the testing, so these settings shouldn't be pushed to GIT. Normally we solve this problem by having two configuration files, where other one is global & in GIT. Another, local one is merged in some way or another to global one and ignored by the git. What I like to achieve would be following structure:
codeception.yml - global settings
codeception.local.yml - local settings witch would be merged to global settings e.g. by the import
There is a include property for config files, but it seems to deal with complete testing suites.
Yes actually. I can't find anything the manual about it, so I dug through the source code and it looks like it reads both codeception.yml and codeception.dist.yml files.
I tested it out it looks like codeception.dist.yml is your global config, and codeception.yml is what you should use for your local and put into .gitignore. Same goes for the suite.yml files. acceptance.suite.dist.yml is the global that gets overwritten by acceptance.suite.yml, etc etc.
my Play using mysql in production. But I am trying to use memory for testing.
I created 2 conf file, 1 is application.conf, the other is application.test.conf (in the same directory).
I tried to do
play -Dconfig.file=conf/application.test.conf test-only
But it still use the default conf file.
I'm just wonder if anyone know how to use a different conf file during testing. (or at least use a different database setting during testing).
If you mean for unit tests then just add
running(FakeApplication(additionalConfiguration = inMemoryDatabase())) { Test code... }
to your tests and they will be done in memory. No need to change conf files.
how to customize the durandal optimizer that it works on file named setup instead of main.js, also how to optimize more than one boot file in case of more than one SPA in a web app?
As of right now you cannot tell the optimizer to use setup.js instead of main.js. There is no hook into the options for that but the project is open source! hurray! And you can pull down the code and change the configuration and build it for your own custom optimizer.
In the Optimizer project for durandal you can change where it finds the main path here.
If I understand your question correctly I think you might be in luck for optimizing more than one boot file for multiple spas. The optimizer does have a configuration called source used like so:
optimizer.exe --source c:\project\spa1
optimizer.exe --source c:\project\spa2
Where spa1 and spa2 are paths to the folder which contains your application source. Which is the root folder for each application where the setup.js files are located.