I'm facing a problem in my scrapy-splash spider for DEPTH_LIMIT not work.
# setting.py
# depth to crawler
#--------------------------------------------------
DEPTH_LIMIT = 5
DEPTH_STATS_VERBOSE = True
DEPTH_PRIORITY = 1
#--------------------------------------------------
here is my full code : https://github.com/scrapy-plugins/scrapy-splash/issues/304
What should I do to make DEPTH_LIMIT work well?
Related
I deployed a gitlab runner and a gitlab instance on the same server using docker, after that, I tried to run a few samples to test my runner, but in the first job it always tells me that it can't access my repository. The weird thing is that it try to access a totally different URL instaed of the one in config.toml.
Here is my config:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "******"
url = "http://172.17.0.3:8010/"
token = "*********"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "ubuntu"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
The http://172.17.0.3:8010/ is exact the ip of my gitlab instance inside the docker network.
Here is where the runner try to get my code:
Running with gitlab-runner 13.12.0 (7a6612da)
on third runner 2ieTUrD1
Preparing the "docker" executor
Using Docker executor with image ubuntu:focal ...
Pulling docker image ubuntu:focal ...
Using docker image sha256:7e0aa2d69a153215c790488ed1fcec162015e973e49962d438e18249d16fa9bd for ubuntu:focal with digest ubuntu#sha256:adf73ca014822ad8237623d388cedf4d5346aa72c270c5acc01431cc93e18e2d ...
Preparing environment
00:01
Running on runner-2ieturd1-project-12-concurrent-0 via dfcd09965d50...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/zh****/e****/.git/
fatal: unable to access 'http://8.136.221.242:8010/zh****/e****.git/': Failed to connect to 8.136.221.242 port 8010: Operation timed out
ERROR: Job failed: exit code 1
Can anyone help me, thank you so much!
Update your /etc/gitlab/gitlab.rb file and set
external_url "http://172.17.0.3:8010/"
Once you've saved it run
sudo gitlab-ctl reconfigure
i am trying to set up gitlab runner for a project, and when i try to run the build i am getting an 403 error.
Where can i check the build logs other than doing systemctl status gitlab-runner?
git-lab-runner Version: 9.5.1
build error:
Running with gitlab-ci-multi-runner 9.5.1 (96b34cc)
on ci (3aa5e67d)
Using Docker executor with image docker.xxx.com/terraform:latest ...
Using docker image sha256:134722953932eb772ab67a8b6e865aae2da7fe6ba3f09e757ea09e0d416ec203 for predefined container...
Pulling docker image docker.xxx.com/terraform:latest ...
Using docker image docker.xxx.com/terraform:latest ID=sha256:fxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx for build container...
Running on runner-3aa5e67d-project-184-concurrent-0 via ip-10-xx-xx-xx...
Cloning repository...
Cloning into '/builds/bi/abc'...
fatal: unable to access 'https://gitlab.xxxx.com/folder/abc.git/': The requested URL returned error: 403
ERROR: Job failed: exit code 1
runner config:
concurrent = 1
check_interval = 0
[[runners]]
name = "ci"
url = "http://gitlab.xxx.com"
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker.xxx.com/terraform:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
I had configured gitlab runner to a different project but it did not ask me to add the user to the members, how can i troubleshoot this error, i believe its an authentication error.
I'm trying to set the configuration file of my odoo server to certain file, I know that running odoo with the -c <path> or --config <path> will do the work, but I'm running it in the server like a service so I can't do this, neither adding the configs to /etc/odoo/odoo.conf because I need 2 configuration files.
Does someone knows how can I can make odoo service run with certain flags(-c and --load)
Here's my config at /etc/odoo/odoo.conf
[options]
addons_path = /usr/lib/python2.7/distpackages/odoo/addons,/opt/odoo/addons/odoodoto
admin_passwd = XXXXXXXXXXXX
data_dir = /var/lib/odoo
db_host = False
db_name = False
db_password = False
db_port = 5432
db_user = False
demo = {}
log_level = warn
logfile = /var/log/odoo/odoo-server.log
logrotate = True
proxy_mode = False
And my second config:
[connector-options]
workers = 4
export ODOO_CONNECTOR_CHANNELS=root:5
export ODOO_CONNECTOR_PORT=8069
log-level = warn
And the --load=web,connector is the other flag I need
Without getting into many details of the init systems of Ubuntu, you should have a bash script inside /etc/init.d (probably /etc/init.d/odoo-server).
Inside that file insert a line:
DAEMON_OPTS="-c /etc/odoo/odoo.conf"
Use commas to pass more parameters
The file I was looking for is /etc/systemd/system/odoo.service there you can specify in the [Service] options something like this ExecStart=/usr/local/bin/odoo --load=web,connector -c /somedir/odoo-server.conf.
You can also configure some of the service settings in /etc/init.d/odoo like George Daramouskas sayed but I don't really know how and if you can add the flags I wanted
When I try to run vagrant up I get the error:
There are errors in the configuration of this machine. Please fix
the following errors and try again:
SSH:
* `private_key_path` file must exist: /home/buildbot/mykey.pem
However, this file definitely exists. If I run ls -lah /home/buildbot/mykey.pem, it's there. It's owned by my user "buildbot". It has the right permissions. Everything looks good, but yet Vagrant can't see it, even though it's running as user "buildbot". Why would this be?
My Vagrantfile is a fairly generic one for AWS:
# -*- mode: ruby -*-
# vi: set ft=ruby :
require 'vagrant-aws'
Vagrant.configure(2) do |config|
config.vm.box = 'aws-dummy'
config.vm.provider :aws do |aws, override|
aws.keypair_name = 'my-key-pair'
aws.security_groups = ['my-security-group']
aws.access_key_id = ENV['AWS_ACCESS_KEY']
aws.secret_access_key = ENV['AWS_SECRET_KEY']
aws.ami = 'ami-43c92455'
override.ssh.username = 'ubuntu'
override.ssh.private_key_path = ENV['AWS_PRIVATE_KEY_PATH']
end
end
In vagrant documentation i did not found a hint on how to reference an included file from a included Vagrantfile within the same baseline box when using "vagrant package". Can anyone help?
Details:
When creating a new baseline box from scratch for vagrant, you are free to use the standard vagrant insecure ssh key or to create a custom new key. I did the last thing. And this new baseline box works fine with my custom key, when i use my Vagrantfile with this additional line:
config.ssh.private_key_path = "custom_key_file"
Now i decided to distribute my baseline box to my team members. Thats no problem. Just enter:
vagrant package --output custom.box
All other team members do copy the "custom_key_file" to the project root dir and create a "Vagrantfile" with this content (done using a version controll system):
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "custombox"
config.ssh.private_key_path = "custom_key_file"
end
When done, each team member enter the following to get a virtual machine based on custom.box fast and easy:
vagrant box add custombox custom.box
vagrant up
Works fine.
Now i want to tune my baseline box a little bit before distributing. I want to include the "custom_key_file" and a "Vagrantfile.pkg" that reads as follows:
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "custombox"
config.ssh.private_key_path = "custom_key_file"
end
To create the tuned baseline box i enter:
vagrant package --output custom2v.box --vagrantfile Vagrantfile.pkg --include custom_key_file
When i extract the custom2v.box i can see there is this tree:
C:.
│ box-disk1.vmdk
│ box.ovf
│ Vagrantfile
│
└───include
custom_key_file
_Vagrantfile
And "include/_Vagrantfile" has the content of my Vagrantfile.pkg. I can add that box as follows:
vagrant box add custombox2v custom2v.box
to a new project it is now very easy to enable it for vagrant. Just add a "Vagrantfile" read as follows:
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "custombox2v"
end
But when i do a:
vagrant up
i get the following error message:
[...]
Bringing machine 'default' up with 'virtualbox' provider...
There are errors in the configuration of this machine. Please fix
the following errors and try again:
SSH:
* `private_key_path` file must exist: custom_key_file
Can anyone help?
The reason is Vagrant's load order and merging of its configs.
What you want to happen is Vagrant to use the private key located inside the box archive.
What really happens when you run "up" is Vagrant merges your config with few others configs on its "load & merge" path.
So in the end of the way you have one big config with the setting:
config.ssh.private_key_path = "custom_key_file"
So Vagrant will look for custom_key_file in the same folder as your Vagrantfile and that's why you get your error.
Check this answer and this issue for information how to config Vagrant to look for the key relatively to box's Vagrantfile.