I am trying to deploy some spiders on Scrapy Cloud, I am getting this message at the end:
Error: Deploy failed: b''
The spiders appear in the dashboard but don't run (doesn't even start to run).
I have this in requirements.txt:
bs4==0.0.1
boto==2.49.0
And this is what scrapinghub.yml:
project: <my project ID>
stacks:
default: scrapy:1.7-py3
requirements:
file: requirements.txt
Related
Hello~ I am trying to use Github workflow to run the unit test for the code in my repository.
So I wrote a yaml file, its function is when I push my code to my repository, it can let me use my local environment to execute my code on Github, and the purpose of these codes is to run the unit tests.
But I can't run this workflow successfully, this error message always appears. I'm curious why I can execute unit test successfully on the local IDE, but it doesn't work when I use workflow to automatically execute it for me.
============================== warnings summary ===============================
..\..\..\..\anaconda3\lib\site-packages\pyreadline\py3k_compat.py:8
C:\Users\COA\anaconda3\lib\site-packages\pyreadline\py3k_compat.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
return isinstance(x, collections.Callable)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ===========================
ERROR test/test_createds_for_fnn.py - AttributeError: type object 'h5py.h5.H5...
ERROR test/test_fnn.py - AttributeError: type object 'h5py.h5.H5PYConfig' has...
ERROR test/test_unet.py - AttributeError: type object 'h5py.h5.H5PYConfig' ha...
!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!
======================== 1 warning, 3 errors in 2.76s =========================
Error: Process completed with exit code 1.
My environment:
Windows 10 OS
Python 3.8.8
Tensorflow-gpu 2.7.0
pytest 6.2.3
h5py 3.6.0
Workflow
# This is a basic workflow to help you get started with Actions
name: CI
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the master branch
push:
branches: [ main ]
pull_request:
branches: [ main ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: self-hosted
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v3
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo Hello, world!
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
echo Add other actions to build,
echo test, and deploy your project
echo train the fnn
pytest
I get the following ERROR when I try to run scripts with brownie, using the following PowerShell command;
brownie run scripts/simple_collectible/deploy_simple
I have look all over stacked and other pages for info on this and I can't seem to find much, I would really like to carry on with my project but I am stuck at this point. any hekp would be wonderful.
Cheers!
INFO MESSAGE:
PS C:\Users\charl\OneDrive\Desktop\NFT Development\NFT-mix-main> brownie run scripts/simple_collectible/deploy_simple
INFO: Could not find files for the given pattern(s).
Brownie v1.17.2 - Python development framework for Ethereum
NftMixMainProject is the active project.
Launching 'ganache-cli.cmd --port 8545 --gasLimit 12000000 --accounts 10 --hardfork istanbul --mnemonic brownie'...
File "C:\Users\charl.local\pipx\venvs\eth-brownie\lib\site-packages\brownie_cli_main_.py", line 64, in main
importlib.import_module(f"brownie._cli.{cmd}").main()
File "C:\Users\charl.local\pipx\venvs\eth-brownie\lib\site-packages\brownie_cli\run.py", line 46, in main
path, _ = _get_path(args[""])
File "C:\Users\charl.local\pipx\venvs\eth-brownie\lib\site-packages\brownie\project\scripts.py", line 130, in _get_path
raise FileNotFoundError(f"Cannot find {path_str}")
FileNotFoundError: Cannot find scripts/simple_collectible/deploy_simple
Terminating local RPC client...
I have the following packages installed:
ganache-cli
pip
pipx
Brownie (installed through pipx, and initialized)
I have run the brownie command to make sure the install is good.
I have install Python Venv
I have tried uninstalling all packaged and reinstalling
I have done the same with my VScode and Vsbuildtools
I have done the same with Python itself (reinstalled from the website)
The code snippet I have for my Script that I am trying to run is here:
#!/usr/bin/python3
import os
from brownie import SimpleCollectible, accounts, config, network
def main():
dev = accounts.add(config["wallets"]["from_key"])
print(network.show_active())
publish_source = True if os.getenv("ETHERSCAN_TOKEN") else False
SimpleCollectible.deploy({"from": dev}, publish_source=publish_source)
And finally for your reference I have my brownie-config.yaml contents here:
# exclude SafeMath when calculating test coverage
# https://eth-brownie.readthedocs.io/en/v1.10.3/config.html#exclude_paths
reports:
exclude_contracts:
- SafeMath
dependencies:
- smartcontractkit/chainlink-brownie-contracts#1.1.1
- OpenZeppelin/openzeppelin-contracts#3.4.0
compiler:
solc:
remappings:
- '#chainlink=smartcontractkit/chainlink-brownie-contracts#1.1.1'
- '#openzeppelin=OpenZeppelin/openzeppelin-contracts#3.4.0'
# automatically fetch contract sources from Etherscan
autofetch_sources: True
dotenv: .env
# set a custom mnemonic for the development network
networks:
default: development
kovan:
vrf_coordinator: '0xdD3782915140c8f3b190B5D67eAc6dc5760C46E9'
link_token: '0xa36085F69e2889c224210F603D836748e7dC0088'
keyhash: '0x6c3699283bda56ad74f6b855546325b68d482e983852a7a82979cc4807b641f4'
fee: 100000000000000000
oracle: '0x2f90A6D021db21e1B2A077c5a37B3C7E75D15b7e'
jobId: '29fa9aa13bf1468788b7cc4a500a45b8'
eth_usd_price_feed: '0x9326BFA02ADD2366b30bacB125260Af641031331'
rinkeby:
vrf_coordinator: '0xb3dCcb4Cf7a26f6cf6B120Cf5A73875B7BBc655B'
link_token: '0x01be23585060835e02b77ef475b0cc51aa1e0709'
keyhash: '0x2ed0feb3e7fd2022120aa84fab1945545a9f2ffc9076fd6156fa96eaff4c1311'
fee: 100000000000000000
oracle: '0x7AFe1118Ea78C1eae84ca8feE5C65Bc76CcF879e'
jobId: '6d1bfe27e7034b1d87b5270556b17277'
eth_usd_price_feed: '0x8A753747A1Fa494EC906cE90E9f37563A8AF630e'
mumbai:
eth_usd_price_feed: '0x0715A7794a1dc8e42615F059dD6e406A6594651A'
binance:
# link_token: ??
eth_usd_price_feed: '0x9ef1B8c0E4F7dc8bF5719Ea496883DC6401d5b2e'
binance-fork:
eth_usd_price_feed: '0x9ef1B8c0E4F7dc8bF5719Ea496883DC6401d5b2e'
mainnet-fork:
eth_usd_price_feed: '0x5f4eC3Df9cbd43714FE2740f5E3616155c5b8419'
matic-fork:
eth_usd_price_feed: '0xF9680D99D6C9589e2a93a78A04A279e509205945'
wallets:
from_key: ${PRIVATE_KEY}
from_mnemonic: ${MNEMONIC}
# You'd have to change the accounts.add to accounts.from_mnemonic to use from_mnemonic
Change scripts/simple_collectible/deploy_simple -> scripts/simple_collectible/deploy_simple.py
Working on a React project.
I start up my environment using npm start. I then start my testing infrastructure using npm test in a different terminal. While my tests are running, errors are generated and printed in the terminal that my environment is running in.
npm start
Compiled successfully!
You can now view v2 in the browser.
Local: http://localhost:3000
Note that the development build is not optimized.
To create a production build, use yarn build.
Open another terminal and run npm test
Test Suites: 1 passed, 1 total
Tests: 3 passed, 3 total
Snapshots: 0 total
Time: 2.415s
Ran all test suites.
Yet, some UI errors that the tests generate appear in the terminal for my local environment
Compiled successfully!
You can now view v2 in the browser.
Local: http://localhost:3000
Note that the development build is not optimized.
To create a production build, use yarn build.
[HPM] Error occurred while trying to proxy request /directory from localhost:3000 to <TARGET> (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors)
[HPM] Error occurred while trying to proxy request /directory from localhost:3000 to <TARGET> (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors)
[HPM] Error occurred while trying to proxy request /directory from localhost:3000 to <TARGET> (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors)
[HPM] Error occurred while trying to proxy request /directory from localhost:3000 to <TARGET> (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors)
[HPM] Error occurred while trying to proxy request /directory from localhost:3000 to <TARGET> (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors)
How do I prevent errors - and anything else for that matter - from being printed out to the terminal where I started my dev environment?
looking at the output of your npm test, it seems like you are using jest, which has a support for configuring setup files
A list of paths to modules that run some code to configure or set up the testing framework before each test file in the suite is executed.
you can utilize it by pointing to a file, where you can mock the implmentation for console.
add to jest.config.js the following
module.exports = {
setupFilesAfterEnv: ['./jest.setup.js'],
};
then in jest.setup.js file
global.beforeAll(() => {
// disable console.error for clarity
console.error = jest.fn().mockImplementation(() => {});
});
note that it will achieve what you desire only if the module that prints to the console during the testing is using console.error(). otherwise, you might need to mock other functions, such as console.info(), console.log(), console.debug()
I'm tring to install apache ambari on my wsl(ubuntu 20.04) as the Ambari User Guides step by step. while install and packing the project to deb files use command:
mvn -B clean install jdeb:jdeb -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" .
got this error
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-timelineservice: An Ant BuildException has occured: Can't get https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz to /root/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded/hbase.tar.gz
[ERROR] around Ant part ...<get usetimestamp="true" src="https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz" dest="/root/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded/hbase.tar.gz"/>... # 5:273 in /root/apache-ambari-2.7.5-src/ambari-metrics/ambari-metrics-timelineservice/target/antrun/build-Download HBase.xml
apache-ambari-2.7.5-src/ambari-metrics/pom.xml defined the nortonworks HDP sources:
<hbase.tar>http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-2.0.0.3.0.0.0-1634</hbase.folder>
<hadoop.tar>http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/hadoop/hadoop-3.1.0.3.0.0.0-1634.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-3.1.0.3.0.0.0-1634</hadoop.folder>
<phoenix.tar>http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/phoenix/phoenix-5.0.0.3.0.0.0-1634.tar.gz</phoenix.tar>
<phoenix.folder>phoenix-5.0.0.3.0.0.0-1634</phoenix.folder>
I tried to download official Hbase-2.3.2, Hadoop-3.3.0,phoenix-5.0.0-HBase-2.0 to instead hotonworks HDP, but failed and got an other error.
I tried to download hortonworks HDP directly use wget and got:
Resolving dev.hortonworks.com.s3.amazonaws.com (dev.hortonworks.com.s3.amazonaws.com)... 52.217.40.204
Connecting to dev.hortonworks.com.s3.amazonaws.com
(dev.hortonworks.com.s3.amazonaws.com)|52.217.40.204|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
where/how can i download these hortonworks HDP files and continue to install ambari?
This solution from the Apache Ambari team fixed this and other similar issues with HDP urls: https://github.com/apache/ambari/pull/3283
You need to make the corresponding code changes: https://github.com/apache/ambari/pull/3283/commits/3dca705f831383274a78a8c981ac2b12e2ecce85
replace the downlink with
<hbase.tar>https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.1-1/tars/hbase/hbase-2.0.2.3.1.4.1-1-bin.tar.gz</hbase.tar>
<hbase.tar>https://private-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.1-1/tars/hbase/hbase-2.0.2.3.1.4.1-1-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-2.0.2.3.1.4.1-1</hbase.folder>
<hadoop.tar>https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.1-1/tars/hadoop/hadoop-3.1.1.3.1.4.1-1.tar.gz</hadoop.tar>
<hadoop.tar>https://private-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.1-1/tars/hadoop/hadoop-3.1.1.3.1.4.1-1.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-3.1.1.3.1.4.1-1</hadoop.folder>
<grafana.folder>grafana-6.4.2</grafana.folder>
<grafana.tar>https://dl.grafana.com/oss/release/grafana-6.4.2.linux-amd64.tar.gz</grafana.tar>
<phoenix.tar>https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.1-1/tars/phoenix/phoenix-5.0.0.3.1.4.1-1.tar.gz</phoenix.tar>
<phoenix.tar>https://private-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.1-1/tars/phoenix/phoenix-5.0.0.3.1.4.1-1.tar.gz</phoenix.tar>
<phoenix.folder>phoenix-5.0.0.3.1.4.1-1</phoenix.folder>
edit file ambari-metrics/pom.xml and replace download link, for example I changed
https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz
for
https://downloads.apache.org/hbase/2.3.5/hbase-2.3.5-src.tar.gz
HDP version: 3.1.0.78, using ambari 2.7.3 to install.
yarn version: 3.1.1
The application is finished with failed status.
Here is the complete log:
yarn logs -applicationId application_1565920695836_0005
19/08/16 11:49:57 INFO client.AHSProxy: Connecting to Application History server at dn1003.jja.bigo/10.221.117.44:10300
Not a valid BCFile.
Can not find any log file matching the pattern: [ALL] for the application: application_1565920695836_0005
Can not find the logs for the application: application_1565920695836_0005 with the appOwner: hadoop
Thanks for your reply.
UPDATE:
After using debug with export YARN_ROOT_LOGGER="DEBUG,console", it says "/data1/app-logs/hadoop/logs/application_1566555356033_0034/meta" is not valid bcfile.