I have a selenium pytest project in pycharm . I want to export it to github to share it to a friend. What instructions should I give to run the project successfully .
Is installing pycharm needed to run the project .
If i generate requirements.txt file , will it be enough to install all the packages needed for running the project in my friends machine
please give a detailed explanation on what my friend should do
You don't need PyCharm.
Your friend will need a version of Python and an OS that is compatible with your program.
Once you have made the git repo and your friend has cloned the repo, have them type:
$ pip install -r requirements.txt
The command line invocation could be many things. You will need to write this part.
Related
How can I run the code in this repository without following their steps, I want to run the main.py file, from the Demo, directly.
What are the setup.py and install.sh files used for?
setup.py is the canonical name for the installer of a Python package. For example, when you run pip install x, pip runs the setup.py of the package you installed. In this case, install.sh is just a shortcut for running setup.py.
There's no way to use a package without installing it, since it won't have what it needs to operate properly.
I am using Mac for development. I installed Rust 1.13.0 using brew install rust and the Rust plugin 0.1.0.1385 for IntelliJ IDEA. I created my first test project with cargo and while opening it with IDEA I got the message
No standard library sources found, some code insight will not work
I haven't found any sources installed, nor the Rust sources package in Homebrew.
How do I provide sources for the project and what are the practical implication if I ignore this step?
As commented, the supported approach is to use rustup:
Navigate to https://rustup.rs/ and follow the installation instructions for your platform.
Add the rust-src component by running: rustup component add rust-src
Create a new Rust project in IntelliJ and choose your existing Rust project source. If the folder already contains previous IntelliJ project files, you may have to delete those first before it will let you proceed.
IntelliJ-Rust should automatically configure the standard library sources to point to the sources downloaded by rustup.
As a reference, since the question title is broad, for Fedora 28 I had to:
dnf install cargo rust-src
sudo ln -s /usr/lib/rustlib/src /usr/lib/rustlib/x86_64-unknown-linux-gnu/
then give /usr/lib/rustlib/x86_64-unknown-linux-gnu/src/rust/src as "Standard library"
Full setup:
Issue opened to simplify the process
When not using the rustup installer, one can install the source package and direct the rust plugin to use those:
(Tested with CLion 2020.2.1, rust-1.46.0-x86_64-pc-windows-gnu.msi, rustc-1.46.0-src.tar.gz. Offline Rust installers and source archive from there: https://forge.rust-lang.org/infra/other-installation-methods.html )
Although the preferred way of installing Rust is by using rustup, as pointed out by the other posts, it is not uncommon to use the packages that your distro makes available.
I use, for example, the packages provided by Gentoo and I share the same problem about the not prefilled field for standard libraries.
Nevertheless, you can easily find out where your standard libraries have been installed by typing the following find command:
find /usr/lib* -type d -name "rust" | grep src
or the following if you installed rust in your home
find -type d -name "rust" | grep src
The previous commands will help, unless, of course, in your distro there is a package for the binaries and one for the source and you only installed the binary one.
I know the question is for MacOS but this answer is shown up when searching for it on Linux. Below I will answer for Ubuntu.
The path is /usr/lib/rustlib/src/rust/src for Ubuntu 20.04
The way I did is:
Installed rustc from the repositories, which includes cargo
sudo apt install rustc
Then installed rust source package
sudo apt install rust-src
I used apt-file (can be installed with sudo apt install apt-file) to search for the install path of the sources
sudo apt-file update
apt-file list rust-src
This show the path as /usr/src/rustc-1.41.0/src .
But a ls -la in /usr/lib/rustlib/ will reveal symlinks and /usr/lib/rustlib/src/rust/src points to the previous found directory.
Using the symlink on IntelliJ will survive new rust versions.
For Fedora 32 install Rust using command:
dnf install cargo rust-src
and the path to standard libary source is:
/usr/lib/rustlib/src/rust
I used Ubuntu. I follow these steps:
sudo apt install rust-src
wait for the install, then
dpkg -L rust-src
copy the last line. For me it is the standard library path:
/usr/lib/rustlib/src/rust
For MacOS, you need to put /opt/homebrew/bin/.
During installation I received the following error:
This script will setup the CNTK prequisites and the CNTK Python
environment onto the machine. More help is given by calling 'get-help
.\install.ps1' in your powershell environment.
The script will analyse your machine and will determine which
components are required. The required components will be downloaded in
[C:\local\Scripts\windows\InstallCache] Repeated operation of this
script will reuse already downloaded components.
If required VS2012 Runtime and VS2013 Runtime will be installed
If required MSMPI will be installed
If required the standard Git tool will be installed
CNTK source will be cloned from Git into [c:\repos\CNTK]
Anaconda3 will be installed into [C:\local\Anaconda3-4.1.1-Windows-x86_64]
A CNTK-PY34 environment will be created in [C:\local\Anaconda3-4.1.1-Windows-x86_64\envs]
CNTK will be installed into the CNTK-PY34 environment
1 - I agree and want to continue Q - Quit the installation process
1 Determining Operations to perform. This will take a moment...
The following operations will be performed: * Setup/Update CNTK Wheel
* Clone CNTK from Github repository
Do you want to continue? (y/n) y Performing download operations
Download operations finished
Performing install operations Setup/Update of CNTK Wheel environment.
Please be patient.... You are using pip version 8.1.2, however version
9.0.0 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. Cloning CNTK (branch v2.0.beta2.0)
repository....
Fatal error during script execution!
System.InvalidOperationException: This command cannot be run due to
the error: The system cannot find the file specifie d. at
System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord
errorRecord)
PS C:\local\Scripts\windows>
I'm not familiar with powershell, so I am not sure where to go from here.
Couple of questions. Are you using beta1 or beta2?
If you are using beta1, just a reminder that there is a beta2 now available with bug fixes and also some improvements to the install script, but I don't think this particular problem has been addressed!
It looks like the clone of the cntk git repository is failing.
Was Git installed on your machine? Is there a GIT.EXE in c:\Program Files\Git\bin\?
I think the install found an existing git.exe (in a different location), and is now trying to call it at the 'wrong' location.
The easiest workaround for you (if you have git installed), from a command prompt:
cd c:/
md \repos
cd repos
clone --branch v2.0.beta1.0 --recursive https://github.com/Microsoft/CNTK
If you have moved to beta2, replace the v2.0.beta1.0 with v2.0.beta2.0
Does anyone have a fully compiled version of pandas that is compatible with AWS Lambda?
After searching around for a few hours, I cannot seem to find what I'm looking for and the documentation on this subject is non-existent.
I need access to the package in a lambda function however I have been unsuccessful at getting the package to compile properly for usage in a Lambda function.
In lieu of the compilation can anyone provide reproducible steps to create the binaries?
Unfortunately I have not been able to successfully reproduce any of the guides on the subjects as they mostly combine pandas with scipy which I don't need and adds an extra layer of burden.
I believe you should be able to use the recent pandas version (or likely, the one on your machine). You can create a lambda package with pandas by yourself like this,
First find where the pandas package is installed on your machine i.e. Open a python terminal and type
import pandas
pandas.__file__
That should print something like '/usr/local/lib/python3.4/site-packages/pandas/__init__.py'
Now copy the pandas folder from that location (in this case '/usr/local/lib/python3.4/site-packages/pandas) and place it in your repository.
Package your Lambda code with pandas like this:
zip -r9 my_lambda.zip pandas/
zip -9 my_lambda.zip my_lambda_function.py
You can also deploy your code to S3 and make your Lambda use the code from S3.
aws s3 cp my_lambda.zip s3://dev-code//projectx/lambda_packages/
Here's the repo that will get you started
After some tinkering around and lot's of googling I was able to make everything work and setup a repo that can just be cloned in the future.
Key takeaways:
All static packages have to be compiled on an ec2 amazon Linux instance
The python code needs to load the libraries in the lib/ folder before executing.
Github repo:
https://github.com/moesy/AWS-Lambda-ML-Microservice-Skeleton
The repo mthenw/awesome-layers lists several publicly available aws lambda layers.
In particular, keithrozario/Klayers has pandas+numpy and is up-to-date as of today with pandas 0.25.
Its ARN is arn:aws:lambda:us-east-1:113088814899:layer:Klayers-python37-pandas:1
I know the question was asked a couple years ago and Lambda was on a different stage back then.
I faced similar issues lately and I thought it would be a good idea to add the newest solution here for future users facing the same problem.
It turns out that amazon released the concept of layers in the re:Invent 2018. It is a great feature. This post in medium describes it much better than I could here: Creating New AWS Lambda Layer For Python Pandas Library
The easiest way to get pandas working in a Lambda function is to utilize Lambda Layers and AWS Data Wrangler. A Lambda Layer is a zip archive that contains libraries or dependencies. According to the AWS documentation, using layers keeps your deployment package small, making development easier.
The AWS Data Wrangler is an open source package that extends the power of pandas to AWS services.
Follow the instructions (under AWS Lambda Layer) here.
Another option is to download the pre-compiled wheel files as discussed on this post: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-package-compatible/
Essentially, you need to go to the project page on https://pypi.org and download the files named like the following:
For Python 2.7: module-name-version-cp27-cp27mu-manylinux1_x86_64.whl
For Python 3.6: module-name-version-cp36-cp36m-manylinux1_x86_64.whl
Then unzip the .whl files to your project directory and re-zip the contents together with your lambda code.
NOTE: The main Python function file(s) must be in the root folder of the resulting deployment package .zip file. Other Python modules and dependencies can be in sub-folders. Something like:
my_lambda_deployment_package.zip
├───lambda_function.py
├───numpy
│ ├───[subfolders...]
├───pandas
│ ├───[subfolders...]
└───[additional package folders...]
#ashtonium's answer actually works and is most likely the easiest, however, a few additional steps are required. Also, Pandas requires Pytz (mentioned in the link provided by #b3rt0) so that package is needed as well.
Download the whl-files from PyPI (the Pandas file ends with ...manylinux1_x86_64.whl, there is only one Pytz file of relevance)
Unzip the whl-files using terminal command, e.g. unzip filename.whl (Linux/MacOS)
Create a new folder structure python/lib/python3.7/site-packages/ (swap 3.7 for version of your choice)
Move folders from step 2 to site-packages folder in step 3
Zip root folder in new structure, i.e. python
Create a new layer in AWS management console where you upload the zip-file
This is a very common question, I hope my solution helps.
Update on Aug 19, 2020:
Wheel-files aren't available for all packages. In these cases you can skip to step 3, go into the site-packages folder and install the package in there with pip3 install PACKAGE_NAME -t . (no venv required). Some packages are easier than others, some are trickier. Psycopg2 for example, requires you to move only one of the two (as of this writing) package folders.
/Cheers
There are some precompiled packages on github by ryfeus.
My solution has been to maintain 2 requirements.txt style files of packages that go in my layer, one named provided_packages.txt and one named provided_linux_installs.txt
Before deployment (if the packages are not already installed) I run:
pip install -r provided_packages.txt -t layer_name/python/lib/python3.8/site-packages/.
pip download -r provided_linux_installs.txt --platform manylinux1_x86_64 --no-deps -d layer_name/python/lib/python3.8/site-packages
cd layer_name/python/lib/python3.8/site-packages
unzip \*.whl
rm *.whl
Then deploy normally (I am using cdk synth & cdk deploy \* --profile profile_name)
In case helpful, my provided_linux_installs.txt looks like this:
pandas==1.1.0
numpy==1.19.1
pytz==2020.1
python-dateutil==2.8.1
I have started to maintain a GitHub repo for easy and quick access to layers. https://github.com/kuharan/Lambda-Layers
I have been using these for my open-source projects and stuff.
I managed to deploy a pandas code in aws lambda using python3.6 runtime . this is the step that i follow :
Add required libraries into requirements.txt
Build project in a docker container (using aws sam cli : sam build --use-container)
Run code (sam local invoke --event test.json)
this is a helper : https://github.com/ysfmag/aws-lambda-py-pandas-template
# all the step are done in AWS EC2 Linux Free tier so that all the Libraries are compatible with the Lambda environment
# install the required packages
mkdir packages
pip3 install -t . pandas
pip3 install -t . numpy --upgrade
pip3 install -t . wikipedia --upgrade
pip3 install -t . sklearn --upgrade
pip3 install -t . pickle-mixin --upgrade
pip3 install -t . fuzzywuzzy --upgrade
# Now remove all unnecessary files
sudo rm -r *.whl *.dist-info __pycache__
# Now make a DIR so that lambda function can reconginzes
sudo mkdir -p build/python/lib/python3.6/site-packages
# Now move all the files from packages folder to site-packages folder
sudo mv /home/ec2-user/packages/* build/python/lib/python3.6/site-packages/
# Now move to the build packages
cd build
# Now zip all the files starting from python folder to site-packages
sudo zip -r python.zip .
upload the zip file to lambda layers
python 3.8 windows 10 lambda aws pandas
You need to do the following steps on a linux machine and python 3.8:
sudo mkdir python
sudo pip3 install --target python pandas
sudo zip -r pandas.zip python
create a public s3 bucket, upload pandas.zip, grab the public URL.
create new lambda layer using s3 URL from above.
add layer to lambda function and import pandas as pd like you normally would
No linux machine? Launch an Ubuntu EC2 instance or container:
sudo apt install python3.8 zip unzip python3-pip
run 1-3 above
Now you need to copy the zip to your local machine. Open a command terminal and change directory to the folder containing your EC2 instance's pem file and run: scp -i yourPemFile.pem ubuntu#'EC2.Instance.IP.Here':/home/path/to/pandas.zip C:\Users\YourUser\Desktop
run steps 4-6 from above
*for number 3 above: you need to grab your EC2 IP and insert it. You may get an error about the permissions on the pem file, if you do then right click the pem file > properties > security > advanced > disable inheritance and make sure only your user is in the "permission entries." Lastly, fix the paths to point to where the pandas.zip file is on the EC2 instance and where you want the file to end up locally.
**pay attention to the python runtime of the lambda function. Make sure it matches the version of python you're using to do the pip stuff (which should be 3.8).
***the original folder name "python" is named that for a reason as per AWS documentation.
After lots of googling on this and messing around, the concept of layers are great and seem to work for me.
This github repo from keithrozario has loads of pre-build layers you can simply add to your lambda via the arn which has some great stuff in there like pandas, requests and sqlalchemy.
I've create a template to compile and upload a layer (containing python dependencies) to lambda using the AWS CLI which you can find in my Gitlab repo here.
I'm running this on an Amazon Linux EC2, using a virtual environment (venv) to install libraries from a requirements.txt file and then load the zipped files to lambda using the AWS CLI.
Note the folder structure my_zip_file/python/binaries which is required for lambda.
Note: Pandas is quite a large library. Your zipped layer file must be below 70mb.
You may also encounter the horrible "OpenBLAS WARNING - could not determine the L2 cache size on this system" error message. I had to increase the memory from the default 128mb in order to the lambda to successfully run.
After searching around for a few hours, I cannot seem to find what I’m looking for and the documentation on this subject is non-existent.
So i decided to build the libraries myself to support the Amazon Linux 2 arch.
Read full blog here https://khanakia.medium.com/add-pandas-and-numpy-python-to-aws-lambda-layers-python-3-7-3-8-694db42f6119
I am using IntelliJ IDEA 12 Ultimate Edition and creating flask project.
I created the virtualenv using IDEA and using that, but my code has dependency on other libraires I as move forward. For example Flask-Restless.
My code in IntelliJ IDEA looks like
Is there a way to install Flak-Restless using IntelliJ IDEA 12?
or
do I need to activate my virtualenv on command-line and install it myself?
Is it something IDEA can provide to me?
In IntelliJ IDEA use Tools | Manage Python Packages dialog to install/uninstall packages for your Python SDK or virtualenv used in project.
Use pip requirements.txt in your repository root. My PyCharm automatically prompt me install absent requirements or if installed versions not equal with requirements.txt.
You can install packages from requirements.txt:
your_python_root_pip install -r requirements.txt
You can get already installed packages with versions:
your_python_root_pip freeze -r requirements.txt
For details see pip help. See requirements.txt example:
flask==0.9
flask-testing==0.4
blinker==1.2
uwsgi==1.4.5
nose
coverage
pep8
You can install all project packages via PyCharm 2017.1 by Tools / Python Integrated Tools / Package requirements file. Get there full path to your requirements.txt file and PyCharm will ask you to install all dependencies.