Can I run Scrapy on the free level of PythonAnywhere? I've looked, but haven't found instructions for installing it there.
If it can't be run on the free level of PythonAnywhere, is there another online environment where I can run Scrapy without needing to install Python and Scrapy on my computer?
EDIT: My question was just about PythonAnywhere, but in finding the answer to the question, I came across Cloud9 and found it to be a preferable alternative, which is explained in the answer.
Short summary:
Scrapy comes preinstalled on PythonAnywhere. No installation required.
I found an alternative that I like better: Cloud9. I was able to install Scrapy on it, but with a security issue that probably won't be a problem for me.
====================================
There were three parts to my question:
Can I run Scrapy in the free level of PythonAnywhere? This part has been answered: Yes, but with debilitating restrictions.
The other two parts have not been answered, but I've found some answers and will share them here.
What other online environments allow me to run Scrapy without needing to install Python and Scrapy on my computer? I haven't found a direct answer to this, but the free tutorial website, Python for Everybody ("Py4E"), has a page, Setting up your Python Development Environment, which lists four online Python environments. It provides a brief tutorial on PythonAnywhere and then just provides links to the other three: Trinket, Cloud9, and CodeAnywhere.
None of those four environments say anything about running Scrapy on them. With some more research, I did find out how to use Scrapy in PythonAnywhere, which I explain next below. Of the other three, Cloud9 is part of Amazon's AWS suite, which is a sophisticated set of software tools that I've used other parts of before. Based on that, I assumed it also accommodates Scrapy, and I checked it out as well. I've added the results of that below as a new part 4 to my question.
Now, the main part of my question: How to install Scrapy on PythonAnywhere? The answer is:
You don't. It's already installed!
It's amazing that PythonAnywhere's otherwise excellent documentation doesn't say anything about this. I found it out by following instructions that I hoped would lead me to installing Scrapy:
First, since I'm new to Python (but not to programming), I ran through Py4E's tutorial on PythonAnywhere, which is really a quick introduction to Python and got me to write a simple program, told me to use the Bash Unix shell instead of the Python interpreter ("$" instead of ">>>"), and had me save a simple program to a file.
Next, I went to Scrapy's installation instructions. It has this wonderful line: "... if you’re already familiar with installation of Python packages, you can install Scrapy and its dependencies from PyPI with: pip install Scrapy". Of course, it doesn't follow that by saying what to do if I'm not familiar with that. Sigh!
After that, I somehow found my way to Python's official instructions on Installing Packages, which starts by explaining that "package" means "a bundle of software to be installed", so I thought that might include Scrapy. So I ran through the instructions there, and about half-way through, it told me to run:
python3 -m pip install "SomeProject"
(* Footnote below on syntax of that command)
The instructions said that "SomeProject" is supposed to be a project that's included in the Python Package Index, so I went there and searched for Scrapy. It gave me a list of 681 projects with "scrapy" in the name, and some of them looked like they might be various versions of Scrapy itself. None of them were called just "Scrapy", but the Scrapy instruction quoted above said to use just that name. So I held my breath and entered:
python3 -m pip install Scrapy
And guess what I got? PythonAnywhere told me:
Requirement already satisfied: Scrapy in /usr/local/lib/python3.9/site-packages (2.5.0)
That was followed by a couple of dozen more lines that all started with "Requirement already satisfied", which I took to be the dependencies required by Scrapy, all of them already present and ready to roll.
So, hmmm, Scrapy is already there? To find out if that's really true, I went to the tutorial on Scrapy's website. The first thing it said was to create a project by using the command:
scrapy startproject tutorial
I entered that, and PythonAnywhere told me that it had successfully created a new project. Since this was a Scrapy command, I conclude that, yes, indeed, I already have Scrapy installed and running on PythonAnywhere. No installation necessary!
What about Cloud9? As I said above in my answer to part 2, when I found out about Cloud9, I was interested because it's part of Amazon Web Services ("AWS"). I've used other parts of AWS before and found them to be sophisticated, complicated, powerful, and well-documented. They are also very economical.
AWS is a commercial system run by Amazon. It charges fees based on usage, with no minimums, and with low-volume usage being free. The pricing page for Cloud9 shows it to be no exception. Cloud9 itself is free to use, but using it calls on other AWS resources that have charges.
The pricing page gives the following example: "If you use the default settings running an IDE for 4 hours per day for 20 days in a month with a 30-minute auto-hibernation setting your monthly charges for 90 hours of usage would be ... $2.05". That's less than half the lowest monthly cost of PythonAnywhere. (As stated in the answer by Giles Thomas, the free level of PythonAnywhere is not very useful for Scrapy.) I'm not sure how the amount of usage in the Cloud9 example compares with the amount of usage allowed by PythonAnywhere's $5/mo service, but my usage is going to be a lot less than either one, so I expect my cost of using Cloud9 to be very low, and possibly nothing. Furthermore, if I only use Scrapy for a project a couple of times a year, with PythonAnywhere, I'd have to close my account in between projects to stop being charged, but AWS doesn't charge me when I'm not using it, so I can keep the account with no cost between projects.
So based on both the quality of the AWS modules I've used and the low usage cost, I was very interested in Cloud9 as an alternative.
And I was not surprised to find that I could use Scrapy in it.
To figure that out, I quickly abandoned the webpage instructions in favor of downloading a pdf of the comprehensive User Guide from the documentation page. Comprehensive = 595 pages! But it's very well organized and cross-referenced, so I was able to learn what I needed by reading about 20 pages, which included a tutorial on using the GUI environment (pg 29..38) and another on using Python in Cloud9 (pg 423..7).
In that second tutorial, I ran:
python3 --version to find out that Python was already installed, version 3.7.10.
python -m pip --version to find out that pip version 20.2.2 is running.
After that tutorial, I was ready to find out if Scrapy is there. I had learned by then about pip show, so I ran:
python -m pip show Scrapy
The answer was no:
WARNING: Package(s) not found: Scrapy
So I repeated the command that I'd done earlier in PythonAnywhere:
python3 -m pip install Scrapy
This time, there were very few "Requirement already satisfied"s and instead there were a lot of "Collecting ... Downloading"s, followed by "Installing collected packages" and then "Successfully installed" with a long list that included Scrapy-2.6.1.
I repeated python -m pip show Scrapy and got several lines of output that told me Scrapy 2.6.1 is installed. Finally, I ran the same test I'd run before in PythonAnywhere, the first instruction in the official Scrapy tutorial:
scrapy startproject tutorial
and got the same output as before, telling me that the project had been created.
Bingo! I have Scrapy running in Cloud9.
On the negative side, there was a problem here. AWS has two levels of sign-in authority, called root users and IAM. For proper security, I should be running Cloud9 as an IAM user, but there was a problem being able to sign in that way. I posted a question on SO about that, but while waiting for an answer, went ahead and started using Cloud9 as the root user. In the course of that, I got the message:
WARNING: Running pip install with root privileges is generally not a good idea.
That warning came with a suggestion of an alternative command that didn't make sense and didn't work when I tried it. So I'm not sure how much I've messed up the security of my AWS account by what I've been doing here. My work is not secretive, so the security may be a non-issue, but I'd still like to figure out how to proceed as an IAM user and clean up any damage I might have caused by what I've been doing as the root user. If anyone knows about that, please respond to the SO question about it linked in the previous paragraph.
So now I've got Scrapy running in Cloud9, and I'm going to go find out if it can get the data I need. I'll make another edit here if there are any surprises in terms of Cloud9 either (a) not being able to do something or (b) resulting in unexpected charges.
====================================
(*) Footnote on syntax of python3 -m pip install "SomeProject":
Since I was working in something called PythonAnywhere, I was tempted to think that this was a Python command. But then I had to remember that, within PythonAnywhere, I was working in Bash, a Unix shell. So Python3 is a Unix command. I haven't found documentation of that exact command, but did of a command it's presumably based on Python. That documentation says, "-m module-name Searches ... for the named module and runs the corresponding .py file as a script." So this means that pip is a Unix module written in Python for installing Python packages. Then install <project name> is a parameter of the pip module. (Somebody please correct me if I've said any of that wrong.)
You can, but free accounts on PythonAnywhere are limited to accessing sites on a whitelist of official public APIs, so you will probably not be able to access non-API sites.
I was trying to install pymc3 with Miniconda in Windows. The installation was certainly non-trivial and took me many hours of trouble-shooting to understand what is going on.
I have read many posts from different people on this question and found this guide to be quite helpful:
pymc3 installation guide in Windows
After following through the steps recommended here, I still came across the warning when importing pymc3 as pm in a jupyter-notebook:
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
Below are some solutions that I have tested and did not work to remove this warning:
conda install -c conda forge libpython blas mkl-service m2w64-toolchain in addition to the packages bundled with pymc3 (theano-pymc3 and arviz) in the first conda environment.
Downloaded GCC compiler and make sure it is on the global path in the correct order as suggested by the first guideline and the official guideline Official pymc3 installation in the second conda environment and installed without m2w64-toolchain.
I have also tried using the GCC compiler from Rtools and updated the path in the environment variable in the 3rd conda environment.
I think I have exhausted all the different options I have read and still didn't manage to remove this warning. Is there anything that I have missed out in trouble-shooting this warning?
I understand that this warning doesn't stop the computation, and would only affect the computational performance. Because I have quite a large dataset, I would certainly appreciate a feasible solution to fix this problem :)
Thank you for any feedback!
According to https://zhuanlan.zhihu.com/p/394033647
to create .theanorc.txt
may solve this problem
I want to do some grasp motion planning experiments, but I failed with compiling the GraspIt! using VS2010 on Win7. Is there has other grasp simulation tool like GraspIt! that can be used in Win7 and VS2010 environment? Thank you!
I had been trying to install Graspit for Windows 7 too, but failed somehow and then switched to OpenRAVE. OpenRAVE is open source and provides an environment for testing, developing, and deploying motion planning algorithms in real-world robotics applications. The installation guide for Windows can be found here. There are a lot of quick start examples, which cover topics to help you get started with Inverse kinematics, testing a grasp, implementing planners and controllers. You could also work in a MATLAB, Octave or Python interface.
You can also find lots of relevant information on the OpenRAVE user mailing list, or ask questions when you need help.
I am running OpenRAVE on Ubuntu 12.04 though, and the installation was fairly easy.
I extensively use Cygwin on a Windows 8 environment (I do not want to go ahead and boot/load Linux directly on the machine). I use the OCamlIDE plug-in for Eclipse and have experienced relatively no problems using this workflow setup.
However, I would like to use Batteries so that I may make use of use of its dynamic arrays among a few other interesting features that will speed up my development process.
I have tried this method: http://ocaml.org/install.html, but I get the following error:
$ sh ./opam_installer.sh /usr/local/bin
No file yet for i686:CYGWIN_NT-6.2-WOW64
What am I missing and how would I configure Cygwin so that it can accept the Opam installer? When I tried yet a different way of building Opam, I got:
'i686-w64-mingw32-gcc' is not recognized as an internal or external command,
as a Makefile error and reason for building failure. It seems something is wrong related to mingw32-gcc, what do I need to install and/or configure for my Cygwin to get it to compile/build things properly. I have wget and curl installed as well.
My overall question: What is the best way to get Batteries installed on my system with the minimum of time spent tracing all of its dependencies by hand? Is there a way I can just build the library module, such as BatDynArray and the includes:
include BatEnum.Enumerable
include BatInterfaces.Mappable
That way I can just call them directly in my code with open...;; and/or include...;;;
OCaml works beautifully on Windows with WODI, which is a Cygwin-based distribution that includes Batteries and tons of other useful packages (which are a pain to install manually on Windows).
I urge you to take a shot at WODI, which I believe to be an indispensable tool for the
rest of us, the forgotten souls, who have to deal with Windows.
First of all, include does not do what you think it does. open Batteries should be exactly what you're looking for. OPAM is not yet solid on windows (maybe Thomas could give an update on where things stand).
Frankly, I would recommend to install a linux on a VM, you should be able to get started with OPAM instantly then. Otherwise, take a look at this package manager for OCaml which focuses on cross platform support: http://yypkg.forge.ocamlcore.org/. I've never tried it myself however. The last package manger you could try is GODI, I'm not sure about its windows support though.
Finally, if none of these options work then it should be possible to install batteries from the source. All you need is OCaml and make. And if there are problems with this approach then you should definitely follow up on them either here or on the bug tracker because batteries does intend to support windows AFAIK.
I installed the OpenNI+NITE+kinect on ubuntu 12.10 today and the samples are working fine. (Ran NITE samples and they work like charm)
I want to start developing in Linux and I like to work with IDE. For ubuntu I have always used Geany or Codeblocks and i was wondering if there was a way to integrate OpenNI,NITE libraries to the IDE's so that I can write, execute, debug the code easily from the IDE itself.
I tried to add the libraries myself in geany but all my attempts failed. :/
Also, when I do make for Sample examples in OpenNI, I get *.d and *.o files but I dont seem to get the executable *.out . So i am not sure how to run the programs.
I am a novice programmer, just starting to learn, So please excuse the noobness in
questions.
Appreciate the Help.
Thanks
Generally if you're doing something non-trivial with Geany you will need to provide your own build system (even simple GNU Make files will do). Geany is intentionally build system agnostic and allows you to run arbitrary build commands which are described in the manual as well as in a helpful article on the wiki.
If you describe in more detail the errors/problems you referred to as "attempts failed" it will likely be possible to provide more concrete solutions.