WARNING: NEWBIE
i had put off upgrading from pandas 0.18 to 0.19 until this morning. this code used to just give a deprication warning:
import pandas.io.data as web
x = web.DataReader('GE','yahoo',(2016, 10, 1), (2016, 11, 1))
now it throws an error and tells me: "The pandas.io.data module is moved to a separate package " builtins.ImportError: The pandas.io.data module is moved to a separate package (pandas-datareader). After installing the pandas-datareader package (https://github.com/pandas-dev/pandas-datareader), you can change the import from pandas.io import data, wb to from pandas_datareader import data, wb."
so, i rewrite my "from...import..." line to:
from pandas_datareader import data, wb
as expected, when i run the code, it throws an error:
builtins.NameError: name 'web' is not defined
when i try this code:
x = wb.pandas-datareader('GE','yahoo',(2016, 10, 1), (2016, 11, 1))
this error is thrown:
builtins.AttributeError: module 'pandas_datareader.wb' has no attribute 'pandas'
when i try this code:
x = wb.Datareader('GE','yahoo',(2016, 10, 1), (2016, 11, 1))
this error is thrown:
builtins.AttributeError: module 'pandas_datareader.wb' has no attribute 'DataReader'
can anyone please tell me how to call the datareader now?
thanks in advance
The old method of using the data reader in pandas should not be used as the first error you encountered. So never use import pandas.io.data as web
The correct way to access the modules of the new package pandas_datareader is what you wrote.
from pandas_datareader import data, wb
data and wb are modules (Python files) with many different functions that you can call to bring in different types of external data into your program. To see all the functions of each module use the dir command.
You can see all the publicly available objects with:
[attribute for attribute in dir(data) if attribute[0] != '_']
Which outputs
['DataReader',
'EurostatReader',
'FamaFrenchReader',
'FredReader',
'GoogleDailyReader',
'OECDReader',
'Options',
'YahooActionReader',
'YahooDailyReader',
'YahooOptions',
'YahooQuotesReader',
'get_components_yahoo',
'get_data_famafrench',
'get_data_fred',
'get_data_google',
'get_data_yahoo',
'get_data_yahoo_actions',
'get_quote_google',
'get_quote_yahoo',
'warnings']
So, these are all the items that you can use after the . in the data module.
If you run the same dir command with the wb module you will see that DataReader does not exist in that module. It exists in the above list in the data module.
Finally, make sure you have spelled your function correctly DataReader has upper case R. Use tab completion to avoid these mistakes or you will get the no attribute error. If the function you want does not get outputted with the dir command then you are using the wrong module.
DataReader also accepts strings as dates so the following will get you what you want.
data.DataReader('GE','yahoo', '2016-10-1', '2016-11-1')
Related
I created a updatesite.nsf from updatesite.ntf template and trying to import a feature and I get an error says "LS2J Error: Java constructor failed to execute in (#304)..." And a long message. Notes and Domino are 12.0.02 both 64 bit...Need help in this . Why am I getting this error ? Nothing in Activity logs either in the updatesite.nsf.
The developer tried to create a JAR and trying to import it the Feature into the updatesite,nsf database and it errors out as above. It should import as I am expecting.
I'm not a Python programmer so apologies if I don't get some of the terminology right (pacakages, modules)!
I have a folder structure that looks something like this:
C:\Test\System\
C:\Test\System\intercepts\
C:\Test\System\intercepts\utils1\
C:\Test\System\intercepts\utils2\
The last three folders each contain an empty __init__.py folder, while the latter two folders (\utils1, \utils2) contain numerous .py modules. For the purposes of my question I'm trying to execute a function within a module called "general.py" that resides in the \utils1 folder.
The first folder (C:\Test\System) contains a file called "entry.py", which imports the .py modules from all those sub-folders:
from intercepts.utils1 import general
from intercepts.utils1 import foobar
from intercepts.utils2 import ...
..etc..
And here is the C# code that executes the above module then attempts to call a function called "startup" in a module called "general.py" in the \utils1 folder:
const string EntryModule = #"C:\Test\System\entry.py";
using (Py.GIL())
{
using (var scope = Py.CreateScope())
{
var code = File.ReadAllText(EntryModule);
var scriptCompiled = PythonEngine.Compile(code, EntryModule);
scope.Execute(scriptCompiled);
dynamic func = scope.Get("general.startup");
func();
}
}
However I get a PythonException on the scope.Execute(...) line, with the following message:
No module named 'intercepts'
File "C:\Test\System\entry.py", line 1, in <module>
from intercepts.utils1 import general
I'm able to do the equivalent of this using IronPython (Python 2.7), so I assume I'm doing something wrong with Python.Net (rather than changes to how packages work in Python 3.x).
I'm using the pythonnet 3.0.0 NuGet package by the way.
Edit
I've tried importing my "entry.py" module as follows:
dynamic os = Py.Import("os");
dynamic sys = Py.Import("sys");
sys.path.append(os.path.dirname(EntryModule));
Py.Import(Path.GetFileNameWithoutExtension(EntryModule));
It now appears to get a little further, however there's a new problem:
In the "entry.py" module you can see that it first imports a module called "general", then a module called "foobar". "foobar.py" contains the line import general.
When I run my C#, the stack trace is now as follows:
No module named 'general'
File "C:\Test\System\intercepts\utils1\foobar.py", line 1, in <module>
import general
File "C:\Test\System\entry.py", line 2, in <module>
from intercepts.utils1 import foobar
Why can't the second imported module ("foobar") "see" the module that was imported immediately before it ("general")? Am I even barking up the right tree by using Py.Import() to solve my original issue?
This turned out to be a change in how Python 3 handles imports, compared to 2, and nothing to do with Python.Net.
In my "foobar.py" module I had to change import general to from . import general. The issue is explained here but I've included the pertinent section below:
I have found that pandas v13.0 for Python 2.7 win32 works for most codes I have written in which I want to use both arcpy and pandas. I put that pandas version into the C:\Python27\ArcGIS10.2\Lib\site-packages directory. I tried other versions, but got miscellaneous errors when trying to run them.
I wrote a new code today, however, that manages to not work. It gives the error:
Access violation at address 1E0ACF39 in module 'python27.dll'. Read of
address 9807D3AF.
with the following code:
cond = dfDSS['OBSERVATION NAME']=='A413011CC1'
dfDSS['GROUP'][cond]='HA273UTheads'
All the code before this to create dfDSS using pd.read_csv and inserting the column 'GROUP' with a value of 'other' everywhere is fine. Only when I try to reset the values using the conditional statement.
The code to this point was written in iPython Notebook using Anaconda, but I now want to do some arcpy stuff with it.
Any suggestions in getting the different versions of Python to work together are appreciated.
2018 Update: Be sure to check all the responses, as the answer to this question has changed multiple times over the years. At the time of this update, the Revise.jl answer is probably the best solution.
I have a file "/SomeAbsolutePath/ctbTestModule.jl", the contents of which are:
module ctbTestModule
export f1
f1(x) = x + 1
end
I fire up Julia in a terminal, which runs "~/.juliarc.jl". The startup code includes the line:
push!(LOAD_PATH, "/SomeAbsolutePath/")
Hence I can immediately type into the Julia console:
using ctbTestModule
to load my module. As expected f1(1) returns 2. Now I suddenly decide I want to edit f1. I open up "/SomeAbsolutePath/ctbTestModule.jl" in an editor, and change the contents to:
module ctbTestModule
export f1
f1(x) = x + 2
end
I now try to reload the module in my active Julia session. I try
using ctbTestModule
but f1(1) still returns 2. Next I try:
reload("ctbTestModule")
as suggested here, but f1(1) still returns 2. Finally, I try:
include("/SomeAbsolutePath/ctbTestModule.jl")
as suggested here, which is not ideal since I have to type out the full absolute path since the current directory might not be "/SomeAbsolutePath". I get the warning message Warning: replacing module ctbTestModule which sounds promising, but f1(1) still returns 2.
If I close the current Julia session, start a new one, and type in using ctbTestModule, I now get the desired behaviour, i.e. f1(1) returns 3. But obviously I want to do this without re-starting Julia.
So, what am I doing wrong?
Other details: Julia v0.2 on Ubuntu 14.04.
The basis of this problem is the confluence of reloading a module, but not being able to redefine a thing in the module Main (see the documentation here) -- that is at least until the new function workspace() was made available on July 13 2014. Recent versions of the 0.3 pre-release should have it.
Before workspace()
Consider the following simplistic module
module TstMod
export f
function f()
return 1
end
end
Then use it....
julia> using TstMod
julia> f()
1
If the function f() is changed to return 2 and the module is reloaded, f is in fact updated. But not redefined in module Main.
julia> reload("TstMod")
Warning: replacing module TstMod
julia> TstMod.f()
2
julia> f()
1
The following warnings make the problem clear
julia> using TstMod
Warning: using TstMod.f in module Main conflicts with an existing identifier.
julia> using TstMod.f
Warning: ignoring conflicting import of TstMod.f into Main
Using workspace()
However, the new function workspace() clears Main preparing it for reloading TstMod
julia> workspace()
julia> reload("TstMod")
julia> using TstMod
julia> f()
2
Also, the previous Main is stored as LastMain
julia> whos()
Base Module
Core Module
LastMain Module
Main Module
TstMod Module
ans Nothing
julia> LastMain.f()
1
Use the package Revise, e.g.
Pkg.add("Revise") # do this only once
include("src/my_module.jl")
using Revise
import my_module
You may need to start this in a new REPL session. Notice the use of import instead of using, because using does not redefine the function in the Main module (as explained by #Maciek Leks and #waTeim).
Other solutions: Two advantages of Revise.jl compared to workspace() are that (1) it is much faster, and (2) it is future-proof, as workspace() was deprecated in 0.7, as discussed in this GitHub issue:
julia> VERSION
v"0.7.0-DEV.3089"
julia> workspace()
ERROR: UndefVarError: workspace not defined
and a GitHub contributor recommended Revise.jl:
Should we add some mesage like "workspace is deprecated, check out Revise.jl instead"?
Even in Julia 0.6.3, the three previous solutions of workspace(), import, and reload fail when a module called other modules, such as DataFrames. With all three methods, I got the same error when I called that module the second time in the same REPL:
ERROR: LoadError: MethodError: all(::DataFrames.##58#59, ::Array{Any,1}) is ambiguous. Candidates: ...
I also got many warning messages such as:
WARNING: Method definition macroexpand(Module, ANY) in module Compat at /Users/mmorin/.julia/v0.6/Compat/src/Compat.jl:87 overwritten in module Compat at /Users/mmorin/.julia/v0.6/Compat/src/Compat.jl:87.
Restarting the Julia session worked, but it was cumbersome. I found this issue in the Reexport package, with a similar error message:
MethodError: all(::Reexport.##2#6, ::Array{Any,1}) is ambiguous.
and followed the suggestion of one contributor:
Does this happen without using workspace()? That function is notorious for interacting poorly with packages, which is partly why it was deprecated in 0.7.
In my humble opinion, the better way is to use import from the very beginning instead of using for the reported issue.
Consider the module:
module ModuleX1
export produce_text
produce_text() = begin
println("v1.0")
end
println("v1.0 loaded")
end
Then in REPL:
julia> import ModuleX1
v1.0 loaded
julia> ModuleX1.produce_text()
v1.0
Update the code of the module and save it:
module ModuleX1
export produce_text
produce_text() = begin
println("v2.0")
end
println("v2.0 loaded")
end
Next, in the REPL:
julia> reload("ModuleX1")
Warning: replacing module ModuleX1
v2.0 loaded
julia> ModuleX1.produce_text()
v2.0
Advantages of using import over using:
avoiding ambiguity in function calls (What to call: ModuleX1.produce_text() or produce_text() after reloading?)
do not have to call workspace() in order to get rid of ambiguity
Disadvantages of using import over using:
a fully qualified name in every call for every exported name is needed
Edited: Discarded "full access to the module, even to the not-exported names" from "Disadvantages..." according to the conversation below.
workspace() has been deprecated.
You can reload("MyModule") in an active REPL session, and it works as expected: changes made to the source file that contains MyModule are reflected in the active REPL session.
This applies to modules that have been brought into scope by either import MyModule or using MyModule
I wanted to create a new module from scratch, and tried the different answers with 1.0 and didn’t get a satisfactory result, but I found the following worked for me:
From the Julia REPL in the directory I want to use for my project I run
pkg> generate MyModule
This creates a subdirectory like the following structure:
MyModule
├── Project.toml
└── src
└── MyModule.jl
I put my module code in MyModule.jl. I change to the directory MyModule (or open it in my IDE) and add a file Scratch.jl with the following code:
using Pkg
Pkg.activate(".")
using Revise
import MyModule # or using MyModule
Then I can add my code to test below and everything updates without reloading the REPL.
I battled to get Revise.jl to work for me, probably because I also use code generation with OOPMacro #class . I had to also call revise(...) See https://timholy.github.io/Revise.jl/stable/limitations/#Limitations-1.
I have been struggling with this problem for up to 5 years and finally got something that works for me that don't involve manually running my main module in the IDE repl after EVERY SMALL CHANGE :'(
Here is some of my code I use to force a revise and exit early when running tests when there were revise errors:
# Common code I include into my test files now:
using Pkg
using Revise
# force include module
Pkg.activate("MyModule")
include("../src/MyModule.jl")
Pkg.activate("MyModule/test")
using Revise
# if there were any revise errors which means some compilation error or
# some change that requires a manual rerun or restart.
# then force you to fix it, rather that running lying tests..
for (k, e) in Revise.queue_errors
if !isnothing(e)
warn(logger, "Something went wrong while revising, you probably have a compile error in this file:")
throw(e)
end
end
module TestSomethingModule
using Revise
using MyModule
revise(MyModule)
...
# then you can call MyModule.doSomithing in a test and it actually updates
end
I'm creating a web-based interface for a number of different command line executables, and am using cherrypy behind apache (using mod_rewrite). I'm very new to this, and am having difficulty getting things configured properly. On my development machine, everything works reasonable well, but when I installed the code on a second machine I can't get anything to work properly.
The basic workflow for the applications is: 1. upload a dataset, 2. process the data (using python with some calls to executables using subprocess.call), 3. display the results on the web page.
After uploading and processing one dataset, everytime I attempt to process a second dataset the system stops responding. I'm not seeing any output in the terminal from the cherrypy process, or in the site log that shows any errors have occurred.
I'm starting cherrypy with the following conf file:
[global]
environment: 'production'
log.error_file: 'logs/site.log'
log.screen: True
tools.sessions.on: True
tools.session.storage_type: "file"
tools.session.storage_path: "sessions/"
tools.sessions.timeout: 60
tools.auth.on: True
tools.caching.on: False
server.socket_host: '0.0.0.0'
server.max_request_body_size: 0
server.socket_timeout: 60
server.thread_pool: 20
server.socket_queue_size: 10
engine.autoreload.on:True
My init.py file:
import cherrypy
import os
import string
from os.path import exists, join
from os import pathsep
from string import split
from mako.template import Template
from mako.lookup import TemplateLookup
from auth import AuthController, require, member_of, name_is
from twopoint import TwoPoint
current_dir = os.path.dirname(os.path.abspath(__file__))
lookup = TemplateLookup(directories=[current_dir + '/templates'])
def findInSubdirectory(filename, subdirectory=''):
if subdirectory:
path = subdirectory
else:
path = os.getcwd()
for root, dirs, names in os.walk(path):
if filename in names:
return os.path.join(root, filename)
return None
class Root:
#cherrypy.expose
#require()
def index(self):
tmpl = lookup.get_template("main.html")
return tmpl.render(usr=WebUtils.getUserName(),source="")
if __name__=='__main__':
conf_path = os.path.dirname(os.path.abspath(__file__))
conf_path = os.path.join(conf_path, "prod.conf")
cherrypy.config.update(conf_path)
cherrypy.config.update({'server.socket_host': '127.0.0.1',
'server.socket_port': 8080});
def nocache():
cherrypy.response.headers['Cache-Control']='no-cache,no-store,must-revalidate'
cherrypy.response.headers['Pragma']='no-cache'
cherrypy.response.headers['Expires']='0'
cherrypy.tools.nocache = cherrypy.Tool('before_finalize',nocache)
cherrypy.config.update({'tools.nocache.on':'True'})
cherrypy.tree.mount(Root(), '/')
cherrypy.tree.mount(TwoPoint(), '/twopoint')
cherrypy.engine.start()
cherrypy.engine.block()
For one example where this occurs, I've got the following javascript function that calls my python code:
function compTwoPoint(dataset,orig){
// call python code to generate images
$.post("/twopoint/compTwoPoint/"+dataset,
function(result){
res=jQuery.parseJSON(result);
if(res.success==true){
showTwoPoint(res.path,orig);
}
else{
alert(res.exception);
$('#display_loading').html("");
}
});
}
This calls the python code:
def twopoint(in_matrix):
"""proprietary code, can't share"""
def twopoint_file(in_file_name,out_file_name):
k = imread(in_file_name);
figure()
imshow(twopoint(k))
colorbar()
savefig(out_file_name,bbox_inches="tight")
close()
class TwoPoint:
#cherrypy.expose
def compTwoPoint(self,dataset):
try:
fnames=WebUtils.dataFileNames(dataset)
twopoint_file(fnames['filepath'],os.path.join(fnames['savebase'],"twopt.png"))
return encoder.iterencode({"success": True})
These functions work together to give the expected result. The problem is that after processing one input file, I am unable to process a second file. I don't seem to get a response from the server.
On the machine where things are working, I'm running python 2.7.6 and cherrypy 3.2.3. On the second machine, I have python 2.7.7 and cherrypy 3.3.0. While this may explain the difference in behavior, I'd like to find a way to make my code portable enough to overcome the difference in version (going from older to newer)
I'm not sure what the problem is, or even what to search for. I would appreciate any guidance or help you can offer.
(edit: Digging a bit more, I discovered something is happening with matplotlib. if I put print statments before and after the figure() command in twopoint_file, only the first one prints. Calling this function directly from a python interpreter (removing cherrypy from the equation) I get the following error:
can't invoke "event" command: application has been destroyed while executing "event generate $w{{ThemeChanged}}"
procedure "ttk::ThemeChanged" line 6 invoked from within "ttk::ThemeChanged"
end edit)
I don't understand what this error means, and haven't had much luck searching.
Old question, but I got the same problem which I fixed by changing backend in Matplotlib:
import matplotlib
matplotlib.use("qt4agg")