In normal Python project we might have a parent py file that imports from a sibling or child folder/module (initialised by __init__.py, from .folder import module etc.
Is there a way to do this using Synapse notebooks within a given (ex. dev) workspace?
Ex. I would like to create a notebook/python module for use in logging - a wrapper to wrap functions. I don't want to have to have to copy-paste this module into 20 different notebooks.
Thanks!
Yes, you can use something called notebook reference
You can use %run magic command to reference another notebook within current notebook's context. All the variables defined in the reference notebook are available in the current notebook. %run magic command supports nested calls but not support recursive calls. You will receive an exception if the statement depth is larger than five.
Example: %run //Notebook1 { "parameterInt": 1, "parameterFloat": 2.5, "parameterBool": true, "parameterString": "abc" }.
Source
Related
I'm not a Python programmer so apologies if I don't get some of the terminology right (pacakages, modules)!
I have a folder structure that looks something like this:
C:\Test\System\
C:\Test\System\intercepts\
C:\Test\System\intercepts\utils1\
C:\Test\System\intercepts\utils2\
The last three folders each contain an empty __init__.py folder, while the latter two folders (\utils1, \utils2) contain numerous .py modules. For the purposes of my question I'm trying to execute a function within a module called "general.py" that resides in the \utils1 folder.
The first folder (C:\Test\System) contains a file called "entry.py", which imports the .py modules from all those sub-folders:
from intercepts.utils1 import general
from intercepts.utils1 import foobar
from intercepts.utils2 import ...
..etc..
And here is the C# code that executes the above module then attempts to call a function called "startup" in a module called "general.py" in the \utils1 folder:
const string EntryModule = #"C:\Test\System\entry.py";
using (Py.GIL())
{
using (var scope = Py.CreateScope())
{
var code = File.ReadAllText(EntryModule);
var scriptCompiled = PythonEngine.Compile(code, EntryModule);
scope.Execute(scriptCompiled);
dynamic func = scope.Get("general.startup");
func();
}
}
However I get a PythonException on the scope.Execute(...) line, with the following message:
No module named 'intercepts'
File "C:\Test\System\entry.py", line 1, in <module>
from intercepts.utils1 import general
I'm able to do the equivalent of this using IronPython (Python 2.7), so I assume I'm doing something wrong with Python.Net (rather than changes to how packages work in Python 3.x).
I'm using the pythonnet 3.0.0 NuGet package by the way.
Edit
I've tried importing my "entry.py" module as follows:
dynamic os = Py.Import("os");
dynamic sys = Py.Import("sys");
sys.path.append(os.path.dirname(EntryModule));
Py.Import(Path.GetFileNameWithoutExtension(EntryModule));
It now appears to get a little further, however there's a new problem:
In the "entry.py" module you can see that it first imports a module called "general", then a module called "foobar". "foobar.py" contains the line import general.
When I run my C#, the stack trace is now as follows:
No module named 'general'
File "C:\Test\System\intercepts\utils1\foobar.py", line 1, in <module>
import general
File "C:\Test\System\entry.py", line 2, in <module>
from intercepts.utils1 import foobar
Why can't the second imported module ("foobar") "see" the module that was imported immediately before it ("general")? Am I even barking up the right tree by using Py.Import() to solve my original issue?
This turned out to be a change in how Python 3 handles imports, compared to 2, and nothing to do with Python.Net.
In my "foobar.py" module I had to change import general to from . import general. The issue is explained here but I've included the pertinent section below:
i'm having some trouble loading module in Julia. I have to module that i cant load in my main file.
So my code (i'm trying to make an octree) look like this:
module Node
export node, contains, intersect
mutable struct node
x::Float64
y::Float64
z::Float64
m::Float64
node(x,y,z) = new(x,y,z,0)
end
end # module
and my other module:
module Tree
include("Node.jl")
using .Node
export tree, insert!, subdivide!
mutable struct tree
capacity::Int64
node::node
divided::Bool
tree(capacity, node) = new(capacity, node, false)
end
end
My problem is when i try to import the module in my main file using something like that:
include("Node.jl")
using .Node
include("Tree.jl")
using .Tree
plop=node(0,0,0)
plip=tree(1,plop)
I get the following error:
ERROR: LoadError: MethodError: Cannot `convert` an object of type node to an object of type Main.Tree.Node.node
I understand its due to the using .Node in my tree module which conflict with the same import in the main file but i'm unable to find a workaround.
One solution would probably be to put everything in the same module but i would like to keep thing separated.
Well, you actually just put the two modules in the same module. Or to be more precise, you have a module Node and a module Tree with a submodule Node in it, thus the Main.Tree.Node.node. This happens because you use include("Node.jl") within your Tree module. The include function works as if it copied the text in the Node.jl file and pasted it into the Tree.jl file. Thus, to use the Node module within Tree without creating a submodule you have to add it.
So, I'd recommend you to generate a package for both the Node and Tree modules. This is done by
julia> using Pkg
julia> Pkg.generate("Node")
Generating project Node:
Node/Project.toml
Node/src/Node.jl
and then copy your Node.jl and Tree.jl files to replace the ones that were created.
Then you can look at this question that tells you how to add a local package.
To sum it up you need to
julia> Pkg.develop(path="/Path/to/Node")
julia> Pkg.develop(path="/Path/to/Tree")
then your /Path/to/Tree/src/Tree.jl looks like
module Tree
using Node
[...]
end
and to run your code you can type
julia> using Node, Tree
julia> plop=node(0,0,0)
node(0.0, 0.0, 0.0, 0.0)
julia> plip=tree(1,plop)
tree(1, node(0.0, 0.0, 0.0, 0.0), false)
Note that it will probably tell you that Tree does not have Node in its dependencies. To solve that you might want to have a look at the documentation of Pkg.jl.
I am currently looking for a way to directly pass in a pandas dataframe or csv file to zipline for simple backtesting WITHOUT having to ingest a data bundle. The reason is that I am planning to generate new data outside of the existing bundle during a backtest and it seems very inefficient to ingest a new bundle for every handle_data call.
I have been looking for this everywhere, including the source codes of zipline. I found that an older version of zipline has a 'data' param in the run_algo function call where you could pass in a df directly, but I can't find that old version at the moment. Is anyone attempting the same thing? Is there any way other than ingesting data bundles in the command line everytime?
I'm using zipline 1.3.0 and it actually does have a data param. This comment is from run_algo.py file of zipline:
data : pd.DataFrame, pd.Panel, or DataPortal, optional
The ohlcv data to run the backtest with.
This argument is mutually exclusive with:
``bundle``
``bundle_timestamp``
Hope it helped
Although this question has been asked before, it appears that much has changed with respect to modules in Julia V1.0.
I'm trying to write a custom module and do some testing on it. From the Julia documentation on Pkg, using the dev command, there is a way to create a git tree and start working.
However, this seems like overkill at this point. I would like to just do a small local file, say mymodule.jl that would be like:
module MyModule
export f, mystruct
function f()
end
struct mystruct
x::Integer
end
end # MyModule
It appears that it used to be possible to load it with
include("module.jl")
using MyModule
entering the include("module.jl"), it appears that the code loads, i.e. there is no error, however, using MyModule gives the error:
ArgumentError: Package MyModule not found in current path:
- Run `import Pkg; Pkg.add("MyModule")` to install the MyModule package.
I notice that upon using include("module.jl"), there is access to the exported function and struct using the full path, MyModule.f() but I would like the shorter version, just f().
My question then is: to develop a module, do I need to use the Pkg dev command or is there a lighter weight way to do this?
In order to use a local module, you must prefix the module name with a ..
using .MyModule
When using MyModule is run (without the .), Julia attempts to find a module called MyModule installed to the current Pkg environment, hence the error.
I'm trying to execute a graph located in a local graphs/ folder using the noflo.asCallback function.
I'm getting an error that the graph or component is not available with the base specified at the baseDir.
What is the correct way to us the asCallback function with a graph?
The issue was related to the package name of the project. The name I was using was noflo-test, and the graph was available as test/GraphName, without the noflo- prepended.