i try to deploy an android app. I work with the kivy framework and buildozer in python. My issue is to include the pandas library. This is my simple and working test code:
from kivy.app import App
from kivy.uix.label import Label
import kivy
kivy.require('1.11.1')
import pandas as pd
class TestLibraries(App):
def build(self):
df = pd.DataFrame()
df.loc[0, 'text'] = 'this is pandas'
return Label(text = df.loc[0, 'text'])
if __name__ == '__main__':
TestLibraries().run()
The next step is do define the buildozer .spec file. Here i see two options:
Via requirements: So i modify the .spec file like this
# (list) Application requirements
# comma separated e.g. requirements = sqlite3,kivy
requirements = python3,kivy==1.11.1,pandas
This works very well.
2. Via recipe: I take the recipe from github. and put it into my folder called recipe. After that i modify the .spec file like this
# (str) The directory in which python-for-android should look for your own build recipes (if any)
p4a.local_recipes = /PATH_TO_FOLDER/recipe/
In the buildozer logfile i can read:
Listing '/PATH_TO_FOLDER/.buildozer/android/app/recipe/pandas'...
Compiling 'PATH_TO_FOLDER/.buildozer/android/app/recipe/pandas/__init__.py'...
So buildozer found the recipe but the library is not installed and the app dosen't works.
And the question is: why not?
You might ask me for using the second option because the first option works very well. In the next step i want to write a new recipe. So i have to learn how to include an existing recipe correctly.
I hope you unterstand my problem an have some advices.
Thanks Capa
Related
I'm not a Python programmer so apologies if I don't get some of the terminology right (pacakages, modules)!
I have a folder structure that looks something like this:
C:\Test\System\
C:\Test\System\intercepts\
C:\Test\System\intercepts\utils1\
C:\Test\System\intercepts\utils2\
The last three folders each contain an empty __init__.py folder, while the latter two folders (\utils1, \utils2) contain numerous .py modules. For the purposes of my question I'm trying to execute a function within a module called "general.py" that resides in the \utils1 folder.
The first folder (C:\Test\System) contains a file called "entry.py", which imports the .py modules from all those sub-folders:
from intercepts.utils1 import general
from intercepts.utils1 import foobar
from intercepts.utils2 import ...
..etc..
And here is the C# code that executes the above module then attempts to call a function called "startup" in a module called "general.py" in the \utils1 folder:
const string EntryModule = #"C:\Test\System\entry.py";
using (Py.GIL())
{
using (var scope = Py.CreateScope())
{
var code = File.ReadAllText(EntryModule);
var scriptCompiled = PythonEngine.Compile(code, EntryModule);
scope.Execute(scriptCompiled);
dynamic func = scope.Get("general.startup");
func();
}
}
However I get a PythonException on the scope.Execute(...) line, with the following message:
No module named 'intercepts'
File "C:\Test\System\entry.py", line 1, in <module>
from intercepts.utils1 import general
I'm able to do the equivalent of this using IronPython (Python 2.7), so I assume I'm doing something wrong with Python.Net (rather than changes to how packages work in Python 3.x).
I'm using the pythonnet 3.0.0 NuGet package by the way.
Edit
I've tried importing my "entry.py" module as follows:
dynamic os = Py.Import("os");
dynamic sys = Py.Import("sys");
sys.path.append(os.path.dirname(EntryModule));
Py.Import(Path.GetFileNameWithoutExtension(EntryModule));
It now appears to get a little further, however there's a new problem:
In the "entry.py" module you can see that it first imports a module called "general", then a module called "foobar". "foobar.py" contains the line import general.
When I run my C#, the stack trace is now as follows:
No module named 'general'
File "C:\Test\System\intercepts\utils1\foobar.py", line 1, in <module>
import general
File "C:\Test\System\entry.py", line 2, in <module>
from intercepts.utils1 import foobar
Why can't the second imported module ("foobar") "see" the module that was imported immediately before it ("general")? Am I even barking up the right tree by using Py.Import() to solve my original issue?
This turned out to be a change in how Python 3 handles imports, compared to 2, and nothing to do with Python.Net.
In my "foobar.py" module I had to change import general to from . import general. The issue is explained here but I've included the pertinent section below:
I would like to remove tensorflow and hugging face models from my laptop.
I did find one link https://github.com/huggingface/transformers/issues/861
but is there not command that can remove them because as mentioned in the link manually deleting can cause problems because we don't know which other files are linked to those models or are expecting some model to be present in that location or simply it may cause some error.
The transformers library will store the downloaded files in your cache. As far as I know, there is no built-in method to remove certain models from the cache. But you can code something by yourself. The files are stored with a cryptical name alongside two additional files that have .json (.h5.json in case of Tensorflow models) and .lock appended to the cryptical name. The json file contains some metadata that can be used to identify the file. The following is an example of such a file:
{"url": "https://cdn.huggingface.co/roberta-base-pytorch_model.bin", "etag": "\"8a60a65d5096de71f572516af7f5a0c4-30\""}
We can now use this information to create a list of your cached files as shown below:
import glob
import json
import re
from collections import OrderedDict
from transformers import TRANSFORMERS_CACHE
metaFiles = glob.glob(TRANSFORMERS_CACHE + '/*.json')
modelRegex = "huggingface\.co\/(.*)(pytorch_model\.bin$|resolve\/main\/tf_model\.h5$)"
cachedModels = {}
cachedTokenizers = {}
for file in metaFiles:
with open(file) as j:
data = json.load(j)
isM = re.search(modelRegex, data['url'])
if isM:
cachedModels[isM.group(1)[:-1]] = file
else:
cachedTokenizers[data['url'].partition('huggingface.co/')[2]] = file
cachedTokenizers = OrderedDict(sorted(cachedTokenizers.items(), key=lambda k: k[0]))
Now all you have to do is to check the keys of cachedModels and cachedTokenizers and decide if you want to keep them or not. In case you want to delete them, just check for the value of the dictionary and delete the file from the cache. Don't forget to also delete the corresponding *.json and *.lock files.
Use
pip install huggingface_hub["cli"]
Then
huggingface-cli delete-cache
You should now see a list of revisions that you can select/deselect.
See this link for details.
pip uninstall tensorflow
pip uninstall tensorflow-gpu
pip uninstall transformers
and find where you have saved gpt-2
model.save_pretrained("./english-gpt2") .???
english-gpt2 = your downloaded model name.
from that path you can manually delete.
After figuring out how to achieve this in a web application using WebPack (https://medium.com/#janos.jarecsni/auto-loading-modules-c8c47472d59b) I was quite sad to see that it is impossible to do in React Native.
I think being able to load all modules in a folder is a rather key technique (this way your code does not have to know about the individual modules), and so it is rather limiting in React Native.
Anyone knows if this is possible using some arcane ways, or any ideas how to work around this?
There's a few ways to achieve this, I have written an article to discuss this in detail. The good news is that there is a Babel plugin that allows you to do this.
https://medium.com/#janos.jarecsni/auto-loading-modules-c8c47472d59b
If you want to import all the modules in the current dir, you can try this one.
import_cur_modules only imports modules in the current dir,
import_sub_modules imports all the modules including sub dirs in the current dir
Note: This function only returns the import lines that need be executed in the __init__.py in the current dir.
import_lines = import_sub_modules(os.path.dirname(__file__))
for line in import_lines:
exec(line)
import os
def import_cur_modules(base_dir: str):
"""import modules in the current dir
Note: This function only returns the import lines that need be executed in the `__init__.py` in base_dir
"""
import_lines = []
for name in os.listdir(base_dir):
if name.endswith('.py') and name != '__init__.py':
import_lines.append(f"from . import {name[:-3]}")
return import_lines
def import_sub_modules(base_dir: str):
"""import all the modules including sub dirs in the current dir
Note: This function only returns the import lines that need be executed in the `__init__.py` in base_dir
"""
import_lines = []
for root, _, files in os.walk(base_dir):
root = os.path.relpath(root, start=base_dir)
for name in files:
if name.endswith('.py') and name != '__init__.py':
path_units = os.path.join(root, name[:-3]).split('/')
if path_units[0] == '.':
path_units.pop(0)
import_lines.append(
f"from .{'.'.join(path_units[:-1])} import {path_units[-1]}"
)
return import_lines
I'm creating a web-based interface for a number of different command line executables, and am using cherrypy behind apache (using mod_rewrite). I'm very new to this, and am having difficulty getting things configured properly. On my development machine, everything works reasonable well, but when I installed the code on a second machine I can't get anything to work properly.
The basic workflow for the applications is: 1. upload a dataset, 2. process the data (using python with some calls to executables using subprocess.call), 3. display the results on the web page.
After uploading and processing one dataset, everytime I attempt to process a second dataset the system stops responding. I'm not seeing any output in the terminal from the cherrypy process, or in the site log that shows any errors have occurred.
I'm starting cherrypy with the following conf file:
[global]
environment: 'production'
log.error_file: 'logs/site.log'
log.screen: True
tools.sessions.on: True
tools.session.storage_type: "file"
tools.session.storage_path: "sessions/"
tools.sessions.timeout: 60
tools.auth.on: True
tools.caching.on: False
server.socket_host: '0.0.0.0'
server.max_request_body_size: 0
server.socket_timeout: 60
server.thread_pool: 20
server.socket_queue_size: 10
engine.autoreload.on:True
My init.py file:
import cherrypy
import os
import string
from os.path import exists, join
from os import pathsep
from string import split
from mako.template import Template
from mako.lookup import TemplateLookup
from auth import AuthController, require, member_of, name_is
from twopoint import TwoPoint
current_dir = os.path.dirname(os.path.abspath(__file__))
lookup = TemplateLookup(directories=[current_dir + '/templates'])
def findInSubdirectory(filename, subdirectory=''):
if subdirectory:
path = subdirectory
else:
path = os.getcwd()
for root, dirs, names in os.walk(path):
if filename in names:
return os.path.join(root, filename)
return None
class Root:
#cherrypy.expose
#require()
def index(self):
tmpl = lookup.get_template("main.html")
return tmpl.render(usr=WebUtils.getUserName(),source="")
if __name__=='__main__':
conf_path = os.path.dirname(os.path.abspath(__file__))
conf_path = os.path.join(conf_path, "prod.conf")
cherrypy.config.update(conf_path)
cherrypy.config.update({'server.socket_host': '127.0.0.1',
'server.socket_port': 8080});
def nocache():
cherrypy.response.headers['Cache-Control']='no-cache,no-store,must-revalidate'
cherrypy.response.headers['Pragma']='no-cache'
cherrypy.response.headers['Expires']='0'
cherrypy.tools.nocache = cherrypy.Tool('before_finalize',nocache)
cherrypy.config.update({'tools.nocache.on':'True'})
cherrypy.tree.mount(Root(), '/')
cherrypy.tree.mount(TwoPoint(), '/twopoint')
cherrypy.engine.start()
cherrypy.engine.block()
For one example where this occurs, I've got the following javascript function that calls my python code:
function compTwoPoint(dataset,orig){
// call python code to generate images
$.post("/twopoint/compTwoPoint/"+dataset,
function(result){
res=jQuery.parseJSON(result);
if(res.success==true){
showTwoPoint(res.path,orig);
}
else{
alert(res.exception);
$('#display_loading').html("");
}
});
}
This calls the python code:
def twopoint(in_matrix):
"""proprietary code, can't share"""
def twopoint_file(in_file_name,out_file_name):
k = imread(in_file_name);
figure()
imshow(twopoint(k))
colorbar()
savefig(out_file_name,bbox_inches="tight")
close()
class TwoPoint:
#cherrypy.expose
def compTwoPoint(self,dataset):
try:
fnames=WebUtils.dataFileNames(dataset)
twopoint_file(fnames['filepath'],os.path.join(fnames['savebase'],"twopt.png"))
return encoder.iterencode({"success": True})
These functions work together to give the expected result. The problem is that after processing one input file, I am unable to process a second file. I don't seem to get a response from the server.
On the machine where things are working, I'm running python 2.7.6 and cherrypy 3.2.3. On the second machine, I have python 2.7.7 and cherrypy 3.3.0. While this may explain the difference in behavior, I'd like to find a way to make my code portable enough to overcome the difference in version (going from older to newer)
I'm not sure what the problem is, or even what to search for. I would appreciate any guidance or help you can offer.
(edit: Digging a bit more, I discovered something is happening with matplotlib. if I put print statments before and after the figure() command in twopoint_file, only the first one prints. Calling this function directly from a python interpreter (removing cherrypy from the equation) I get the following error:
can't invoke "event" command: application has been destroyed while executing "event generate $w{{ThemeChanged}}"
procedure "ttk::ThemeChanged" line 6 invoked from within "ttk::ThemeChanged"
end edit)
I don't understand what this error means, and haven't had much luck searching.
Old question, but I got the same problem which I fixed by changing backend in Matplotlib:
import matplotlib
matplotlib.use("qt4agg")
I am trying to run tests by adding a version of tornado downloaded from github.com in the sys.path.
[tests]
recipe = zc.recipe.testrunner
extra-paths = ${buildout:directory}/parts/tornado/
defaults = ['--auto-color', '--auto-progress', '-v']
But when I run bin/tests I get the following error :
ImportError: No module named tornado
Am I not understanding how to use extra-paths ?
Martin
Have you tried looking into generated bin/tests script if it contains your path? It will tell definitely if your buildout.cfg is correct or not. Maybe problem is elsewhere. Because it seem that your code is ok.
If you happen to regularly include various branches from git/mercurial or elsewhere to buildout, you might be interested in mr.developer. mr.developer can download and add package to develop =. You wont need to set extra-path in every section.