I have written following simple socket application in which object defined in separate file and it works.
# models_object.py
class Person(object):
def __init__(self,name,surname,age):
self.name = name
self.surname = surname
self.age = age
def __str__(self):
return self.name + " " + self.surname
#client.py
import cPickle,socket
from model_objects import Person
def client():
client = socket.socket()
host = socket.gethostname()
port = 8000
adem_kerenci = Person("Adem","Kerenci",22)
serialized_object = cPickle.dumps(adem_kerenci)
client.connect((host, port))
client.send(serialized_object)
client.close()
client()
#service.py
import cPickle,socket
def server():
server = socket.socket()
host = socket.gethostname()
port = 8000
server.bind((host, port))
server.listen(1)
connection, address = server.accept()
recieved_data = connection.recv(2**20)
print cPickle.loads(recieved_data)
connection.close()
server()
However, if I write Person class definition into client.py, code raises following error.
Traceback (most recent call last):
File "service.py", line 14, in <module>
server()
File "service.py", line 11, in server
print cPickle.loads(recieved_data)
AttributeError: 'module' object has no attribute 'Person'
Is there any relation between importing and serializing?
Note: We tried python 2.7
From documentation:
Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.
This means that the object you are unpickling in the server should be importable too from the server code with the same "fully qualified" name which in the first example is: model_objects.Person. This is apparent in the first case if you try splitting the client.py and model_objects.py in one folder and service.py in another folder which will give you the same error as it is trying to import model_objects.py inside service.py but it couldn't find it.
So in the second case you can copy the class Person as well to the server code and it will work but this is bad practice or make a shared folder for both client and server which is shared production code as well (better practice).
This is because during unpickling python needs the class of the object to reinstantiate the object and use it.
Related
I have two classes: class One that does some stuff and class Wrapper that translates the API of One. To test the Wrapper I want to mock the class One. My problem is that One sets a number of attributes in it's __init__ method and I'd like my test to throw an error when these attributes are changed (e.g. renamed or deleted) in One but not in Wrapper.
I have tried to initialize an instance of One and to use it as a spec, but the resulting mock is non-callable:
from unittest import mock
import pytest
class One:
def __init__(self, a):
self.a = a
spec_obj = One(a='foo')
# working code, the test should pass
class Wrapper:
def __init__(self, wa):
self._one = One(a=wa)
self.wa = self._one.a
#mock.patch(__name__ + '.One', autospec=spec_obj)
def test_wrapper(MockOne):
MockOne.return_value.a = test_a = 'bar'
wrapper_obj = Wrapper(wa=test_a)
MockOne.assert_called_once_with(a=test_a)
assert wrapper_obj.wa == test_a
which throws the error:
TypeError: 'NonCallableMagicMock' object is not callable
since the spec spec_obj is non-callable.
If I set autospec=True, everything works but the test passes even when the One's attribute is renamed.
MockOne.return_value produces a Mock that you configure, but when the mock is actually called, you get a different Mock that hasn't been properly configured. You need to configure MockOne.return_value directly.
#mock.patch('One', autospec=True)
def test_wrapper(MockOne):
MockOne.return_value.a = test_a = 'bar'
wrapper_obj = Wrapper(wa=test_a)
MockOne.assert_called_once_with(a=test_a)
assert wrapper_obj.wa == test_a
If I want to serialize an array in Godot, I can do this:
var a1 = [1 ,2 ,3]
# save
var file = File.new()
file.open("a.sav", File.WRITE)
file.store_var(a1, true)
file.close()
# load
file.open("a.sav", File.READ)
var a2 = file.get_var(true)
file.close()
print(a1)
print(a2)
output (it works as expected):
[1, 2, 3]
[1, 2, 3]
But if I want to serialize an object, like this class in A.gd:
class_name A
var v = 0
Same test, with an instance of A:
# instance
var a1 = A.new()
a1.v = 10
# save
var file = File.new()
file.open("a.sav", File.WRITE)
file.store_var(a1, true)
file.close()
# load
file.open("a.sav", File.READ)
var a2 = file.get_var(true)
file.close()
print(a1.v)
print(a2.v)
output:
10
error (on line print(a2.v)):
Invalid get index 'v' (on base: 'previously freed instance').
From the online docs:
void store_var(value: Variant, full_objects: bool = false)
Stores any Variant value in the file. If full_objects is true, encoding objects is allowed (and can potentially include code).
Variant get_var(allow_objects: bool = false) const
Returns the next Variant value from the file. If allow_objects is true, decoding objects is allowed.
Warning: Deserialized objects can contain code which gets executed. Do not use this option if the serialized object comes from untrusted sources to avoid potential security threats such as remote code execution.
Isn't it supposed to work with full_objects=true? Otherwise, what's the purpose of this parameter?
My classes contains many arrays of arrays and other stuff. I guess Godot handle this kind of basic serialization functionality (of course, devs will often have to save complex data at one point), so, maybe I'm just not doing what I'm supposed to do.
Any idea?
For full_objects to work, your custom type must extend from Object (if you don't specify what your class extends, it extends Reference). And then, the serialization will be based on exported variables (or whatever you say in _get_property_list). By the way, this can, and in your case it likely is, serializing the whole script of your custom type. You can verify by looking at the saved file.
Thus, full_objects is not useful to serialize a type that extends Resource (which does not extend Object). Instead Resource serialization works with ResourceSaver, and ResourceLoader. Also with load and preload. And yes, this is how you would store or load scenes, and scripts (and textures, and meshes, and so on…).
I believe the simpler solution for your code is to use the functions str2var and var2str. These will save you a lot of headache:
# save
var file = File.new()
file.open("a.sav", File.WRITE)
file.store_pascal_string(var2str(a1))
file.close()
# load
file.open("a.sav", File.READ)
var a2 = str2var(file.get_pascal_string())
file.close()
print(a1.v)
print(a2.v)
That solution will work regardless of what is it you are storing.
Perhaps this is a solution (I haven't tested)
# load
file.open("a.sav", File.READ)
var a2 = A.new()
a2=file.get_var(true)
file.close()
print(a1.v)
print(a2.v)
I would like to enhance the class pathlib.Path but the simple example above dose not work.
from pathlib import Path
class PPath(Path):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
test = PPath("dir", "test.txt")
Here is the error message I have.
Traceback (most recent call last):
File "/Users/projetmbc/test.py", line 14, in <module>
test = PPath("dir", "test.txt")
File "/anaconda/lib/python3.4/pathlib.py", line 907, in __new__
self = cls._from_parts(args, init=False)
File "/anaconda/lib/python3.4/pathlib.py", line 589, in _from_parts
drv, root, parts = self._parse_args(args)
File "/anaconda/lib/python3.4/pathlib.py", line 582, in _parse_args
return cls._flavour.parse_parts(parts)
AttributeError: type object 'PPath' has no attribute '_flavour'
What I am doing wrong ?
You can subclass the concrete implementation, so this works:
class Path(type(pathlib.Path())):
Here's what I did with this:
import pathlib
class Path(type(pathlib.Path())):
def open(self, mode='r', buffering=-1, encoding=None, errors=None, newline=None):
if encoding is None and 'b' not in mode:
encoding = 'utf-8'
return super().open(mode, buffering, encoding, errors, newline)
Path('/tmp/a.txt').write_text("я")
Here is the definition of the Path class. It does something rather clever. Rather than directly returning an instance of Path from its __new__(), it returns an instance of a subclass, but only if it's been invoked directly as Path() (and not as a subclass).
Otherwise, it expects to have been invoked via either WindowsPath() or PosixPath(), which both provide a _flavour class attribute via multiple inheritance. You must also provide this attribute when subclassing. You'll probably need to instantiate and/or subclass the _Flavour class to do this. This is not a supported part of the API, so your code might break in a future version of Python.
TL;DR: This idea is fraught with peril, and I fear that my answers to your questions will be interpreted as approval rather than reluctant assistance.
You may be able to simplify your life depending on why you want to extend Path (or PosixPath, or WindowsPath). In my case, I wanted to implement a File class that had all the methods of Path, and a few others. However, I didn't actually care if isinstance(File(), Path).
Delegation works beautifully:
class File:
def __init__(self, path):
self.path = pathlib.Path(path)
...
def __getattr__(self, attr):
return getattr(self.path, attr)
def foobar(self):
...
Now, if file = File('/a/b/c'), I can use the entire Path interface on file, and also do file.foobar().
Combining some of the previous answers you could also just write:
class MyPath(pathlib.Path):
_flavour = type(pathlib.Path())._flavour
I have been struggling with this too.
Here is what i did, studying from the pathlib module.
Seems to me that is the cleaner way to do it, but if the pathlib module changes its implementation, it probably won't hold.
from pathlib import Path
import os
import pathlib
class PPath(Path):
_flavour = pathlib._windows_flavour if os.name == 'nt' else pathlib._posix_flavour
def __new__(cls, *args):
return super(PPath, cls).__new__(cls, *args)
def __init__(self, *args):
super().__init__() #Path.__init__ does not take any arg (all is done in new)
self._some_instance_ppath_value = self.exists() #Path method
def some_ppath_method(self, *args):
pass
test = PPath("dir", "test.txt")
Note
I have opened a bug track here after a little discussion on the Python dev. list.
A temporary solution
Sorry for this double answer but here is a way to achieve what I want. Thanks to Kevin that points me to the source of pathlib and the fact we have here constructors.
import pathlib
import os
def _extramethod(cls, n):
print("=== "*n)
class PathPlus(pathlib.Path):
def __new__(cls, *args):
if cls is PathPlus:
cls = pathlib.WindowsPath if os.name == 'nt' else pathlib.PosixPath
setattr(cls, "extramethod", _extramethod)
return cls._from_parts(args)
test = PathPlus("C:", "Users", "projetmbc", "onefile.ext")
print("File ?", test.is_file())
print("Dir ?", test.is_dir())
print("New name:", test.with_name("new.name"))
print("Drive ?", test.drive)
test.extramethod(4)
This prints the following lines.
File ? False
Dir ? False
New name: C:/Users/projetmbc/new.name
Drive ?
=== === === ===
In order to inherit from pathlib.Path, you need to specify which OS, or "flavour" you're representing. All you need to do is specify that you are using either Windows or Unix (seems to be Unix based on your traceback) by inheriting from pathlib.PosixPath or pathlib.WindowsPath.
import pathlib
class PPath(pathlib.PosixPath):
pass
test = PPath("dir", "test.txt")
print(test)
Which outputs:
dir\test.txt
Using type(pathlib.Path()) as proposed in this answer does the exact same thing as directly inheriting from pathlib.PosixPath or pathlib.WindowsPath since instantiating pathlib.Path "creates either a PosixPath or a WindowsPath" (pathlib documentation).
If you know your application will not be cross-platform, it is simpler to directly inherit from the flavor Path that represents your OS.
Here is a simple way to do things regarding to the observation made by Kevin.
class PPath():
def __init__(self, *args, **kwargs):
self.path = Path(*args, **kwargs)
Then I will need to use a trick so as to automatically bind all the Path's methods to my PPpath class. I think that will be funny to do.
It's work too.
from pathlib import Path
class SystemConfigPath(type(Path())):
def __new__(cls, **kwargs):
path = cls._std_etc()
return super().__new__(cls, path, **kwargs)
#staticmethod
def _std_etc():
return '/etc'
name = SystemConfigPath()
name = name / 'apt'
print(name)
Printed:
/etc/apt
#staticmethod can be replaced by #classmethod
I need to add the following class method to my existing pipeline
http://doc.scrapy.org/en/latest/faq.html#i-m-getting-an-error-cannot-import-name-crawler
i am not sure how to have 2 of these class methods in my class
from twisted.enterprise import adbapi
import MySQLdb.cursors
class MySQLStorePipeline(object):
"""A pipeline to store the item in a MySQL database.
This implementation uses Twisted's asynchronous database API.
"""
def __init__(self, dbpool):
self.dbpool = dbpool
#classmethod
def from_settings(cls, settings):
dbargs = dict(
host= settings['DB_HOST'],
db= settings['DB_NAME'],
user= settings['DB_USER'],
passwd= settings['DB_PASSWD'],
charset='utf8',
use_unicode=True,
)
dbpool = adbapi.ConnectionPool('MySQLdb', **dbargs)
return cls(dbpool)
def process_item(self, item, spider):
pass
From my understanding of class methods, several class methods in a python class should just be fine. It just depends on which one the caller requires. However, I have only seen from_crawler until now in scrapy pipelines. From there you can get access to the settings via crawler.settings
Are you sure that from_settings is required? I did not check all occurences, but in middleware.py priority seems to apply: If a crawler object is available and a from_crawler method exists, this is taken. Otherwise, if there is a from_settings method, that is taken. Otherwise, the raw constructor is taken.
if crawler and hasattr(mwcls, 'from_crawler'):
mw = mwcls.from_crawler(crawler)
elif hasattr(mwcls, 'from_settings'):
mw = mwcls.from_settings(settings)
else:
mw = mwcls()
I admit, I do not know if this is also the place where pipelines get created (I guess not, but there is no pipelines.py), but the implementation seems very reasonable.
So, I'd just either:
reimplement the whole method as from_crawler and only use that one
add method from_crawler and use both
The new method could look like follows (to duplicate as little code as possible):
#classmethod
def from_crawler(cls, crawler):
obj = cls.from_settings(crawler.settings)
obj.do_something_on_me_with_crawler(crawler)
return obj
Of course this depends a bit on what you need.
In openerp 6.1 windows I tried to capture public IP address when people sign-in and sign-out from the HR module. So, in the hr_attendance module I've added ip_address field. Client Public IP address available in the web\common\http.py environ['REMOTE_ADDR']
from osv import fields, osv
from web.common.http import Root
class hr_attendance(osv.osv):
def _get_ipaddress(self, cr, uid, context=None):
ip = None
try:
# environ['REMOTE_ADDR'] how to get
except Exception, e:
pass
return ip
_inherit = "hr.attendance"
_columns = {
'ip_address' : fields.char('IP Address', readonly=True, size=64)
}
_defaults = {
'ip_address': _get_ipaddress,
}
hr_attendance()
How can I get the client IP address or environ['REMOTE_ADDR"] in the default method? Which class and method do I need to override in hr_attendance module?
In my doubt is which method need to call in http.py and what are the parameter to pass get environ['REMOTE_ADDR'] in the HR Module
That looks like it might be difficult to do without changes to the web layer, because the module code doesn't really know anything about the web server layer. The best I could find after a few minutes of digging was a place where you could copy the IP address from the HTTP request to the OpenERP context dictionary. I think if you added this line to the WebRequest.init() method, it might do what you need.
self.context['remote_addr'] = self.httprequest.remote_addr
I think that context will be passed down to the module code where you can read it.