Subclass `pathlib.Path` fails - oop

I would like to enhance the class pathlib.Path but the simple example above dose not work.
from pathlib import Path
class PPath(Path):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
test = PPath("dir", "test.txt")
Here is the error message I have.
Traceback (most recent call last):
File "/Users/projetmbc/test.py", line 14, in <module>
test = PPath("dir", "test.txt")
File "/anaconda/lib/python3.4/pathlib.py", line 907, in __new__
self = cls._from_parts(args, init=False)
File "/anaconda/lib/python3.4/pathlib.py", line 589, in _from_parts
drv, root, parts = self._parse_args(args)
File "/anaconda/lib/python3.4/pathlib.py", line 582, in _parse_args
return cls._flavour.parse_parts(parts)
AttributeError: type object 'PPath' has no attribute '_flavour'
What I am doing wrong ?

You can subclass the concrete implementation, so this works:
class Path(type(pathlib.Path())):
Here's what I did with this:
import pathlib
class Path(type(pathlib.Path())):
def open(self, mode='r', buffering=-1, encoding=None, errors=None, newline=None):
if encoding is None and 'b' not in mode:
encoding = 'utf-8'
return super().open(mode, buffering, encoding, errors, newline)
Path('/tmp/a.txt').write_text("я")

Here is the definition of the Path class. It does something rather clever. Rather than directly returning an instance of Path from its __new__(), it returns an instance of a subclass, but only if it's been invoked directly as Path() (and not as a subclass).
Otherwise, it expects to have been invoked via either WindowsPath() or PosixPath(), which both provide a _flavour class attribute via multiple inheritance. You must also provide this attribute when subclassing. You'll probably need to instantiate and/or subclass the _Flavour class to do this. This is not a supported part of the API, so your code might break in a future version of Python.
TL;DR: This idea is fraught with peril, and I fear that my answers to your questions will be interpreted as approval rather than reluctant assistance.

You may be able to simplify your life depending on why you want to extend Path (or PosixPath, or WindowsPath). In my case, I wanted to implement a File class that had all the methods of Path, and a few others. However, I didn't actually care if isinstance(File(), Path).
Delegation works beautifully:
class File:
def __init__(self, path):
self.path = pathlib.Path(path)
...
def __getattr__(self, attr):
return getattr(self.path, attr)
def foobar(self):
...
Now, if file = File('/a/b/c'), I can use the entire Path interface on file, and also do file.foobar().

Combining some of the previous answers you could also just write:
class MyPath(pathlib.Path):
_flavour = type(pathlib.Path())._flavour

I have been struggling with this too.
Here is what i did, studying from the pathlib module.
Seems to me that is the cleaner way to do it, but if the pathlib module changes its implementation, it probably won't hold.
from pathlib import Path
import os
import pathlib
class PPath(Path):
_flavour = pathlib._windows_flavour if os.name == 'nt' else pathlib._posix_flavour
def __new__(cls, *args):
return super(PPath, cls).__new__(cls, *args)
def __init__(self, *args):
super().__init__() #Path.__init__ does not take any arg (all is done in new)
self._some_instance_ppath_value = self.exists() #Path method
def some_ppath_method(self, *args):
pass
test = PPath("dir", "test.txt")

Note
I have opened a bug track here after a little discussion on the Python dev. list.
A temporary solution
Sorry for this double answer but here is a way to achieve what I want. Thanks to Kevin that points me to the source of pathlib and the fact we have here constructors.
import pathlib
import os
def _extramethod(cls, n):
print("=== "*n)
class PathPlus(pathlib.Path):
def __new__(cls, *args):
if cls is PathPlus:
cls = pathlib.WindowsPath if os.name == 'nt' else pathlib.PosixPath
setattr(cls, "extramethod", _extramethod)
return cls._from_parts(args)
test = PathPlus("C:", "Users", "projetmbc", "onefile.ext")
print("File ?", test.is_file())
print("Dir ?", test.is_dir())
print("New name:", test.with_name("new.name"))
print("Drive ?", test.drive)
test.extramethod(4)
This prints the following lines.
File ? False
Dir ? False
New name: C:/Users/projetmbc/new.name
Drive ?
=== === === ===

In order to inherit from pathlib.Path, you need to specify which OS, or "flavour" you're representing. All you need to do is specify that you are using either Windows or Unix (seems to be Unix based on your traceback) by inheriting from pathlib.PosixPath or pathlib.WindowsPath.
import pathlib
class PPath(pathlib.PosixPath):
pass
test = PPath("dir", "test.txt")
print(test)
Which outputs:
dir\test.txt
Using type(pathlib.Path()) as proposed in this answer does the exact same thing as directly inheriting from pathlib.PosixPath or pathlib.WindowsPath since instantiating pathlib.Path "creates either a PosixPath or a WindowsPath" (pathlib documentation).
If you know your application will not be cross-platform, it is simpler to directly inherit from the flavor Path that represents your OS.

Here is a simple way to do things regarding to the observation made by Kevin.
class PPath():
def __init__(self, *args, **kwargs):
self.path = Path(*args, **kwargs)
Then I will need to use a trick so as to automatically bind all the Path's methods to my PPpath class. I think that will be funny to do.

It's work too.
from pathlib import Path
class SystemConfigPath(type(Path())):
def __new__(cls, **kwargs):
path = cls._std_etc()
return super().__new__(cls, path, **kwargs)
#staticmethod
def _std_etc():
return '/etc'
name = SystemConfigPath()
name = name / 'apt'
print(name)
Printed:
/etc/apt
#staticmethod can be replaced by #classmethod

Related

How to create a callable mock from a non-callable spec?

I have two classes: class One that does some stuff and class Wrapper that translates the API of One. To test the Wrapper I want to mock the class One. My problem is that One sets a number of attributes in it's __init__ method and I'd like my test to throw an error when these attributes are changed (e.g. renamed or deleted) in One but not in Wrapper.
I have tried to initialize an instance of One and to use it as a spec, but the resulting mock is non-callable:
from unittest import mock
import pytest
class One:
def __init__(self, a):
self.a = a
spec_obj = One(a='foo')
# working code, the test should pass
class Wrapper:
def __init__(self, wa):
self._one = One(a=wa)
self.wa = self._one.a
#mock.patch(__name__ + '.One', autospec=spec_obj)
def test_wrapper(MockOne):
MockOne.return_value.a = test_a = 'bar'
wrapper_obj = Wrapper(wa=test_a)
MockOne.assert_called_once_with(a=test_a)
assert wrapper_obj.wa == test_a
which throws the error:
TypeError: 'NonCallableMagicMock' object is not callable
since the spec spec_obj is non-callable.
If I set autospec=True, everything works but the test passes even when the One's attribute is renamed.
MockOne.return_value produces a Mock that you configure, but when the mock is actually called, you get a different Mock that hasn't been properly configured. You need to configure MockOne.return_value directly.
#mock.patch('One', autospec=True)
def test_wrapper(MockOne):
MockOne.return_value.a = test_a = 'bar'
wrapper_obj = Wrapper(wa=test_a)
MockOne.assert_called_once_with(a=test_a)
assert wrapper_obj.wa == test_a

Can`t get QTreeView model of QSortFilterProxyModel() outside of the init function

I have an application which displays JSON-file to the QTreeView. For this I created my own QJsonTreeModel(QAbstractItemModel) class for QTreeView model
In my MainWindow.py I implemented it like this:
class MainWindow(QMainWindow):
def __init__(self, json_text: dict) -> None:
self.tree_view = QTreeView()
self.model = QJsonTreeModel()
self.tree_view.setContextMenuPolicy(Qt.CustomContextMenu)
self.tree_view.customContextMenuRequested.connect(self.open_right_click_menu)
self.tree_view.setModel(self.model)
self.model.load(self.json_text)
def open_right_click_menu(self, position) -> None:
index = self.tree_view.selectionModel().currentIndex()
parent = index.parent()
if not index.isValid():
return
# workes fine
print(self.model.data(self.tree_view.selectedIndexes()[2], Qt.EditRole))
It works perfect but recently I needed to add search QLineEdit() field to find the input element in the tree. And for this task I found soultion with QSortFilterProxyModel(). Which I implemented like this:
class MainWindow(QMainWindow):
def __init__(self, json_text: dict) -> None:
self.tree_view = QTreeView()
self.line_edit = QLineEdit()
self.model = QJsonTreeModel()
self.model.load(self.json_text)
self.tree_view.setContextMenuPolicy(Qt.CustomContextMenu)
self.tree_view.customContextMenuRequested.connect(self.open_right_click_menu)
self.filter_proxy_model = QSortFilterProxyModel()
self.filter_proxy_model.setSourceModel(self.model)
self.filter_proxy_model.setFilterCaseSensitivity(Qt.CaseInsensitive) # Qt.CaseSensitive
self.filter_proxy_model.setRecursiveFilteringEnabled(True)
self.filter_proxy_model.setFilterKeyColumn(-1)
self.tree_view.setModel(self.filter_proxy_model)
self.line_edit.textChanged.connect(self.filter_proxy_model.setFilterRegExp)
def open_right_click_menu(self, position) -> None:
index = self.tree_view.selectionModel().currentIndex()
parent = index.parent()
if not index.isValid():
return
# not working rn. Segmentation fault
print(self.model.data(self.tree_view.selectedIndexes()[2], Qt.EditRole))
# also not working
print(self.filter_proxy_model.sourceModel().data(self.tree_view.selectedIndexes()[2], Qt.EditRole))
It works fine, the input text is searched in the tree and displayed normally, but the problem is in other place. I have implemented right click menu for my QTreeView:
self.tree_view.setContextMenuPolicy(Qt.CustomContextMenu)
self.tree_view.customContextMenuRequested.connect(self.open_right_click_menu)
And in open_right_click_menu function I need to call self.model.data(self.tree_view.selectedIndexes()[2], Qt.EditRole) (index is correct, there are three columns) function or some another function of QJsonTreeModel() class which worked perfectly before adding QSortFilterProxyModel(). But after implementing QSortFilterProxyModel() I have got Segmentation fault when I am trying to call some function of QJsonTreeModel() class. Also I tried self.filter_proxy_model.sourceModel().data(self.tree_view.selectedIndexes()[2], Qt.EditRole) but it also causes Segmentation fault. Where am I wrong and how do I call functions of QJsonTreeModel() class properly?
I didn't find this question before I asked mine, but the problem was that I did`t map QSortFilterProxyModel back. So the solution is this:
def open_right_click_menu(self, position) -> None:
# not working version
print(self.model.data(self.tree_view.selectedIndexes()[2], Qt.EditRole))
# working version
print(self.model.data(
self.filter_proxy_model.mapToSource(self.tree_view.selectedIndexes()[2]), Qt.EditRole))

How to do filtering according to log in user?

I want to get hobbys according to log in user but I am always getting
TypeError at /backend/api/hobbys/
init() takes 1 positional argument but 2 were given
this is my views.py
class ListCreateHobbyView(GenericAPIView):
queryset = Hobby.objects.all()
serializer_class = HobbySerializer
# Filtering by logged in user
def get(self, request, *args, **kwargs):
queryset = Hobby.objects.filter(user=request.user)
serializer = self.get_serializer(queryset, many=True)
return Response(serializer.data)
What can be wrong?
Please do not override the entire .get(…) method. This means that (nearly) all boilerplate code that Django has written is no longer applied. You filter in the .get_queryset(…) method [drf-doc]:
from rest_framework.generics import ListAPIView
class ListCreateHobbyView(ListAPIView):
queryset = Hobby.objects.all()
serializer_class = HobbySerializer
def get_queryset(self, *args, **kwargs):
return super().get_queryset(*args, **kwargs).filter(
user=self.request.user
)
You can also make a custom filter backend: this is a reusable component that you then can use in other views. This thus looks like:
from rest_framework import filters
class UserFilterBackend(filters.BaseFilterBackend):
def filter_queryset(self, request, queryset, view):
return queryset.filter(user=request.user)
then you use this as filter backend:
from rest_framework.generics import ListAPIView
class ListCreateHobbyView(ListAPIView):
queryset = Hobby.objects.all()
serializer_class = HobbySerializer
filter_backends = [UserFilterBackend]
The advantage of this approach is that if you later construct extra views that need the same filtering, you can simply "plug" these in.

Python do not serialize object which is not imported in socket

I have written following simple socket application in which object defined in separate file and it works.
# models_object.py
class Person(object):
def __init__(self,name,surname,age):
self.name = name
self.surname = surname
self.age = age
def __str__(self):
return self.name + " " + self.surname
#client.py
import cPickle,socket
from model_objects import Person
def client():
client = socket.socket()
host = socket.gethostname()
port = 8000
adem_kerenci = Person("Adem","Kerenci",22)
serialized_object = cPickle.dumps(adem_kerenci)
client.connect((host, port))
client.send(serialized_object)
client.close()
client()
#service.py
import cPickle,socket
def server():
server = socket.socket()
host = socket.gethostname()
port = 8000
server.bind((host, port))
server.listen(1)
connection, address = server.accept()
recieved_data = connection.recv(2**20)
print cPickle.loads(recieved_data)
connection.close()
server()
However, if I write Person class definition into client.py, code raises following error.
Traceback (most recent call last):
File "service.py", line 14, in <module>
server()
File "service.py", line 11, in server
print cPickle.loads(recieved_data)
AttributeError: 'module' object has no attribute 'Person'
Is there any relation between importing and serializing?
Note: We tried python 2.7
From documentation:
Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.
This means that the object you are unpickling in the server should be importable too from the server code with the same "fully qualified" name which in the first example is: model_objects.Person. This is apparent in the first case if you try splitting the client.py and model_objects.py in one folder and service.py in another folder which will give you the same error as it is trying to import model_objects.py inside service.py but it couldn't find it.
So in the second case you can copy the class Person as well to the server code and it will work but this is bad practice or make a shared folder for both client and server which is shared production code as well (better practice).
This is because during unpickling python needs the class of the object to reinstantiate the object and use it.

How do Scrapy from_settings and from_crawler class methods work?

I need to add the following class method to my existing pipeline
http://doc.scrapy.org/en/latest/faq.html#i-m-getting-an-error-cannot-import-name-crawler
i am not sure how to have 2 of these class methods in my class
from twisted.enterprise import adbapi
import MySQLdb.cursors
class MySQLStorePipeline(object):
"""A pipeline to store the item in a MySQL database.
This implementation uses Twisted's asynchronous database API.
"""
def __init__(self, dbpool):
self.dbpool = dbpool
#classmethod
def from_settings(cls, settings):
dbargs = dict(
host= settings['DB_HOST'],
db= settings['DB_NAME'],
user= settings['DB_USER'],
passwd= settings['DB_PASSWD'],
charset='utf8',
use_unicode=True,
)
dbpool = adbapi.ConnectionPool('MySQLdb', **dbargs)
return cls(dbpool)
def process_item(self, item, spider):
pass
From my understanding of class methods, several class methods in a python class should just be fine. It just depends on which one the caller requires. However, I have only seen from_crawler until now in scrapy pipelines. From there you can get access to the settings via crawler.settings
Are you sure that from_settings is required? I did not check all occurences, but in middleware.py priority seems to apply: If a crawler object is available and a from_crawler method exists, this is taken. Otherwise, if there is a from_settings method, that is taken. Otherwise, the raw constructor is taken.
if crawler and hasattr(mwcls, 'from_crawler'):
mw = mwcls.from_crawler(crawler)
elif hasattr(mwcls, 'from_settings'):
mw = mwcls.from_settings(settings)
else:
mw = mwcls()
I admit, I do not know if this is also the place where pipelines get created (I guess not, but there is no pipelines.py), but the implementation seems very reasonable.
So, I'd just either:
reimplement the whole method as from_crawler and only use that one
add method from_crawler and use both
The new method could look like follows (to duplicate as little code as possible):
#classmethod
def from_crawler(cls, crawler):
obj = cls.from_settings(crawler.settings)
obj.do_something_on_me_with_crawler(crawler)
return obj
Of course this depends a bit on what you need.