marshmallow - include_fk fail if foreign_key is not int - serialization

On serializing a related database entity with sql-alchemy and marshmallow I encountered following issue:
On dumping this schema the debugger raises a ValueError with the message
self._serialize(d, many=False)
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
return self._serialize(value, attr, obj, **kwargs)
ret = self._format_num(value) # type: _T
return self.num_type(value)
ValueError: invalid literal for int() with base 10: 'sub'
It seems the library tries to cast the the key into an Integer. For readability reasons the key is a String in this case, so the cast obviously fails.
Ist there a flag to avoid casting the foreign_key?
Here the models for reference:
Parent Class
class Operation(db.Model):
__tablename__ = "operation"
key = db.Column(db.String(64), primary_key=True)
label = db.Column(db.String(128), nullable=False)
rules = db.relationship('app.models.rule.Rule', backref="operation")
Child Class
class Rule(db.Model):
__tablename__ = 'rule'
id = db.Column(db.Integer, primary_key=True, autoincrement="auto")
operation_key = db.Column(db.Integer, ForeignKey('operation.key'))
class RuleSchema(ma.SQLAlchemyAutoSchema):
class Meta:
model = Rule
include_fk = True

Found a workaround. Adding a nested field with an only condition for the schema. Api. The serialised json nests the foreign key into a object. 'operation': {'key': 'sub'}
class RuleSchema(ma.SQLAlchemyAutoSchema):
class Meta:
model = Rule
# include_fk = True
operation = ma.Nested(OperationSchema, only=('key',))

Related

Cannot resolve keyword into field error while using drf model serializer and search_fields within field set?

SerializerClass:
class VacancySerializer(serializers.ModelSerializer):
organization_small_name = serializers.CharField(source='organization.short_name', read_only=True)
class Meta:
model = Vacancy
fields = ['organization', 'description', 'specs', 'type', 'publication_date',
'is_published', 'withdrawal_data', 'organization_small_name', ]
read_only_fields = ['publication_date', 'is_published', 'withdrawal_data',]
ViewSet:
class VacancyViewSet(viewsets.ModelViewSet):
queryset = Vacancy.objects.all()
serializer_class = VacancySerializer
filter_backends = [filters.SearchFilter]
search_fields = ['organization_small_name']
...
Model:
class Vacancy(models.Model):
organization = models.OneToOneField(DictOrganization, on_delete=models.CASCADE, related_name='vacancies')
description = models.TextField('Описание')
specs = models.ManyToManyField(DictSpec, blank=True)
type = models.CharField('Тип', max_length=20, choices=VacancyType.choices(), default=VacancyType.PRACTICE.value)
publication_date = models.DateField('Дата публикации', null=True, blank=True)
is_published = models.BooleanField('Опубликовано', default=False)
withdrawal_data = models.DateField('Дата снятия с публикации', null=True, blank=True)
My goal is to make API search by 'organization_small_name' field
that is in VacancySerializer.
Server runs successfully, but as soon as i add ?search parameter, i get next error:
Why it doesn't recognize 'organization_small_name' field, even thought it is decribed in serializer, and how can i fix it?

Field bindings in peewee

How I can link fields in peewee as in sqlalchemy:
parent_id = Column(Integer, ForeignKey("parent.id"))
How to convert this code to peewee?
The very first document in the documentation describes creating a foreign key relationship. Instead of taking the time to write your question and post it here, you could have read that doc. I'm linking it here: http://docs.peewee-orm.com/en/latest/peewee/quickstart.html#model-definition
Example models:
from peewee import *
db = SqliteDatabase('people.db')
class Person(Model):
name = CharField()
birthday = DateField()
class Meta:
database = db # This model uses the "people.db" database.
class Pet(Model):
owner = ForeignKeyField(Person, backref='pets')
name = CharField()
animal_type = CharField()
class Meta:
database = db # this model uses the "people.db" database
For a self-referential foreign key: http://docs.peewee-orm.com/en/latest/peewee/models.html#self-referential-foreign-keys
Example:
class Category(Model):
name = CharField()
parent = ForeignKeyField('self', null=True, backref='children')

How to perform a data migration with Alembic and two versions of my Table?

I'm trying to refactor a database model; separate one column out of a table into another new one. I'd like to do this using existing SQLAlchemy Core models & Alembic. I'd also like to use server-side INSERT ... FROM SELECT ...-style query to migrate data (docs). By avoiding having to copy all the gazillion of rows to Python-world I hope to have maximum scalability, maximum performance and minimum downtime.
My problem is the programmatic use of SQLAlchemy running on two versions of the same table name in a single Metadata context. Should I resort to using an textual SQL instead? 😕
schema.py before:
class User(Base):
__tablename__ = "users"
id = Column(BigInteger, primary_key=True, autoincrement=False, nullable=False)
[...]
profile_picture_url = Column(String, nullable=True)
schema.py after:
class User(Base):
__tablename__ = "users"
id = Column(BigInteger, primary_key=True, autoincrement=False, nullable=False)
[...]
class UserProfileExtras(Base):
__tablename__ = "user_profile_extras"
user_id = Column(BigInteger, ForeignKey("users.id"), index=True, nullable=False)
profile_picture_url = Column(String, nullable=False)
So here's my attempt to create an Alembic upgrade script:
# Import the new/current-in-code models.
from ... import User, UserProfileExtras
# Define the previous User model in order to operate on the current/old schema.
class UserBeforeUpgrade(Base):
__tablename__ = "users"
id = Column(BigInteger, primary_key=True, autoincrement=False, nullable=False)
[...]
profile_picture_url = Column(String, nullable=True)
table_before_upgrade: Table = UserBeforeUpgrade.__table__
new_target_table = UserProfileExtras.__table__
[...]
def upgrade() -> None:
op.create_table(
"user_profile_extras",
sa.Column("user_id", sa.BigInteger(), autoincrement=False, nullable=False),
sa.Column("profile_picture_url", sa.VARCHAR(), nullable=False),
[...]
)
from_user_table = (select([table_before_upgrade.c.id, table_before_upgrade.c.profile_picture_url])
.where(table_before_upgrade.c.profile_picture_url != None))
insert_from = (
new_target_table.insert().from_select(
[new_target_table.c.user_id, new_target_table.c.profile_picture_url],
from_user_table)
)
op.execute(insert_from))
[...]
[...]
Error:
sqlalchemy.exc.InvalidRequestError: Table 'users' is already defined for this MetaData instance.
Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

How to remove SQLAlchemy Many-To-Many Orphans from database?

Context
I have a simple MySQL database written with SQLAlchemy. The following are my two models, Subreddit and Keyword, that have a many-to-many relationship, along with their association table:
subreddits_keywords = db.Table('subreddits_keywords', db.Model.metadata,
db.Column('subreddit_id', db.Integer, db.ForeignKey('subreddits.id', ondelete='CASCADE')),
db.Column('keyword_id', db.Integer, db.ForeignKey('keywords.id', ondelete='CASCADE')),
)
class Subreddit(db.Model, JsonSerializer):
__tablename__ = 'subreddits'
id = db.Column(db.Integer, primary_key=True)
subreddit_name = db.Column(db.String(128), index=True)
# Establish a parent-children relationship (subreddit -> keywords).
keywords = db.relationship('Keyword', secondary=subreddits_keywords, backref='subreddits', cascade='all, delete', passive_deletes=True, lazy='dynamic')
// ...
class Keyword(db.Model, JsonSerializer):
__tablename__ = 'keywords'
id = db.Column(db.Integer, primary_key=True)
keyword = db.Column(db.String(128), index=True)
// ...
As test data, I've created the following data set:
Subreddit:
test_subreddit
Keywords:
test_keyword1
test_keyword2
test_keyword3
In other words, test_subreddit.keywords should return [test_keyword1, test_keyword2, test_keyword3].
Problem
When I remove test_subreddit, test_keyword1, test_keyword2, test_keyword3 still persist in the database.
I understand that with many-to-many relationships, there is technically no parent so cascade's technically will not work according to this post:
https://stackoverflow.com/a/803584/10426919.
What I've Tried
I followed this link: https://github.com/sqlalchemy/sqlalchemy/wiki/ManyToManyOrphan.
This link provides a library function that should fix my exact problem.
However, the function does not work when integrated into my Model file in the following ways:
Method #1:
from app.extensions import db
from werkzeug.security import generate_password_hash, check_password_hash
from sqlalchemy.inspection import inspect
from sqlalchemy_utils import auto_delete_orphans <------ # library
subreddits_keywords = db.Table('subreddits_keywords', db.Model.metadata,
db.Column('subreddit_id', db.Integer, db.ForeignKey('subreddits.id', ondelete='CASCADE')),
db.Column('keyword_id', db.Integer, db.ForeignKey('keywords.id', ondelete='CASCADE')),
)
class Subreddit(db.Model, JsonSerializer):
__tablename__ = 'subreddits'
id = db.Column(db.Integer, primary_key=True)
subreddit_name = db.Column(db.String(128), index=True)
# Establish a parent-children relationship (subreddit -> keywords).
keywords = db.relationship('Keyword', secondary=subreddits_keywords, backref='subreddits', cascade='all, delete', passive_deletes=True, lazy='dynamic')
// ...
class Keyword(db.Model, JsonSerializer):
__tablename__ = 'keywords'
id = db.Column(db.Integer, primary_key=True)
keyword = db.Column(db.String(128), index=True)
// ...
auto_delete_orphans(Subreddit.keywords) <------ # Library function
However, this function does not seem to do anything. There is no error that is output to help guide me towards the right direction. When I check my database in MySQL workbench, the Subreddit, test_subreddit, is deleted, but the keywords [test_keyword1, test_keyword2, test_keyword3] are still in the database under the Keywords table.
Method #2:
I tried integrating the actual function, that the library function is based on, into my code as well:
from app.extensions import db
from werkzeug.security import generate_password_hash, check_password_hash
from sqlalchemy.inspection import inspect
from sqlalchemy_utils import auto_delete_orphans
# for deleting many-to-many "orphans".
from sqlalchemy import event, create_engine
from sqlalchemy.orm import attributes, sessionmaker
subreddits_keywords = db.Table('subreddits_keywords', db.Model.metadata,
db.Column('subreddit_id', db.Integer, db.ForeignKey('subreddits.id', ondelete='CASCADE')),
db.Column('keyword_id', db.Integer, db.ForeignKey('keywords.id', ondelete='CASCADE')),
)
class Subreddit(db.Model, JsonSerializer):
__tablename__ = 'subreddits'
id = db.Column(db.Integer, primary_key=True)
subreddit_name = db.Column(db.String(128), index=True)
# Establish a parent-children relationship (subreddit -> keywords).
keywords = db.relationship('Keyword', secondary=subreddits_keywords, backref='subreddits', cascade='all, delete', passive_deletes=True, lazy='dynamic')
// ...
class Keyword(db.Model, JsonSerializer):
__tablename__ = 'keywords'
id = db.Column(db.Integer, primary_key=True)
keyword = db.Column(db.String(128), index=True)
// ...
engine = create_engine("mysql://", echo=True)
Session = sessionmaker(bind=engine)
#event.listens_for(Session, 'after_flush')
def delete_tag_orphans(session, ctx):
# optional: look through Session state to see if we want
# to emit a DELETE for orphan Tags
flag = False
for instance in session.dirty:
if isinstance(instance, Subreddit) and \
attributes.get_history(instance, 'keywords').deleted:
flag = True
break
for instance in session.deleted:
if isinstance(instance, Subreddit):
flag = True
break
# emit a DELETE for all orphan Tags. This is safe to emit
# regardless of "flag", if a less verbose approach is
# desired.
if flag:
session.query(Keyword).\
filter(~Keyword.subreddits.any()).\
delete(synchronize_session=False)
Again, the keywords persisted despite being attached to no parent.
What I'm trying to accomplish
When children in the database no longer have a parent, I would like them to be removed from the database. What am I doing wrong?
Rather than using auto_delete_orphans, I created a method that I can call when I want to delete children. This method checks the child in question, and sees if it has any parents. If it does have a parent, we leave it be, but if it does not have a parent, we then delete the children.
Here is how I implemented this method, given that a Subreddit is a parent and a Keyword is a child of Subreddit.
def check_for_keyword_orphans(keyword):
# check if each keyword has an associated subreddit
if len(keyword.subreddits) == 0:
db.session.delete(keyword)
return True # keyword deleted
else:
return False # keyword has an associated subreddit
And here is how I used the method in my API route:
keywords = subreddit.keywords
for keyword in keywords:
check_for_keyword_orphans(keyword)
db.session.commit()

Got AttributeError when attempting to get a value for field `email` on serializer `UserSerializer`

Creating Django REST FRamework API. Using Abstract User and email is the default logging parameter using.
AttributeError at /post/
Got AttributeError when attempting to get a value for field email on serializer UserSerializer.
The serializer field might be named incorrectly and not match any attribute or key on the Post instance.
Original exception text was: 'Post' object has no attribute 'email'.
class PostSerializer(ModelSerializer):
category = ReadOnlyField(source='category.name')
author = UserSerializer(source='user.email')
#question = serializers.CharField(source='question.text', read_only=True)
class Meta:
model = Post
fields = '__all__'
class User(AbstractUser):
username = models.CharField("Username", max_length=50, unique=True)
email = models.EmailField("Email Address", max_length=254, unique=True)
USERNAME_FIELD = 'email'
REQUIRED_FIELDS = ['username', 'first_name', 'last_name']
class Post(models.Model):
STATUS_CHOICES = (
('draft', 'Draft'),
('published', 'Published'),
)
title = models.CharField(max_length=250)
slug = models.SlugField(max_length=250, unique_for_date='publish')
author = models.ForeignKey(User, related_name='posts', on_delete=models.CASCADE, to_field='email')
category = models.ForeignKey(Category, related_name='categorys', on_delete=models.CASCADE)
body = models.TextField()
image = models.ImageField(upload_to='blog/%Y/%m/%d', blank=True)
publish = models.DateTimeField(default=timezone.now)
rating = models.IntegerField("Rumor Rate", validators=[MaxValueValidator(5), MinValueValidator(0)], default=1, null=True)
created = models.DateTime[![enter image description here][1]][1]Field(auto_now_add=True)
updated = models.DateTimeField(auto_now=True)
status = models.CharField(max_length=10, choices=STATUS_CHOICES, default='draft')
I suppose you want to return the author's email in the author field when you request the Post endpoint. You may also convert the author field in the serializer to a ReadOnlyField (as category) and specify source='author.email', since your model foreign key is named like that:
class PostSerializer(ModelSerializer):
category = ReadOnlyField(source='category.name')
author = ReadOnlyField(source='author.email')
# ...