I'd like to represent the following json by Pydantic model:
{
"sip" {
"param1": 1
}
"param2": 2
...
}
Means json may contain sip field and some other field, any number any names, so I'd like to have model which have sip:Optional[dict] field and some kind of "rest", which will be correctly parsed from/serialized to json. Is it possible?
Maybe you are looking for the extra model config:
extra
whether to ignore, allow, or forbid extra attributes during model initialization. Accepts the string values of 'ignore', 'allow', or 'forbid', or values of the Extra enum (default: Extra.ignore). 'forbid' will cause validation to fail if extra attributes are included, 'ignore' will silently ignore any extra attributes, and 'allow' will assign the attributes to the model.
Example:
from typing import Any, Dict, Optional
import pydantic
class Foo(pydantic.BaseModel):
sip: Optional[Dict[Any, Any]]
class Config:
extra = pydantic.Extra.allow
foo = Foo.parse_raw(
"""
{
"sip": {
"param1": 1
},
"param2": 2
}
"""
)
print(repr(foo))
print(foo.json())
Output:
Foo(sip={'param1': 1}, param2=2)
{"sip": {"param1": 1}, "param2": 2}
Related
I have a complex SCIM attribute that looks like follows:
"myattr1": {
"subattr1": 5,
"subattr2": [1, 2, 3]
}
I want to modify this to become
"myattr1": {
"subattr1": 5,
"subattr2": [1, 3]
}
How can I do this using PATCH ? Should I replace the entire sub-attribute or can I just remove the value 2 from it using PATCH ?
I know how to do this with multi-valued attributes. But I don't know how to do it for sub-attributes.
[EDIT: This is wrong..]
I believe this will work:
PATCH /resource/id
{ "schemas":
["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
"Operations":[
{
"op":"remove",
"path":"myattr1[subattr2 eq \"2\"]"
}
]
}
Example on path was taken from https://datatracker.ietf.org/doc/html/rfc7644#page-33 where it mentions "path":"members[value eq
"2819c223-7f76-453a-919d-413861904646"].displayName" as a way to target the displayName sub-attribute for the complex multi-valued group attribute "members".
The escaped quotes around 2 are necessary if it's a string - if the value is in fact an integer, they won't be necessary.
As per the PingIdentity documentation https://github.com/pingidentity/scim2/wiki/Working-with-SCIM-paths#the-value-sub-attribute, Simple multivalued attributes have a special implicit sub-attribute called "value".
If that is the way, your PATCH request payload should be as follows.
{
"schemas": [
"urn:ietf:params:scim:api:messages:2.0:PatchOp"
],
"Operations": [
{
"op": "remove",
"path": "myattr1.subattr2[value eq \"2\"]"
}
]
}
However, this type of patch operation is not clearly defined in RFC 7644 (https://datatracker.ietf.org/doc/html/rfc7644#section-3.5.2)
You would be able to confirm this if the question is raised in SCIM mailing list https://mailarchive.ietf.org/arch/browse/scim/
Introduction
Let's say I have a Kotlin enum class:
enum class Type {
ONE,
TWO,
THREE,
FOUR
}
and a following data class:
data class Item(
val name: String,
val type: Type
)
Then I have a Single that emits a list of Items – can by anything but for example purposes, let's say it looks like that:
val itemsSingle = Single.just(listOf(
Item("A", Type.ONE),
Item("B", Type.ONE),
Item("C", Type.TWO),
Item("D", Type.THREE),
))
Problem
What I'd like to achieve is to have an RxJava stream that will output a map where keys come from Type and values are lists of Items matching a given Type value (where an undetermined, unsorted list of Items is provided by some other Single stream). The signature would be:
Single<Map<Type, List<Item>> // or Observable<Map<Type, List<Item>>
One additional requirement is that the map's keys should always exhaust all values from Type enum even if the itemsSingle stream contains no items for some Type values (or no items at all). So, for the provided itemsSingle example stream the returned map should look like this:
{
ONE: [ Item(name: "A", type: ONE), Item(name: "B", type: ONE) ],
TWO: [ Item(name: "C", type: TWO) ],
THREE: [ Item(name: "D", type: THREE) ],
FOUR: []
}
Attempt
With all the above, I've kinda achieved the desired result with following steps:
To satisfy the requirement of exhausting all Type enum values I first create a map that has an empty list for all possible Type values:
val typeValuesMap = Type.values().associate { it to emptyList<Item>() }
val typeValuesMapSingle = Single.just(typeValuesMap)
// result: {ONE=[], TWO=[], THREE=[], FOUR=[]}
I can get a map that contains items from itemsSingle grouped under respective Type value keys:
val groupedItemsMapSingle = itemsSingle.flattenAsObservable { it }
.groupBy { it.type }
.flatMapSingle { it.toList() }
.toMap { list -> list[0].type } // the list is guaranteed to have at least one item
// result: {ONE=[Item(name=A, type=ONE), Item(name=B, type=ONE)], THREE=[Item(name=D, type=THREE)], TWO=[Item(name=C, type=TWO)]}
finally I can combine both lists using the combineLatest operator and overwriting initial empty list of items for a given Type value if itemsSingle contained any Items for this Type value:
Observable.combineLatest(
typeValuesMapSingle.flattenAsObservable { it.entries },
groupedItemsMapSingle.flattenAsObservable { it.entries }
) { a, b -> listOf(a, b) }
.defaultIfEmpty(typeValuesMap.entries.toList()) // in case itemsSingle is empty
.flatMapIterable { it }
.collect({mutableMapOf<Type, List<Item>>()}, { a, b -> a[b.key] = b.value})
// result: {FOUR=[], ONE=[Item(name=A, type=ONE), Item(name=B, type=ONE)], THREE=[Item(name=D, type=THREE)], TWO=[Item(name=C, type=TWO)]}
Summary
As you can see, it's quite a lot of code for a seemingly simple operation. So my question is – is there a simpler way to achieve the result I'm after?
Just merge a map of empty lists with a map of filled lists
val result = itemsSingle.map { items->
Type.values().associateWith { listOf<Item>() } + items.groupBy { it.type }
}
Request Json Looks like the below :-
{
"eNumber": 8506493,
"details": [
{
"id":12345,
"name": xyz123
}
]
}
As part of requirement, I need to check the "details" array that either "id" or "name" field must present. if "id" field is present, "name" is non-mandatory. if "name" field is present, "id" is non-mandatory. Throws error if not met.
I tried few options using filter the details array and check the size of the filtered array in the validation component. It does not seems to be working well. if anyone has got better solutions.please share it here.
This example code will return true or false if it passes your condition
%dw 2.0
import * from dw::core::Arrays
output application/json
---
payload.details every ((item) -> item.id? or item.name?)
The function I'm using is every that checks that all elements in an array passes the given criteria.
Later you can use a choice and a use the raise error or you can use the fail dw function with an if else.
You can restrict it at RAML level.
Sample RAML -
#%RAML 1.0
title: api
types:
test1:
type: object
properties:
id:
type: string
example: 123a
test2:
type: object
properties:
name:
type: string
example: xyz124
test3:
type: object
properties:
enumber:
type: string
example: 8506493a
details :
type: [test1 | test2]
/test:
post:
body:
application/json:
type: test3
I've got an API endpoint POST /data.
The received data is formatted in a certain way which is different from the way I store it in the db.
I'll use geometry type from postgis as an example.
class MyPostgisModel(models.Model):
...
position = models.PointField(null=True)
my_charfield = models.CharField(max_length=10)
...
errors = JSONField() # Used to save the cleaning and validation errors
class MyPostgisSerializer(serializers.ModelSerializer):
class Meta:
model = MyPostgisModel
fields = [
...
"position",
...
"my_charfield",
"errors",
]
def to_internal_value(self, data):
...
# Here the data is coming in the field geometry but in the db, it's called
# position. Moreover I need to apply the `GEOSGeometry(json.dumps(...))`
# method as well.
data["position"] = GEOSGeometry(json.dumps(data["geometry"]))
return data
The problem is that there is not only one field like position but many. And I would like (maybe wrongly) to do like the validate_*field_name* scheme but for cleaning (clean_*field_name*).
There is another problem. In this scheme, I would like to still save the rest of the data in the database even if some fields have raised ValidationError (eg: a CharField that is too long) but are not part of the primary_key/a unique_together constraint. And save the related errors into a JSONField like this:
{
"cleaning_errors": {
...
"position": 'Invalid format: {
"type": "NotAValidType", # Should be "Point"
"coordinates": [
4.22,
50.67
]
}'
...
},
"validating_errors": {
...
"my_charfield": "data was too long: 'this data is way too long for 10 characters'",
...
}
}
For the first problem, I thought of doing something like this:
class BaseSerializerCleanerMixin:
"""Abstract Mixin that clean fields."""
def __init__(self, *args, **kwargs):
"""Initialize the cleaner strategy."""
# This is the error_dict to be filled by the `clean_*field_name*`
self.cleaning_error_dict = {}
super().__init__(*args, **kwargs)
def clean_fields(self, data):
"""Clean the fields listed in self.fields_to_clean before validating them."""
cleaned_data = {}
for field_name in getattr(self.Meta, "fields", []):
cleaned_field = (
getattr(self, "clean_" + field_name)(data)
if hasattr(self, "clean_" + field_name)
else data.get(field_name)
)
if cleaned_field is not None:
cleaned_data[field_name] = cleaned_field
return cleaned_data
def to_internal_value(self, data):
"""Reformat data to put it in the database."""
cleaned_data = self.clean_fields(data)
return super().to_internal_value(cleaned_data)
I'm not sure that's a good idea and maybe there is an easy way to deal with such things.
For the second problem ; catching the errors of the validation without specifying with is_valid() returning True when no primary_key being wrongly formatted, I'm not sure how to proceed.
I am trying to create a Schema class using nested dictionaries that has some list as elements. However when I do a dumps() Only the top level elements are dumped.
Have a rest api that returns a list of certain things,eg. list of users. but the schema is such that certain aggregate details are sent at the top level, the data looks something like this. This is what i am expecting as output:
{
"field1": 5,
"field2": false,
"field3": {
"field4": 40,
"field5": [
{
"field6": "goo goo gah gah",
"field7": 99.341879,
"field8": {
"field9": "goo goo gah gah",
"field10": "goo goo gah gah"
}
}]
}
}
Heres my code:
MySchema = Schema.from_dict(
{
"field1": fields.Int(),
"field2": fields.Bool(),
"field3": {
"field4": fields.Int(),
"field5": [
{
"field6": fields.Str(),
"field7": fields.Float(),
"field8": {
"field9": fields.Str(),
"field10": fields.Str()
}
}]
}
}
)
#Then use it like:
response = MySchema().dumps(data)
Actual result:
"{\"field1\": 5, \"field2\": false}"
Option 1
You're looking for several nested schemas, interconnected through fields.Nested:
from marshmallow import Schema, fields
Field8Schema = Schema.from_dict({
"field9": fields.Str(),
"field10": fields.Str()
})
Field5Schema = Schema.from_dict({
"field6": fields.Str(),
"field7": fields.Float(),
"field8": fields.Nested(Field8Schema),
})
Field3Schema = Schema.from_dict({
"field4": fields.Int(),
"field5": fields.List(fields.Nested(Field5Schema))
})
MySchema = Schema.from_dict({
"field1": fields.Int(),
"field2": fields.Bool(),
"field3": fields.Nested(Field3Schema),
})
MySchema().dump(data)
# {'field2': False,
# 'field1': 5,
# 'field3': {'field4': 40,
# 'field5': [{'field6': 'goo goo gah gah',
# 'field8': {'field9': 'goo goo gah gah', 'field10': 'goo goo gah gah'},
# 'field7': 99.341879}]}}
Option 2
If the nesting won't be that deep, it might be simpler to use decorators, i.e. nest and unnest data as suggested in the docs:
class UserSchema(Schema):
#pre_load(pass_many=True)
def remove_envelope(self, data, many, **kwargs):
namespace = 'results' if many else 'result'
return data[namespace]
#post_dump(pass_many=True)
def add_envelope(self, data, many, **kwargs):
namespace = 'results' if many else 'result'
return {namespace: data}
It feels it fits your case nicely.
Comments
I'd suggest not to use from_dict as it is less readable for such a complex data, and instead switch to a class-based schema.
There's plenty of good examples of nesting in the docs.