Document serialization with Doctrine MongoDB ODM - serialization

I'm trying to code a class handling serialization of documents by reading their metadata. I got inspired by this implementation for entities with Doctrine ORM and modified it to match how Doctrine ODM handles documents. Unfortunatly something is not working correctly as one document is never serialized more than once even if it is refered a 2nd time thus resulting on incomplete serialization.
For example, it outputs this (in json) for a user1 (see User document) that belongs to some place1 (see Place document). Then it outputs the place and the users belonging to it where we should see the user1 again but we don't :
{
id: "505cac0d6803fa1e15000004",
login: "user1",
places: [
{
id: "505cac0d6803fa1e15000005",
code: "place1",
users: [
{
id: "505c862c6803fa6812000000",
login: "user2"
}
]
}
]
}
I guess it could be related to something preventing circular references but is there a way around it ?
Also, i'm using this in a ZF2 application, would there be a better way to implement this using the ZF2 Serializer ?
Thanks for your help.

I have a serializer already written for DoctrineODM. You can find it in http://github.com/superdweebie/DoctrineExtensions - look in lib/Sds/DoctrineExtensions/Serializer.
If you are are using zf2, then you might also like http://github.com/superdweebie/DoctrineExtensionsModule, which configures DoctrineExtensions for use in zf2.
To use the Module, install it with composer, as you would any other module. Then add the following to your zf2 config:
'sds' => [
'doctrineExtensions' => [
'extensionConfigs' => [
'Sds\DoctrineExtensions\Serializer' => null,
),
),
),
To get the serializer use:
$serializer = $serivceLocator->get('Sds\DoctrineExtensions\Serializer');
To use the serializer:
$array = $serializer->toArray($document)
$json = $serializer->toJson($document)
$document = $serializer->fromArray($array)
$document = $serializer->fromJson($json)
There are also some extra annotations available to control serialization, if you want to use them:
#Sds\Setter - specify a non standard setter for a property
#Sds\Getter - specify a non standard getter fora property
#Sds\Serializer(#Sds\Ignore) - ignore a property when serializing
It's all still a work in progress, so any comments/improvements would be much appreciated. As you come across issues with these libs, just log them on github and they will get addressed promptly.
Finally a note on serializing embedded documents and referenced documents - embedded documents should be serialized with their parent, while referenced documents should not. This reflects the way data is saved in the db. It also means circular references are not a problem.
Update
I've pushed updates to Sds/DoctrineExtensions/Serializer so that it can now handle references properly. The following three (five) methods have been updated:
toArray/toJson
fromArray/fromJson
applySerializeMetadataToArray
The first two are self explainitory - the last is to allow serialization rules to be applied without having to hydrate db results into documents.
By default references will be serialized to an array like this:
[$ref: 'CollectionName/DocumentId']
The $ref style of referencing is what Mongo uses internally, so it seemed appropriate. The format of the reference is given with the expectation it could be used as a URL to a REST API.
The default behaviour can be overridden by defineing an alternative ReferenceSerializer like this:
/**
* #ODM\ReferenceMany(targetDocument="MyTargetDocument")
* #Sds\Serializer(#Sds\ReferenceSerializer('MyAlternativeSerializer'))
*/
protected $myDocumentProperty;
One alternate ReferenceSerializer is already included with the lib. It is the eager serializer - it will serialize references as if they were embedded documents. It can be used like this:
/**
* #ODM\ReferenceMany(targetDocument="MyTargetDocument")
* #Sds\Serializer(#Sds\ReferenceSerializer('Sds\DoctrineExtensions\Serializer\Reference\Eager'))
*/
protected $myDocumentProperty;
Or an alternate shorthand annotation is provided:
/**
* #ODM\ReferenceMany(targetDocument="MyTargetDocument")
* #Sds\Serializer(#Sds\Eager))
*/
protected $myDocumentProperty;
Alternate ReferenceSerializers must implement Sds\DoctrineExtensions\Serializer\Reference\ReferenceSerializerInterface
Also, I cleaned up the ignore annotation, so the following annotations can be added to properties to give more fine grained control of serialization:
#Sds\Serializer(#Sds\Ignore('ignore_when_serializing'))
#Sds\Serializer(#Sds\Ignore('ignore_when_unserializing'))
#Sds\Serializer(#Sds\Ignore('ignore_always'))
#Sds\Serializer(#Sds\Ignore('ignore_never'))
For example, put #Sds\Serializer(#Sds\Ignore('ignore_when_serializing')) on an email property - it means that the email can be sent upto the server for update, but can never be serialized down to the client for security.
And lastly, if you hadn't noticed, sds annotations support inheritance and overriding, so they play nice with complex document structures.

Another very simple, framework independent way to transforming Doctrine ODM Document to Array or JSON - http://ajaxray.com/blog/converting-doctrine-mongodb-document-tojson-or-toarray
This solution gives you a Trait that provides toArray() and toJSON() functions for your ODM Documents. After useing the trait in your Document, you can do -
<?php
// Assuming in a Symfony2 Controller
// If you're not, then make your DocmentManager as you want
$dm = $this->get('doctrine_mongodb')->getManager();
$report = $dm->getRepository('YourCoreBundle:Report')->find($id);
// Will return simple PHP array
$docArray = $report->toArray();
// Will return JSON string
$docJSON = $report->toJSON();
BTW, it will work only on PHP 5.4 and above.

Related

Laravel Scout toSearchableArray attribute is not filterable

I've been doing some testing with laravel scout and according to the documentation (https://laravel.com/docs/8.x/scout#configuring-searchable-data), I've mapped my User model as such:
/**
* Get the indexable data array for the model.
*
* #return array
*/
public function toSearchableArray()
{
$data = $this->toArray();
return array_merge($data, [
'entity' => 'An entity'
]);
}
Just for the sake of testing, this is literally what I came down to on the debugging.
After importing the User model with this mapping, I can see on the meilisearch dashboard it is indeed showing the user data + the entity = 'An entity'.
However, when applying this:
User::search('something')->where('entity', 'An entity')->get()
It produces the following error:
"message": " --> 1:1\n |\n1 | entity=\"An entity\"\n | ^----^\n |\n = attribute `entity` is not filterable, available filterable attributes are: ",
"exception": "MeiliSearch\\Exceptions\\ApiException",
"file": "/var/www/api/vendor/meilisearch/meilisearch-php/src/Http/Client.php",
Tracing back to view the 'filterable attributes', I've ended at the conclusion that:
$client = app(\MeiliSearch\Client::class);
dump($client->index('users')->getFilterableAttributes()); // Returns []
$client->index('users')->updateFilterableAttributes(['entity']);
dump($client->index('users')->getFilterableAttributes()); // Returns ['entity']
Forcing the updateFilterableAttributes now allows me to complete the search as intended, but I don't feel this should be the regular behaviour? If its mapped on the searchableArray, it should be searchable? What am I not understanding and what other approaches are there to achieve this goal?
This is actually not an issue but a requirement of meilisearch in particular. Scout under the hood uses different drivers for indexing - "algolia", "meilisearch", "database", "collection" and even "null", all of them have different indexing methods unifing of which would be troublesome and inefficient for scout I believe.
So filtering or a faceted search, as meilisearch refers to it, requires us to establish filtering criteria first, which is empty by default for document (models in laravel) fields.
Quoting from the docs:
This step is mandatory and cannot be done at search time. Filters need
to be properly processed and prepared by Meilisearch before they can
be used.
Updating filterableAttributes requires recreating the entire
index. This may take a significant amount of time depending on your
dataset size.
For more info please refer to meilisearch official docs https://docs.meilisearch.com/learn/advanced/filtering_and_faceted_search.html

How to get all FAL File Objects which are referenced?

I'm trying to make a extbase extension for TYPO3 to get alle file objects with mimetype image/... which referenced by any content, plugin or fluid in typo3.
But i don't know which is the best way to get these data. How should i create a model in my extension and how should i create the correct repository?
If i create a custom query i'm not sure how to return a complete FAL Object which contains any data (like metadata) etc.
hope someone could help me to find the right way, and maybe has a example or something.
thanks a lot
You could do it like this, details are at the bottom:
Get all file references.
Go through them, retrieve the referenced file for each of them and retain only the ones where the field mime_type starts with image/.
There are two things you probably need to watch out for:
The field mime_type needs to be up to date. Check the FAL scheduler indexing task for that.
Performance. Depending on the number of files you have, it could be much faster to do this with a custom SQL statement which makes use of a JOIN. But you should only do that if performance is a problem.
How to get all file references:
First, build your own empty file reference class:
namespace Vendor/Extkey/Domain/Model;
class FileReference extends \TYPO3\CMS\Extbase\Domain\Model\FileReference {}
Make sure to configure it in your TypoScript to be serialized to the table sys_file_reference:
config.tx_extbase.persistence {
classes {
Vendor\Extkey\Domain\Model\FileReference {
mapping {
tableName = sys_file_reference
}
}
}
}
Add a repository for the references:
namespace Vendor/Extkey/Domain/Repository;
class FileReferenceRepository extends \TYPO3\CMS\Extbase\Persistence\Repository {
public function initializeObject() {
/** #var \TYPO3\CMS\Extbase\Persistence\Generic\QuerySettingsInterface */
$defaultQuerySettings = $this->objectManager->get('TYPO3\\CMS\\Extbase\\Persistence\\Generic\\QuerySettingsInterface');
$defaultQuerySettings->setRespectStoragePage(FALSE);
$this->setDefaultQuerySettings($defaultQuerySettings);
}
}
The reference class can be empty, but the repository needs it to be there in order to work correctly. Make sure you add the default query settings to ignore the page id, so you get all non-hidden and non-deleted file references by calling $fileReferenceRepository->findAll().
How to check the MIME-type of each referenced file:
You can get the MIME-type of each reference by calling
$fileReference->getOriginalResource()->getMimeType()
This should automatically fetch the original file from storage and read its MIME-type.

zend acl dynamic assertions : when/how to load the resource?

i'm creating a zend framework 2 application and i'm sort of trying to implement what is explained here:
http://ralphschindler.com/2009/08/13/dynamic-assertions-for-zend_acl-in-zf
The demonstration that the code works is really nice, but it doesn't really apply to how a framework (utilizing mvc) works. Or maybe i'm just on the wrong track...
i've created a RouteListener like this :
class RouteListener implements ListenerAggregateInterface
{
public function attach(EventManagerInterface $events)
{
$this->listeners[] = $result = $events->attach(
MvcEvent::EVENT_DISPATCH, array($this, "checkAcl"), 100
);
}
}
the method checkAcl then checks if you're allowed to do what you want to do.
The resource and action are determined like this:
$resource = $routeMatch->getParam("controller");
$action = $routeMatch->getParam("action");
And the role is determined by the identity stored in the session (which implements Zend\Permissions\Acl\Role\RoleInterface)
Following the example: how do i determine if a user is allowed to edit a certain blog-post?
By the time acl is doing it's checking, the controller hasn't loaded the blogpost yet, so i'm not sure how to approach this. Unless i duplicate the retrieval of the blogpost in the assertion, but that i'm hoping there is a better way.
I'm also using doctrine for my persistence layer and in the end i've solved this problem using doctrine's Lifecycle Events. This allows you to trigger the acl-check whenever you want: when a entity (p.e. a blog-post) is loaded, or saved, etc.

RavenDB Index created incorrectly

I have a document in RavenDB that looks looks like:
{
"ItemId": 1,
"Title": "Villa
}
With the following metadata:
Raven-Clr-Type: MyNamespace.Item, MyNamespace
Raven-Entity-Name: Doelkaarten
So I serialized with a type MyNamespace.Item, but gave it my own Raven-Entity-Name, so it get its own collection.
In my code I define an index:
public class DoelkaartenIndex : AbstractIndexCreationTask<Item>
{
public DoelkaartenIndex()
{
// MetadataFor(doc)["Raven-Entity-Name"].ToString() == "Doelkaarten"
Map = items => from item in items
where MetadataFor(item)["Raven-Entity-Name"].ToString() == "Doelkaarten"
select new {Id = item.ItemId, Name = item.Title};
}
}
In the Index it is translated in the "Maps" field to:
docs.Items
.Where(item => item["#metadata"]["Raven-Entity-Name"].ToString() == "Doelkaarten")
.Select(item => new {Id = item.ItemId, Name = item.Title})
A query on the index never gives results.
If the Maps field is manually changed to the code below it works...
from doc in docs
where doc["#metadata"]["Raven-Entity-Name"] == "Doelkaarten"
select new { Id = doc.ItemId, Name=doc.Title };
How is it possible to define in code the index that gives the required result?
RavenDB used: RavenHQ, Build #961
UPDATE:
What I'm doing is the following: I want to use SharePoint as a CMS, and use RavenDB as a ready-only replication of the SharePoint list data. I created a tool to sync from SharePoint lists to RavenDB. I have a generic type Item that I create from a SharePoint list item and that I serialize into RavenDB. So all my docs are of type Item. But they come from different lists with different properties, so I want to be able to differentiate. You propose to differentiate on an additional property, this would perfectly work. But then I will see all list items from all lists in one big Items collection... What would you think to be the best approach to this problem? Or just live with it? I want to use the indexes to create projections from all data in an Item to the actual data that I need.
You can't easily change the name of a collection this way. The server-side will use the Raven-Entity-Name metadata, but the client side will determine the collection name via the conventions registered with the document store. The default convention being to use the type name of the entity.
You can provide your own custom convention by assigning a new function to DocumentStore.Conventions.FindTypeTagName - but it would probably be cumbersome to do that for every entity. You could create a custom attribute to apply to your entities and then write the function to look for and understand that attribute.
Really the simplest way is just to call your entity Doelkaarten instead of Item.
Regarding why the change in indexing works - it's not because of the switch in linq syntax. It's because you said from doc in docs instead of from doc in docs.Items. You probably could have done from doc in docs.Doelkaartens instead of using the where clause. They are equivalent. See this page in the docs for further examples.

RavenDB does not set all the properties on store

I have a really weird scenario where I try to store domain events (I'm trying to learn CQRS and RavenDB at the same time). The basic structure of the documents I try to store are:
public interface IDomainEvent { ... }
public abstract class BaseDomainEvent : IDomainEvent { ... }
public class DomainEventA : BaseDomainEvent { ... }
public class DomainEventB : BaseDomainEvent { ... }
Given that I want to store DomainEventA and DomainEventB in the same collection in RavenDB and I have managed to do so. But the problem is that in the collection I am missing the properties of DomainEventB, and not all properties are set even though I have checked that the properties are set before I commit the transaction where I store the objects. The following gist shows a working example of what I want to do: https://gist.github.com/2830093, and the test code that fails me is found in this test: https://github.com/mastoj/TJ.CQRS/blob/ravenfail/TJ.CQRS.RavenEvent.Tests/RavenEventStoreTests.cs that is using this RavenDB code: https://github.com/mastoj/TJ.CQRS/blob/ravenfail/TJ.CQRS.RavenEvent/RavenEventStore.cs.
I really can't get my head around this one.
EDIT 1: I can add that in the failing scenario the metadata of the stored object says it is one type but the properties for that type is not stored.
I planned to delete or vote for close but I think more than me might experience this problem at some point. I found the solution in my case and it was that the objects I added to RavenDB had a faulty equals method so RavenDB thought that all my objects were the same one. When I added one more property to check in the equals method everything start working as expected.