Related
We are having issues indexing metadata with items added to Google Cloud Search with 3rd-party datasources, using the REST API. We are following this guide: https://developers.google.com/cloud-search/docs/guides/schema-guide.
What we tried: uploaded the schema successfully ✅, indexed some items ✅, made them available in the search UI ✅, but it seems the properties and facets not showing up whether it’s in the green metadata bits in search results or in API results -- we get an empty structuredData object. Search results seem to only query the content.inlineContent and nothing else. We’re not getting any errors from the API when we make the index request (using https://developers.google.com/cloud-search/docs/reference/rest/v1/indexing.datasources.items/index_). It seems like it’s just ignoring everything in structuredData.
Other attempts: In addition to the REST API, we’ve also used the official nodejs SDK, (which has additional validation because it’s in TypeScript), to no avail. We’ve verified that auth params are correct (it returns 401 otherwise) and that there are no other validation issues (it returns 400 otherwise). Requests return 200 and do upload successfully, just not the structuredData. The original schema upload was also successful (200), and the response included the full schema we gave.
What are we missing?
schema JSON:
{
"objectDefinitions": [{
"name": "page",
"options": {
"displayOptions": {
"objectDisplayLabel": "Page",
"metalines": [
{
"properties": [
{
"propertyName": "title"
},
{
"propertyName": "content"
},
{
"propertyName": "author"
},
{
"propertyName": "siteSection"
}
]
}
]
}
},
"propertyDefinitions": [
{
"name": "title",
"isReturnable": true,
"isWildcardSearchable": true,
"isSuggestable": true,
"isRepeatable": false,
"isFacetable": false,
"textPropertyOptions": {
"operatorOptions": {
"operatorName": "title"
}
},
"displayOptions": {
"displayLabel": "Title"
}
},
{
"name": "content",
"isReturnable": true,
"isRepeatable": false,
"isFacetable": false,
"htmlPropertyOptions": {
"retrievalImportance": {
"importance": "DEFAULT"
}
},
"displayOptions": {
"displayLabel": "Content"
}
},
{
"name": "author",
"isReturnable": true,
"isRepeatable": true,
"isFacetable": true,
"textPropertyOptions": {
"operatorOptions": {
"operatorName": "author"
}
},
"displayOptions": {
"displayLabel": "Author(s)"
}
},
{
"name": "siteSection",
"isReturnable": true,
"isWildcardSearchable": false,
"isSuggestable": false,
"isRepeatable": false,
"isFacetable": true,
"textPropertyOptions": {
"operatorOptions": {
"operatorName": "sitesection"
}
},
"displayOptions": {
"displayLabel": "Site Section"
}
}
]
}]
}
indexing code:
const version = '4';
const apiUrl = `https://cloudsearch.googleapis.com/v1/indexing/datasources/${sourceId}/items/exampleItem:index`;
const title = "Example Item";
const url = "https://example.com";
fetch(apiUrl, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: "Bearer " + token,
},
body: JSON.stringify({
item: {
name: `datasource/${sourceId}/items/exampleItem`,
acl: {
readers: [
{
gsuitePrincipal: {
gsuiteDomain: true,
},
},
],
},
metadata: {
title: title,
sourceRepositoryUrl: url,
objectType: "page",
},
structuredData: {
object: {
properties: [
{
name: "title",
textValues: {
values: [title],
},
},
{
name: "author",
textValues: {
values: ["Unknown Author"],
},
},
{
name: "siteSection",
textValues: {
values: ["exampleSection"],
},
},
{
name: "content",
htmlValues: {
values: [exampleContentHTML],
},
},
],
},
},
content: {
inlineContent: b64(exampleContentHTML),
contentFormat: "TEXT",
},
version: b64(version),
itemType: "CONTENT_ITEM",
},
mode: "SYNCHRONOUS",
}),
})
indexing result:
{
"name": "datasources/DATASOURCE_ID/items/exampleItem",
"acl": {
"readers": [{
"gsuitePrincipal": {
"gsuiteDomain": true
}
}]
},
"metadata": {
"title": "Example Item",
"sourceRepositoryUrl": "https://example.com",
"objectType": "page"
},
"structuredData": {
"object": {}
},
"content": {
"inlineContent": "... base 64 encoded content...",
"contentFormat": "TEXT"
},
"version": "NQ==",
"status": {
"code": "ACCEPTED"
},
"queue": "default",
"itemType": "CONTENT_ITEM"
}
I am currently building a web application where you can create setlists (arrays) with an array of lyric objectId's inside, that you can then sort / order into how you want it. So if you would like the 3rd list item to become the first, then you simply drag and drop it to the first line.
I now have a problem in my mongoose schema. I am looking for a way to implement an order attribute or something that would allow me to add a order value such as 0 or 1 depending on the position of the lyrics. Does any of you know how to best implement such order?
Here is a copy of my schema. Currently lyrics is an array of lyric objectId's. But in there i would need an "Order" as well, so that i can sort the array according to the order value.
const mongoose = require("mongoose");
const SetlistSchema = new mongoose.Schema({
setlistName: { type: String, required: true },
lastEdited: { type: Date },
createdAt: { type: Date, default: Date.now },
lyrics: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'Lyric'
}],
author: {
id: {
type: mongoose.Schema.Types.ObjectId,
ref: "User"
},
username: String
}
});
module.exports = mongoose.model("Setlist", SetlistSchema);
Here is the Lyrics schema.
const mongoose = require("mongoose");
const LyricSchema = new mongoose.Schema({
lyricName: { type: String, required: true },
lyricContent: { type: String, required: true },
lastEdited: { type: Date },
createdAt: { type: Date, default: Date.now },
author: {
id: {
type: mongoose.Schema.Types.ObjectId,
ref: "User"
},
username: String
}
});
module.exports = mongoose.model("Lyric", LyricSchema);
If adding an order number isn't the best practice, what can you then recommend as a way of keeping track of which order the user would like the lyrics to show up?
You can use aggregation framework to sort lyrics by order field. You first need to add a sort field with Number type.
Setlist.aggregate([
{
$unwind: "$lyrics"
},
{
$lookup: {
from: "lyrics", // MUST be the PHYSICAL collection name
localField: "lyrics",
foreignField: "_id",
as: "lyrics"
}
},
{
$sort: {
"lyrics.order": 1
}
},
{
"$group": {
"_id": "$_id",
"lyrics": {
"$push": "$lyrics"
},
"allFields": {
"$first": "$$ROOT"
}
}
},
{
"$replaceRoot": {
"newRoot": {
"$mergeObjects": [
"$allFields",
{
"lyrics": "$lyrics"
}
]
}
}
}
])
Playground
Sample documents:
db={
"lists": [
{
"_id": ObjectId("5a934e000102030405000000"),
"setlistName": "list1",
"lastEdited": ISODate("2020-03-18T23:11:56.443+03:00"),
"createdAt": ISODate("2020-03-15T23:11:56.443+03:00"),
"lyrics": [
ObjectId("6a934e000102030405000000"),
ObjectId("6a934e000102030405000001"),
ObjectId("6a934e000102030405000002")
]
},
{
"_id": ObjectId("5a934e000102030405000001"),
"setlistName": "list2",
"lastEdited": ISODate("2020-03-11T23:11:56.443+03:00"),
"createdAt": ISODate("2020-03-11T23:11:56.443+03:00"),
"lyrics": [
ObjectId("6a934e000102030405000003"),
ObjectId("6a934e000102030405000004")
]
}
],
"lyrics": [
{
"_id": ObjectId("6a934e000102030405000000"),
"name": "Lyric 1",
"order": 3
},
{
"_id": ObjectId("6a934e000102030405000001"),
"name": "Lyric 2",
"order": 1
},
{
"_id": ObjectId("6a934e000102030405000002"),
"name": "Lyric 3",
"order": 2
},
{
"_id": ObjectId("6a934e000102030405000003"),
"name": "Lyric 4",
"order": 2
},
{
"_id": ObjectId("6a934e000102030405000004"),
"name": "Lyric 5",
"order": 1
}
]
}
Output: (as you see lyrics are sorted by order field value)
[
{
"_id": ObjectId("5a934e000102030405000000"),
"createdAt": ISODate("2020-03-15T20:11:56.443Z"),
"lastEdited": ISODate("2020-03-18T20:11:56.443Z"),
"lyrics": [
[
{
"_id": ObjectId("6a934e000102030405000001"),
"name": "Lyric 2",
"order": 1
}
],
[
{
"_id": ObjectId("6a934e000102030405000002"),
"name": "Lyric 3",
"order": 2
}
],
[
{
"_id": ObjectId("6a934e000102030405000000"),
"name": "Lyric 1",
"order": 3
}
]
],
"setlistName": "list1"
},
{
"_id": ObjectId("5a934e000102030405000001"),
"createdAt": ISODate("2020-03-11T20:11:56.443Z"),
"lastEdited": ISODate("2020-03-11T20:11:56.443Z"),
"lyrics": [
[
{
"_id": ObjectId("6a934e000102030405000004"),
"name": "Lyric 5",
"order": 1
}
],
[
{
"_id": ObjectId("6a934e000102030405000003"),
"name": "Lyric 4",
"order": 2
}
]
],
"setlistName": "list2"
}
]
I'm just reading through the more advanced validators in the Schema definition of the Swagger specification:
{
"Schema":{
"type":"object",
"properties":{
"title":{
"type":"string"
},
"multipleOf":{
"type":"number",
"minimum":0,
"exclusiveMinimum":true
},
"maximum":{
"type":"number"
},
"exclusiveMaximum":{
"type":"boolean",
"default":false
},
"minimum":{
"type":"number"
},
"exclusiveMinimum":{
"type":"boolean",
"default":false
},
"maxLength":{
"type":"integer",
"minimum":0
},
"minLength":{
"type":"integer",
"minimum":0,
"default":0
},
"pattern":{
"type":"string",
"format":"regex"
},
"maxItems":{
"type":"integer",
"minimum":0
},
"minItems":{
"type":"integer",
"minimum":0,
"default":0
},
"uniqueItems":{
"type":"boolean",
"default":false
},
"maxProperties":{
"type":"integer",
"minimum":0
},
"minProperties":{
"type":"integer",
"minimum":0,
"default":0
},
"required":{
"type":"array",
"items":{
"type":"string"
},
"minItems":1,
"uniqueItems":true
},
"enum":{
"type":"array",
"items":{
},
"minItems":1,
"uniqueItems":true
},
"type":{
"type":"string",
"enum":[
"array",
"boolean",
"integer",
"number",
"object",
"string"
]
},
"not":{
"oneOf":[
{
"$ref":"#/definitions/Schema"
},
{
"$ref":"#/definitions/Reference"
}
]
},
"allOf":{
"type":"array",
"items":{
"oneOf":[
{
"$ref":"#/definitions/Schema"
},
{
"$ref":"#/definitions/Reference"
}
]
}
},
"oneOf":{
"type":"array",
"items":{
"oneOf":[
{
"$ref":"#/definitions/Schema"
},
{
"$ref":"#/definitions/Reference"
}
]
}
},
"anyOf":{
"type":"array",
"items":{
"oneOf":[
{
"$ref":"#/definitions/Schema"
},
{
"$ref":"#/definitions/Reference"
}
]
}
},
"items":{
"oneOf":[
{
"$ref":"#/definitions/Schema"
},
{
"$ref":"#/definitions/Reference"
}
]
},
"properties":{
"type":"object",
"additionalProperties":{
"oneOf":[
{
"$ref":"#/definitions/Schema"
},
{
"$ref":"#/definitions/Reference"
}
]
}
},
"additionalProperties":{
"oneOf":[
{
"$ref":"#/definitions/Schema"
},
{
"$ref":"#/definitions/Reference"
},
{
"type":"boolean"
}
],
"default":true
},
"description":{
"type":"string"
},
"format":{
"type":"string"
},
"default":{
},
"nullable":{
"type":"boolean",
"default":false
},
"discriminator":{
"$ref":"#/definitions/Discriminator"
},
"readOnly":{
"type":"boolean",
"default":false
},
"writeOnly":{
"type":"boolean",
"default":false
},
"example":{
},
"externalDocs":{
"$ref":"#/definitions/ExternalDocumentation"
},
"deprecated":{
"type":"boolean",
"default":false
},
"xml":{
"$ref":"#/definitions/XML"
}
},
"patternProperties":{
"^x-":{
}
},
"additionalProperties":false
}
}
The thing that I am thinking about is combinations of the anyOf, allOf, oneOf and not keywords. I have two questions.
The first question is: "can they be used in conjunction"? Like so:
{
"allOf" : [
{
"minItems" : 0
},
{
"maxItems" : 10
}
],
"anyOf" : [
{
"type" : "array",
"items" : {
"type" : "string"
}
},
{
"type" : "array",
"items" : {
"type" : "integer"
}
}
]
}
This example is, of course, needlessly complicated. But is it valid? Or can you only use one modifier but not the others?
The second question is, can anybody point me to a real-world example where one of these operators have been used in conjunction?
The thing that I am thinking about is combinations of the anyOf, allOf, oneOf and not keywords. I have two questions.
The first question is: "can they be used in conjunction"?
Yes, allOf, anyOf, oneOf and not can be used in conjunction. OpenAPI Specification follows the rules of JSON Schema here, and in JSON Schema adjacent keywords work as branches of an implicit allOf (source). So your example is equivalent to:
{
"allOf": [
{
"allOf": [
{
"minItems": 0
},
{
"maxItems": 10
}
]
},
{
"anyOf": [
{
"type": "array",
"items": {
"type": "string"
}
},
{
"type": "array",
"items": {
"type": "integer"
}
}
]
}
]
}
That said, this example is too complex and can be simplified into:
{
"minItems": 0,
"maxItems": 10,
"type": "array",
"items": {
"oneOf": [
{
"type": "string"
},
{
"type": "integer"
}
]
}
}
I got in trouble with very simple code:
property var pagesAllModels: {
ru: [
{ title: qsTr("New"), url: "http://bash.im" },
{ title: qsTr("Random"), url: "http://bash.im/random" },
{ title: qsTr("Best"), url: "http://bash.im/best" },
{ title: qsTr("By rating"), url: "http://bash.im/byrating" },
{ title: qsTr("Abyss"), url: "http://bash.im/abyss" },
{ title: qsTr("Abyss top"), url: "http://bash.im/abysstop" },
{ title: qsTr("Abyss best"), url: "http://bash.im/abyssbest" },
],
eng: [
{ title: "Latest", url: "http://bash.org/?latest" },
{ title: "Browse", url: "http://bash.org/?browse" },
{ title: "Random", url: "http://bash.org/?random" },
{ title: "Top", url: "http://bash.org/?top" }
]
}
That code in QML gives me error at line "eng: [" with error "expected lexem ," but in pure javascript everything work fine. What's wrong?
That's because your code is illegal JSON array definition. Web browsers are accepting it because they aren't strict about JS syntax, but QML engine is really strict :
In a key:value pair in associative array, the key must be a string, so it must have quotes around it, else it would be confused with a (non-existing) variable name.
property var pagesAllModels: {
"ru": [
{ "title": qsTr("New"), "url": "http://bash.im" },
{ "title": qsTr("Random"), "url": "http://bash.im/random" },
{ "title": qsTr("Best"), "url": "http://bash.im/best" },
{ "title": qsTr("By rating"), "url": "http://bash.im/byrating" },
{ "title": qsTr("Abyss"), "url": "http://bash.im/abyss" },
{ "title": qsTr("Abyss top"), "url": "http://bash.im/abysstop" },
{ "title": qsTr("Abyss best"), "url": "http://bash.im/abyssbest" }
],
"eng": [
{ "title": "Latest", "url": "http://bash.org/?latest" },
{ "title": "Browse", "url": "http://bash.org/?browse" },
{ "title": "Random", "url": "http://bash.org/?random" },
{ "title": "Top", "url": "http://bash.org/?top" }
]
}
And it works !
Try this
property var pagesAllModels: {
ru: [
{ title: qsTr("New"), url: "http://bash.im" },
{ title: qsTr("Random"), url: "http://bash.im/random" },
{ title: qsTr("Best"), url: "http://bash.im/best" },
{ title: qsTr("By rating"), url: "http://bash.im/byrating" },
{ title: qsTr("Abyss"), url: "http://bash.im/abyss" },
{ title: qsTr("Abyss top"), url: "http://bash.im/abysstop" },
{ title: qsTr("Abyss best"), url: "http://bash.im/abyssbest" }
],
eng: [
{ title: "Latest", url: "http://bash.org/?latest" },
{ title: "Browse", url: "http://bash.org/?browse" },
{ title: "Random", url: "http://bash.org/?random" },
{ title: "Top", url: "http://bash.org/?top" }
]
}
Hope this will resolve your issue
associative arrays separates by semicolon ";"
Try this
property var pagesAllModels: {
ru: [
{ title: qsTr("New"), url: "http://bash.im" },
{ title: qsTr("Random"), url: "http://bash.im/random" },
{ title: qsTr("Best"), url: "http://bash.im/best" },
{ title: qsTr("By rating"), url: "http://bash.im/byrating" },
{ title: qsTr("Abyss"), url: "http://bash.im/abyss" },
{ title: qsTr("Abyss top"), url: "http://bash.im/abysstop" },
{ title: qsTr("Abyss best"), url: "http://bash.im/abyssbest" },
];
eng: [
{ title: "Latest", url: "http://bash.org/?latest" },
{ title: "Browse", url: "http://bash.org/?browse" },
{ title: "Random", url: "http://bash.org/?random" },
{ title: "Top", url: "http://bash.org/?top" }
]
}
I am fetching nested data to be shown as nested list but whenever I tap on top level item, it again shows same top level list instead of showing children list and a ajax request is fired to fetch json data again. Here is the store:
Ext.define('MyTabApp.store.CategoriesStore',{
extend:'Ext.data.TreeStore',
config:{
model : 'MyTabApp.model.Category',
autoLoad: false,
storeId : 'categoriesStore',
proxy: {
type: 'ajax',
url: 'resources/data/catTree.json',
reader: {
type: 'json',
rootProperty: 'data.categories'
}
},
listeners:{
load: function( me, records, successful, operation, eOpts ){
console.log("categories tree loaded");
console.log(records);
}
}
}
});
and here is the data in that file which I am using to mock service:
{
"data":{
"categories": [
{
"name": "Men",
"categories": [
{
"name": "Footwear",
"categories": [
{ "name": "Casual Shoes", "leaf": true },
{ "name": "Sports Shoes", "leaf": true }
]
},
{
"name": "Clothing",
"categories": [
{ "name": "Casual Shirts", "leaf": true },
{ "name": "Ethnic", "leaf": true }
]
},
{ "name": "Accessories", "leaf": true }
]
},
{
"name": "Women",
"categories": [
{ "name": "Footwear", "leaf": true },
{ "name": "Clothing", "leaf": true },
{ "name": "Accessories", "leaf": true }
]
},
{
"name": "Kids",
"categories": [
{
"name": "Footwear",
"categories": [
{ "name": "Casual Shoes", "leaf": true },
{ "name": "Sports Shoes", "leaf": true }
]
},
{ "name": "Clothing", "leaf": true }
]
}
]
}
}
This is the list:
Ext.define('MyTabApp.view.CategoriesList', {
extend: 'Ext.dataview.NestedList',
alias : 'widget.categorieslist',
config: {
height : '100%',
title : 'Categories',
displayField : 'name',
useTitleAsBackText : true,
style : 'background-color:#999 !important; font-size:75%',
styleHtmlContent : true,
listConfig: {
itemHeight: 47,
itemTpl : '<div class="nestedlist-item"><div>{name}</div></div>',
height : "100%"
}
},
initialize : function() {
this.callParent();
var me = this;
var catStore = Ext.create('MyTabApp.store.CategoriesStore');
catStore.load();
me.setStore(catStore);
}
});
The list starts working properly without any ajax request on each tap if I remove data wrapper over top categories array and change rootProperty to categories instead of data.categories. Since server is actually returning categories in data object I cannot remove it so how do I fix the store in that case? Also why is that additional ajax request to fetch the file?
[EDIT]
Tried to create a fiddle http://www.senchafiddle.com/#d16kl which is similar but not same because it is using 2.0.1 and data is not loaded from external file or server.
Last time I had this exact situation, it was because one of my top level category was a leaf but I had not set leaf:true. Doing so recalled the top level of the nested list as if it was a child.
It seems from your Fiddle that if your data is in this following format, it would work fine:
{
"categories" : [{
"name" : "Foo",
"categories" : [{
...
}]
}]
}
That is, just remove the "data" property and make defaultRootProperty: 'categories' & rootProperty: 'categories'. Check this: http://www.senchafiddle.com/#d16kl#tIhTp
It works with external data file as well.