Solr delta import does not work - sql

I am using solr 4.2. Note that full import works but somehow delta import doesn't. Delta import does not give any error but never fetches any changes. Here's the data config file.
<dataConfig>
<dataSource type="JdbcDataSource"
driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver:testsql;databaseName=test"
user="dba"
password="dba"/>
<script>
<![CDATA[
function metadataTransformer (row) {
var attributeName = row.get("attribute_name");
var attributeValue = row.get("attribute_value");
row.put(attributeName, attributeValue);
return row;
}
]]>
</script>
<document name="PRODUCTS">
<entity name="product" query="select distinct p.product_id as id from products p
inner join products_meta pm on p.product_id = pm.product_id
inner join meta m on pm.meta_id = m.meta_id
where m.meta_type_id = 11 order by id desc"
deltaImportQuery="select distinct p.product_id as id from products p
inner join products_meta pm on p.product_id = pm.product_id
inner join meta m on pm.meta_id = m.meta_id
where m.meta_type_id = 11 and p.product_id='${dih.delta.product_id}'"
deltaQuery= "select distinct product_id as id from products
where updtime > '${dih.last_index_time}'">
<field column="id" name="id"/>
<entity name="attribute" query="select attribute_name,attribute_value from solr_import
where id =${product.id}" transformer= "script:metadataTransformer">
</entity>
</entity>
</document>
</dataConfig>
Here's what I have tried without any luck.
changing p.product_id='${dih.delta.product_id} to p.product_id='${dih.delta.id} and other way also.
changing where updtime > '${dih.last_index_time}'" to where updtime > '${dih.last_index_time}'"
Please help.

This is fixed. The issue was solr box was on UTC timezone. Once I converted that to ET timezone, delta import started working fine.
Hope this helps someone else.

If you don't want to use ${dih.last_index_time} (server's time config problem), you can try mysql query like:
deltaImportQuery="SELECT * FROM table_name where id='${dataimporter.delta.id}'"
deltaQuery="select id from table_name where update_time > SUBTIME( NOW( ) , '0:15:0' )
I used with crontab and worked fine!

Related

How do I produce an XML response from PostgreSQL 1:n tables?

I have the table "client" with:
id name registered_on status
-- ------- ------------- ------
1 Alice 2020-03-04 a
2 Vincent 2020-03-05 p
3 Anne 2020-03-06 a
And the table "account" with:
client_id account_number type balance
--------- -------------- ---- -------
1 300-1 CHK 100
2 307-5 SAV 24
2 307-6 CHK 350
I created them in DB Fiddle (for a similar question I asked before about producing JSON).
Now, I need a SQL query to produce the 1:n XML document:
<client id="1" name="Alice" registered_on="2020-03-04" status="a">
<account account_number="300-1" type="CHK" balance="100" />
</client>
<client id="2" name="Vincent" registered_on="2020-03-05" status="p">
<account account_number="307-5" type="SAV" balance="24" />
<account account_number="307-6" type="CHK" balance="350" />
</client>
<client id="3" name="Anne" registered_on="2020-03-06" status="a" />
There's a 1:n relationship between the tables and some clients may not have an account (such as "Anne"). The result is a simple join (probably an outer join) that I know how to do. I just don't get how to produce a XML document from it.
If it's makes it easier/shorter I'm open to an alternative XML result, as long as it represents the same data; using tags, instead of attributes, for example.
After trying a bunch of options I was able to find the answer(s).
Original Format: With Attributes
It's possible to produce the XML result using an outer join:
select
xmlserialize(content -- remove this line to keep as XML instead of VARCHAR
xmlagg(r)
as text) -- remove this line to keep as XML instead of VARCHAR
from (
select
xmlelement(name client,
xmlattributes(c.id, c.name, c.registered_on, c.status),
case when count(a.client_id) > 0 then
xmlagg(xmlelement(name account,
xmlattributes(a.account_number, a.type, a.balance) ))
end
) as r
from client c
left join account a on a.client_id = c.id
group by c.id
) s
Or using subqueries (shorter but less performant):
select
xmlserialize(content -- remove this line to keep as XML instead of VARCHAR
xmlagg(
xmlelement(name client, xmlattributes(id, name, registered_on, status),
( select xmlagg(xmlelement(name account,
xmlattributes(a.account_number, a.type, a.balance)
)) from account a where a.client_id = c.id
)
))
as text) -- remove this line to keep as XML instead of VARCHAR
from client c;
Result:
<client id="1" name="Alice" registered_on="2020-03-04" status="a">
<account account_number="300-1" type="CHK" balance="100.00" />
</client>
<client id="2" name="Vincent" registered_on="2020-03-05" status="p">
<account account_number="307-5" type="SAV" balance="24.00" />
<account account_number="307-6" type="CHK" balance="350.00" />
</client>
<client id="3" name="Anne" registered_on="2020-03-06" status="a" />
Alternative Format: Without Attributes
Some people prefer to avoid attributes altogether and always use tags. That can also be done, using:
select
xmlserialize(content -- remove this line to keep as XML instead of VARCHAR
xmlagg(xmlelement(name client,
xmlforest(id, name, registered_on, status),
( select xmlagg(xmlelement(name account,
xmlforest(a.account_number, a.type, a.balance)))
from account a where a.client_id = c.id
)
))
as text) -- remove this line to keep as XML instead of VARCHAR
from client c;
Result:
<client>
<id>1</id>
<name>Alice</name>
<registered_on>2020-03-04</registered_on>
<status>a</status>
<account>
<account_number>300-1</account_number>
<type>CHK</type>
<balance>100.00</balance>
</account>
</client>
<client>
<id>2</id>
<name>Vincent</name>
<registered_on>2020-03-05</registered_on>
<status>p</status>
<account>
<account_number>307-5</account_number>
<type>SAV</type>
<balance>24.00</balance>
</account>
<account>
<account_number>307-6</account_number>
<type>CHK</type>
<balance>350.00</balance>
</account>
</client>
<client>
<id>3</id>
<name>Anne</name>
<registered_on>2020-03-06</registered_on>
<status>a</status>
</client>

How can I output XML with different queries

I am struggling at outputting XML the correct way. I'm trying to generate XML document with SQL Server query, which gives my the following:
CODE USED:
SELECT Plate, tbl_veiculos.ID, Brand, Model, Origin, Color
FROM tbl_veiculos, tbl_veiculo_spec
WHERE tbl_veiculos.ID = tbl_veiculo_spec.ID AND tbl_veiculos.ID = 1
FOR XML PATH ('Vehicle'), TYPE, ROOT('VehicleList')
RESULT:
<VehicleList>
<Vehicle>
<Plate>34-23-nd</Plate>
<ID>1</ID>
<Brand>Mercedes-Benz</Brand>
<Model>A140</Model>
<Origin>Germany</Origin>
<Color>Red</Color>
</Vehicle>
</VehicleList>
Which is to a certain way what I need. The problem comes when I try to merge it with other query output. I know this doesn't explain well so I'll show you an harcoded version of what I want.
<VehicleList>
<Vehicle>
<Plate>34-23-nd</Plate>
<ID>1</ID>
<Brand>Mercedes-Benz</Brand>
<Model>A140</Model>
<SellerInfo>
<Name>Someone Special</Name>
<Street>Oxfod Court 1231</Street>
</SellerInfo>
<Origin>Germany</Origin>
<Color>Red</Color>
</Vehicle>
</VehicleList>
<SellerInfo> comes from other table.
You can use subquery as below:
SELECT Plate, tbl_veiculos.ID, Brand, Model,
(select Name, Street from sellerInfo where id = t1.id --use id to join sellerInfo table
for xml Path(''), type) as SellerInfo,
Origin, Color,
FROM tbl_veiculos t1 inner join tbl_veiculo_spec ts
on tbl_veiculos.ID = tbl_veiculo_spec.ID AND tbl_veiculos.ID = 1
FOR XML PATH ('Vehicle'), TYPE, ROOT('VehicleList')

Get XML node values using SQL

I'm using SQL Server 2014 and I need help with constructing a query that will give me all of the values for a particular XML node. I have a table that has information stored in it in xml. The xml data is like below:
<category>
<text>Laser Printers</text>
<title id="1">
<text>HP OfficeJet Pro 8610</text>
</text>
<title id="2">
<text>HP OfficeJet Pro 8700</text>
</text>
<title id="3">
<text>Canon PIXMA MX 922</text>
</text>
</category>
I'm only able to get the value from the first node for each row in the table using this query.
SELECT it.contents.value('(category/title/text)[1]', 'NVARCHAR(255)') as Printer
FROM inventory i
INNER JOIN store_products p ON i.inventoryId = p.inventoryId
INNER JOIN items it ON p.itemId = it.itemId
So my result is:
Printer
HP OfficeJet Pro 8610
What I need is the following:
Printer
HP OfficeJet Pro 8610, HP OfficeJet Pro 8700, Canon PIXMA MX 922
Is this even possible?
You need to use CROSS APPLY and the .nodes() XQuery function - something like this:
SELECT
XC.value('(text)[1]', 'varchar(100)') AS Printer
FROM
inventory i
INNER JOIN
store_products p ON i.inventoryId = p.inventoryId
INNER JOIN
items it ON p.itemId = it.itemId
CROSS APPLY
it.contents.nodes('/category/title') AS XT(XC)

SQL to XML transfer of data

Ok have been trying to learn SQL to XML the last few days and this is what I have been able to teach my self thus far.
`SELECT distinct StudentItem.foldername AS "foldername", StudentItem.status, StudentItem.vhrid, StudentItem.firstname, StudentItem.middleinitial, StudentItem.lastname,
dbo.getEnumDescript(StudentType, 'StudentType') AS title,
StudentItem.email,
dbo.getEnumDescript(OfficeLocation, 'OfficeLocation') AS Office,
practices.id as 'StudentItem/practices/practice/id',
practices.name as 'StudentItem/practices/practice/name',
schoolItem.Name as 'StudentItem/bio/schools/schoolItem/schoolname',
schoolItem.schoolYear as 'lawyerItem/bio/schools/schoolItem/schoolyear'
FROM [dbo].[Student] as lawyerItem
LEFT JOIN [dbo].[StudentGroups] as aprac on StudentItem.vhrid = aprac.vhrid
INNER JOIN [dbo].[PracticeGroups] as practices on aprac.PracticeGroupID = practices.ID
LEFT JOIN [dbo].[StudentEducation] as schoolItem on StudentItem.vhrid = schoolItem.vhrid
where StudentItem.vhrid='50330'
FOR XML path, ROOT ('StudentItem'), ELEMENTS;`
What I get is this
`<StudentItems>
<row>
<foldername>susan.wissink</foldername>
<status>1</status>
<vhrid>50330</vhrid>
<firstname>Susan</firstname>
<middleinitial>M.</middleinitial>
<lastname>Wissink</lastname>
<title>Student leader</title>
<email>swissink#blank.com</email>
<Office>Phoenix</Office>
<StudentItem>
<practices>
<practice>
<id>681</id>
<name>Real Estate Finance and Lending</name>
</practice>
</practices>
<bio>
<schools>
<schoolItem>
<schoolname><i>Best in America®</i>, ASU</schoolname>
<schoolyear>2016</schoolyear>
</schoolItem>
</schools>
</bio>
</StudentItem>
</row>
<row>
<foldername>susan.wissink</foldername>
<status>1</status>
<vhrid>50330</vhrid>
<firstname>Susan</firstname>
<middleinitial>M.</middleinitial>
<lastname>Wissink</lastname>
<title>Student leader</title>
<email>swissink#blank.com</email>
<Office>Phoenix</Office>
<StudentItem>
<practices>
<practice>
<id>681</id>
<name>Real Estate Finance and Lending</name>
</practice>
</practices>
<bio>
<schools>
<schoolItem>
<schoolname><i>Best in America®</i>, UOP</schoolname>
<schoolyear>2011-2015</schoolyear>
</schoolItem>
</schools>
</bio>
</StudentItem>
</row>`
But I'm trying to get the all the practices and schools to show up as one entry for the guy that. More or less I'm trying to get it to look like below.
`<StudentItems>
<row>
<foldername>susan.wissink</foldername>
<status>1</status>
<vhrid>50330</vhrid>
<firstname>Susan</firstname>
<middleinitial>M.</middleinitial>
<lastname>Wissink</lastname>
<title>Student leader</title>
<email>swissink#blank.com</email>
<Office>Phoenix</Office>
<StudentItem>
<practices>
<practice>
<id>681</id>
<name>Real Estate Finance and Lending</name>
<id>683</id>
<name>Business and Finance</name>
</practice>
</practices>
<bio>
<schools>
<schoolItem>
<schoolname><i>Best in America®</i>, UOP</schoolname>
<schoolyear>2011-2015</schoolyear>
<schoolname><i>Best in America®</i>, ASU</schoolname>
<schoolyear>2016</schoolyear>
</schoolItem>
</schools>
</bio>
</StudentItem>
</row>`
Any help would be welcome. Thank You.
Without sample data, it's difficult to write code and test for. But generally what you need to do is to create sub-queries to create your practice and schoolItem XML nodes. Something like this:
SELECT distinct StudentItem.foldername AS "foldername",
StudentItem.status,
StudentItem.vhrid,
StudentItem.firstname,
StudentItem.middleinitial,
StudentItem.lastname,
dbo.getEnumDescript(StudentType, 'StudentType') AS title,
StudentItem.email,
dbo.getEnumDescript(OfficeLocation, 'OfficeLocation') AS Office,
(
select practices.id, practices.name
from [dbo].[StudentGroups] as aprac
INNER JOIN [dbo].[PracticeGroups] as practices
on aprac.PracticeGroupID = practices.ID
where StudentItem.vhrid = aprac.vhrid
FOR XML path(''), type
) 'StudentItem/practices/practice',
(
select Name schoolname, schoolYear
from [dbo].[StudentEducation] schoolItem
where StudentItem.vhrid = schoolItem.vhrid
FOR XML path(''), type
) 'StudentItem/bio/schools/schoolItem'
FROM [dbo].[Student] as StudentItem
where StudentItem.vhrid='50330'
FOR XML path, ROOT ('StudentItem');

Get value from specific xml name value pair using sql

I am writing a query in SQL to combine a few tables of data and have got to one where it is stored in XML in the following format:
<CustomDetails xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Fields>
<Field>
<Name>Selected City</Name>
<Value>Central</Value>
</Field>
<Field>
<Name>Address Provided</Name>
<Value>New Address</Value>
</Field>
</Fields>
</CustomDetails>
The XML is stored in a table and I have managed to get the Address Provided field using the following code
select
o.OrderID,
od.CustomDetails.query('data(/CustomDetails/Fields/Field[2]/Value)') as 'Address Provided'
from
dbo.[Order] o on
o.OrderID = s.OrderID
join
dbo.OrderData od on
od.OrderID = o.OrderID
I would like to know if there is a better way of doing this and also if there is a way to guarantee that I will get the Address Provided field even if it appears first or if there are other fields in front of it.
Hopefully that all makes sense, Any help is appreciated.
Thanks
After some research I managed to get a solution.
select
o.OrderID,
cast(od.CustomDetails.query('
for $CD in /CustomDetails/Fields/Field,
$Name in $CD/Name
where contains($Name, "Address Provided")
return data($CD/Value)
') as varchar(50)) as addressProvided
from
dbo.[Order] o on
o.OrderID = s.OrderID
join
dbo.OrderData od on
od.OrderID = o.OrderID