Tuning infinispan/hibernate search indexing - indexing

Using
Infinispan 6.0.0
Hibernate Search 4.4.0
But indexing seems extremely slow.
Here are the jmx stats from the DefaultCacheManager...
averageReadTime: 0
averageWriteTime: 9903
I tried max_merge_docs and merge_factor and ram_buffer as indicated in the docs here: http://docs.jboss.org/hibernate/search/4.4/reference/en-US/html/search-configuration.html#d0e2031
But that seems to do nothing. The only thing that sort of works better is the number of shards...
<namedCache name="myCache">
<clustering mode="distribution">
<async/>
<hash numOwners="1"/>
</clustering>
<indexing enabled="true" indexLocalOnly="true">
<properties>
<property name="default.directory_provider" value="ram" />
<property name="default.indexwriter.merge_factor" value="30000" />
<property name="default.indexwriter.ram_buffer_size" value="1024" />
<!--<property name="default.sharding_strategy.nbr_of_shards" value="4" /> -->
</properties>
</indexing>
</namedCache>
The values above are the last ones I tried. I tried different ones to no avail. Only nbr_of_shards seems to speed things up just a tad.
The model...
#Indexed
#ProvidedId
public class MyModel implements Serializable
{
#DocumentId
Integer id;
Integer prop1;
Integer prop2;
#Field(analyze = Analyze.NO)
String prop3;
#Field(analyze = Analyze.NO)
String prop4;
#Field(analyze = Analyze.NO)
String prop5;
#Field(analyze = Analyze.NO)
String prop6;
#Field(analyze = Analyze.NO)
String prop7;
#Field(analyze = Analyze.NO)
String prop9;
#Field(analyze = Analyze.NO)
Long prop10;
}
And in my servlet doPost()
MyModel model = new MyModel();
// Get POST params here and set the model properties...
cache.put(id, model);
out.println("Added!");

Related

Why we should write List and DetailedUser in the variable <variable type = "list<DetailedUser" name="">

I don't understand this line of the code why we have list and DetailedUser?
<data>
//app class
<import type = "com.androidistanbul.databindingdemo.layoutdetails.DetailedUser/>
// java class
<import type= "java.util.List"/>
<variable
name = "userList"
type = "detailedUser" /> // import class
<variable
name="userList"
type = "list<DetailedUser" />
</data>
// List index
<TextView
// android:layout_midth = "wrap_content"
// android:layout_height= "wrap_content"
// android:layout_marginTop = "8dp"
android:text="#{userList[index].name + "" + userList[index].surnane}"/>
< and > are html entities and they represent < and >, respectively.
So List<DetailedUser> translates to List<DetailedUser>

VTD-XML JAVA XMLModifier remove whitespace issue

I am trying to remove some nodes using XMLModifier using the following code. I am ending with white spaces in between. How can I get rid of this?
import java.nio.file.Files;
import java.nio.file.Paths;
import com.ximpleware.AutoPilot;
import com.ximpleware.VTDGen;
import com.ximpleware.VTDNav;
import com.ximpleware.XMLModifier;
public class VTDWhiteSpaceIssue {
public static void main(String[] args) throws Exception {
byte[] encoded = Files.readAllBytes(Paths.get("Sample.xml"));
String query = new String(encoded, "UTF-8");
VTDGen vtdGenDoc = new VTDGen();
vtdGenDoc.setDoc(query.getBytes());
vtdGenDoc.parse(false);
VTDNav vtdNav = vtdGenDoc.getNav();
AutoPilot autoPilot = new AutoPilot(vtdNav);
XMLModifier xmlModifier = new XMLModifier(vtdNav);
autoPilot.selectXPath("//product/catalog_item");
if (autoPilot.evalXPath() != -1 && vtdNav.toElement(VTDNav.FIRST_CHILD)) {
do {
String nodeName = vtdNav.toRawString(vtdNav.getCurrentIndex());
if (!"price".equals(nodeName) && !"item_number".equals(nodeName)) {
System.out.println("Removing node " + nodeName);
xmlModifier.remove();
}
} while (vtdNav.toElement(VTDNav.NEXT_SIBLING));
}
System.out.println();
System.out.println("==============================================================");
// normalizedQueryNav = normalizedQueryModifier.outputAndReparse();
xmlModifier.output(System.out);
System.out.println("==============================================================");
}
}
Sample.xml
<catalog>
<product description="Cardigan Sweater" product_image="cardigan.jpg">
<catalog_item gender="Men's">
<title>Cardigan Sweater</title>
<item_number>QWZ5671</item_number>
<size description="Medium">
<color_swatch image="red_cardigan.jpg">Red</color_swatch>
<color_swatch image="burgundy_cardigan.jpg">Burgundy</color_swatch>
</size>
<size description="Large">
<color_swatch image="red_cardigan.jpg">Red</color_swatch>
<color_swatch image="burgundy_cardigan.jpg">Burgundy</color_swatch>
</size>
<price>39.95</price>
</catalog_item>
</product>
I tried using the following code instead of xmlModifier.remove()
long elementFragment = vtdNav.getElementFragment();
xmlModifier.remove(vtdNav.expandWhiteSpaces(elementFragment));
It fails with the following exception:-
com.ximpleware.ModifyException: Invalid insertion/deletion condition detected between offset 189 and offset 373
at com.ximpleware.XMLModifier.check2(XMLModifier.java:888)
at com.ximpleware.XMLModifier.output(XMLModifier.java:1977)
at vtd.VTDWhiteSpaceIssue.main(VTDWhiteSpaceIssue.java:40)
Note:- Sample code executed with vtd-xml_2_13.jar
The exception is caused by fragments overlapping... obviously when you call expandWhiteSpaces on node named "size." The trailing white spaces of first Size will overlap with the leading white spaces of the second "size" element. The fix is to call
public final long expandWhiteSpaces(long l,
short actionType)
For action Type, use WS_LEADING. That should do it for you.

Deploy BrowserFormWebPart declaratively without BinarySerializedWebPart Element

Does anyone know if there is a way to deploy a BrowserFormWebPart (custom InfoPath form for a list content type) using standard AllUsersWebPart element and a CDATA section for the properties? So far I have tried without success. Any help is appreciated.
After 2 days of research - Following code works
private void UpdateInfoPathForms(SPSite oSite)
{
UpdateInfoPath(oSite, "Lists/Audit Calendar/Item/newifs.aspx");
UpdateInfoPath(oSite, "Lists/Audit Calendar/Item/displayifs.aspx");
UpdateInfoPath(oSite, "Lists/Audit Calendar/Item/editifs.aspx");
}
private void UpdateInfoPath(SPSite oSite, string formFileLocation)
{
var file = oSite.RootWeb.GetFile(formFileLocation);
using (SPLimitedWebPartManager manager = file.GetLimitedWebPartManager(System.Web.UI.WebControls.WebParts.PersonalizationScope.Shared))
{
try
{
var wp1 = new Microsoft.Office.InfoPath.Server.Controls.WebUI.BrowserFormWebPart();
wp1.SubmitBehavior = Microsoft.Office.InfoPath.Server.Controls.WebUI.SubmitBehavior.FormDefault;
wp1.FormLocation = "~list/Item/template.xsn";
wp1.ContentTypeId = oSite.RootWeb.Lists["Audit Calendar"].ContentTypes["Item"].Id.ToString();
IListWebPart listWebpart = wp1 as IListWebPart;
listWebpart.ListId = oSite.RootWeb.Lists["Audit Calendar"].ID;
if (formFileLocation.Contains("newifs.aspx"))
{
listWebpart.PageType = PAGETYPE.PAGE_NEWFORM;
}
else if (formFileLocation.Contains("displayifs.aspx"))
{
wp1.ListFormMode = Microsoft.Office.InfoPath.Server.Controls.WebUI.ListFormMode.ReadOnly;
listWebpart.PageType = PAGETYPE.PAGE_DISPLAYFORM;
}
else if (formFileLocation.Contains("editifs.aspx"))
{
listWebpart.PageType = PAGETYPE.PAGE_EDITFORM;
}
listWebpart.ViewFlags = SPViewFlags.None;
manager.AddWebPart(wp1, "Main", 0);
manager.SaveChanges(wp1);
}
finally
{
manager.Web.Dispose();
}
}
I have had the same problem. Here is what I tried:
<AllUsersWebPart WebPartZoneID="Main" WebPartOrder="2">
<![CDATA[
<webParts>
<webPart xmlns="http://schemas.microsoft.com/WebPart/v3">
<metaData>
<type name="Microsoft.Office.InfoPath.Server.Controls.WebUI.BrowserFormWebPart, Microsoft.Office.InfoPath.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
<importErrorMessage>Cannot import this Web Part.</importErrorMessage>
</metaData>
<data>
<properties>
<property name="ChromeType" type="chrometype">None</property>
<property name="HelpMode" type="helpmode">Modeless</property>
<property name="ChromeState" type="chromestate">Normal</property>
</properties>
</data>
</webPart>
</webParts>]]>
</AllUsersWebPart>
I then have a Feature Receiver that configures the web part:
using (SPLimitedWebPartManager manager = file.GetLimitedWebPartManager(PersonalizationScope.Shared))
{
try
{
BrowserFormWebPart webpart = GetWebPart(manager);
webpart.SubmitBehavior = SubmitBehavior.FormDefault;
webpart.FormLocation = "~list/MyList/template.xsn";
webpart.ContentTypeId = "0x01003C8AD6E14DAD5342BBFAA84E63F8022C";
manager.SaveChanges(webpart);
}
finally
{
manager.Web.Dispose();
}
}
The BrowserFormWebPart properties are required for getting the form to display, but for some reason, setting those properties in the AllUsersWebPart section did not work. The form displays and I can fill it out, but the values from the form do not get inserted into the fields of the list item. I added the following section to the Feature Receiver to try to get the form to tie into the fields of the list item:
IListWebPart listWebpart = webpart as IListWebPart;
listWebpart.PageType = PAGETYPE.PAGE_EDITFORM;
listWebpart.ViewFlags = SPViewFlags.None;
Unfortunately, no joy. And that is as far as I got. Hopefully you'll have better luck.

WCF DataContractSerializer Behavior

I'm seeing some unusual behavior when using the DataContractSerializer. I have defined a message contract like so:
namespace MyNamespace.DataContracts
{
[MessageContract(WrapperName = "order", WrapperNamespace = #"http://example.com/v1/order")]
public class MyOrder
{
[MessageBodyMember(Namespace = #"http://example.com/v1/order", Order = 1)]
public MyStore store;
[MessageBodyMember(Namespace = #"http://example.com/v1/order", Order = 2)]
public MyOrderHeader orderHeader;
[MessageBodyMember(Namespace = #"http://example.com/v1/order", Order = 3)]
public List<MyPayment> payments;
[MessageBodyMember(Namespace = #"http://example.com/v1/order", Order = 4)]
public List<MyShipment> shipments;
}
.
.
I'm sending it an XML message that looks like this:
<?xml version="1.0" encoding="utf-8"?>
<order xmlns="http://example.com/v1/order>
<store>
...
</store>
<orderHeader>
...
</orderHeader>
<payments>
<payment>
...
</payment>
</payments>
<shipments>
<shipment>
...
</shipment>
</shipments>
</order>
My service deserializes this XML as expected. Inside my service, I'm using the DataContractSerializer to create an XML string and that's where things get weird. I'm using the serializer like this:
DataContractSerializer serializer = new DataContractSerializer(typeof(MyOrder));
using (MemoryStream ms = new MemoryStream())
{
serializer.WriteObject(ms, order);
ms.Position = 0;
StreamReader sr = new StreamReader(ms);
string outputMessage = sr.ReadToEnd();
}
Once this finishes, the outputMessage contains the following XML:
<?xml version="1.0" encoding="utf-8"?>
<MyOrder xmlns="http://example.com/v1/order" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<order>
<store>
...
</store>
<orderHeader>
...
</orderHeader>
<payments>
<payment>
...
</payment>
</payments>
<shipments>
<shipment>
...
</shipment>
</shipments>
</order>
</MyOrder>
Needless to say, anything expecting to receive the original XML message will fail to parse this. So I guess I have two questions:
Why is the DataContractSerializer
adding the extra outer node to my
XML output?
Is there a way to stop it from doing
this?
Thanks.
I should probably add this is with .NET 4.
You could try using WriteObjectContent instead of WriteObject, but I'm unable to reproduce your problem using the code you supplied. All the extra class defintions that are part of your message contract are empty in my definition, but this is the XML I am getting:
<MyOrder xmlns="http://schemas.datacontract.org/2004/07/SandboxApp"
xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<orderHeader i:nil="true"/>
<payments i:nil="true"/>
<shipments i:nil="true"/>
<store i:nil="true"/>
</MyOrder>
Which also seems odd, since it seems to ignore the WrapperName. Same result in .NET 3.5 SP1 and .NET 4.0.

How do I view the SQL that is generated by nHibernate?

How do I view the SQL that is generated by nHibernate? version 1.2
You can put something like this in your app.config/web.config file :
in the configSections node :
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net"/>
in the configuration node :
<log4net>
<appender name="NHibernateFileLog" type="log4net.Appender.FileAppender">
<file value="logs/nhibernate.txt" />
<appendToFile value="false" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%d{HH:mm:ss.fff} [%t] %-5p %c - %m%n" />
</layout>
</appender>
<logger name="NHibernate.SQL" additivity="false">
<level value="DEBUG"/>
<appender-ref ref="NHibernateFileLog"/>
</logger>
</log4net>
And don't forget to call
log4net.Config.XmlConfigurator.Configure();
at the startup of your application, or to put
[assembly: log4net.Config.XmlConfigurator(Watch=true)]
in the assemblyinfo.cs
In the configuration settings, set the "show_sql" property to true.
I am a bit late I know, but this does the trick and it is tool/db/framework independent.
Instead of those valid options, I use NH Interceptors.
At first, implement a class which extends NHibernate.EmptyInterceptor and implements NHibernate.IInterceptor:
using NHibernate;
namespace WebApplication2.Infrastructure
{
public class SQLDebugOutput : EmptyInterceptor, IInterceptor
{
public override NHibernate.SqlCommand.SqlString
OnPrepareStatement(NHibernate.SqlCommand.SqlString sql)
{
System.Diagnostics.Debug.WriteLine("NH: " + sql);
return base.OnPrepareStatement(sql);
}
}
}
Then, just pass an instance when you open your session. Be sure to do it only when in DEBUG:
public static void OpenSession() {
#if DEBUG
HttpContext.Current.Items[SessionKey] = _sessionFactory.OpenSession(new SQLDebugOutput());
#else
HttpContext.Current.Items[SessionKey] = _sessionFactory.OpenSession();
#endif
}
And that's it.
From now on, your sql commands like these...
var totalPostsCount = Database.Session.Query<Post>().Count();
var currentPostPage = Database.Session.Query<Post>()
.OrderByDescending(c => c.CreatedAt)
.Skip((page - 1) * PostsPerPage)
.Take(PostsPerPage)
.ToList();
.. are shown straight in your Output window:
NH: select cast(count(*) as INT) as col_0_0_ from posts post0_
NH:select post0_.Id as Id3_, post0_.user_id as user2_3_, post0_.Title as
Title3_, post0_.Slug as Slug3_, post0_.Content as Content3_,
post0_.created_at as created6_3_, post0_.updated_at as updated7_3_,
post0_.deleted_at as deleted8_3_ from posts post0_ order by
post0_.created_at desc limit ? offset ?
In the configuration settings, set the "show_sql" property to true.
This will cause the SQL to be output in NHibernate's logfiles courtesy of log4net.
Use sql server profiler.
EDIT (1 year later): As #Toran Billups states below, the NHibernate profiler Ayende wrote is very very cool.
You can also try NHibernate Profiler (30 day trial if nothing else). This tool is the best around IMHO.
This will not only show the SQL generated but also warnings/suggestions/etc
There is a good reference for NHibernate logging at: How to configure Log4Net for use with NHibernate. It includes info on logging all NHibernate-generated SQL statements.
Nhibernate Profiler is an option, if you have to do anything serious.
If you're using SQL Server (not Express), you can try SQL Server Profiler.
Or, if you want to show the SQL of a specific query, use the following method (slightly altered version of what suggested here by Ricardo Peres) :
private String NHibernateSql(IQueryable queryable)
{
var prov = queryable.Provider as DefaultQueryProvider;
var session = prov.Session as ISession;
var sessionImpl = session.GetSessionImplementation();
var factory = sessionImpl.Factory;
var nhLinqExpression = new NhLinqExpression(queryable.Expression, factory);
var translatorFactory = new NHibernate.Hql.Ast.ANTLR.ASTQueryTranslatorFactory();
var translator = translatorFactory.CreateQueryTranslators(nhLinqExpression, null, false, sessionImpl.EnabledFilters, factory).First();
var sql = translator.SQLString;
var parameters = nhLinqExpression.ParameterValuesByName;
if ( (parameters?.Count ?? 0) > 0)
{
sql += "\r\n\r\n-- Parameters:\r\n";
foreach (var par in parameters)
{
sql += "-- " + par.Key.ToString() + " - " + par.Value.ToString() + "\r\n";
}
}
return sql;
}
and pass to it a NHibernate query, i.e.
var query = from a in session.Query<MyRecord>()
where a.Id == "123456"
orderby a.Name
select a;
var sql = NHibernateSql(query);
You are asking only for viewing; but this answer explains how to log it to file. Once logged, you can view it in any text editor.
Latest versions of NHibernate support enabling logging through code. Following is the sample code that demonstrates this. Please read the comments for better understanding.
Configuration configuration = new Configuration();
configuration.SetProperty(NHibernate.Cfg.Environment.Dialect, ......);
//Set other configuration.SetProperty as per need
configuration.SetProperty(NHibernate.Cfg.Environment.ShowSql, "true"); //Enable ShowSql
configuration.SetProperty(NHibernate.Cfg.Environment.FormatSql, "true"); //Enable FormatSql to make the log readable; optional.
configuration.AddMapping(......);
configuration.BuildMappings();
ISessionFactory sessionFactory = configuration.BuildSessionFactory();
//ISessionFactory is setup so far. Now, configure logging.
Hierarchy hierarchy = (Hierarchy)LogManager.GetRepository(Assembly.GetEntryAssembly());
hierarchy.Root.RemoveAllAppenders();
FileAppender fileAppender = new FileAppender();
fileAppender.Name = "NHFileAppender";
fileAppender.File = logFilePath;
fileAppender.AppendToFile = true;
fileAppender.LockingModel = new FileAppender.MinimalLock();
fileAppender.Layout = new PatternLayout("%d{yyyy-MM-dd HH:mm:ss}:%m%n%n");
fileAppender.ActivateOptions();
Logger logger = hierarchy.GetLogger("NHibernate.SQL") as Logger;
logger.Additivity = false;
logger.Level = Level.Debug;
logger.AddAppender(fileAppender);
hierarchy.Configured = true;
You can further play with FileAppender and Logger as per your need. Please refer to this answer and this resource for more details. This explains the same with XML configuration; but the same should equally apply to code.