I am using Lucene 8.10.1.
I have the following code :
IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get(index)));
try
{
Fields fields = MultiFields.getFields(reader);
for (String termfield : fields) {
Terms terms = fields.terms(termfield);
TermsEnum termsEnum = terms.iterator(null);
int count = 0;
while (termsEnum.next() != null) {
count++;
}
System.out.println(count);
}
}
But I am running into this error.
java: cannot find symbol
symbol: method getFields(org.apache.lucene.index.IndexReader)
location: class org.apache.lucene.index.MultiFields
Couldn't find anything in the docs of 8.10.1.
Please help! Thank you!
Related
I am using the highlighting feature of Lucene to isolate matching terms for my query, but some of the matched terms are excessive.
I have some simple test cases which are delivered in an Ant project (download details below).
Materials
You can download the test case here: mydemo_with_libs.zip
That archive includes the Lucene 8.6.3 libraries which my test uses; if you prefer a copy without the JAR files you can download that from here: mydemo_without_libs.zip
The necessary libraries are: core, analyzers, queries, queryparser, highlighter, and memory.
You can run the test case by unzipping the archive into an empty directory and running the Ant command ant synsearch
Input
I have provided a short synonym list which is used for indexing and analysing in the highlighting methods:
cope,manage
jobs,tasks
simultaneously,at once
and there is one document being indexed:
Queues are a useful way of grouping jobs together in order to manage a number of them at once. You can:
hold or release multiple jobs at the same time;
group multiple tasks (for the same event);
control the priority of jobs in the queue;
Eventually log all events that take place in a queue.
Use either job.queue or task.queue in specifications.
Process
When building the index I am storing the text field, and using a custom analyzer. This is because (in the real world) the content I am indexing is technical documentation, so stripping out punctuation is inappropriate because so much of it may be significant in technical expressions. My analyzer uses a TechTokenFilter which breaks the stream up into tokens consisting of strings of words or digits, or individual characters which don't match the previous pattern.
Here's the relevant code for the analyzer:
public class MyAnalyzer extends Analyzer {
public MyAnalyzer(String synlist) {
if (synlist != "") {
this.synlist = synlist;
this.useSynonyms = true;
}
}
public MyAnalyzer() {
this.useSynonyms = false;
}
#Override
protected TokenStreamComponents createComponents(String fieldName) {
WhitespaceTokenizer src = new WhitespaceTokenizer();
TokenStream result = new TechTokenFilter(new LowerCaseFilter(src));
if (useSynonyms) {
result = new SynonymGraphFilter(result, getSynonyms(synlist), Boolean.TRUE);
result = new FlattenGraphFilter(result);
}
return new TokenStreamComponents(src, result);
}
and here's my filter:
public class TechTokenFilter extends TokenFilter {
private final CharTermAttribute termAttr;
private final PositionIncrementAttribute posIncAttr;
private final ArrayList<String> termStack;
private AttributeSource.State current;
private final TypeAttribute typeAttr;
public TechTokenFilter(TokenStream tokenStream) {
super(tokenStream);
termStack = new ArrayList<>();
termAttr = addAttribute(CharTermAttribute.class);
posIncAttr = addAttribute(PositionIncrementAttribute.class);
typeAttr = addAttribute(TypeAttribute.class);
}
#Override
public boolean incrementToken() throws IOException {
if (this.termStack.isEmpty() && input.incrementToken()) {
final String currentTerm = termAttr.toString();
final int bufferLen = termAttr.length();
if (bufferLen > 0) {
if (termStack.isEmpty()) {
termStack.addAll(Arrays.asList(techTokens(currentTerm)));
current = captureState();
}
}
}
if (!this.termStack.isEmpty()) {
String part = termStack.remove(0);
restoreState(current);
termAttr.setEmpty().append(part);
posIncAttr.setPositionIncrement(1);
return true;
} else {
return false;
}
}
public static String[] techTokens(String t) {
List<String> tokenlist = new ArrayList<String>();
String[] tokens;
StringBuilder next = new StringBuilder();
String token;
char minus = '-';
char underscore = '_';
char c, prec, subc;
// Boolean inWord = false;
for (int i = 0; i < t.length(); i++) {
prec = i > 0 ? t.charAt(i - 1) : 0;
c = t.charAt(i);
subc = i < (t.length() - 1) ? t.charAt(i + 1) : 0;
if (Character.isLetterOrDigit(c) || c == underscore) {
next.append(c);
// inWord = true;
}
else if (c == minus && Character.isLetterOrDigit(prec) && Character.isLetterOrDigit(subc)) {
next.append(c);
} else {
if (next.length() > 0) {
token = next.toString();
tokenlist.add(token);
next.setLength(0);
}
if (Character.isWhitespace(c)) {
// shouldn't be possible because the input stream has been tokenized on
// whitespace
} else {
tokenlist.add(String.valueOf(c));
}
// inWord = false;
}
}
if (next.length() > 0) {
token = next.toString();
tokenlist.add(token);
// next.setLength(0);
}
tokens = tokenlist.toArray(new String[0]);
return tokens;
}
}
Examining the index I can see that the index contains the separate terms I expect, including the synonym values. For example the text at the end of the first line has produced the terms
of
them
at , simultaneously
once
.
You
can
:
and the text at the end of the third line has produced the terms
same
event
)
;
When the application performs a search it analyzes the query without using the synonym list (because the synonyms are already in the index), but I have discovered that I need to include the synonym list when analyzing the stored text to identify the matching fragments.
Searches match the correct documents, but the code I have added to identify the matching terms over-performs. I won't show all the search method here, but will focus on the code which lists matched terms:
public static void doSearch(IndexReader reader, IndexSearcher searcher,
Query query, int max, String synList) throws IOException {
SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter("\001", "\002");
Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query));
Analyzer analyzer;
if (synList != null) {
analyzer = new MyAnalyzer(synList);
} else {
analyzer = new MyAnalyzer();
}
// Collect all the docs
TopDocs results = searcher.search(query, max);
ScoreDoc[] hits = results.scoreDocs;
int numTotalHits = Math.toIntExact(results.totalHits.value);
System.out.println("\nQuery: " + query.toString());
System.out.println("Matches: " + numTotalHits);
// Collect matching terms
HashSet<String> matchedWords = new HashSet<String>();
int start = 0;
int end = Math.min(numTotalHits, max);
for (int i = start; i < end; i++) {
int id = hits[i].doc;
float score = hits[i].score;
Document doc = searcher.doc(id);
String docpath = doc.get("path");
String doctext = doc.get("text");
try {
TokenStream tokens = TokenSources.getTokenStream("text", null, doctext, analyzer, -1);
TextFragment[] frag = highlighter.getBestTextFragments(tokens, doctext, false, 100);
for (int j = 0; j < frag.length; j++) {
if ((frag[j] != null) && (frag[j].getScore() > 0)) {
String match = frag[j].toString();
addMatchedWord(matchedWords, match);
}
}
} catch (InvalidTokenOffsetsException e) {
System.err.println(e.getMessage());
}
System.out.println("matched file: " + docpath);
}
if (matchedWords.size() > 0) {
System.out.println("matched terms:");
for (String word : matchedWords) {
System.out.println(word);
}
}
}
Problem
While the correct documents are selected by these queries, and the fragments chosen for highlighting do contain the query terms, the highlighted pieces in some of the selected fragments extend over too much of the input.
For example, if the query is
+text:event +text:manage
(the first example in the test case) then I would expect to see 'event' and 'manage' in the highlighted list. But what I actually see is
event);
manage
Despite the highlighting process using an analyzer which breaks terms apart and treats punctuation characters as single terms, the highlight code is "hungry" and breaks on whitespace alone.
Similarly if the query is
+text:queeu~1
(my final test case) I would expect to only see 'queue' in the list. But I get
queue.
job.queue
task.queue
queue;
It is so nearly there... but I don't understand why the highlighted pieces are inconsistent with the index, and I don't think I should have to parse the list of matches through yet another filter to produce the correct list of matches.
I would really appreciate any pointers to what I am doing wrong or how I could improve my code to deliver exactly what I need.
Thanks for reading this far!
I managed to get this working by replacing the WhitespaceTokenizer and TechTokenFilter in my analyzer with a PatternTokenizer; the regular expression took a bit of work but once I had it all the matching terms were extracted with pinpoint accuracy.
The replacement analyzer:
public class MyAnalyzer extends Analyzer {
public MyAnalyzer(String synlist) {
if (synlist != "") {
this.synlist = synlist;
this.useSynonyms = true;
}
}
public MyAnalyzer() {
this.useSynonyms = false;
}
private static final String tokenRegex = "(([\\w]+-)*[\\w]+)|[^\\w\\s]";
#Override
protected TokenStreamComponents createComponents(String fieldName) {
PatternTokenizer src = new PatternTokenizer(Pattern.compile(tokenRegex), 0);
TokenStream result = new LowerCaseFilter(src);
if (useSynonyms) {
result = new SynonymGraphFilter(result, getSynonyms(synlist), Boolean.TRUE);
result = new FlattenGraphFilter(result);
}
return new TokenStreamComponents(src, result);
}
My question is a bit similar to this one : Change PDF Annotation properties using iTextSharp C#
But I want to specifically change font size of pdf annotation using iText 7. I have searched a lot online but haven't been able to find any great examples or documentation regarding this. Following is the code I have used.
static void EditAnnot(string PDF)
{
string OutPDF = #"C:\Users\AP037X\Desktop\test.pdf";
iText.Kernel.Pdf.PdfDocument pdfDoc = new iText.Kernel.Pdf.PdfDocument(new iText.Kernel.Pdf.PdfReader(PDF), new iText.Kernel.Pdf.PdfWriter(OutPDF));
iText.Kernel.Pdf.PdfDictionary pageDict = pdfDoc.GetPage(1).GetPdfObject();
iText.Kernel.Pdf.PdfArray annots = pageDict.GetAsArray(iText.Kernel.Pdf.PdfName.Annots);
if (annots != null)
{
for (int i = 0; i < annots.Size(); i++)
{
Console.WriteLine("Scan..");
if (annots.GetAsDictionary(i) == null)
{
Console.WriteLine("1");
//return;
}
iText.Kernel.Pdf.PdfString t = annots.GetAsDictionary(i).GetAsString(iText.Kernel.Pdf.PdfName.Contents);
if (t == null)
{
Console.WriteLine("2");
//return;
}
Console.WriteLine(t);
if (Convert.ToString(t).Trim() == "Change")
{
Console.WriteLine("Found");
Console.WriteLine(annots.Size());
iText.Kernel.Geom.Rectangle rect = annots.GetAsDictionary(i).GetAsRectangle(iText.Kernel.Pdf.PdfName.Rect);
iText.Kernel.Pdf.PdfString cont = new iText.Kernel.Pdf.PdfString("New String");
iText.Kernel.Pdf.Annot.PdfFreeTextAnnotation NewAnnot = new iText.Kernel.Pdf.Annot.PdfFreeTextAnnotation(rect,cont);
float[] color = { 1f,1f,0f};
NewAnnot.SetColor(color);
NewAnnot.Put(iText.Kernel.Pdf.PdfName.Contents, new iText.Kernel.Pdf.PdfString("lion"));
NewAnnot.Put(iText.Kernel.Pdf.PdfName.Font, *What to type here?*);
annots.Remove(i);
annots.Add(i, NewAnnot.GetPdfObject());
}
}
}
pdfDoc.Close();
CompressPDF(OutPDF);
}
I am trying to work with payloads in Lucene 6 but I am having troubles. The idea is to index payloads and use them in a CustomScoreQuery to check if the payload of a query term matches the payload for the document term.
Here is my payload filter:
#Override
public final boolean incrementToken() throws IOException {
if (!this.input.incrementToken()) {
return false;
}
// get the current token
final char[] token = Arrays.copyOfRange(this.termAtt.buffer(), 0, this.termAtt.length());
String stoken = String.valueOf(token);
String[] parts = stoken.split(Constants.PAYLOAD_DELIMITER);
if (parts.length > 1 && parts.length == 2){
termAtt.setLength(parts[0].length());
// the rest is the payload
BytesRef br = new BytesRef(parts[1]);
System.out.println(br);
payloadAtt.setPayload(br);
}else if (parts.length > 1){
// skip
}else{
// no payload here
payloadAtt.setPayload(null);
}
return true;
}
It seems to be adding the payload, however when I try to access the payload in CustomScoreQuery it just keeps returning null.
public float determineBoost(int doc) throws IOException{
float boost = 1f;
LeafReader reader = this.context.reader();
System.out.println("Has payloads:" + reader.getFieldInfos().hasPayloads());
// loop through each location of the term and boost if location matches the payload
if (reader != null){
PostingsEnum posting = reader.postings(new Term(this.field, term.getTerm()), PostingsEnum.POSITIONS);
System.out.println("Term: " + term.getTerm());
if (posting != null){
// move to the document currently looking at
posting.advance(doc);
int count = 0;
while (count < posting.freq()){
BytesRef load = posting.getPayload();
System.out.println(posting);
System.out.println(posting.getClass());
System.out.println(posting.attributes());
System.out.println("Load: " + load);
// if the location matches in the term location than boos the term by the boost factor
try {
if(load != null && term.containLocation(new Payload(load))){
boost = boost * this.boost;
}
} catch (PayloadException e) {
// do not care too much, the payload is unrecognized
// this is not going to change the boost factor
}
posting.nextPosition();
count += 1;
}
}
}
return boost;
}
For my two tests it keeps stating the load is null. Any suggestions or help?
Using OWL API 3.4.9.
Given an OWLClass and on ontology, how can I get <rdfs:label> of that OWLClass in that ontology?
I hope to get the label in the type of String.
Inspired from the guide to the OWL-API, the following code should work (not tested):
//Initialise
OWLOntologyManager m = create();
OWLOntology o = m.loadOntologyFromOntologyDocument(pizza_iri);
OWLDataFactory df = OWLManager.getOWLDataFactory();
//Get your class of interest
OWLClass cls = df.getOWLClass(IRI.create(pizza_iri + "#foo"));
// Get the annotations on the class that use the label property (rdfs:label)
for (OWLAnnotation annotation : cls.getAnnotations(o, df.getRDFSLabel())) {
if (annotation.getValue() instanceof OWLLiteral) {
OWLLiteral val = (OWLLiteral) annotation.getValue();
// look for portuguese labels - can be skipped
if (val.hasLang("pt")) {
//Get your String here
System.out.println(cls + " labelled " + val.getLiteral());
}
}
}
The accepted answer is valid for OWLAPI version 3.x (3.4 and 3.5 versions) but not for OWL-API 4.x and newer.
To retrieve rdfs:label values asserted against OWL classes, try this instead:
OWLClass c = ...;
OWLOntology o = ...;
IRI cIRI = c.getIRI();
for(OWLAnnotationAssertionAxiom a : ont.getAnnotationAssertionAxioms(cIRI)) {
if(a.getProperty().isLabel()) {
if(a.getValue() instanceof OWLLiteral) {
OWLLiteral val = (OWLLiteral) a.getValue();
System.out.println(c + " labelled " + val.getLiteral());
}
}
}
EDIT
As Ignazio has pointed out, EntitySearcher can also be used, for example:
OWLClass c = ...;
OWLOntology o = ...;
for(OWLAnnotation a : EntitySearcher.getAnnotations(c, o, factory.getRDFSLabel())) {
OWLAnnotationValue val = a.getValue();
if(val instanceof OWLLiteral) {
System.out.println(c + " labelled " + ((OWLLiteral) val).getLiteral());
}
}
Here is a method I wrote to extract labels from a class.
private List<String> classLabels(OWLClass class){
List<String> labels;
labels = ontologiaOWL.annotationAssertionAxioms(class.getIRI())
//get only the annotations with rdf Label property
.filter(axiom -> axiom.getProperty().getIRI().getIRIString().equals(OWLRDFVocabulary.RDFS_LABEL.getIRI().getIRIString()))
.map(axiom -> axiom.getAnnotation())
.filter(annotation-> annotation.getValue() instanceof OWLLiteral)
.map(annotation -> (OWLLiteral) annotation.getValue())
.map(literal -> literal.getLiteral())
.collect(Collectors.toList());
return labels;
}
Can someone give me hints to apply pseudo-feedback in lucene. I can not find much help on google. I am using Similarity classes.
Is there any class in lucene which I can extend to implement feedback ?
thanks.
Assuming you are referring to this relevance feedback method, Once you have the TopDocs for your original query, iterate through how ever many (let's say we'll want the top 25 terms for the top 25 docs of the original query) records you desire, and call IndexReader.getTermVectors(int), which will grab the information you need. Iterate through each. while storing the term frequencies in a hash map would be the implementation the immediately occurs to me.
Something like:
//Get the original results
TopDocs docs = indexsearcher.search(query,25);
HashMap<String,ScorePair> map = new HashMap<String,ScorePair>();
for (int i = 0; i < docs.scoreDocs.length; i++) {
//Iterate fields for each result
FieldsEnum fields = indexreader.getTermVectors(docs.scoreDocs[i].doc).iterator();
String fieldname;
while (fieldname = fields.next()) {
//For each field, iterate it's terms
TermsEnum terms = fields.terms().iterator();
while (terms.next()) {
//and store it
putTermInMap(fieldname, terms.term(), terms.docFreq(), map);
}
}
}
List<ScorePair> byScore = new ArrayList<ScorePair>(map.values());
Collections.sort(byScore);
BooleanQuery bq = new BooleanQuery();
//Perhaps we want to give the original query a bit of a boost
query.setBoost(5);
bq.add(query,BooleanClause.Occur.SHOULD);
for (int i = 0; i < 25; i++) {
//Add all our found terms to the final query
ScorePair pair = byScore.get(i);
bq.add(new TermQuery(new Term(pair.field,pair.term)),BooleanClause.Occur.SHOULD);
}
}
//Say, we want to score based on tf/idf
void putTermInMap(String field, String term, int freq, Map<String,ScorePair> map) {
String key = field + ":" + term;
if (map.containsKey(key))
map.get(key).increment();
else
map.put(key,new ScorePair(freq,field,term));
}
private class ScorePair implements Comparable{
int count = 0;
double idf;
String field;
String term;
ScorePair(int docfreq, String field, String term) {
count++;
//Standard Lucene idf calculation. This is calculated once per field:term
idf = (1 + Math.log(indexreader.numDocs()/((double)docfreq + 1))) ^ 2;
this.field = field;
this.term = term;
}
void increment() { count++; }
double score() {
return Math.sqrt(count) * idf;
}
//Standard Lucene TF/IDF calculation, if I'm not mistaken about it.
int compareTo(ScorePair pair) {
if (this.score() < pair.score()) return -1;
else return 1;
}
}
(I make no claim that this is functional code, in it's current state.)