I need a way to score the lucene documents using term frequency only. Is there any flag that needs to be changed for this? - lucene

Question

If i have two documents with D1 having term "lucene" twice and D2 having term "lucene" thrice. I want lucene to score D2 higher than D1. Note here that, D1 has only two words (i.e. lucene lucene) while D3 has 100 words out of which 3 words are lucene. Default lucene scoring model will score D1 higher than D2. I want to disable this mode and rank D2 higher than D1. That's my project requirement.

Answer 1

You'll need to implement a Similarity which does what you want. You could implement directly on Similarity, but you'll probably find it's simpler to just copy ClassicSimilarity (DefaultSimilarity, before version 5.4), and stub out the things you don't want to impact your score (ie. return a constant). For instance, here's a very simple implementation that would simply return the frequency of the terms in the query:
import org.apache.lucene.index.FieldInvertState;
import org.apache.lucene.search.similarities.TFIDFSimilarity;
import org.apache.lucene.util.BytesRef;
public class SimpleSimilarity extends TFIDFSimilarity {
//Comments describe briefly what these methods do in the *standard* implementation.
//Not what they do in this implementation (which, for most of them, is nothing at all)
public SimpleSimilarity() {}
//boosts results which match more query terms
#Override
public float coord(int overlap, int maxOverlap) {
return 1f;
}
//constant per query, normalizes scores somewhat based on query
#Override
public float queryNorm(float sumOfSquaredWeights) {
return 1f;
}
//Norms should be disabled when using this similarity
//They are useless to it, and would just be wasted space.
#Override
public final long encodeNormValue(float f) {
return 1L;
}
#Override
public final float decodeNormValue(long norm) {
return 1f;
}
//Weighs shorter fields more heavily
#Override
public float lengthNorm(FieldInvertState state) {
return 1f;
}
//Higher frequency terms (more matches) scored higher
#Override
public float tf(float freq) {
//return (float)Math.sqrt(freq); The standard tf impl
return freq;
}
//Scores closer matches higher when using a sloppy phrase query
#Override
public float sloppyFreq(int distance) {
return 1.0f;
}
//ClassicSimilarity doesn't really do much with payloads. This is unmodified
#Override
public float scorePayload(int doc, int start, int end, BytesRef payload) {
return 1f;
}
//Weigh matches on rarer terms more heavily.
#Override
public float idf(long docFreq, long numDocs) {
return 1f;
}
#Override
public String toString() {
return "SimpleSimilarity";
}
}