(pure) Lucene: Counting documents having a timestamp in longfield, grouped by year - lucene

Question

My documents structure is:
[text:TextField,date:LongField]
I am looking for a 'statistic' query on my documents, based on a precision level on the dateTime field. This means counting documents grouped by the LongField date, ignoring some bytes at the right of the date.
For a given precision, I am looking for how many documents match for each different values of this precision.
Assuming the precision 'year' is grouping by "date/10000"
With the following data:
{text:"text1",dateTime:(some timestamp where year is 2015 like 20150000)}
{text:"text2",dateTime:(some timestamp where year is 2010 like 20109878)}
{text:"text3",dateTime:(some timestamp where year is 2015 like 20150024)}
{text:"text14,dateTime:(some timestamp where year is 1997 like 19970987)}
The result should be:
[{bracket:1997, count:1}
{bracket:2010, count:1}
{bracket:2015, count:2}]
While NumericRangeQuery allow to create 1 (or some) range, is it possible for lucene to generate the ranges based on a precision step?
I can handle this by creating a new field for each precision level that I need, but maybe this kind of things allready exists.
It's a kind of faceted search where the facet is the time. The use case should be:
-give me document count for each milleniums,
-then give me document count for each centuries (inside a millenium)
-then give me document count for each year (inside a century)
-then give me document count for each days (inside a year)
when 0 documents exists inside a bucket, the result should not be in the results.
Regards

Answer 1

Collector can do this without any trick, here is the working code:
public class GroupByTest1 {
private RAMDirectory directory;
private IndexSearcher searcher;
private IndexReader reader;
private Analyzer analyzer;
private class Data {
String text;
Long dateTime;
private Data(String text, Long dateTime) {
this.text = text;
this.dateTime = dateTime;
}
}
#Before
public void setUp() throws Exception {
directory = new RAMDirectory();
analyzer = new WhitespaceAnalyzer();
IndexWriter writer = new IndexWriter(directory, new IndexWriterConfig(analyzer));
Data datas[] = {
new Data("A", 2012L),
new Data("B", 2012L),
new Data("C", 2012L),
new Data("D", 2013L),
};
Document doc = new Document();
for (Data data : datas) {
doc.clear();
doc.add(new TextField("text", data.text, Field.Store.YES));
doc.add(new LongField("dateTime", data.dateTime, Field.Store.YES));
writer.addDocument(doc);
}
writer.close();
reader = DirectoryReader.open(directory);
searcher = new IndexSearcher(reader);
}
#Test
public void test1() throws Exception {
final Map<Integer, Long> map = new HashMap<>();
Collector collector = new SimpleCollector() {
int base = 0;
#Override
public void collect(int doc) throws IOException {
String year = reader.document(doc + base).get("dateTime");
if (!map.containsKey(Integer.valueOf(year))) {
map.put(Integer.valueOf(year), 1L);
} else {
long l = map.get(Integer.valueOf(year));
map.put(Integer.valueOf(year), ++l);
}
}
#Override
public boolean needsScores() {
return false;
}
#Override
protected void doSetNextReader(LeafReaderContext context) throws IOException {
base = context.docBase;
}
};
searcher.search(new MatchAllDocsQuery(), collector);
for (Integer integer : map.keySet()) {
System.out.print("year = " + integer);
System.out.println(" count = " + map.get(integer));
}
}
}
Output I get is following:
year = 2012 count = 3
year = 2013 count = 1
This may run slow depending on how many records you have. It loads every single document to know what is the year on it and groups based on that. There is also grouping module which you may also look into it.