Use Solr to index/search txt file content - apache

Question

I'm making a study to compare different search platforms' performance over Twitter's tweets. For my purpose I have collected a set of tweets (around 50,000) and saved them in a single text (.txt) file in a format similar to the following:
Tweet ID User Tweet Content Tweet Time-stamp
The data would look like this:
31261817690923008 username1 tweet 1 content goes here 1482180069
31132193287839744 username2 tweet 2 content goes here 1274400000
Now, using Solr 6.3.0, is it possible to index each line of content separately? Instead, should I use XML or JSON? or do I have to store each line (tweet) in a different file?

Answer 1

You can use the CSV Update Handler, which will result in a single document for each row.
To adjust the parsing to the structure you've used, you can use separator (TAB? %09) to provide the separator used between fields / columns, encapsulator to set the value used to encapsulate a single field value (it doesn't seem you've used any) and fieldnames to provide a proper field name for each column, unless they're in the first row - in that case set header to true (and don't provide fieldnames).

Answer 2

Assuming two things:
#1 You do not want to do an awful lot of coding for the data entry.
#2 Your text file is TAB or comma separated.
If so, you can easily turn it into an XML that can be added via the Admin interface.
A few things to keep in mind:
Enclose your data in <add> ... </add> blocks of a reasonable size. Ideally not 50K. Experiment a little.
Enclose each entry - line in your case in <doc> ... <doc>
Each column needs to have its own field as in
<field name="id"> ... </field>
<field name="username"> ... </field>
...
All need unique IDs.
For practical purposes, if you can open the textfile in a spreadsheet, add the tag columns in between your data and then concatenate the lines, it is relatively easy even if a little labour intensive for 50K.
A doc set of two would look something like:
<add>
<doc>
<field name="id"> ... </field>
<field name="user"> ... </field>
<field name="content"> ... </field>
<field name="time_stamp"> ... </field>
</doc>
<doc>
<field name="id"> ... </field>
<field name="user"> ... </field>
<field name="content"> ... </field>
<field name="time_stamp"> ... </field>
</doc>
</add>