solr indexing custom json - creates empty documents - indexing

Question

I am quite new to solr. I currently have it running in cloud mode using docker compose (my configuration can be seen at the end of the question)
I created a collection called audittrail using default configuration. The idea is that I'll send event logging info from another app to solr. It has a convenient looking schema full of dynamic fields by default. (I know I shouldn't just use default settings in production, right now I'm looking for a proof of concept).
Now I'm following this document in an attempt to index some of my data: https://lucene.apache.org/solr/guide/7_2/transforming-and-indexing-custom-json.html#mapping-parameters
> curl 'http://0.0.0.0:8983/api/collections/audittrail/update/json'\
'?split=/events&'\
'f=action_kind_s:/action_kind_s&'\
'f=time_dt:/events/time_dt'\
'&echo=true' \ ########## NOTE this means we're running in debug more. solr returns the documents it should be creating
-H 'Content-type:application/json' -d '{
"action_kind_s": "task_exec",
"events": [
{
"event_kind_s": "start",
"in_transaction_b": false,
"time_dt": "2018-03-09T12:57:07Z"
},
{
"event_kind_s": "start_txn",
"in_transaction_b": true,
"time_dt": "2018-03-09T12:57:07Z"
},
{
"event_kind_s": "diff",
"in_transaction_b": true,
"key_s": "('MerchantWorkerProcess', 5819715045818368L)",
"property_s": "claim_time",
"time_dt": "2018-03-09T12:57:07Z",
"value_dt": "2018-03-09T12:57:07Z"
},
],
"final_status_s": "COMPLETE",
"request_s": "1dfda9955dac6f3cfd76fbedee98b15f6edc0db",
"task_name_s": "0p5k20100CcnMVxaxoWl32WlfPixjV1OFKgv0k1KZ0m_acc_work"
}'
# response:
{
"responseHeader":{
"status":0,
"QTime":1},
"docs":[{},
{},
{}]}
That's three empty documents...
So I thought maybe it was because I wasn't specifying an id. So I gave each event a unique id and tried again with the added &f=id:/events/id. Same result
Originally I tried using wildcards (&f=/**) with the same effect.
There is obviously something missing in my understanding.
So my question is:
What should I do to get my documents populated correctly?
EDIT
Also, my solr node logs arent turnng up any errors. Here's a sample:
2018-03-09 14:30:50.770 INFO (qtp257895351-21) [c:audittrail s:shard2 r:core_node4 x:audittrail_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [audittrail_shard2_replica_n2] webapp=null path=/update/json params={split=/events}{add=[78953602-6b02-4948-8443-fd1ebc340921 (1594470800573857792)]} 0 3
2018-03-09 14:31:05.770 INFO (commitScheduler-14-thread-1) [c:audittrail s:shard2 r:core_node4 x:audittrail_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1594470816305643520,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2018-03-09 14:31:05.770 INFO (commitScheduler-14-thread-1) [c:audittrail s:shard2 r:core_node4 x:audittrail_shard2_replica_n2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter#13d117d6 commitCommandVersion:1594470816305643520
2018-03-09 14:31:05.918 INFO (commitScheduler-14-thread-1) [c:audittrail s:shard2 r:core_node4 x:audittrail_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher#4edc35b0[audittrail_shard2_replica_n2] realtime]
2018-03-09 14:31:05.921 INFO (commitScheduler-14-thread-1) [c:audittrail s:shard2 r:core_node4 x:audittrail_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
docker-compose.yml
version: '3'
services:
zookeeper:
image: zookeeper:3.4.11
ports:
- "2181:2181"
hostname: "zookeeper"
container_name: "zookeeper"
solr1:
image: solr:7.2.1
ports:
- "8983:8983"
container_name: solr1
links:
- zookeeper:ZK
command: /opt/solr/bin/solr start -f -z zookeeper:2181
solr2:
image: solr:7.2.1
ports:
- "8984:8983"
container_name: solr2
links:
- zookeeper:ZK
command: /opt/solr/bin/solr start -f -z zookeeper:2181
Here are the exact steps I go through to index some data.
This does not actually index anything and I want to know why
docker-compose up
create the collection
curl -X POST 'http://0.0.0.0:8983/solr/admin/collections?action=CREATE&name=audittrail&numShards=2'
{
"responseHeader":{
"status":0,
"QTime":6178},
"success":{
"172.24.0.3:8983_solr":{
"responseHeader":{
"status":0,
"QTime":3993},
"core":"audittrail_shard1_replica_n1"},
"172.24.0.4:8983_solr":{
"responseHeader":{
"status":0,
"QTime":4399},
"core":"audittrail_shard2_replica_n2"}},
"warning":"Using _default configset. Data driven schema functionality is enabled by default, which is NOT RECOMMENDED for production use. To turn it off: curl http://{host:port}/solr/audittrail/config -d '{\"set-user-property\": {\"update.autoCreateFields\":\"false\"}}'"}
curl to create some data ( this is the same curl as in the main question. but not in debug mode:
curl 'http://0.0.0.0:8983/api/collections/audittrail/update/json?split=/events&f=action_kind_s:/action_kind_s&f=time_dt:/events/time_dt' -H 'Content-type:application/json' -d '{ "action_kind_s": "task_exec", "events": [{"event_kind_s": "start","in_transaction_b": false, "time_dt": "2018-03-09T12:57:07Z"},{"event_kind_s": "start_txn", "in_transaction_b": true,"time_dt": "2018-03-09T12:57:07Z"},{"event_kind_s": "diff", "in_transaction_b": true,"key_s": "('MerchantWorkerProcess', 5819715045818368L)","property_s": "claim_time","time_dt": "2018-03-09T12:57:07Z","value_dt": "2018-03-09T12:57:07Z"},], "final_status_s": "COMPLETE", "request_s": "xxx", "task_name_s": "xxx"}'
{
"responseHeader":{
"status":0,
"QTime":126}}
Do the query:
curl 'http://0.0.0.0:8983/solr/audittrail/select?q=*:*'
{
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":12,
"params":{
"q":"*:*"}},
"response":{"numFound":0,"start":0,"maxScore":0.0,"docs":[]
}}

Answer 1

It seems that it's only the echo parameter that doesn't do what you expect it to do - remove that, and add commit=true to your URL to make Solr commit the documents to the index as soon as possible before returning, and you can then find documents (by searching for *:* in the admin interface under collection -> query with your fields present in the index:
{
"action_kind_s":"task_exec",
"time_dt":"2018-03-09T12:57:07Z",
"id":"b56100f5-ff61-45e7-8d6b-8072bac6c952",
"_version_":1594486636806144000},
{
"action_kind_s":"task_exec",
"time_dt":"2018-03-09T12:57:07Z",
"id":"f49fc3cb-eac6-4d02-bcdf-b7c1a34782e3",
"_version_":1594486636807192576}