How to set PartitionKey property from stream analytics on the output message to service bus? - azure-stream-analytics

Question

I have following setup:
Event hub
Service bus topic with a single subscription that accepts all messages from the topic (for my POC)
Above service bus subscription is setup with sessions enabled
Stream analytics (SA) job that moves the events from event hub (the input) into service bus topic (the output). Here is my SA query:
SELECT *, LOWER(source) as Partner
INTO [sb-output]
FROM [test-input]
The above job also sets the partition key for service bus. Used following json in the System Properties of [sb-output] based on the documentation at https://learn.microsoft.com/en-us/azure/stream-analytics/service-bus-topics-output#custom-metadata-properties-for-output:
{ "PartitionKey" : "Partner" }
What I did:
Sent an event to event hub without partition key. This was successful.
{
"specversion": "1.0",
"id": "c8c4faad-9f53-4e43-95ca-c318d673660a",
"type": "CustomerChanged",
"time": "2020-09-09T22:25:40.0148301Z",
"source": "ABCD",
"subject": "system-1",
"datacontenttype": "application/json",
"dataschema": "1.0",
"data": {
"customerNumber": "7879875212123",
"firstName": "John",
"lastName" : "Kennedy"
}
}
The SA successfully moved the event from event hub to service bus.
The service bus subscription succcessfully received the message as shown below:
{
"specversion": "1.0",
"id": "c8c4faad-9f53-4e43-95ca-c318d673660a",
"type": "CustomerChanged",
"time": "2020-09-09T23:22:13.3647825Z",
"source": "ABCD",
"subject": "system-1",
"datacontenttype": "application/json",
"dataschema": "1.0",
"data": {
"customerNumber": "7879875212123",
"firstName": "John",
"lastName": "Kennedy"
},
"EventProcessedUtcTime": "2020-09-09T23:22:14.3776603Z",
"PartitionId": 0,
"EventEnqueuedUtcTime": "2020-09-09T23:22:14.3080000Z",
"Partner": "abcd"
}
As can be seen, the property Partner is at the end of the message.
However the Service Bus Explorer tool shows me that the PartitionKey property has not been set to "abcd" but to some other random string.
Troubleshooting:
To make sure that I can send a message to service bus topic with a specific PartitionKey key, I wrote a sample code that submits the message to service bus topic by explicitly setting the session id property on the message. The service bus explorer showed me that both SessionId as well as PartitionKey properties are set to the correct value.
In ASA output configuration, tried to set following system properties json. Neither worked.
{ "SessionId" : "Partner" }
{ "PartitionKey" : "Partner", "SessionId" : "Partner" }
In ASA output configuration, tried setting both Property Columns (to Partner) as well as System Property Columns (to { "PartitionKey" : "Partner" }). That did not work.
Question:
What am I doing wrong with ASA output configuration that is circumventing the propagation of PartitionKey value from my custom field to service bus message?
Also is there a reason why the System Property Columns does not show the json text that I entered after saving?

Answer 1

Sharing the answer as per the comment by the original poster:
Currently, Microsoft is working on a fix.
Until then, you can use the suggested work around: { "PartitionKey" : "Partner", "SessionId" : "Partner", "Label": "Partner" } in system properties.