GAWK - Finding a corresponding Action - gawk

Question

I have a large file of sequentiually added lines of data that relate to parcels going through a system, this file is added to as each action takes place and builds throughout the day. What I need to do it check the file say every minute and check for items that have not yet made it to there chute i.e. do not have an "DISCHARGE_VERIFIED". The example below is one complete record but this could be spread throughout the file as thousands of items are processed at the same time.
170209 043314 0887 DE(N) ItemHandler.ItemLog event=<ITEM_AT_INDUCTION>, *********************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: ProjectIdle>, inductionId=<3: IU04>, position=<sorter#0.induction#3: IU04>, itemRevisionNumber=<0> ##[
170209 043314 0888 DE(N) ItemHandler.ItemLog event=<SET_ITEM_ID>, ***************************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: ProjectIdle>, itemRevisionNumber=<0> ##[
170209 043317 0314 DE(N) ItemHandler.ItemLog event=<SCANNER_RESULT>, ************************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForData>, barcodeCount=<3>, barcodes=<[ProxyWrapperBarcode(barcode=<JJD014600004054211864>, type=<C0>, result=<OK>, ccType=<>), ProxyWrapperBarcode(barcode=<1910456693>, type=<A0>, result=<OK>, ccType=<>), ProxyWrapperBarcode(barcode=<2LAU2000+52000000>, type=<C0>, result=<OK>, ccType=<>)]>, codeSource=<ohscan>, scannerId=<4001: IU04-SCAN02>, scannerStatus=<0>, position=<sorter#0.scanner#4001: IU04-SCAN02>, itemRevisionNumber=<2> ##[
170209 043317 0315 DE(N) ItemHandler.ItemLog event=<DESTINATION_REQUEST>, *******************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForData>, barcodes=<[ProxyWrapperBarcode(barcode=<JJD014600004054211864>, type=<C0>, result=<OK>, ccType=<>), ProxyWrapperBarcode(barcode=<1910456693>, type=<A0>, result=<OK>, ccType=<>), ProxyWrapperBarcode(barcode=<2LAU2000+52000000>, type=<C0>, result=<OK>, ccType=<>)]>, ccReason=<SCANNER_DATA_ADDED>, PreviousccResult=<>, sortSchemeId=<-1>, sortSchemeName=<>, logicalDestination=<>, BatchCountItem=<true>, collectionId=<-1>, goodsId=<>, position=<sorter#0.scanner#4001: IU04-SCAN02>, dynamicDataCount=<0>, dynamicData=<{}>, carrierId=<159>, carrierCount=<-1>, itemRevisionNumber=<2> ##[
170209 043317 0322 DE(N) ItemHandler.ItemLog event=<DESTINATION_REPLY>, *********************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, ccReason=<SCANNER_DATA_ADDED>, PendingccResult=<OK>, Pendingstrategy=<notSpecified>, PendingchuteGroup=<[3000]: Parked0>, PendingNotChutedestinationId=<-1>, PendingsortSchemeId=<-1>, PendingsortSchemeName=<>, PendinglogicalDestination=<>, PendinggoodsId=<>, PendingBatchCountItem=<true>, PendingcollectionId=<-1>, position=<sorter#0.scanner#4001: IU04-SCAN02>, dynamicDataCount=<0>, dynamicData=<{}>, itemRevisionNumber=<4> ##[
170209 043317 0322 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM>, *************************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, PendingchuteGroup=<[3000]: Parked0>, Pendingstrategy=<notSpecified>, CscdestinationId=<-1: UnDef>, CmcdestinationId=<-1: UnDef>, position=<sorter#0.scanner#4001: IU04-SCAN02>, itemRevisionNumber=<4> ##[
170209 043317 0484 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM_REPLY>, *******************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, CscdestinationId=<3000: Parked0>, chuteGroup=<[3000]: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0.scanner#4001: IU04-SCAN02>, chuteListStartPoint=<-1>, itemRevisionNumber=<6> ##[
170209 043317 0486 DE(N) ItemHandler.ItemLog event=<RECONVERT>, *****************************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForData>, CscdestinationId=<3000: Parked0>, chuteGroup=<[3000]: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0.scanner#4001: IU04-SCAN02>, chuteListStartPoint=<-1>, itemRevisionNumber=<7> ##[
170209 043317 0486 DE(N) ItemHandler.ItemLog event=<DESTINATION_REQUEST>, *******************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForData>, barcodes=<[ProxyWrapperBarcode(barcode=<JJD014600004054211864>, type=<C0>, result=<OK>, ccType=<>), ProxyWrapperBarcode(barcode=<1910456693>, type=<A0>, result=<OK>, ccType=<>), ProxyWrapperBarcode(barcode=<2LAU2000+52000000>, type=<C0>, result=<OK>, ccType=<>)]>, ccReason=<SCANNER_DATA_ADDED>, PreviousccResult=<OK>, sortSchemeId=<-1>, sortSchemeName=<>, logicalDestination=<>, BatchCountItem=<true>, collectionId=<-1>, goodsId=<>, position=<sorter#0.scanner#4001: IU04-SCAN02>, dynamicDataCount=<0>, dynamicData=<{}>, carrierId=<159>, carrierCount=<-1>, itemRevisionNumber=<7> ##[
170209 043317 0492 DE(N) ItemHandler.ItemLog event=<DESTINATION_REPLY>, *********************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, ccReason=<SCANNER_DATA_ADDED>, PendingccResult=<OK>, Pendingstrategy=<priority>, PendingchuteGroup=<[121]: FLY425>, PendingNotChutedestinationId=<-1>, PendingsortSchemeId=<-1>, PendingsortSchemeName=<>, PendinglogicalDestination=<FLY425>, PendinggoodsId=<>, PendingBatchCountItem=<true>, PendingcollectionId=<-1>, position=<sorter#0.scanner#4001: IU04-SCAN02>, dynamicDataCount=<0>, dynamicData=<{}>, itemRevisionNumber=<9> ##[
170209 043317 0492 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM>, *************************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, PendingchuteGroup=<[121]: FLY425>, Pendingstrategy=<priority>, CscdestinationId=<3000: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0.scanner#4001: IU04-SCAN02>, itemRevisionNumber=<9> ##[
170209 043317 0666 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM_REPLY>, *******************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, CscdestinationId=<121: FLY425>, chuteGroup=<[121]: FLY425>, CmcdestinationId=<121: FLY425>, position=<sorter#0.scanner#4001: IU04-SCAN02>, chuteListStartPoint=<121>, itemRevisionNumber=<11> ##[
170209 043317 0667 DE(N) ItemHandler.ItemLog event=<ITEM_INDUCTED>, *************************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: ProjectHeadingForChute>, inductionId=<3: IU04>, inductionMode=<SCANNER>, inductStatus=<NORMAL_ITEM>, carrierId=<159>, carrierCount=<1>, CmcdestinationId=<121: FLY425>, position=<sorter#0: MS01>, itemRevisionNumber=<12> ##[
170209 043327 0379 DE(N) ItemHandler.ItemLog event=<ITEM_DISCHARGED>, ***********************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: ProjectHeadingForChute>, CscdestinationId=<121: FLY425>, chuteGroup=<[121]: FLY425>, CmcdestinationId=<121: FLY425>, ccResult=<OK>, sortSchemeId=<-1>, logicalDestination=<FLY425>, goodsId=<>, carrierId=<159>, length=<-1 mm>, width=<-1 mm>, height=<-1 mm>, volume=<-1 mm3>, position=<sorter#0.chute#121: FLY425>, itemRevisionNumber=<13> ##[
170209 043339 0765 DE(N) ItemHandler.ItemLog event=<DISCHARGE_VERIFIED>, ********************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: ProjectWaitForVerification>, CscdestinationId=<121: FLY425>, chuteGroup=<[121]: FLY425>, CmcdestinationId=<121: FLY425>, position=<sorter#0.chute#121: FLY425>, itemRevisionNumber=<14> ##[
Once an item has made it to it's chute it is denoted as "DISCHARGE_VERIFIED" so I need to be able to pull out of the log file, the lines the that have an "itemId" but with no corresponding "DISCHARGE_VERIFIED" at that point in time and display the chute it is heading for which is held in the line "ITEM_INDUCTED", in the element "CmcdestinationId=<121: FLY425>" see below
170209 043317 0667 DE(N) ItemHandler.ItemLog event=<ITEM_INDUCTED>, *************************, itemId=<12562305>, globalId=<12562305>, cmcIndex=<750>, sorter=<0: MS01>, state=<CSC: ProjectHeadingForChute>, inductionId=<3: IU04>, inductionMode=<SCANNER>, inductStatus=<NORMAL_ITEM>, carrierId=<159>, carrierCount=<1>, CmcdestinationId=<121: FLY425>, position=<sorter#0: MS01>, itemRevisionNumber=<12> ##[
I'm doing this on a windows machine using gawk so have all the usual quotation issues.
Any help would be appreciated
Thanks
Phil

Answer 1

I think I have a working script. Save it as inducted.awk for example, so you skip all the windows quotation issues you describe.
inducted.awk:
/ITEM_INDUCTED/ {
match($0, /itemId=<([^>]+)>/, ary1)
match($0, /CmcdestinationId=<([^>]+)>/, ary2)
dest[ary1[1]] = ary2[1]
}
/DISCHARGE_VERIFIED/ {
match($0, /itemId=<([^>]+)>/, ary1)
delete dest[ary1[1]]
}
END {
for (id in dest) {
print id " -- " dest[id]
}
}
So basically, when a line with ITEM_INDUCTED is found, it adds the itemid and destination to an array.
And when a line has DISCHARGED_VERIFIED, that information is deleted from the array.
At the end of the script, remaining inducted but not discharged ids are printed with their destinations.
To execute it:
gawk -f .\inducted.awk large_log_file
if the awk file is not in the same folder as large_log_file, specify the full path for it.