I am trying to add 2 different interceptors on the same source and send the intercepted data to 2 different channels.
But, I was not able to configure the same. Couldn't find any documentation about the same. Also, I am having some issues with the channel selectors. Not sure how to select a channel with the different interceptors.
Here is my code so far:
a1.sources = syslog_udp
a1.channels = chan1 chan2
a1.sinks = sink1 sink2 //both are different kafka sinks
a1.sources.syslog_udp.type = syslogudp
a1.sources.syslog_udp.port = 514
a1.sources.syslog_udp.host = 0.0.0.0
a1.sources.syslog_udp.keepFields = true
a1.sources.syslog_udp.interceptors = i1 i2
a1.sources.syslog_udp.interceptors.i1.type = regex_filter
a1.sources.syslog_udp.interceptors.i1.regex = '<regex_string1>'
a1.sources.syslog_udp.interceptors.i1.excludeEvents = false
a1.sources.syslog_udp.interceptors.i2.type = regex_filter
a1.sources.syslog_udp.interceptors.i2.regex = '<regex_string1>'|'<regex_string2>'
a1.sources.syslog_udp.interceptors.i2.excludeEvents = false
a1.sources.syslog_udp.selector.type = multiplexing
a1.sources.syslog_udp.channels = chan1 chan2
a1.channels.chan1.type = memory
a1.channels.chan1.capacity = 200
a1.channels.chan2.type = memory
a1.channels.chan2.capacity = 200
Seems like there is no straight-forward setup for this.
A work-around for this kind of layout is to have a single/wider channel interceptor in one agent, pipe the output to an avro-sink and setup a new agent for the avro-source and set-up the new channel interceptor on that.
I migrated app from vb6 to vb.net, every thing is working fine except getting the message from the MQ server, when MQGet called, I get 2033 error (no Message)
also attached the capture message that sent to MQ from working vb6 and from not working vb.net, please help?enter image description here
'***********************************
'Send(MQPUT) to MQSeries and CICS
'***********************************
PutMsgOpts = MQPMO_DEFAULT
MsgDesc = MQMD_DEFAULT
MsgDesc.Persistence = MQPER_PERSISTENT
MsgDesc.MsgId = MQMI_NONE.Value
MsgDesc.CorrelId = "AMQ!NEW_SESSION_CORRELID" 'if using MQBridge
MsgDesc.ReplyToQ = gReplyToQ
MsgDesc.ReplyToQMgr = gMQRplyMgrName
MsgDesc.Format_Renamed = MQFMT_STRING
'sPutMsg is composed of 8 byte program name(host/cics)
' plus data desired to pass as string only (dfcommarea)
'************ MAX LENGTH IS 32776 *******************
sPutMsg = gCICSPrgName & gsHost_Msg
If Len(sPutMsg) <> 32768 Then
sPutMsg = sPutMsg & Space(32768 - Len(sPutMsg))
End If
sMsgIdGet.Value = MsgDesc.MsgId
'*******************************************
'MQClose the queue for request sent to host
'*******************************************
'HOST-BYPASS
MQCLOSE(Hconn, Hobj, MQCO_NONE, cC, Reason)
'HOST-BYPASS
'**********************************************************
'MQOpen the queue for receiving the request from the host
'**********************************************************
'set up the queue name
ObjDesc = MQOD_DEFAULT
ObjDesc.ObjectName = gReplyToQ
'Open
'HOST-BYPASS
MQOPEN(Hconn, ObjDesc, MQOO_INPUT_AS_Q_DEF Or MQOO_FAIL_IF_QUIESCING, Hobj, cC, Reason)
'HOST-BYPASS
'Hconn is set by MQseries in the Connect
'**************************************
'Receive (MQGET) from MQSeries and CICS
'**************************************
GetMsgOpts = MQGMO_DEFAULT
GetMsgOpts.Options = MQGMO_WAIT Or MQGMO_CONVERT
GetMsgOpts.WaitInterval = 10000 '20 seconds
MsgDesc = MQMD_DEFAULT
MsgDesc.Format_Renamed = MQFMT_STRING
MsgDesc.Persistence = MQPER_PERSISTENT
MsgDesc.CorrelId = sMsgIdGet.Value 'set correlId with MQGET msgid returned
MsgDesc.MsgId = MQMI_NONE.Value
MsgDesc.ReplyToQ = gReplyToQ
MsgDesc.ReplyToQMgr = gMQRplyMgrName
'** sGetMsg is composed of 8 byte program name(host/cics)
'** plus data desired to pass as string only (dfcommarea)
'************ MAX LENGTH IS 32776 *******************
sGetMsg = New String(" ", 32768)
'Command to receive from to MQSeries and CICS
retryCount = 0
cC = MQCC_OK + 1
Reason = 2033
Do While cC <> MQCC_OK And Reason = 2033
MQGET(Hconn, Hobj, MsgDesc, GetMsgOpts, Len(sGetMsg), sGetMsg, readlen, cC, Reason)
retryCount = retryCount + 1
If retryCount > 1 Then Exit Do
Loop
'************************
'MQClose queue for input
'************************
MQCLOSE(Hconn, Hobj, MQCO_NONE, cC, Reason)
If you get 2033, then you know the queue is there, but the message you are trying to retrieve is not. Use MQ Explorer or plain MQ samples along with runmqsc to check if there are any messages on that queue, and if there are, if their correlation id matches what your program is supplying.
If there are no messages there, then check that your messages have been indeed received by the CICS program your are sending them to, and if that program responded.
A 2033 problem may be your coding problem, but more probably it's something in the configuration and setup that has changed along with your conversion. You may have a differently configured queue manager, with differently configured channels etc. Check the entire chain.
I am trying to test sending data to heka's UDPInput with no success. I decided to try to use the heka-flood tool to mimic UPD traffic also with no success. I am using 0.10 version of heka. My heka.toml :
[UdpInput]
address = "127.0.0.1:4880"
net = "udp"
splitter = "udp_splitter"
decoder = "ProtobufDecoder"
set_hostname = true
# I have also tried not setting this as well
[udp_splitter]
type = "HekaFramingSplitter"
[ProtobufDecoder]
[LogOutput]
type = "LogOutput"
message_matcher = "Logger == 'UdpInput'"
encoder = "PayloadEncoder"
and my flood.toml:
[udp_proto]
ip_address = "127.0.0.1:4880"
sender = "udp"
pprof_file = ""
encoder = "protobuf"
num_messages = 1000
corrupt_percentage = 0.0001
signed_percentage = 0.00011
variable_size_messages = false
ascii_only = true
max_message_size = 32000
If I add another input, like say a log tailer and add it to the message matcher for the LogOutput, those messages end up being logged out. I never see anything from the UpdInput. What am I doing wrong?
I have a setup where celery_result_backend has been configured to 'amqp'. I can see my tasks getting executed by the worker in logs. But
It is creating the queue with task id but its status is expired.I am not getting the result (result = AsyncResult(taskid); result.get() hangs). I tried all the backed supported:
1)Mysql: It is not putting data to the celery created tables
2) Redis: It is not putting data to the db
I two centos system.
1) I am calling the delay method to send the task to proper rabbitmq. And the worker is listening to the queue, from there it will pick the task and process(I can see task in the queue and getting executed by the worker in machine 2 But the result is not being put into the backend.
).Here I am doing the result.get() It hangs.
2) The worker is running on it to execute the task.It executes the task but I think not able to put the rersult
Settings:
RABBITMQ_BROKER_HOST = '10.213.166.133'
RABBITMQ_BROKER_PORT = dqms_settings.RABBITMQ_BROKER_PORT
RABBITMQ_BROKER_VHOST = dqms_settings.RABBITMQ_BROKER_VHOST
RABBITMQ_BROKER_USERNAME = dqms_settings.RABBITMQ_BROKER_USERNAME
RABBITMQ_BROKER_PASSWORD = dqms_settings.RABBITMQ_BROKER_PASSWORD
BROKER_URL = 'amqp://%s:%s#%s:%s/%s' % (RABBITMQ_BROKER_USERNAME,
RABBITMQ_BROKER_PASSWORD,
RABBITMQ_BROKER_HOST,
RABBITMQ_BROKER_PORT,
RABBITMQ_BROKER_VHOST)
#CELERY_TASK_RESULT_EXPIRES = 18000
#CELERY_IGNORE_RESULT = True
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
#CELERY_RESULT_BACKEND = 'db+mysql://svcacct-dqms:s3cretP#ssw0rd#10.213.166.202:3306/dqms'
#CELERY_RESULT_BACKEND = 'amqp'
#CELERY_AMQP_TASK_RESULT_EXPIRES = 1000
#CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = TIME_ZONE
CELERYD_PREFETCH_MULTIPLIER = dqms_settings.CELERYD_PREFETCH_MULTIPLIER
CELERY_DEFAULT_QUEUE = dqms_settings.CELERY_DEFAULT_QUEUE
CELERY_DEFAULT_EXCHANGE_TYPE = dqms_settings.CELERY_DEFAULT_EXCHANGE_TYPE
CELERY_DEFAULT_ROUTING_KEY = dqms_settings.CELERY_DEFAULT_ROUTING_KEY
CELERY_QUEUES = dqms_settings.CELERY_QUEUES
CELERY_ROUTES = dqms_settings.CELERY_ROUTES
CELERYD_HIJACK_ROOT_LOGGER = dqms_settings.CELERYD_HIJACK_ROOT_LOGGER
CELERY_ACKS_LATE = dqms_settings.CELERY_ACKS_LATE
CELERY_RESULT_BACKEND = 'redis://:s3cretP#ssw0rd#10.213.166.204:6379/5' #'djcelery.backends.database.DatabaseBackend'
#CELERY_REDIS_MAX_CONNECTIONS = 6
#CELERY_ALWAYS_EAGER = False
Can some one help why it is not putting the result in the queue?
This is a issue which is happening quite common now.
setting CELERY_ALWAYS_EAGER to TRUE will do the work
However this is not the best solution in production scenario.
I have a web service , i add some extra class which have message contract and after that it changed the way we access some of the methods( and i have not added message contract to these classes these are data contracts ), earlier i.e before we could create one object for request and response (like see the Before part) we are creating a single object for OrderStatusResponse Class. But if you see now the After(we have to create separate objects for request and response).
is this a side effect of enabling "Always generate message contract?"
Before
SmartConnect.Service1Client Client =
new SmartConnectClient.SmartConnect.Service1Client();
SmartConnect.OrderStatusResponse Status =
new SmartConnectClient.SmartConnect.OrderStatusResponse();
Status.UserID = "1234";
Status.Password = "abcd";
Status.SoftwareKey = "abc";
Status.OrderNumber = "1234";
Status = Client.GetOrderStatus(Status);
lbl_OS.Text = Status.Status.ToString();
lbl_RM.Text = Status.ReturnMessage.ToString();
After
SmartConnectRepublic.SmartConnectClient SmartClient =
new WCF_Client.SmartConnectRepublic.SmartConnectClient();
//SmartConnectRepublic.OrderStatusResponse Status =
new WCF_Client.SmartConnectRepublic.OrderStatusResponse();
WCF_Client.SmartConnectRepublic.GetOrderStatusRequest request =
new WCF_Client.SmartConnectRepublic.GetOrderStatusRequest();
request.status = new WCF_Client.SmartConnectRepublic.OrderStatusResponse();
request.status.OrderNumber = "1055055";
request.status.UserID = "1234";
request.status.Password = "dfsdfsd";
request.status.SoftwareKey = "sdfsdfsdfs";
WCF_Client.SmartConnectRepublic.GetOrderStatusResponse response =
new WCF_Client.SmartConnectRepublic.GetOrderStatusResponse();
response = SmartClient.GetOrderStatus(request);
lbl_Status.Text = response.GetOrderStatusResult.Status;
lbl_RC.Text = response.GetOrderStatusResult.ReturnCode.ToString();
lbl_RM.Text = response.GetOrderStatusResult.ReturnCode.ToString();
Yes, I suspect it is a difference with using message contracts. You seem to have figured it out, though.