I am trying to add 2 different interceptors on the same source and send the intercepted data to 2 different channels.
But, I was not able to configure the same. Couldn't find any documentation about the same. Also, I am having some issues with the channel selectors. Not sure how to select a channel with the different interceptors.
Here is my code so far:
a1.sources = syslog_udp
a1.channels = chan1 chan2
a1.sinks = sink1 sink2 //both are different kafka sinks
a1.sources.syslog_udp.type = syslogudp
a1.sources.syslog_udp.port = 514
a1.sources.syslog_udp.host = 0.0.0.0
a1.sources.syslog_udp.keepFields = true
a1.sources.syslog_udp.interceptors = i1 i2
a1.sources.syslog_udp.interceptors.i1.type = regex_filter
a1.sources.syslog_udp.interceptors.i1.regex = '<regex_string1>'
a1.sources.syslog_udp.interceptors.i1.excludeEvents = false
a1.sources.syslog_udp.interceptors.i2.type = regex_filter
a1.sources.syslog_udp.interceptors.i2.regex = '<regex_string1>'|'<regex_string2>'
a1.sources.syslog_udp.interceptors.i2.excludeEvents = false
a1.sources.syslog_udp.selector.type = multiplexing
a1.sources.syslog_udp.channels = chan1 chan2
a1.channels.chan1.type = memory
a1.channels.chan1.capacity = 200
a1.channels.chan2.type = memory
a1.channels.chan2.capacity = 200
Seems like there is no straight-forward setup for this.
A work-around for this kind of layout is to have a single/wider channel interceptor in one agent, pipe the output to an avro-sink and setup a new agent for the avro-source and set-up the new channel interceptor on that.
Related
I am trying to test sending data to heka's UDPInput with no success. I decided to try to use the heka-flood tool to mimic UPD traffic also with no success. I am using 0.10 version of heka. My heka.toml :
[UdpInput]
address = "127.0.0.1:4880"
net = "udp"
splitter = "udp_splitter"
decoder = "ProtobufDecoder"
set_hostname = true
# I have also tried not setting this as well
[udp_splitter]
type = "HekaFramingSplitter"
[ProtobufDecoder]
[LogOutput]
type = "LogOutput"
message_matcher = "Logger == 'UdpInput'"
encoder = "PayloadEncoder"
and my flood.toml:
[udp_proto]
ip_address = "127.0.0.1:4880"
sender = "udp"
pprof_file = ""
encoder = "protobuf"
num_messages = 1000
corrupt_percentage = 0.0001
signed_percentage = 0.00011
variable_size_messages = false
ascii_only = true
max_message_size = 32000
If I add another input, like say a log tailer and add it to the message matcher for the LogOutput, those messages end up being logged out. I never see anything from the UpdInput. What am I doing wrong?
I have a setup where celery_result_backend has been configured to 'amqp'. I can see my tasks getting executed by the worker in logs. But
It is creating the queue with task id but its status is expired.I am not getting the result (result = AsyncResult(taskid); result.get() hangs). I tried all the backed supported:
1)Mysql: It is not putting data to the celery created tables
2) Redis: It is not putting data to the db
I two centos system.
1) I am calling the delay method to send the task to proper rabbitmq. And the worker is listening to the queue, from there it will pick the task and process(I can see task in the queue and getting executed by the worker in machine 2 But the result is not being put into the backend.
).Here I am doing the result.get() It hangs.
2) The worker is running on it to execute the task.It executes the task but I think not able to put the rersult
Settings:
RABBITMQ_BROKER_HOST = '10.213.166.133'
RABBITMQ_BROKER_PORT = dqms_settings.RABBITMQ_BROKER_PORT
RABBITMQ_BROKER_VHOST = dqms_settings.RABBITMQ_BROKER_VHOST
RABBITMQ_BROKER_USERNAME = dqms_settings.RABBITMQ_BROKER_USERNAME
RABBITMQ_BROKER_PASSWORD = dqms_settings.RABBITMQ_BROKER_PASSWORD
BROKER_URL = 'amqp://%s:%s#%s:%s/%s' % (RABBITMQ_BROKER_USERNAME,
RABBITMQ_BROKER_PASSWORD,
RABBITMQ_BROKER_HOST,
RABBITMQ_BROKER_PORT,
RABBITMQ_BROKER_VHOST)
#CELERY_TASK_RESULT_EXPIRES = 18000
#CELERY_IGNORE_RESULT = True
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
#CELERY_RESULT_BACKEND = 'db+mysql://svcacct-dqms:s3cretP#ssw0rd#10.213.166.202:3306/dqms'
#CELERY_RESULT_BACKEND = 'amqp'
#CELERY_AMQP_TASK_RESULT_EXPIRES = 1000
#CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = TIME_ZONE
CELERYD_PREFETCH_MULTIPLIER = dqms_settings.CELERYD_PREFETCH_MULTIPLIER
CELERY_DEFAULT_QUEUE = dqms_settings.CELERY_DEFAULT_QUEUE
CELERY_DEFAULT_EXCHANGE_TYPE = dqms_settings.CELERY_DEFAULT_EXCHANGE_TYPE
CELERY_DEFAULT_ROUTING_KEY = dqms_settings.CELERY_DEFAULT_ROUTING_KEY
CELERY_QUEUES = dqms_settings.CELERY_QUEUES
CELERY_ROUTES = dqms_settings.CELERY_ROUTES
CELERYD_HIJACK_ROOT_LOGGER = dqms_settings.CELERYD_HIJACK_ROOT_LOGGER
CELERY_ACKS_LATE = dqms_settings.CELERY_ACKS_LATE
CELERY_RESULT_BACKEND = 'redis://:s3cretP#ssw0rd#10.213.166.204:6379/5' #'djcelery.backends.database.DatabaseBackend'
#CELERY_REDIS_MAX_CONNECTIONS = 6
#CELERY_ALWAYS_EAGER = False
Can some one help why it is not putting the result in the queue?
This is a issue which is happening quite common now.
setting CELERY_ALWAYS_EAGER to TRUE will do the work
However this is not the best solution in production scenario.
I am integrating
rails application to Quickbooks online
using API.
Use gem Quickbooks-ruby
But want to add discount, add taxes into invoice but not success even not found how to pass in API.
invoice = Quickbooks::Model::Invoice.new
invoice.customer_id = 1
invoice.txn_date = Date.civil(2014, 3, 27)
invoice.doc_number = "001"
transaction_tax = Quickbooks::Model::TransactionTaxDetail.new
# Point to a saved tax code in QBO, e.g. this points to id = 2,
# which is a NYC tax code saved on QBO account = 10% sales tax
transaction_tax.txn_tax_code_id = 2
transaction_tax.total_tax = 134.10
invoice.txn_tax_detail = transaction_tax
sales_line_item = Quickbooks::Model::InvoiceLineItem.new
sales_line_item.amount = 1490
sales_line_item.description = "CCM ice skates"
sales_line_item.sales_item! do |detail|
detail.unit_price = 149
detail.quantity = 10
detail.item_id = 1 # Item ID here
detail.tax_code_id = 'TAX' # for US must be 'NON' or 'TAX'
end
discount_line_item = Quickbooks::Model::InvoiceLineItem.new
discount_line_item.amount = 149
discount_line_item.discount_item! do |detail|
detail.discount_percent = 10
detail.percent_based = true
detail.discount_account_id = 99
end
invoice.line_items << sales_line_item
invoice.line_items << discount_line_item
service = Quickbooks::Service::Invoice.new
service.access_token = OAuth::AccessToken.new($qb_oauth_consumer, "token", "secret")
service.company_id = "9991111222"
created_invoice = service.create(invoice)
There should be something like JAXB in ruby for object serialization/deserialization.
From the following link, you can download QB endpoint definations and data class defination(as XSD). You need to generate data classes from it.
https://developer.intuit.com/docs/#api/deki/files/2466/v3.1_dataservices.zip
Then using any standard ruby based OAuth lib, you can make call to QB API Endpoints.
You can use the setter methods of the data class( in your case - invoice ) to populate data/to construct the payload. ( I don't have any ready example of this. But I guess it is not hard to find in net)
For doc, you can refer the following two links.
https://developer.intuit.com/docs/0025_quickbooksapi/0050_data_services/030_entity_services_reference/invoice
https://developer.intuit.com/docs/0025_quickbooksapi/0050_data_services/030_entity_services_reference/invoice#DiscountLineDetail
Thanks
I am testing flume to load data into hHase and thinking about parallel data loading with using flume's selector and inteceptor, because of speed gap between source and sink.
So, what I want to do with flume are
creating Event's header with interceptors's regex_extractor type
multiplexing Event with header to more than two channels with selector's multiplexing type
in one source-channel-sink.
and tried configuration as below.
agent.sources = tailsrc
agent.channels = mem1 mem2
agent.sinks = std1 std2
agent.sources.tailsrc.type = exec
agent.sources.tailsrc.command = tail -F /home/flumeuser/test/in.txt
agent.sources.tailsrc.batchSize = 1
agent.sources.tailsrc.interceptors = i1
agent.sources.tailsrc.interceptors.i1.type = regex_extractor
agent.sources.tailsrc.interceptors.i1.regex = ^(\\d)
agent.sources.tailsrc.interceptors.i1.serializers = t1
agent.sources.tailsrc.interceptors.i1.serializers.t1.name = type
agent.sources.tailsrc.selector.type = multiplexing
agent.sources.tailsrc.selector.header = type
agent.sources.tailsrc.selector.mapping.1 = mem1
agent.sources.tailsrc.selector.mapping.2 = mem2
agent.sinks.std1.type = file_roll
agent.sinks.std1.channel = mem1
agent.sinks.std1.batchSize = 1
agent.sinks.std1.sink.directory = /var/log/flumeout/1
agent.sinks.std1.rollInterval = 0
agent.sinks.std2.type = file_roll
agent.sinks.std2.channel = mem2
agent.sinks.std2.batchSize = 1
agent.sinks.std2.sink.directory = /var/log/flumeout/2
agent.sinks.std2.rollInterval = 0
agent.channels.mem1.type = memory
agent.channels.mem1.capacity = 100
agent.channels.mem2.type = memory
agent.channels.mem2.capacity = 100
But, it doesn't work!
when selector part is removed, there are some interceptor debugging message in flume's log.
but when selector and interceptor are together, there are nothing.
Is there any wrong expression or something I missed?
Thanks for reading. :)
I found it.
In the flume log, there are warning message as below.
2013-10-10 16:34:20,514 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSources(FlumeConfiguration.java:571)] Removed tailsrc due to Failed to configure component!
so I had attached below line
agent.sources.tailsrc.channels = mem1 mem2
and then It works!!!!
I have a web service , i add some extra class which have message contract and after that it changed the way we access some of the methods( and i have not added message contract to these classes these are data contracts ), earlier i.e before we could create one object for request and response (like see the Before part) we are creating a single object for OrderStatusResponse Class. But if you see now the After(we have to create separate objects for request and response).
is this a side effect of enabling "Always generate message contract?"
Before
SmartConnect.Service1Client Client =
new SmartConnectClient.SmartConnect.Service1Client();
SmartConnect.OrderStatusResponse Status =
new SmartConnectClient.SmartConnect.OrderStatusResponse();
Status.UserID = "1234";
Status.Password = "abcd";
Status.SoftwareKey = "abc";
Status.OrderNumber = "1234";
Status = Client.GetOrderStatus(Status);
lbl_OS.Text = Status.Status.ToString();
lbl_RM.Text = Status.ReturnMessage.ToString();
After
SmartConnectRepublic.SmartConnectClient SmartClient =
new WCF_Client.SmartConnectRepublic.SmartConnectClient();
//SmartConnectRepublic.OrderStatusResponse Status =
new WCF_Client.SmartConnectRepublic.OrderStatusResponse();
WCF_Client.SmartConnectRepublic.GetOrderStatusRequest request =
new WCF_Client.SmartConnectRepublic.GetOrderStatusRequest();
request.status = new WCF_Client.SmartConnectRepublic.OrderStatusResponse();
request.status.OrderNumber = "1055055";
request.status.UserID = "1234";
request.status.Password = "dfsdfsd";
request.status.SoftwareKey = "sdfsdfsdfs";
WCF_Client.SmartConnectRepublic.GetOrderStatusResponse response =
new WCF_Client.SmartConnectRepublic.GetOrderStatusResponse();
response = SmartClient.GetOrderStatus(request);
lbl_Status.Text = response.GetOrderStatusResult.Status;
lbl_RC.Text = response.GetOrderStatusResult.ReturnCode.ToString();
lbl_RM.Text = response.GetOrderStatusResult.ReturnCode.ToString();
Yes, I suspect it is a difference with using message contracts. You seem to have figured it out, though.