I am not using props.conf. So I guess it is the default behavior.
Below is the single log:
2018-07-19 13:30:40.293 +0000 [http8080] INFO RequestFilter- {
"transaction_id" : "aaaaaaaaawwwwwwww",
"http_method" : "POST",
"date_time" : "2018-07-19 13:30:34.694 +0000",
"requestId" : "20180719-dc7bc01d-b02c-43c8-932b-42af542ccefb"
}
But it is coming in 2 events
2018-07-19 13:30:40.293 +0000 [http8080] INFO RequestFilter- {
"transaction_id" : "aaaaaaaaawwwwwwww",
"http_method" : "POST",
And
"date_time" : "2018-07-19 13:30:34.694 +0000",
"requestId" : "20180719-dc7bc01d-b02c-43c8-932b-42af542ccefb"
}
It is always breaking from "date_time"
Any suggestions? how can i fix it?
You will need to adjust your props.conf to change the event break logic. By default, it will break whenever it detects a valid timestamp which suits most log formats.
This regex should match just the initial row:
LINE_BREAKER = \d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}.\d{3}\s+\+\d{4}\s+\[.*\]
SHOULD_LINEMERGE = false
Related
I've been struggling with a problem using kafka connect and the S3 sink.
First the structure:
{
Partition: number
Offset: number
Key: string
Message: json string
Timestamp: timestamp
}
Normally when posting to Kafka, the timestamp should be set by the producer. Unfortunately there seems to be cases where this didn't happen. This means that the Timestamp might sometimes be null
To extract this timestamp the connector was set to the following value:
"timestamp.extractor":"Record".
Now it is always certain that the Message field itself always contains a timestamp as well.
Message:
{
timestamp: "2019-04-02T06:27:02.667Z"
metadata: {
creationTimestamp: "1554186422667"
}
}
The question however is that now, I would like to use that field for the timestamp.extractor
I was thinking that this would suffice, but this doesn't seem to work:
"timestamp.extractor":"RecordField",
"timestamp.field":"message.timestamp",
This results in a NullPointer as well.
Any ideas as to how to use the timestamp from the kafka message payload itself, instead of the default timestamp field that is set for kafka v0.10+
EDIT:
Full config:
{ "name": "<name>",
"config": {
"connector.class":"io.confluent.connect.s3.S3SinkConnector",
"tasks.max":"4",
"topics":"<topic>",
"flush.size":"100",
"s3.bucket.name":"<bucket name>",
"s3.region": "<region>",
"s3.part.size":"<partition size>",
"rotate.schedule.interval.ms":"86400000",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "false",
"value.converter.schemas.enable": "false",
"storage.class":"io.confluent.connect.s3.storage.S3Storage",
"format.class":"io.confluent.connect.s3.format.json.JsonFormat",
"locale":"ENGLISH",
"timezone":"UTC",
"schema.generator.class":"io.confluent.connect.storage.hive.schema.TimeBasedSchemaGenerator",
"partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"partition.duration.ms": "3600000",
"path.format": "'year'=YYYY/'month'=MM/'day'=dd",
"timestamp.extractor":"RecordField",
"timestamp.field":"message.timestamp",
"max.poll.interval.ms": "600000",
"request.timeout.ms": "610000",
"heartbeat.interval.ms": "6000",
"session.timeout.ms": "20000",
"s3.acl.canned":"bucket-owner-full-control"
}
}
EDIT 2:
Kafka message payload structure:
{
"reference": "",
"clientId": "",
"gid": "",
"timestamp": "2019-03-19T15:27:55.526Z",
}
EDIT 3:
{
"transforms": "convert_op_creationDateTime",
"transforms.convert_op_creationDateTime.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.convert_op_creationDateTime.target.type": "Timestamp",
"transforms.convert_op_creationDateTime.field": "timestamp",
"transforms.convert_op_creationDateTime.format": "yyyy-MM-dd'T'HH:mm:ss.SSSXXX"
}
So I tried doing a transform on the object, but it seems like I've been stuck again on this thing. The pattern seems to be invalid. Looking around the internet it does seem like this is a valid SimpleDatePattern. It seems to be complaining about the 'T'. Updated the message schema as well.
Based on the schema you've shared, you should be setting:
"timestamp.extractor":"RecordField",
"timestamp.field":"timestamp",
i.e. no message prefix to the timestamp field name.
If the data is a string, then Connect will try to parse as milliseconds - source code here.
In any case, message.timestamp assumes the data looks like { "message" : { "timestamp": ... } }, so just timestamp would be correct. And having nested fields didn't use to be possible anyway, so you might want to clarify which version of Connect you have.
I'm not entirely sure how you would get instanceof Date to evalutate to true when using JSON Converter, and even if you had set schema.enable = true, then also in the code, you can see there is only conditions for schema types of numbers and strings, but still assumes that it is milliseconds.
You can try using the TimestampConverter transformation to convert your date string.
I am writing a script in python to create an event.Newbie at this.Below is the entire script code for reference.
import requests,json
from datetime import datetime
event={}
event['name']={}
print("Enter the following:\ntitle of event")
event['name']['html']=input()
event['description']={}
event['description']['html']=input("Event description:\n")
event['start']={}
startdate=datetime.strptime(input("start datetime eg :Jun 1 2005 1:33PM :\n"),'%b %d %Y %I:%M%p')
event['start']['utc']=str(startdate.date())+'T'+str(startdate.time())+'Z'
#event['start']['utc']=startdate.isoformat()+"Z"
#Turning datetime in YYYY-MM-DDThh:mm:ssZ format
event['start']['timezone']=input("timezone eg Asia/kolkata\n")
event['end']={}
enddate=datetime.strptime(input("end datetime eg :Jun 1 2005 1:33PM\n"),'%b %d %Y %I:%M%p')
event['end']['utc']=str(enddate.date())+'T'+str(enddate.time())+'Z'
#event['end']['utc']=enddate.isoformat()+"Z"
event['end']['timezone']=event['start']['timezone']
event['currency']=input("3 letter code")
response = requests.post("https://www.eventbriteapi.com/v3/events/",
headers = {
"Authorization": "Bearer NC.....",
"Content-Type" : "application/json"
},
data=json.dumps({"event":event}),
verify = True, # Verify SSL certificate
)
Strictly followed the docs https://www.eventbrite.com/developer/v3/endpoints/events/
According to docs datatype of event.start.utc and event.start.end data shall be datetime i.e "2010-01-31T13:00:00Z"
We can see in the comments I also tried with isoformat function.
On printing event object I found the same format as specified in the docs.
But receiving response.content as event.start invalid argument or event.start.utc as datetime wrong format use instead "YYYY-MM-DDThh:mm:ssZ" !!
I ran into date issues with Eventbrite too; after debugging found this to work for events:
{ event:
{ name: { html: 'Postman API Event!' },
description:
{ html: 'My fav event is The Winter Formal.' },
start: { timezone: 'America/Los_Angeles', utc: '2018-09-06T00:19:53Z' },
currency: 'USD',
listed: false,
end: { timezone: 'America/Los_Angeles', utc: '2018-09-06T00:20:53Z' } } }
and of course, different date formats in another part of API ;-)
Regarding discounts / cross event discount, where the docs specify a
Naive Local ISO8601 date and time format
First I've heard of the 'Naive' format? Why no example Eventbrite? Here's what worked:
2018-10-11T12:13:14
I am having trouble to get my filter of a defect store working. I am trying to to get all the defect that are created after a specific date( Release.ReleaseDate ).
_getFilters: function(startDate) {
var dateFilter = Ext.create('Rally.data.wsapi.Filter', {
property: 'CreationDate',
operator: '>',
value: startDate
});
console.log('startDate is: ', startDate); //Sat Aug 23 2014 02:59:59 GMT-0400 (Eastern Daylight Time)
console.log(dateFilter.toString());
return dateFilter;
},
With the above filter, I always get an empty result, even thought I have verified that there are indeed some defects that are created after the startDate.
Note: If I remove the filter in the store config, I do see a list of defects.
Am I missing anything?
Thanks
The date has to be converted to ISO format. Assuming you got the release object:
var releaseStartDate = release.get('ReleaseStartDate');
var releaseStartDateISO = Rally.util.DateTime.toIsoString(releaseStartDate,true);
your filter will include this:
{
property : 'CreationDate',
operator : '>',
value : releaseStartDateISO
}
A full example is in this github repo.
It turns out that I need to 'reformat' the startDate to the following form before passing it to the filter.
YYYY-MM-DD
I'm trying to configure logstash with redis and elasticsearch.
I have a problem with the #timestamp field.
The value of #timestamp is always the real event timestamp -2 hrs.
I have a shipper configured like this :
input{ file {...}}
filter{
if [type]=="apachelogs"{
grok{
match => [ "message", "%{COMBINEDAPACHELOG}"]
}
date {
locale => "en"
timezone => "Europe/Brussels"
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
output{ redis{...}}
and a logstash-indexer like this :
input{ redis {...}}
output { elasticsearch {...}}
The result of an event in ES looks like this :
"#timestamp": "2014-05-21T13:29:53.000Z"
...
"timestamp": "21/May/2014:15:29:53 +0200"
So as you can see there is always a 2hrs offset in the #timestamp and I can't figure out why.
I've tried different things such as changing the timezone etc. without success.
Any idea about this one ?
Thanks
You can use this filter to change timezone.
Change
"#timestamp": "2014-04-23T13:40:29.000Z"
to
"#timestamp": "2014-04-23T15:40:29.000+0200"
Try to use this filter
filter {
ruby {
code => "
event['#timestamp'] = event['#timestamp'].localtime('+02:00')
"
}
}
Hope this can help you.
timezone should be work.
What's wrong with your result? Values in following two fields indicate the same time point.
"#timestamp": "2014-05-21T13:29:53.000Z"
"timestamp": "21/May/2014:15:29:53 +0200"
Where Z stands for +0000.
I have setup a test where I try to PUT an item after fetching it. It fails on the dates when PUT'ing though. The date fields below use DS.attr('date').
Versions:
Ember : 1.1.1
Ember Data : 1.0.0-beta.4+canary.c15b8f80
Handlebars : 1.0.0
jQuery : 1.9.1
Here's my code:
BuildingsController
App.BuildingsController = Ember.Controller.extend({
actions: {
createBuilding: function() {
var store = this.get('store');
store.find('building', 1729).then(function(building) {
building.set('title', 'Test 123');
building.save();
});
}
}
});
Data returned from API when calling store.find:
{
"building":{
"id":"1729",
"name":"Test 123",
"sort":"1",
"published_at":"2013-09-26 11:00:27",
"source":"source test",
"content":"<p>content test<\/p>",
"excerpt":"<p>excerpt test<\/p>",
"lat":"62.39039989300704",
"lon":"17.341790199279785",
"address":"address",
"build_start":"2013-09-22",
"build_end":"2013-09-23",
"created_at":"2013-09-26 11:00:28",
"updated_at":"2013-09-26 11:00:28"
}
}
Data PUT to API:
{
"address" : "address",
"build_end" : "Mon, 23 Sep 2013 00:00:00 GMT",
"build_start" : "Sun, 22 Sep 2013 00:00:00 GMT",
"content" : "<p>content test</p>",
"created_at" : "undefined, NaN undefined NaN NaN:NaN:NaN GMT",
"excerpt" : "<p>excerpt test</p>",
"lat" : 62.39039989300704,
"lon" : 17.341790199279785,
"name" : "Test 123",
"published_at" : "undefined, NaN undefined NaN NaN:NaN:NaN GMT",
"sort" : 1,
"source" : "source test",
"updated_at" : "undefined, NaN undefined NaN NaN:NaN:NaN GMT"
}
You can use a custom attribute type (rather than DS.attr('date')) by registering a custom transform and handle serialization / deserialization manually:
DS.RESTAdapter.registerTransform("mydate", {
deserialize: function(serialized) {
// deserialize date here
return deserialized;
},
serialize: function(deserialized) {
// serialize date here
return serialized;
}
});
As described in this answer to What is the best way to modify the date format when ember-data does serialization?
I've been working with other projects and now I updated Ember from 1.1.1 to 1.1.2 which seem to have magically solved this problem. I'm guessing the update didn't have anything to do with my problem, though.
Thanks for taking your time.
// Stefan