I have indexed data on splunk but i can see the _time(indexed time) is showing wrong like.
I had indexed this data on 19th oct but this is showing like it is indexed on 18th oct.
Please suggest what would be the solution or i need to manually overwrite the _time key with current date time.
Thanks
_time is not the time the event was indexed - that's _index_time. _time is the time the event happened, which usually is different from when it was indexed (because of transport/processing delays).
From your screenshot I see what I presume is the event time ('date' field) differs from _time. That often happens when the time zone is incorrect or is not interpreted correctly. Were that the case here, however, I would expect the difference between date and _time to be a multiple of 30 minutes.
From what I see in the question, it's possible the props.conf settings are causing Splunk to interpret the wrong field as _time. Closer inspection shows the sourcetype ends with "too_small". This is an indication that Splunk does not have specific settings for the sourcetype so it's trying to guess at where the timestamp is (and getting it wrong, obviously).
The solution is to create a props.conf stanza for the sourcetype. It should be something like this:
[json]
TIME_PREFIX = date:
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z
MAX_TIMESTAMP_LOOKAHEAD = 26
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 10000
Put this settings on your indexer and restart it. Events that arrive after that should have the right time on them.
I'm trying to find a simple yet robust way to convert time between arbitrary time zones.
This: http://www.cpearson.com/excel/TimeZoneAndDaylightTime.aspx explains only how to convert betwen my (current) TZ and another TZ.
Those two SO articles (Getting Windows Time Zone Information (C++/MFC) and How do you get info for an arbitrary time zone in Windows?) talk about getting the information from the registry.
That sounds a bit too convoluted and time-consuming; moreover, it appears that Windows stores TZs in their "full names" (such as (UTC-08:00) Pacific Time (US & Canada)) and I'd rather refer to TZs using abbreviations (such as EDT). Moreover, relying on Windows registry could also be unsafe: different users might have different versions and some might not be up to date. That would mean a report run by two persons might provide two different results!
Is there a simpler way that will also be robust? Writing a lookup table could work for some time but then it will be broken when a government decides to abolish DST or change anything else.
Maybe get a list of TZs from Internet and parse it? Would that be safe enough?
Update 1
I've made my research and explored the possibilities, but this problem is not as trivial as it might seem. If you think that the function shall look like bTime = aTime + 3, then please reconsider. Timezones and DSTs are in a state of constant flux.
Read this for reference: list of pending / proposed timezone changes. Note that some countries are actually changing their timezones, not just DST settings! And Brazil changed the date on which they change their clocks to winter time! A static lookup table would be broken very quickly by all those changes.
Update 2
I'm not looking into a quick and dirty hack, I can come up with that myself. I'm not wanting to write something and forget about it; I'd like to create a function once that could be safely used by other people for different internal projects without the maintenance nightmare. Hard-coding constants that are known to change once in a while is a very bad software design (think Y2K bug caused by a very, very old piece of code).
Update 3
This database looks good (although I'm not sure if it's stable enough): https://timezonedb.com/api. They even have a TZ conversion call - exactly what I need! I will probably try to parse XML from VBA and share my results.
The API at https://timezonedb.com/references/convert-time-zone is indeed a great place to get the correct worldwide time, timezone, and timezone-offset between two locations, taking into account past/future Daylight Savings changes.
A problem with your suggested method of specifying only the Time Zone Abbreviations (such as "convert PST to EST") is that this API takes your zones literally, even if they are incorrect.
So, if Toronto is currently on EDT but you specify EST, you'll probably get the incorrect time. Using "full names" like (UTC-08:00) Pacific Time (US & Canada) would have the same issue.
A way around that is to specify the time zone names like America/Vancouver (as listed here), or else specify the city, country and/or region name with the appropriate parameters.
I wrote a function to figure it out but it only applies to certain countries (see further down).
What time was it in Toronto last Halloween at 11:11pm Vancouver time?
http://api.timezonedb.com/v2/convert-time-zone?key=94RKE4SAXH67&from=America/Vancouver&to=America/Toronto&time=1509516660
Result: (Default is XML but JSON is also available.)
<result>
<status>OK</status>
<message/>
<fromZoneName>America/Vancouver</fromZoneName>
<fromAbbreviation>PDT</fromAbbreviation>
<fromTimestamp>1509516660</fromTimestamp>
<toZoneName>America/Toronto</toZoneName>
<toAbbreviation>EDT</toAbbreviation>
<toTimestamp>1509527460</toTimestamp>
<offset>10800</offset>
</result>
Getting the data programmatically:
There are plenty of options and lookup methods you will have to decide upon, but here's one example using a VBA Function:
What will be the time difference between Vancouver & Berlin on Christmas Day?
Input Time: 2018-12-25 00:00:00 = Vancouver Local Unix time 1545724800
Function GetTimeZoneOffsetHours(fromZone As String, _
toZone As String, UnixTime As Long) As Single
Const key = "94RKE4SAXH67"
Const returnField = "<offset>"
Dim HTML As String, URL As String
Dim XML As String, pStart As Long, pStop As Long
URL = "http://api.timezonedb.com/v2/convert-time-zone?key=" & key & _
"&from=" & fromZone & "&to=" & toZone & "&time=" & UnixTime
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", URL, False
.Send
XML = .ResponseText
End With
pStart = InStr(XML, returnField)
If pStart = 0 Then
MsgBox "Something went wrong!"
Exit Function
End If
pStart = pStart + Len(returnField) + 1
pStop = InStr(pStart, XML, "</") - 1
GetTimeZoneOffsetHours = Val(Mid(XML, pStart, pStop - pStart)) / 60
End Function
Sub testTZ()
Debug.Print "Time Zone Offset (Vancouver to Berlin) = " & _
GetTimeZoneOffsetHours("America/Vancouver", _
"Europe/Berlin", 1545724800) & " hours"
End Sub
Unix/UTC Timestamps:
Unix time is defined as "the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970."
You can convert times between Unix and/or UTC or Local time at: epochconverter.com ... the site also has conversion formulas for several programming languages.
For example, the formua to convert Unix time to GMT/UTC in Excel is:
=(A1 / 86400) + 25569
You could also download static files (in SQL or CSV format) here instead of caling the API, and the page also has sample queries. However use caution: it's easier to make mistakes with Daylight Savings (as mentioned above).
I made a dummy account to get the "demo" used in the examples, but you should get your own (free) key for long-term use. (I'm not responsible if it gets locked out from over-use!)
An good alternative Time Zone API is Google Maps Time Zone API. The difference is that you specify Latitude & Longitude. It seems to work just fine without a key You'll need to register for a key.
What will the Time Zone Offset be on June 1st at the White House?
https://maps.googleapis.com/maps/api/timezone/json?location=38.8976,-77.0365×tamp=1527811200&key={YourKeyHere}
Result:
{
"dstOffset" : 0,
"rawOffset" : -18000,
"status" : "OK",
"timeZoneId" : "America/Toronto",
"timeZoneName" : "Eastern Standard Time"
}
The Offset will be -18000 seconds (-5 hours).
Determining when Daylight Savings is in effect
Below is a function I put together so I could "trust" the Daylight Savings (DST) values I was getting from a different API, however (as discussed by others) the rules have no pattern plus are constantly changing country by country, even town by town in some parts of the world, so this only will work in countries where:
DST begins on the Second Sunday of March every year
DST end on the First Sunday of November every year
The applicable countries are Bahamas, Bermuda, Canada, Cuba, Haiti, St. Pierre & United States. (Source: Daylight saving time by country**)
Function IsDST(dateTime As Date) As Boolean
'Returns TRUE if Daylight Savings is in effect during the [dateTime]
'DST Start (adjust clocks forward) Second Sunday March at 02:00am
'DST end (adjust clocks backward) First Sunday November at 02:00am
Dim DSTStart As Date, DSTstop As Date
DSTStart = DateSerial(Year(dateTime), 3, _
(14 - Weekday(DateSerial(Year(dateTime), 3, 1), 3))) + (2 / 24)
DSTstop = DateSerial(Year(dateTime), 11, _
(7 - Weekday(DateSerial(Year(dateTime), 11, 1), 3))) + (2 / 24)
IsDST = (dateTime >= DSTStart) And (dateTime < DSTstop)
End Function
And a couple examples of how I could use function IsDST*:
Public Function UTCtoPST(utcDateTime As Date) As Date
'Example for 'PST' time zone, where Offset = -7 during DST, otherwise if -8
If IsDST(utcDateTime) Then
UTCtoPST = utcDateTime - (7 / 24)
Else
UTCtoPST = utcDateTime - (8 / 24)
End If
End Function
Function UTCtimestampMStoPST(ByVal ts As String) As Date
'Example for 'PST', to convert a UTC Unix Time Stamp to 'PST' Time Zone
UTCtimestampMStoPST = UTCtoPST((CLng(Left(ts, 10)) / 86400) + 25569)
End Function
* Note that function IsDST is incomplete: It does not take into account the hours just before/after IsDST takes actually effect at 2am. Specifically when, in spring, the clock jumps forward from the last instant of 01:59 standard time to 03:00 DST and that day has 23 hours, whereas in autumn the clock jumps backward from the last instant of 01:59 DST to 01:00 standard time, repeating that hour, and that day has 25 hours ...but, if someone wants to add that functionality to update the function, feel free! I was having trouble wrapping my head around that last part, and didn't immediately need that level of detail, but I'm sure others would appreciate it!
Finally one more alternative is an API that I use to for polling current/future/historical weather data for various purposes — and also happens to provide Timezone Offset — is DarkSky.
It queries by latitude/longitude and is free (up to 1000 calls/day) and provides "ultra-accurate weather data" (more-so in the USA, where it predicts weather down to the minute and to the square-yard! — but quite accurate I've seen for the unpredictable Canadian West Coast Canada!)
Response is in JSON only, and the very last line is Time Zone Offset versus UTC/GMT time.
DarkSky Sample Call:
https://api.darksky.net/forecast/85b57f827eb89bf903b3a796ef53733c/40.70893,-74.00662
It says it's supposed to rain for the next 60 hours at Stack Overflow's Head Office. ☂
...but I dunno, it looks like a pretty nice day so far! ☀
(flag)
Im afraid anything to do with timezones is never a simple task (ask any web designer and they will say it is a massive challenge)
there are 2 ways to solve your problem
1) The Easy way - Create a central list which all other workbooks are linked to. This can be saved on SharePoint or on a shared drive, then all you have to do is update this one table
2) The hard way - Use a website API to get the latest timezone data. https://www.amdoren.com/ is a good site, you can get a free API key by signing up. The only issue is you then have to parse the Json file from the website. This isn't easy but if you google "vba parse json" you will find some solutions (it generally requires importing some libraries and using other peoples code as a starting point)
Hope you find the right solution, and if you do might be worth sharing it as im sure there will be others with same issue.
While searching for items in the inbox that have been received after a particular time frame (as mentioned in the code below). It searches for the date but it is also returning the email with the specified timestamp. I want the emails only after the specified timestamp.
SearchFilter greaterthanfilter = new SearchFilter.IsGreaterThan(ItemSchema.DateTimeReceived,
Convert.ToDateTime(lastUploadedEmailtimeStamp));
mailItems = inbox.FindItems(greaterthanfilter, view);
Not sure if anyone has faced any similar issues? Basically I want to search for items that were received after a particular mm/dd/yyyy hh:mm:ss.
Exchange stores the datetimes with a precision down to the Millisecond, EWS only give you a precision on datetimes to the second however the Searchfilters do have a precision of milliseconds with Date time. So if you datetime stamps your using only have a precision of seconds then you need to use something like this eg where you wanted all email that was received after 7:43 and 8 seconds
SearchFilter sfs = new SearchFilter.IsGreaterThan(ItemSchema.DateTimeReceived, DateTime.ParseExact("2014/12/29 07:43:08.999", "yyyy/MM/dd HH:mm:ss.fff", null));
FindItemsResults<Item> femaa = service.FindItems(WellKnownFolderName.Inbox,sfs, iItemView);
If you want to look at the actual precision on your messages you need to use a MAPI editor like OutlookSpy of MFCMapi. You can then look at the PT_Systime value which are FileTime "8 bytes; a 64-bit integer representing the number of 100-nanosecond intervals since January 1, 1601" see http://msdn.microsoft.com/en-us/library/ee157583(v=EXCHG.80).aspx
Cheers
Glen
I would like to display the user's time zone.
#user.time_zone will display the city name "Brussels"
How could I get the full time zone with GMT like this :
(GMT+01:00) Brussels
I finally found the trick
ActiveSupport::TimeZone[#user.time_zone]
display the full user time zone
(GMT+01:00) Brussels
I've participated the Rails Rumble this year and I found myself tangled by time zone issues.
I uploaded my app in linode and I live in EST time zone or -4 to UTC.
I have a model and it saves things by doing the following:
def self.processing_creation(user_id, home_id, chore_id)
registered_chore = RegisteredChore.where("DATE(created_at) = ?", Date.today).find_by_user_id_and_home_id_and_chore_id(user_id, home_id, chore_id)
unless registered_chore
registered_chore = RegisteredChore.create(user_id: user_id, home_id: home_id, chore_id: chore_id, created_at: Time.zone.now)
user = registered_chore.user
user.add_one_chore_point
end
registered_chore
end
RegisteredChore.where("DATE(created_at) = ?", Date.today).find_by_user_id_and_home_id_and_chore_id(user_id, home_id, chore_id)
returns false even if the data was just created.
I noticed that "create" saves in UTC but Date.today uses user's local time zone.
What's the best way to handle this?
Another example to illustrate the issue:
I want to register a chore at 11 PM in EST time.
The server is already in the next day (e.g: 12).
Rails saves the data in the next day date (12).
But user is still in day 11.
Technically, user with the current method, could save the entry again because the days are different from the db and user interface.
How to solve this?
Rails will, by default, save times to the database as UTC. If you're using DATE([Date.today]), then it's going to be looking up records on the 11th, not the 12th. To get the right date, you probably want to set Time.zone to the user's current timezone, and then do the query.
I built a gem called by_star to handle this kind of date searching, and I reckon you should use it. With it, your query would be this:
RegisteredChore.today.where(:user_id => user.id,
:home_id => home.id,
:chore_id => chore.id)
Rails 3 by default saves all the time in database relative in GMT+00:00 time zone. So you will have to set an offset while computing time depending on your timezone. Else, you can change default Activerecord time zone by adding following to application.rb
config.time_zone = 'Your time zone' (Example: 'Eastern Time (US & Canada)')
config.active_record.default_timezone = 'Your time zone' (Example: 'Eastern Time (US & Canada)')