I'm using BigQuery to store logs. I recently realized BigQuery web UI seems to convert number with 16digits into date (like "12/3/2017, 10:00:13 AM") even though actual column type is "string".
Is there any way to stop web UI from doing this?
Data seems fine once I export it into CSV. It's just web UI issue I'm seeing.
After some investigation, it turned out to be the Streak extension which was doing this automatic conversion into timestamp format. One of its features is the following:
Automatically Convert Timestamps to Human Readable Format
AppEngine and BigQuery store timestamps as epoch time which is great for computers and computation, but not so good for a human to read. The extension automatically finds, parses and replaces epoch timestamps with a human friendly date/time rendering.
Related
I’m using Azure storage explorer to find specific files. The files loaded are very old and short by date looks not showing the correct results.
Is there any way in which I can give date range and it will show me the files in between those date range?
Is there any way in which I can give date range and it will show me
the files in between those date range?
Unfortunately no. Azure Blob Storage has very limited server-side filtering capabilities and filtering by date is not one of them. What you have to do is list all blobs in the container and then apply filtering on the client side.
An alternative would be to import this information in an Azure Cognitive Search Index and there you will be able to perform all kinds of filtering. I wrote a blog post about it long time ago that you may find useful: https://gauravmantri.com/2014/08/25/making-azure-blob-storage-searchable-using-azure-search-service/.
Thanks for adding DateTimeInput! I have a bit of a problem with it, though. It converts the entered time into UTC using my local timezone (decreases 2 hours from the entered time). Is there a way to make it simply save the time that's entered?
I didn't find any info about any options this component accepts.
EDIT:
Nobody has any answer to this problem? I'm sure I'm not the only one who wants to be able to store date/time values from the UI "as-is", instead of some automatic conversions.
In our case we enter certain measurement data from different locations. The location of the UI user here is irrelevant and the automatic conversion simply messes things up.
I'm adding some testing to my current project which uses Azure blob storage to store telemetry data coming from a stream analytics job. I want to do testing of the routines that get the telemetry data, so I created a separate container for test data. I downloaded a sample set of data, modified the data to serve my needs and re-uploaded (using Azure storage explorer) everything back into the new container.
The tests were immediately failing and I quickly found out that this is because the LastModified date of the files changed into the date/time of upload. This is fine, but the sequence of the upload was also different. My code uses the modified date of the file to find out which one is the most recent, which would now return a different file based on the new dates.
I found that you cannot modify this property, although you can change another property to have it update. So I know the solution: I could write a quick script which gets the sequence of files from my production instance and then touches every file in the test instance in the same sequence.
But... I was wondering whether this is the best option. I also read it's 'best practice' to store a custom datetime in a separate property, but I don't think I can do that straight from Stream Analytics (which is writing the blobs). I also considered using an Azure Function to do this (new blob => update property), but I'm than adding complexity and something that might fail for whatever reason.
So I'm looking for the best way to solve this problem. Anyone?
Update: this one probably deserves a tiny bit more explanation. Apart from using the LastModified date to sort on, I also use it to filter blobs. The blobs themselves are CSV files containing ASA output data, so telemetry records. Each record has a timestamp, but that information is IN the file. When retrieving data, I don't want to have to dive into each file to find out what the timestamp is of those records. So I use a prefilter to filter out the blobs within a certain timespan, and then only download / open those file to the records inside.
This works perfectly as long as you do not touch any of the blob, but obviously it stops working as soon as any of the blobs gets modified for whatever reason. So I'm now convinced that I need a different / better way to solve this issue; but how?
It seems to me that you have two separate things: the data that you want to store in blob storage and metadata about the blob such as the timestamp. I would create a different (azure) database for the metadata or even simpler just add metadata to the (block)blob:
blockBlob.Metadata.Add("from", dateTime.ToString());
blockBlob.Metadata.Add("to", dateTime.ToString());
blockBlob.Metadata.Add("order", "1");
For sorting I would just add a simple order property.
The comment by #Vignesh deserves the credit here, but in order to get this one marked answer I'll provide it myself.
With ASA, you can set the output to be structured by date/time. That means in this case, data is written to the blob store with a directory structure such as:
2016 / 06 / 27 / 15 / 23 (= 27-06-2016 15:23)
2016 / 06 / 28 / 11 / 02 (= 28-06-2016 11:02)
The ASA output allow you to specify how granular you want the structure to be, in my case I chose to store it by day (so not including a time path). The ASA runtime will now ensure that data from a certain point in time is stored within a blob in that resides in the correct path.
Then I subsequently changed my logic to not use the datetime stamp of the individual blob files any more, but simply read just the files from the folders that are within the timerange I'm interested in. That assures we only get data that was produced within that timerange. And if there's more than one file in a folder, I need to load them both since both were in the same timerange anyway. As long as minutes are enough granularity for you, this works excellent even though it might feel a bit strange to use a folder structure for such a thing.
Having a seperate 'index' for blobs which tracks their datetime would work too of course, but adds complexity which in this case I don't really need.
We're using an Access application that stores and retrieves information from SharePoint, and the times and dates get viewed through the 'filter' of the Time Zone settings on SharePoint.
This has started to cause a problem when just trying to enter a date with no time. People marked as CST will see 7/1/2014 0:00, but those in PST will see 6/30/2014 22:00. Calculations that organise metrics by date would then show the same entry in June for PST users and July for CST.
Is there a way to adjust for this? I don't WANT to be capturing a time in this field, but since it's a Date/Time field on SharePoint, it's attaching a time anyway. Would changing the field in question to 'String' work or would that cause more problems than a more adaptive solution?
(I've read links that popped up in 'Questions that may already have your answer', conducted other searches on and off Stack.)
After a bit of testing after all my Users went home except for one, I found that just changing the type on SharePoint for the fields in question from 'Date/Time' to 'Text (Single Line)' worked perfectly. My Tester had no issues with the VB used to store information and bringing up the raw tables showed the 'proper' date stamps after a restart of the client database.
It seems a little too good to be true, but it appears to be a victory! I'll definitely come back here with an update if anything blows up.
Im using a web service to populate a tableview, but the data in my web database changes very rarely, and i don't need to always download the same data.
I need to storage the data in the device, and verify if the data on the device is different than the one on the web.
What is the best practice to do that? Parse a JSON object, containing the last version and comparing to the local version and then parsing the data if its needed ?
Is there a way to automate the last version?
If you have control over the web APIs then have a separate api that returns a timestamp for the last web data change and check this. If the date is not newer than the date of last download use the local data. imho.
I personally used local storage for the same situation on iOS and it worked great.
You just need to provide an URL which responds with the last version of the data (maybe a date) and then locally in JavaScript you first check if version online is newer than your local stored one, if it is you redownload data and place it into localstorage, otherwise you just keep old data (and don't download anything at all).
Of course everything must be done through Ajax.