I'm trying to pass a DateTime values from a Python (jython) component to a Python (python 3) component but it doesn't seem to work
There is a Matillion Job Variable myvar with type DateTime and default value 2028-01-28 12:00:00
The Jython code is as follow:
from dateutil.parser import parse
from datetime import timedelta
print('class', myvar.getClass())
myvar = parse(str(myvar.toInstant())).replace(tzinfo=None)
print('myvar', repr(myvar))
myvar2 = myvar + timedelta(days=1)
print('myvar2', repr(myvar2))
context.updateVariable('myvar', myvar2)
print('myvar', repr(myvar))
I can see already that myvar has the wrong type (str) the end of the Jython execution
('class', <type 'com.matillion.bi.emerald.server.scripting.MatillionDate'>)
('myvar', 'datetime.datetime(2028, 1, 28, 12, 0)')
('myvar2', 'datetime.datetime(2028, 1, 29, 12, 0)')
('myvar', u'2028-01-29 12:00:00.0')
I assume that I can't update the variable passing a datetime instance. I guess I must convert the datetime back to a com.matillion.bi.emerald.server.scripting.MatillionDate since that seems to be the type of myvar inside Jython.
The question is how do I convert my datetime to a MatillionDate?
I think you're seeing the string value because you're using the repr() function of the variable.
I've taken your example and added a few additional lines to show what I mean.
from datetime import datetime
print('myvar', type(myvar))
context.updateVariable('myvar', datetime.now())
print('myvar', myvar)
print('myvar', type(myvar))
You can see that the 'type' is actually a Java Timestamp, which makes sense because Jython executes in the JVM. You can update the variable with a Python datetime just as easily where the resulting type is also a Java Timestamp.
To answer your question though, if you set the Matillion variable to the value of str(myvar2), you should be able to access the same/updated value in the Python3 component afterwards (as a datetime.datetime object), without the necessity of creating another MatillionDate value. This would just require yet another type conversion when consuming that value from Python3.
Related
I have a csv file where the a timestamp column is coming with values in the following format: 2022-05-12T07:09:33.727-07:00
When I try something like:
df['timestamp'] = pd.to_datetime(df['timestamp'])
It seems to fail silently as the dtype of that column is still object. I am wondering how I can parse such a value.
Also, what is the strategy so that it remains robust to a variety of input time formats?
final_month = pd.Timestamp('2018-02-01')
df_final_month = df[df['week'] >= final_month]
df_final_month.iloc[:, 1:].sum().argmax()
index = df.set_index('week')
index['storeC'].argmin()
the code above is correct, i just don't exactly understand how does it work inside. i have some questions:
1.the type(week) is datetime, the reason why set final_month as Timestamp is that the datetime is almost as same as Timestamp, they recognise each other in Python?
2.about the argmax(), and argmin(), for the df_final_month.iloc[:, 1:].sum().argmax(), i removed sum() and tried like df_final_month.iloc[:, 1:].argmax(), it returns
`AttributeError: 'DataFrame' object has no attribute 'argmax'`
why is it? why the second code doesn't need a max() or something to call argmin(), what's the requirement for using argmin()/argmax() ?
please explaining the details of how python or pandas deal with these data, the more detail the better.
thanks!
i am new in Python.
Is Timestamp almost as same as datetime?
Here is quote from pandas documentation itself:
TimeStamp is the pandas equivalent of python’s Datetime and is interchangable with it in most cases
In fact, if you look at source code of pandas you will see that Timestamp actually inherits from datetime. Here is code to check these statements are true:
dt = datetime.datetime(2018, 1, 1)
ts = pd.Timestamp('2018-01-01')
dt == ts # True
isinstance(ts, datetime.datetime) # True
Why calling argmax method on DataFrame, without calling sum throws an error?
Because DataFrame object doesn't have argmax method, only Series do. And sum, in your case, returns a Series instance.
I'm doing some ETL from a CSV file in GCS to BQ, everything works fine, except for dates. The field name in my table is TEST_TIME and the type is DATE, so in the TableRow I tried passing a java.util.Date, a com.google.api.client.util.DateTime, a String, a Long value with the number of seconds, but none worked.
I got error messages like these:
Could not convert non-string JSON value to DATE type. Field: TEST_TIME; Value: ...
When using DateTime I got this error:
JSON object specified for non-record field: TEST_TIME.
//tableRow.set("TEST_TIME", date);
//tableRow.set("TEST_TIME", new DateTime(date));
//tableRow.set("TEST_TIME", date.getTime()/1000);
//tableRow.set("TEST_TIME", dateFormatter.format(date)); //e.g. 05/06/2016
I think that you're expected to pass a String in the format YYYY-MM-DD, which is similar to if you were using the REST API directly with JSON. Try this:
tableRow.set("TEST_TIME", "2017-04-06");
If that works, then you can convert the actual date that you have to that format and it should also work.
While working with google cloud dataflow, I used a wrapper from Google for timestamp - com.google.api.client.util.DateTime.
This worked for me while inserting rows into Big Query tables. So, instead of
tableRow.set("TEST_TIME" , "2017-04-07");
I would recommend
tableRow.set("TEST_TIME" , new DateTime(new Date()));
I find this to be a lot cleaner than passing timestamp as a string.
Using the Java class com.google.api.services.bigquery.model.TableRow, to set milliseconds since UTC into a BigQuery TIMESTAMP do this:
tableRow.set("timestamp", millisecondsSinceUTC / 1000.0d);
tableRow.set() expects a floating point number representing seconds since UTC with up to microsecond precision.
Very non-standard and undocumented (set() boxes the value in an object, so it's unclear what data types set() accepts. The other proposed solution of using com.google.api.client.util.DateTime did not work for me.)
I put the data into my QTableWidget tableView using a loop:
for i in range(0, len(res)):
self.tableView.setItem(i, 2, QTableWidgetItem(str(create_dataframe(res)[2][i])))
self.tableView.setItem(i, 3, QTableWidgetItem(str(create_dataframe(res)[3][i])))
where create_dataframe(res)[2][i] returns value of class 'int' and create_dataframe(res)[3][i] returns value of class 'datetime.datetime' (like '2017-03-25 16:51:24'). The question is: how do I make these items properly sortable through self.tableView.setSortingEnabled(True), i.e. not as strings, but as integers and datetime respectively? I know that I should use setData and Qt.DisplayRole, but could you please give an example using this short piece of code?
Thank you.
OK, here is the answer I came up with by myself:
self.tableView.setItem(i, 2, QTableWidgetItem(str(create_dataframe(res)[2][i])))
it3 = QTableWidgetItem()
it3.setData(Qt.EditRole, QVariant(create_dataframe(res)[3][i]))
self.tableView.setItem(i, 3, it3)
I.e. there is no need to transform datetime value: it can be properly sorted in string form. As for integer values, I have to create an instance of QTableWidgetItem(), then use .setData with QVariant on it. After this I can setItem to the table.
How to get the current time with GHCJS? Should i try to access Date or use Haskell base libraries? Is there an utility function somewhere in the GHCJS base libraries?
The Data.Time.Clock module seems to work well:
import Data.Time.Clock (getCurrentTime)
import Data.Time.Format -- Show instance
main = do
now <- getCurrentTime
print now
The solution i found currently is quite ugly, but it works for me, so maybe it can save some time to somebody:
{-# LANGUAGE JavaScriptFFI #-}
import GHCJS.Types( JSVal )
import GHCJS.Prim( fromJSString )
foreign import javascript unsafe "Date.now()+''" dateNow :: IO (JSVal)
asInteger = read (fromJSString dateNow) :: Integer -- this happens in IO
The ugliness comes from not finding a JSInteger type in GHCJS, which would be needed in order to get the result of Date.now() which is a long integer. So i need to produce a string concatenating a string to the result of Date.now() in Javascript. At this point i could get a JSString as result, but that would not be an instance of Read so using read would not work. So i get a JSValue and convert it to String using fromJSString.
Eventually there might be a JSInteger in GHCJS, or JSString might become an instance of Read, so if you are reading this from the future try out something more elegant!