I have hosted my data on S3 Bucket in parquet format and i am trying to access it using Athena. I can see i can successfully access the hosted table. I detected something fishy when i try to access a column "createdon".
createdon is a timestamp column and it reflects same on Athena table, but when i try to query it using the provided SQL below query
SELECT createdon FROM "uat-raw"."opportunity" limit 10;
Unexpected output :
+51140-02-21 19:00:00.000
+51140-02-21 21:46:40.000
+51140-02-22 00:50:00.000
+51140-02-22 03:53:20.000
+51140-02-22 06:56:40.000
+51140-02-22 09:43:20.000
+51140-02-22 12:46:40.000
Expected output:
2019-02-21 19:00:00.000
2019-02-21 21:46:40.000
2019-02-22 00:50:00.000
2019-02-22 03:53:20.000
2019-02-22 06:56:40.000
2019-02-22 09:43:20.000
2019-02-22 12:46:40.000
can any one help me with the same ?? and also i have attached pic for more information.
I expect an SQL query which i can use to query my data on S3 from Athena.
Related
I have some tables on SAP HANA and „create column table“ to combine multiple „raw tables“.
In one table, there are duplicate rows, to be more specific, every Information (column) is the same but the date/time is not.So the source System has this weird habit to create one entry, several times (which is wrong). I do not have the possibility to manipulate data in the source System.
The table looks something like:
Table name: Testsubject_status
Column: Status....info....Timestamp
Test me...............bla.......05.01.2017 05:05:00
Test me...............bla......01.01.2017 11:15:00
Test him………..blub…..01.01.2017 11:17:00
Test her ………..blab.....01.01.2017 11:25:00
Test me ………..bla.......01.01.2017 11:35:00
Test it………......blue......01.01.2017 12:15:00
Test me ………..bla.......07.01.2017 12:15:00
All duplicates after the first entry (date/time whise) should be not considered in the newly created table.
Table name: Testsubject_status_NEW
Column: Status....info....Timestamp
Test me...............bla......01.01.2017 11:15:00
Test him………..blub…..01.01.2017 11:17:00
Test her ………..blab.....01.01.2017 11:25:00
Test it………......blue......01.01.2017 12:15:00
This problem does appear multiple times, not only with Test me.
Is the solution something like:
Select
xxx AS "tri"
yyy AS "tre"
zzz AS "tru"
Case when Testsubject_status.Status Count > 1 Then "take first entry"
From ...
Where …
???
I am glad for every help or advice.
Based on the description it should be sufficient to aggregate for the maximum date:
SELECT tri, tre, tru,
max(timestamp)
FROM
....
That works if the „de-duplication“ indeed should happen based on all remaining columns except timestamp.
My table size is big.I use two query i think that is costly. I need efficient query.
Sample
id qt created_at
--- ---- -----------
1 1 2017-07-28
2 2 2017-07-28
3 3 2017-07-28
4 1 2017-07-29
5 2 2017-07-29
6 3 2017-07-20
And i want to display latest
4 1 2017-07-29
5 2 2017-07-29
I do this using two query. i think it's not efficient. i want efficient query.
my code
$last_time = Table::orderBy('created_at','desc')->first()->created_at;
$latest_records = Table::where('created_at','=',$last_time)->get();
First, make a column Unix Timestamp for created_at!
Please read about Laravel date mutators: https://laravel.com/docs/5.1/eloquent-mutators#date-mutators
By default, timestamps are formatted as 'Y-m-d H:i:s'. If you need to customize the timestamp format, set the $dateFormat property on your model. This property determines how date attributes are stored in the database, as well as their format when the model is serialized to an array or JSON
Also, you can override getDateFormat():
protected function getDateFormat()
{
return 'U';
}
And use this in your migration files:
$table->integer('updated_at');
$table->integer('created_at');
And, if you use soft deletes:
$table->integer('deleted_at');
https://laravel.com/docs/4.2/eloquent#timestamps
Then you can get record like:
$record = DB::table('your_table_name')->orderBy('created_at', 'desc')->get(); dd($record);
Hope this helps you!
My environment:
hadoop 1.0.4
hive 0.12
hbase 0.94.14
presto 0.56
All packages are installed on pseudo machine. The services are not running on localhost but
on the host name with a static IP.
presto conf:
coordinator=false
datasources=jmx,hive
http-server.http.port=8081
presto-metastore.db.type=h2
presto-metastore.db.filename=/root
task.max-memory=1GB
discovery.uri=http://<HOSTNAME>:8081
In presto cli I can get the table in hive successfully:
presto:default> show tables;
Table
-------------------
ht1
k_business_d_
k_os_business_d_
...
tt1_
(11 rows)
Query 20140114_072809_00002_5zhjn, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:11 [11 rows, 291B] [0 rows/s, 26B/s]
but when I try to query data from any table the result always be empty: (no error information)
presto:default> select * from k_business_d_;
key | business | business_name | collect_time | numofalarm | numofhost | test
-----+----------+---------------+--------------+------------+-----------+------
(0 rows)
Query 20140114_072839_00003_5zhjn, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
0:02 [0 rows, 0B] [0 rows/s, 0B/s]
If I executed the same sql in HIVE, the result show there are 1 row in the table.
hive> select * from k_business_d_;
OK
9223370648089975807|2 2 测试机 2014-01-04 00:00:00 NULL 1.0 NULL
Time taken: 2.574 seconds, Fetched: 1 row(s)
Why presto can't fetch from HIVE tables?
It looks like this is an external table that uses HBase via org.apache.hadoop.hive.hbase.HBaseStorageHandler. This is not supported yet, but one mailing list post indicates it might be possible if you copy the appropriate jars to the Hive plugin directory: https://groups.google.com/d/msg/presto-users/U7vx8PhnZAA/9edzcK76tD8J
I need to rearrange rows and columns as per the requirements.
I have data in tables as:
SBU_ID FAC_ID Month Year Venting Combustion Comments
3001 4001 1 2009 5.31 207.11 ABCD
3002 4002 2 2009 15.24 45.12 XYZ
3003 4003 1 2010 8.56 5.00 PQRS
Required format:
Jan-2009 Feb-2009 Jan-2010
SBU_ID FAC_ID Metric Result Comment Result Comment Result Comment
3001 4001 Venting 5.31 ABCD 15.24 XYZ 8.56 PQRS
3001 4001 Combustion 207.11 ABCD 45.12 XYZ 55.00 PQRS
Please advice if this is possible. Thanks.
Google for PIVOT/UNPIVOT functions (if you're talking about MS SQL) or it's analogs for Oracle and other databases.
We don't know if you want to just show the data, or completely transform the table design?
Normally (in first case), you should perform a query (or series of queries) to retrieve particular results from your source data.
Getting the data in required format is not simple using simple query (you don't mention any particular DB Engine). To be honest, I don't even know if it's technically possible to create queries to present columns entries (Venting, Combustion) as rows (series of "group by"s maybe?) without writing a simple application to convert the data.
At SQL Query level don't try to solve this.
Once the query returns all the rows, do this gimmick in your high level code, java or c or what ever.
I have data in an MYSQL database that looks like this:
Project Date Time
A 2009-01-01 15
A 2009-01-02 10
B 2009-01-02 30
A 2009-01-09 15
C 2009-01-07 5
I would like to produce output from this data like this:
Date Project A Time Project B Time Project C Time
2009-01-01 15 0 0
2009-01-02 10 30 0
2009-01-07 15 0 5
Can this be done with an SQL query, or do I need to write an external script to itterate through the DB and organize the output?
(Also, if someone has a better suggestion for a subject line let me know and I'll edit the question; I'm not sure of the proper terms to describe the current and desired formats, which makes searching for this information difficult)
You're looking for pivot / crosstab support. Here is a good link.
http://en.wikibooks.org/wiki/MySQL/Pivot_table
I believe this is called Pivot table. Just google for it.
I've been looking for something like this and found a MySQL stored procedure that works perfectly:
http://forums.mysql.com/read.php?98,7000,250306#msg-250306
The result you're looking for could be obtained from the following simple call:
call pivotwizard('date','project','time','from_table','where_clause')