I have a task to extract data from BigQuery to PostgreSQL and I would like to know if it is possible to do this extraction using Google Dataform. I know the reverse (Postgres to BigQuery) is possible, but what about BigQuery to Postgres?
Thanks
Related
I have hundreds of Hive Queries(HQLs, using Hive functions like date_sub, lead, lag etc) which I need to convert to Redshift Spectrum, is there any tool which helps in this?
is it possible to query in memory arrow table using presto or is there some way to use a pandas data frame as a data source for presto query engine ?
Actually I have parquet files which I want to convert to arrow and query that thorough presto is something like this possible ?
As of now, this is not possible. However, there is a Jira open for this feature request. https://github.com/prestodb/presto/issues/12201
I'm aware that dataflow can use a BigQuery table (or an SQL style query of a BigQuery table) for input, but is it possible to refer to a saved view as input to dataflow?
Yes you can use bigquery legacy view in dataflow but you can't use standard sql view as it is not supported.
My suggestion is that you should use BigQuery job to convert bigquery view to bigquery table then use it into dataflow.
You can read from Bigquery view in Dataflow. Try the Python code below:
beam.io.Read(
beam.io.BigQuerySource(use_standard_sql=True,
query='SELECT * FROM `project.dataset.view` limit 1000000'))
I have a visio .vdx for the design of the my data warehouse with Lucidchart. Is there a way to generate redshift sql from that ?
What would be the best tool to work with Redshift data modeling ?
If those sql generator can generate tables for special visio stencil, like http://www.visualdatavault.com
Amazon Redshift is (mostly) compatible with PostgreSQL, so any tool that can introspect PostgreSQL tables should work with Redshift.
One things to note -- constraints and foreign keys are not enforced in Redshift.
Is it possible to execute Bigquery's query using Google cloud data flow directly and fetch data, not reading data from table then putting conditions?
For example, PCollections res=p.apply(BigqueryIO.execute("Select col1,col2 from publicdata:samples.shakeseare where ...."))
Instead of reinventing using iterative method what Bigquery queries already implemented, we can use the same directly.
Thanks and Regards
Ajay K N
BigQueryIO currently only supports reading from a Table and not a Query or View (FAQ).
One way to work around this is in your main program to create a BigQuery permanent table by issuing a query before you run your Dataflow job. After, your job runs you could delete the table.