I am creating a web app in which there is an analytical dashboard that is allowed to generate a query. This query will be forward to backend server through an exposed API. The controller (written in Node.js) behind this API will execute this query on AWS Athena to fetch the required data.
Now the problem is that how should I send query to backend server. Should I use JSON format? Then at backend How should I convert JSON to SQL Query? Do I need to write custom solution or is there any supported library available?
Is there any better way of doing this?
I have tried some JavaScript libraries like JSON-SQL, JSON-SQL-Builder2 but these doesn't support the format of Query that will be executed by Athena. Athena uses Presto engine to run a query.
Good evening,
If your problem is sending a query to the database from Node, then it seems like the AWS SDK for JavaScript in Node.js.
Your workflow will likely be something like:
Start Query Execution
Get Query Execution
Get Data from S3
Related
I want to build a small api for an existing MS Sql server in my local network. I am not planning this api to do INSERT, UPDATE query on the sql server. Just a small SELECT query. Mostly for counting the rows in a table for a date range.
I don't want the api to create any default tables in the database either while setting up. Just to perform a select query on get request. The sql raw query is just fine for me.
Which APIs should I look for this purpose?
My intention is to get live production from the server and display those in mobile/desktop app.
Currently I have achieved this using pyodbc package in python and I can make a tkinter app out of it. But I want more advanced setup for a flutter app. So building an api is much better I suppose.
I need to understand the below:
1.) How does one BigQuery connect to another BigQuery and apply some logic and create another BigQuery. For e.g if i have a ETL tool like Data Stage and we have some data been uploaded for us to consume in form of a BigQuery. So in DataStage or using any other technology how do i design the job so that the source is one BQ and the Target is another BQ.
2.) I want to achieve like my input will be a VIEW (BigQuery) and then need to run some logic on the BigQuery View and then load into another BigQuery view.
3.) What is the technology used to connected one BigQuery to another BigQuery is it https or any other technology.
Thanks
If you have a large amount of data to process (many GB), you should do the transformation of the data directly in the Big Query database. It would be very slow to extract all the data, run it through something locally, and send it back. You don't need any outside technology to make one view depend on another view, besides access to the relevant data.
The ideal job design will be an SQL query that Big Query can process. If you are trying to link tables/views across different projects then the source BQ table must be listed in fully-specified form projectName.datasetName.tableName in the FROM clauses of the SQL query. Project names are globally unique in Google Cloud.
Permissions to access the data must be set up correctly. BQ provides fine-grained control over who can access, and it is in the BQ documentation. You can also enable public access to all BQ users if that is appropriate.
Once you have that SQL query, you can create a new view by sending your SQL to Google BigQuery either through the command line (the bq tool), the web console, or an API.
1) You can use BigQuery Connector in DataStage to read and write to bigquery.
2) Bigquery use namespaces in the format project.dataset.table to access tables across projects. This allows you to manipulate your data in GCP as it were in the same database.
To manipulate your data you can use DML or standard SQL.
To execute your queries you can use the GCP Web console or client libraries such as python or java.
3) BigQuery is a RESTful web service and use HTTPS
I want to design Web UI which fetches data from HDFS. I want to generate some reports using this data which is stored in HDFS. I have my own custom reports format. I am writing REST API's to fetch data. But running HIVE queries gives latency issues Hence I want different approach for this, I could think of two.
Using IMPALA to create tables. But I am not sure about REST support for IMPALA.
Using HIVE but instead of MR use SPARK as execution engine. .
spark-job-server provides REST support, and fetch data with SPARK-SQL.
Which of the approach will be suitable or is there any better approach for this?
Please can anyone help as I am very new in this.
I'd prefer to choose impala if latency is the main consideration. It's dedicated to SQL processing on hdfs and does it well. About REST api and the application logic you are achieving, this seems to be a good example
In my use case, ElasticSearch is already configured and has data that can be queried via REST API. I'm wondering if there is a way to write SQL statements that can query this data that ElasticSearch is already configured on.
Example, configure an adapter to ElasticSearch in MS SQLServer and use linkedserver to connect and run normal SQL statements.
I have read about the "river", but it seems to do the opposite of what I'm looking for. Any pointers will really help.
SQL Server is a relational database. It operates with tables in common. Posting requests to some URI is very unusual work for SQL Server. And there is no standard mechanism to do this.
What can you do:
Write a CLR-function to send post-requests
Map result json to some table (it can be difficult, because Elastic Search is document-oriented and SQL Server is not)
So, as for me, this way is very complicated. I advice to use some hand-written service to operate with DB and Elastic Search, and don't try to put all logic to SQL Server.
Something like Elasticsearch-SQL plugin might be of interest to you.
This won't allow you to use it as a linked server in MSSQL, but it will allow whatever application you have, to send SQL queries to the sql API on your ElasticSearch server and get results.
Example:
http://localhost:9200/_sql?sql=select * from indexName limit 10
We have been working on some of the database in CSV format and putting them into Google Fusion Tables. But, now is it possible if we can SQL Server to get the data from the database and output a layer with markers to the user.
We want to avoid ASP.NET and C# if possible. We would prefer JS And PHP, but can't seem to find much help regarding the matter.
Does anyone have input on using SQL Server with Google Maps to plot points based on data in the DB?
I think what you could do is write a PHP file that queries SQL Server and returns a JSON array. You could then use JS to plot your map points.
Take a look at this blog post: SQL Server JSON to Table and Table to JSON.
It talks about SQL Server to JSON. They are using it with PowerShell, but one could use it as a starting point for PHP.