I have a Synapse Git Project that has SQL Scripts created in the Azure Portal like so Microsoft Docs SQL Scriptand the challenge is that in GIT they appear as this kinda bulky JSON file and I would love to read it as SQL File DBEAVER or IntelliJ …
Any way to do this without having to manually select the Query Section of the file and kinda clean it?
First Some Background
Synapse stores all artifacts in JSON format. In the Git repo, they are in the following folder structure:
Inside each folder are the JSON files that define the artifacts. Folder sqlscript contains the JSON for SQL Scripts in the following format:
NOTE: the Synapse folder of the script is just a property - this is why all SQL Script names have to be unique across the entire workspace.
Extracting the script
The workspace does allow you to Export SQL to a .sql file:
There are drawbacks: you have to do it manually, 1 file at a time, and you cannot control the output location or SQL file name.
To pull the SQL back out of the JSON, you have to access the properties.content.query property value and save it as a .sql file. As far as I know, there is no built in feature to automatically save a Script as SQL. Simple Copy/Paste doesn't really work because of the \ns.
I think you could automate at least part of this with an Azure DevOps Pipeline (or a GitHub Action). You might need to copy the JSON file out to another location, and then have a process (Data Factory, Azure Function, Logic App, etc.) read the file and extract the query.
Related
I have several packages/projects deployed in the SSISDB catalog. I want to find which packages have a specific expression attached to a variable value. Is there a TSQL way of doing this? I know the package data is encrypted in [internal].[packages]. But with the proper credentials, is it possible to decrypt it? Or what are the other options I have? Can I use some C#/Powershell script to search?
There is no tables that stores this kind of information. You should search for it by reading the package XML. Using SSISDB, it is not possible to read the package XML using T-SQL since SSISDB encrypts the whole project as binary.
To read the package's XML, you should extract the project binary using the SSISDB.cataloag.get_project stored procedure. Change the extract file extension to .zip and extract its content. Then, loop over packages to check if the variable is used in each package. This can be done using C# or PowerShell:
Extracting dtsx from Integration Service Catalog from C#
Get Package XML from SSIS Catalog with PowerShell
In case you are storing the database within SQL Server (Msdb) you can read the package XML data from the msdb.dbo.sysssispackages table.
I'm fairly new to using DBs and have been tasked to create an automated process that uploads Excel-Files into a Oracle Database.
I was told that the User should put the files into a dedicated folder and then a process should automatically upload the files. After checking Stackoverflow and the Internet, however it looks to me like there is no way to do the upload with just PL/SQL.
Do i need to use other external tools to achive this or am i just looking the wrong way?
The reason i want to do it with just PL/SQL is that i don't have sys rights on the server or a way to install any tools right now.
You can upload CSV files stored in a folder using SQL Loader directly into Oracle Tables. But you need to have those files stored as CSV not XLS and transfer them to a folder in a server which contains at least an Oracle client. In this case, your user should save the files as csv, and then you must have a pick up process to move them to a server where you can run the sql loader process.
However, if you want to keep using Excel, and you have no option to move the files, Oracle Application Express which is free and included with Oracle Database contains a plugin to upload directly and automatically excel files into tables. You would have to create a small application in Apex with a page for doing this. It is totally out-of-the-box and quite easy. If you use Apex 18c or higher, it is there. If you use Apex 5.1.4 you need to install a plugin. In this case, the user is responsible to upload the excel file by the web apex application, or you can use the API APEX_DATA_PARSER package for doing so without manual intervention. However, keep in mind that if you use the API, you need to have the files accessible for the database.
Apex Data Parser 19c
Let me know if you have more doubts about it.
Regards
Is there any way possible, to upload a SQL DB file stored locally on disk to Azure SQL Server? Its already populated with data and I'd like to use it for learning the Azure environment and creating web apps alongside it. NOTE: this is not a SQL Server DB file, its just a regular standalone db file with data populated into it.
There's a tool called Data Migration Assistant, you can download here. It's quite intuitive to use.
https://www.microsoft.com/en-us/download/details.aspx?id=53595
I am using Intellij 14.1.4. I am able to run SQLs by custom defined db data sources in database console.
I have some sql files in my project and would like to execute them directly instead of copying them to the database console. However, I am not able to use the same data sources that I created for the console when setting up connections from the DB Connections drop down.
I wonder how to run SQL statements from sql files with the same data sources as the ones I defined in the db console?
Thanks
You can right-click on an sql file in the editor and there is an option "Run myfile.sql"
To run SQL statements from sql files You must associate with file type
see how
I've a list of .sql script files to create Stored Procedures which I'm using the Eclipse DTP to develop. Currently to create/update all these Stored Procedures, I've to open & run
one by one from the Data Perspective.
Is there a way to create a batch file that run the scripts along the lines of
run createSP1.sql
run createSP2.sql
...
run createSPn.sql
and run it in the Eclipse DTP to avail of the DB connection defined there?
why not just create a batch file that merges all of your .sql files together into a single procs.sql file as part of the build process. I don't know what platform you're running on but in Windows you could have a .bat file that does something like this:
type *.sql > proc.sql
then to apply it to the database, why not do it outside Eclipse and connect to the database via the command line. You could bundle this all up as a single batch file that gets the latest version of your stored procedures from source control, merges them into a single file and then applies it to the database.
Part I
As far as I know the developers of Eclipse DTP
have not yet implemented a command line SQL execution
interface through the Eclipse console view.
See the following URL on the eclipse DTP developer forum
http://dev.eclipse.org/newslists/news.eclipse.dtp/msg00304.html
Part II
While the Eclipse DTP people are working on it,
you can use a database specific tool to load
a master SQL file (all SQL proc files
appended together)
There are database specific console
tools that will load your master SQL file
command line.
(ie. SQL*Plus for Oracle, ij for Apache Derby)
Part III
An improvement over DOS batch is using Cygwin bash
or python or perl to merge all of your sql files
together into a master file.
I found that the text processing tools available
in UNIX (awk,sed,cat...) are great for this sort
of thing.