I have a table with a lot of data that I don't need, but I have to save that, however to do my job i need only some data.
so, to limit the amount of data that my query will return I have to use a filter to not get some files like the strings .js, .css, .log etc.
I'm using the "Microsoft Azure Explorer - Azure Table Storage - Query Editor" to do this, but I don't know how to use a function like SQL function "like" to filter that strings.
follow a example of data that i don't need in my return:
lg.folha.views.demonstrativodemedias.editar.min.css
27042018_144634_536_6A2FAE_AMB..xml
is it possible to filter that files using only the query builder options?
Query Builder Options
Query builder documentation: https://learn.microsoft.com/en-us/rest/api/storageservices/Querying-Tables-and-Entities?redirectedfrom=MSDN#Anchor_2
Fro now azure table doesn't support wildcard query. You could get it from here:Filtering on String Properties.
Note that the Table service does not support wildcard queries.
So you have to use eq, ne, gt, ge, lt, le these operators.
You could use property ge 'test' and RowKey lt 'test~' to get query entities whose property starts with test.
So, the Azure Table service doesn't support the function like, so I did an application in C# to do this.
doing that I solved two problems:
1 - The like function - I created a function to exclud the worong files.
2 - the excel line limit - Excel have a limit of 1.048.576 and some of my stracts return more than this, so a did a function to split the list in N Excel's File.
Related
As the title suggests, I am building a power bi report that queries against an azure SQL database. One query is able to take in user defined parameters from the Transform Data tab of power bi. One of the parameters is a long string that contains ', [, ], and ", which need to be replaced with % before the SQL query is submitted.
Does anyone know of any ways to perform a replace automatically before the parameter is passed into the SQL query?
I have looked into the Replace values option for queries, however this cannot see my parameter.
Thanks!
As an answer to what I did, I placed a series of text.replace() inside the SQL query within power bi when passing the parameter in. That way the special characters were replaced with % (SQL wildcard) before we sent it to SQL.
Hello for demonstration purposes I trimmed out my actual sql query.
I have a SQL query
SELECT *
FROM dbdev.training.courses
where dbdev is my DEV database table name. When I migrate to TEST env, I want my query to dynamically change to
SELECT *
FROM dbtest.training.courses
I tried using input parameters like {env: p('db_name')} and using in the query as
SELECT * FROM :env.training.courses
or
SELECT * FROM (:env).training.courses
but none of them worked. I don't want my SQL query in properties file.
Can you please suggest a way to write my SQL query dynamically based on environment?
The only alternative way is to deploy separate jars for different environments with different code.
You can set the value of the property to a variable and then use the variable with string interpolation.
Warning: creating dynamic SQL queries using any kind of string manipulation may expose your application to SQL injection security vulnerabilities.
Example:
#['SELECT * FROM $(vars.database default "dbtest").training.courses']
Actually, you can do a completely dynamic or partially dynamic query using the MuleSoft DB connector.
Please see this repo:
https://github.com/TheComputerClassroom/dynamicSQLGETandPATCH
Also, I'm about to post an update that allows joins.
At a high level, this is a "Query Builder" where the code that builds the query is written in DataWeave 2. I'm working on another version that allows joins between entities, too.
If you have questions, feel free to reply.
One way to do it is :
Create a variable before DB Connector:
getTableName - ${env}.training.courses
Write SQL Query :
Select * from $(getTableName);
I am quite new to GBQ and any help is appreciated it.
I have a query below:
#Standard SQL
create or replace table `xxx.xxx.applications`
as select * from `yyy.yyy.applications`
What I need to do is to add today's date at the end of the table name so it is something like xxx.xxx.applications_<todays date>
basically create a filename with Application but add date at the end of the name applications.
I am writing a procedure to create a table every time it runs but need to add the date for audit purposes every time I create the table (as a backup).
I searched everywhere and can't get the exact answer, is this possible in Query Editor as I need to store this as a Proc.
Thanks in advance
BigQuery doesn't support dynamic SQL at the moment which means that this kind of construction is not possible.
Currently BigQuery supports Parameterized Queries but its not possible to use parameters to dynamically change the source table's name as you can see in the provided link.
BigQuery supports query parameters to help prevent SQL injection when
queries are constructed using user input. This feature is only
available with standard SQL syntax. Query parameters can be used as
substitutes for arbitrary expressions. Parameters cannot be used as
substitutes for identifiers, column names, table names, or other parts
of the query.
If you need to build a query based on some variable's value, I suggest that you use some script in SHELL, Python or any other programming language to create the SQL statement and then execute it using the bq command.
Another approach could be using the BigQuery client library in some of the supported languages instead of the bq command.
I've been reading up on how to query a wildcard table in BigQuery, but Data Studio doesn't seem to recognize the _TABLE_SUFFIX keyword.
Google Data Studio on using parameters
Google BigQuery docs on querying wildcard tables
I'm trying to use the recently added date parameters for a custom query in Data Studio. The goal is to prevent the custom query from scanning all partitions to save time.
When using the following query:
SELECT
*
FROM
`project-name.analytics_196324132.events_*`
WHERE
_TABLE_SUFFIX BETWEEN DS_START_DATE AND DS_END_DATE
I receive the following error:
Unrecognized name: _TABLE_SUFFIX
I expected the suffix keyword to be recognized so that the custom query is more efficient. But I get this error message. Does Data Studio not yet support this? Or is there another way?
It could be possible that you are setting the query in the wrong place. I created a DataSource from a Custom Query and the wildcard worked. The query I tested was the following, similar to yours since _TABLE_SUFFIX is a wildcard that is available in standardSQL in BigQuery:
select
*
from
`training_project.training_dataset.table1_*`
where
_TABLE_SUFFIX BETWEEN '20190625' AND '20190626'
As per your comments you are trying to add a query in the formula field of a custom parameter, however the formula field only accepts basic math operations, functions, and branching logic.
The workaround I can see is to build a select query and use it as a Custom Query in the Data Source definition so that the query can calculate any extra fields in advance (steps 5,6 and 7 from this tutorial).
I'm attempting to run a simple statement against an Access DB to find records.
Data validation in the records was horrible, and I cannot sanitize it. Meaning, it must be preserved as is.
I need to be able to search against a string with white space and hyphen characters removed. The following statement will work in Access 2010 direct:
select * from dummy where Replace(Replace([data1],' ',''),'-','') = 'ABCD1234';
Running it from an ODBC connection via PHP will not. It produces the following error:
SQL error: [Microsoft][ODBC Microsoft Access Driver] Undefined function 'Replace' in expression., SQL state 37000 in SQLExecDirect
Creating a query in the database that runs the function and attempting to search its values indirectly causes the same error:
select * from dummy_indirect where Expr1 = 'ABCD1234';
I've attempted to use both ODBC drivers present. ODBCJR32.dll (03/22/2010) and ACEODBC.dll (02/18/2007). To my knowledge these should be current as it was installed with the full Access 2010 and Access 2010 Database Engine.
Any ideas on how to work around this error and achieve the same effect are welcome. Please note, that I cannot alter the database in way, shape, or form. That indirect query was created in another mdb file that has the original tables linked from the original DB.
* Update *
OleDB did not really affect anything.
$dsn= "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=c:\dummy.mdb;";
I'm not attempting to use it as a web backend either. I'm not a sadomasochist.
There is a legacy system that I must support that does use Access as a backend. Data gets populated there from other old systems that I must integrate into more modern systems. Hence, the creation of an API with Apache/PHP that is running on the server supporting the legacy system.
I need to be able to search a table that has an alphanumeric case identifier to get a numeric identifier that is unique and tied to a generator (Autonumber in access). Users have been using it a trash box for years (inconsistent data entry with sporadic notations) so the only solution I have is to strip everything except alphanumeric out of both the field value and the search value and attempt to perform a LIKE comparison against it.
If not replace() which is access supported, what ODBC compatible functions exist that I can use do the same kind of comparison?
Just to recap, the Access db engine will not recognize the Replace() function unless your query is run from within an Access application session. Any attempt from outside Access will trigger that "Undefined function" error message. You can't avoid the error by switching from ODBC to OleDb as the connection method. And you also can't trick the engine into using Replace() by hiding it in separate query (in the same or another Access db) and using that query as the data source for your main query.
This behavior is determined by Access' sandbox mode. That linked page includes a list of functions which are available in the default sandbox mode. That page also describes how you can alter the sandbox mode. If you absolutely must have Replace() available for your query, perhaps the lowest setting (0) would allow it. However, I'm not recommending you do that. I've never done it myself, so don't know anything about the consequences.
As for alternatives for Replace(), it would help to know about the variability in the values you're searching. If the space or dash characters appear in only one or a few consistent positions, you could do a pattern match with a Like expression. For example, if the search field values consist of 4 letters, an optional space or dash, followed by 4 digits, a WHERE clause like this should work for the variations of "ABCD1234":
SELECT * FROM dummy
WHERE
data1 = 'ABCD1234'
OR data1 Like 'ABCD[- ]1234';
Another possibility is to compare against a list of values:
SELECT * FROM dummy
WHERE
data1 IN ('ABCD1234','ABCD 1234','ABCD-1234');
However if your search field values can include any number of spaces or dashes at any position within the string, that approach is no good. And I would look real hard for some way to make the query task easier:
You can't clean the stored values because you're prohibited from altering the original Access db in any way. Perhaps you could create a new Access db, import the data, and clean that instead.
Set up the original Access db as a linked server in SQL Server and build your query to take advantage of SQL Server features.
Surrender. :-( Pull in a larger data set to your PHP client code, and evaluate which rows to use vs. which to ignore.
I'm not sure you can do this with ODBC and your constraints. The MS Access driver is limited (by design; MS wants you to use SQL Server for back ends).
Can you use OLEDB? that might be an option.