I am trying to find a "quick" way to lower case all the data (strings) in a table's column inside Google Cloud BigQuery.
Before going into building a script, I'm looking for more shorter way like Query.
How can I query BigQuery to lower case entire column?
You can use an UPDATE statement:
UPDATE YourTable
SET string_column = LOWER(string_column)
WHERE true;
How can I query BigQuery to lower case entire column?
Definitely LOWER is the function to use
For example
#standardSQL
WITH `dataset.table` AS (
SELECT
'https://stackoverflow.com/q/44970976/5221944' AS url,
'How to lower case entire column data in Google Cloud BigQuery' AS title
)
SELECT * REPLACE(LOWER(title) AS title)
FROM `dataset.table`
I am trying to find a "quick" way to lower case all the data
From what I see in your question - I would not recommend using DML's UPDATE as it is Costly and not necessarily "quick" and for sure not flexible in case if you later will want to change your mind let say to have it UPPER or somehow differently (camel style for example)
The quick way in your case I see in creating view like below. It is cheap ($0.00) and flexible to accomodate any logic of transforming columns in original table
SELECT * REPLACE(LOWER(title) AS title)
FROM `dataset.table`
Found out my self. it can be done using UPDATE as Elliott suggested. but it must use standartSQL. I used the #standardSQL declration for that.
#standardSQL
UPDATE dataset.table
SET field = LOWER(field)
WHERE TRUE
Related
I would like to create a "view" table where I would like to rename 2 rows (2 different names).
Unfortunately, when I type this command, it doesn't work:
SELECT torch_cooling AS MASTER,
REPLACE (REPLACE(torch_cooling, 'gas', 'Gasgekühlt')'water', 'Wassergekühlt') AS TEXT
FROM to_torches
I would like "gas" to be "Gasgekühlt" and "Water" to be "Wassergekühlt".
SELECT torch_cooling AS MASTER,
REPLACE (REPLACE(torch_cooling, 'gas', 'Gasgekühlt') 'water', 'Wassergekühlt') AS TEXT
FROM to_torches`
I think REPLACE is not the right tool for this, consider using a CASE for this.
SELECT torch_cooling AS MASTER,
CASE torch_cooling
WHEN 'gas' THEN 'Gasgekühlt'
WHEN 'water' THEN 'Wassergekühlt'
END AS TEXT
FROM t;
This is not really scalable, if you want to scale this, you should use a mapping table.
Beside the option to use a CASE WHEN construct for that (which is absolutely fine), you can also use a function like DECODE. Usually, your DB provides such a function.
The example according to your situation:
SELECT torch_cooling AS master,
DECODE (torch_cooling ,'gas','Gasgekühlt', 'water','Wassergekühlt') AS text
FROM to_torches;
Please note the exact syntax and function name for this way depend on your DB type since the function name differs.
DECODE will be used in Oracle DB's, but there are similar functions in other DB types, too.
Thank you for checking my question out!
I'm trying to write a query for a very specific problem we're having at my workplace and I can't seem to get my head around it.
Short version: I need to be able to target columns by their name, and more specifically by a part of their name that will be consistent throughout all the columns I need to combine or compare.
More details:
We have (for example), 5 different surveys. They have many questions each, but SOME of the questions are part of the same metric, and we need to create a generic field that keeps it. There's more background to the "why" of that, but it's pretty important for us at this point.
We were able to kind of solve this with either COALESCE() or CASE statements but the challenge is that, as more surveys/survey versions continue to grow, our vendor inevitably generates new columns for each survey and its questions.
Take this example, which is what we do currently and works well enough:
CASE
WHEN SURVEY_NAME = 'Service1' THEN SERV1_REC
WHEN SURVEY_NAME = 'Notice1' THEN FNOL1_REC
WHEN SURVEY_NAME = 'Status1' THEN STAT1_REC
WHEN SURVEY_NAME = 'Sales1' THEN SALE1_REC
WHEN SURVEY_NAME = 'Transfer1' THEN Null
ELSE Null
END REC
And also this alternative which works well:
COALESCE(SERV1_REC, FNOL1_REC, STAT1_REC, SALE1_REC) as REC
But as I mentioned, eventually we will have a "SALE2_REC" for example, and we'll need them BOTH on this same statement. I want to create something where having to come into the SQL and make changes isn't needed. Given that the columns will ALWAYS be named "something#_REC" for this specific metric, is there any way to achieve something like:
COALESCE(all columns named LIKE '%_REC') as REC
Bonus! Related, might be another way around this same problem:
Would there also be a way to achieve this?
SELECT (columns named LIKE '%_REC') FROM ...
Thank you very much in advance for all your time and attention.
-Kendall
Table and column information in Db2 are managed in the system catalog. The relevant views are SYSCAT.TABLES and SYSCAT.COLUMNS. You could write:
select colname, tabname from syscat.tables
where colname like some_expression
and syscat.tabname='MYTABLE
Note that the LIKE predicate supports expressions based on a variable or the result of a scalar function. So you could match it against some dynamic input.
Have you considered storing the more complicated properties in JSON or XML values? Db2 supports both and you can query those values with regular SQL statements.
I need to filter out records based on some text matching in nvarchar(1000) column.
Table has more than 400 thousands records and growing. For now, I am using Like condition:-
SELECT
*
FROM
table_01
WHERE
Text like '%A1%'
OR Text like '%B1%'
OR Text like '%C1%'
OR Text like '%D1%'
Is there any preferred work around?
SELECT
*
FROM
table_01
WHERE
Text like '%[A-Z]1%'
This will check if the texts contains A1, B1, C1, D1, ...
Reference to using the Like Condition in SQL Server
You can try the following if you know the exact position of your sub string:
SELECT
*
FROM
table_01
WHERE
SUBSTRING(Text,1,2) in ('B1','C1','D1')
Have a look at LIKE on msdn.
You could reduce the number filters by combining more details into a single LIKE clause.
SELECT
*
FROM
table_01
WHERE
Text like '%[ABCD]1%'
If you can create a FULLTEXT INDEX on that column of your table (that assumes a lot of research on performance and space), then you are probably going to see a big improvement on performance on text matching. You can go to this link to see what FULLTEXT SEARCH is
and this link to see how to create a FULLTEXT INDEX.
I needed to do this so that I could allow two different databases in a filter for the DatabaseName column in an SQL Server Profiler Trace Template.
All you can do is fill in the body of a Like clause.
Using the reference in John Hartscock's answer, I found out that the like clause uses a sort of limited regex pattern.
For the OP's scenario, MSMS has the solution.
Assuming I want databases ABCOne, ABCTwo, and ABCThree, I come up with what is essentially independent whitelists for each character:
Like ABC[OTT][NWH][EOR]%
Which is easily extensible to any set of strings. It won't be ironclad, that last pattern would also match ABCOwe, ABCTnr, or ABCOneHippotamus, but if you're filtering a limited set of possible values there's a good chance you can make it work.
You could alternatively use the [^] operator to present a blacklist of unacceptable characters.
In the process of fixing a poorly imported database with issues caused by using the wrong database encoding, or something like that.
Anyways, coming back to my question, in order to fix this issues I'm using a query of this form:
UPDATE table_name SET field_name =
replace(field_name,’search_text’,'replace_text’);
And thus, if the table I'm working on has multiple columns I have to call this query for each of the columns. And also, as there is not only one pair of things to run the find and replace on I have to call the query for each of this pairs as well.
So as you can imagine, I end up running tens of queries just to fix one table.
What I was wondering is if there is a way of either combine multiple find and replaces in one query, like, lets say, look for this set of things, and if found, replace with the corresponding pair from this other set of things.
Or if there would be a way to make a query of the form I've shown above, to run somehow recursively, for each column of a table, regardless of their name or number.
Thank you in advance for your support,
titel
Let's try and tackle each of these separately:
If the set of replacements is the same for every column in every table that you need to do this on (or there are only a couple patterns), consider creating a user-defined function that takes a varchar and returns a varchar that just calls replace(replace(#input,'search1','replace1'),'search2','replace2') nested as appropriate.
To update multiple columns at the same time you should be able to do UPDATE table_name SET field_name1 = replace(field_name1,...), field_name2 = replace(field_name2,...) or something similar.
As for running something like that for every column in every table, I'd think it would be easiest to write some code which fetches a list of columns and generates the queries to execute from that.
I don't know of a way to automatically run a search-and-replace on each column, however the problem of multiple pairs of search and replace terms in a single UPDATE query is easily solved by nesting calls to replace():
UPDATE table_name SET field_name =
replace(
replace(
replace(
field_name,
'foo',
'bar'
),
'see',
'what',
),
'I',
'mean?'
)
If you have multiple replaces of different text in the same field, I recommend that you create a table with the current values and what you want them replaced with. (Could be a temp table of some kind if this is a one-time deal; if not, make it a permanent table.) Then join to that table and do the update.
Something like:
update t1
set field1 = t2.newvalue
from table1 t1
join mycrossreferncetable t2 on t1.field1 = t2.oldvalue
Sorry didn't notice this is MySQL, the code is what I would use in SQL Server, my SQL syntax may be different but the technique would be similar.
I wrote a stored procedure that does this. I use this on a per database level, although it would be easy to abstract it to operate globally across a server.
I would just paste this inline, but it would seem that I'm too dense to figure out how to use the markdown deal, so the code is here:
http://www.anovasolutions.com/content/mysql-search-and-replace-stored-procedure
I am doing a simple SELECT statement in an Oracle DB and need to select the columns in a somewhat-specific order. Example:
Table A has 100 attributes, one of which is "chapter" that occurs somewhere in the order of columns in the table. I need to select the data with "chapter" first and the remaining columns after in no particular order. Essentially, my statement needs to read something like:
SELECT a.chapter, a. *the remaining columns* FROM A
Furthermore, I cannot simply type:
SELECT a.chapter, a.*
because this will select "chapter" twice.
I know the SQL statement seems simple, but if I know how to solve this problem, I can extrapolate this thought into more complicated areas. Also, let's assume that I can't just scroll over to find the "chapter" column and drag it to the beginning.
Thanks.
You should not select * in a program. As your schema evolves it will bring in things you do not know yet. Think about what happens when someone add a column with the whole book in it? The query you thought would be very cheap suddenly starts to bring in megabytes of data.
That means you have to specify every column you need.
Your best bet is just to select each column explicitly.
A quickie way to get around this would be SELECT a.chapter AS chapterCol, a.* FROM table a; This means there will be one column name chapterCol (assuming there's not a column already there named chapterCol. ;))
If your going to embed the 'SELECT *' into program code, then I would strongly recommend against doing that. As noted by the previous authors, your setting up the code to break if a column is ever added to (or removed from) the table. The simple advice is don't do it.
If your using this in development tools (viewing the data, and the like). Then, I'd recommend creating a view with the specific column order you need. Capture the output from 'SELECT COLUMN_NAME FROM ALL_TAB_COLUMNS' and create a select statement for the view with the column order you need.
This is how I would build your query without having to type all the names in, but with some manual effort.
Start with "Select a.chapter"
Now perform another select on your data base as follows :
select ','|| column_name
from user_tab_cols
where table_name = your_real_table_name
and column_name <> 'CHAPTER';
now take the output from that, in a cut-and-paste manner and append it to what you started with. Now run that query. It should be what you asked for.
Ta-da!
Unless you have a very good reason to do so, you should not use SELECT * in queries. It will break your application every time the schema changes.