Related
I am working on a sql database which will provide with data some grid. The grid will enable filtering, sorting and paging but also there is a strict requirement that users can enter free text to a text input above the grid for example
'Engine 1001 Requi' and that the result will contain only rows which in some columns contain all the pieces of the text. So one column may contain Engine, other column may contain 1001 and some other will contain Requi.
I created a technical column (let's call it myTechnicalColumn) in the table (let's call it myTable) which will be updated each time someone inserts or updates a row and it will contain all the values of all the columns combined and separated with space.
Now to use it with entity framework I decided to use a table valued function which accepts one parameter #searchQuery and it will handle it like this:
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS #Result TABLE
( ... here come columns )
AS
BEGIN
DECLARE #searchToken TokenType
INSERT INTO #searchToken(token) SELECT value FROM STRING_SPLIT(#searchText,' ')
DECLARE #searchTextLength INT
SET #searchTextLength = (SELECT COUNT(*) FROM #searchToken)
INSERT INTO #Result
SELECT
... here come columns
FROM myTable
WHERE (SELECT COUNT(*) FROM #searchToken WHERE CHARINDEX(token, myTechnicalColumn) > 0) = #searchTextLength
RETURN;
END
Of course the solution works fine but it's kinda slow. Any hints how to improve its efficiency?
You can use an inline Table Valued Function, which should be quite a lot faster.
This would be a direct translation of your current code
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS TABLE
AS RETURN
(
WITH searchText AS (
SELECT value token
FROM STRING_SPLIT(#searchText,' ') s(token)
)
SELECT
... here come columns
FROM myTable t
WHERE (
SELECT COUNT(*)
FROM searchText
WHERE CHARINDEX(s.token, t.myTechnicalColumn) > 0
) = (SELECT COUNT(*) FROM searchText)
);
GO
You are using a form of query called Relational Division Without Remainder and there are other ways to cut this cake:
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS TABLE
AS RETURN
(
WITH searchText AS (
SELECT value token
FROM STRING_SPLIT(#searchText,' ') s(token)
)
SELECT
... here come columns
FROM myTable t
WHERE NOT EXISTS (
SELECT 1
FROM searchText
WHERE CHARINDEX(s.token, t.myTechnicalColumn) = 0
)
);
GO
This may be faster or slower depending on a number of factors, you need to test.
Since there is no data to test, i am not sure if the following will solve your issue:
-- Replace the last INSERT portion
INSERT INTO #Result
SELECT
... here come columns
FROM myTable T
JOIN #searchToken S ON CHARINDEX(S.token, T.myTechnicalColumn) > 0
For a project, we are using a table (named txtTable) that contains all the texts. And each column contains a different language (for example column L9 is English, column L7 is German, etc..).
TextID L9 L7 L16 L10 L12
------------------------------------------------------
26 Archiving Archivierung NULL NULL NULL
27 Logging Protokollierung NULL NULL NULL
28 Comments Kommentar NULL NULL NULL
This table is located in a database on a Microsoft SQL Server 2005. The big problem is that this database name changes each time the program is restarted. This is a behavior typically for this third-party program and cannot be changed.
Next to this database and on the same server is our own database. In this database are several tables that point to the textID for generating data for reporting (SQL Server Reporting Services) in the correct language. This database contains also a table "ProjectSettings" with some properties like the name of the texttable database, and the stored procedures to generate the reporting data.
The way we now are requesting the right texts of the right language from this table with the changing database name is by creating a dynamic SQL query and execute it in a stored procedure.
Now we were wondering if there is a cleaner way to get the texts in the right language. We were thinking about creating a function with the textID and the language as a parameter, but we cannot find a good way to do this. We thought about a function so we just can use it in the select statement, but this doesn’t work:
CREATE FUNCTION [dbo].[GetTextFromLib]
(
#TextID int,
#LanguageColumn Varchar(5)
)
RETURNS varchar(255)
AS
BEGIN
-- return variables
DECLARE #ResultVar varchar(255)
-- Local variables
DECLARE #TextLibraryDatabaseName varchar(1000)
DECLARE #nvcSqlQuery varchar(1000)
-- get the report language database name
SELECT #TextLibraryDatabaseName = TextLibraryDatabaseName FROM ProjectSettings
SET #nvcSqlQuery = 'SELECT #ResultVar =' + #LanguageColumn + ' FROM [' + #TextLibraryDatabaseName + '].dbo.TXTTable WHERE TEXTID = ' + cast(#TextID as varchar(30))
EXEC(#nvcSqlQuery)
-- Return the result of the function
RETURN #ResultVar
END
Is there any way to work around this so we don’t have to use the dynamic sql in our stored procedures so it is only ‘contained’ in 1 function?
Thanks in advance & kind regards,
Kurt
Yes, it is possible with the help of synonym mechanism introduced with SQL Server 2005. So, you can create synonym during your setting up procedure based on data from ProjectSettings table and you can use it in your function. Your code will look something like this:
UPDATE: The code of function is commented here because it still contains dynamic SQL which does not work in function as Kurt said in his comment. New version of function is below this code.
-- Creating synonym for TXTTable table
-- somewhere in code when processing current settings
-- Suppose your synonym name is 'TextLibrary'
--
-- Drop previously created synonym
IF EXISTS (SELECT * FROM sys.synonyms WHERE name = N'TextLibrary')
DROP SYNONYM TextLibrary
-- Creating synonym using dynamic SQL
-- Local variables
DECLARE #TextLibraryDatabaseName varchar(1000)
DECLARE #nvcSqlQuery varchar(1000)
-- get the report language database name
SELECT #TextLibraryDatabaseName = TextLibraryDatabaseName FROM ProjectSettings
SET #nvcSqlQuery = 'CREATE SYNONYM TextLibrary FOR [' + #TextLibraryDatabaseName + '].dbo.TXTTable'
EXEC(#nvcSqlQuery)
-- Synonym created
/* UPDATE: This code is commented but left for discussion consistency
-- Function code
CREATE FUNCTION [dbo].[GetTextFromLib]
(
#TextID int,
#LanguageColumn Varchar(5)
)
RETURNS varchar(255)
AS
BEGIN
-- return variables
DECLARE #ResultVar varchar(255)
-- Local variables
DECLARE #nvcSqlQuery varchar(1000)
SET #nvcSqlQuery = 'SELECT #ResultVar =' + #LanguageColumn + ' FROM TextLibrary WHERE TEXTID = ' + cast(#TextID as varchar(30))
EXEC(#nvcSqlQuery)
-- Return the result of the function
RETURN #ResultVar
END
*/
UPDATE This is one more attempt to solve the problem. Now it uses some XML trick:
-- Function code
CREATE FUNCTION [dbo].[GetTextFromLib]
(
#TextID int,
#LanguageColumn Varchar(5)
)
RETURNS varchar(255)
AS
BEGIN
-- return variables
DECLARE #ResultVar varchar(255)
-- Local variables
DECLARE #XmlVar XML
-- Select required record into XML variable
-- XML has each table column value in element with corresponding name
SELECT #XmlVar = ( SELECT * FROM TextLibrary
WHERE TEXTID = #TextID
FOR XML RAW, ELEMENTS )
-- Select value of required element from XML
SELECT #ResultVar = Element.value('(.)[1]', 'varchar(255)')
FROM #XmlVar.nodes('/row/*') AS T(Element)
WHERE Element.value('local-name(.)', 'varchar(50)') = #LanguageColumn
-- Return the result of the function
RETURN #ResultVar
END
Hope this helps.
Credits to answerer of this question at Stackoverflow - How to get node name and values from an xml variable in t-sql
To me, it sounds like a total PITA... However, how large is this database of "words" you are dealing with. Especially if it is not changing much and remains pretty constant. Why not have on some normal cycle (such as morning), just have one dynamic query generated that queries the one that changes and synchronize it to a "standard" table name in YOUR database that won't change. Then, all your queries run against YOUR version and completely remove the constant dynamic queries every time. Yes there would need to be this synchronizing stored procedure to run, but if it can be run on a schedule, you should be fine, and again, how large is the table of "words" for proper language context.
sql2005
This is my simplified example:
(in reality there are 40+ tables in here, I only showed 2)
I got a table called tb_modules, with 3 columns (id, description, tablename as varchar):
1, UserType, tb_usertype
2, Religion, tb_religion
(Last column is actually the name of a different table)
I got an other table that looks like this:
tb_value (columns:id, tb_modules_ID, usertype_OR_religion_ID)
values:
1111, 1, 45
1112, 1, 55
1113, 2, 123
1114, 2, 234
so, I mean 45, 55, 123, 234 are usertype OR religion ID's
(45, 55 usertype, 123, 234 religion ID`s)
Don't judge, I didn't design the database
Question
How can I make a select, showing * from tb_value, plus one column
That one column would be TITLE from the tb_usertype or RELIGIONNAME from the tb_religion table
I would like to make a general thing.
Was thinking initially about maybe a SQL function that returns a string, but I think I would need dynamic SQL, which is not ok in a function.
Anyone a better idea ?
At the beginning we have this -- which is quite messy.
To clean-up a bit I add two views and a synonym:
create view v_Value as
select
ID as ValueID
, tb_modules_ID as ModuleID
, usertype_OR_religion_ID as RemoteID
from tb_value ;
go
create view v_Religion as
select
ID
, ReligionName as Title
from tb_religion ;
go
create synonym v_UserType for tb_UserType ;
go
And now the model looks like
It is easier now to write the query
;
with
q_mod as (
select
m.ID as ModuleID
, coalesce(x1.ID , x2.ID) as RemoteID
, coalesce(x1.Title , x2.Title) as Title
, m.Description as ModuleType
from tb_Modules as m
left join v_UserType as x1 on m.TableName = 'tb_UserType'
left join v_Religion as x2 on m.TableName = 'tb_Religion'
)
select
a.ModuleID
, v.ValueID
, a.RemoteID
, a.ModuleType
, a.Title
from q_mod as a
join v_Value as v on (v.ModuleID = a.ModuleID and v.RemoteID = a.RemoteID) ;
There is an obvious pattern in this query, so it can be created as dynamic sql if you have to add another module-type table. When adding another table, use ID and Title to avoid having to use a view.
EDIT
To build dynamic sql (or query on application level)
Modify lines 6 and 7, the x-index is tb_modules.id
coalesce(x1. , x2. , x3. ..)
Add lines to the left join (below line 11)
left join v_SomeName as x3 on m.TableName = 'tb_SomeName'
The SomeName is tb_modules.description and x-index is matching tb_modules.id
EDIT 2
The simplest would probably be to package the above query into a view and then each time the schema changes dynamically crate and run ALTER VIEW. This way the query would not change from the point of the application.
Since we're all agreed the design is flaky, I'll skip any comments on that. The pattern of the query is this:
-- Query 1
select tb_value.*,tb_religion.religion_name as ANY_DESCRIPTION
from tb_value
JOIN tb_religion on tb_value.ANY_KIND_OF_ID = tb_religion.id
WHERE tb_value.module_id = 2
-- combine it with...
UNION ALL
-- ...Query 2
select tb_value.*,tb_religion.title as ANY_DESCRIPTION
from tb_value
JOIN tb_userType on tb_value.ANY_KIND_OF_ID = tb_userType.id
WHERE tb_value.module_id = 1
-- combine it with...
UNION ALL
-- ...Query 3
select lather, rinse, repeat for 40 tables!
You can actually define a view that hardcodes all 40 cases, and then put filters onto queries for the particular modules you want.
To do this dynamically you need to be able to create a sql statement that looks like this
select tb_value.*, tb_usertype.title as Descr
from tb_value
inner join tb_usertype
on tb_value.extid = tb_usertype.id
where tb_value.tb_module_id = 1
union all
select tb_value.*, tb_religion.religionname as Descr
from tb_value
inner join tb_religion
on tb_value.extid = tb_religion.id
where tb_value.tb_module_id = 2
-- union 40 other tables
Currently you can not do that because you do not have any information in the db telling you which column to use from tb_religion and tb_usertype etc. You can add that as a new field in tb_module.
If you have fieldname to use in tb_module you can build a view that does what you want.
And you could add a trigger to table tb_modules that alters the view whenever tb_modules is modified. That way you do not need to use dynamic sql from the client when doing queries. The only thing you need to worry about is that the table needs to be created in the db before you add a new row to tb_modules
Edit 1
Of course the code in the trigger needs to dynamically build the alter view statement.
Edit 2 You also need to have a field with information about what column in tb_usertype and tb_religion etc. to join against tb_value.extid (usertype_OR_religion_ID). Or you can assume that the field will always be called id
Edit 3 Here is how you could build the trigger on tb_module that alters the view v_values. I have added fieldname as a column in tb_modules and I assume that the id field in the related tables is called id.
create trigger tb_modules_change on tb_modules after insert, delete, update
as
declare #sql nvarchar(max)
declare #moduleid int
declare #tablename varchar(50)
declare #fieldname varchar(50)
set #sql = 'alter view v_value as '
declare mcur cursor for
select id, tablename, fieldname
from tb_modules
open mcur
fetch next from mcur into #moduleid, #tablename, #fieldname
while ##FETCH_STATUS = 0
begin
set #sql = #sql + 'select tb_value.*, '+#tablename+'.'+#fieldname+' '+
'from tb_value '+
'inner join '+#tablename+' '+
'on tb_value.extid = '+#tablename+'.id '+
'where tb_value.tb_module_id = '+cast(#moduleid as varchar(10))
fetch next from mcur into #moduleid, #tablename, #fieldname
if ##FETCH_STATUS = 0
begin
set #sql = #sql + ' union all '
end
end
close mcur
deallocate mcur
exec sp_executesql #sql
Hm..there are probably better solutions available but here's my five cents:
SELECT
id,tb_modules_ID,usertype_OR_religion_ID,
COALESCE(
(SELECT TITLE FROM tb_usertype WHERE Id = usertype_OR_religion_ID),
(SELECT RELIGIONNAME FROM tb_religion WHERE Id = usertype_OR_religion_ID),
'N/A'
) AS SourceTable
FROM tb_valuehere
Note that I don't have the possibility to check the statement right now so I'm reserving myself for any syntax errors...
First, using your current design the only reasonable solution is dynamic SQL. You should write a module in your middle-tier that queries for the appropriate table names and builds the queries on the fly. Trying to accomplish that in T-SQL will be a nightmare. T-SQL was not designed for string construction.
The right solution is to build a new database designed properly, migrate the data and scrap the existing design. The problems you will encounter with your current design will simply grow. It will be harder for new developers to learn the new system. It will be prone to errors. There will be no data integrity (e.g. forcing the attribute "Start Date" to be parsable as a date). Custom queries will be a chore to write and so on. Eventually, you will hit the day when the types of information desired from the system are simply too difficult to extract given the current design.
First take the undesigner out the back and put them out of their misery. They are hurting people.
Due to their incompetence, every time you add a row to Module, you have to modify every query that uses it. Good for www.dailywtf.com.
You do not have Referential Integrity either, because you cannot define an FK on the this_or_that column. Your data is exposed, probably to "code" written by the same undesigner. No doubt you are aware that this is where the deadlocks are created.
That it is a "judgement", that is so that you understand the gravity of the undesign, and you can justify replacing it, to your managers.
SQL was designed for Relational Databases, that means Normalised. It is not good for mangled files. Sure, some queries may be better than others (just look at the answers), but there is no way to get around the undesign, any SQL query will be hamstrung, and need change whenever a Module row is added.
"Dynamic" is reserved for Databases, not possible for flat flies.
Two answers. One to stop the continuing idiocy of changing the existing queries every time a Module row is added (you're welcome); the second to answer your question.
Safe Future Queries
CREATE VIEW UserReligion_vw AS
SELECT [XxxxId] = id, -- replace Xxxx
[ReligionId] = usertype_OR_religion_ID
FROM tb_value
WHERE tb_modules_ID = 1
CREATE VIEW UserReligion_vw AS
SELECT [XxxxId] = id,
[ReligionId] = usertype_OR_religion_ID
FROM tb_value
WHERE tb_modules_ID = 2
From now on, make sure the all queries currently using the undesign, are modified to use the correct View instead. Do not use the Views for Update/Delete/Insert.
Answer
Ok, now for the main question. I can think of other approaches, but this one is the best. You have stated, you want the third column to also be an unnormalised piece of chicken excreta and the supply Title for [EITHER_Religion_OR_UserType_OR_This_OR_That]. Right, so you are teaching the user to be confused as well; when the no of modules grow, they will have great fun figuring out what the column contains. Yes a problem does always compound itself.
SELECT [XxxxId] = id,
[Whatever] = CASE tb_modules_ID
WHEN 1 THEN ( SELECT name -- title, whatever
FROM tb_religion
WHERE id = V.usertype_OR_religion_ID
)
WHEN 2 THEN ( SELECT name -- title, whatever
FROM tb_usertype
WHERE id = V.usertype_OR_religion_ID
)
ELSE "(UnknownModule)" -- do not remove the brackets
END
FROM tb_value V
WHERE conditions... -- you need something here
This is called a Correlated Scalar Subquery.
It works on any version of Sybase since 4.9.2 with no limitations. And SQL 2005 (last time I looked, anyway, Aug 2009). But on MS you will get a StackTrace if the volume of tb_value is large, so make sure the WHERE clause has some conditions on it.
But MS have broken the server with their "new" 2008 codeline, so it does not work in all circumstances (the worse your mangled files, the less likely it will work; the better your database design, the more likely it will work). That is why some MS people pray every day for the next Service pack, and others never attend church.
I guess you want something like this:
Adding tables and one row per table into tb_modules is straight forward.
SET NOCOUNT ON
if OBJECT_ID('tb_modules') > 0 drop table tb_modules;
if OBJECT_ID('tb_value') > 0 drop table tb_value;
if OBJECT_ID('tb_usertype') > 0 drop table tb_usertype;
if OBJECT_ID('tb_religion') > 0 drop table tb_religion;
go
create table dbo.tb_modules (
id int,
description varchar(20),
tablename varchar(255)
);
insert into tb_modules values ( 1, 'UserType', 'tb_usertype');
insert into tb_modules values ( 2, 'Religion', 'tb_religion');
create table dbo.tb_value(
id int,
tb_modules_ID int,
usertype_OR_religion_ID int
);
insert into tb_value values ( 1111, 1, 45);
insert into tb_value values ( 1112, 1, 55);
insert into tb_value values ( 1113, 2, 123);
insert into tb_value values ( 1114, 2, 234);
create table dbo.tb_usertype(
id int,
UserType varchar(30)
);
insert into tb_usertype values ( 45, 'User_type_45');
insert into tb_usertype values ( 55, 'User_type_55');
create table dbo.tb_religion(
id int,
Religion varchar(30)
);
insert into tb_religion values ( 123, 'Religion_123');
insert into tb_religion values ( 234, 'Religion_234');
-- start of query
declare #sql varchar(max) = null
Select #sql = case when #sql is null then ' ' else #sql + char(10) + 'union all ' end
+ 'Select ' + str(id) + ' type, id, ' + description + ' description from ' + tablename from tb_modules
set #sql = 'select v.id, tb_modules_ID , usertype_OR_religion_ID , t.description
from tb_value v
join ( ' + #sql + ') as t
on v.tb_modules_ID = t.type and v.usertype_OR_religion_ID = t.id
'
Print #sql
exec( #sql)
I think it's intended to be used with dynamic sql.
Maybe break out each tb_value.tb_modules_ID row into its own temp table, named with the tb_modules.tablename.
Then have an sp iterate through the temp tables matching your naming convention (by prefix or suffix) building the sql and doing your join.
We all know that to select all columns from a table, we can use
SELECT * FROM tableA
Is there a way to exclude column(s) from a table without specifying all the columns?
SELECT * [except columnA] FROM tableA
The only way that I know is to manually specify all the columns and exclude the unwanted column. This is really time consuming so I'm looking for ways to save time and effort on this, as well as future maintenance should the table has more/less columns.
You can try it this way:
/* Get the data into a temp table */
SELECT * INTO #TempTable
FROM YourTable
/* Drop the columns that are not needed */
ALTER TABLE #TempTable
DROP COLUMN ColumnToDrop
/* Get results and drop temp table */
SELECT * FROM #TempTable
DROP TABLE #TempTable
No.
Maintenance-light best practice is to specify only the required columns.
At least 2 reasons:
This makes your contract between client and database stable. Same data, every time
Performance, covering indexes
Edit (July 2011):
If you drag from Object Explorer the Columns node for a table, it puts a CSV list of columns in the Query Window for you which achieves one of your goals
If you don't want to write each column name manually you can use Script Table As by right clicking on table or view in SSMS like this:
Then you will get whole select query in New Query Editor Window then remove unwanted column like this:
Done
The automated way to do this in SQL (SQL Server) is:
declare #cols varchar(max), #query varchar(max);
SELECT #cols = STUFF
(
(
SELECT DISTINCT '], [' + name
FROM sys.columns
where object_id = (
select top 1 object_id from sys.objects
where name = 'MyTable'
)
and name not in ('ColumnIDontWant1', 'ColumnIDontWant2')
FOR XML PATH('')
), 1, 2, ''
) + ']';
SELECT #query = 'select ' + #cols + ' from MyTable';
EXEC (#query);
A modern SQL dialect like BigQuery proposes an excellent solution.
SELECT * EXCEPT(ColumnNameX, [ColumnNameY, ...])
FROM TableA
This is a very powerful SQL syntax to avoid a long list of columns that need to be updated all the time due to table column name changes. And this functionality is missing in the current SQL Server implementation, which is a pity. Hopefully, one day, Microsoft Azure will be more data scientist-friendly.
Data scientists like to have a quick option to shorten a query and remove some columns (due to duplication or any other reason).
https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#select-modifiers
You could create a view that has the columns you wish to select, then you can just select * from the view...
Yes it's possible (but not recommended).
CREATE TABLE contact (contactid int, name varchar(100), dob datetime)
INSERT INTO contact SELECT 1, 'Joe', '1974-01-01'
DECLARE #columns varchar(8000)
SELECT #columns = ISNULL(#columns + ', ','') + QUOTENAME(column_name)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'contact' AND COLUMN_NAME <> 'dob'
ORDER BY ORDINAL_POSITION
EXEC ('SELECT ' + #columns + ' FROM contact')
Explanation of the code:
Declare a variable to store a comma separated list of column names. This defaults to NULL.
Use a system view to determine the names of the columns in our table.
Use SELECT #variable = #variable + ... FROM to concatenate the
column names. This type of SELECT does not not return a result set. This is perhaps undocumented behaviour but works in every version of SQL Server. As an alternative you could use SET #variable = (SELECT ... FOR XML PATH('')) to concatenate strings.
Use the ISNULL function to prepend a comma only if this is not the
first column name.
Use the QUOTENAME function to support spaces and punctuation in column names.
Use the WHERE clause to hide columns we don't want to see.
Use EXEC (#variable), also known as dynamic SQL, to resolve the
column names at runtime. This is needed because we don't know the column names at compile time.
Like the others have said there is no way to do this, but if you're using Sql Server a trick that I use is to change the output to comma separated, then do
select top 1 * from table
and cut the whole list of columns from the output window. Then you can choose which columns you want without having to type them all in.
Basically, you cannot do what you would like - but you can get the right tools to help you out making things a bit easier.
If you look at Red-Gate's SQL Prompt, you can type "SELECT * FROM MyTable", and then move the cursor back after the "*", and hit <TAB> to expand the list of fields, and remove those few fields you don't need.
It's not a perfect solution - but a darn good one! :-) Too bad MS SQL Server Management Studio's Intellisense still isn't intelligent enough to offer this feature.......
Marc
DECLARE #SQL VARCHAR(max), #TableName sysname = 'YourTableName'
SELECT #SQL = COALESCE(#SQL + ', ', '') + Name
FROM sys.columns
WHERE OBJECT_ID = OBJECT_ID(#TableName)
AND name NOT IN ('Not This', 'Or that');
SELECT #SQL = 'SELECT ' + #SQL + ' FROM ' + #TableName
EXEC (#SQL)
UPDATE:
You can also create a stored procedure to take care of this task if you use it more often.
In this example I have used the built in STRING_SPLIT() which is available on SQL Server 2016+,
but if you need there are pleanty of examples of how to create it manually on SO.
CREATE PROCEDURE [usp_select_without]
#schema_name sysname = N'dbo',
#table_name sysname,
#list_of_columns_excluded nvarchar(max),
#separator nchar(1) = N','
AS
BEGIN
DECLARE
#SQL nvarchar(max),
#full_table_name nvarchar(max) = CONCAT(#schema_name, N'.', #table_name);
SELECT #SQL = COALESCE(#SQL + ', ', '') + QUOTENAME([Name])
FROM sys.columns sc
LEFT JOIN STRING_SPLIT(#list_of_columns_excluded, #separator) ss ON sc.[name] = ss.[value]
WHERE sc.OBJECT_ID = OBJECT_ID(#full_table_name, N'u')
AND ss.[value] IS NULL;
SELECT #SQL = N'SELECT ' + #SQL + N' FROM ' + #full_table_name;
EXEC(#SQL)
END
And then just:
EXEC [usp_select_without]
#table_name = N'Test_Table',
#list_of_columns_excluded = N'ID, Date, Name';
no there is no way to do this. maybe you can create custom views if that's feasible in your situation
EDIT May be if your DB supports execution of dynamic sql u could write an SP and pass the columns u don't want to see to it and let it create the query dynamically and return the result to you. I think this is doable in SQL Server atleast
If you are using SQL Server Management Studio then do as follows:
Type in your desired tables name and select it
Press Alt+F1
o/p shows the columns in table.
Select the desired columns
Copy & paste those in your select query
Fire the query.
Enjoy.
If you want to exclude a sensitive case column like the password for example, I do this to hide the value :
SELECT * , "" as password FROM tableName;
The best way to solve this is using view you can create view with required columns and retrieve data form it
example
mysql> SELECT * FROM calls;
+----+------------+---------+
| id | date | user_id |
+----+------------+---------+
| 1 | 2016-06-22 | 1 |
| 2 | 2016-06-22 | NULL |
| 3 | 2016-06-22 | NULL |
| 4 | 2016-06-23 | 2 |
| 5 | 2016-06-23 | 1 |
| 6 | 2016-06-23 | 1 |
| 7 | 2016-06-23 | NULL |
+----+------------+---------+
7 rows in set (0.06 sec)
mysql> CREATE VIEW C_VIEW AS
-> SELECT id,date from calls;
Query OK, 0 rows affected (0.20 sec)
mysql> select * from C_VIEW;
+----+------------+
| id | date |
+----+------------+
| 1 | 2016-06-22 |
| 2 | 2016-06-22 |
| 3 | 2016-06-22 |
| 4 | 2016-06-23 |
| 5 | 2016-06-23 |
| 6 | 2016-06-23 |
| 7 | 2016-06-23 |
+----+------------+
7 rows in set (0.00 sec)
In summary you cannot do it, but I disagree with all of the comment above, there "are" scenarios where you can legitimately use a *
When you create a nested query in order to select a specific range out of a whole list (such as paging) why in the world would want to specify each column on the outer select statement when you have done it in the inner?
In SQL Management Studio you can expand the columns in Object Explorer, then drag the Columns tree item into a query window to get a comma separated list of columns.
If we are talking of Procedures, it works with this trick to generate a new query and EXECUTE IMMEDIATE it:
SELECT LISTAGG((column_name), ', ') WITHIN GROUP (ORDER BY column_id)
INTO var_list_of_columns
FROM ALL_TAB_COLUMNS
WHERE table_name = 'PUT_HERE_YOUR_TABLE'
AND column_name NOT IN ('dont_want_this_column','neither_this_one','etc_column');
Postgres sql has a way of doing it
pls refer:
http://www.postgresonline.com/journal/archives/41-How-to-SELECT-ALL-EXCEPT-some-columns-in-a-table.html
The Information Schema Hack Way
SELECT 'SELECT ' || array_to_string(ARRAY(SELECT 'o' || '.' || c.column_name
FROM information_schema.columns As c
WHERE table_name = 'officepark'
AND c.column_name NOT IN('officeparkid', 'contractor')
), ',') || ' FROM officepark As o' As sqlstmt
The above for my particular example table - generates an sql statement that looks like this
SELECT o.officepark,o.owner,o.squarefootage
FROM officepark As o
Is there a way to exclude column(s) from a table without specifying
all the columns?
Using declarative SQL in the usual way, no.
I think your proposed syntax is worthy and good. In fact, the relational database language 'Tutorial D' has a very similar syntax where the keywords ALL BUT are followed by a set of attributes (columns).
However, SQL's SELECT * already gets a lot a flak (#Guffa's answer here is a typical objection), so I don't think SELECT ALL BUT will get into the SQL Standard anytime soon.
I think the best 'work around' is to create a VIEW with only the columns you desire then SELECT * FROM ThatView.
I do not know of any database that supports this (SQL Server, MySQL, Oracle, PostgreSQL). It is definitely not part of the SQL standards so I think you have to specify only the columns you want.
You could of course build your SQL statement dynamically and have the server execute it. But this opens up the possibility for SQL injection..
Right click table in Object Explorer, Select top 1000 rows
It'll list all columns and not *. Then remove the unwanted column(s). Should be much faster than typing it yourself.
Then when you feel this is a bit too much work, get Red Gate's SQL Prompt, and type ssf from tbl, go to the * and click tab again.
I know this is a little old, but I had just run into the same issue and was looking for an answer. Then I had a senior developer show me a very simple trick.
If you are using the management studio query editor, expand the database, then expand the table that you are selecting from so that you can see the columns folder.
In your select statement, just highlight the referenced columns folder above and drag and drop it into the query window. It will paste all of the columns of the table, then just simply remove the identity column from the list of columns...
A colleage advised a good alternative:
Do SELECT INTO in your preceding query (where you generate or get the
data from) into a table (which you will delete when done). This will
create the structure for you.
Do a script as CREATE to new query
window.
Remove the unwanted columns. Format the remaining columns
into a 1 liner and paste as your column list.
Delete the table you
created.
Done...
This helped us a lot.
Actually snowflake just released exclude so now you'd just:
SELECT * EXCLUDE [columnA,columnB,...] FROM tableA
Well, it is a common best practice to specify which columns you want, instead of just specifying *. So you should just state which fields you want your select to return.
That what I use often for this case:
declare #colnames varchar(max)=''
select #colnames=#colnames+','+name from syscolumns where object_id(tablename)=id and name not in (column3,column4)
SET #colnames=RIGHT(#colnames,LEN(#colnames)-1)
#colnames looks like column1,column2,column5
I did it like this and it works just fine (version 5.5.41):
# prepare column list using info from a table of choice
SET #dyn_colums = (SELECT REPLACE(
GROUP_CONCAT(`COLUMN_NAME`), ',column_name_to_remove','')
FROM `INFORMATION_SCHEMA`.`COLUMNS` WHERE
`TABLE_SCHEMA`='database_name' AND `TABLE_NAME`='table_name');
# set sql command using prepared columns
SET #sql = CONCAT("SELECT ", #dyn_colums, " FROM table_name");
# prepare and execute
PREPARE statement FROM #sql;
EXECUTE statement;
Sometimes the same program must handle different database stuctures. So I could not use a column list in the program to avoid errors in select statements.
* gives me all the optional fields. I check if the fields exist in the data table before use. This is my reason for using * in select.
This is how I handle excluded fields:
Dim da As New SqlDataAdapter("select * from table", cn)
da.FillSchema(dt, SchemaType.Source)
Dim fieldlist As String = ""
For Each DC As DataColumn In DT.Columns
If DC.ColumnName.ToLower <> excludefield Then
fieldlist = fieldlist & DC.Columnname & ","
End If
Next
In Hive Sql you can do this:
set hive.support.quoted.identifiers=none;
select
`(unwanted_col1|unwanted_col2|unwanted_col3)?+.+`
from database.table
this gives you the rest cols
The proposed answer (stored procedure) from BartoszX didn't work for me when using a view instead of a real table.
Credit for the idea and the code below (except for my fix) belongs to BartoszX.
In order that this works for tables as well as for views, use the following code:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[select_without]
#schema_name sysname = N'dbo',
#table_name sysname,
#list_of_columns_excluded nvarchar(max),
#separator nchar(1) = N','
AS
BEGIN
DECLARE
#SQL nvarchar(max),
#full_table_name nvarchar(max) = CONCAT(#schema_name, N'.', #table_name);
SELECT #SQL = COALESCE(#SQL + ', ', '') + QUOTENAME([Name])
FROM sys.columns sc
LEFT JOIN STRING_SPLIT(#list_of_columns_excluded, #separator) ss ON sc.[name] = ss.[value]
WHERE sc.OBJECT_ID = OBJECT_ID(#full_table_name)
AND ss.[value] IS NULL;
SELECT #SQL = N'SELECT ' + #SQL + N' FROM ' + #full_table_name;
EXEC(#SQL)
END
GO
Is there any RDBMS that implements something like SELECT * EXCEPT? What I'm after is getting all of the fields except a specific TEXT/BLOB field, and I'd like to just select everything else.
Almost daily I complain to my coworkers that someone should implement this... It's terribly annoying that it doesn't exist.
Edit: I understand everyone's concern for SELECT *. I know the risks associated with SELECT *. However, this, at least in my situation, would not be used for any Production level code, or even Development level code; strictly for debugging, when I need to see all of the values easily.
As I've stated in some of the comments, where I work is strictly a commandline shop, doing everything over ssh. This makes it difficult to use any gui tools (external connections to the database aren't allowed), etc etc.
Thanks for the suggestions though.
As others have said, it is not a good idea to do this in a query because it is prone to issues when someone changes the table structure in the future. However, there is a way to do this... and I can't believe I'm actually suggesting this, but in the spirit of answering the ACTUAL question...
Do it with dynamic SQL... this does all the columns except the "description" column. You could easily turn this into a function or stored proc.
declare #sql varchar(8000),
#table_id int,
#col_id int
set #sql = 'select '
select #table_id = id from sysobjects where name = 'MY_Table'
select #col_id = min(colid) from syscolumns where id = #table_id and name <> 'description'
while (#col_id is not null) begin
select #sql = #sql + name from syscolumns where id = #table_id and colid = #col_id
select #col_id = min(colid) from syscolumns where id = #table_id and colid > #col_id and name <> 'description'
if (#col_id is not null) set #sql = #sql + ','
print #sql
end
set #sql = #sql + ' from MY_table'
exec #sql
Create a view on the table which doesn't include the blob columns
Is there any RDBMS that implements something like SELECT * EXCEPT?
Yes, Google Big Query implements SELECT * EXCEPT:
A SELECT * EXCEPT statement specifies the names of one or more columns to exclude from the result. All matching column names are omitted from the output.
WITH orders AS(
SELECT 5 as order_id,
"sprocket" as item_name,
200 as quantity
)
SELECT * EXCEPT (order_id)
FROM orders;
Output:
+-----------+----------+
| item_name | quantity |
+-----------+----------+
| sprocket | 200 |
+-----------+----------+
EDIT:
H2 database also supports SELECT * EXCEPT (col1, col2, ...) syntax.
Wildcard expression
A wildcard expression in a SELECT statement. A wildcard expression represents all visible columns. Some columns can be excluded with optional EXCEPT clause.
EDIT 2:
Hive supports: REGEX Column Specification
A SELECT statement can take regex-based column specification in Hive releases prior to 0.13.0, or in 0.13.0 and later releases if the configuration property hive.support.quoted.identifiers is set to none.
The following query selects all columns except ds and hr.
SELECT `(ds|hr)?+.+` FROM sales
EDIT 3:
Snowflake also now supports: SELECT * EXCEPT (and a RENAME option equivalent to REPLACE in BigQuery)
EXCLUDE col_name EXCLUDE (col_name, col_name, ...)
When you select all columns (SELECT *), specifies the columns that should be excluded from the results.
RENAME col_name AS col_alias RENAME (col_name AS col_alias, col_name AS col_alias, ...)
When you select all columns (SELECT *), specifies the column aliases that should be used in the results.
and so does Databricks SQL (since Runtime 11.0)
star_clause
[ { table_name | view_name } . ] * [ except_clause ]
except_clause
EXCEPT ( { column_name | field_name } [, ...] )
and also DuckDB
-- select all columns except the city column from the addresses table
SELECT * EXCLUDE (city) FROM addresses;
-- select all columns from the addresses table, but replace city with LOWER(city)
SELECT * REPLACE (LOWER(city) AS city) FROM addresses;
-- select all columns matching the given regex from the table
SELECT COLUMNS('number\d+') FROM addresses;
DB2 allows for this. Columns have an attribute/specifier of Hidden.
From the syscolumns documentation
HIDDEN
CHAR(1) NOT NULL WITH DEFAULT 'N'
Indicates whether the column is implicitly hidden:
P Partially hidden. The column is implicitly hidden from SELECT *.
N Not hidden. The column is visible to all SQL statements.
Create table documentation As part of creating your column, you would specify the IMPLICITLY HIDDEN modifier
An example DDL from Implicitly Hidden Columns follows
CREATE TABLE T1
(C1 SMALLINT NOT NULL,
C2 CHAR(10) IMPLICITLY HIDDEN,
C3 TIMESTAMP)
IN DB.TS;
Whether this capability is such a deal maker to drive the adoption of DB2 is left as an exercise to future readers.
Is there any RDBMS that implements something like SELECT * EXCEPT
Yes! The truly relational language Tutorial D allows projection to be expressed in terms of the attributes to be removed instead of the ones to be kept e.g.
my_relvar { ALL BUT description }
In fact, its equivalent to SQL's SELECT * is { ALL BUT }.
Your proposal for SQL is a worthy one but I heard it has already been put to the SQL standard's committee by the users' group and rejected by the vendor's group :(
It has also been explicitly requested for SQL Server but the request was closed as 'won't fix'.
Yes, finally there is :) SQL Standard 2016 defines Polymorphic Table Functions
SQL:2016 introduces polymorphic table functions (PTF) that don't need to specify the result type upfront. Instead, they can provide a describe component procedure that determines the return type at run time. Neither the author of the PTF nor the user of the PTF need to declare the returned columns in advance.
PTFs as described by SQL:2016 are not yet available in any tested database.10 Interested readers may refer to the free technical report “Polymorphic table functions in SQL” released by ISO. The following are some of the examples discussed in the report:
CSVreader, which reads the header line of a CVS file to determine the number and names of the return columns
Pivot (actually unpivot), which turns column groups into rows (example: phonetype, phonenumber) -- me: no more harcoded strings :)
TopNplus, which passes through N rows per partition and one extra row with the totals of the remaining rows
Oracle 18c implements this mechanism. 18c Skip_col Polymorphic Table Function Example Oracle Live SQL and Skip_col Polymorphic Table Function Example
This example shows how to skip data based on name/specific datatype:
CREATE PACKAGE skip_col_pkg AS
-- OVERLOAD 1: Skip by name
FUNCTION skip_col(tab TABLE, col columns)
RETURN TABLE PIPELINED ROW POLYMORPHIC USING skip_col_pkg;
FUNCTION describe(tab IN OUT dbms_tf.table_t,
col dbms_tf.columns_t)
RETURN dbms_tf.describe_t;
-- OVERLOAD 2: Skip by type --
FUNCTION skip_col(tab TABLE,
type_name VARCHAR2,
flip VARCHAR2 DEFAULT 'False')
RETURN TABLE PIPELINED ROW POLYMORPHIC USING skip_col_pkg;
FUNCTION describe(tab IN OUT dbms_tf.table_t,
type_name VARCHAR2,
flip VARCHAR2 DEFAULT 'False')
RETURN dbms_tf.describe_t;
END skip_col_pkg;
and body:
CREATE PACKAGE BODY skip_col_pkg AS
/* OVERLOAD 1: Skip by name
* NAME: skip_col_pkg.skip_col
* ALIAS: skip_col_by_name
*
* PARAMETERS:
* tab - The input table
* col - The name of the columns to drop from the output
*
* DESCRIPTION:
* This PTF removes all the input columns listed in col from the output
* of the PTF.
*/
FUNCTION describe(tab IN OUT dbms_tf.table_t,
col dbms_tf.columns_t)
RETURN dbms_tf.describe_t
AS
new_cols dbms_tf.columns_new_t;
col_id PLS_INTEGER := 1;
BEGIN
FOR i IN 1 .. tab.column.count() LOOP
FOR j IN 1 .. col.count() LOOP
tab.column(i).pass_through := tab.column(i).description.name != col(j);
EXIT WHEN NOT tab.column(i).pass_through;
END LOOP;
END LOOP;
RETURN NULL;
END;
/* OVERLOAD 2: Skip by type
* NAME: skip_col_pkg.skip_col
* ALIAS: skip_col_by_type
*
* PARAMETERS:
* tab - Input table
* type_name - A string representing the type of columns to skip
* flip - 'False' [default] => Match columns with given type_name
* otherwise => Ignore columns with given type_name
*
* DESCRIPTION:
* This PTF removes the given type of columns from the given table.
*/
FUNCTION describe(tab IN OUT dbms_tf.table_t,
type_name VARCHAR2,
flip VARCHAR2 DEFAULT 'False')
RETURN dbms_tf.describe_t
AS
typ CONSTANT VARCHAR2(1024) := upper(trim(type_name));
BEGIN
FOR i IN 1 .. tab.column.count() LOOP
tab.column(i).pass_through :=
CASE upper(substr(flip,1,1))
WHEN 'F' THEN dbms_tf.column_type_name(tab.column(i).description)
!=typ
ELSE dbms_tf.column_type_name(tab.column(i).description)
=typ
END /* case */;
END LOOP;
RETURN NULL;
END;
END skip_col_pkg;
And sample usage:
-- skip number cols
SELECT * FROM skip_col_pkg.skip_col(scott.dept, 'number');
-- only number cols
SELECT * FROM skip_col_pkg.skip_col(scott.dept, 'number', flip => 'True')
-- skip defined columns
SELECT *
FROM skip_col_pkg.skip_col(scott.emp, columns(comm, hiredate, mgr))
WHERE deptno = 20;
I highly recommend to read entire example(creating standalone functions instead of package calls).
You could easily overload skip method for example: skip columns that does not start/end with specific prefix/suffix.
db<>fidde demo
Related: How to Dynamically Change the Columns in a SQL Query By Chris Saxon
Stay away from SELECT *, you are setting yourself for trouble. Always specify exactly which columns you want. It is in fact quite refreshing that the "feature" you are asking for doesn't exist.
I believe the rationale for it not existing is that the author of a query should (for performance sake) only request what they're going to look at/need (and therefore know what columns to specify) -- if someone adds a couple more blobs in the future, you'd be pulling back potentially large fields you're not going to need.
Temp table option here, just drop the columns not required and select * from the altered temp table.
/* Get the data into a temp table */
SELECT * INTO #TempTable
FROM
table
/* Drop the columns that are not needed */
ALTER TABLE #TempTable
DROP COLUMN [columnname]
SELECT * from #TempTable
declare #sql nvarchar(max)
#table char(10)
set #sql = 'select '
set #table = 'table_name'
SELECT #sql = #sql + '[' + COLUMN_NAME + '],'
FROM INFORMATION_SCHEMA.Columns
WHERE TABLE_NAME = #table
and COLUMN_NAME <> 'omitted_column_name'
SET #sql = substring(#sql,1,len(#sql)-1) + ' from ' + #table
EXEC (#sql);
I needed something like what #Glen asks for easing my life with HASHBYTES().
My inspiration was #Jasmine and #Zerubbabel answers. In my case I've different schemas, so the same table name appears more than once at sys.objects. As this may help someone with the same scenario, here it goes:
ALTER PROCEDURE [dbo].[_getLineExceptCol]
#table SYSNAME,
#schema SYSNAME,
#LineId int,
#exception VARCHAR(500)
AS
DECLARE #SQL NVARCHAR(MAX)
BEGIN
SET NOCOUNT ON;
SELECT #SQL = COALESCE(#SQL + ', ', ' ' ) + name
FROM sys.columns
WHERE name <> #exception
AND object_id = (SELECT object_id FROM sys.objects
WHERE name LIKE #table
AND schema_id = (SELECT schema_id FROM sys.schemas WHERE name LIKE #schema))
SELECT #SQL = 'SELECT ' + #SQL + ' FROM ' + #schema + '.' + #table + ' WHERE Id = ' + CAST(#LineId AS nvarchar(50))
EXEC(#SQL)
END
GO
It's an old question, but I hope this answer can still be helpful to others. It can also be modified to add more than one except fields. This can be very handy if you want to unpivot a table with many columns.
DECLARE #SQL NVARCHAR(MAX)
SELECT #SQL = COALESCE(#SQL + ', ', ' ' ) + name FROM sys.columns WHERE name <> 'colName' AND object_id = (SELECT id FROM sysobjects WHERE name = 'tblName')
SELECT #SQL = 'SELECT ' + #SQL + ' FROM ' + 'tblName'
EXEC sp_executesql #SQL
Stored Procedure:
usp_SelectAllExcept 'tblname', 'colname'
ALTER PROCEDURE [dbo].[usp_SelectAllExcept]
(
#tblName SYSNAME
,#exception VARCHAR(500)
)
AS
DECLARE #SQL NVARCHAR(MAX)
SELECT #SQL = COALESCE(#SQL + ', ', ' ' ) + name from sys.columns where name <> #exception and object_id = (Select id from sysobjects where name = #tblName)
SELECT #SQL = 'SELECT ' + #SQL + ' FROM ' + #tblName
EXEC sp_executesql #SQL
For the sake of completeness, this is possible in DremelSQL dialect, doing something like:
WITH orders AS
(SELECT 5 as order_id,
"foobar12" as item_name,
800 as quantity)
SELECT * EXCEPT (order_id)
FROM orders;
Result:
+-----------+----------+
| item_name | quantity |
+-----------+----------+
| foobar12 | 800 |
+-----------+----------+
There also seems to be another way to do it here without Dremel.
Your question was about what RDBMS supports the * EXCEPT (...) syntax, so perhaps, looking at the jOOQ manual page for * EXCEPT can be useful in the future, as that page will keep track of new dialects supporting the syntax.
Currently (mid 2022), among the jOOQ supported RDBMS, at least BigQuery, H2, and Snowflake support the syntax natively. The others need to emulate it by listing the columns explicitly:
-- ACCESS, ASE, AURORA_MYSQL, AURORA_POSTGRES, COCKROACHDB, DB2, DERBY, EXASOL,
-- FIREBIRD, HANA, HSQLDB, INFORMIX, MARIADB, MEMSQL, MYSQL, ORACLE, POSTGRES,
-- REDSHIFT, SQLDATAWAREHOUSE, SQLITE, SQLSERVER, SYBASE, TERADATA, VERTICA,
-- YUGABYTEDB
SELECT LANGUAGE.CD, LANGUAGE.DESCRIPTION
FROM LANGUAGE
-- BIGQUERY, H2
SELECT * EXCEPT (ID)
FROM LANGUAGE
-- SNOWFLAKE
SELECT * EXCLUDE (ID)
FROM LANGUAGE
Disclaimer: I work for the company behind jOOQ
As others are saying: SELECT * is a bad idea.
Some reasons:
Get only what you need (anything more is a waste)
Indexing (index what you need and you can get it more quickly. If you ask for a bunch of non-indexed columns, too, your query plans will suffer.