I'm using sql server 2010 that contains bulk amount of data. i am facing error while updating the siz of one of the field in one of table.
code
CREATE TABLE [dbo].[Ayyat_Translation_Language_old_20131209] (
[Ayat_Translation_Language_ID] INT IDENTITY (1, 1) NOT NULL,
[Translation_Laanguage_ID] INT NULL,
[Juz_ID] INT NULL,
[Surah_ID] INT NOT NULL,
[Ayat_Description] NVARCHAR (3900) COLLATE Arabic_CI_AI_KS_WS NOT NULL,
[Ayat_No] INT NULL,
PRIMARY KEY CLUSTERED ([Ayat_Translation_Language_ID] ASC),
CONSTRAINT [fkey2] FOREIGN KEY ([Translation_Laanguage_ID]) REFERENCES [dbo].[Translation_Language] ([TransLation_Language_ID]) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT [fkey0] FOREIGN KEY ([Juz_ID]) REFERENCES [dbo].[Juz] ([Juz_ID]),
CONSTRAINT [fkey1] FOREIGN KEY ([Surah_ID]) REFERENCES [dbo].[Surah] ([Surah_ID]) ON DELETE CASCADE ON UPDATE CASCADE
);
when i try to change the size of [Ayat_Description] from 3900 to 5000, it shows random error. how to fix it ?
nvarchar [ ( n | max ) ]
Variable-length Unicode string data. n defines the string length and can be a value from 1 through 4,000. max indicates that the maximum storage size is 2^31-1 bytes (2 GB). The storage size, in bytes, is two times the actual length of data entered + 2 bytes.
You will need to enter either 4000 or max, in case you need to store more than 4000 characters
Related
I am calculating Euclidean distance between faces and want to store results in a table.
Current setup :
Each face is stored in Objects table and Distances between faces is stored in Faces_distances table.
The object table has the following columns objects_id, face_encodings, description
The faces_distances table has the following columns face_from, face_to, distance
In my my data set I have around 22 231 face objects which result in 494 217 361 pairs of faces - Although I understand it could be divided by 2 because
distance(face_from, face_to) = distance(face_to, face_from)
The database is Postgres 12.
The request below enables to insert the pairs of faces (without performing the distance calculation) that have not been calculated yet, but the execution time is very very very long (started 4 Days ago and still not done). Is there a way to optimize it ?
'''
-- public.objects definition
-- Drop table
-- DROP TABLE public.objects;
CREATE TABLE public.objects
(
objects_id int4 NOT NULL DEFAULT
nextval('objects_in_image_objects_id_seq'::regclass),
filefullname varchar(2303) NULL,
bbox varchar(255) NULL,
description varchar(255) NULL,
confidence numeric NULL,
analyzer varchar(255) NOT NULL DEFAULT 'object_detector'::character
varying,
analyzer_version int4 NOT NULL DEFAULT 100,
x int4 NULL,
y int4 NULL,
w int4 NULL,
h int4 NULL,
image_id int4 NULL,
derived_from_object int4 NULL,
object_image_filename varchar(2023) NULL,
face_encodings _float8 NULL,
face_id int4 NULL,
face_id_iteration int4 NULL,
text_found varchar NULL COLLATE "C.UTF-8",
CONSTRAINT objects_in_image_pkey PRIMARY KEY (objects_id),
CONSTRAINT objects_in_images FOREIGN KEY (objects_id) REFERENCES
public.objects(objects_id)
);
CREATE TABLE public.face_distances
(
face_from int8 NOT NULL,
face_to int8 NOT NULL,
distance float8 NULL,
CONSTRAINT face_distances_pk PRIMARY KEY (face_from, face_to)
);
-- public.face_distances foreign keys
ALTER TABLE public.face_distances ADD CONSTRAINT face_distances_fk
FOREIGN KEY (face_from) REFERENCES public.objects(objects_id);
ALTER TABLE public.face_distances ADD CONSTRAINT face_distances_fk_1
FOREIGN KEY (face_to) REFERENCES public.objects(objects_id);
Indexes
CREATE UNIQUE INDEX objects_in_image_pkey ON public.objects USING btree (objects_id);
CREATE INDEX objects_description_column ON public.objects USING btree (description);
CREATE UNIQUE INDEX face_distances_pk ON public.face_distances USING btree (face_from, face_to);
Query to add all pair of faces that are not already in the table.
insert into face_distances (face_from,face_to)
select t1.face_from , t1.face_to
from (
select f_from.objects_id face_from,
f_from.face_encodings face_from_encodings,
f_to.objects_id face_to,
f_to.face_encodings face_to_encodings
from objects f_from,
objects f_to
where f_from.description = 'face'
and f_to.description = 'face' ) as t1
left join face_distances on (
t1.face_from= face_distances.face_from
and t1.face_to = face_distances.face_to )
where face_distances.face_from is null;
try this simplified query.
It took only 5 minutes on my apple M1, SQLServer, with 22231 objects 'face', generated 247.097.565 pairs, which is excatly C(22231,2) number. The syntax is compatible with postgressql.
optimizations: join instead of the old jointure way, ranking functions to remove duplicates permutations (A,B)=(B,A),
removed the last [left join face_distance]: an empty table to recompute is a lot faster than checking for existance as an index search key lookup would be initiated for each key pair
insert into face_distances (face_from,face_to)
select f1,f2
from(
select --only needed fields here as this will fill temporary tables
f1.objects_id f1
,f2.objects_id f2
,dense_rank()over(order by f1.objects_id) rank1
,rank()over(partition by f2.objects_id order by f1.objects_id) rank2
from objects f1
-- generates all permutations
join objects f2 on f2.objects_id <> f1.objects_id and f2.description = 'face'
where f1.description = 'face'
)a
where rank2 >= rank1 --removes duplicate permutations
I discovered the normal 3072 byte key limitation in MariaDB 10.3 while running this statement:
ALTER TABLE table1 ADD UNIQUE INDEX idx_my_composite_uniqueai_usr_email_uniq (table2_uuid ASC, table3_uuid ASC, text_column ASC)
Which gave the expected error:
ERROR 1071 (42000) at line 31: Specified key was too long; max key length is 3072 bytes
DDL for table1:
CREATE TABLE table1
(
uuid char(36) CHARACTER SET ascii NOT NULL,
table2_fk_uuid char(36) CHARACTER SET ascii NOT NULL,
table3_fk_uuid char(36) CHARACTER SET ascii NOT NULL,
text_field varchar(1024) DEFAULT NULL,
PRIMARY KEY (uuid),
KEY fk_table2_fk_uuid (table2_fk_uuid),
KEY fk_table3_fk_uuid (table3_fk_uuid),
CONSTRAINT fk_table1_table2 FOREIGN KEY (table2_uuid) REFERENCES table2 (uuid) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT fk_table1_table3 FOREIGN KEY (table3_uuid) REFERENCES table3 (uuid) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3;
My understanding is that the total key length for this unique key is 1024 * 4 (because text_field column is CHARSET=utfmb3) + 36 * 2 = 4168 bytes.
I realize that shortening the text_field column width if possible is the best way to stay within the key length limitation. However, prior to understanding this I upgraded my MariaDB server to 10.5 and also to 10.6 and the ALTER TABLE command executes just fine and the key is created.
Has the key limitation been expanded in the later db version? I'm not see that it did in the MariaDB documentation. If not, what else am I missing?
The limitation was removed back in 10.4. Release notes mention it :
"Unique indexes can be created on BLOB or TEXT fields (MDEV-371)"
I have a table with 300000 rows and when i run a simple query like
select * from diario_det;
it leaves 41041 ms to return rows. It's fine that? How i can optimize the query?
I use Postgresql 9.3 in Centos 7.
Here's is my table
CREATE TABLE diario_det
(
cod_empresa numeric(2,0) NOT NULL,
nro_asiento numeric(8,0) NOT NULL,
nro_secue_pase numeric(4,0) NOT NULL,
descripcion_pase character varying(150) NOT NULL,
monto_debe numeric(16,3),
monto_haber numeric(16,3),
estado character varying(1) NOT NULL,
cod_pcuenta character varying(15) NOT NULL,
cod_local numeric(2,0) NOT NULL,
cod_centrocosto numeric(4,0) NOT NULL,
cod_ejercicio numeric(4,0) NOT NULL,
nro_comprob character varying(15),
conciliado_por character varying(10),
CONSTRAINT fk_diario_det_cab FOREIGN KEY (cod_empresa, cod_local, cod_ejercicio, nro_asiento)
REFERENCES diario_cab (cod_empresa, cod_local, cod_ejercicio, nro_asiento) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION,
CONSTRAINT fk_diario_det_pc FOREIGN KEY (cod_empresa, cod_pcuenta)
REFERENCES plan_cuenta (cod_empresa, cod_pcuenta) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
)
WITH (
OIDS=TRUE
);
ALTER TABLE diario_det
OWNER TO postgres;
-- Index: pk_diario_det_ax
-- DROP INDEX pk_diario_det_ax;
CREATE INDEX pk_diario_det_ax
ON diario_det
USING btree
(cod_pcuenta COLLATE pg_catalog."default", cod_local, estado COLLATE pg_catalog."default");
Very roughly size of one row is 231 bytes, times 300000... It's 69300000 bytes (~69MB) that has to be transferred from server to client.
I think that 41 seconds is a bit long, but still the query has to be slow because of amount of data that has to be loaded from disk and transferred.
You can optimise query by
selecting just columns you that are going to use not all of them (if you need just cod_empresa it would reduce total amount of transferred data to ~1.2MB, but server would still have to iterate trough all records - slow)
filter only rows that are going to use - using WHERE on columns with indexes can really speed the query up
If you want to know what is happening in your query, play around with EXPLAIN and EXPLAIN EXECUTE.
Also, if you're running dedicated database server, be sure to configure it properly to use a lot of system resources.
I have a table with the following definition:
CREATE TABLE url_tracker (
id int not null identity(1, 1),
active bit not null,
install_date int not null,
partner_url nvarchar(512) not null,
local_url nvarchar(512) not null,
public_url nvarchar(512) not null,
primary key(id)
);
And I have a requirement that these three URLs always be unique - any individual URL can appear many times, but the combination of the three must be unique (for a given day).
Initially I thought I could do this:
CREATE UNIQUE INDEX uniques ON url_tracker
(install_date, partner_url, local_url, public_url);
However this gives me back the warning:
Warning! The maximum key length is 900 bytes. The index 'uniques' has maximum
length of 3076 bytes. For some combination of large values, the insert/update
operation will fail.
Digging around I learned about the INCLUDE argument to CREATE INDEX, but according to this question converting the command to use INCLUDE will not enforce uniqueness on the URLs.
CREATE UNIQUE INDEX uniques ON url_tracker (install_date)
INCLUDE (partner_url, local_url, public_url);
How can I enforce uniqueness on several relatively large nvarchar fields?
Resolution
So from the comments and answers and more research I'm concluding I can do this:
CREATE TABLE url_tracker (
id int not null identity(1, 1),
active bit not null,
install_date int not null,
partner_url nvarchar(512) not null,
local_url nvarchar(512) not null,
public_url nvarchar(512) not null,
uniquehash AS HashBytes('SHA1',partner_url+local_url+public_url) PERSISTED,
primary key(id)
);
CREATE UNIQUE INDEX uniques ON url_tracker (install_date,uniquehash);
Thoughts?
I would make a computed column with the hash of the URLs, then make a unique index/constraint on that. Consider making the hash a persisted computed column. It shouldn't have to be recalculated after insertion.
Following the ideas from the conversation in the comments. Assuming that you can change the datatype of the URL to be VARCHAR(900) (or NVARCHAR(450) if you really think you need Unicode URLs) and be happy with the limitation on the length of the URL, this solution could work. This also assumes SQL Server 2008 or better. Please, always specify what version you're working with; sql-server is not specific enough, since solutions can vary greatly depending on the version.
Setup:
USE tempdb;
GO
CREATE TABLE dbo.urls
(
id INT IDENTITY(1,1) PRIMARY KEY,
url VARCHAR(900) NOT NULL UNIQUE
);
CREATE TABLE dbo.url_tracker
(
id INT IDENTITY(1,1) PRIMARY KEY,
active BIT NOT NULL DEFAULT 1,
install_date DATE NOT NULL DEFAULT CURRENT_TIMESTAMP,
partner_url_id INT NOT NULL REFERENCES dbo.urls(id),
local_url_id INT NOT NULL REFERENCES dbo.urls(id),
public_url_id INT NOT NULL REFERENCES dbo.urls(id),
CONSTRAINT unique_urls UNIQUE
(
install_date,partner_url_id, local_url_id, public_url_id
)
);
Insert some URLs:
INSERT dbo.urls(url) VALUES
('http://msn.com/'),
('http://aol.com/'),
('http://yahoo.com/'),
('http://google.com/'),
('http://gmail.com/'),
('http://stackoverflow.com/');
Now let's insert some data:
-- succeeds:
INSERT dbo.url_tracker(partner_url_id, local_url_id, public_url_id)
VALUES (1,2,3), (2,3,4), (3,4,5), (4,5,6);
-- fails:
INSERT dbo.url_tracker(partner_url_id, local_url_id, public_url_id)
VALUES(1,2,3);
GO
/*
Msg 2627, Level 14, State 1, Line 3
Violation of UNIQUE KEY constraint 'unique_urls'. Cannot insert duplicate key
in object 'dbo.url_tracker'. The duplicate key value is (2011-09-15, 1, 2, 3).
The statement has been terminated.
*/
-- succeeds, since it's for a different day:
INSERT dbo.url_tracker(install_date, partner_url_id, local_url_id, public_url_id)
VALUES('2011-09-01',1,2,3);
Cleanup:
DROP TABLE dbo.url_tracker, dbo.urls;
Now, if 900 bytes is not enough, you could change the URL table slightly:
CREATE TABLE dbo.urls
(
id INT IDENTITY(1,1) PRIMARY KEY,
url VARCHAR(2048) NOT NULL,
url_hash AS CONVERT(VARBINARY(32), HASHBYTES('SHA1', url)) PERSISTED,
CONSTRAINT unique_url UNIQUE(url_hash)
);
The rest doesn't have to change. And if you try to insert the same URL twice, you get a similar violation, e.g.
INSERT dbo.urls(url) SELECT 'http://www.google.com/';
GO
INSERT dbo.urls(url) SELECT 'http://www.google.com/';
GO
/*
Msg 2627, Level 14, State 1, Line 1
Violation of UNIQUE KEY constraint 'unique_url'. Cannot insert duplicate key
in object 'dbo.urls'. The duplicate key value is
(0xd111175e022c19f447895ad6b72ff259552d1b38).
The statement has been terminated.
*/
After fixing Error 1253 (MySQL: Unable to fulling forward engineering Sakila (sample) into server), I have Error 1628.
Executing SQL script in server
ERROR: Error 1628: Comment for table 'customer' is too long (max = 60)
Scripts:
CREATE TABLE IF NOT EXISTS `sakila`.`customer` (
`customer_id` SMALLINT(5) UNSIGNED NOT NULL AUTO_INCREMENT ,
`store_id` TINYINT(3) UNSIGNED NOT NULL ,
`first_name` VARCHAR(45) NOT NULL ,
`last_name` VARCHAR(45) NOT NULL ,
`email` VARCHAR(50) NULL DEFAULT NULL ,
`address_id` SMALLINT(5) UNSIGNED NOT NULL ,
`active` TINYINT(1) NOT NULL DEFAULT TRUE ,
`create_date` DATETIME NOT NULL ,
`last_update` TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP ,
PRIMARY KEY (`customer_id`) ,
INDEX `idx_fk_store_id` (`store_id` ASC) ,
INDEX `idx_fk_address_id` (`address_id` ASC) ,
INDEX `idx_last_name` (`last_name` ASC) ,
CONSTRAINT `fk_customer_address`
FOREIGN KEY (`address_id` )
REFERENCES `sakila`.`address` (`address_id` )
ON DELETE RESTRICT
ON UPDATE CASCADE,
CONSTRAINT `fk_customer_store`
FOREIGN KEY (`store_id` )
REFERENCES `sakila`.`store` (`store_id` )
ON DELETE RESTRICT
ON UPDATE CASCADE)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COMMENT 'Table storing all customers. Holds foreign keys to the address table and the store table where this customer is registered.\n\nBasic information about the customer like first and last name are stored in the table itself. Same for the date the record was created and when the information was last updated.'
SQL script execution finished: statements: 3 succeeded, 1 failed
As an addition: More current versions (5.6.X) allow longer comments. Unfortunately this length differs from the type of comment:
For tables: "A comment for the table, up to 2048 characters long."
For columns: "A comment for a column can be specified with the COMMENT option, up to 1024 characters long."
For INDEX: "In MySQL 5.6, index definitions can include an optional comment of up to 1024 characters."
For PARTITION: "Beginning with MySQL 5.6.6, the maximum length for a partition comment is 1024 characters. (Previously, this limit was not explicitly defined.)"
Source: http://dev.mysql.com/doc/refman/5.6/en/create-table.html
As stated in the MySQL docs, a comment is limited to 255 characters: http://dev.mysql.com/doc/refman/5.1/en/create-table.html#id3411882. Your comment is 305 characters, and it would seem, from the error message, that your particular MySQL install has a 60 character limit.