Memo fields showing chinese characters or symbols instead of actual value when records number reach a certain point - sql

This is my first post here and I am a database newbie. I tried to find an answer, but I'm not sure I understand how it applies to my case.
My access program generates a query with different types of fields (Autonumber, shorttext, memo) which is then used to create a report.
It has been working fine until now, but since the DB has grown I run into a problem.
I use the ID (primary key) IN () the where condition to filter the report. I make a long string that get all the id of the selected records:
WHERE ID IN (1200,1201,1203,1226,1227,1228,1229,...)
When a certain amount of characters in the query is reached (around 4000), I get Chinese characters instead of all the memo fields, and only the memo fields, of the query results (and then in the report).
Is there a limit in the query size? Isn't it 32000 characters?
Why do these characters shows only if I select too many records?
Is there a substite to IN () that could help me reduce the query lengh or should I completely avoid memo fields?
EDIT : That's the query (stripped down a little to be readable) :
SELECT ObjetsLegislatifs.IDobjet, ObjetsLegislatifs.TitreObjet, ObjetsLegislatifs.ContenuObjet
FROM ObjetsLegislatifs
WHERE IDobjet IN(1200,1201,1203,1226,1227,1228,1229,1230,1231,1232,)
GROUP BY ObjetsLegislatifs.IDobjet, ObjetsLegislatifs.TitreObjet, ObjetsLegislatifs.ContenuObjet
ORDER BY ObjetsLegislatifs.IDobjet;
Basicaly, th IDobjet is a autonumber, the "TitreObjet" and "ContenuObjet" are Memos fields.
While IDobjet always shows the proper number, the memo fields start showing chinese when the query is too long, when a certain threshold is reached. I tried with a text field instead in the query and they work fine.

The best thing for you to do is create a temporary table (which is a special type of table - see here) with a single ID column.
When generating the query, you can then insert your ID into this table and join to it instead of using an IN list, something like this:
select a.*
from table a
INNER JOIN temp_table b ON a.ID = b.ID

Related

How to return all records where id contains a certain sequence of some digits PostgreSQL query

I have a query that brings back a value based on the user id. The id is int8. How could I make the query to return all the records that contain a certain sequence of the numbers, in other words.
the full user id presented as..
where up.user_id in (7401195021087888999)
and then where u only have a segment of the user id
where up.user_id Contains(74011950210878)
to bring back several records (hopefully). Contains is not correct, just to explain.
Hope this makes sense
One option would be to cast the integer column to text and then use LIKE:
SELECT *
FROM yourTable
WHERE user_id::text LIKE '%74011950210878%';
Note that your requirement might imply that the user_id column should actually be a text column, and not numeric.

optimizing PGSQL SQL search queries on big texts ('like', full text search, ... )

We have a software solution which is used by +200 customers. We recently switched to pgsql, because our former database was too slow handeling the search queries our customers use.
Our dabatabase looks like this:
TABLE A
1. ID
(+ some other fields which aren't important here)
TABLE B
This table is used to store 'data' on the items in table A. This is different for every customer. For example 'Type' can be 'CLIENTNAME' and value 'AZERTY'. One record on TABLE A can have infinite records in TABLE B. Mostly 1 record in Table A has between 5 - 10 records on Table B.
1. ID TABLE A
2. TYPE
3. VALUE
TABLE C
1. TABLE A ID
2. VERSIONNR
3. DESCRIPTION
This file has the different verions of the records in TABLE A. Each of these versions has an extended description. This can range from 0 characters to infinite.
Our problem: our customers are used on 'google-like' searching. For example: they type 'AZERTY' and we show all the records from TABLE A where the ID of TABLE A:
'AZERTY' is in the description of the most recent version of TABLE C
'AZERTY' is in one of the values of TABLE B
Additional problem: this search is a 'contains'. If they search 'ZER', they should also find the records with 'AZERTY' in it. Multiple arguments are an 'AND', if they search for 'ZER 123', we need to show all records where the description matches 'ZER' and '123' or the values match 'ZER' and '123'.
What we have done so far:
There is an option a user can check in/out whether they want to search the description or not. We mosty advice them to only search for the values and only use the description in case of need.
We make several search threads to the database for one search query, because searching all documents at once would take too much time.
Some time ago, on our former slow database engine, a collegue of mine made 'search tables', basically this is a table which contains all values on a TABLE A ID so there isn't need for any join in the SQL query when searching. It looks like this:
TABLE D
TABLE A ID
VALUES (all values from TABLE B for this TABLE A ID, seperated by a ' ')
DESCRIPTION (the description of the most recent version for this TABLE A ID)
Example record:
- 1
- ZER 123 CLIENT NAME NUMBER 7856 jsdfjklf 4556423
- DESCRIPTION CAN BE VERY LONG.
If a customer searches for 'ZER 123' this becomes:
"select TABLE_A_ID from TABLE_D where values like '%ZER%' and values like '%123%'"
Important:
Some of our customers have alot of records in TABLE A. +5.000.000, which means there are alot of records in TABLE B (+/- 50.000.000). Most of our customers have between 300.000 and 500.000 records in TABLE A.
My questions:
Is there a better / faster way to search through all the values then that search table? Without the search table i would have to make a join for every ' ' in the search argument of the customer, which will work too slow (i think?) if they have alot of records in TABLE A. For example:
select ID from TABLE_A
INNER JOIN TABLE_B Sub1 ON TABLE_A.ID = Sub1.TABLE_A_ID and Sub1.VALUE like '%ZER%'
INNER JOIN TABLE_B Sub2 on FILE_A.ID = Sub2.TABLE_A_ID and Sub2.VALUE like '%123%'
I have taken a look at the full text search in PGSQL. I don't think i can use it since you can't use it as like (= 'contains') ?
Is there any index I can use on the values (FILE B or search file) and description (FILE C or search file) to make the searches faster? I've read on it and i don't think there is any, because indexes aren't used when searching with "like '%ZER%'" ?
I hope i've explained this cleary.
Thanks in advance!
Your terminology is confusing, but I assume you mean "tables" when you write "files".
You cannot reasonably search in several tables with a single query, but you can search in several columns of a single table at the same time.
Based on your description, I would say that you need a trigram index on the concatenation of the relevant string columns in the table.

SQL Server match partial text in the comma separated varchar column

I have a VARCHAR column category_text in the table that contain tags to a notification stored. I have three tags Query, Complaint and Suggestion and column can have one or more values separated by comma. I am applying a filter and filter can have one or more values as well in comma separated pattern.
Now what I want is to retrieve all the rows that contain at least one tag based on the filter user is applying, for instance user can select 'query,suggestion' as a filter and result would be all the rows that contain one of the tags i.e. query or suggestion.
select
t.category_text
from
real_time_notifications t
where
charindex('query, suggestion, complaints', t.category_text) > 0
order by
t.id desc
Create a new table, like user_category (user.id link to user table, category) and create an index on both. It will speed up a lot for searching and ease your future maintenance a lot.
If you still persist to do that, create an inline function to split string to records and then merge to test.

SQL or statement vs multiple select queries

I'm having a table with an id and a name.
I'm getting a list of id's and i need their names.
In my knowledge i have two options.
Create a forloop in my code which executes:
SELECT name from table where id=x
where x is always a number.
or I'm write a single query like this:
SELECT name from table where id=1 OR id=2 OR id=3
The list of id's and names is enormous so i think you wouldn't want that.
The problem of id's is the id is not always a number but a random generated id containting numbers and characters. So talking about ranges is not a solution.
I'm asking this in a performance point of view.
What's a nice solution for this problem?
SQLite has limits on the size of a query, so if there is no known upper limit on the number of IDs, you cannot use a single query.
When you are reading multiple rows (note: IN (1, 2, 3) is easier than many ORs), you don't know to which ID a name belongs unless you also SELECT that, or sort the results by the ID.
There should be no noticeable difference in performance; SQLite is an embedded database without client/server communication overhead, and the query does not need to be parsed again if you use a prepared statement.
A "nice" solution is using the INoperator:
SELECT name from table where id in (1,2,3)
Also, the IN operator is syntactic sugar built for exactly this purpose..
SELECT name from table where id IN (1,2,3,4,5,6.....)
Hoping that you are getting the list of ID's on which you have to perform a query for names as input temp table #InputIDTable,
SELECT name from table WHERE ID IN (SELECT id from #InputIDTable)

SQL function CONTAINS() does not return expected results?

I have a table 'Asset' with a column 'AssetDescription'. Every row of it has some group of words/sentences, seprated by comma.
row1: - flowers, full color, female, Trend
row2:- baby smelling flowers, heart
Now if I put a search query like:-
select * from Asset where contains(AssetDescription,'flower')
It returns nothing.
I have one more table 'SearchData' with column 'SearchCol', having similar rows as mentioned above in table 'Asset'.
Now if a put a search query like:-
select * from SearchData where contains(SearchCol,'flower')
It returns both the rows.
QUESTION:-
Why first query doesn't return any result, but second one does correctly.
If 'Full Text Search' has something to do with 1st ques, than what to do regarding that. As I'm using SQL server 2000.
Clearing a comment doubt on my question:-
Table 'SearchData' has more than 100,000 rows and so as the table 'Asset'.
Those two tables are NOT identical. But their respective columns has rows that contains some group of words seperated by commas. (So flowers, flower etc etc are in plenty in both of those columns.)
Screenshot of the Indexes of both the tables (Asset and SearchData):-
it probably has something to do with your full-text index configuration.
Can you post some info on your index and catalog?
If you read the article on CONTAINS you will see that when searching for
contains(AssetDescription,'flower')
'flower' is treated as a simple term which
matches an exact word or phrase
However for
contains(AssetDescription,'flower*')
'flower' is treated as a prefix term which
specifies a match of words or phrases beginning with the specified
text
and will match 'flowers' in your data.
So, are you sure that data in your two tables is the same, or does 'Asset' contain 'flowers' and 'SearchData' contain 'flower'?
GOT THE SOLUTION.
THANKS TO ALL and especially DIEGO for support.
To allow the FULL TEXT SEARCH (FTS) to work properly:-
Enable FTS for the required table.
Enable FTS for the required column under that table.
Open the properties for the same table and check if attributes 'Full-text change tracking' and 'Full-text update-index' are enabled. If not, enable them.
DONE.
: )