I'm not sure if it is possible to do what I'm trying to do, but I thought i would give it a shot anyway. Also, I am fairly new to the SQL Server world and this is my fist post, so I apologize if my wording is poor or if I leave information out. Also, I am working with SQL Server 2005.
Let's say I have a table named "table" and a column named "column" The contents of column is a jumbled mess of characters (ntext data type). These characters were all drawn in from multiple entry fields in a front end application. Now one of those entry fields was for sensitive information that we no longer need and would like to get rid of but I can't just get rid of the whole column because it also contains other valuable information. Most of the solutions I have found so far only deal with columns that have short entries so they are just able to update the whole string, but for mine I think I need to identify the the beginning and the end of the specific substring that I need and replace it or delete it somehow. This is the closest I have gotten to at least selecting the data that I need... AAA and /AAA mark the beginning and the end of the substring that I need.
select
substring (column, charindex ('AAA', column), charindex ('/AAA',column))
from table
where column like '%/AAA%'
The problems I am having with this one are that the substring doesn't stop at /AAA, it just keeps going, and some of the results are just blank so it looks something like:
AAA 12345 /AAA abcdefghijklmnop
AAA 12346 /AAA qrstuvwxyzabcdef
AAA 12347 /AAA abcdefghijklmnop
With the characters in bold being the information I need to get rid of. Also even though row 3 is blank, it still does contain the info that I need but I'm guessing that it isn't returning it because it has a different amount of characters before it (for example, rows 1, 2, and 4 might have 50 characters before them but row 3 might have 100 characters before it), at least that's the only reason that I could think of.
So I suppose the first step would probably be to actually select the right substring, then to either delete it or replace it with a different, meaningless substring like "111111" or something.
If there is more information that you need to be provided with or if I was unclear about anything please let me know and thank you for taking the time to read through (and hopefully answer) my question!
Edit: Another one that gets close to the right results goes something like this
select substring(column,charindex('AAA',column),20) from table
where column like '%/AAA%'
I'm not sure if this approach would work better since the substring i'm looking for is always going to have the same amount of characters. The problem with this one though, is that instead of having blank rows, they are replaced with irrelevant substrings from that column, but all of the other rows do return exactly what I want.
First of all, check your usage of SUBSTRING(). The third argument is for length, not end character, so you would need to alter your query to something like:
select substring (column, charindex ('AAA',column)
, charindex ('/AAA',column)-charindex ('AAA',column))
from table where column like '%/AAA%'
Yes your approach of finding it and then either deleting or replacing it is sound.
If some of the results are blank, it's possible that you are finding and replacing the entire string. If it had not found the correct regular expression in there, you would have not returned the row at all, which is different from returning a black value in that column.
Related
I need to make some changes in SQL within a CURSOR. Previously, the maximum value for column 'code' was 4 characters (e.g. K100, K101,....K999) but now it needs to be 8 characters (e.g. K1000, K1001, K1002,....K1000000).
CURSOR c_code(i_prefix VARCHAR2)
IS
SELECT NVL(MAX(SUBSTR(code,2))+1,100) code
FROM users
WHERE code LIKE i_prefix||'___';
The 'code' column value starts from 100 and increments +1 each time a new record is inserted. Currently, the maximum value is 'K999' and I would like it to be K1000, K1001, K1002 and so on.
I have altered and modified the 'code' column to VARCHAR(8) in the users table.
Note: i_prefix value is always 'K'.
I have tried to amend the SQL -
CURSOR c_code(i_prefix VARCHAR2)
IS
SELECT NVL(MAX(SUBSTR(code,2))+1,100) code
FROM users
WHERE code LIKE i_prefix||'________';
However, it restarts from 100 and not from K1000, K1001, K1002, etc. each time a record is inserted.
I have been suggested to use REGEXP_LIKE() but not sure how to properly use it to get the desired outcome in this case.
Can you please guide me on how can we get this result using REGEXP_LIKE().
Thank you.
Your old code
WHERE code LIKE i_prefix||'___';
will match K followed by exactly three characters, which is what you had. Your new code
WHERE code LIKE i_prefix||'________';
will match K followed by exactly eight characters, which is one too many for a start, since you said the total length was eigh - which means you need sever wilcard placeholders:
WHERE code LIKE i_prefix||'_______';
... but that still won't work at the moment since your existing values aren't that long. As all your current values are at least four, you could do:
WHERE code LIKE i_prefix||'___%';
which will match K followed by three or more characters - with no upper limit, but your column is restricted to eight too anyway.
If you did want to use a regular expression, which are generally slower, you could do:
WHERE REGEXP_LIKE(code, i_prefix||'.{3,7}');
which would match K followed by three to seven characters, or:
WHERE REGEXP_LIKE(code, i_prefix||'\d{3,7}');
which would only match K followed by three to seven digits.
fiddle
However, I would suggest you use a sequence to generate the numeric part, and just prefix that with the K character. The sequence could start from 100 on a new system with no data, or from the current maximum number in an existing system with data.
I would also consider zero-padding the data, including all the existing values, to allow them to be compared; so update K100 to K0000100. Or if you can't do that, once you get past K199 jump to K2000000. Either would then allow the values to be sorted easily as strings. Or, perhaps, add a virtual column that extracts the numeric part as a number.
If I have a field that includes many values like
67495837431978432 ABC1234
and other values like
1234 Something Street
(I know the data shouldn't be like this. It isn't my database. Can't be changed. Please disregard.)
How can I only return records that have data in this field like the former?
I've tried
where SUBSTRING(field, 1, 10) LIKE '%[0-9]%'
with my thought being if the first type of value starts with 17 numbers, and the next type is a street address with many letters in the first 10 characters, I should be able to check if the first 10 characters only include numbers, and return the field based on that, and I should only get back records like
67495837431978432 ABC1234
as I desire to. Not the cleanest, but should work fine for my situation. But that didn't work. I can't see why. Am I making an error I'm not seeing? Is there a better way to do this that is relatively simple?
If you are using SQL Server and you want to check that the first ten characters are digits, you can use like with character classes:
where field LIKE '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]%'
I am learning basics of SQL through W3School and during understanding basics of wildcards I went through the following query:
--Finds any values that start with "a" and are at least 3 characters in length
WHERE CustomerName LIKE 'a_%_%'
as per the example following query will search the table where CustomerName column start with 'a' and have at least 3 characters in length.
However, I try the following query also:
WHERE CustomerName LIKE 'a__%'
The above query also gives me the exact same result.
I want to know whether there is any difference in both queries? Does the second query produce a different output in some specific scenario? If yes what will be that scenario?
Both start with A, and end with %. In the middle part, the first says "one char, then between zero and many chars, then one char", while the second one says "one char, then one char".
Considering that the part that comes after them (the final part) is %, which means "between zero and many chars", I can only see both clauses as identical, as they both essentially just want a string starting with A then at least two following characters. Perhaps if there were at least some limitations on what characters were allowed by the _, then maybe they could have been different.
If I had to choose, I'd go with the second one for being more intuitive. After all, many other masks (e.g. a%%%%%%_%%_%%%%%) will yield the same effect, but why the weird complexity?
For Like operator a single underscore "_" means, any single character, so if you put One underscore like
ColumnName LIKE 'a_%'
you basically saying you need a string where first letter is 'a' then followed by another single character and then followed by anything or nothing.
ColumnName LIKE 'a__%' OR ColumnName LIKE 'a_%_%'
Both expressions mean first letter 'a' then followed by two characters and then followed by anything or nothing. Or in simple English any string with 3 or more character starting with a.
I have a PostgreSQL column of type text that contains data like shown below
(32.85563, -117.25624)(32.855470000000004, -117.25648000000001)(32.85567, -117.25710000000001)(32.85544, -117.2556)
(37.75363, -121.44142000000001)(37.75292, -121.4414)
I want to convert this into another column of type text like shown below
(-117.25624, 32.85563)(-117.25648000000001,32.855470000000004 )(-117.25710000000001,32.85567 )(-117.2556,32.85544 )
(-121.44142000000001,37.75363 )(-121.4414,37.75292 )
As you can see, the values inside the parentheses have switched around. Also note that I have shown two records here to indicate that not all fields have same number of parenthesized figures.
What I've tried
I tried extracting the column to Java and performing my operations there. But due to sheer amount of records I have, I will run out of memory. I also cannot do this method in batched due to time constraints.
What I want
A SQL query or a sequence of SQL queries that will achieve the result that I have mentioned above.
I am using PostgreSQL9.4 with PGAdmin III as the client
this is a type of problem that should not be solved by sql, but you are lucky to use Postgres.
I suggest the following steps in defining your algorithm.
First part will be turning your strings into a structured data, second will transform structured data back to string in a format that you require.
From string to data
First, you need to turn your bracketed values into an array, which can be done with string_to_array function.
Now you can turn this array into rows with unnest function, which will return a row per bracketed value.
Finally you need to slit values in each row into two fields.
From data to string
You need to group results of the first query with results wrapped in string_agg function that will combine all numbers in rows into string.
You will need to experiment with brackets to achieve exactly what you want.
PS. I am not providing query here. Once you have some code that you tried, let me know.
Assuming you also have a PK or some unique column, and possibly other columns, you can do as follows:
SELECT id, (...), string_agg(point(pt[1], pt[0])::text, '') AS col_reversed
FROM (
SELECT id, (...), unnest(string_to_array(replace(col, ')(', ');('), ';'))::point AS pt
FROM my_table) sub
GROUP BY id; -- assuming id is PK or no other columns
PostgreSQL has the point type which you can use here. First you need to make sure you can properly divide the long string into individual points (insert ';' between the parentheses), then turn that into an array of individual points in text format, unnest the array into individual rows, and finally cast those rows to the point data type:
unnest(string_to_array(replace(col, ')(', ');('), ';'))::point AS pt
You can then create a new point from the point you just created, but with the coordinates reversed, turn that into a string and aggregate into your desired output:
string_agg(point(pt[1], pt[0])::text, '') AS col_reversed
But you might also move away from the text format and make an array of point values as that will be easier and faster to work with:
array_agg(point(pt[1], pt[0])) AS pt_reversed
As I put in the question, I tried extracting the column to Java and performing my operations there. But due to sheer amount of records I have, I will run out of memory. I also cannot do this method in batched due to time constraints.
I ran out of memory here as I was putting everything in a Hashmap of
< my_primary_key,the_newly_formatted_text >. As the text was very long sometimes and due to the sheer number of records that I had, it wasnt surprising that I got an OOM.
Solution that I used:
As suggested my many folks here, this solution was better solved with a code. I wrote a small script that formatted the text as per my liking and wrote the primary key and the newly formatted text to a file in tsv format. Then I imported the tsv in a new table and updated the original table from the new one.
Query:
SELECT StartDate, EndDate, RIGHT(Sector, 1 )
FROM Table1
ORDER BY Right(Sector, 1), StartDate
By looking at this, the query should order everything by sector, followed by the start date. This query has worked for quiet awhile until yesterday where it did not order it properly, for some reason, Sector 2 came before Sector 1.
The data type for Sector is of type int, not null. After inserting a TRIM function into Sector it seems to work fine afterwards.
New Query:
SELECT StartDate, EndDate, RIGHT(Sector, 1 )
FROM Table1
ORDER BY Right(TRIM(Sector), 1), StartDate
Which I found really weird since it's suppose to only pick out one character, so why is there leading spaces?
Is there an issue with using RIGHT function on a int before converting the type? Or is it something else?
Thanks for the help everyone!
-Edit- The RIGHT function should return either 1,2,3 or 4 however when ordering it, 2 comes before 1.
To clarify, the column Sector contains an int value, we can determine it's location by obtaining the last digit (Which is why the previous coder did)
MS Access 2003 has a curious little feature (I can't speak for the other versions):
Make a simple query. Sort by Column A Ascending. Save the query.
Run the query. When you see the output, sort by Column A Descending using the toolbar option (see pic below). Save & close.
Run the query again. Your new sort will have overridden the sort that you saved in the query.
I think you or someone else probably just opened the query out of curiosity, sorted by Sector Descending, and when prompted to save Design Changes, you chose Yes (even though technically you didn't make any). The easiest way I found to restore the original sort is to edit the query and save it.
You've got your data stored wrong if you need to sort on a subcharacter of a numeric field.
That said, in certain context, VBA functions reserve a space in string representations of numbers for the sign. A nonsensical example of this would be:
?Len("12345")
5
Notice the space at the beginning (where the - would be if the number returned by Len() could be negative). I thought this was a result of coercing a number to a string value, but that's not it, and I couldn't replicate the problem. But that would likely be the source of the problem, and, of course, trimming off the leading space would take care of the issue.
But that's two function calls for each line, and then you're sorting by it, and that means no use of indexes, so it's going to be slow relative to a SORT BY that can use indexes. So, I'd conclude you have a schema error, in that you're giving meaning to a subpart of the data stored in the field.
It seems pretty obvious that you have a blank space at the end of the Sector field that the trim is removing.