How to exclude certain table(names) in Nifi's CaptureChangeMySQL processor by passing a table name pattern?
For e.g. I have 500 tables and their corresponding history tables.
Capture change should work for Employee, Order etc. but not for their corresponding tables EmployeeHistory, OrderHistory and so on.
In short, tables with postfix 'History' should be filtered by the processor.
I tried
1) $.table_name:equals('DeviceHistory'):not() - not worked
2) ${table_name:equals('*History'):not()} - not worked either
From the NiFi CaptureChangeMySQL processor documentation, the Table Name Pattern field is set as:
A regular expression (regex) for matching CDC events affecting matching tables. The regex must match the table name as it is stored in the database. If the property is not set, no events will be filtered based on table name.
This should be a Java regex string. Looking at the NiFi CaptureChangeMySQL processor source code, here is a code snippet of how this value is used:
// Should we skip this table? Yes if we've specified a DB or table name pattern and they don't match
skipTable = (databaseNamePattern != null && !databaseNamePattern.matcher(data.getDatabase()).matches())
|| (tableNamePattern != null && !tableNamePattern.matcher(data.getTable()).matches());
where tableNamePattern holds Pattern.compile(YOUR_TABLE_NAME_PATTERN).
I wrote a sample program based on this and got the desired behavior using this regex string:
^(?:(?!History).)*$
Here is a demo: https://regex101.com/r/VWuSTy/1/tests
Related
I have used before LIKE command to match patterns to a specific SQL table column. For example need all the rows which all have name started with "A". But this case I am trying to solve things reversed , have a column "Description" where all the regular expressions row wise. And need to return specific rows which all match with a input text.
Table A
Description
============
\b[0-9A-Z ]*WT[A-Z0-9 ]*BALL\b
\b[0-9A-Z ]*WG[A-Z0-9 ]*BAT\b
\b[0-9A-Z ]*AX[A-Z0-9 ]*BAR\b
So Description column has these regular expressions and the input text "BKP 200 WT STAR BALL" So need to return the first row after select query since that the only match. If any one can help with the select query or idea, would be very helpful. If more details required please mention also.
Cross join you regex table to one that you searching within. Then just match two columns against each other.
Here's the example how you can match any of your expressions.
Here how you can match all of them
I have a requirement where I have to track changes on a column and display changed and new value
I have a source file with employee attributes. I did a lookup on employee table and returned employee attributes on which I am tracking changes. I created a flag where I am comparing columns from source and look. I also have a router to filter out update_flag='Y'
employee_id,name,old_department_id,new_department_id,old_salary,new_salary
1,SAM,10,20,100,200
to
employee_id,employee_attribute,old_value,new_value
1,SAM,department_id,10,20
1,SAM,salary,100,200
You could use a Router with an output for each record you want to create. The Router can output multiple records for a single input record.
Group1 : old_department_id != new_department_id
Group2 : old_salary != new_salary
After the Router you use Expressions to build the record values you want like adding the column employee_attribute. Then you combine the outputs of each Expression with a Union.
I have a weird issue with my FTS4 index in SQLite3, namely that a MATCH query for one term returns not the exact match but another, similar one, and vice versa.
Here is an example to illustrate it better:
SELECT name FROM test_idx WHERE name MATCH 'lehmbruck-museum';
-- "Lehmbruck-Archiv"
SELECT name FROM test_idx WHERE name MATCH 'lehmbruck-archiv';
-- "Lehmbruck-Museum"
It seems to have something to do with the dash, here is a similar case that exhibits the same behavior:
SELECT name FROM test_idx WHERE name MATCH 'some-thing';
-- "some-thang"
SELECT name FROM test_idx WHERE name MATCH 'some-thang';
-- "some-thing"
Here is how this test database is built, in case somebody wants to have a go at reproducing it:
CREATE VIRTUAL TABLE test_idx USING fts4(name);
INSERT INTO test_idx (name) VALUES
('some-thing'), ('some-thang'),
('Lehmbruck-Museum'), ('Lehmbruck-Archiv');
SELECT name FROM test_idx WHERE name MATCH 'lehmbruck-museum';
What you pass to MATCH here is a full text query expression. The - character is a unary operator in that expression language that is a stand in for the NOT set operation, and is certainly giving you your unexpected results. Notably - the exact opposite of what you expect! Of course, it is finding exactly what the query is instructed to find - the string lehmbruck and NOT museum at the end!
You'll need to escape it to get the results you want - or perhaps employ the LIKE operator if you are looking at a single column in a table.
Some more information on this expression language can be found in section 3 of the FTS3 and FTS4 documentation on the SQLite doc site here.
I have a table with column mapping which store record: "IV=>0J,IV=>0Q,IV=>2,V=>0H,V=>0K,VI=>0R,VI=>1,"
What is the sql to check whether or not a substring is in column mapping.
so, I would like this:
if I have "IV=>0J" would return true, because IV=>0J is exact in string "mapping"
if I have "IV=>01" would return false. And so on...
I try this:
SELECT * FROM table WHERE charindex('IV=>0J',mapping)
But when I have "IV=>0", it returns TRUE. But, it should return FALSE.
Thank You..
You can search with commas included. Just also add one at beginning and end of mapping:
SELECT * FROM table WHERE charindex(',IV=>0J,',',' + mapping + ',') <> 0
or
SELECT * FROM table WHERE ',' + mapping + ',' LIKE '%,IV=>OJ,%'
This should do the trick:
SELECT * FROM table
WHERE
mapping LIKE '%,IV=>0J,%'
OR mapping LIKE '%,IV=>0J'
OR mapping LIKE 'IV=>0J,%'
OR mapping = 'IV=>0J'
But you should really normalize the database - you are currently violating the principle of atomicity, and therefore the 1NF. Your current difficulties in querying and the future difficulties with performance that you are about to encounter all stem from this root problem...
While you can search by including a comma in the string, this is a bad design for several reasons.
You are unable to take advantage of indexing
You force a full scan of the table, which will lead to bad performance AND excessive blocking.
You have to make sure that there is always a leading or a trailing comma (depends on what you expect in your LIKE expression).
You are no longer able to edit a single entry, you'll have to replace the entire string each time you want to change even a single mapping.
You open yourself to a concurrency nightmare if more that one users try to update different mappings that just happen to be stored in the same column.
Your table isn't even in 1st normal form any more, which is why you have such difficulties
You should normalize your mapping column, by extracting the data to a different mapping table, with at least the From and To columns you require. You can then add these columns to an index an convert your query using only a single index seek.
You can also add the ID values of your source table to the Mappings table and the index. This will allow you to convert the lookup for a source row to a join between the two tables that takes advantage of indexing
charindex returns the position of the text, not Boolean.
to check if the text exists, compare to 0:
SELECT * FROM table WHERE charindex('IV=>0J',mapping) <> 0
I think you're missing something here, the Charindex function does not return TRUE or FALSE.
It returns the starting point of the substring inside master string, or if the substring is not present, then -1.
So you query should read,
SELECT * FROM table WHERE charindex('IV=>0J',mapping) > 0
I have a ruby on rails app in which I am querying for a boolean column, Flag. The code is:
Merchant.where("Flag=?",false)
However this does not work at all and the only result is that the Merchants table does not have a column name "flag". Is there any way to fix this? The column name starts with an uppercase letter, but the search is being done for a lower case "flag"
If the column name was quoted when the table was created then you will have to quote it forever. So, if you started with this:
create table merchants (
-- ...
"Flag" boolean
-- ...
)
Then you'll have to refer to it using
Merchant.where('"Flag" = ?', false)
PostgreSQL normalizes all unquoted identifiers to lower case (not upper case as the standard says), that's why the error message complains about flag rather than Flag.
If you can, you might want to rebuild your table with only lower case column names.