Potential Duplicates Detection, with 3 Severity Level - vba

I wanna make a program that detect a potential duplicates with 3 severity level.
let consider my data is only in two column, but with thousands row.
data in second column delimited only with comma. data example :
Number | Material
1 | helmet,valros,42
2 | helmet,iron,knight
3 | valros,helmet,42
4 | knight,helmet
5 | valros,helmet,42
6 | plain,helmet
7 | helmet, leather
and my 3 levels is :
very high : A,B,C vs A,B,C
high : A,B,C vs B,C,A
so so : A,B,C vs A,B
so far i just can make the first level, i don't know how to do the 2nd and the 3rd level.
what I've tried.
Sub duplicates_separation()
Dim duplicate(), i As Long
Dim delrange As Range, cell As Long
Dim shtIn As Worksheet, shtOut As Worksheet
Set shtIn = ThisWorkbook.Sheets("input")
Set shtOut = ThisWorkbook.Sheets("output")
x = 2
y = 1
Set delrange = shtIn.Range("b1:b10000") 'set your range here
ReDim duplicate(0)
'search duplicates in 2nd column
For cell = 1 To delrange.Cells.Count
If Application.CountIf(delrange, delrange(cell)) > 1 Then
ReDim Preserve duplicate(i)
duplicate(i) = delrange(cell).Address
i = i + 1
End If
Next
'print duplicates
For i = UBound(duplicate) To LBound(duplicate) Step -1
shtOut.Cells(x, 1).EntireRow.Value = shtIn.Range(duplicate(i)).EntireRow.Value
End Sub
duplicates detected by the program :
3 | valros,helmet,42
5 | valros,helmet,42
what i expected:
Number | Material
1 | helmet,valros,42
3 | valros,helmet,42
5 | valros,helmet,42
4 | knight,helmet
2 | helmet,iron,knight
i have an idea for detecting duplicates lv 2, but I think it will be so complicated and make the program slow.
turn column 2 to columns with "text to columns" command
sort column from A to Z (alphabetically)
concatenate the column
do countif like done in detecting duplicates lv 1
is there a way to detect the 2nd & 3rd level duplicates?
UPDATE
Yesterday I went to friend's house to consult about this problem, but his solution is in JAVA languange.. >which i don't understand
public class ali {
static void sPrint(String[] Printed) {
for (int iC = 0; iC < Printed.length; iC++) {
System.out.print(String.valueOf(Printed[iC]) + " | ");
}
System.out.println();
}
public static void main(String Args[]) {
int defaultLength = 10;
int indexID = 0;
int indexDesc = 1;
String[] DETECTORP1 = new String[defaultLength];
String[] DETECTORP2 = new String[defaultLength];
String[] DETECTORP3 = new String[defaultLength];
String[] DETECTORP4 = new String[defaultLength];
String[][] theString = new String[5][2];
theString[0] = new String[]{"1", "A, B, C, D"};
theString[1] = new String[]{"2", "A, B, C, D"};
theString[2] = new String[]{"3", "A, B, C, D, E"};
theString[3] = new String[]{"4", "A, B, D, C, E"};
theString[4] = new String[]{"5", "A, B, D, C, E, F"};
int P1 = 0;
int P2 = 0;
int P3 = 0;
int P4 = 0;
for (int iC = 0; iC < theString.length; iC++) {
System.out.println(theString[iC][indexID] + " -> " + theString[iC][indexDesc]);
}
for (int iC = 0; iC < theString.length; iC++) {
int LEX;
String theReference[] = theString[iC][indexDesc].replace(",", ";;").split(";;");
for (int iD = 0; iD < theString.length; iD++) {
if (iC != iD) {
String theCompare[] = theString[iD][1].replace(",", ";;").split(";;");
if (theReference.length == theCompare.length) {
LEX=0;
int theLength = theReference.length;
for (int iE = 0; iE < theLength; iE++) {
if (theReference[iE].equals(theCompare[iE])) {
LEX += 1;
}
}
if (LEX == theLength) {
DETECTORP1[P1] = theString[iC][indexID] + " WITH " + theString[iD][indexID];
P1 += 1;
} else {
LEX = 0;
for (int iF = 0; iF < theReference.length; iF++) {
for (int iG = 0; iG < theCompare.length; iG++) {
if (theReference[iF].equals(theCompare[iG])) {
LEX += 1;
break;
}
}
}
if (LEX == theReference.length) {
DETECTORP2[P2] = theString[iC][indexID] + " WITH " + theString[iD][indexID];
P2 += 1;
}
}
} else {
LEX = 0;
if (theReference.length > theCompare.length) {
for (int iF = 0; iF < theReference.length; iF++) {
for (int iG = 0; iG < theCompare.length; iG++) {
if (iG == iF) {
if (theReference[iF].equals(theCompare[iF])) {
LEX += 1;
break;
}
}
}
}
if (LEX <= theReference.length && LEX >= theCompare.length) {
DETECTORP3[P3] = theString[iC][indexID] + " WITH " + theString[iD][indexID];
P3 += 1;
}
} else {
LEX =0;
for (int iF = 0; iF < theCompare.length; iF++) {
for (int iG = 0; iG < theReference.length; iG++) {
if (iG == iF) {
if (theCompare[iF].equals(theReference[iF])) {
LEX += 1;
// System.out.println(theReference[iG] + "==" + theCompare[iG]);
break;
}
}
}
}
if (LEX <= theCompare.length && LEX >= theReference.length) {
DETECTORP3[P3] = theString[iC][indexID] + " WITH " + theString[iD][indexID];
P3 += 1;
}
}
}
}
}
}
sPrint(DETECTORP1);
sPrint(DETECTORP2);
sPrint(DETECTORP3);
}
}
how to do this in VBA?

Really, it depends how you want to define "severity level". Here's one way to do it, not necessarily the best: Use the Levensthein distance.
Represent each of your items by a one-character attribute symbol, e.g.
H helmet
K knight
I iron
$ Leather
^ Valros
╔ Plain
¢ Whatever
etc.
Then convert your Material lists into a string containing sequence of characters representing these attributes:
HIK = helmet,iron,knight
¢H = plain,helmet
Then compute the Levenshtein distance between those two strings. That will be your "severity level".
Debug.Print LevenshteinDistance("HIK","¢H")
'returns 3
Two implementations of the Levenshtein distance are shown in Wikipedia. And indeed you are in luck: someone on StackOverflow ported this to VBA.
In the comments section below, you say you don't like having to represent each of your possible attributes by one-character symbols. Fair enough; I agree this is a bit silly. Workaround: It is, in fact, possible to adapt the Levenshtein Distance algorithm to look not at each character in a string, but at each element of an array instead, and do comparisons based on that. I show how to make this change in my answer to your follow-up question.

Related

Why doesn't my number sequence print from the 2d arraylist correctly?

I cannot get the loop to work in the buildDimArray method to store the number combinations "11+11", "11+12", "11+21", "11+22", "12+11", "12+12", "12+21", "12+22", "21+11", "21+12", "21+21", "21+22", "22+11", "22+12", "22+21", and "22+22" into the 2d arraylist with each expression going into one column of the index dimBase-1 row. The loop may work for other people, but for some reason mine isn't functioning correctly. The JVM sees the if dimBase==1 condition, but refuses to check the other conditions. The "WTF" not being printed as a result from the buildDimArray method. If dimBase=1, it prints successfully, but doesn't for the other integers. The dimBase==3 condition needs a loop eventually. The "WTF" is for illustrative purposes. I could get away with a 1d arraylist, but in the future I will likely need the 2d arraylist once the program is completed.
package jordanNumberApp;
import java.util.Scanner;
import java.util.ArrayList;
/*
* Dev Wills
* Purpose: This code contains some methods that aren't developed. This program is supposed to
* store all possible number combinations from numbers 1-dimBase for the math expression
* "##+##" into a 2d arraylist at index row dimBase-1 and the columns storing the
* individual combinations. After storing the values in the arraylist, the print method
* pours the contents in order from the arraylist as string values.
*/
public class JordanNumberSystem {
// a-d are digits, assembled as a math expression, stored in outcomeOutput, outcomeAnswer
public static int dimBase, outcomeAnswer, a, b, c, d;
public static String inputOutcome, outcomeOutput;
public static final int NUM_OF_DIMENSIONS = 9; //Eventually # combinations go up to 9
public static ArrayList<ArrayList<String>> dimBaseArray;
public static Scanner keyboard;
/*
* Constructor for JordanNumber System
* accepts no parameters
*/
public JordanNumberSystem() // Defunct constructor
{
// Declare and Initialize public variables
this.dimBase = dimBase;
this.outcomeOutput = outcomeOutput;
this.outcomeAnswer = outcomeAnswer;
}
// Set all values of variable values
public static void setAllValues()
{
// Initialize
dimBase = 1;
outcomeAnswer = 22; // variables not used for now
outcomeOutput = "1"; // variables not used for now
//a = 1;
//b = 1;
//c = 1;
//d = 1;
dimBaseArray = new ArrayList<ArrayList<String>>();
keyboard = new Scanner(System.in);
}
public static void buildDimArray(int dim)
{
dimBase = dim;
try
{
//create first row
dimBaseArray.add(dimBase-1, new ArrayList<String>());
if( dimBase == 1)
{
a = b = c = d = dimBase ;
dimBaseArray.get(0).add(a+""+b+"+"+c+""+d);
System.out.println("WTF"); // SHOWS
}
else if (dimBase == 2)
{ // dim = 2
a = b = c = d = 1 ;
System.out.println("WTF"); // doesn't show
// dimBaseArray.get(dimBase-1).add(a+""+b+"+"+c+""+d);
for( int i = 1 ; i <= dim ; i++)
a=i;
for( int j = 1 ; j <= dim ; j++)
b=j;
for( int k = 1 ; k <= dim ; k++)
c=k;
for( int l = 1 ; l <= dim ; l++)
{
d=l;
dimBaseArray.get(dim-1).add(a+""+b+"+"+c+""+d);
}
}
else if (dimBase == 3)
{
a = b = c = d = dimBase;
dimBaseArray.get(2).add(a+""+b+"+"+c+""+d);
System.out.println("WTF");
}
}catch (IndexOutOfBoundsException e)
{
System.out.println(e.getMessage());
}
}
public static void printArray(int num) // Prints the contents of the array
{ // Fixing the printing method
try
{
int i = num-1;
for( String string : dimBaseArray.get(i))
{
System.out.println(string);
System.out.println("");
}
} catch (IndexOutOfBoundsException e)
{
System.out.println(e.getMessage());
}
}
public static void main(String[] args) throws java.lang.IndexOutOfBoundsException
{
setAllValues(); // sets the initial a,b,c,d values and dimBase, initializes 2d arraylist
// Get the Dimension Base number
System.out.println("Enter Dimension Base Number. Input an integer: ");
int dimBaseInput = keyboard.nextInt(); // Receives integer
dimBase = dimBaseInput;
if( dimBase != 1 && dimBase != 2 && dimBase != 3)
{// Error checking
System.out.println("invalid Dimension Base Number should be 1 or 2 ");
System.exit(1);
}
// Build the arraylist, print, clear, exit
buildDimArray(dimBase);
printArray(dimBase);
dimBaseArray.clear();
System.exit(1);
}
}// End of class

What is the time complexity of this function?

Here's a sample solution for Sliding Window Maximum problem in Java.
Given an array nums, there is a sliding window of size k which is
moving from the very left of the array to the very right. You can only
see the k numbers in the window. Each time the sliding window moves
right by one position.
I want to get the time and space complexity of this function. Here's what I think would be the answer:
Time: O((n-k)(k * logk)) == O(nklogk)
Space (auxiliary): O(n) for return int[] and O(k) for pq. Total of O(n).
Is this correct?
private static int[] maxSlidingWindow(int[] a, int k) {
if(a == null || a.length == 0) return new int[] {};
PriorityQueue<Integer> pq = new PriorityQueue<Integer>(k, new Comparator<Integer>() {
// max heap
public int compare(Integer o1, Integer o2) {
return o2 - o1;
}
});
int[] result = new int[a.length - k + 1];
int count = 0;
// time: n - k times
for (int i = 0; i < a.length - k + 1; i++) {
for (int j = i; j < i + k; j++) {
// time k*logk (the part I'm not sure about)
pq.offer(a[j]);
}
// logk
result[count] = pq.poll();
count = count + 1;
pq.clear();
}
return result;
}
You're right in most of the part except -
for (int j = i; j < i + k; j++) {
// time k*logk (the part I'm not sure about)
pq.offer(a[j]);
}
Here total number of executions is log1 + log2 + log3 + log4 + ... + logk. The summation of this series -
log1 + log2 + log3 + log4 + ... + logk = log(k!)
And second thought is, you can do it better than your linearithmic time solution using double-ended queue property which will be O(n). Here is my solution -
public int[] maxSlidingWindow(int[] nums, int k) {
if (nums == null || k <= 0) {
return new int[0];
}
int n = nums.length;
int[] result = new int[n - k + 1];
int indx = 0;
Deque<Integer> q = new ArrayDeque<>();
for (int i = 0; i < n; i++) {
// remove numbers out of range k
while (!q.isEmpty() && q.peek() < i - k + 1) {
q.poll();
}
// remove smaller numbers in k range as they are useless
while (!q.isEmpty() && nums[q.peekLast()] < nums[i]) {
q.pollLast();
}
q.offer(i);
if (i >= k - 1) {
result[indx++] = nums[q.peek()];
}
}
return result;
}
HTH.

Conditional Formatting With Icons Epplus

I need to achieve something like this with Epplus.
Can someone guide me with the code I need to use.
Following is the code to do exactly what you want but it is for three icon set you can change it based on your icons. I'm setting red color arrow if value is greater than 4,yellow color arrow if value is between 1 and 4 and, finally, green color if it is less than 1. Just change "AddThreeIconSet" to your icons. You should get the idea with this.
for (int j = 2; j <= 9; j++) // Loop through columns
{
for (int i = 3; i <= 12; i++) // Loop through rows
{
// gets only the current cell as range
ExcelRange rng = worksheet.Cells[i, j, i, j];
ExcelAddress address = new ExcelAddress(rng.Address);
// Get the value of the current cell
if(Convert.ToDouble(worksheet.Cells[i, j].Value) >= 4.0)
{
var v = worksheet.ConditionalFormatting.AddThreeIconSet(address, eExcelconditionalFormatting3IconsSetType.Arrows);
v.Reverse = true;
v.Icon1.Type = eExcelConditionalFormattingValueObjectType.Num;
}
else if (Convert.ToDouble(workSheet.Cells[i, j].Value) > 1.0 && Convert.ToDouble(workSheet.Cells[i, j].Value) < 4.0)
{
var v = worksheet.ConditionalFormatting.AddThreeIconSet(address , eExcelconditionalFormatting3IconsSetType.Arrows);
v.Icon3.Type = eExcelConditionalFormattingValueObjectType.Num;
}
else if (Convert.ToDouble(workSheet.Cells[i, j].Value) < 1.0)
{
var v = worksheet.ConditionalFormatting.AddThreeIconSet(address , eExcelconditionalFormatting3IconsSetType.Arrows);
v.Icon2.Type = eExcelConditionalFormattingValueObjectType.Num;
}
}
}

Output in the memo field

void __fastcall TForm1::Step(int _Sum, int _Num, int _Val)
{
if (_Sum <= 0 || _Num <= 0 || _Sum < _Val) return;
if (_Num == 1)
{
data[theK - 1] = _Sum;
Memo1->Lines->Add("{");
for (int i = 0; i < theK; ++i)
Memo1->Lines->Add(data[i]);
sc++;
Memo1->Lines->Add("}");
Memo1->Lines->Add("\n");
Label4->Caption = sc;
return;
}
for (int i = _Val; i < _Sum; ++i)
{
data[theK - _Num] = i;
Step(_Sum - i, _Num - 1, i);
}
}
I have output like this:
1 2 3 4 5 6 7 8 9 10
11 12
But I need output like:
1,2,3,4 5,6,7,8 9,10,11,12
How to do it in C++ Builder? I have Memo1 or ListBox. I need formatting output.
I usually build a UnicodeString for each line of output like so:
UnicodeString Str;
for (int I =0; I<5;i++)
{
Str += IntToStr(I)+","
};
Str.SetLength(Str.Length -1); // trim off last comma
Memo1->Add(Str);

apply bind pose to a kinect skeleton

I want to normalize a skeleton in order to make it invariant to the size of the person
in front of the kinect; in the same way as the aveteering example.
But I don't want to animate a 3D model using XNA, the only thing I need is to normalize an
skeleton.
So in order to do this task, I have divided it in two functions:
(a) apply a bind pose to an skeleton in order to see how to work this matrix. Obviously this is not what i want to do, but it is a first step in order to
know how to work whit matrix, and so on.
(b) apply any arbitrary pose to a normalized-size-skeleton
First of all, I want to apply a bind pose to an skeleton (a).
First, I have to load the matrix that describe the bone length/ offset between bones and store it in
List BindPose.
Due to I have no idea how to do it, I modified the Aveteering example and write in a file all the Matrix that define
the BindPose, InverseBindPose and SkeletonHierarchy of the dude. I only need BindPose to this first task, but I have the
code prepared in order to do the second task (b)
The file looks like this:
1,331581E-06;-5,551115E-17;1;0;1;-4,16881E-11;-1,331581E-06;0;4,16881E-11;1;8,153579E-23;0;0,03756338;37,46099;2,230549;1
1,110223E-16;-4,435054E-22;1;0;1;1,426127E-06;-2,220446E-16;0;-1,426127E-06;1;-7,654181E-22;0;-0,9558675;-4,079016E-08;-6,266987E-12;1
0,9954988;-0,09477358;1,501821E-06;0;0,09477358;0,9954988;-4,019565E-06;0;-1,114112E-06;4,143805E-06;1;0;3,786007;-0,003599779;5,107028E-06;1
0,9948416;-0,101441;-3,23556E-07;0;0,101441;0,9948416;-2,266755E-08;0;3,241862E-07;-1,027114E-08;1;0;4,543321;-0,00359975;-1,33061E-07;1
0,9950595;0,09927933;2,388133E-07;0;-0,09927933;0,9950595;-2,333792E-08;0;-2,399506E-07;-4,86646E-10;1;0;4,544049;-0,003599948;6,324596E-08;1
0,9992647;0,02747673;0,02674458;0;-0,02928042;0,9971476;0,06956656;0;-0,02475683;-0,07029849;0,9972187;0;4,543965;-0,004398902;2,258555E-07;1
0,9154034;0,4025377;1,107153E-06;0;-0,4025377;0,9154033;-2,437432E-07;0;-1,109319E-06;-2,115673E-07;1;0;5,536249;-0,00288291;1,332601E-07;1
0,9812952;-0,1925096;-4,732622E-07;0;0,1925095;0,9812951;-3,00921E-08;0;4,697166E-07;-5,889972E-08;1;0;3,953898;1,702301E-07;4,88653E-08;1
.......
So each line is a 4X4 matrix defining the BindPose.
To generate this file, the code is like this:
private void ViewSkinningData(SkinningData data)
{
string nameFile = "bind_pose_transformations";
bool append = false;
// The using statement automatically closes the stream and calls IDisposable.Dispose on the stream object.
using (System.IO.StreamWriter file = new System.IO.StreamWriter(#nameFile, append))
{
for (int i = 0; i < data.BindPose.Count; i++)
{
Matrix m = data.BindPose[i];
string matrixString = MatrixToString(m);
file.WriteLine(matrixString);
}
for (int i = 0; i < data.InverseBindPose.Count; i++)
{
Matrix m = data.InverseBindPose[i];
string matrixString = MatrixToString(m);
file.WriteLine(matrixString);
}
for (int i = 0; i < data.SkeletonHierarchy.Count; i++)
{
file.Write(data.SkeletonHierarchy[i] + ";");
}
}
}
string MatrixToString(Matrix m)
{
string result;
result = m.M11 + ";" + m.M12 + ";" + m.M13 + ";" + m.M14 + ";" + m.M21 + ";" + m.M22 + ";" + m.M23 + ";" + m.M24 + ";" + m.M31 + ";" + m.M32 + ";" + m.M33 + ";" + m.M34 + ";" + m.M41 + ";" + m.M42 + ";" + m.M43 + ";" + m.M44;
return result;
}
Next step is to load all this Skinning data in my program:
private void InitializeSkinningDataFromFile()
{
string filename = "bind_pose_transformations";
int number_avatar_joints = 58;
List<Matrix> binpose = new System.Collections.Generic.List<Matrix>();
List<Matrix> inversebindpose = new System.Collections.Generic.List<Matrix>();
List<int> skeletonhierarchy = new System.Collections.Generic.List<int>();
// The using statement automatically closes the stream and calls IDisposable.Dispose on the stream object.
using (System.IO.StreamReader file = new System.IO.StreamReader(filename))
{
string s;
int count = 0;
while (!String.IsNullOrEmpty(s = file.ReadLine()))
{
string[] values = s.Split(';');
Matrix m = BuildMatrix(values);
binpose.Add(m);
count++;
if (count == number_avatar_joints)
{
break;
}
}
count = 0;
while (!String.IsNullOrEmpty(s = file.ReadLine()))
{
string[] values = s.Split(';');
Matrix m = BuildMatrix(values);
inversebindpose.Add(m);
count++;
if (count == number_avatar_joints)
{
break;
}
}
string[] skeletonHierarchy = file.ReadLine().Split(';'); //lee un caracter de separacion al final...
//for (int i = 0; i < skeletonHierarchy.Count(); i++)
for (int i = 0; i < number_avatar_joints; i++)
{
skeletonhierarchy.Add(int.Parse(skeletonHierarchy[i]));
}
}
skinningDataValue = new SkinningData(binpose, inversebindpose, skeletonhierarchy);
}
After, I have to construct boneTransforms structure:
// Bone matrices for the "dude" model
this.boneTransforms = new Matrix[skinningDataValue.BindPose.Count];
this.skinningDataValue.BindPose.CopyTo(this.boneTransforms, 0);
Now boneTransforms have the transformation for my skeleton. So now, i have to apply these trasnformations to an skeleton
Skeleton skeleton = new Skeleton();
foreach (Joint joint in skeleton.Joints)
{
int indexMatrix = AvatarBoneToNuiJointIndex(joint.JointType);
Matrix transform;
if (indexMatrix >= 0)
{
transform = this.boneTransforms[indexMatrix];
}
else
{
transform = Matrix.Identity;
}
Joint aux = ApplyMatrixTransformationToJoint(joint, transform);
normalizeSkel.Joints[joint.JointType] = aux;
}
This is a helper function AvatarBoneToNuiJointIndex:
public int AvatarBoneToNuiJointIndex(JointType jointType)
{
switch (jointType)
{
case JointType.HipCenter:
return 1;
case JointType.Spine:
return 4;
case JointType.ShoulderCenter:
return 6;
case JointType.Head:
return 7;
case JointType.ShoulderLeft:
return 12;
case JointType.ElbowLeft:
return 13;
case JointType.WristLeft:
return 14;
case JointType.HandLeft:
return 15;
case JointType.ShoulderRight:
return 31;
case JointType.ElbowRight:
return 32;
case JointType.WristRight:
return 33;
case JointType.HandRight:
return 34;
case JointType.KneeLeft:
return 50;
case JointType.AnkleLeft:
return 51;
case JointType.FootLeft:
return 52;
case JointType.KneeRight:
return 54;
case JointType.AnkleRight:
return 55;
case JointType.FootRight:
return 56;
default: return -1;
}
}
This is a helper function ApplyMatrixTransformationToJoint:
public Joint ApplyMatrixTransformationToJoint(Joint skeletonJoint, Matrix tranformations)
{
Vector3 pos = SkeletonPointToVector3(skeletonJoint.Position);
Vector3 result = ApplyMatrixTransformationToVector(pos, tranformations);
SkeletonPoint newPosition = new SkeletonPoint()
{
X = result.X,
Y = result.Y,
Z = result.Z
};
skeletonJoint.Position = newPosition;
return skeletonJoint;
}
This is the code for ApplyMatrixTransformationToVector:
static Vector3 ApplyMatrixTransformationToVector(Vector3 v, Matrix m)
{
return Vector3.Transform(v, m);
}
But the problem is that I can't see anything.
I don't know if this approach is correct.
Any help would be fantastic.
Many thanks!