The problem we have, arises from the fact that the online sign-up form we use, can be accessed multiple times by the same user. We do not want to use the option to restrict that, as e.g. parents/guardians can sign up both themselves and their children, so that is a valid use case for us.
However, some of our clients (due perhaps to poor web connectivity or other issues) end up submitting he same person more than once. So now out lists are inflated and we are double counting these people.
In the past I have in SQL joined a table against itself, as a stratagem to poke out such duplicates. But from Googling this a bit, the SQL type queries available in Google Sheets do not seem to lend themselves very easily to joins.
Any suggestions?
If all duplicate entries are identical, then try:
=UNIQUE([data range])
If you wanted to use the query function you could count use select and group by to group together all duplicates. You could count them too.
Example:
=QUERY(A:B,"select A, B, count(A) where A!='' group by A, B",1)
This will output a list of unique rows and their count.
References:
Group by
I have two CSV files, the first like so:
Book1:
ID,TITLE,SUBJECT
0001,BLAH,OIL
0002,BLAH,HAMSTER
0003,BLAH,HAMSTER
0004,BLAH,PLANETS
0005,BLAH,JELLO
0006,BLAH,OIL
0007,BLAH,HAMSTER
0008,BLAH,JELLO
0009,BLAH,JELLO
0010,BLAH,HAMSTER
0011,BLAH,OIL
0012,BLAH,OIL
0013,BLAH,OIL
0014,BLAH,JELLO
0015,BLAH,JELLO
0016,BLAH,HAMSTER
0017,BLAH,PLANETS
0018,BLAH,PLANETS
0019,BLAH,HAMSTER
0020,BLAH,HAMSTER
And then a second CSV with items associated with the first list, with ID being the common attribute between the two.
Book2:
ID,ITEM
0001,PURSE
0001,STEAM
0001,SEASHELL
0002,TRUMPET
0002,TRAMPOLINE
0003,PURSE
0003,DOLPHIN
0003,ENVELOPE
0004,SEASHELL
0004,SERPENT
0004,TRUMPET
0005,CAR
0005,NOODLE
0006,CANNONBALL
0006,NOODLE
0006,ORANGE
0006,SEASHELL
0007,CREAM
0007,CANNONBALL
0007,GUM
0008,SERPENT
0008,NOODLE
0008,CAR
0009,CANNONBALL
0009,SERPENT
0009,GRAPE
0010,SERPENT
0010,CAR
0010,TAPE
0011,CANNONBALL
0011,GRAPE
0012,ORANGE
0012,GUM
0012,SEASHELL
0013,NOODLE
0013,CAR
0014,STICK
0014,ORANGE
0015,GUN
0015,GRAPE
0015,STICK
0016,BASEBALL
0016,SEASHELL
0017,CANNONBALL
0017,ORANGE
0017,TRUMPET
0018,GUM
0018,STICK
0018,GRAPE
0018,CAR
0019,CANNONBALL
0019,TRUMPET
0019,ORANGE
0020,TRUMPET
0020,CHERRY
0020,ORANGE
0020,GUM
The real datasets are millions of records, so I'm sorry in advance for my simple example.
The problem I need to solve is getting the data merged and collated in a way where I can see which item groupings most commonly appear together on the same ID. (e.g. GRAPE,GUM,SEASHELL appear together 340 times, ORANGE and STICK 89 times, etc...)
Then I need to see if there is any change/deviation to the general results in common appearance when grouped by SUBJECT.
Tools I'm familiar with are Excel and SQL, but I also have PowerBI and Alteryx at my disposal.
Full disclosure: Not homework, or work, but a volunteer project, thus my unfamiliarity with this kind of data manipulation.
Thanks in advance.
An Alteryx solution:
Drag the two .csv files onto your canvas (seen as book1.csv and book2.csv in my picture; Alteryx will create "Input" tools for you.
Drag a "Join" tool on and connect the two .csv files to its inputs; select "ID" as the join field; unselect the "Right_ID" as output since it's merely a duplicate of "ID"
Drag a "Summary" tool on and connect the Join tool's output to the Summary tool's input; select all three of the outputs and add as a "group by"... then add the ID column with a "count"
Drag a browse tool on and connect the summary's output to the browse tool's input.
run the workflow
After all that, click on the browse tool and you should see what is seen in my screenshot: (which is showing just the first ten rows of output):
+1 for taking on a volunteer project - I think anyone who knows data can have a big impact in support of their favourite group or cause.
I would just pull the 2 files into Power BI as 2 separate tables (Get Data / From File). Create a relationship between the 2 tables based on ID (it might get auto-generated). It should be one to many.
Then I would add a Calculated Column to the Book1 table to Concatenate the related ITEM values, eg.
Items =
CALCULATE (
CONCATENATEX (
DISTINCT ( 'Book2'[ITEM] ),
'Book2'[ITEM],
", ",
'Book2'[ITEM], ASC
)
)
Now you can use that Items field in visuals (e.g. a Table), along with Count of ID to get the frequency.
Adding Subject to a copy of the table (e.g. to the Columns well of a Matrix) will produce your grouped scenario, or you could add a Subject Slicer.
As you will be comparing subsets of varying size, I would change Count of ID to Show value as - % of grand total.
Little different solution using Alteryx.
With this dataset, there are very few repeating 3 or 4 item groups. You can do the two item affinity analysis and get a probability of 3 or 4 item groups, or you can count the 3 and 4 item groups individually. I believe what you want is the latter as your probability of getting grapes with oranges may be altered by whether you have bananas in the cart or not.
Anyway, I did not join in the subject until after finding all of my combinations. I found all the combinations by taking the Cartesian join of two, then three, then four of the original set. I then removed all duplicates by ensuring items were always in alphabetical order in each row. I then counted occurrences of each combination. More joins can be added in the same pattern to count groups of 5,6,7...
Once you have the counts of occurrences, then I would join back with the subjects and perform this analysis on each group and compare to the overall results.
I'm supposed to disclose that I work for Alteryx.
first of all if you are using windows
just navigate to the directory which contains the CSV and write the following command:
copy pattern newfileName.csv
#example
copy *.csv merged.csv
now you created one csv file, the file is too large now you can't process it once, depending on your programming language you can use appropriate way, for python you can use generators to process line by line, or pandas you can read chunk by chunk it will be easy.
I hope this help you.
I have two tables
Email Contact History
Place of Service
that share a primarymembercustomerid. The Email Contact History has three fields:
Campaigncode
Primarymembercustomerid
maildate
and the Place of Service table has three fields
primarymembercustomerid
servicedate
serviceshortDesc
primarymembercustomerids are selected for E-mail campaigns, then if they walk into one of our branch offices and receive services, they show up in the Place of Service table. I want to count the number of primarymembercustomer ids that are mailed, and right next to it I want to have a count of primarymembercustomerids that showed up to a branch office.
What I have so far:
select
ch.campaigncode,
pos.serviceshortdesc,
count(ch.primarymembercustomerid),
count(pos.primarymembercustomerid)
from mktprodvm.cdmv_prmmbr_contacthist_email ch
right outer join mktprodvm.cdmv_pos pos on ch.primarymembercustomerid = pos.primarymembercustomerid
where ch.campaigncode = 'EDT_ALLACMO'
and pos.servicedate between '2017-02-01' and '2017-02-28'
group by 1,2
What I'm ending up with is a count of primarymembercustomerids that walk into a branch for that time period, but I'm not getting the total count of primarymembercustomerids that were E-mailed. I thought that by doing a right outer join I would get the total number of primarymembercustomerids that were mailed, but it's not working for me. I feel like I need to do some kind of subquery or correlated subquery, but I've ready about how to use them and I don't think that's right. I've never used them before and to be quite honest I'm not that great of a SQL coder either. Thanks for any help!
Because my low reputation on this site I can't comment, so I'm writing this as an answer.
I think that you are using the wrong type of join (or writing the tables in the wrong order). If you don't want to lose rows of your main table, the Email Contact History, you have to do a LEFT JOIN not a RIGHT JOIN.
Also, I don't know if it's possible, but I'm guessing that a primarymembercustomerid can have more than one service and since you are selecting the serviceshortdesc, a single Email might count in different rows of your anwerset and the total won't be accurate. According to what you said you want, I don't see a reason for including the service description in the SELECT.
I have a list of customers and their revenue. For certain reasons, I want to display one section per customer, for the TOP 10 customers (in regards to their revenues). Sounds simple.
However, when I activate the ranking for the sections, the customer with the 7th highest revenue is placed on the top of the list, and is given the running count '8'.
I tried everything I could think of to put them back into order, with no success. Any ideas?
I finally found a solution - the reason for this strange behaviour is the fact that section filters don't always seem to work.
Let me explain:
I had a filter on both the section and the table therein, limiting customers only to a certain subgroup thereof. The filter on the table worked, so that the displayed revenue represented the correct revenue I expected. However, the filter on the section did NOT work, so that the ranking took into consideration the entire revenue, and not only the subgroup's revenue. Therefore Customer 8 landed on first place, because in total, he had the highest revenue.
So if you have this problem, check your filters.
Next, I'll be trying to find out why the section filters don't apply...
This is not really an issue that affects the code but rather a question of the table's appearance.
So, the table is the summary of records for income and expenses of different business departments. Let's call each department a type of the record. Each of those types has subtype1. Each subtype1 has subtypes2 and each subtype2 has subtypes3.
So the sample data would be something like this.
1, Type1, sum of subtypes1
1.1, Subtype1, sum of subtypes2
1.1.1 Subtype2, sum of subtypes3
1.1.1.1 Subtype3, amount
1.1.1.2 Subtype3, amount
1.2, Subtype1, sum of subtypes2
1.2.1, Subtype2, sum of subtypes3
1.2.1.1, Subtype3, amount
Each subtype can have different number of "children subtypes". Children subtypes can't go further than subtype3.
Then I am using VBA script to group the records of the same subtype under their direct parent up to the main type. Everything works fine, I can expand or hide every single level of this structure.
However, logically the group outline on the left side of the table for rows should show 4 levels. Instead it shows 8 levels of groups. First 4 do exactly what you would expect, show or hide respective subtypes while the other 4 levels do absolutely nothing which is also expected because I don't see a reason for them to be there.
Any ideas why extra levels have been created and how to get rid of them?
I might have explained this in a not very clear way so feel free to ask for further information.
Try stepping through your code in trace mode to watch the groups being set up. (open the VBA window and use the F8 key to loop one line at a time)
This may reveal why the extra groups are being defined and suggest what to change.