I've been trying to build a query over a custom log of mine where I sort the users based on certain criteria to have some overview of them.
My log contains a entry for each time a user tries to download a file, that entry contains date, ip, a custom generated token and how many times that user has tried.
The token is stored by SESSION and a token is only valid for 5 attempts of downloading, So that means that one ip can have multiple users(with different tokens) that each have different amount of attempts.
What I want to achieve is rather simple, I want to group the users by ip, and then count their amount of attempts, and then find out how many users there are.
The amount is not counted per IP but rather per token meaning a log entry may look like this:
IP TOKEN ATTEMPT
111.111.111.111 DK1234 a1
111.111.111.111 DK9876 a1
111.111.111.111 DK9876 a2
222.222.222.222 DK5432 a1
Below is my latest attempts of trying to achieve this, but while I try to make the logic behind it work it just isn't what I want.
(The fields involved are: Ip, Token and Attempt (The attempt value looking like this: a1, a2, a3 and so on for each attempt the user makes).)
SELECT
Ip,
CASE TO_INT(replace_chr(Attempt, 'a', ''))
WHEN 1
THEN
'MUL'
ELSE
'ONE'
END
AS Users,
SUM(TO_INT(replace_chr(Attempt, 'a', ''))) AS Attempts
FROM
--LOG PATH
WHERE
Status = 'SUCCESS'
and
TO_DATE(TO_TIMESTAMP(LDate, 'dd/MM/yyyy-hh:mm:ss')) > SUB( TO_LOCALTIME(SYSTEM_TIMESTAMP()), TIMESTAMP('8','d') )
GROUP BY
Ip,
Users
If I could somehow store a value to increase for each unique Token per IP and store it with the results, but I cannot / do not know a way to achieve this either.
Using DISTINCT won't work either because when I do I get a error saying that DISTINCT cannot work with GROUP BY and my SUM() / Possible COUNT() won't work when Ip isn't in a GROUP BY
(The snippet below is what I have tried with DISTINCT / count)
SELECT
Ip,
COUNT(DISTINCT Token),
SUM(TO_INT(replace_chr(Attempt, 'a', ''))) AS Attempts
FROM
--Log Path
WHERE
Status = 'SUCCESS'
and
TO_DATE(TO_TIMESTAMP(LDate, 'dd/MM/yyyy-hh:mm:ss')) > SUB( TO_LOCALTIME(SYSTEM_TIMESTAMP()), TIMESTAMP('8','d') )
GROUP BY
Ip
How I'd like my result grid to end up: (Without the explanation text of course)
IP Users Attempts
123.456.789.012 4 4 (4 users each trying one time)
120.987.654.321 2 5 (2 users, One user tried once and the other user tried 4 times)
444.444.444.444 1 1 (One user, one attempt)
I hope I'm making sense, otherwise I'll be happy to elaborate / explain anything needed :)
I believe you need two stages. The first stage collapses the entries per-user:
SELECT
Ip,
Token,
MAX(TO_INT(replace_chr(Attempt, 'a', ''))) AS Attempts
FROM
...
GROUP BY
Ip,
Token
The second stage then rolls up by Ip:
SELECT
Ip,
COUNT(*) AS Users,
SUM(Attempts) As TotalAttempts
FROM
...
GROUP BY
Ip
Related
I'm pretty new to KQL, and running into a problem trying to format my data in Azure Sentinel.
I have a query with these columns I'm interested in: Email and IP.
If I run something like summarize count() by Email, IP I get almost what I want, however in some cases, the email value will be the same, but could be coming from a different IP.
Is there a way to have the output formatted so it will show the email value, then under that, list all the IP count values associated with the email?
You can easily create a set (unique values) of IPs per Email
// Data sample generation. Not part of the solution.
let t = range i from 1 to 30 step 1 | extend Email = strcat("email_", tostring(toint(rand(3))), "#", dynamic(["gmail", "outlook", "hotmail"])[toint(rand(3))], ".com"), IP = strcat_delim(".", tostring(toint(rand(256))), tostring(toint(rand(256))), tostring(toint(rand(256))), tostring(toint(rand(256))));
// Solution starts here
t
| summarize make_set(IP) by Email
Email
set_IP
email_0#outlook.com
["22.0.72.237","32.17.234.224","84.232.201.220","181.161.231.252","121.190.204.101"]
email_1#gmail.com
["187.58.44.239","95.117.156.141","16.245.100.138"]
email_2#outlook.com
["154.46.54.212","178.139.208.204","204.197.11.160","160.96.246.141","173.141.14.145","100.35.29.216"]
email_0#gmail.com
["230.16.241.147","173.164.214.236","95.194.124.236","186.101.39.234"]
email_1#hotmail.com
["19.214.101.122","168.72.148.236"]
email_2#hotmail.com
["136.190.117.24","113.147.42.218","224.220.103.201"]
email_0#hotmail.com
["126.176.108.237","201.222.155.151"]
email_2#gmail.com
["132.67.147.234","2.101.57.210"]
email_1#outlook.com
["6.173.214.26","18.169.68.195","87.141.157.8"]
Fiddle
This is the description of my problem on this page
trying to get data like this didnt work for me :
tod_session_req = await session.execute(
select(
Users.firstname
).join(
Users, Users.id == PrivateTods.fu_id
).outerjoin(
Blocks, Users.id == Blocks.blocker
).where(
Blocks.blocked != user.id, # other user didnt block current
)
)
tod_session = tod_session_req.fetchone()
I could use raw queries or other method or any help. thanks alot
In my bot,
If a user wants to play, the bot must check the Tods table to see if there is a row where Tods.su_id value is None.
We consider the Tods.su_id that wants to be connected to the Tods.fu_id to be 100
If there was a colum Tods.su_id with None value, the bot should check the two conditions in the Blocks table
One is whether the Blocks.blocked column is equal to 100 in the rows where the Blocks.blocker column is equal to Tods.fu_id or not. (Did the user Users.fu_id block the user with ID 100 or not).
if user 100 wasnt blocked, go to the next condition.
If it was: read the last two lines
The next condition is :
In the Blocks Table rows, where the Blocks.blocker column is equal to 100, is the blocked column equal to Users.fu_id or not? (Did the user 100 block user Users.fu_id or not).
If not, select that row and give it to me
If it was blocked, continue browsing so that there is no user who can play with it, and in that case it will be added to the tod table because this section is related to Python, not the database.
For AWS ALB access logs (https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html), I would like an Athena SQL query example to sort descending/ascending by the count of the client:port field for elb_status_code/target_status_code during a start and end date (DD-MM-YYYY HH-MM).
The result of the query for target_status_code=500 to be like:
client:port
count of target_status_code=500
70.132.2.XX:port
2570
70.132.2.XX:port
2315
80.122.1.XX:port
1750
...
...
The point would be to find the top clients:port (The IP address and port of the requesting client) with the elb_status_code/target_status_code=4xx or 5xx (https://en.wikipedia.org/wiki/List_of_HTTP_status_codes).
Using the table described in Querying Classic Load Balancer Logs , assuming you partition it by date (the partition key is called date_partition_key below), you could do something like this:
SELECT
CONCAT(request_ip, ':', CAST(request_port AS VARCHAR)) AS client_port,
COUNT(*) AS count_of_status_500
FROM elb_logs
WHERE elb_response_code = '500'
AND date_partition_key BETWEEN '2022-01-01' AND '2022-01-03'
GROUP BY 1
ORDER BY 2 DESC
The 1 and 2 in the group and order by clauses refer back to the first and second items in the select list, i.e. the client port and the count, respectively. It's just a convenient way of not having to repeat the function calls etc.
Meanwhile I found this link
https://aws.amazon.com/premiumsupport/knowledge-center/athena-analyze-access-logs/
with some ALB access logs queries examples. This may be useful for users not very familiar with SQL queries (like me).
I'm trying to count the number of occurrences separate log messages appears per client
My table is in this structure
EventTime - Logmessage - HostName - Client
This query gives me a number of logs for each client:
SELECT Count([Log Message]) AS Count
,[Client]
FROM [test1].[dbo].[logs_test]
Group By Client
How would I go down into a lower level and get the number of times a log appears per client? The output I'm looking to achieve is something like the below
Log Message Count Client
NON ATTEMPT 12 TestClient
Appreciate any help
You will need to change what you are counting and add another level to your grouping...
SELECT LogMessage
, Count(EventTime) AS Count
, Client
FROM [test1].[dbo].[logs_test]
Group By Client, LogMessage
I've got a table with client codes and client names. The client codes used before were e.g.M1003993. They were updated, and now client codes are created starting with 71003993. The client names linked to these client codes (both M, 7) are different clients names, so I cannot filter discrepancy in client codes based on the same client name.
What I need is how to get extracted 1 DISTINCT client code with linked to it client name (M1003993) and join it to the similar DISTINCT client code with its own DISTINCT client name (71003993) meaning that I need a reference list of codes where 1st letter or number of the client code is different whilst the rest 7 numbers are the same? All this is done within 1 table.
My guess is that the same table should be joined back to itself, but how to create this cross-reference when we have (71003993) and (M1003993) codes with different client names.
Image below shows SELECT, maybe it will help to explain what I mean
I need to identify client names for both codes M and 7, so 71003993 (client name 1 ) = M1003993 (client name 2), and this is for 260,000 records
if the last 7 letters of the client code is same ,use the below code.
SELECT *
FROM yourTable y1
JOIN yourTable y2 On RIGHT(y1.clientcode,7)= RIGHT(y2.clientcode,7)
WHERE y1.clientcode like 'M%' AND y2.clientcode like '7%'