Creating daily bins for blockchain transactions - google-bigquery

Question

after some manipulation, I ended up with a table in GBQ that lists all transactions made on blockchain (around 280 million rows):
+-------+-------------------------+--------+-------+----------+
| Linha | timestamp | sender | value | receiver |
+-------+-------------------------+--------+-------+----------+
| 1 | 2018-06-28 01:31:00 UTC | User1 | 1.67 | User2 |
| 2 | 2017-04-06 00:47:29 UTC | User3 | 0.02 | User4 |
| 3 | 2013-11-27 13:22:05 UTC | User5 | 0.25 | User6 |
+-------+-------------------------+--------+-------+----------+
Since this table has all transactions, if I sum all the values for each user up to a given date, I may have his balance, and once I have close to 22 million users, I want to binarize them by the amount of coin they have. I used this code to go through all the dataset:
#standardSQL
SELECT
COUNT(val) AS num,
bin
FROM (
SELECT
val,
CASE
WHEN val > 0 AND val <= 1 THEN '0_to_1'
WHEN val > 1
AND val <= 10 THEN '1_to_10'
WHEN val > 10 AND val <= 100 THEN '10_to_100'
WHEN val > 100
AND val <= 1000 THEN '100_to_1000'
WHEN val > 1000 AND val <= 10000 THEN '1000_to_10000'
WHEN val > 10000 THEN 'More_10000'
END AS bin
FROM (
SELECT
max(timestamp),
receiver,
SUM(value) as val
FROM
`table.transactions`
WHERE
timestamp < '2011-02-12 00:00:00'
group by
receiver))
GROUP BY
bin
Which gives me something like:
+-------+-------+---------------+
| Linha | num | bin |
+-------+-------+---------------+
| 1 | 11518 | 1_to_10 |
| 2 | 9503 | 100_to_1000 |
| 3 | 18070 | 10_to_100 |
| 4 | 20275 | 0_to_1 |
| 5 | 1781 | 1000_to_10000 |
| 6 | 158 | More_10000 |
+-------+-------+---------------+
Now I want to iterate through the rows of my transactions tables checking the number of users in each bin at the end of every day. The final table should be something like this:
+-------------------------+---------+-----------+-----------+-------------+---------------+------------+
| timestamp | 0_to_1 | 1_to_10 | 10_to_100 | 100_to_1000 | 1000_to_10000 | More_10000 |
+-------------------------+---------+-----------+-----------+-------------+---------------+------------+
| 2009-01-09 00:00:00 UTC | 1 | 1 | 0 | 0 | 0 | 0 |
| 2009-01-10 00:00:00 UTC | 0 | 2 | 0 | 0 | 0 | 0 |
| ... | ... | ... | ... | ... | ... | ... |
| 2018-09-10 00:00:00 UTC | 2342823 | 124324325 | 43251315 | 234523555 | 2352355556 | 12124235231|
+-------------------------+---------+-----------+-----------+-------------+---------------+------------+
I can't order by timestamp to make my life easier because the dataset is too large, so I would appreciate some ideas. I wonder if there is ome way to improve performance and save resources using pagination, for example. I've heard about it, but don't have a clue how to use it.
Thanks in advance!
UPDATE: after some work, now I do have a transactions table ordered by timestamps.

Answer 1

The query below should give you the count of transactions within each bin by timestamp. Now, keep in mind that this query will evaluate the value of a transaction at the row level.
SELECT
timestamp,
COUNT(DISTINCT(CASE
WHEN value > 0 AND value <= 1 THEN receiver
END)) AS _0_to_1,
COUNT(DISTINCT(CASE
WHEN value > 1 AND value <= 10 THEN receiver
END)) AS _1_to_10,
COUNT(DISTINCT(CASE
WHEN value > 10 AND value <= 100 THEN receiver
END)) AS _10_to_100,
COUNT(DISTINCT(CASE
WHEN value > 100 AND value <= 1000 THEN receiver
END)) AS _100_to_1000,
COUNT(DISTINCT(CASE
WHEN value > 1000 AND value <= 10000 THEN receiver
END)) AS _1000_to_100000,
COUNT(DISTINCT(CASE
WHEN value > 10000 THEN receiver
END)) AS More_10000
FROM `table.transactions`
WHERE timestamp = TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
GROUP BY 1
Regarding your question for performance, one area you may want to explore (if possible) is to create a partitioned version of this big table. This will help you 1) improve performance, and 2) reduce cost of query the data for a specific data range. You can find more info here
EDIT
I added a WHERE clause to the query to filter for the previous day. I am assuming you will run your query, for example, today to get the data from the previous day. Now you may need to adjust CURRENT_TIMESTAMP() to your time zone by adding an additional TIMESTAMP_SUB(...., INTERVAL X HOUR or TIMESTAMP_ADD(...., INTERVAL X HOUR, where X is the number of hours that need to be subtracted or added to match the time zone of the data you are analyzing.
Also, you may need to CAST(timestamp AS TIMESTAMP) depending on the type of your field.