I'm a noob at Ethereum and need to make statistics from blockchain data.
I have a full node up and running with RPC ready, but can't find a clue in the doc on how to query :
Total number of transactions in/out for a contract/address
Total number of ether sent/received for a contract/address
The only solution I found was to parse all blocks and transactions in a double for loop and the performance is way too slow to do this.
Can someone help me ?
TL;DR : How do you do getTotalAmountSentTo(address) and getTotalNumberOfTransactionsFor(address) in Ethereum?
If you are using web3JS you can use web3.eth.filter function to find all the transactions you are interested in.
Related
I noticed that running a SELECT count(*) FROM myTable on my larger BQ tables yields long running times, upwards of 30/40 seconds despite the validator claiming the query processes 0 bytes. This doesn't seem quite right when 500 GB queries run faster. Additionally, total row counts are listed under details -> Table Info. Am I doing something wrong? Is there a way to get total row counts instantly?
When you run a count BigQuery still needs to allocate resources (such as: slot units, shards etc). You might be reaching some limits which cause a delay. For example, the slots default per project is 2,000 units.
BigQuery execution plan provides very detail information about the process which can help you better understand the source of the delay.
One way to overcome this is to use an approximate method described in this link
This Slide by Google might also help you
For more details see this video about how to understand the execution plan
I am currently studying the bitcoin and litecoin to try and get a better understanding of cryptocurrencies, and blockchains in general - and I have spotted something in the code that I have a question about.
in src/amount.h - I see the following code...
/** No amount larger than this (in satoshi) is valid.
*
* Note that this constant is *not* the total money supply, which in Bitcoin
* currently happens to be less than 21,000,000 BTC for various reasons, but
* rather a sanity check. As this sanity check is used by consensus-critical
* validation code, the exact value of the MAX_MONEY constant is consensus
* critical; in unusual circumstances like a(nother) overflow bug that allowed
* for the creation of coins out of thin air modification could lead to a fork.
* */
static const CAmount MAX_MONEY = 84000000 * COIN;
Now, the comment here, seems to suggest that this code does not actually define what the total supply of the currency will be, even though the amount of Litecoin available is in fact 84,000,000...
So, my real question :
Is the real total supply held in another piece of code? If so, what am I missing, where can I find this code, and if I were to be trying to edit this (I'm not - but I want to understand what is going on here) - would I need to edit code in multiple places?
NOTE : Tagged bitcoin even though this is litecoin souce in the question, because litecoin doesn't appear to have a stackoverflow tag, and the two codebases are similar anyway.
EDIT : I also wanted to add, that I performed a grep for "84000000" - and only really found that one line of code to be relevant... So I must be missing something...
EDIT 2 : According to literally every coin out there on git that I have looked at - this is the number that they change when adjusting the total supply - so is the comment just wrong - or did I misunderstand it?
I realise this is an old question, but since it hasn't been updated I'll provide an answer.
As the source suggests, MAX_MONEY is simply a sanity check. If someone tries to create a transaction spending 500 million Bitcoin, and it somehow manages to bypass all other sanity checks, the network will still reject it because the amount exceeds MAX_MONEY. So MAX_MONEY is not directly related to total supply, but as you have observed, many alts will set MAX_MONEY to the expected total supply over the lifetime of the coin.
For a pure proof-of-work coin with consistent reward scheme (eg halving every X blocks) the total supply can be pre-calculated, but a future fork could change that.
For a typical proof-of-stake or hybrid proof-of-work and proof-of-stake coin, the maximum supply can be estimated by simulation, but the exact amount will vary depending on network activity.
(This assumes there is not another part of the code that cuts off all rewards after a limit is reached.)
End goal is to find all used addresses of a xpub programmatically, in batches, by querying a local node.
This PHP tool ( https://github.com/dan-da/hd-wallet-addrs ) extracts regular bitcoin addresses from an HD bitcoin wallet . I have to query a local bitcoin node to find out whether the extracted addresses have been used or not.This can easily be found out by querying the node one address at a time ( https://bitco.in/en/developer-reference#getreceivedbyaddress ). It works, but is too slow.
How can the same be done in batches ? i.e is there a Core function call or something to check info about a group of addresses ?
or any other way this can be accomplished ? I am relatively new to bitcoin and don't fully understand its inner workings.
P.S: can't use an external API like blockchain.info
With importAddress and rescan (3 paramter) set to true, you can add the addresses that you want watch.
Than with listTransactions you can list the last transactions that affect the imported address.
Hi,there.
Recently,I want to run a query in bigquery web UI by using "group by" over some tables(tables' name suits xxx_mst_yyyymmdd).The rows will be over 10 million. Unhappily,the query failed with this error:
Query Failed
Error: Resources exceeded during query execution.
I did some improvements with my query language,the error may not happen for this time.But with the increasement of my data, the Error will also appear in the future.So I checked the latest release of Bigquery,maybe there two ways to solve this:
1.After 2016/01/01,Bigquery will change the Query pricing tiers to satisfy the "High Compute Tiers" so that the "resourcesExceeded error" will not happen again.
2.BigQuery Slots.
I checked some documents in Google and didn't find a way on how to use BigQuery Slots.Is there any sample or usecase of BigQuery Slots?Or I have to contact with BigQuery Team to open the function?
Hope someone can help me to answer this question,thanks very much!
A couple of points:
I'm surprised that a GROUP BY with a cardinality of 10M failed with resources exceeded. Can you provide a job id of the failed query so we can investigate? You mention that you're concerned about hitting these errors more often as your data size increases; you should likely be able to increase your data size by a few more orders of magnitude without seeing this; likely you've encountered either a bug or something was strange with either your query or your data.
"High Compute Tiers" won't necessarily get rid of resourcesExceeded. For the most part, resourcesExceeded means that BigQuery ran into memory limitations; high compute tiers only address CPU usage. (and note, they haven't been enabled yet).
BigQuery slots enable you to process data faster and with more reliable performance. For the most part, they also wouldn't help prevent resourcesExceeded errors.
There is currently (as of Nov 5) a bug where you may need to provide an EACH keyword with a GROUP BY. Recent changes should enable BigQuery to automatically select the execution strategy, so EACH shouldn't be needed, but there are a couple of cases where it doesn't pick the right one. When in doubt, add an EACH to your JOIN and GROUP BY operations.
To get your project eligible for using slots you need to contact support.
Because of memory limitation i need to split a result from sql-component (List<Map<column,value>>) into smaller chunks (some thousand).
I know about
from(sql:...).split(body()).streaming().to(...)
and i also know
.split().tokenize("\n", 1000).streaming()
but the latter is not working with List<Map<>> and is also returning a String.
Is there a out of the Box way to create those chunks? Or do i need to add a custom aggregator just behind the split? Or is there another way?
Edit
Additional info as requested by soilworker:
At the moment the sql endpoint is configured this way:
SqlEndpoint endpoint = context.getEndpoint("sql:select * from " + lookupTableName + "?dataSource=" + LOOK_UP_DS,
SqlEndpoint.class);
// returns complete result in one list instead of one exchange per line.
endpoint.getConsumerProperties().put("useIterator", false);
// poll interval
endpoint.getConsumerProperties().put("delay", LOOKUP_POLL_INTERVAL);
The route using this should poll once a day (we will add CronScheduledRoutePolicy soon) and fetch a complete table (view). All the data is converted to csv with a custom processor and sent via a custom component to proprietary software. The table has 5 columns (small strings) and around 20M entries.
I don't know if there is a memory issue. But i know on my local machine 3GB isn't enough. Is there a way to approximate the memory footprint to know if a certain amount of Ram would be enough?
thanks in advance
maxMessagesPerPoll will help you get the result in batches