End goal is to find all used addresses of a xpub programmatically, in batches, by querying a local node.
This PHP tool ( https://github.com/dan-da/hd-wallet-addrs ) extracts regular bitcoin addresses from an HD bitcoin wallet . I have to query a local bitcoin node to find out whether the extracted addresses have been used or not.This can easily be found out by querying the node one address at a time ( https://bitco.in/en/developer-reference#getreceivedbyaddress ). It works, but is too slow.
How can the same be done in batches ? i.e is there a Core function call or something to check info about a group of addresses ?
or any other way this can be accomplished ? I am relatively new to bitcoin and don't fully understand its inner workings.
P.S: can't use an external API like blockchain.info
With importAddress and rescan (3 paramter) set to true, you can add the addresses that you want watch.
Than with listTransactions you can list the last transactions that affect the imported address.
Related
I'm a noob at Ethereum and need to make statistics from blockchain data.
I have a full node up and running with RPC ready, but can't find a clue in the doc on how to query :
Total number of transactions in/out for a contract/address
Total number of ether sent/received for a contract/address
The only solution I found was to parse all blocks and transactions in a double for loop and the performance is way too slow to do this.
Can someone help me ?
TL;DR : How do you do getTotalAmountSentTo(address) and getTotalNumberOfTransactionsFor(address) in Ethereum?
If you are using web3JS you can use web3.eth.filter function to find all the transactions you are interested in.
I'm trying to get a list of addresses that have made transactions with a given bitcoin address for a project that examines how people use bitcoin for non-nefarious purposes. I've got a lot of addresses so a web based blockchain explorer like blockchain.info isn't practical.
I've downloaded the blockchain and used bitcoin-abe to dump it into a sqlite database. However I'm not finding addresses anywhere. Are the actual addresses called something different in the blockchain?
The spending conditions, i.e., who is able to spend a given output, are encoded as scripts in the output. What is commonly referred to as a Bitcoin address is little more than a default script format (either pay-to-pubkey or pay-to-pubkey-hash) which require a signature from a private key matching the pubkey in the script. For example P2PKH scripts look like this:
OP_DUP OP_HASH160 <PubkeyHash> OP_EQUALVERIFY OP_CHECKSIG
This checks that the pubkey on the stack matches the hash, and then checks that the signature and pubkey are valid for the transaction.
ABE stores the output scripts, but appears not to create an index for the addresses. So you probably want to convert the addresses that you're looking for into the script version (see the wiki for details on how to extract the pubkey hash or pubkey from the address). Once you have the pubkey hash or pubkey you construct a binary script similar to this (hexencoded):
76a914<pubkey-hash>88ac
You should then be able to search for these in the database ABE gives you.
You need to write a cron job using the BTC address of the user and check whether the transaction is made or not.
https://www.blockchain.com/explorer
Ex. https://github.com/bitpay/insight
I've got a problem i have been trying to work out I have googled a few things that are similar to what i want to do but can't work out exactly how to do it,
I have around 250M ip address's and i want to look that up against the maxmind geolite2 data so that i can tell what country each IP-address originates from,
I have imported all the data into my Redshift cluster with talend,
table a has ID and 'ipaddress' ie 10.0.0.5
table b (maxmind) has country name and IP range as expressed as 10.0.0.0/24
how could i use Redshift SQL to match these two considering the size of my source data source?
edit: heres the link to the geolite2 data https://dev.maxmind.com/geoip/geoip2/geolite2/
You could try using Amazon Redshift's ability to Import Custom Python Library Modules to load the netaddr library. Then, you could use the library within a User Defined Function written in Python.
See also: IP Range to CIDR conversion in Python?
I am new to redis and I am trying to figure out how redis can be used.
So please let me know if this is a right way to build an application.
I am building an application which has got only one data source. I am planning to run a job on nightly basis to get data into a file.
Now I have a front end application, that needs to render this data in different formats.
Example application use case
Download processed applications by a university on nightly basis.
Display how many applications got approved or rejected.
Display number of applications by state.
Let user search for an application by application id.
Instead of using postgres/mysql like relational database, I am thinking about using redis. I am planning to store data in following ways.
Application id -> Application details
State -> List of application ids
Approved -> List of application ids (By date ?)
Declined -> List of application ids (By date ?)
Is this correct way to store data into redis?
Also if someone queries for all applications in california for a certain date,
I will be able to pull application ids in one call but to get details for each application, do I need to make another request?
Word of caution:
Instead of using postgres/mysql like relational database, I am thinking about using redis.
Why? Redis is an amazing database, but don't use the right hammer for the wrong nail. Use Redis if you need real time performance at scale, but don't try make it replace an RDBMS if that's what you need.
Answer:
Fetching data efficiently from Redis to answer your queries depends on how you'll be storing it. Therefore, to determine the "correct" data model, you first need to define your queries. The data model you proposed is just a description of the data - it doesn't really say how you're planning to store it in Redis. Without more details about the queries, I would store the data as follows:
Store the application details in a Hash (e.g. app:<id>)
Store the application IDs in a per state in Set (e.g. apps:<state>)
Store the approved/rejected applications in two Sorted Sets, the id being the member and the date being the score
Also if someone queries for all applications in california for a certain date, I will be able to pull application ids in one call but to get details for each application, do I need to make another request?
Again, that depends on the data model but you can use Lua scripts to embed this logic and execute it in one call to the database.
First of all you can use a Hash to store structured Data. With Lists (ZSets) and Sets you can create indexes for an ordered or unordered access. (Depending on your requirements of course. Make a list of how you want to access your data).
It is possible to get all data as json of an index in one go with a simple redis script (example using an unordered set):
local bulkToTable = function(bulk)
local retTable = {};
for index = 1, #bulk, 2 do
local key = bulk[index];
local value = bulk[index+1];
retTable[key] = value;
end
return retTable;
end
local functionSet = redis.call("SMEMBERS", "app:functions")
local returnObj = {} ;
for index = 1, #functionSet, 1 do
returnObj[index] = bulkToTable(redis.call("HGETALL", "app:function:" .. functionSet[index]));
returnObj[index]["functionId"] = functionSet[index];
end
return cjson.encode(returnObj);
more information about redis scripts see here : http://www.redisgreen.net/blog/intro-to-lua-for-redis-programmers/
Has anyone come up with an elegant way to search data stored on Authorize.net's Customer Information Manager (CIM)?
Based on their XML Guide there doesn't appear to be any search capabilities at all. That's a huge short-coming.
As I understand it, the selling point for CIM is that the merchant doesn't need to store any customer information. They merely store a unique identifier for each and retrieve the data as needed. This may be great from a PCI Compliance perspective, but it's horrible from a flexibility standpoint.
A simple search like "Show me all orders from Texas" suddenly becomes very complicated.
How are the rest of you handling this problem?
The short answer is, you're correct: There is no API support for searching CIM records. And due to the way it is structured, there is no easy way to use CIM alone for searching all records.
To search them in the manner you describe:
Use getCustomerProfileIdsRequest to get all the customer profile IDs you have stored.
For each of the CustomerProfileIds returned by that request, use getCustomerProfileRequest to get the specific record for that client.
Examine each record at that time, looking for the criterion you want, storing the pertinent records in some other structure; a class, a multi-dimensional array, an ADO DataTable, whatever.
Yes, that's onerous. But it is literally the only way to proceed.
The previously mentioned reporting API applies only to transactions, not the Customer Information Manager.
Note that you can collect the kind of data you want at the time of recording a transaction, and as long as you don't make it personally identifiable, you can store it locally.
For example, you could run a request for all your CIM customer profile records, and store the state each customer is from in a local database.
If all you store is the state, then you can work with those records, because nothing ties the state to a specific customer record. Going forward, you could write logic to update the local state record store at the same time customer profile records are created / updated, too.
I realize this probably isn't what you wanted to hear, but them's the breaks.
This is likely to be VERY slow and inefficient. But here is one method. Request an array of all the customer Id's, and then check each one for the field you want... in my case I wanted a search-by-email function in PHP:
$cimData = new AuthorizeNetCIM;
$profileIds = $cimData->getCustomerProfileIds();
$profileIds = $cimData->getCustomerProfileIds();
$array = $profileIds->xpath('ids');
$authnet_cid = null;
/*
this seems ridiculously inefficient...
gotta be a better way to lookup a customer based on email
*/
foreach ( $array[0]->numericString as $ids ) { // put all the id's into an array
$response = $cimData->getCustomerProfile($ids); //search an individual id for a match
//put the kettle on
if ($response->xml->profile->email == $email) {
$authnet_cid = $ids;
$oldCustomerProfile = $response->xml->profile;
}
}
// now that the tea is ready, cream, sugar, biscuits, you might have your search result!
CIM's primary purpose is to take PCI compliance issues out of your hands by allowing you to store customer data, including credit cards, on their server and then access them using only a unique ID. If you want to do reporting you will need to keep track of that kind of information yourself. Since there's no PCI compliance issues with storing customer addresses, etc, it's realistic to do this yourself. Basically, this is the kind of stuff that needs to get flushed out during the design phase of the project.
They do have a new reporting API which may offer you this functionality. If it does not it's very possible it will be offered in the near future as Authnet is currently actively rolling out lots of new features to their APIs.