How can I get summary of transaction history from etherscan to example Time Diff between first and last for a fraud detection model - pandas

I want to create data prep script that pulls live data from the blokchcain (possibly using a centralized cache or API like etherscan). I'm wondering what is easy way to scrape all this data from blockchain in Python? I need the following columns for my fraud detection model.
Here is a description of the rows in the dataset:
Address: The address of the ethereum account
FLAG: Whether the transaction is fraud or not (This is our "target" column) Avg min between sent tnx: Average time between sent transactions for account in minutes
Avgminbetweenreceivedtnx: Average time between received transactions for account in minutes
TimeDiffbetweenfirstand_last(Mins): Time difference between the first and last transaction
Sent_tnx: Total number of sent normal transactions
Received_tnx: Total number of received normal transactions
NumberofCreated_Contracts: Total Number of created contract transactions
UniqueReceivedFrom_Addresses: Total Unique addresses from which account received transactions
UniqueSentTo_Addresses20: Total Unique addresses from which account sent transactions
MinValueReceived: Minimum value in Ether ever received
MaxValueReceived: Maximum value in Ether ever received
AvgValueReceived: Average value in Ether ever received
MinValSent: Minimum value of Ether ever sent
MaxValSent: Maximum value of Ether ever sent
AvgValSent: Average value of Ether ever sent
MinValueSentToContract: Minimum value of Ether sent to a contract
ERC20MostRecTokenType: Most received token for account via ERC20 transactions
The dataset comes from this link: https://www.kaggle.com/vagifa/ethereum-frauddetection-dataset/version/1

Related

Seller/Buyer initiated information

I am trying to calculate stock market liquidity by using the effective spread, quoted spread and price impact. one of the variables that I need is seller and buyer initiated trades. How am I able to extract that information from Bloomberg ?

Aggregate several bitcoin addresses into a single wallet

I have a few bitcoin addresses (and private keys) with small amounts on them (0.001 to 0.01 BTC). Obviously sending them one by one to exchange does not worth their value due to high transaction fee.
Is there a way to aggregate them all into a single address to exchange the total amount then
You don't need an exchange to perform bitcoin to bitcoin transactions. You can send the all small amounts to one bitcoin address using any wallet, and the only fee you are facing will be the miner's fee, which is in your control. You can set a very small fee, or no fee at all - that will simply move your transactions to a lower priority. Wait a few days, and they will be confirmed.
If you want to do it using code, check https://github.com/primal100/pybitcointools

Hyperledger Online payment use case: front end GO implementation

I am new to hyperledger and online payment.
One scenario I have in mind is: if A is an online shop, B is a consumer. B orders an item from A's shop and pays with Bitcoin/hyperledger. A ships the item to B once the payment is received.
With hyperledger, the process goes:
B sends A the payment via blockchain;
A is able to query how much money A owns;
But the question is:
How does A know if the transaction was from B?
How does A know how much money was sent by B?
How does A know if the transaction from B is for the specific ordered item?
Thanks,
What you'd usually do in Bitcoin is to create a new payment address that is bound to the specific order. The invoice would include that address with instructions to send the specified amount to that address and the recipient could simply wait for an incoming transaction with an output destined for that address.
At this point you'd know which customer has paid for which order, to check the amount just check the output value field in the transaction and compare it to the invoice.

Predicting WorldPay transaction ID with Shopify

We're using the WorldPay payment processor with Shopify.
When a transaction is created in WorldPay it's given an ID based on its order number in Shopify. For example:
#1010.1 for order number 1010
#1011.1 for order number 1011
#1013.2 for order number 1013
For most orders the number after the dot is a "1". But for at least one it's "2". Is there any way of predicting what this number is, or of fetching it via the API?
The part after the #1013 is the transaction number. If the transaction fails for whatever reason that number is incremented when the user attempts the transaction again. You can retrieve the transactions for an order using the transactions endpoint. http://api.shopify.com/transactions.html

Dealing with Amazon Product Advertising API Throttle limits

For those of you who use the Amazon Product Advertising API, what experience have you had with running into their throttle? Supposedly, the limit is set at 1 request per second, is that your experience?
I want my site to grow to be nation-wide, but I'm concerned about its capability to make all the Amazon API requests without getting throttled. We cache all the responses for 24 hours, and also throttle our own users who make too many searches within a short period.
Should I be concerned? Any suggestions?
I believe they have changed it. Per this link:
https://forums.aws.amazon.com/message.jspa?messageID=199771
Hourly request limit per account = 2,000 + 500 * [Average associate revenue driven per day over the past 30 days period]/24 to a maximum of 25,000 requests per hour.
Here is the latest on request limits that I could find, effective Sept 3rd, 2012.
If your application is trying to submit requests that exceed the
maximum request limit for your account, you may receive error messages
from Product Advertising API. The request limit for each account is
calculated based on revenue performance. Each account used to access
the Product Advertising API is allowed an initial usage limit of 1
request per second. Each account will receive an additional 1 request
per second (up to a maximum of 10 requests per second) for every
$4,600 of shipped item revenue driven per hour in a trailing 30-day
period.
https://affiliate-program.amazon.com/gp/advertising/api/detail/faq.html
They have updated their guidelines, you now have more requests when you sell more items.
Effective 23-Jan-2019, the request limit for each account is calculated based on revenue performance attributed to calls to the
Product Advertising API (PA API) during the last 30 days.
Each account used for Product Advertising API is allowed an initial
usage limit of 8640 requests per day (TPD) subject to a maximum of 1
request per second (TPS). Your account will receive an additional 1
TPD for every 5 cents or 1 TPS (up to a maximum of 10) for every $4320
of shipped item revenue generated via the use of Product Advertising
API for shipments in the last 30 days.
Source: https://docs.aws.amazon.com/AWSECommerceService/latest/DG/TroubleshootingApplications.html
Amazon enforces limits on how many calls you can make per hour and per second.
You can increase the former by following the sanctioned route (increase commission revenue) or by privately petitioning Amazon with a valid reason. When whitelisted, your limit will go up to 25,000 calls per hour, which is more than good enough for the vast majority of projects I can think of.
The latter limit is murkier and enforced depending on the type of query you make. My interpretation is that it is meant to keep serial crawlers who do batch item lookups in check. If you are simply doing keyword searches etc., I would not worry so much about it. Otherwise, the solution is to distribute your calls across multiple IPs.
One other point to keep in mind if you are querying multiple locales is to use separate accounts per locale. Some locales are grouped and will count to the same call quota. European Amazons, for instance, form such a pool.