I have a list of Bitcoin transaction hashes like this one: a4ad67ea73c88b635e003700290239e6beab7dc5c9b31f58cd34084418b7316d
I would like to convert this hash to raw format (in hexcode). For example, the blockchain API converts this transaction hash to the following (https://blockchain.info/rawtx/a4ad67ea73c88b635e003700290239e6beab7dc5c9b31f58cd34084418b7316d?format=hex):
01000000000101b6e447e3730b6c22a4312c51e98a013b8e1514ebe592a75767349b659dd1eb4b0000000000ffffffff020000000000000000536a4c50000d2ab10002ce909734abc6014d89e07b7d1d5aa1d324eb6af71e2860a470d612483853e078120e105d3ea910720edbb89fc9025e3b4d8e0701e44510686281d5484fbb48444129251371047bf8ad5b5fb9010000000000160014841996f8ff255c875c4f8875a7bd036bf64209210246304302203c5ef41b9f17525714ab840dbd1716c2baae14e14db84a18716f97b5d1c3aa3c021f6be45f733d3ce5094b470385d997e797ffab976610c015833b395197be586601210380a033803cdcfae4dda162741774cbf38af31ebdd11e9bba414590d7fe36835400000000
One way to get the raw transaction bytes isto query the API, but this is very time consuming since my dataset is large. I am therefore wondering if there is an easier way to do this conversion. I am using Python.
Thanks in advance for your help.
As you have a local copy of the blockchain you can use getrawtransaction and decoderawtransaction from python-bitcoinlib.
from bitcoinlib.services.bitcoind import BitcoindClient
bdc = BitcoindClient.from_config('/usr/local/src/.bitcoinlib/config/bitcoin.conf') txid = 'a4ad67ea73c88b635e003700290239e6beab7dc5c9b31f58cd34084418b7316d'
rt = bdc.getrawtransaction(txid)
print("Raw: %s" % rt)
> Raw: 01000000000101b6e447e3730b6c22a4312c51e98a013b8e1514ebe592a75767349b659dd1eb4b0000000000ffffffff020000000000000000536a4c50000d2ab10002ce909734abc6014d89e07b7d1d5aa1d324eb6af71e2860a470d612483853e078120e105d3ea910720edbb89fc9025e3b4d8e0701e44510686281d5484fbb48444129251371047bf8ad5b5fb9010000000000160014841996f8ff255c875c4f8875a7bd036bf64209210246304302203c5ef41b9f17525714ab840dbd1716c2baae14e14db84a18716f97b5d1c3aa3c021f6be45f733d3ce5094b470385d997e797ffab976610c015833b395197be586601210380a033803cdcfae4dda162741774cbf38af31ebdd11e9bba414590d7fe36835400000000
Remember, in Bitcoin transaction hashes are shown little-endian rather than big-endian.
Related
I would like to compute the sha256 of a string in a Free TON-Solidity contract, I do it by storing the string in a TvmBuilder and then passing it as a TvmSlice to sha256(), but the result is not correct (it doesn't match the one computed by sha256sum in my shell). Any idea why ?
Is the TvmBuilder adding some bits that are passed in the slice ?
Yes, tvm builder is kind of TL-B scheme serializer as far as I understand
sha256() function in Free TON Solidity API only takes a TvmBuilder as input, You can compute the hash for a raw string.
Hashing arbitrary string is hashing its BOC , because BOC is the only structure tvm can understand
i think you may want to build BOC out of this string. builder builds cells and cell layout is made of slices + refs. it results in a tree structure of slices mixed with refs, which resolve in blockchain state.
your approach should work for small strings as well as it works with the whole blockchain state. that’s the only way tvm understands data
so hash of string is hash of a cell which has proofs for underlying cells
that’s the way i now understand it, hope that helps.
and if you have string less than 127 bytes, you can pass bytes instead and hash bytes packed in a single cell
tg #freeton_smartcontracts here where clever SmC guys can clarify , because i’m self learner, not really hands on SmC pro
https://github.com/move-ton/ton-client-py/blob/b06b333e6f5582aa1888121cca80424b614e092c/tonclient/abi.py#L49
maybe this or rust core sdk help you
Can I write around 10MB of value (JSON Data as a string) against a key (string - xyz) in Azure redis cache
Size- Standard 1 GB
Version - 4.0.14
I am able to insert 3MB of value , but while inserting 7MB of value it gives network error.
I am using StackExchange.Redis.2.1.58 client from .net console app.
From Redis website:
Strings are the most basic kind of Redis value. Redis Strings are
binary safe, this means that a Redis string can contain any kind of
data, for instance a JPEG image or a serialized Ruby object.
A String value can be at max 512 Megabytes in length.
You could put 'syncTimeout' parameter in ConnectionString like this "RedisConfiguration": {"ConnectionString": "mycache.redis.cache.windows.net:6380,password=$$$$$$$$$$$$$$$$$$=,ssl=True,abortConnect=False,syncTimeout=150000"," DatabaseNumber ": 1}. This parameter sets the "Time (ms) to allow synchronous operations", as can be seen at https://stackexchange.github.io/StackExchange.Redis/Configuration.html. I had this problem when I stored items that took more than 5 seconds to be processed, since this is the default value. You can try increasing this value using the parameter inserted in the connection string. I hope I can help you with this, regards.
I'm currently using Erlang for a big project but i have a question regarding a proper proceeding.
I receive bytes over a tcp socket. The bytes are according to a fixed protocol, the sender is a pyton client. The python client uses class inheritance to create bytes from the objects.
Now i would like to (in Erlang) take the bytes and convert these to their equivelant messages, they all have a common message header.
How can i do this as generic as possible in Erlang?
Kind Regards,
Me
Pattern matching/binary header consumption using Erlang's binary syntax. But you will need to know either exactly what bytes or bits your are expecting to receive, or the field sizes in bytes or bits.
For example, let's say that you are expecting a string of bytes that will either begin with the equivalent of the ASCII strings "PUSH" or "PULL", followed by some other data you will place somewhere. You can create a function head that matches those, and captures the rest to pass on to a function that does "push()" or "pull()" based on the byte header:
operation_type(<<"PUSH", Rest/binary>>) -> push(Rest);
operation_type(<<"PULL", Rest/binary>>) -> pull(Rest).
The bytes after the first four will now be in Rest, leaving you free to interpret whatever subsequent headers or data remain in turn. You could also match on the whole binary:
operation_type(Bin = <<"PUSH", _/binary>>) -> push(Bin);
operation_type(Bin = <<"PULL", _/binary>>) -> pull(Bin).
In this case the "_" variable works like it always does -- you're just checking for the lead, essentially peeking the buffer and passing the whole thing on based on the initial contents.
You could also skip around in it. Say you knew you were going to receive a binary with 4 bytes of fluff at the front, 6 bytes of type data, and then the rest you want to pass on:
filter_thingy(<<_:4/binary, Type:6/binary, Rest/binary>>) ->
% Do stuff with Rest based on Type...
It becomes very natural to split binaries in function headers (whether the data equates to character strings or not), letting the "Rest" fall through to appropriate functions as you go along. If you are receiving Python pickle data or something similar, you would want to write the parsing routine in a recursive way, so that the conclusion of each data type returns you to the top to determine the next type, with an accumulated tree that represents the data read so far.
I only covered 8-bit bytes above, but there is also a pure bitstring syntax, which lets you go as far into the weeds with bits and bytes as you need with the same ease of syntax. Matching is a real lifesaver here.
Hopefully this informed more than confused. Binary syntax in Erlang makes this the most pleasant binary parsing environment in a general programming language I've yet encountered.
http://www.erlang.org/doc/programming_examples/bit_syntax.html
I have a problem regarding a selective read-in routine while using h5py.
f = h5py.File('file.hdf5','r')
data = f['Data']
I have several positive values in the 'Data'- dataset and also some placeholders with -9999.
How I can get only all positive values for calculations like np.min?
np.ma.masked_array creates a full copy of the array and all the benefits from using h5py are lost ... (regarding memory usage). The problem is, that I get errors if I try to read data sets that exceed 100 millions of values per data set using data = f['Data'][:,0]
Or if this is not possible is something like that possible?
np.place(data[...], data[...] <= -9999, float('nan'))
Thanks in advance
You could use:
mask = f['Data'] >= 0
data = f['Data'][mask]
although I am not sure how much memory the mask calculation itself uses.
when I call getwork on my bitcoind server, I get the following:
./bitcoind getwork
{
"midstate" : "695d56ae173bbd0fd5f51d8f7753438b940b7cdd61eb62039036acd1af5e51e3",
"data" : "000000013d9dcbbc2d120137c5b1cb1da96bd45b249fd1014ae2c2b400001511000000009726fba001940ebb5c04adc4450bdc0c20b50db44951d9ca22fc5e75d51d501f4deec2711a1d932f00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000",
"hash1" : "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000",
"target" : "00000000000000000000000000000000000000000000002f931d000000000000"
}
This protocol does not seem to be documented. How do I compute the hash from this data. I think that this data is in little endian. So the first step is to convert everything to big endian? Once that is done, I calculate the sha256 of the data. The data can be divided in two chuncks of 64 bytes each. The hash of the first chuck is given by midstate and therefore does not have to be computed.
I must therefore hash the chunck #2 with sha256, using the midstate as the initial hash values. Once that is done, I end up with a hash of chunk 2, which is 32 bytes. I calculate the hash of this chunk one more time to get a final hash.
Then, do I convert everything to little endian and submit the work?
What is hash1 used for?
The hash calculation is documented at Block hashing algorithm.
Start there for the relatively simple basics. The basic data structures are documented in Protocol specification - Bitcoin Wiki. Note that the protocol definition (and the definition of work) more or less assumes that SHA-256 hashes are 256-bit little-endian values, rather than big-endian as the standard implies. See also
Getwork is more complicated and runs into more serious endian/byte ordering confusion.
First note that the getwork API is optimized to speed up the initial steps of mining.
The midstate and hash1 values are for these performance optimizations and can be ignored. Just look at the "data".
And when a standard sha256 implementation is used, only the first 80 bytes (160 hex characters) of the "data" are hashed.
Unfortunately, the JSON data presented in the getwork data structure has different endian characteristics than what is needed for hashing in the block example above.
They all say to go to the source for the answer, but the C++ source can be big and confusing. A simple alternative is the poold.py code. There is discussion of it here: New mining pool for testing. You only need to look at the first few lines of the "checkwork" routine, and the "bufreverse" and "bytereverse" functions, to get the byte ordering right. In the end it is just a matter of doing a reversal of the bytes in each 32-bit segment of the data. Yes - very odd. But endian issues are tricky and can end up that way....
Some other helpful information on the way "getwork" works can be found in discussions at:
Do I understand header hashing?
Stupid newbie question about the nonce
Note that finding the signal to noise in the original Bitcoin forum is getting very hard, and there is currently an Area51 proposal for a StackExchange site for Bitcoin and Crypto Currency in general. Come join us!
It sounds right, there is a script in javascript that do calculate the hash but I do not fully understand it so I don't know, maybe you understand it better if you look.
this.tryHash = function(midstate, half, data, hash1, target, nonce){
data[3] = nonce;
this.sha.reset();
var h0 = this.sha.update(midstate, data).state; // compute first hash
for (var i = 0; i < 8; i++) hash1[i] = h0[i]; // place it in the h1 holder
this.sha.reset(); // reset to initial state
var h = this.sha.update(hash1).state; // compute final hash
if (h[7] == 0) {
var ret = [];
for (var i = 0; i < half.length; i++)
ret.push(half[i]);
for (var i = 0; i < data.length; i++)
ret.push(data[i]);
return ret;
} else return null;
};
SOURCE: https://github.com/jwhitehorn/jsMiner/blob/4fcdd9042a69b309035dfe9c9ddf716119831a16/engine.js#L149-165
Frankly speaking
Bitcoin block hashing algorithm is not officially described by any source.
"
The hash calculation is documented at Block hashing algorithm.
"
should read
The hash calculation is "described" at Block hashing algorithm.
en.bitcoin.it/wiki/Block_hashing_algorithm
btw the example code in PHP comes with a bug (typo)
the example code in Python generates errors when run by Python3.3 for Windows XP 32
(missing support for string.decode)