I have a mining pool that is going to take several days to synch the blockchain, I'm wondering if I can have miners mining on it before the blockchain synchs or if I have to wait before blocks will be generated. I suspect that I do, but perhaps this lovely site will erase that suspicion entirely :)
Are you talking about bitcoin network or any private blockchain network?
Yes, you have to sync entire blockchain wallet.
As we know that mining is the process of adding transactions to the large distributed public ledger of existing transactions and cryptocurrency wallet blockchain remains in syncing process automatically.
Mining and syncing of blocks works simultaneously, can you please explain what you exactly want to achieve?
Related
I am trying to go over all transactions data from every block on the bitcoin blockchain from the previous 4 years. With almost 2k transaction per block, it will take a lot of queries per block.
I have a full node running locally and I tried two ways:
Python with RPC: This is very slow and keeps losing connection after some time (httpx.ReadTimeout)
Python with os.popen commands: Doesn't have the connection problem, but still very slow.
Would there be any other way? Any recommendation on how to analyze bulk data from the blockchain? The methods listed above are unfeasible given the time it would take.
EDIT: The problem isn't memory, but the time the bitcoin node takes to answer the queries.
Hey there are differents ways to fetch bitcoin blockchain data:
Network level using P2P messages (this method doesn't require to setup a node)
Parsing .blk files which are synchronized by your node
Querying the application interface RPC
P2P messages and .blk files are raw encoded, so you will need to decode blocks and transactions.
The RPC interface abstract the raw decoding but it's slower (because it decodes).
We wrote a paper with Matthieu Latapy to give instructions about collecting the whole Bitcoin blockchain and indexing in order to make parsing efficient.
Step-by-step procedure
Full paper
Repository
Website
I'm developing a web/mobile app similar to dropbox or drive, but I'm finding problems about storage cost.
As I said, my application lets the user storage files and retrieve it later, but my users pay only one time, so I've found Amazon S3 and GCS too expensive, because they charge every month, also they charge per transaction and download bandwith, so it would be unaffordable.
In my search I've wondered how could work a website like youtube considering that the cost is too much.
I've found Backblaze and It would be cheaper for my needs, but still goes very expensive.
I've considered using Youtube API for upload videos and reduce costs, but my application would work offline too (It would sync frequently) so I don't think youtube works for offline playing.
Could you help me please?
Thank you.
This is not really an answer, but your situation is of interest to me as I am asked this constantly by customers. What is the cheapest solution, and not what is the most appropriate solution?
When you try to reduce storage costs too far, reliability will usually drop significantly. The cost for S3 is dirt cheap to me and Backblaze is 4 times cheaper (I don't have personal experience with Backblaze).
Think about your business model a bit. If the service that you are offering cannot offer the reliability that will be required, you will quickly fail. A couple of data loss situations and poof, your business is gone.
I understand bitcoin uses blockchain technology to maintain a decentralised ledger of all transactions. I have also read many posts eluding to future applications of blockchain technology, none of which have been very clear to me.
Is blockchain technology simply a decentralised database with consensus validation of the data? If this was the case surely the db would grow to be too large to be effectively decentralised?
To help me understand, can anyone point me to a clear example of a non-bitcoin blockchain application?
Yes, its true that the blockchain database would grow overtime which is what is called "blockchain bloat". Currently the blockchain growth of Bitcoin is roughly less than 100mb a day. Today (2016) the bitcoin blockchain takes up about 60-100GB of space which took about 6 years to accumulate. It is indeed growing faster, but also limited by the blocksize cap of 1MB per block (every 10 minutes). Some proposed solutions have been:
SPV nodes: This is how your phone doesn't need to download the entire blockchain, but retrieve its data from SPV nodes that have the entire blockchain.
Lightning network - This is how Bitcoin can overcome the 1MB memory cap.
Those are just some of the solutions for bitcoin that I know of, as for altcoin related solutions. NXT/Ardor has implemented the solution of pruned data, because NXT/Ardor gives the ability to upload arbitrary data and messages onto its blockchain, the bloat is much more apparent in this scenario. The NXT/Ardor blockchain has the ability to delete previous data every 2 weeks and only keep the hash of its data on the blockchain which only takes a few KB. They also have the ability to retain all of the blockchain data with the pruning turned off which would mark a node as an Archival Node and other nodes can replicate this node and become an Archival node.
From my understanding NXT/Ardor has been one of the few blockchains that has production ready decentralized data storage system, marketplace, stock exchange, and messaging system built into its blockchain.
Blockchain is not just a decentralised database, but it is much more than that. While the original Bitcoin blockchain allowed only value to be transferred, along with limited data with every transaction, several new blockchains have been developed in the past 2-3 years, which have much more advanced native scripting and programming capabilities.
Apart from the Bitcoin blockchain, I would say that there a few other major blockchains like Ethereum, Ripple, R3's Corda, Hyperledger. Although Ethereum has a crypto-currency called Ether, it is actually a Turing complete EVM (Ethereum Virtual Machine). Using Ethereum, you can create Smart Contracts that would themselves run in a decentralised manner. As a developer, it opens up completely new avenues for you and changes your perspective of writing programs. While Ripple is mainly geared towards payments, Corda and Hyperledger are built with a view of being private/permissioned blockchains, to solve the issues such as scalability, privacy, and identity. The target markets for Hyperledger and Corda are mostly banks and other Financial Institutions.
As for the non-bitcoin application of blockchain, you can certainly look at some companies like Consensys (multiple different use cases on blockchain), Digix Global (gold tokens on the blockchain), Everledger (tracking of diamonds on the blockchain), Otonomos (Company registration on the blockchain), OT Docs (Trade Finance and document versioning on the blockchain) amongst others.
Blockchain is:
Name for a data structure,
Name for an algorithm,
Name for a suite of Technologies,
An umbrella term for purely distributed peer-to-peer systems with a common application area,
A peer-to-peer-based operating system with its own unique rule set that utilizes hashing to provide unique data transactions with a distributed ledger
Blockchain is much more than a "database". Yes the blocks on the chain stores data but it is more like a service. There are many applications of blockchain. Read about them: here. If you want to see the code of a blockchain application, try this one: here.
Blockchain is combination of p2p network, decentralised database and asymmetric cryptography
P2P network means you can transfer data between two deferent network nodes without any middleman, decentralised db means every nodes of network has one replica of network db and asymmetric cryptography means you can use digital signature to validate the authenticity and integrity of a messages
In current project we currently use 8 worker role machines side by side that actually work a little different than azure may expect it.
Short outline of the system:
each worker start up to 8 processes that actually connect to cloud queue and processes messages
each process accesses three different cloud queues for collecting messages for different purposes (delta recognition, backup, metadata)
each message leads to a WCF call to an ERP system to gather information and finally add retreived response in an ReDis cache
this approach has been chosen over many smaller machines due to costs and performance. While 24 one-core machines would perform by 400 calls/s to the ERP system, 8 four-core machines with 8 processes do over 800 calls/s.
Now to the question: when even increasing the count of machines to increase performance to 1200 calls/s, we experienced outages of Cloud Queue. In same moment of time, 80% of the machines' processes don't process messages anymore.
Here we have two problems:
Remote debugging is not possible for these processes, but it was possible to use dile to get some information out.
We use GetMessages method of Cloud Queue to get up to 4 messages from queue. Cloud Queue always answers with 0 messages. Reconnect the cloud queue does not help.
Restarting workers does help, but shortly lead to same problem.
Are we hitting the natural end of scalability of Cloud Queue and should switch to Service Bus?
Update:
I have not been able to fully understand the problem, I described it in the natual borders of Cloud Queue.
To summarize:
Count of TCP connections have been impressive. Actually too impressive (multiple hundreds)
Going back to original memory size let the system operate normally again
In my experience I have been able to get better raw performance out of Azure Cloud Queues than service bus, but Service Bus has better enterprise features (reliable, topics, etc). Azure Cloud Queue should process up to 2K/second per queue.
https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/
You can also try partitioning to multiple queues if there is some natural partition key.
Make sure that your process don't have some sort of thread deadlock that is the real culprit. You can test this by connecting to the queue when it appears hung and trying to pull messages from the queue. If that works it is your process, not the queue.
Also take a look at this to setup some other monitors:
https://azure.microsoft.com/en-us/documentation/articles/storage-monitor-storage-account/
It took some time to solve this issue:
First a summarization of the usage of the storage account:
We used the blob storage once a day pretty heavily.
The "normal" diagonistics that Azure provides out of the box also used the same storage account.
Some controlling processes used small tables to store and read information once an hour for ca. 20 minutes
There may be up to 800 calls/s that try to increase a number to count calls to an ERP system.
When recognizing that the storage account is put under heavy load we split it up.
Now there are three physical storage accounts heaving 2 queues.
The original one still keeps up to 800/s calls for increasing counters
Diagnositics are still on the original one
Controlling information has been also moved
The system runs now for 2 weeks, working like a charm. There are several things we learned from that:
No, the infrastructure is "not just there" and it doesn't scale endlessly.
Even if we thought we didn't use "that much" summarized we used quite heavily and uncontrolled.
There is no "best practices" anywhere in the net that tells the complete story. Esp. when start working with the storage account a guide from MS would be quite helpful
Exception handling in storage is quite bad. Even if the storage account is overused, I would expect some kind of exception and not just returning zero message without any surrounding information
Read complete story here: natural borders of cloud storage scalability
UPDATE:
The scalability has a lot of influences. You may are interested in Azure Service Bus: Massive count of listeners and senders to be aware of some more pitfalls.
I'm curious how banks handle recurring tasks of their users. And if they have thousands of those tasks, for example paypal recurring payments, how does paypal handle them? What software they use? I dont think they use cron\quartz for such tasks.
What do they do if system goes down and unable to process user tasks?
Is there any solution like mysql has with event scheduler? Have anyone tried RabbitMQ to process various events?
CICS, COBOL and over-night batch jobs still abound.
Banks and telecom sector have very power full system and they are just develop for this specific purpose we just need to integrate them.
When i was working in a telecom company they have lot of servers and all of them have 32+GB of ram etc.
For banks they also have very power full system, When ever their system crash automatically other server start working. Banks have central repository where they save their data due to the central system they didn't get data consistency issue.