It is my understanding that in distributed networks such as those created using blockchain technology, each peer has a shared ledger of all previous transactions since the inception of the chain. As time progresses on, this ledger will grow in size (assuming further uptake of the technology).
If each peer keeps a copy of the ledger locally to ensure the 'decentralization' of the system, what happens when the ledger file becomes far too large to be considered feasible to store locally on a peer machine? Does the storage of different replications of the ledger rest in the hands of people/organizations that can cope/support with the incessant increase in ledger size?
I can imagine the case that if the ledger becomes too large to store on my local machine, and I can no longer store anything but this data, I'd have to abandon the system or invest in dedicated machinery to store these extremely large files... From MB to GB and beyond!
This post, Blockchains don't scale. Not today, at least. But there’s hope does a pretty good job at describing the scaling problem in blockchain and presenting a few possible solutions.
Related
I am currently working on a website where, roughly 40 million documents and images should be served to it's users. I need suggestions on which method is the most suitable for storing content with subject to these requirements.
System should be highly available, scale-able and durable.
Files have to be stored permanently and users should be able to modify them.
Due to client restrictions, 3rd party object storage providers such as Amazon S3 and CDNs are not suitable.
File size of content can vary from 1 MB to 30 MB. (However about 90% of the files would be less than 2 MB)
Content retrieval latency is not much of a problem. Therefore indexing or caching is not very important.
I did some research and found out about the following solutions;
Storing content as BLOBs in databases.
Using GridFS to chunk and store content.
Storing content in a file server in directories using a hash and storing the metadata in a database.
Using a distributed file system such as GlusterFS or HDFS and storing the file metadata in a database.
The website is developed using PHP and Couchbase Community Edition is used as the database.
I would really appreciate any input.
Thank you.
I have been working on a similar system for last two years, the work is still in progress. However, requirements are slightly different from yours: modifications are not possible (I will try to explain why later), file sizes fall in range from several bytes to several megabytes, and, the most important one, the deduplication, which should be implemented both on the document and block levels. If two different users upload the same file to the storage, the only copy of the file should be kept. Also if two different files partially intersect with each other, it's necessary to store the only copy of the common part of these files.
But let's focus on your requirements, so deduplication is not the case. First of all, high availability implies replication. You'll have to store your file in several replicas (typically 2 or 3, but there are techniques to decrease data parity) on independent machines in order to stay alive in case if one of the storage servers in your backend dies. Also, taking into account the estimation of the data amount, it's clear that all your data just won't fit into a single server, so vertical scaling is not possible and you have to consider partitioning. Finally, you need to take into account concurrency control to avoid race conditions when two different clients are trying to write or update the same data simultaneously. This topic is close to the concept of transactions (I don't mean ACID literally, but something close). So, to summarize, these facts mean that you're are actually looking for distributed database designed to store BLOBs.
On of the biggest problems in distributed systems is difficulties with global state of the system. In brief, there are two approaches:
Choose leader that will communicate with other peers and maintain global state of the distributed system. This approach provides strong consistency and linearizability guarantees. The main disadvantage is that in this case leader becomes the single point of failure. If leader dies, either some observer must assign leader role to one of the replicas (common case for master-slave replication in RDBMS world), or remaining peers need to elect new one (algorithms like Paxos and Raft are designed to target this issue). Anyway, almost whole incoming system traffic goes through the leader. This leads to the "hot spots" in backend: the situation when CPU and IO costs are unevenly distributed across the system. By the way, Raft-based systems have very low write throughput (check etcd and consul limitations if you are interested).
Avoid global state at all. Weaken the guarantees to eventual consistency. Disable the update of files. If someone wants to edit the file, you need to save it as new file. Use the system which is organized as a peer-to-peer network. There is no peer in the cluster that keeps the full track of the system, so there is no single point of failure. This results in high write throughput and nice horizontal scalability.
So now let's discuss the options you've found:
Storing content as BLOBs in databases.
I don't think it's a good option to store files in traditional RDBMS because they provide optimizations for structured data and strong consistency, and you don't need neither of this. Also you'll have difficulties with backups and scaling. People usually don't use RDBMS in this way.
Using GridFS to chunk and store content.
I'm not sure, but it looks like GridFS is built on the top of MongoDB. Again, this is document-oriented database designed to store JSONs, not BLOBs. Also MongoDB had problems with a cluster for many years. MongoDB passed Jepsen tests only in 2017. This may mean that MongoDB cluster is not mature yet. Make performance and stress tests, if you go this way.
Storing content in a file server in directories using a hash and storing the metadata in a database.
This option means that you need to develop object storage on your own. Consider all the problems I've mentioned above.
Using a distributed file system such as GlusterFS or HDFS and storing the file metadata in a database.
I used neither of these solutions, but HDFS looks like overkill, because you get dependent on Hadoop stack. Have no idea about GlusterFS performance. Always consider the design of distributed file systems. If they have some kind of dedicated "metadata" serves, treat it as a single point of failure.
Finally, my thoughts on the solutions that may fit your needs:
Elliptics. This object storage is not well-known outside of the russian part of the Internet, but it's mature and stable, and performance is perfect. It was developed at Yandex (russian search engine) and a lot of Yandex services (like Disk, Mail, Music, Picture hosting and so on) are built on the top of it. I used it in previous project, this may take some time for your ops to get into it, but it's worth it, if you're OK with GPL license.
Ceph. This is real object storage. It's also open source, but it seems that only Red Hat people know how to deploy and maintain it. So get ready to a vendor lock. Also I heard that it have too complicated settings. Never used in production, so don't know about performance.
Minio. This is S3-compatible object storage, under active development at the moment. Never used it in production, but it seems to be well-designed.
You may also check wiki page with the full list of available solutions.
And the last point: I strongly recommend not to use OpenStack Swift (there are lot of reasons why, but first of all, Python is just not good for these purposes).
One probably-relevant question, whose answer I do not readily see in your post, is this:
How often do users actually "modify" the content?
and:
When and if they do, how painful is it if a particular user is served "stale" content?
Personally (and, "categorically speaking"), I prefer to tackle such problems in two stages: (1) identifying the objects to be stored – e.g. using a database as an index; and (2) actually storing them, this being a task that I wish to delegate to "a true file-system, which after all specializes in such things."
A database (it "offhand" seems to me ...) would be a very good way to handle the logical ("as seen by the user") taxonomy of the things which you wish to store, while a distributed filesystem could handle the physical realities of storing the data and actually getting it to where it needs to go, and your application would be in the perfect position to gloss-over all of those messy filesystem details . . .
We have use-cases where we would like to store a large volume of data in Google Bigtable for long periods:
during product development
for performance tuning
for demos
We need to store the data but we don't really need it to be "online" all the time. The current cost bottleneck seems to be the cost of nodes which in these cases are idle for long periods.
How is Google Bigtable being used during product development? I am aware of the development mode (and the emulator) and they are fine for some use-cases but we still need the production environment for other use-cases.
Really, what would be ideal is the ability to switch "off" Bigtable (while still paying for data stored but not for nodes) and bring up the nodes when needed. I don't believe this feature exists. In its absence are there other possible workarounds/alternatives?
It's an interesting question. I've done it with smaller projects using Datastore with much smaller sizes (~2Gb) that have hung around for years after disabling billing. Given how much it costs to do backup/restores on those projects, I could imagine this would be cost prohibitive solution in the BigTable world. It is disappointing that Google hasn't provided a better solution for this. They do talk about different storage classes so I'd imagine disabling a project would move its assets to coldline - but that is just rank speculation on my part.
Recently I had a discussion with our Network and System team about putting SQL files on different SAN LUNs. They believed that now a days because of SAN EMC Management process it is wasting the time and energy to put SQL files (Data/Log/Lob/Index/Backups especially TLogs) on separate drives with different spindles. So, could you help me by participating and stating your idea and vision about this discussion please.
I would tend to agree with your SAN administrators on this one. Most SANs today are running RAID-10 or similar technologies, spanning many drives and handling very high IOPS. Physically separating spindles for SQL Server data and logs goes back to the days of local storage with low numbers of drives and low IOPS capabilities.
so - its definitely worth placing your SQL data on separate LUNs, if only for scaling purposes. Definitely don't partition a LUN into multiple filesystems, I have seen that - its a road to destruction.
Putting different volumes on different physical spindles - this depends on a lot of factors.
Whats the workload, OLTP or OLAP (transactional or analytics)?
Whats the storage array? Is it traditional RAID (LUNs are on raidgroups) or is it virtualized provisioning (LUNs on "extents" Pools, extents on raidgroups (example VNX, VMAX, Unity))?
Are you using thin provisioning?
How are you going to scale?
Measure what workload are you dishing out to your current storage devices. Main thing, measure IOPS but also IO block size. IOPS alone is a meaningless number, you want to know the size of IO operations to determine volume placement. Determine what technology are you using, traditional or virtualized. Use latency as the ultimate performance measure.
This should get the conversation with your storage guys started.
does anyone have a way to measure a Bloomberg Terminals general usage or excel API data usage.
This is the most informed way I have seen so far, anyone agree, disagree or have a better way.
https://www.howtogeek.com/howto/43713/how-to-monitor-the-bandwidth-consumption-of-individual-applications/
Thanks
Agree if you're just trying to assess your own server bandwidth usage.
If you're referring to actual data consumption, it depends a little on what platform you're doing it on. You can't infer cost utilization from bandwidth usage, because some fields/data is significantly more expensive, so you may have a person using relatively less data, but incurring notably larger costs due to the entitlements they have requested or are using.
For the enterprise license, you can get a monthly usage summary in arrears for the utilization so it's pretty easy to track.
For B-Pipe you can run BPUR or DFRP on the terminal to run the usage stats, but you need to be permissioned.
On a straight terminal as you mentioned, this data is tracked by Bloomberg to ensure compliance with TOS so you can talk to your rep about getting some usage utilization. The limits team at Bloomberg though sees it binary, so you're either over or you're not.
I have terabytes of files and database dumps that I need to backup off-site.
What's the best way to accomplish this?
I'm currently weighing rsyinc to Amazon EBS or getting an appliance (eg barracuda).
I called a buddy of mine, and he said he uses backula to get all the files on a single disk, then backs that disk up to tape, then sends the tapes off to iron mountain.
Still waiting to hear back from other sysadmins I've contacted. Will post results here.
One common solution to offsite backups that is worth considering is performing the backup onsite and then physically transporting the backup elsewhere, either via secure snail mail or with a service designed for that purpose. If bandwidth is an issue, this may be more practical.
Instead of tapes, I use hard drives that I physically swap out every week. It is less expensive than tape equipment, and easier to plug into another system when necessary.
Back in the late 80s I worked at a place where every week we received a box of tapes of various sorts every monday - we would do one set of weekly backups on the tapes on that box and send them off-site. Evidently they had two of these boxes, one that was in our office and the other they kept locked up somewhere. Then we got an Exabyte drive which had a single tape capacity greater than that whole box of TK-50s, QIC-40s and mag tapes, and it was just simpler to send a single tape home with one of the manager every week.
I'm sure there are still off-site backup systems like that, but I find it easier to keep cycling a couple of 500Gb drives from my home system to my desk at work.
Why not encrpyt it and actually upload to a third party vendor?
I am thinking of doing this with my data at home but have not found a vendor that will just let me do a dump...They all want to install client side apps...
Admittedly, I have not looked that hard...
We use a couple of solutions. We have an offsite backup with another company that we do. We also use several portable hard drives and swap them out each day. Neither solution really handles multiple terabytes of data. More like gigabytes.
In the future, however, we will probably be looking at going the tape router, or something else that is similarly permanent and storable. Terabytes of data is too much to transfer over the wire. When bluray discs become reasonably priced and commercially viable, it may be a good idea to look into the 400GB discs that were touted not long ago. Those would be extremely storage friendly (both in the physical sense and the file size sense), and depending on the longevity stats, may keep for a while, similar to tapes.
I would recommend using a local san from a company like EMC that provides compressed snapshot based replication to remote facilities. It's an expensive solution, but it works.
http://www.emc.com/products/family/emc-centera-family.htm
Over the weekend, I've heard back from a couple of my sysadmin buddies.
It seems the best practice is to backup all machines to a central large disk, then back that disk up to tape, then send the tapes off site (all have used Iron Mountain).
Tapes hold 400-800G and cost $30-$80 per tape.
A tape changer seems to go for $10k on up.
Not sure how much the off-site shipping costs.
I'm scared of tape. I think it gives a false sense of data security. In my own experience from backing up dozens of terrabytes across hundreds of tapes, we discovered that the data recovery rate after a few years fell to about 70%.
To be fair, that was with a now discontinued technology (AIT), but it pretty much put me off tape for life unless it sits on a 1" spool and is reassuringly expensive.
These days, multiple hard drives, multiple locations, and yes, a fall back into Amazon S3 or other cloud provider does no harm (apart from being a tad expensive).