A new stream called stream 1 is created.
I deliver some changes to stream 1.
Later, I delete stream 1.
So:
Is there a possibility to restore a deleted stream?
If I am not able to restore the stream then, will I loose my changes delivered to it?
Is there a possibility to restore a deleted stream?
Not easily unless you had created snapshots (we covered snapshots in your previous question "Consistency of snapshot code in rtc?"): in that case, when you delete a stream, RTC would ask you to select another existing stream in order to keep ownership of those snapshots.
If you do, then it is trivial to re-create a new stream from a snapshot, assuring you to recover all components at their exact state as recorded by the snapshot.
But if you didn't set any snapshot, then you have to manually re-enter all the components, and set them to (for instance) their most recent baselines.
If I am not able to restore the stream then, will I loose my changes delivered to it?
In any case, as mentioned in the thread "Delete a Stream - any side-effects?"
Change-sets exist independently of any stream, so deleting a
stream does not delete any change-sets.
It will just be harder to get the exact list of change sets back to a new stream if they were only delivered to stream 1 (that you deleted).
Especially if those change set were never grouped inside a baseline (for a given component) or, as explained above, with a snapshot.
But those change sets are not gone.
Related
I'm trying to use S3 as a datastore for an application (for reasons that are not relevant to this question, I can't use a proper DBMS).
My issue is that I want read-after-update consistency: if I try to read an object immediately after updating it, I want to be guaranteed to get the updated object.
S3 guarantees read-after-write consistency (i.e if you try to read an object immediately after creating it, you're guaranteed to succeed).
However, it doesn't guarantee read-after-update consistency (i.e if you try to read an object immediately after updating it, you may get a previous version of the object).
Additionally it has some more caveats.
What can I add to my solution so that it would have read-after-update consistency?
I've thought of a couple of options:
A write-through cache: each write updates a cache first (thinking of using mongoDB as I'm most familiar with it, and it has fast reads for recent objects), then to S3. Each read tries the cache first, and falls back to S3 if not found.
However, this solution doubles the chances of failure (i.e if mongoDB is down, or S3 is down, then my whole database is down). Ideally I'd like to have only S3 as a point of failure.
Always write new objects, never update: I thought I could work around this by utilizing the read-after-write guarantee; instead of creating object /my-bucket/obj-id.json, I'll create /my-bucket/obj-id/v1.json. When I update I'll just create V2 etc.
When reading, I'll need to list the keys with the obj-id path, and choose the latest version.
However, one of the above caveats is:
A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the change is fully propagated, the object might not appear in the list.
Is there any way to "cheat" S3's eventual consistency? Or Am I stuck with a cache component?
Or maybe there's a third, better, idea?
thanks!
Let's assume we have an Aggregate User which has a UserPortraitImage and a Contract as a PDF file. I want to store files in a dedicated document-based store and just hold process-relevant data in the event (with a link to the BLOB data).
But how do I avoid a two-phase commit when I have to store the files and store the new event?
At first I'd store the documents and then the event; if the first transaction fails it doesn't matter, the command failed. If the second transaction fails it also doesn't matter even if we generated some dead files in the store, the command fails; we could even apply a rollback.
But could there be an additional problem?
The next question is how to design the aggregate and the event. If the aggregate only holds a reference to the BLOB storage, what is the process after a SignUp command got called?
SignUpCommand ==> Store documents (UserPortraitImage and Contract) ==> Create new User aggregate with the given BLOB storage references and store it?
Is there a better design which unburdens the aggregate of knowing that BLOB data is saved in another store? And who is responsible for storing BLOB data and forwarding the reference to the aggregate?
Sounds like you are working with something analogous to an AtomPub media-entry/media-link-entry pair. The blob is going into your data store, the meta data gets copied into the aggregate history
But how do I avoid a two-phase commit when I have to store the files and store the new event?
In practice, you probably don't.
That is to say, if the blob store and the aggregate store happen to be the same database, then you can update both in the same transaction. That couples the two stores, and adds some pretty strong constraints to your choice of storage, but it is doable.
Another possibility is that you accept that the two changes that you are making are isolated from one another, and therefore that for some period of time the two stores are not consistent with each other.
In this second case, the saga pattern is what you are looking for, and it is exactly what you describe; you pair the first action with a compensating action to take if the second action fails. So "manual" rollback.
Or not - in a sense, the git object database uses a two phase commit; an object gets copied into the object store, and then the trees get updated, and then the commit... garbage collection comes along later to discard the objects that you don't need.
who is responsible for storing BLOB data and forwarding the reference to the aggregate?
Well, ultimately it is an infrastructure concern; does your model actually need to interact with the document, or is it just carrying a claim check that can be redeemed later?
At first I'd store the documents and then the event; if the first
transaction fails it doesn't matter, the command failed. If the second
transaction fails it also doesn't matter even if we generated some
dead files in the store, the command fails; we could even apply a
rollback. But could there be an additional problem?
Not that I can think of, aside from wasted disk space. That's what I typically do when I want to avoid distributed transactions or when they're not available across the two types of data stores. Oftentimes, one of the two operations is less important and you can afford to let it complete even if the master operation fails later.
Cleaning up botched attempts can be done during exception handling, as an out-of-band process or as part of a Saga as #VoiceOfUnreason explained.
SignUpCommand ==> Store documents (UserPortraitImage and Contract) ==>
Create new User aggregate with the given BLOB storage references and
store it?
Yes. Usually the Application layer component (Command handler in your case) acts as a coordinator betweeen the different data stores and gets back all it needs to know from one store before talking to the other or to the Domain.
If we enable the AppendFileOnly in the redis.conf file, every operation which changes the redis database is loggged in that file.
Now, Suppose Redis has used all the memory allocated to it in the "maxmemory" direcive in the redis.conf file.
To store more data., it starts removing data by any one of the behaviours(volatile-lru, allkeys-lru etc.) specified in the redis.conf file.
Suppose some data gets removed from the main memory, But its log will still be there in the AppendOnlyFile(correct me if I am wrong). Can we get that data back using this AppendOnlyFile ?
Simply, I want to ask that if there is any way we can get that removed data back in the main memory ? Like Can we store that data into disk memory and load that data in the main memory when required.
I got this answer from google groups. I'm sharing it.
----->
Eviction of keys is recorded in the AOF as explicit DEL commands, so when
the file is replayed fully consistency is maintained.
The AOF is used only to recover the dataset after a restart, and is not
used by Redis for serving data. If the key still exists in it (with a
subsequent eviction DEL), the only way to "recover" it is by manually
editing the AOF to remove the respective deletion and restarting the
server.
-----> Another answer for this
The AOF, as its name suggests, is a file that's appended to. It's not a database that Redis searches through and deletes the creation record when a deletion record is encountered. In my opinion, that would be too much work for too little gain.
As mentioned previously, a configuration that re-writes the AOF (see the BGREWRITEAOF command as one example) will erase any keys from the AOF that had been deleted, and now you can't recover those keys from the AOF file. The AOF is not the best medium for recovering deleted keys. It's intended as a way to recover the database as it existed before a crash - without any deleted keys.
If you want to be able to recover data after it was deleted, you need a different kind of backup. More likely a snapshot (RDB) file that's been archived with the date/time that it was saved. If you learn that you need to recover data, select the snapshot file from a time you know the key existed, load it into a separate Redis instance, and retrieve the key with RESTORE or GET or similar commands. As has been mentioned, it's possible to parse the RDB or AOF file contents to extract data from them without loading the file into a running Redis instance. The downside to this approach is that such tools are separate from the Redis code and may not always understand changes to the data format of the files the way the Redis server does. You decide which approach will work with the level of speed and reliability you want.
But its log will still be there in the AppendOnlyFile(correct me if I am wrong). Can we get that data back using this AppendOnlyFile ?
NO, you CANNOT get the data back. When Redis evicts a key, it also appends a delete command to AOF. After rewriting the AOF, anything about the evicted key will be removed.
if there is any way we can get that removed data back in the main memory ? Like Can we store that data into disk memory and load that data in the main memory when required.
NO, you CANNOT do that. You have to take another durable data store (e.g. Mysql, Mongodb) for saving data to disk, and use Redis as cache.
I'm going to be on vacations for a month, the stream needs to be supported. How to allow other users to change my Accurev stream (change time label, backed stream, etc.)?
By default anyone can change the time basis, rename a stream, apply locks, etc..to streams. The only way to prevent these operations from occurring is thorough the server_admin_trig. If you have enabled this trigger to block users from performing these acts, then I suggest you create a group, add whoever needs to admin this stream for you and update the script to allow this group to perform those operations.
I'm trying to do some automation by using the Azure Rest API (using win 8 app) to try to read a blob being written to by another worker role. This other logging worker role is "PUT"ting all these blocks to a blob, and then at the top of the hour commits all these blocks, compresses them then starts a new one. I don't own this logger and would like to try to do this around it without bothering anything.
Here is what I'm doing:
Get the list of all uncommitted blocks from that blob file
Commit that list to that blob and then read/transform as I need
My current approach works for me but it causes issues with the logger, possibly crashing an instance, and definitely losing data when it commits its blob. To prevent this, I've been trying and failing to commit these uncommitted blocks to another temp blob and then reading it from there.
It always returns with a 400: Invalid Block list, when I call PutBlockList. The same exact call succeeds when using the blob file that I got the uncommitted blocks from. It also succeeds to the temp blob if the list of uncommitted blocks is empty.
I've tried to copy the blob, but it fails with a 404: BlobNotFound since it has no content until the blob is committed.
I've tried snapshots, but the documentation states that it omits uncommitted blocks. There is also no documentation of getting the actual data of the uncommitted blocks instead of just the blockids.
So is it possible to commit a blob's uncommitted blocks to a different blob?