I use rabbitmq-server-3.6.1.
It seems there are two options how to set synchronization batch size.
First one is policy option "ha-sync-batch-size". It is described in RabbitMQ documentation here.
Second one is configuration file option "mirroring_sync_batch_size", which I cannot find described anywhere and do not understand from RabbitMQ source code how it works.
The question is related to the mirroring synchronisation and consists of several subquestions:
Does "mirroring_sync_batch_size" influence the synchronisation batch size in the same way as "ha-sync-batch-size" does?
If answer to question #1 is yes, which of the options takes precedence when both are present?
If answer to question #1 is no, what does "mirroring_sync_batch_size" do?
What is purpose and function of configuration file option "mirroring_flow_control"?
Thanks in advance for your answers.
mirroring_sync_batch_size configuration variable is used as default value if no policy is set.
mirroring_flow_control option enables or disables flow control mechanism between channel and mirror slave processes. Flow control is described in this blog post
Related
I am converting an old tcp only server to use SSL (via IBM's GSkit), and one of the problems is getting the SSL handle into the spawned program. The original code passes the raw socket in via calls to the givedescriptor() and then uses takedescriptor() to get and then use the passed in socket.
Is there a GSKit/SSL equivalent of the give/take descripter methods?
givedescriptor() API documentation
UPDATE:
The issue is that the socket and the SSLHandle are created in one process, which initialized the SSL environment, and then need to be passed on to another process entirely - hence the need to give/take descriptor, as the socket / SSLHandle need to 'given' to the new process (it is actually an RPG program that is submitted and runs independently from the original program).
UPDATE 2:
Looks similar to this question, so I'll have a read of that as well.
From the other article (which doesn't have a code based answer, but a written solution)
"It looks like the session handles are just pointers to some storage
in heap. Due to the design of Single Level Store, you could copy them
via shared memory (memmap, shmget/shmat, ...). You just have to ensure
that the process that opened the GSK environment doesn't die or the
activation group will get cleaned up and those pointers will become
invalid. You also will probably need to put a mutex or some other
locking primitive around them if you're going to have multiple threads
accessing the shared data structure."
UPDATE 3:
This is the example I am using to share the memory between processes - Example: Using semaphore set and shared memory functions, still not exactly solved the issue yet though.
UPDATE 4:
I thought I'd add more details of why I need to ask the question. I am changing a non-blocking TCP server that is used as a connection point to an IBM i. It has the 'standard' mechanism for handling connections as they come it, creating threads and negotiating the connections in these threads. The threads then create independent process (via sbmjob). In the vanilla TCP version we can then give the running job the handle of the socket via the give/takedescriptor function, and will merrily write to and from the socket.
So I need an equivalent way of getting the independently running program to be able to write to SSL.
It maybe that this is not possible with the current mechanism.
There is no such thing as an 'SSL handle' known to the operating system and inheritable by child processes or transferable to other processes. The 'SSL handle' will inevitably be a pointer into some opaque data structure in the originating process, as SSL is an application layer protocol, and therefore implemented in the process, not in the kernel. So you can't 'give' an 'SSL handle' to another process and expect it to work.
EDIT
The answers here don't really answer the underlying question, which how I should do this, so although the bounty has been awarded, I can't accept the only answer.
The answer is that you can't do it.
It maybe that this is not possible with the current mechanism.
Correct. As you've foreseen this possiblity in your question, it is difficult to understand why you can't accept it in an answer.
In principle your idea is not impossible! If you believe that is possible try to find the answer!
If all answerer from the SO will say that is impossible it is not all time the true!
For example: 15 years ago I had tried to find the answer how can I write an Java-applet which can write and read images on a server.
Everybody had said to me that is impossible, but I did not believed it. I
tried to find my answer again and again. And I had found the answer: I
disassembled one online apllet from one specialist and in the source
code I find my answer: using PHP server we can do it. I asked the
owner from this applet about details of communication between
Java-applet and PHP server and he has helped me.
You have to find your specialist. That is the first rule to find the correct answer. May be on the IBM forum you will find someone.
The second rule is to read a lot of books from specialist about this. Not only one book. May be three of them or more.
I would recommend you also to read How do I ask a good question?, because in your question you do not have any computer language. And I think we have on SO someone specialist which could give you the correct answer.
The first rule on SO for finding of correct specialist is to set the correct tags. Without correct tags only few people see your question and it is only question of luck that somebody from them is the correct specialist for you.
Be optimistic and try to believe in you! Good luck and success!
Is there a generic way to tell whether snakemake was executed in cluster mode (e.g., --cluster, --cluster-sync, --drmaa)?
The use case is that in the cluster case I'd like to copy data from/to storage in some rules.
No, this information is not intentionally exposed. A more snakemakeish way would be to keep such special handling out of the workflow definition. This way, the scalability of the workflow is not limited and the code is also not crowded with platform-specific stuff. Instead, you can use the --default-remote-provider argument if your storage protocol is supported, see here. Another possibility is to copy the files in the jobscript. Both strategies can be very flexibly implemented via configuration profiles, see here. A good example for a comprehensive profile that performs a similar task is this one.
Would it help to use an input function to copy the files?
This solution helped me in a slightly related case:
Snakemake: Generic input function for different file locations
I'm very new to Apache ActiveMQ and still in learning phase so please bear with my silly or with not very precise questions. I would like to know is there is any possibility of message lost if we keep enablebatch property to true?
As per my knowledge enabling batch processing is always helpful and enhance your performance too in terms of processing transaction but I would like to know negative side of this property which may lead to message lost issue. Also if you can let me know whether there are some additional properties too which may need to set if we enable batch as true ?
Will appreciate your response
Regards,
Deepak Hardasani
Don't try to pre-tune! It only leads to de-tuning. Most Enterprise message flows are sufficiently handled with defaults on modern server systems without jumping through a lot of hoops and adding complexity.
Build up a representative test case.. run it. If the numbers meet your requirements, move on to more useful tasks. If not, then worry about tuning.
I'm hoping to use something like nanomsg (or ZeroMQ). Specifically the pipeline pattern. Instead of a round-robin method for sending out the data, I would like to load balance based on the data.Any suggestions?
I found an answer to the ZeroMQ use case here: ZMQ sockets with custom load-balancing
Ultimately though, I think what I was really looking for was better served with a DDS solution as opposed to ZeroMQ or nanomsg. I found this question and answer very helpful: WHY / WHEN using rather DDS instead of ZeroMQ?
Almost every application out there performs i/o operations, either with disk or over network.
As my applications work fine under the development-time environment, I want to be sure they will still do when the Internet connection is slow or unstable, or when the user attempts to read data from badly-written CD.
What tools would you recommend to simulate:
slow i/o (opening files, closing files, reading and writing, enumeration of directory items)
occasional i/o errors
occasional 'access denied' responses
packet loss in tcp/ip
etc...
EDIT:
Windows:
The closest solution to do the job as described seems to be holodeck, commercial software (>$900).
Linux:
Open solution wasn't found by now, but the same effect
can be achived as specified by smcameron and krosenvold.
Decorator pattern is a good idea.
It would require to wrap my i/o classes, but resulting in a testing framework.
The only remaining untested code would be in 3rd party libraries.
Yet I decided not to go this way, but leave my code as it is and simulate i/o errors from outside.
I now know that what I need is called 'fault injection'.
I thought it was a common production-line part with plenty of solutions I just didn't know.
(By the way, another similar good idea is 'fuzz testing', thanks to Lennart)
On my mind, the problem is still not worth $900.
I'm going to implement my own open-source tool based on hooks (targeting win32).
I'll update this post when I'm done with it. Come back in 3 or 4 weeks or so...
What you need is a fault injecting testing system. James Whittaker's 'How to break software' is a good read on this subject and includes a CD with many of the tools needed.
If you're on linux you can do tons of magic with iptables;
iptables -I OUTPUT -p tcp --dport 7991 -j DROP
Can simulate connections up/down as well. There's lots of tutorials out there.
Check out "Fuzz testing": http://en.wikipedia.org/wiki/Fuzzing
At a programming level many frameworks will let you wrap the IO stream classes and delegate calls to the wrapped instance. I'd do this and add in a couple of wait calls in the key methods (writing bytes, closing the stream, throwing IO exceptions, etc). You could write a few of these with different failure or issue type and use the decorator pattern to combine as needed.
This should give you quite a lot of flexibility with tweaking which operations would be slowed down, inserting "random" errors every so often etc.
The other advantage is that you could develop it in the same code as your software so maintenance wouldn't require any new skills.
You don't say what OS, but if it's linux or unix-ish, you can wrap open(), read(), write(), or any library or system call etc, with an LD_PRELOAD-able library to inject faults.
Along these lines:
http://scaryreasoner.wordpress.com/2007/11/17/using-ld_preload-libraries-and-glibc-backtrace-function-for-debugging/
I didn't go writing my own file system filter, as I initially thought, because there's a simpler solution.
1. Network i/o
I've found at least 2 ways to simulate i/o errors here.
a) Running a virtual machine (such as vmware) allows to configure bandwidth and packet loss rate. Vmware supports on-machine debugging.
b) Running a proxy on the local machine and tunneling all the traffic through it. For the case of upd/tcp communications a proxifier (e.g. widecap) can be used.
2. File i/o
I've managed to deduce this scenario to the previous one by mapping a drive letter to a network share which resides inside the virtual machine. The file i/o will be slow.
A cheaper alternative exists: to set up a local ftp server (e.g. FileZilla), configure speeds and use Novell's NetDrive to access it.
You'll wanna setup a test lab for this. What type of application are you building anyway? Are you really expecting the application be fed corrupt data?
A test technique I know the Microsoft Exchange Server people tried was sending noise to the server. Basically feeding every possible input with seemingly random data. They managed to crash the server quite often this way.
But still, if you can't trust input that hasn't been signed then general rules apply. Track every operation which could potentially be untrusted (result of corrupt data) and you should be able to handle most problems gracefully.
Just test your application behavior on random input, that should catch most problems but you'll never be able to fully protect your self from corrupt data. That's just not possible, as the data could be part of some internal buffer being handed off within the application itself.
Be mindful of when and how you decode data. That is all.
The first thing you'll need to do is define what "correct" means under these circumstances. You can only test against a definition of what behaviour is intended.
The tactics of testing will depend on technology. In the context of automated unit testing, I have found it very useful, in OO languages such as Java, to use various flavors of "mocking" or "stubbing" to pass e.g. misbehaving InputStreams to parts of my code that used file I/O.
Consider holodeck for some of the fault injection, if you have access to spare hardware you can simulate network impairment using Netem or a commercial product based on it the Mini-Maxwell, which is much more expensive than free but possibly easier to use.