When I run snakemake --dag or snakemake --rulegraph multiple times on the same workflow, I get an identical layout but different, apparently random, colors. Is there a way I can provide a seed to get deterministic behavior?
Background: For one of my workflows, I use a git pre-commit hook to auto-generate a plot of the workflow diagram. This is convenient, as updates to the workflow are automatically handled. But because Snakemake generates random colors each time, the diagram graphic file is updated essentially every commit even when no changes have been made to the workflow. I'm contemplating possible ways to update the graphic only when the workflow structure has changed.
Related
I'm in the processing of learning Vulkan, and I have just integrated ImGui into my code using the Vulkan-GLFW example in the original ImGui repo, and it works fine.
Now I want to render both the GUI and my 3D model on the screen at the same time, and since the GUI and the model definitely needs different shaders, I need to use multiple pipelines and submit multiples commands. The GUI is partly transparent, so I would like it to be rendered after the model. The Vulkan specs states that the execution order of commands are not likely to be the order that I record the commands, thus I need synchronization of some kind. In this Reddit post several methods of exactly achieving my goals was proposed, and I once believed that I must use multiple subpasses (together with subpass dependency) or barriers or other synchronization methods like that to solve this problem.
Then I had a look at SaschaWillems' Vulkan examples, in the ImGui example though, I see no synchronization between the two draw calls, it just record the command to draw the model first, and then the command to draw the GUI.
I am confused. Is synchronization really needed in this case, or did I misunderstand something about command re-ordering or blending? Thanks.
Think about what you're doing for a second. Why do you think there needs to be synchronization between the two sets of commands? Because the second set of commands needs to blend with the data in the first set, right? And therefore, it needs to do a read/modify/write (RMW), which must be able to read data written by the previous set of commands. The data being read has to have been written, and that typically requires synchronization.
But think a bit more about what that means. Blending has to read from the framebuffer to do its job. But... so does the depth test, right? It has to read the existing sample's depth value, compare it with the incoming fragment, and then discard the fragment or not based on the depth test. So basically every draw call that uses a depth test contains a framebuffer read/modify/wright.
And yet... your depth tests work. Not only do they work between draw calls without explicit synchronization, they also work within a draw call. If two triangles in a draw call overlap, you don't have any problem with seeing the bottom one through the top one, right? You don't have to do inter-triangle synchronization to make sure that the previous triangles' writes are finished before the reads.
So somehow, the depth test's RMW works without any explicit synchronization. So... why do you think that this is untrue of the blend stage's RMW?
The Vulkan specification states that commands, and stages within commands, will execute in a largely unordered way, with several exceptions. The most obvious being the presence of explicit execution barriers/dependencies. But it also says that the fixed-function per-sample testing and blending stages will always execute (as if) in submission order (within a subpass). Not only that, it requires that the triangles generated within a command also execute these stages (as if) in a specific, well-defined order.
That's why your depth test doesn't need synchronization; Vulkan requires that this is handled. This is also why your blending will not need synchronization (within a subpass).
So you have plenty of options (in order from fastest to slowest):
Render your UI in the same subpass as the non-UI. Just change pipelines as appropriate.
Render your UI in a subpass with an explicit dependency on the framebuffer images of the non-UI subpass. While this is technically slower, it probably won't be slower by much if at all. Also, this is useful for deferred rendering, since your UI needs to happen after your lighting pass, which will undoubtedly be its own subpass.
Render your UI in a different render pass. This would only really be needed for cases where you need to do some full-screen work (SSAO) that would force your non-UI render pass to terminate anyway.
In VkSubmitInfo, when pWaitDstStageMask[0] is VK_PIPLINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT, vulkan implementation executes pipeline stages without waitng for pWaitSemaphores[0] until it reaches Color Attachment Stage.
However, if the command buffer has multiple subpasses and multiple draw commands, then does WaitDstStageMask mean the stages of all draw commands?
If I want vulkan implementation to wait the semaphore when it reaches color attachment output stage of the last subpass, what should I do?
You probably don't actually want to do this. On hardware that benefits from multi-subpass renderpasses, the fragment work for the entire renderpass will be scheduled and execute as essentially a single monolithic chunk of work. E.g. all subpasses will execute for some pixel (x,y) regions before any subpasses are executed for some other pixel (x,y) regions. So it doesn't really make sense to say insert a synchronization barrier on an external event between two subpasses. So you need to think about what your renderpass is doing and whether it is really open to the kinds of optimizations they were designed for.
If not, then treating the subpasses (or at least the final one) as independent renderpasses isn't going to be a loss anyway, so you might as well just put it in a separate renderpass in a separate submit batch, and put the semaphore wait before it.
If so, then you just want to do the semaphore wait before the COLOR_ATTACHMENT stage for the whole renderpass anyway.
In such situation You have (I think) two options:
You can split render pass - exclude the last subpass and submit it's commands in a separate render pass recorded in a separate command buffer so You can specify semaphore for which it should wait on (but this doesn't sound too reasonably) or...
You can use events - You should signal events after the commands which generate results later commands require, and then, in the last sub-pass You should wait on that event just before the commands that indeed need to wait.
The second approach is probably preferable (despite You are not using submission's pWaitSemaphores and pWaitDstStageMask fields), but it also has it's restrictions:
vkCmdWaitEvents must not be used to wait on event signal operations occuring on other queues.
And I'm not sure, but maybe subpass dependencies may also help You here. Clever definitions of submission's pWaitSemaphores and render pass's subpass dependencies may be enough to do the job. But I'm not too confident in explaining subpass dependencies (I'm not sure I fully understand them) so don't rely on this. Maybe someone can confirm this. Bot the above two option definitely will do the trick.
Is there a good way to modularize and re-use code in IPython Notebook (Jupyter) when doing the same analysis on two different sets of data?
For example, I have a notebook with a lot of cells doing analysis on a data file. I have another data file of the same format, and I'd like to run the same analysis and compare the output. None of these options looks particularly appealing for this:
Copy and paste the cells to a second notebook. The analysis code is now duplicated and harder to update.
Move the analysis code into a module and run it for both files. This would lose the cell-by-cell format of the figures that are currently generated and simply jumble them all together in one massive cell.
Load both files in one notebook and run the analyses side by side. This also involves a lot of copy-and-pasting, and doesn't generalize well to 3 or 4 different data files.
Is there a better way to do this?
You could lace demo directives into the standalone module, as per the IPython Demo Mode example.
Then when actually executing it in the notebook, you make a call to the demo object wrapper each time you want to step to the next important part. So your cells would mostly consist of calls to that demo wrapper object.
Option 2 is clearly the best for code re-use, it is the de facto standard arguably in all of software engineering.
I argue that the notebook concept itself doesn't scale well to 3, 4, 5, ... different data files. Notebook presentations are not meant to be batch processing receptacles. If you find yourself needing to do parameter sweeps across different data sets, and wanting to re-run analyses on top of the different data loaded for each parameter group (even when the 'parameters' might be as simple as different file names) it raises a bad code smell. It likely means the level of analysis being performed in an 'interactive' way is wrong. Witnessing analysis 'interactively' and at the same time performing batch processing are two pretty much incompatible goals. A much better idea would be to batch process all of the parameter sets separately, 'offline' from the point of view of any presentation, and then build a set of stand-alone functions that can produce visual results from the computed and stored batch results. Then the notebook will just be a series of function calls, each of which produces summary data (some of which could be examples from a selection of parameter sets during batch processing) across all of the parameter sets at once to invite the necessary comparisons and meaningfully present the result data side-by-side.
'Witnessing' an entire interactive presentation that performs analysis on one parameter set, then changing some global variable / switching to a new notebook / running more cells in the same notebook in order to 'witness' the same presentation on a different parameter set sounds borderline useless to me, in the sense that I cannot imagine a situation where that mode of consuming the presentation is not strictly worse than consuming a targeted summary presentation that first computed results for all parameter sets of interest and assembled important results into a comparison.
Perhaps the only case I can think of would be toy pedagogical demos, like some toy frequency data and a series of notebooks that do some simple Fourier analysis or something. But that's exactly the kind of case that begs for the analysis functions to be made into a helper module, and the notebook itself just lets you selectively declare which toy input file you want to run the notebook on top of.
I have four child pipelines in my project and the output for these is ingested in the main pipeline. I want the output files for the child pipelines to be automatically backed up after baseline in some directory. This will help if i disable individual forges and want to restore previous index. Please help
It should be straight-forward to add copy or archive jobs to your baseline update script to execute this sort of thing before your main forge is run.
radimbe, as forge itself is single-threaded (except for the rarely implemented multi-threaded java manipulator) and monolithic, this sort of architecture is commonly used to take better advantage of multi-processor machines and multi-threaded CPU cores. In addition, if data is made available at different times or with different frequencies, you can decouple child forges from your main baseline process, improving its overall turnaround time. And from a strategy POV, this approach could decompose what might be a large, unwieldy job into perhaps simpler, more focused and more easily maintained components.
Is there a programmatic way to convert two images into an animation sequence (e.g., an animated GIF) like the following example?
This image sequence, taken from a http://memrise.com course, doesn't seem to have manually-edited frames, but seems automatically transformed using some kind shape morphing algorithm. Is there a common term used to describe such an animation or algorithm? Is there a feature in ImageMagick or Photoshop/Gimp that generates such animations, given a pair of images?
Ideally the technique could be scriptable so I could create animations for several pairs of start-end images.
Edit: I have just been told about Gimp's tool under Filters->Animation->Blend, which appears to do the same thing as jQuery morph: each frame i is start + (finish - start)/N*i. In other words, you're transitioning each pixel independently from the start value to the finish value, without any shape morphing. The example gives is more complicated, as it modifies the contours of both images to achieve its compelling effect.
Other examples:
http://static.memrise.com/uploads/mems/32000121024054535.gif
http://static.memrise.com/uploads/mems/225428000121109232837.gif
I have written a tool that doesn't require setting manual keypoints and is not restricted to a domain (like faces). Anyway, the images have to be similar (e.g. two faces or two cars from the same perspective).
https://github.com/kallaballa/Poppy
There is also a web-version created with emscripten.
I generated the above animation using following command line:
poppy flame.png glyph.png flame.png
Although this is an old question, since ImageMagick is mentioned, for anyone who comes here from google it may be worth looking at this imagemagick plugin called shapemorph.
GIMP can't do that directly, but over the years a series of (now poorly maintaind) plug-ins to do that where released by third parties. The keyword for searching for this is "morph" - you should find a bunch of stand alone programs to do that as well, from "gratis" to full fledged Free Software, such as xmorph
Given pairs of vector files (.wmf extension) it is possible to use linear interpolation of shapenodes in Visual Basic for Applications to create frames for GIF animations , though this would take along time to explain. For some examples see
http://www.giless.co.uk/animatorMorphGIFs.htm (it is like a slideshow)
I have made some improvements since then, as well!