Let's just say I have different versions of render passes depending on how many views/layers I want to render to, let's say 1 (for normal single-plane rendering), 2 (for VR), and 6 for cubemaps. So for each render pass type I might have these three sub-types. Each sub-type has this information given to create the render pass:
typedef struct VkRenderPassMultiviewCreateInfo {
VkStructureType sType;
const void* pNext;
uint32_t subpassCount;
const uint32_t* pViewMasks;
uint32_t dependencyCount;
const int32_t* pViewOffsets;
uint32_t correlationMaskCount;
const uint32_t* pCorrelationMasks;
} VkRenderPassMultiviewCreateInfo;
My question is, when we create a graphics pipeline you have to give a sub-pass. Does filling the above struct with different information result in the sub-passes in the render pass created not compatible with each other? And so necessitating the creation of separate graphics pipeline objects for each of the multiview types?
With certain very minor exceptions (view count not being one of them), if the render passes are different, then you must use a different pipeline.
It should also be noted that multiview and layered rendering use the same underlying resources. That is, multiview is a form of layered rendering. As such, if a render pass/pipeline expects multiview, it cannot use layered rendering for non-multiview purposes. That is, in multiview each layer you render to is considered a "view" for multiview purposes. Which means that it automatically routes primitives to specific layers based on the view mask (all rendering commands in a multi-view subpass are automatically repeated for multiple views, and you're not allowed to write to Layer). In layered rendering, it's up to the user code to which layer(s) a rendering command renders to)
Lastly, let us look at dynamic rendering in Vulkan 1.3 (aka: rendering without a render pass). With this feature, you don't have to build a render pass, and pipelines meant to be used for such rendering are not built against a render pass. However, this won't exactly fix your problem.
Even though you aren't building your pipelines against an entire VkRenderPass structure, you still need to give the pipeline some idea of what you're rendering to. This is provided by the VkPipelineRenderingCreateInfo at pipeline creation time. Among other things, this struct specifies the viewMask for that subpass.
And this value must exactly match the value you used with vkCmdBeginRendering. So you will need pipelines to be built against a specific multiview setup, and therefore, you will need separate pipelines for each multiview setup.
Single-view rendering should not be considered a special case of multi-view rendering. They are two different ways of rendering.
Related
So a VkSampler is created with a VkSamplerCreateInfo that just has a bunch of configuration settings, that as far as I can see would just define a pure function of some input image.
They are described as:
VkSampler objects represent the state of an image sampler which is used by the implementation to
read image data and apply filtering and other transformations for the shader.
One use (possibly only use) of VkSampler is to write them to descriptors (such as VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER) for use in descriptor sets that are bound to pipelines/shaders.
My question is: can you write the same VkSampler to multiple different descriptors? from the same or multiple different descriptor pools? even if one of the current descriptors is in use in some currently executing render pass?
Can you use the same VkSampler concurrently from multiple different render passes / subpasses / pipelines?
Put another way, are VkSamplers stateless? or do they represent some stateful memory on the device and so you shouldn't use the same one concurrently?
VkSampler objects definitely have data associated with them, so it would be wrong to call them "stateless". What they are is immutable. Like VkRenderPass, VkPipeline, and similar objects, once they are created, their contents cannot be changed.
Synchronization between accesses is (generally) needed only for cases when one of the accesses is a modification operation. Since VkSamplers are immutable, there are no modification operations. So synchronization is not needed for cases where you're accessing a VkSampler from different threads, commands, or whathaveyou.
The only exception is the obvious one: vkDestroySampler, which requires that submitted commands that use the sampler have completed before calling the function.
I'm learning Metal, and there's a conceptual question that I'm trying to wrap my head around: at what level, exactly, should my code handle successive drawing operations that require different pipeline states? As I understand it (from answers like this: https://stackoverflow.com/a/43827775/2752221), I can use a single MTLRenderCommandEncoder and change its pipeline state, the vertex buffer it's using, etc., between calls to drawPrimitives:, and the encoder state that was current at the time of each call to drawPrimitives: will be preserved. So that's great. But it also seems like the design of Metal is such that one can make multiple MTLRenderCommandEncoder instances, and use them to sequentially throw batches of commands into a MTLCommandBuffer. Given that the former works – using one MTLRenderCommandEncoder and changing its state – why would one do the latter? Under what circumstances is it correct to do the former, and under what circumstances is it necessary to do the latter? What is an example of a situation where the latter would be necessary/appropriate?
If it matters, I'm working on a macOS app, using Objective-C. Thanks.
Ignoring multithreaded encoding cases, which are somewhat advanced, the main reason you'd want to create multiple render command encoders during a frame is because you need to change which textures you're rendering to.
You'll notice that you need to provide a render pass descriptor when creating a render command encoder. For this reason, we often say that the sequence of commands belonging to a particular encoder constitute a render pass. The attachments of that descriptor refer to the textures that will be written to by the commands encoded by the encoder.
Many different techniques, including shadow mapping and postprocessing effects like bloom require multiple passes to produce. Since you can't change attachments in the midst of a pass, creating a new encoder is the only way to encode multiple passes in a frame.
Relatedly, you should ordinarily use one command buffer per frame. You can, however, sometimes reduce frame time by splitting your passes across multiple command buffers, but this is highly dependent on the shape of your workload and should only be done in tandem with profiling, as it's not always an optimization.
In addition to Warren's answer, another way to look at the question is by examining the API. A number of Metal objects are created from descriptors. The properties of the descriptor at the time an object is created from it govern that object for its lifetime. Those are aspects of the object that can't be changed after creation.
By contrast, the object will have various setter methods to modify other properties over its lifetime.
For a render command encoder, the properties that are fixed for its lifetime are those specified by the MTLRenderPassDescriptor used to create it. If you want to render with different values for any of those properties, the only way to do so is to create a new encoder from a different descriptor. On the other hand, if you can do everything you need/want to do by using the encoder's setter methods, then you don't need a new encoder.
I am writing a C++ OpenCV-based computer vision program. The basic idea of the program could be described as follows:
Read an image from a camera.
Do some magic to the image.
Display the transformed image.
The implementation of the core logic of the program (step 2) falls into a sequential calling of OpenCV functions for image processing. It is something about 50 function calls. Some temporary image objects are created to store intermediate results, but, apart from that, no additional entities are created. The functions from step 2 are used only once.
I am confused about organising this type of code (which feels more like a script). I used to create several classes for each logical step of the image processing. Say, here I could create 3 classes like ImagePreprocessor, ImageProcessor, and ImagePostprocessor, and split the abovementioned 50 OpenCV calls and temorary images correspondingly between them. But it doesn't feel like a resonable OOP design. The classes would be nothing more than a way to store the function calls.
The main() function would still just create a single object of each class and call thier methods consequently:
image_preprocessor.do_magic(img);
image_processor.do_magic(img);
image_postprocessor.do_magic(img);
Which is, to my impression, essentially the same thing as callling 50 OpenCV functions one by one.
I start to question whether this type of code requiers an OOP design at all. After all, I can simply provide a function do_magic(), or three functions preprocess(), process(), and postprocess(). But, this approach doesn't feel like a good practice as well: it is still just a pile of function calls, separated into a different function.
I wonder, are there some common practices to organise this script-like kind of code? And what would be the way if this code is a part of a large OOP system?
Usually, in Image Processing, you have a pipeline of various Image Processing Modules. Same is applicable on Video Processing, where each Image is processed according to its timestamp order in the video.
Constraints to consider before designing such pipeline:
Order of Execution of these modules is not always same. Thus, the pipeline should be easily configurable.
All modules of the pipeline should be executable in parallel with each other.
Each module of the pipeline may also have a multithreaded operation. (Out of scope of this answer, but is a good idea when a single module becomes the bottleneck for the pipeline).
Each module should easily adhere to the design and have the flexibility of internal implementation changes without affecting other modules.
The benefit of preprocessing of a frame by one module should be available to later modules.
Proposed Design.
Video Pipeline
A video pipeline is a collection of modules. For now, assume module is a class whose process method is called with some data. How each module can be executed will depend on how such modules are stored in VideoPipeline! To further explain, see below two categories:-
Here, let’s say we have modules A, B, and C which always execute in same order. We will discuss the solution with a video of Frame 1, 2 and 3.
a. Linked List: In a single-threaded application, frame 1 is first executed by A, then B and then C. The process is repeated for next frame and so on. So linked list seems like an excellent choice for the single threaded application.
For a multi-threaded application, speed is what matters. So, of course, you would want all your modules running 128-core machine. This is where Pipeline class comes into play. If each Pipeline object runs in a separate thread, the whole application which may have 10 or 20 modules starts running multithreaded. Note that the single-thread/multithread approach can be made configurable
b. Directed Acyclic Graph: Above-linked list implementation can be further improved when you have high processing power and want to reduce the lag between input and response time of pipeline. Such a case is when module C does not depend on B, but on A. In such case, any frame can be parallelly processed by module B and module C using a DAG based implementation. However, I wouldn’t recommend this as the benefits are not so great compared to the increased complexity, as further management of output from module B and C needs to be done by say module D where D depends on B or C or both. The number of scenarios increases.
Thus, for simplicity sake, let’s use LinkedList based design.
Pipeline
Create a linked list of PipelineElement.
Make process method of pipeline call process method of the first element.
PipelineElement
First, the PipelineElement processes the information by calling its ImageProcessor(read below). The PipelineElement will pass a Packet(of all data, read below) to ImageProcessor and receives the updated packet.
If next element is not null, call next PipelineElement process and pass updated packet.
If next element of a PipelineElement is null, stop. This element is special as it has an Observer object. Other PipelineElement will be set to null for Observer field.
FrameReader(VIdeoReader/ImageReader)
For video/image reader, create an abstract class. Whether you process video or image or multiple, processing is done one frame at a time, so create an abstract class(interface) ImageProcessor.
A FrameReader object stores reference to the pipeline.
For each frame, it pushes the information in by calling process method of Pipeline.
ImageProcessor
There is no Pre and Post ImageProcessor. For example, retinex processing is used as Post Processing but some application can use it as PreProcessing. Retinex processing class will implement ImageProcessor. Each element will hold Its ImageProcessor and Next PipeLineElement object.
Observer
A special class which extends PipelineElement and provides a meaningful output using GUI or disk.
Multithreading
1. Make each method run in its thread.
2. Each thread will poll messages from a BlockingQueue( of small size like 2-3 Frames) to act as a buffer between two PipelineElements. Note: The queue helps in averaging the speed of each module. Thus, small jitters(a module taking too long time for a frame) does not affect video output rate and provides smooth playback.
Packet
A packet will store all the information such as input or Configuration class object. This way you can store intermediate calculations as well as observe a real-time effect of changing configuration of an algorithm using a Configuration Manager.
To conclude, each element can now process in parallel. The first element will process nth frame, the second element will process n-1th frame, and soon, but with this, a lot more issues such as pipeline bottlenecks and additional delays due to less core power available to each element will pop up.
This structure lends itself to the pipes and filters architecture (see Pattern-Oriented Software Architecture Volume 1: A System of Patterns by Frank Buschmann):
The Pipes and Filters architectural pattern provides a structure for
systems that process a stream of data. Each processing step is
encapsulated in a filter component. Data is passed through pipes
between adjacent filters. Recombining filters allows you to build
families of related systems.
See also this short description (with images) from the Enterprise Integration Patterns book.
For a 2D game I'm making (for Android) I'm using a component-based system where a GameObject holds several GameComponent objects. GameComponents can be things such as input components, rendering components, bullet emitting components, and so on. Currently, GameComponents have a reference to the object that owns them and can modify it, but the GameObject itself just has a list of components and it doesn't care what the components are as long as they can be updated when the object is updated.
Sometimes a component has some information which the GameObject needs to know. For example, for collision detection a GameObject registers itself with the collision detection subsystem to be notified when it collides with another object. The collision detection subsystem needs to know the object's bounding box. I store x and y in the object directly (because it is used by several components), but width and height are only known to the rendering component which holds the object's bitmap. I would like to have a method getBoundingBox or getWidth in the GameObject that gets that information. Or in general, I want to send some information from a component to the object. However, in my current design the GameObject doesn't know what specific components it has in the list.
I can think of several ways to solve this problem:
Instead of having a completely generic list of components, I can let the GameObject have specific field for some of the important components. For example, it can have a member variable called renderingComponent; whenever I need to get the width of the object I just use renderingComponent.getWidth(). This solution still allows for generic list of components but it treats some of them differently, and I'm afraid I'll end up having several exceptional fields as more components need to be queried. Some objects don't even have rendering components.
Have the required information as members of the GameObject but allow the components to update it. So an object has a width and a height which are 0 or -1 by default, but a rendering component can set them to the correct values in its update loop. This feels like a hack and I might end up pushing many things to the GameObject class for convenience even if not all objects need them.
Have components implement an interface that indicates what type of information they can be queried for. For example, a rendering component would implement the HasSize interface which includes methods such as getWidth and getHeight. When the GameObject needs the width, it loops over its components checking if they implement the HasSize interface (using the instanceof keyword in Java, or is in C#). This seems like a more generic solution, one disadvantage is that searching for the component might take some time (but then, most objects have 3 or 4 components only).
This question isn't about a specific problem. It comes up often in my design and I was wondering what's the best way to handle it. Performance is somewhat important since this is a game, but the number of components per object is generally small (the maximum is 8).
The short version
In a component based system for a game, what is the best way to pass information from the components to the object while keeping the design generic?
We get variations on this question three or four times a week on GameDev.net (where the gameobject is typically called an 'entity') and so far there's no consensus on the best approach. Several different approaches have been shown to be workable however so I wouldn't worry about it too much.
However, usually the problems regard communicating between components. Rarely do people worry about getting information from a component to the entity - if an entity knows what information it needs, then presumably it knows exactly what type of component it needs to access and which property or method it needs to call on that component to get the data. if you need to be reactive rather than active, then register callbacks or have an observer pattern set up with the components to let the entity know when something in the component has changed, and read the value at that point.
Completely generic components are largely useless: they need to provide some sort of known interface otherwise there's little point them existing. Otherwise you may as well just have a large associative array of untyped values and be done with it. In Java, Python, C#, and other slightly-higher-level languages than C++ you can use reflection to give you a more generic way of using specific subclasses without having to encode type and interface information into the components themselves.
As for communication:
Some people are making assumptions that an entity will always contain a known set of component types (where each instance is one of several possible subclasses) and therefore can just grab a direct reference to the other component and read/write via its public interface.
Some people are using publish/subscribe, signals/slots, etc., to create arbitrary connections between components. This seems a bit more flexible but ultimately you still need something with knowledge of these implicit dependencies. (And if this is known at compile time, why not just use the previous approach?)
Or, you can put all shared data in the entity itself and use that as a shared communication area (tenuously related to the blackboard system in AI) that each of the components can read and write to. This usually requires some robustness in the face of certain properties not existing when you expected them to. It also doesn't lend itself to parallelism, although I doubt that's a massive concern on a small embedded system...?
Finally, some people have systems where the entity doesn't exist at all. The components live within their subsystems and the only notion of an entity is an ID value in certain components - if a Rendering component (within the Rendering system) and a Player component (within the Players system) have the same ID, then you can assume the former handles the drawing of the latter. But there isn't any single object that aggregates either of those components.
Like others have said, there's no always right answer here. Different games will lend themselves towards different solutions. If you're building a big complex game with lots of different kinds of entities, a more decoupled generic architecture with some kind of abstract messaging between components may be worth the effort for the maintainability you get. For a simpler game with similar entities, it may make the most sense to just push all of that state up into GameObject.
For your specific scenario where you need to store the bounding box somewhere and only the collision component cares about it, I would:
Store it in the collision component itself.
Make the collision detection code work with the components directly.
So, instead of having the collision engine iterate through a collection of GameObjects to resolve the interaction, have it iterate directly through a collection of CollisionComponents. Once a collision has occurred, it will be up to the component to push that up to its parent GameObject.
This gives you a couple of benefits:
Leaves collision-specific state out of GameObject.
Spares you from iterating over GameObjects that don't have collision components. (If you have a lot of non-interactive objects like visual effects and decoration, this can save a decent number of cycles.)
Spares you from burning cycles walking between the object and its component. If you iterate through the objects then do getCollisionComponent() on each one, that pointer-following can cause a cache miss. Doing that for every frame for every object can burn a lot of CPU.
If you're interested I have more on this pattern here, although it looks like you already understand most of what's in that chapter.
Use an "event bus". (note that you probably can't use the code as is but it should give you the basic idea).
Basically, create a central resource where every object can register itself as a listener and say "If X happens, I want to know". When something happens in the game, the responsible object can simply send an event X to the event bus and all interesting parties will notice.
[EDIT] For a more detailed discussion, see message passing (thanks to snk_kid for pointing this out).
One approach is to initialize a container of components. Each component can provide a service and may also require services from other components. Depending on your programming language and environment you have to come up with a method for providing this information.
In its simplest form you have one-to-one connections between components, but you will also need one-to-many connections. E.g. the CollectionDetector will have a list of components implementing IBoundingBox.
During initialization the container will wire up connections between components, and during run-time there will be no additional cost.
This is close to you solution 3), expect the connections between components are wired only once and are not checked at every iteration of the game loop.
The Managed Extensibility Framework for .NET is a nice solution to this problem. I realize that you intend to develop on Android, but you may still get some inspiration from this framework.
I have been developing a web application in Seaside+Squeak recently, and have found it to be a wonderful experience. Seaside really is head and shoulders above every other framework out there, and I feel as though I am working at a higher level of abstraction (above the HTTP request/response cycle and HTML templating that other frameworks make you deal with).
That said, I'm a little confused about Seaside components. I recently had to display a list of objects on a component (similar to the stackoverflow front page). At first I made each object a component (a subclass of WAComponent), but this proved to be really wasteful, and I had to set #children dynamically in the parent component for it to work at all. I then tried making them render objects (objects that aren't subclasses of WAComponent, and render using renderOn: instead of renderContentOn:, like components do). This worked, but now they could no longer access global state in the session object as components can (using #session). Then I discovered "WACurrentSession value", which gives any object access to the current Seaside session object. I was now able to make them render objects. In addition, I discovered that I could rewrite a lot of my other, more minor components as render objects, too.
Besides needing call/answer or backtracking state, what other reasons are there for using components over render objects?
This is a frequent point of confusion for new Seaside users. We have tried hard to make this clearer in Seaside 2.9, which is currently in Alpha, but I will try to focus on 2.8 here since it sounds like that is what you are using.
First of all, you are correct that you do not need to use a Component in order to access the Session. Seaside 2.9 moves #session up to a new class WAObject which makes it accessible to almost all Seaside objects (including Components) but you can definitely refer to WACurrentSession yourself for now in 2.8.
Components provide roughly the following functionality in 2.8:
#renderContentOn: is called with an instance of whatever renderer class you specify in #rendererClass (instead of whatever renderer is in use when your object is asked to render itself)
A hook (#updateUrl:) to allow updating the URL used by the renderer to generate links
Hooks (#updateRoot:, #style, #script) to allow updating the HEAD section of the HTML document
The ability to be the root of an Application
Hooks (#updateStates:, #states) to make state backtracking easier
A hook (#initialRequest:) to allow initialization based on the request that caused the Session to be created
A facility (#children) to make sure all components below you will also have the above hooks called on them
The ability to add Decorations
The ability to show/answer/call (uses Decorations)
Some convenience methods (#inform:, #isolate:, etc)
If you don't need any of the above, you don't need a Component. If you need any of the above, you pretty much do need a Component unless you want to implement the equivalent functionality on your own class.
The simplest metric is probably: if you intend to keep the object around between HTTP requests it should be a Component; if you intend to throw the object away and create it on each rendering pass it probably doesn't need to be. If you imagine an application that was displaying blog pages, you'd probably have Components for a menu, a blog roll, the blog body, each comment, and so on. You might want to factor out the reading of the blog's markup and generation of its HTML so that you could support different markups or different renderers and so that the comment Components could support the same markup. This could be done with a non-Component class that implements #renderOn: and could be created and used by other Components as needed.
Seaside 2.9 currently splits up the above functionality by making WAPresenter concrete and introducing WAPainter as its superclass. 1-3 above are implemented on WAPainter and 4-7 on WAPresenter so you have your choice of what to subclass depending on your needs. It also removes a lot of methods from WAPresenter and WAComponent in an effort to make them easier for end users to understand.
Hope that helps.