Does soneone of you have a good solution for external rendering for Microsoft HoloLens Apps? Specified: Is it possible to let my laptop render an amount of 3D objects that is too much for the HoloLens GPU and then display them with the HoloLens by wifi including the spatial mapping and interaction?
It's possible to render remotely both directly from the unity editor and from a built application.
While neither achieves your goal of a "good solution" they both allow very intensive applications to at least run at all.
This walks you through how to add it to an app you're building.
https://learn.microsoft.com/en-us/windows/mixed-reality/add-holographic-remoting
This is for running directly from the editor:
https://blogs.unity3d.com/2018/05/30/create-enhanced-3d-visuals-with-holographic-emulation-in-uwp/
I don't think this is possible since, you can't really access the OS or the processor at all on the HoloLens. Even if you do manage to send the data to a 3rd party to process, the data will still need to be run back through the HoloLens which is really just the same as before.
You may find a way to perhaps hook up a VR backpack to it but even then, I highly doubt it would be possible.
If you are having trouble rendering 3D objects, then you should reduce the number of triangles, get a lower resolution shader on it, or reduce the size of the object. The biggest factor in processing 3D objects on the HoloLens is how much space is being drawn on the lens. If your object takes up 25% of the view instead of 100% it will be easier to process on the HoloLens.
Also if you can't avoid a lot of objects in the scene maybe check out LOD, which reduces the resolution of objects based off of distance to it and vice versa.
Is it possible to specify a (rectangular) area for skeleton tracking with the Kinect (using any of the available SDKs)? I want to make sure that only users inside that designated area are tracked and that the sensor is not distracted by people outside it. Think of a game zone, in which a player interacts with the Kinect and where bystanders outside of the zone should be ignored lest they confuse the sensor.
The reason I want this is that many times the Kinect "locks" onto someone or even something, whether it should or not, and then it's difficult for the sensor to track other individuals, who come into tracking range. I want to avoid that by defining this zone.
It's not possible to specify a target area for the skeleton tracking with Microsoft's official SDKs, but there are some potential workarounds.
(Note that I'm not familiar with other SDKs for the Kinect, and note that I'm not sure if you are using the Kinect v1 or v2.)
If you are using the Kinect v1, note that it can track 6 players simultaneously (with a skeleton body position), but it can only provide full-body skeletal tracking for 2 players at a time. It's possible to specify which 2 players you want full skeletal tracking for in the official SDK, and you can do this based on which skeletons are in your target game zone.
If this isn't the problem, and the problem is that the Kinect (v1 or v2) has already detected 6 players and it can't detect a 7th individual that's in your game zone, then that is a more difficult problem. With the official SDK, you have no control over which 6 players are selected to be tracked. The sensor will lock onto the first 6 players it finds, so if a 7th player walks in, there is no simple way to lock onto that player.
However, there are some possible workarounds that involve resetting the sensor to clear all skeletons to re-select the 6 tracked skeletons (see the thread Skeleton tracking in crowds - Kinect v2):
Kinect body tracking is always scanning and finding candidate bodies
to track. The body tracking only locks on when it detects head and
shoulders of the person facing the camera. You could do something like
look for stable blob points in the target area and if there isn't a
tracked body, reset the Kinect Monitor service.
The SDK is resilient to this type of failure of the runtime, but it is
a hard approach. Additionally, you could employ a way to cover the
depth camera (your hand) to reset the tracking since this will make
all depth/ir invalid and will need to rebuild.
-- Carmine Sirignano - MSFT
In the same thread, RobAcheson points out that restarting the sensor is another workaround:
I've been using the by-hand method successfully for a while and that
definitely works - when I'm in the crowd :)
I have started calling KinectSensor.Close() and KinectSensor.Open()
when there are >6 skeletons if none are in the target area. That seems
to be working well too. Now I just need a crowd to test with.
-- RobAcheson
I'm working on an arcade cabinet that will be able to play various video game consoles (real hardware, not emulated.) There will be a PC inside to run a selection menu. I'll have to write that myself. I'll also need program a PLC which will do various things like control the relays which switch audio/video/controls between the PC and the various consoles, etc. I'll need help with those two tasks in time, but they are not what I'm working on right now.
What I'm working on as a starting point has to do with the controller encoding. Basically, the controls for each player consist of a few buttons and a joystick. These use momentary, normally-open contact switches, one for each button, and one for each cardinal direction on the joystick. Pressing the button, or joystick direction, closes the switch. The state of the buttons is then communicated to the console by an encoder.
The encoder has a connection for each button and joystick direction which is connected to 5 volts ("high") through a pull-up resistor. When a button or direction is pressed, a connection to ground is made through the momentary switch. When the encoder reads ground ("low") on a button connection, it knows that a button has been pressed and it communicates this to the console.
I already have all this working with the various consoles, but I've thought of some features that would be nice to add. This is where my current task comes in.
The first feature is button remapping. Some of these games were designed with controllers in mind, so when you use them with an arcade control panel, some of the buttons may not be where you want them. Some games allow buttons to be remapped via software, but others do not. My idea is to add a PLC in between the joystick and buttons and the encoder. I'll call this PLC a "pre-encoder."
The pre-encoder would read the states of the buttons on some input pins, then write these states back to some output pins, relaying them to the encoder. The advantage is that its programming could associate any input pin with any output pin, effectively remapping the buttons. Whenever a console is selected via the computer's menu, a button-mapping profile associated with a particular game could be selected as well, and forwarded to the pre-encoder.
Of course, the pre-encoder's routine which reads the buttons and relays their states to the encoder must repeat very quickly for smooth control. These games will be running at about 50 to 60Hz, meaning a new a video frame every 16.67ms or less. Ideally, the pre-encoder will be able to repeat this routine many, many times per frame to ensure the absolute minimum input lag. I want to ensure that the code and hardware selection is optimized to run as fast as possible.
The second feature is turbo buttons. Some games, especially arcade games, require a fire button to be pressed repeatedly every time you want to fire your gun, or your ship's cannons, etc, even if you have unlimited ammo. This seems unnecessary, and it will tire your fingers out pretty quickly. A turbo button is one that can be held down continuously, yet the game is being told that you are rapidly pressing and releasing it. This could be done in software for anything running on the PC, or with an analog solution like a 555 timer, but the best method is to synchronize the turbo button timing with the video refresh rate. By feeding the vertical sync pulse from the PC or video game console's video output to a PLC, it will know exactly how often a frame of video is rendered. Turbo button timing can then be controlled by defining, in numbers of frames, the periods when the button should be pressed and released. Timing information could also be included with the game-specific button profiles.
The third feature is slow buttons. Actually, this would probably only be applied to the joystick, but I'm referring to the switches for its cardinal directions as buttons. In certain games (it will probably only be used in shmups) it is sometimes needed to move your character (ship/plane) through very tight spaces. If movement is too fast in response to even minimal joystick input, you may go too far and crash. The idea is that, while a slow activation button is held, the joystick will be made less responsive by rapidly activating and deactivating it in the same manner as the turbo buttons.
I'm not sure if I want the pre-encoder itself to be watching the vertical sync pulse or if it will slow it down too much. My current thinking is that a seperate PLC will be responsible for general management of the cab itself; watching the "on" button, switching relays, communicating directly with the PC, watching the vertical sync pulse, etc. This will free up the pre-encoder to run more quickly.
Here is some example "code" for the pre-encoder. Obviously, it's just a rough outline of what I have in mind, as I don't even know what language it will be. This example assumes that a dedicated PLC will be used just as the pre-encoder. A separate PLC will be responsible for watching the vertical sync pulse, in addition to other tasks, like getting a game profile from the computer and passing some of that info to the pre-encoder. That PLC will know what the frame timing should be for turbo and slow functions, it will count frames, and during frames when turbo buttons should be disabled, it outputs high to a pin on the pre-encoder PCB, letting it know to disable turbo buttons. During frames when it should be enabled, it outputs low to that pin. Same idea with the slow buttons. There is also a pin which the pre-encoder checks at the end of its routine, so it can be told to stop and await a different game profile.
get info from other PLC (which got it from the computer, from a user-selected game profile):
array containing list of turbo buttons (buttons are identified by what input pin they are connected to)
array containing list of slow buttons (will probably only be the joystick directions, if any)
array containing list of slow activation buttons (should normally be only one button, if any)
array containing list of normal buttons (not turbo or slow)
array containing which output pin to use for each button (this determines remapping)
Begin Loop
if turbo pin is high
for each turbo button
output pin = high
next
else
for each turbo button
output pin = input pin
next
end if
if slow pin is high and slow activation button is pressed
for each slow button
output pin = high
next
else
for each slow button
output pin = input pin
next
end if
for each normal button
output pin = input pin
next
Restart Loop unless stop pin is low
If you've read all this, thank you for your time. So (finally), here are my questions:
What are your overall thoughts; on my idea in general, feasibility, etc.?
What kind of PLC should I use for the pre-encoder? I was originally thinking of trying an Arduino, but my reading indicates that it will be much too slow, due to its use of high-level programming libraries. I don't have a problem building my own board around another PLC.
What language should I use to program the PLC? I don't mind learning a new language. There's no time limit on this project, and I'll put it in whatever it takes to get the pre-encoder running as fast as possible.
What will I need to flash my program onto the PLC?
At run-time, how should these PLC's communicate with each other, and with the PC?
Am I asking in the right place; right forum, right section, etc.? Anywhere else I should ask?
Awaiting your response eagerly,
-Rob
I have some thoughts that might be useful to you:
What are your overall thoughts; on my idea in general, feasibility, etc.?
This project sounds like you want to cheat at Defender, like I used to do with a 555 timer chip in my Atari joystick when I was a kid.
The project is feasible but you will need a pretty fast PLC.
You might spend a lot of time making this work, like a quest.
What kind of PLC should I use for the pre-encoder? I was originally thinking of trying an Arduino, but my reading indicates that it will be much too slow, due to its use of high-level programming libraries. I don't have a problem building my own board around another PLC.
As I thought of what PLC might be fast enough, a few things came to mind.
If you use a PLC that has a task architecture, you can use an event to trigger a task on the v-sync pulse, and another event to trigger on console activity. If you use a PLC without a task architecture, the user might recognize the variable latency that will occur as the program scan moves in and out of phase with the v-sync and the activity in the game. This might not be true if the PLC is fast enough, say 1ms scan time.
Most inexpensive PLCs are never going to make it. The overhead and performance will keep most PLCs around 5-10ms per scan. However, a PC-based PLC might work well. So maybe a Beckhoff controller will work nicely. If you use something like a CX2000, it has Windows 7, USB, DVI for the user interface, and an Ethercat bus on the side to attach physical I/O cards for the controller and console connections. See about the software below. There are many non-PC-based PLCs that would work fine, but these will likely be expensive and harder to integrate.
The Arduino solution should work if you are using a fast enough model. But your development time will be higher because it doesn't come with anything but a blank screen and a bunch of libraries. Troubleshooting is much more of a pain-in-the-neck than PLCs that really shine. You'll need to plan carefully to get the Arduino to work. Also, hardware interfacing a microcontroller is harder and you'll have to manage debouncing the switches in your code. Every PLC has filtering in its inputs, and the variety of I/O makes design easy. But, the Arduino or other microcontroller is really the choice if money is an issue. A fast PLC can be real expensive ($800 to $20k, think around $1500). If you are going to build more than a few systems, the Arduino might be better.
What language should I use to program the PLC? I don't mind learning a new language. There's no time limit on this project, and I'll put it in whatever it takes to get the pre-encoder running as fast as possible.
IEC61131 is a standard for PLC programming languages. In the USA most PLCs are programmed in ladder logic because it is really easy to learn and quicker to troubleshoot and maintain in machinery. Structured text has its advantages too, particularly in performance. It looks like some amalgamation of basic/C/Java, easy to learn and looks almost like your pseudocode example. As for your project, I think it could be programmed in either language. I would never use the other IEC61131 languages for this task.
Beckhoff TwinCAT3 uses MS Visual Studio as the IDE, where you can write both the selection menu (in VB/C++/C#) and the PLC code (in IEC61131) in the same project. The runtime license for TwinCAT (on the CX2000 unit) runs in kernel mode, providing processing performance to Windows 7 whenever it is not doing something else more important. I've used a few CX1020 models and they were great performers. The scan times were around 5ms with a significant amount of code. Faster units will scan <1ms.
What will I need to flash my program onto the PLC?
PLCs don't "flash" like microcontrollers. Whatever software you use to write the software will have a way to connect to the controller. The term "go online" makes the connection. The terms "download" and "upload" refer to transferring the program between the development computer and the PLC. The term "online edit" refers to making code changes while the PLC is executing the code. When modern PLCs are powered down, they use a battery to copy program and user RAM to flash. When they power up, they copy the flash back to RAM. To make a connection to any modern PLC, you will use a USB or Ethernet cable.
At run-time, how should these PLC's communicate with each other, and with the PC?
You plan more than one PLC? A PLC connection to a PC is a complicated subject. The term "OPC Server" refers to some [expensive] software that lets your custom Windows PC application access memory in PLCs. The Beckhoff solution glues all that together nicely without buying more stuff. PLC to PLC communication is easier. The method is usually by ethernet and varies widely as to the details.
Am I asking in the right place; right forum, right section, etc.? Anywhere else I should ask?
Sure, there is some PLC activity on this forum, which appears to tend toward hardcore PC/Web/Mobile development. I come here for awesomely intelligent answers to my deeper software questions.
You could try plctalk.net, a forum that is a little more geared toward nuts-and-bolts engineers and service techs with wild connectivity and compatibility questions related to machinery and automation. You might get some blank stares about vertical sync pulses. Their skill sets revolve around an industrial paradigm, where reliability is probably their highest calling.
You might also ask questions about performance on an Arduino or Microchip/Atmel/ARM forums. If you tell them that a PLC is faster than their hardware, that will rile them up real good! They might tell you that you can get microsecond performance numbers, which you can if you are using hardware interrupts and lots of physical circuitry to make that a reality, and you are able to cope with the sleepless nights of troubleshooting.
-Dennis
I'm am an experienced flash developer who's been learning objective-c for the last 5 months.
I am beginning the development of an app previously prototyped in Flash and I'm trying to guess what could be the best approach to port it to iOS.
My app is kind of a music game. It consists of some dynamic graphics (circles growing and rotating), with typography also changing and rotating. Everything moves in sync with music. And at the same time the user can interact with the app (moving and rotating things) and some sounds will change depending on his actions.
Graphics can't be bitmaps because they get redrawn every frame.
This was easy to develop with Flash due to its management of vector graphics. But I'm not sure what would be the best way to do it in objective-c.
My options, I guess are things like: Core Graphics, OpenGL, maybe Cocos2D (not sure if that would be to kill a flea with a sledgehammer). Or even things like OpenFrameworks or Cinder, but I rather use objective-c other than c++.
Any hint on where to look at will be appreciated.
EDIT:
I can't really put a real screenshot due to confidentiality issues. But it is something similar to this
But it will be interactive and sections would change size and disappear depending on the music and user interaction.
Which graphics library should you use? The answer is going to depend a lot on what you know or could learn. OpenGL will use hardware acceleration, so it's probably fastest. But OpenGL doesn't have built-in functions for drawing arc segments or any curves or text at all, so you'd probably have to do it yourself. Also, OpenGL is notoriously difficult to learn.
Core Graphics has many cool methods for drawing vector graphics (rectangles, arcs, general paths, etc.), but might be slower than you want, depending on what you're trying to do. Without having code to actually run it's hard to say.
It looks like Cocos2D is built on OpenGL and is made to be simple. I see lots of mention of sprites on their website, but nothing about vector graphics. (I've never used it, so it could be there and I'm just not seeing it.)
If I were in your position, I'd look into cocos2d and see if it does vector graphics at all. If not, I might give Core Graphics a try and see what performance was like. I know OpenGL can do what you want, but it can be difficult to learn, so I'd probably do that last.
I am writing a simple application for streaming video over the network, using a slightly different from the ordinary "H.264 over RTP" approach (i am using my own codecs).
To achieve this, i need raw frames and raw audio samples that QTMovie, when playing back a movie, implicitly sends to QTMovieView.
The most common way to retrieve raw video frames is to use VisualContext - and then, using a display link callback, i "generate" a CVPixelBufferRef, using this VisualContext. So i am getting frames with some frequency that is synchronized with my current refresh rate (not that i need this synchronization - i only need to have a "stream" of frames that i can transmit over the network - but CoreVideo Programming Guide and most Apple samples related to video promote this approach).
The first problem i have faced with - is when i attach a VisualContext to a QTMovie, the picture can't be rendered onto the QTMovieView anymore. I don't know why does this happen (i guess it's related to the idea of GWorld and the rendering being "detached" from it when i attach VisualContext). Ok, at least i have frames, which i could render onto a simple NSView (though this sounds wrong, and performance-unfriendly. Am i doing it right?)
What about the sound, i have no idea what to do. I need to get raw samples of sound as the movie being played (ideally - something similar to what QTCaptureDecompressedAudioOutput returns in its callback).
I have prepared myself to delving into deprecated Carbon QuickTime APIs, if there is no other way. But I don't know even where to start. Should i use the same CoreVideo Display link and periodically retrieve sound somehow? Should i get QTDataReference and locate the sound frames manually?
I am actually a beginner with programming video and audio services. If you could share some experience i would REALLY appreciate any idea you could share with me :)
Thank you,
James