Why do successive calls to ImageCapture.takePhoto() fail in my PWA? - webrtc

I am writing a PWA at the moment that uses the camera to do some light computer vision on successive frames from a mobile device camera. When calling takePhoto() a second (or sometimes 3rd time) I get the following error: Uncaught (in promise) DOMException: The associated Track is in an invalid state.
It is noted in the spec that when calling takePhoto()
devices MAY temporarily stop streaming data, reconfigure themselves with the appropriate photo settings, take the photo, and then resume streaming." It is also noted that "If the operation cannot be completed for any reason (for example, upon invocation of multiple takePhoto() method calls in rapid succession), then reject p with a new DOMException whose name is UnknownError, and abort these steps.
Why? Is there a way around this with grabFrame()? Will it change in the future? Is there a discussion within the Chrome team?
I note that I am able to successfully take the photos in quick succession using a Webcam attached to my PC, but NOT using the camera in my phone.

Related

What affects geofence triggering? How to troubleshoot?

I am getting very inconsistent plot results with different devices. Here are some of the issues I have noticed:
The same fences and triggers are deployed to both devices, but:
1) One device is only triggering the exit event on a particular fence, the other doesn't fire the exit event at all.
2) Both devices are not firing the geofence enter or exit events for a series of geofences I have created. I have tried going into the fence and waiting to see if the enter ever fires and does not.
I am retrieving the list of active triggers using the plugin to ensure the triggers are registered with the device.
What can affect geofence triggering behaviour and how do you troubleshoot?
Which devices are you testing with? And are you testing with the Plot Projects demo app or your own app with Plot integrated?
Whether a geofences triggers or not is dependent on a number of things, such as accuracy of the location update. You can troubleshoot the behaviour of our plugin by checking the debug log. How you can use this is explained in this blog post: https://www.plotprojects.com/how-to-test-plot-integration-with-debug-log/
In case you are using Titanium, we expose a mailDebugLog() method for that framework, this is explained in the documentation here: https://github.com/Plotprojects/plot-titanium-module#function-reference

How to change dynamic video resolution during a call (in WebRTC)

I have been using SimpleWebRTC lib for my project.
How to change dynamic remote video resolution during a call (like google hangout when resizing browser)
hangout browser resizing will change remote video resolution size (.videoWidth .videoHeight)
Is this associated with webrtc plan b?
I would like to know how it is implemented for many peer connection.
Tell the sending end (say via DataChannels) to change resolution to NxM. At the sending end, until the new APIs are available to change a getUserMedia/MediaStream capture size on the fly, you can request a second camera/mic stream and replace the existing streams with them. (Note: this will cause onnegotiationneeded i.e. renegotiation, and the far side would see a new output stream.)
Smoother (but only in Firefox thus far -- in the standardization process) would be to use RTPSender.replaceTrack() to change the video track without touching audio or renegotiating.
Another option that will exist (though doesn't yet in either browser) is to use RTPSender.width/height (or whatever syntax gets agreed) to scale outgoing video before encoding.
Plan B for multistream/BUNDLE (which Chrome implements) was not adopted; Firefox has now (in Fx38 which goes out in a few days) implemented the Unified Plan; expect to see a blog post soon from someone on how to force the two to work together (until Chrome gets to implementing Unified Plan)

Google Play Services Multiplayer matching players errors

I developed a game for android with the google play services realtime multiplayer feature. I currently have a problem when matching the players. I dont use any invite feature, so all players just use the automatch functionality.
My game can be played with 4 players, but games with just 3 or 2 players are also possible. For my testing with 2 devices i use:
RoomConfig.createAutoMatchCriteria(minNumberOfOpponents, maxNumberOfOpponents, 0);
If i keep starting, ending and restarting games for a number of times, it often happens that the clients are not connected correctly. In the working cases the games onRoomConnected is called correctly and the game starts. In some cases tho, this is not happening. In theses cases, one device finds the other device and its onPeerJoined() and onRoomConnecting() callback is called. onRoomConnected() is never called tho. Thats because the other device gets no information whatsoever, just the roomCreated callback is called, and thats it.
So one device finds the other, and gets the information that another device joined the room. It also gets informed when this device leaves the room again. But the other device doesnt recognize any of this.
If this helps. i had some issues with losing connection before, and fixed it by restarting the apiClient everytime a room left on any clint. I dont think this is related tho.
I thought that might be a problem with leaving the current room correctly, and somehow joining the old room again, but it also happens ehen starting the app for the first time. Also the apiClient reconnect should avoid this problem
Thx in advance
Edit: It seems like its just my nexus 5 which produces the error. Every other device i tested works fine. The Nexus 5 does too in most cases. If the clients get connected and the game starts, there has never been any problem. The errrr just happens on this one device, and only in maybe 5 out of 6 cases, when searching an online game.
It just stops getting any callbacks called, sometimes right after the onRoomCreated(), sometime after he found another peer and onRoomConnecting(), and sometimes after onRoomConnected() has been called.
The other device gets its appropiate callbacks called tho in these cases.
So if the error device stops at onRoomCreated() the other device finds the client.
If the error device finds the other device and gets onRoomConnecting() called and stops after, the other device gets its onRoomConnected().
And if the error device gets its onRoomConnected() called, it sometimes even stops getting any messages from there on, while the other device is already in the game.
Maybe this helps someone. i'm not 100 % sure i fixed my problem. Haven't tested it in depth, but it seems everything is working fine now.
My problem was, that i have 2 different threads in my application, where the standard activity GUI thread starts the apiClient and handles the callbacks, while the gameengine thread initiates the room creation and sends the reliable messages via the apiClient.
It seems, sometimes things mess up while the peers connect and trade their first data. Currently i avoided directly calling any apiClient actions from the gameengine thread, but use runOnGuiThread to handle these actions on the Activity Gui Thread.

iBeacons with app killed

I am testing our iBeacons on iOS 7.1 and I can detect beacons correctly when I am in foreground and invoke the app from the background. However the issue comes when user has killed the app. The "didDetermineState" callback get invoked twice when I am in the beacon region while the app is killed. Am I missing the obvious or has anyone experience this same behavior ?
I have seen this happen not just when the app is completely killed, but in other cases, too. You will need to add filtering logic to your region callback methods, so if you get multiple calls it doesn't cause trouble in your app.
Another common issue is that you will get an exit region notification, followed within a couple of seconds by an entry region notification. Again a software filter is the way to deal with this. An example of a software filter for iBeacon callbacks is described here.

Reducing memory size to ensure backgrounding in iOS

When developing an app that uses Bluetooth Low Energy, there comes a time when the iOS device loses connection to the peripheral. (Sometimes for hours.)
In order to reconnect to an existing peripheral, the app must constantly scan in the background at a specific rate throughout the day(s), even when the app is backgrounded.
The problem is, iOS will not guarantee that your app will not get killed, due to memory constraints, etc.
Information found in the iPhone OS Programming guide states that:
Apps that work with Bluetooth peripherals can ask to be woken up if
the peripheral delivers an update when the app is suspended. This
support is important for Bluetooth-le accessories that deliver data at
regular intervals, such as a Bluetooth heart rate belt. When an app
includes the UIBackgroundModes key with the bluetooth-central value in
its Info.plist file, the Core Bluetooth framework keeps open any
active sessions for the corresponding peripheral. In addition, new
data arriving from the peripheral causes the system to wake up the app
so that it can process the data. The system also wakes up the app to
process accessory connection and disconnection notifications.
The problem does not arise when the phone is connected to a device and the application is background. It does happen, however, when the device is disconnected and the app is backgrounded. In this specific case, the phone is no longer connected to the peripheral, and therefore no longer getting notifications.
Many people have discussed this before, either on Stack Overflow or the Apple forums, and I believe one of the Apple developers has responded saying:
We're aware of this issue and are trying to come up with a solution.
Currently, there is no workaround."
My question is, is there a way to at least improve your chances of not getting killed by iOS due to memory constraints?
For example, an instant messaging app (IMO) seems to run quite nicely in the background. After days and days of not being used, the app will wake up and display a gChat message.
I’m questioning things such as
Strong pointers
Overall memory size
Reducing memory size when app is backgrounded or minimized
Reducing frequency of background operation
Etc.
Why do you need background execution even when the bluetooth hardware is disconnected?
I don't think that you need to "rescan continuously" to reconnect again, if the hardware is "paired" with the iPhone/iPad, it will reconnect itself. Like a bluetooth headset. Or not?
AFAIK you have no chances to accomplish what you are asking for.
A normal App is always suspended when the user go back to home. The app has approx. 5 secs of background time to stop timers, save state ecc ecc.
There are special background modes that allows you to have more background time, and each of this mode (explained in the page you linked) has a different behavior.
About the bluetooth mode:
The descripted behavior is not an issue, but it's by design:
the app is suspended
when the app is suspended, it can be killed by the OS to free ram (and there are no tricks to avoid this), but the system will wake up if needed.
the app is then awaken every time a notification is received (awaken from suspended state or lauched from "previously-killed" state)
the app has 10 seconds to do tasks (save informations, ecc ecc). Moreover, can request +10 mins. of background time for a particular task
after the 10 secs (or 10 min) the app is suspended again
The example you wrote about the chat app is incorrect: chat apps usually doesn't use any background mode, simply they forward you messages using push notifications. When you open the app, the app connect to a server that stores all your messages and download it.
You can get "more uptime" using location background mode (routing app can work in background), or using a combination of significative location changes (the app is awaken) and the 10 minutes background time, but I think that Apple will reject an app that "abuse" this.
Shortly, you have to design your app to support this behavior.
I found this in some more Apple documentation:
Memory Usage for Background Apps
Every app should free up as much memory as is practical upon entering
the background. The system tries to keep as many apps in memory at the
same time as it can, but when memory runs low it terminates suspended
apps to reclaim that memory. Apps that consume large amounts of memory
while in the background are the first apps to be terminated.
Practically speaking, your app should remove strong references to
objects as soon as they are no longer needed. Removing strong
references gives the compiler the ability to release the objects right
away so that the corresponding memory can be reclaimed. However, if
you want to cache some objects to improve performance, you can wait
until the app transitions to the background before removing references
to them.
Some examples of objects that you should remove strong references to
as soon as possible include:
Image objects
Large media or data files that you can load again from disk Any other
objects that your app does not need and can recreate easily later To
help reduce your app’s memory footprint, the system automatically
purges some data allocated on behalf of your app when your app moves
to the background.
The system purges the backing store for all Core Animation layers.
This effort does not remove your app’s layer objects from memory, nor
does it change the current layer properties. It simply prevents the
contents of those layers from appearing onscreen, which given that the
app is in the background should not happen anyway. It removes any
system references to cached images. (If your app does not have a
strong reference to the images, they are subsequently removed from
memory.) It removes strong references to some other system-managed
data caches.
From the apple documentation i have always assumed the following:
connectPeripheral:options:
Establish a connection to the peripheral.
- (void)connectPeripheral:(CBPeripheral *)peripheral options:(NSDictionary *)options;
Parameters
peripheral
The peripheral to connect to.
options
A dictionary to customize the behavior of the connection. See CBConnectPeripheralOptionNotifyOnDisconnectionKey.
Discussion
**This never times out**. Use cancelPeripheralConnection: to cancel a pending connection.'
With the important part being on the fact that this never times out. I would assume this would hand off to the system so that it will connect automatically to the peripheral when it comes into range, thus removing the need for full backgrounding. Someone correct me if im wrong though!