I am creating a messaging app with SwiftUI, and I want to add video calling function to that. I used SkyWay webRTC API (https://webrtc.ecl.ntt.com/en/) to achieve this and I could build an example project written in swift code. Now what I am trying is linking local stream with SKWVideo view and wrapping it up into UIViewRepresentable. But I got stuck with bellow error message.
import SwiftUI
import SkyWay
import UIKit
struct ContentView: View {
#State var video = SKWVideo()
var body: some View {
VideoView(localStreamView: $video)
}
}
struct VideoView: UIViewRepresentable {
#Binding var localStreamView: SKWVideo
func makeUIView(context: Context) -> SKWVideo {
let option: SKWPeerOption = SKWPeerOption.init()
option.key = "xxxx"
option.domain = "localhost"
let peer = SKWPeer(options: option)
SKWNavigator.initialize(peer!)
let constraints: SKWMediaConstraints = SKWMediaConstraints()
let localStream = SKWNavigator.getUserMedia(constraints)
localStream?.addVideoRenderer(localStreamView, track: 0)
return localStreamView
}
func updateUIView(_ uiView: SKWVideo, context: Context) {
//
}
}
I got this error
Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Modifications to the layout engine must not be performed from a background thread after it has been accessed from the main thread.'
I ran it on a iPhone 7 real device, and I wrote plist settings. I have completely no idea right now. Please help me..
iOS 13.0
xcode 11.0
Related
I would appreciated some help please even if this is maybe a trivial question.
I've written a SwiftUI app that reads the media library from the device and plays it depending on user settings. That is all fine.
The problem I have is that if you install the app for the first time, the user needs to grant permission to access the media library. This appears to be a system generated dialog but I cannot see which step in the also triggers it. I tried to have the access request be triggered code generated but that doesn't seem to trigger the pop up but it still only appears at a later stage in the app load process. The code seems to recognise though that the user reacted to the access request pop up and does select the correct switch case.
What it does not seem to do though is that it still can't read the media library. The MPMediaQuery returns nil.
My suspicion is that it somehow connected to the fact that the access request doesn't run on the main thread but I am not experienced enough in Swift programming to know what the problem is. I would be most grateful for some helpful hints.
Here is my code:
import MediaPlayer
import SwiftUI
import Foundation
class Library {
var artists : [Artist] = []
#EnvironmentObject var settings : UserSettings
var counter : Float = 0
init() {
switch MPMediaLibrary.authorizationStatus() {
case .authorized:
print("authorized")
case .denied:
print("denied")
return
case .notDetermined:
print("not determined")
MPMediaLibrary.requestAuthorization() { granted in
if granted != .authorized {
return
}
}
case .restricted:
print("restricted")
#unknown default:
print("default")
}
if MPMediaLibrary.authorizationStatus() == .notDetermined { return }
let filter : Set<MPMediaPropertyPredicate> = [MPMediaPropertyPredicate(value: MPMediaType.music.rawValue, forProperty: MPMediaItemPropertyMediaType)]
let mediaQuery = MPMediaQuery(filterPredicates: filter )
var artistsInCollection : [Artist] = []
guard let _ = mediaQuery.items?.count else { return }
for item in mediaQuery.items! {
//here I do something but that's not relevant to my question
}
self.artists = artistsInCollection
}
}
I have a very big objective-c project and am trying to incorporate SwiftUI slowly. I have been able to successfully implement some screens in swiftUI classes which are called from Objective-c classes.
One strange bug am facing now is that List is not visible when the app is run. It is visible in preview though. And also when I replace List with VStack, it appears fine when the app is run. If I make a standalone SwiftUI app and run this code, then too it runs fine. The issue is only when I call this class from objective-c class.
Attached images
Xcode Preview Screenshot
Device Screenshot
Code
import SwiftUI
struct InfoShareView: View {
var body: some View {
List{
Text("First Line")
Text("Second Line")
}
.padding()
.background(Color.white)
}
}
struct InfoShareView_Previews: PreviewProvider {
static var previews: some View {
InfoShareView()
}
}
And here is the call to InfoShareView
import UIKit
import SwiftUI
class AboutViewControllerSwift: UIViewController {
override func viewDidLoad() {
super.viewDidLoad()
let swiftController = UIHostingController(rootView: InfoShareView())
addChild(swiftController)
swiftController.view.translatesAutoresizingMaskIntoConstraints = false
view.addSubview(swiftController.view)
swiftController.view.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
swiftController.view.centerYAnchor.constraint(equalTo: view.centerYAnchor).isActive = true
}
}
And here is the call from objective-c class
AboutViewControllerSwift *aboutViewController = [[AboutViewControllerSwift alloc] initWithNibName:nil bundle:nil];
self.nvgController=[[UINavigationController alloc]initWithRootViewController:aboutViewController];
self.window.rootViewController=self.nvgController;
[self.window makeKeyAndVisible];
I think this is due to constraints, try use instead following
...
view.addSubview(swiftController.view)
swiftController.view.topAnchor.constraint(equalTo: view.topAnchor).isActive = true
swiftController.view.bottomAnchor.constraint(equalTo: view.bottomAnchor).isActive = true
swiftController.view.leadingAnchor.constraint(equalTo: view.leadingAnchor).isActive = true
swiftController.view.trailingAnchor.constraint(equalTo: view.trailingAnchor).isActive = true
Adding this to the last line of viewdidload worked for me:
swiftController.view.frame = self.view.frame
I'm trying to create a video recorder in a nativescript plugin on the ios side which means that I am using the native Objective C Classes inside of the plugin to have a shared interface in the nativescript app with the android implementation.
I have the camera view loaded and I am trying to get access to the video frames from the AVCaptureSession. I created an object that implements the protocol to get the frames and for the first second the function captureOutput which has a parameter DidOutputSampleBuffer outputs the frames. But from then on all the frames are dropped and I do not know why. I can view that they are dropped because the function in the protocol, captureOutput with parameter DidDropSampleBuffer runs for every frame.
I tried changing the order of intializion for the avcapturesession but that didn't change anything.
Below is the main function with the main code to create the capture session and capture object. While this is typescript nativescript allows you to call native Objective C functions and classes so the logic is the same in objective c. I also create a VideoDelegate object in nativescript which corresponds to a class in Objective C which allows me to implement the protocol to get the video frames from the capture device output.
this._captureSession = AVCaptureSession.new();
//Get the camera
this._captureSession.sessionPreset = AVCaptureSessionPreset640x480;
let inputDevice = null;
this._cameraDevice = AVCaptureDevice.defaultDeviceWithMediaType(AVMediaTypeVideo);
//Get the camera input
let error: NSError = null;
this._captureInput = AVCaptureDeviceInput.deviceInputWithDeviceError(this._cameraDevice);
if(this._captureSession.canAddInput(this._captureInput)){
this._captureSession.addInput(this._captureInput);
}
else{
console.log("couldn't add input");
}
let self = this;
const VideoDelegate = (NSObject as any).extend({
captureOutputDidOutputSampleBufferFromConnection(captureOutput: any,sampleBuffer: any, connection:any): void {
console.log("Captureing Frames");
if(self.startRecording){
self._mp4Writer.appendVideoSample(sampleBuffer);
console.log("Appending Video Samples");
}
},
captureOutputDidDropSampleBufferFromConnection(captureOutput: any,sampleBuffer: any, connection:any): void {
console.log("Dropping Frames");
},
videoCameraStarted(date){
// console.log("CAMERA STARTED");
}
}, {
protocols: [AVCaptureVideoDataOutputSampleBufferDelegate]
});
this._videoDelegate = VideoDelegate.new();
//setting up camera output for frames
this._captureOutput = AVCaptureVideoDataOutput.new();
this._captureQueue = dispatch_queue_create("capture Queue", null);
this._captureOutput.setSampleBufferDelegateQueue(this._videoDelegate,this._captureQueue);
this._captureOutput.alwaysDiscardsLateVideoFrames = false;
this._framePixelFormat = NSNumber.numberWithInt(kCVPixelFormatType_32BGRA);
this._captureOutput.videoSettings = NSDictionary.dictionaryWithObjectForKey(this._framePixelFormat,kCVPixelBufferPixelFormatTypeKey);
this._captureSession.addOutput(this._captureOutput);
this._captureSession.startRunning();
So far tvOS supports two ways to make tv apps, TVML and UIKit, and there is no official mentions about how to mix up things to make a TVML (that is basically XML) User Interface with the native counter part for the app logic and I/O (like playback, streaming, iCloud persistence, etc).
So, which is the best solution to mix TVML and UIKit in a new tvOS app?
In the following I have tried a solution following code snippets adapted from Apple Forums and related questions about JavaScriptCore to ObjC/Swift binding.
This is a simple wrapper class in your Swift project.
import UIKit
import TVMLKit
#objc protocol MyJSClass : JSExport {
func getItem(key:String) -> String?
func setItem(key:String, data:String)
}
class MyClass: NSObject, MyJSClass {
func getItem(key: String) -> String? {
return "String value"
}
func setItem(key: String, data: String) {
print("Set key:\(key) value:\(data)")
}
}
where the delegate must conform a TVApplicationControllerDelegate:
typealias TVApplicationDelegate = AppDelegate
extension TVApplicationDelegate : TVApplicationControllerDelegate {
func appController(appController: TVApplicationController, evaluateAppJavaScriptInContext jsContext: JSContext) {
let myClass: MyClass = MyClass();
jsContext.setObject(myClass, forKeyedSubscript: "objectwrapper");
}
func appController(appController: TVApplicationController, didFailWithError error: NSError) {
let title = "Error Launching Application"
let message = error.localizedDescription
let alertController = UIAlertController(title: title, message: message, preferredStyle:.Alert ) self.appController?.navigationController.presentViewController(alertController, animated: true, completion: { () -> Void in
})
}
func appController(appController: TVApplicationController, didStopWithOptions options: [String : AnyObject]?) {
}
func appController(appController: TVApplicationController, didFinishLaunchingWithOptions options: [String : AnyObject]?) {
}
}
At this point the javascript is very simple like. Take a look at the methods with named parameters, you will need to change the javascript counter part method name:
App.onLaunch = function(options) {
var text = objectwrapper.getItem()
// keep an eye here, the method name it changes when you have named parameters, you need camel case for parameters:
objectwrapper.setItemData("test", "value")
}
App. onExit = function() {
console.log('App finished');
}
Now, supposed that you have a very complex js interface to export like
#protocol MXMJSProtocol<JSExport>
- (void)boot:(JSValue *)status network:(JSValue*)network user:(JSValue*)c3;
- (NSString*)getVersion;
#end
#interface MXMJSObject : NSObject<MXMJSProtocol>
#end
#implementation MXMJSObject
- (NSString*)getVersion {
return #"0.0.1";
}
you can do like
JSExportAs(boot,
- (void)boot:(JSValue *)status network:(JSValue*)network user:(JSValue*)c3 );
At this point in the JS Counter part you will not do the camel case:
objectwrapper.bootNetworkUser(statusChanged,networkChanged,userChanged)
but you are going to do:
objectwrapper.boot(statusChanged,networkChanged,userChanged)
Finally, look at this interface again:
- (void)boot:(JSValue *)status network:(JSValue*)network user:(JSValue*)c3;
The value JSValue* passed in. is a way to pass completion handlers between ObjC/Swift and JavaScriptCore. At this point in the native code you do all call with arguments:
dispatch_async(dispatch_get_main_queue(), ^{
NSNumber *state = [NSNumber numberWithInteger:status];
[networkChanged.context[#"setTimeout"]
callWithArguments:#[networkChanged, #0, state]];
});
In my findings, I have seen that the MainThread will hang if you do not dispatch on the main thread and async. So I will call the javascript "setTimeout" call that calls the completion handler callback.
So the approach I have used here is:
Use JSExportAs to take car of methods with named parameters and avoid to camel case javascript counterparts like callMyParam1Param2Param3
Use JSValue as parameter to get rid of completion handlers. Use callWithArguments on the native side. Use javascript functions on the JS side;
dispatch_async for completion handlers, possibly calling a setTimeout 0-delayed in the JavaScript side, to avoid the UI to freeze.
[UPDATE]
I have updated this question in order to be more clear. I'm finding a technical solution for bridging TVML and UIKit in order to
Understand the best programming model with JavaScriptCode
Have the right bridge from JavaScriptCore to ObjectiveC and
viceversa
Have the best performances when calling JavaScriptCode from Objective-C
This WWDC Video explains how to communicate between JavaScript and Obj-C
Here is how I communicate from Swift to JavaScript:
//when pushAlertInJS() is called, pushAlert(title, description) will be called in JavaScript.
func pushAlertInJS(){
//allows us to access the javascript context
appController!.evaluateInJavaScriptContext({(evaluation: JSContext) -> Void in
//get a handle on the "pushAlert" method that you've implemented in JavaScript
let pushAlert = evaluation.objectForKeyedSubscript("pushAlert")
//Call your JavaScript method with an array of arguments
pushAlert.callWithArguments(["Login Failed", "Incorrect Username or Password"])
}, completion: {(Bool) -> Void in
//evaluation block finished running
})
}
Here is how I communicate from JavaScript to Swift (it requires some setup in Swift):
//call this method once after setting up your appController.
func createSwiftPrint(){
//allows us to access the javascript context
appController?.evaluateInJavaScriptContext({(evaluation: JSContext) -> Void in
//this is the block that will be called when javascript calls swiftPrint(str)
let swiftPrintBlock : #convention(block) (String) -> Void = {
(str : String) -> Void in
//prints the string passed in from javascript
print(str)
}
//this creates a function in the javascript context called "swiftPrint".
//calling swiftPrint(str) in javascript will call the block we created above.
evaluation.setObject(unsafeBitCast(swiftPrintBlock, AnyObject.self), forKeyedSubscript: "swiftPrint" as (NSCopying & NSObjectProtocol)?)
}, completion: {(Bool) -> Void in
//evaluation block finished running
})
}
[UPDATE] For those of you who would like to know what "pushAlert" would look like on the javascript side, I'll share an example implemented in application.js
var pushAlert = function(title, description){
var alert = createAlert(title, description);
alert.addEventListener("select", Presenter.load.bind(Presenter));
navigationDocument.pushDocument(alert);
}
// This convenience funnction returns an alert template, which can be used to present errors to the user.
var createAlert = function(title, description) {
var alertString = `<?xml version="1.0" encoding="UTF-8" ?>
<document>
<alertTemplate>
<title>${title}</title>
<description>${description}</description>
</alertTemplate>
</document>`
var parser = new DOMParser();
var alertDoc = parser.parseFromString(alertString, "application/xml");
return alertDoc
}
You sparked an idea that worked...almost. Once you have displayed a native view, there is no straightforward method as-of-yet to push an TVML-based view onto the navigation stack. What I have done at this time is:
let appDelegate = UIApplication.sharedApplication().delegate as! AppDelegate
appDelegate.appController?.navigationController.popViewControllerAnimated(true)
dispatch_async(dispatch_get_main_queue()) {
tvmlContext!.evaluateScript("showTVMLView()")
}
...then on the JavaScript side:
function showTVMLView() {setTimeout(function(){_showTVMLView();}, 100);}
function _showTVMLView() {//push the next document onto the stack}
This seems to be the cleanest way to move execution off the main thread and onto the JSVirtualMachine thread and avoid the UI lockup. Notice that I had to pop at the very least the current native view controller, as it was getting sent a deadly selector otherwise.
It appears to me that the documentation PubNub has for getting started in Swift don't apply to versions earlier than PubNub 4.0. I can't successfully establish a callback to register with PubNub.
My code:
class Communicator: NSObject, PNObjectEventListener {
var pubNubClient: PubNub
override init(){
let config = PNConfiguration(
publishKey: "my_publish_key",
subscribeKey: "my_subscribe_key"
)
pubNubClient = PubNub.clientWithConfiguration(config);
super.init()
pubNubClient.addListener(self)
pubNubClient.subscribeToChannels(["my_channel"], withPresence: false)
}
func didReceiveMessage(client: PubNub!, message: PNMessageResult!){
/* THIS METHOD NEVER GETS REACHED */
}
}
Digging into the PubNub source a bit, this is the area that seems to be having problems:
- (void)addListener:(id <PNObjectEventListener>)listener {
dispatch_async(self.resourceAccessQueue, ^{
if ([listener respondsToSelector:#selector(client:didReceiveMessage:)]) {
/* this block is never reached!!! */
[self.messageListeners addObject:listener];
}
/* Remaining Lines Stripped Away */
});
}
I'm still relatively new to Swift and integrating with Objective C. I'm curious if there's a problem with the respondsToSelector since the Objective C code is referencing Swift code.
The messages are definitely getting passed; there's another lower level function in the PubNub library that's logging all the messages received.
Any help would be much appreciated.
Versions prior to 4.0 are deprecated and wont work exactly how they used to.
I would recommend migrating over to the newest (4.0) SDK entirely, the new iOS SDK has removed a lot of bloat and compiles much faster. To get started view this tutorial.
To summarize, instantiating a PubNub client look as follows:
let config = PNConfiguration(
publishKey: "Your_Pub_Key",
subscribeKey: "Your_Sub_Key")
client = PubNub.clientWithConfiguration(config)
client?.addListener(self)
client?.subscribeToChannels(["Your_Channel"], withPresence: false)
And the new didReceiveMessage function looks as follows:
func client(client: PubNub!, didReceiveMessage message: PNMessageResult!, withStatus status: PNErrorStatus!) {
//Do Something like
//println(message)
}
Resolved by adding:
func client(client: PubNub!, didReceiveMessage message: PNMessageResult!) {
}
The documentation on how to parse the received PNMessageResult is scant. Here's how I handled it:
func client(client: PubNub!, didReceiveMessage message: PNMessageResult!) {
let encodedMessage = message.data.valueForKey("message") as! NSDictionary
let messageType = encodedMessage["meta"]! as! String
let messageString = encodedMessage["data"]!["msg"]! as! String
print("PubNub: [\(messageType)] \(messageString)")
}
add _ client works for me!
func client(_ client: PubNub, didReceiveMessage message: PNMessageResult) {
print("Pubnub Message: \(message)")
}