I wonder if anyone is able to help with this conundrum...? On a RPi 4, running the AWS Labs WebRTC SDK's sample app (https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/blob/master/samples/kvsWebRTCClientMasterGstreamerSample.c) I have edited the gstreamer pipeline to send 3x video streams from USB webcams/HDMI capture and audio stream from one of the camera's mic. It's working really nicely... except:
When testing only the video streams (via https://matwerber1.github.io/aws-kinesisvideo-webrtc-react/) , latency is very low, but once I add in the audio, it starts off in sync but gradually the video becomes approx 2 second delayed. One alternative pipeline setup with a single camera had the opposite effect of the audio gradually slipping out of sync to about 2 secs latency.
This is my pipeline as added to the sample app:
"v4l2src do-timestamp=TRUE device=/dev/video0 ! "
"video/x-raw,width=720,height=480 ! "
"videomixer name=mix sink_1::ypos=10 sink_1::xpos=10 sink_2::ypos=10 sink_2::xpos=180 ! "
"queue ! videoconvert ! "
"x264enc bframes=0 speed-preset=veryfast bitrate=1024 byte-stream=TRUE tune=zerolatency ! "
"video/x-h264,stream-format=byte-stream,alignment=au,profile=high,framerate=30/1 ! "
"appsink sync=TRUE emit-signals=TRUE name=appsink-video "
"v4l2src device=/dev/video2 ! "
"queue ! videoconvert ! video/x-raw,width=160,height=120 ! mix.sink_1 "
"v4l2src device=/dev/video4 ! "
"queue ! videoconvert ! video/x-raw,width=160,height=120 ! mix.sink_2 "
"alsasrc device=hw:2,0 !"
"queue ! audioconvert ! audioresample ! opusenc ! "
"audio/x-opus,rate=48000,channels=1 ! appsink sync=TRUE emit-signals=TRUE name=appsink-audio"
I have tried adjusting nearly all parameters with no improvement. Do I have the queue ! elements in the right places? Do I need to employ some buffering, if so, where? I have tried adding framerates to the caps but that stops all streams working completely.
Any recommendations or suggestions appreciated.
Thanks
Related
I am succesfully processing my video into x264. I am happy with the output and file size I am generating, but the video seems to take as long as the video is to process.
For example, if the video is 10 seconds long, it will take 10 seconds to process the video, etc...
Even if I put -crf 50, it will take the same time. I find this odd behaviour.
Note: I am using FFmpeg with FFmpegKit (React Native): https://github.com/tanersener/ffmpeg-kit
I am using full-gpl in order to be able to encode to x264. I am not sure what is wrong here or if this is normal behaviour?
Here is my FFmpeg command I am executing:
`-y -i ${media.path} -c:v libx264 -preset veryfast -tune fastdecode -crf 20 -vf "crop=1350:1080, scale=960:780" -c:a copy -movflags faststart ${path}`
I appreciate all the help I can get here to speed this up / fix this issue.
Cheers!
According to Kurento documentation: http://doc-kurento.readthedocs.io/en/stable/mastering/kurento_API.html
GstreamerFilter is a generic filter interface that allow use GStreamer filter in Kurento Media Pipelines.
I was trying to find Gstreamer filters on google, all I found was Gstreamer plugins. (https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/
Does this mean I can use the Kurento Gstreamer filter, to add plugins such as rtph264depay and rtmpsink with it?
e.g.
WebRTC endpoint > RTP Endpoint > (rtph264depay) Gstreamer filter (rtmpsink) > RTMP server.
All without installing Gstreamer separately?
GstreamerFilter allows you to configure a filter using a native GStreamer filter (the same way than when you are using gst-launch-1.0). For example, the following Kurento filter allows to rotate horizontally your media within KMS:
GStreamerFilter filter = new GStreamerFilter.Builder(pipeline, "videoflip method=horizontal-flip").build();
Said that, and regarding your question, for the best of my knowledge, I think so, you can use GstreamerFilter to use rtph264depay and rtmpsink.
Boni Garcia 's code is right.
But if you replace "videoflip method=horizontal-flip" as "rtmpsink location=rtmp://deque.me/live/test01", you will get a error message: "Given command is not valid, pad templates does not match".
You can go deeper to check kms-filter source code from https://github.com/Kurento/kms-filters, in kms-filters/src/server/implementation/objects/GStreamerFilterImpl.cpp there is a line:
99 throw KurentoException (MARSHALL_ERROR,
100 "Given command is not valid, pad templates does not match");
I afraid you can't use GstreamerFilter to send data to rtmp server, maybe you should modify the source code a bit.
Kurento
Just looking at the source - the GStreamerFilter is limited to simple GStreamer plugins. They reject bins and I don't see how you would specify/isolate multiple pads so it probably won't do it.
(EDIT: Maybe I'm wrong here - I'm still learning. I see the mixer example isolating media types and that makes me think it may be possible)
gstreamer
On the other hand installing gstreamer shouldn't really be that much overhead - then link the output RTP connection to a gst-launch pipeline that can output RTMP. It just sucks you can't manage the full pipeline using kurento.
(I don't know what that pipeline would look like - investigating it myself. It's something like this:
gst-launch-1.5 -v \
udpsrc port=9999 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! mux. \
multifilesrc location=sample.aac loop=1 ! aacparse ! mux. \
mpegtsmux name=mux mux. ! rtpmp2tpay ! queue ! udpsink host=10.20.20.20 port=5000
But I'm faking audio in this and haven't gotten the full stream working)
back to kurento
Further exploration suggested maybe the Composite MediaElement would work (tl;dr: no):
Composite composite = new Composite.Builder(pipeline).build();
HubPort in_audio = new HubPort.Builder(composite).build();
HubPort in_video = new HubPort.Builder(composite).build();
HubPort out_composite = new HubPort.Builder(composite).build();
GStreamerFilter filter = new GStreamerFilter.Builder(pipeline, "rtmpsink location=rtmp://127.0.0.1/live/live_stream_720p").build();
webRtcEndpoint.connect(in_audio, MediaType.AUDIO);
webRtcEndpoint.connect(in_video, MediaType.VIDEO);
out_composite.connect(filter);
results in (kurento logs):
...15,011560 21495 [0x4f01700] debug KurentoWebSocketTransport WebSocketTransport.cpp:422 processMessage() Message: >{"id":28,"method":"create","params":{"type":"GStreamerFilter","constructorParams":{"mediaPipeline":"5751ec53_kurento.MediaPipeline","command":"rtmpsink location=rtmp://127.0.0.1/live/live_stream_720p"},"properties":{},"sessionId":"d8abb1d8"},"jsonrpc":"2.0"}<
...15,011862 21495 [0x4f01700] debug KurentoGStreamerFilterImpl GStreamerFilterImpl.cpp:47 GStreamerFilterImpl() Command rtmpsink location=rtmp://127.0.0.1/live/live_stream_720p
...15,015698 21495 [0x4f01700] error filterelement kmsfilterelement.c:148 kms_filter_element_set_filter() <kmsfilterelement0> Invalid factory "rtmpsink", unexpected pad templates
...15,016841 21495 [0x4f01700] debug KurentoWebSocketTransport WebSocketTransport.cpp:424 processMessage() Response: >{"error":{"code":40001,"data":{"type":"MARSHALL_ERROR"},"message":"Given command is not valid, pad templates does not match"},"id":28,"jsonrpc":"2.0"}
I.e. failure.
I am streaming a .ts file which contains both video and audio over UDP by using below pipeline,
GST_DEBUG=6 gst-launch-1.0 filesrc location=vafpd.ts ! tsdemux program-number=10 name=demux demux. ! queue ! h264parse ! muxer.sink_300 mpegtsmux name=muxer prog-map=program_map,sink_300=10,sink_301=10 ! rtpmp2tpay ! udpsink host=192.168.1.139 port=8765 sync=true async=true qos=true demux. ! queue ! faad ! faac ! aacparse ! muxer.sink_301
It is working fine and receives EOS, whenever the both audio and video durations are same.
But when i streamed a .ts file which has audio and video. and The audio duration is more than the video duration, then It never ends and never caught EOS.
When ever the video ends, then below INFO stops printing,
0:00:03.392743787 3358 0xa90050 INFO h264parse gsth264parse.c:1335:gst_h264_parse_update_src_caps:<h264parse0> PAR 1/1
0:00:04.059453486 3358 0xa90050 INFO baseparse gstbaseparse.c:3644:gst_base_parse_set_latency:<h264parse0> min/max latency 0:00:00.033333333, 0:00:00.033333333
These are the log messages,,, After end of the video duration,
0:05:19.714110763 3323 0x10ef990 LOG baseparse gstbaseparse.c:2919:gst_base_parse_chain:<h264parse0> chain leaving
0:05:19.714128717 3323 0x10ef990 LOG GST_SCHEDULING gstpad.c:3834:gst_pad_chain_data_unchecked:<h264parse0:sink> called chainfunction &gst_base_parse_chain with buffer 0x7f92a8013c00, returned ok
0:05:19.714152324 3323 0x10ef990 DEBUG queue_dataflow gstqueue.c:1277:gst_queue_loop:<queue0> queue is empty
0:05:19.714172126 3323 0x10ef990 LOG queue_dataflow gstqueue.c:1286:gst_queue_loop:<queue0> (queue0:src) wait for ADD: 0 of 0-200 buffers, 0 of 0-10485760 bytes, 0 of 0-1000000000 ns, 0 items
When GST-DEBUG=3,
0:00:00.031284464 3365 0x18ba920 WARN basesrc gstbasesrc.c:3483:gst_base_src_start_complete:<filesrc0> pad not activated yet
Pipeline is PREROLLING ...
0:00:00.033063992 3365 0x18bc450 WARN h264parse gsth264parse.c:1025:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 9 AU delimiter, Size: 2 will be dropped
0:00:00.047101000 3365 0x18bc4f0 FIXME basesink gstbasesink.c:3064:gst_base_sink_default_event:<udpsink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
How to resolve this issue?
I have instaled ARToolKit on Ubuntu 12.10 on a 64bit Asus. The install gave no errors so I think I'm ok. But when I want to try one af the examples it can't find the camera. If I don't fill anything in at char *vconf = ""; I get
No video config string supplied, using defaults.
ioctl failed
The most often found solution implies
char *vconf = "v4l2src device=/dev/video0 use-fixed-fps=false ! ffmpegcolorspace ! capsfilter caps=video/x-raw-rgb,width=640,height=480 ! identity name=artoolkit ! fakesink";
But this doesn't work for me. I get
r#r-K55VD:~/Downloads/Artoolkit-on-Ubuntu-12.04-master/bin$ ./simpleTest
Using supplied video config string [v4l2src device=/dev/video0 use-fixed-fps=false ! ffmpegcolorspace ! capsfilter caps=video/x-raw-rgb,width=640,height=480 ! identity name=artoolkit ! fakesink].
ARVideo may be configured using one or more of the following options,
separated by a space:
DEVICE CONTROLS:
-dev=filepath
specifies device file.
-channel=N
specifies source channel.
-noadjust
prevent adjusting the width/height/channel if not suitable.
-width=N
specifies expected width of image.
-height=N
specifies expected height of image.
-palette=[RGB|YUV420P]
specifies the camera palette (WARNING:all are not supported on each camera !!).
IMAGE CONTROLS (WARNING: every options are not supported by all camera !!):
-brightness=N
specifies brightness. (0.0 <-> 1.0)
-contrast=N
specifies contrast. (0.0 <-> 1.0)
-saturation=N
specifies saturation (color). (0.0 <-> 1.0) (for color camera only)
-hue=N
specifies hue. (0.0 <-> 1.0) (for color camera only)
-whiteness=N
specifies whiteness. (0.0 <-> 1.0) (REMARK: gamma for some drivers, otherwise for greyscale camera only)
-color=N
specifies saturation (color). (0.0 <-> 1.0) (REMARK: obsolete !! use saturation control)
OPTION CONTROLS:
-mode=[PAL|NTSC|SECAM]
specifies TV signal mode (for tv/capture card).
What is a methodological way of finding out what exactly to put at char *vconf = " " ?Because I feel I tried a lot of variations at random, but nothing works. I know it needs a path like /dev/video0, but what else seems up in the air to me.
char *vconf = "v4l2src device=/dev/video0 use-fixed-fps=false !
ffmpegcolorspace ! capsfilter
caps=video/x-raw-rgb,width=640,height=480 ! identity name=artoolkit !
fakesink";
The above configuration you tried is for GStreamer driver.
Since you are using VideoLinuxV4L , instead of the above use:
char *vconf = "-dev=/dev/video0 ";
For more you can refer to "{ARtoolkit Folder}/doc/video/index.html"
is there a way for discovering all the available encodings of a certain webcam (e.g x-raw-rgb -xraw-yuv)?
Morevoer, I would like to discover also the available resolutions.
Thanks!
Yes, set the v4l2src element to ready and check the caps on the src pad. The element will narrow the list of caps down to the ones actually supported when it has opened and queried an actual device. That happens in READY state.
What I do is the following (command line):
GST_DEBUG=v4l2src:3 gst-launch v4l2src ! decodebin2 ! xvimagesink
If the video source in onboard else change the "v4l2src". This will show ALOT of info, from "probed caps:" it will long line of possible formats the video source supports.
Here is a same copy/paste from my machine:
probed caps: video/x-raw-yuv, format=(fourcc)YUY2, width=(int)1280,
height=(int)720, interlaced=(boolean)false,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 10/1 };
video/x-raw-yuv, format=(fourcc)YUY2, width=(int)640, height=(int)480,
interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1,
framerate=(fraction){ 30/1 };
So the info your looking for is:
! video/x-raw-yuv, framerate=30/1, width=640, height=480, interlaced=false !
If anything NOT from the probed list will result in error:
could not negotiate format