The processing speed between swift and objective-c in Metal - objective-c

I have written a demo base on apple official document in swift. And I found the usage of CPU is lower in Objective-c than in swift.
Does it mean Objective-c is much effective than swift while handling metal app?
I'm confused because many people says that swift is faster than Objective-c in general. Or it is just an exception?
The demo involved pointer management. I know it is trouble handle pointer with swift. Maybe it is the reason why the app is cost much resource in swift? I am still finding.
The demo below is a triple buffer model which renders hundreds of small quads, updates their positions at the start of each frame and writes them into a vertex buffer. And uses semaphores to wait for full frame completions in case the CPU is running too far ahead of the GPU.
This is the part of code from apple official document
- (void)updateState
{
AAPLVertex *currentSpriteVertices = _vertexBuffers[_currentBuffer].contents;
NSUInteger currentVertex = _totalSpriteVertexCount-1;
NSUInteger spriteIdx = (_rowsOfSprites * _spritesPerRow)-1;
for(NSInteger row = _rowsOfSprites - 1; row >= 0; row--)
{
float startY = _sprites[spriteIdx].position.y;
for(NSInteger spriteInRow = _spritesPerRow-1; spriteInRow >= 0; spriteInRow--)
{
vector_float2 updatedPosition = _sprites[spriteIdx].position;
if(spriteInRow == 0)
{
updatedPosition.y = startY;
}
else
{
updatedPosition.y = _sprites[spriteIdx-1].position.y;
}
_sprites[spriteIdx].position = updatedPosition;
for(NSInteger vertexOfSprite = AAPLSprite.vertexCount-1; vertexOfSprite >= 0 ; vertexOfSprite--)
{
currentSpriteVertices[currentVertex].position = AAPLSprite.vertices[vertexOfSprite].position + _sprites[spriteIdx].position;
currentSpriteVertices[currentVertex].color = _sprites[spriteIdx].color;
currentVertex--;
}
spriteIdx--;
}
}
}
- (void)drawInMTKView:(nonnull MTKView *)view
{
dispatch_semaphore_wait(_inFlightSemaphore, DISPATCH_TIME_FOREVER);
_currentBuffer = (_currentBuffer + 1) % MaxBuffersInFlight;
[self updateState];
id<MTLCommandBuffer> commandBuffer = [_commandQueue commandBuffer];
commandBuffer.label = #"MyCommand";
__block dispatch_semaphore_t block_sema = _inFlightSemaphore;
[commandBuffer addCompletedHandler:^(id<MTLCommandBuffer> buffer)
{
dispatch_semaphore_signal(block_sema);
}];
MTLRenderPassDescriptor *renderPassDescriptor = view.currentRenderPassDescriptor;
if(renderPassDescriptor != nil)
{
id<MTLRenderCommandEncoder> renderEncoder =
[commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
renderEncoder.label = #"MyRenderEncoder";
[renderEncoder setCullMode:MTLCullModeBack];
[renderEncoder setRenderPipelineState:_pipelineState];
[renderEncoder setVertexBuffer:_vertexBuffers[_currentBuffer]
offset:0
atIndex:AAPLVertexInputIndexVertices];
[renderEncoder setVertexBytes:&_viewportSize
length:sizeof(_viewportSize)
atIndex:AAPLVertexInputIndexViewportSize];
[renderEncoder drawPrimitives:MTLPrimitiveTypeTriangle
vertexStart:0
vertexCount:_totalSpriteVertexCount];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:view.currentDrawable];
}
[commandBuffer commit];
}
and this is in swift
func updateSprite() {
let currentSpriteVertices = vertexBuffer[currentBuffer].contents().bindMemory(to: DFVertex.self, capacity: totalSpriteVertexCount * MemoryLayout<DFVertex>.size)
var currentVertex = totalSpriteVertexCount - 1
var spriteIdx = (rowsOfSprites * spritePerRow) - 1
for _ in stride(from: rowsOfSprites - 1, through: 0, by: -1) {
let startY = sprites[spriteIdx].position.y
for spriteInRow in stride(from: spritePerRow - 1, through: 0, by: -1) {
var updatePosition = sprites[spriteIdx].position
if spriteInRow == 0 {
updatePosition.y = startY
} else {
updatePosition.y = sprites[spriteIdx - 1].position.y
}
sprites[spriteIdx].position = updatePosition
for vertexOfSprite in stride(from: DFSprite.vertexCount - 1, through: 0, by: -1) {
currentSpriteVertices[currentVertex].position = DFSprite.vertices[vertexOfSprite].position + sprites[spriteIdx].position
currentSpriteVertices[currentVertex].color = sprites[spriteIdx].color
currentVertex -= 1
}
spriteIdx -= 1
}
}
}
func draw(in view: MTKView) {
inFlightSemaphore.wait()
currentBuffer = (currentBuffer + 1) % maxBufferInFlight
updateSprite()
let commandBuffer = commandQueue.makeCommandBuffer()
if commandBuffer == nil {
print("create command buffer failed.")
}
commandBuffer!.label = "command buffer"
commandBuffer!.addCompletedHandler { (buffer) in
self.inFlightSemaphore.signal()
}
if let renderPassDescriptor = view.currentRenderPassDescriptor,
let renderEncoder = commandBuffer!.makeRenderCommandEncoder(descriptor: renderPassDescriptor) {
renderEncoder.setCullMode(.back)
renderEncoder.setRenderPipelineState(pipelineState!)
renderEncoder.setVertexBuffer(vertexBuffer[currentBuffer],
offset: 0,
index: DFVertexInputIndex.vertex.rawValue)
renderEncoder.setVertexBytes(&viewportSize,
length: MemoryLayout.size(ofValue: viewportSize),
index: DFVertexInputIndex.viewportSize.rawValue)
renderEncoder.drawPrimitives(type: .triangle,
vertexStart: 0,
vertexCount: totalSpriteVertexCount)
renderEncoder.endEncoding()
commandBuffer!.present(view.currentDrawable!)
}
commandBuffer!.commit()
}
the result is that the app written in objective-c cost about 40% cpu usage while about 100% in swift. I thought swift would be faster.

Related

How to render Vulkan's scene to ImGui window

https://github.com/ocornut/imgui/issues/5110
Version: 1.87
Branch: docking
Back-end/Renderer/Compiler/OS
Back-ends: imgui_impl_Vulkan.cpp + imgui_impl_Vulkan.cpp
Operating System: MacOS
My Issue/Question:
How to render Vulakn's scene to ImGui window?
want to achieve this:
image
I have read various documents and #914,but I'm still at a loss.
Maybe I'm close to the answer?
Here is my project code, built quickly via CMake. It is implemented with reference to the Vulkan tutorial.
Also would appreciate any help, thanks a lot.
The key code is here:
m_Dset = ImGui_ImplVulkan_AddTexture(m_TextureSampler, m_TextureImageView, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL);
...
ImGui::Begin("Viewport");
ImVec2 viewportPanelSize = ImGui::GetContentRegionAvail();
ImGui::Image(m_Dset, ImVec2{viewportPanelSize.x, viewportPanelSize.y});
ImGui::End();
I tried several ways:
According to this code I got it:
m_Dset = ImGui_ImplVulkan_AddTexture(m_TextureSampler, m_TextureImageView, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL);
...
ImGui::Begin("Viewport");
ImVec2 viewportPanelSize = ImGui::GetContentRegionAvail();
ImGui::Image(m_Dset, ImVec2{viewportPanelSize.x, viewportPanelSize.y});
ImGui::End();
image
According to this code I got it:
m_Dset = ImGui_ImplVulkan_AddTexture(m_TextureSampler, m_SwapChainImageViews[currentFrame], VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL);
// std::vector<VkImageView> m_SwapChainImageViews;
...
ImGui::Begin("Viewport");
ImVec2 viewportPanelSize = ImGui::GetContentRegionAvail();
ImGui::Image(m_Dset, ImVec2{viewportPanelSize.x, viewportPanelSize.y});
ImGui::End();
image
But when I drag the ImGui window to the main window:
image
3.last attempt:
// remove m_Dset = ImGui_ImplVulkan_AddTexture(...);
...
ImGui::Begin("Viewport");
ImVec2 viewportPanelSize = ImGui::GetContentRegionAvail();
ImGui::Image(m_DescriptorSets[currentFrame], ImVec2{viewportPanelSize.x, viewportPanelSize.y});
ImGui::End();
I got it:
image
Hope you understand what I mean......Any help would be greatly appreciated.It can be quickly reproduced with the items provided above, if you have the time.
I've been stuck for days.
ps: I'm working tirelessly on problems I've encountered while learning Vulkan and ImGui. I have a cold because of it. 😭😭😭😭😭
I must be low IQ.
I have these variables:
private:
GLFWwindow *m_Window;
VkInstance m_Instance;
VkDebugUtilsMessengerEXT m_DebugMessenger;
VkPhysicalDevice m_PhysicalDevice = VK_NULL_HANDLE;
VkDevice m_Device;
VkQueue m_GraphicsQueue;
VkQueue m_PresentQueue;
VkSurfaceKHR m_Surface;
VkSwapchainKHR m_SwapChain;
std::vector<VkImage> m_SwapChainImages;
VkFormat m_SwapChainImageFormat;
VkExtent2D m_SwapChainExtent;
std::vector<VkImageView> m_SwapChainImageViews;
VkRenderPass m_RenderPass;
VkDescriptorSetLayout m_DescriptorSetLayout;
VkPipelineLayout m_PipelineLayout;
VkPipeline m_GraphicsPipeline;
std::vector<VkFramebuffer> m_SwapChainFramebuffers;
VkCommandPool m_CommandPool;
// for imgui
VkDescriptorPool m_ImGuiDescriptorPool;
VkRenderPass m_ImGuiRenderPass;
std::vector<VkFramebuffer> m_ImGuiFramebuffers;
VkCommandPool m_ImGuiCommandPool;
std::vector<VkCommandBuffer> m_ImGuiCommandBuffers;
VkImage m_TextureImage;
VkDeviceMemory m_TextureImageMemory;
VkImageView m_TextureImageView;
VkSampler m_TextureSampler;
VkImage m_DepthImage;
VkDeviceMemory m_DepthImageMemory;
VkImageView m_DepthImageView;
VkBuffer m_VertexBuffer;
VkDeviceMemory m_VertexBufferMemory;
VkBuffer m_IndexBuffer;
VkDeviceMemory m_IndexBufferMemory;
// UniformBuffer
std::vector<VkBuffer> m_UniformBuffers;
std::vector<VkDeviceMemory> m_UniformBuffersMemory;
VkDescriptorPool m_DescriptorPool;
std::vector<VkDescriptorSet> m_DescriptorSets;
std::vector<VkCommandBuffer> m_CommandBuffers;
std::vector<VkSemaphore> m_ImageAvailableSemaphores;
std::vector<VkSemaphore> m_RenderFinishedSemaphores;
std::vector<VkFence> m_InFlightFences;
QueueFamilyIndices m_QueueFamilyIndices;
uint32_t currentFrame = 0;
uint32_t m_ImageCount = 2;
VkDescriptorSet m_Dset;
void initVulkan()
{
createInstance();
setupDebugMessenger();
createSurface();
pickPhysicalDevice();
createLogicalDevice();
createSwapChain();
createImageViews();
createRenderPass();
createDescriptorSetLayout();
createGraphicsPipeline();
createCommandPool(&m_CommandPool);
createDepthResources();
createFramebuffers();
createTextureImage();
createTextureImageView();
createTextureSampler();
createVertexBuffer();
createIndexBuffer();
createUniformBuffers();
createDescriptorPool();
createDescriptorSets();
createCommandBuffers();
createSyncObjects();
IMGUI_CHECKVERSION();
ImGui::CreateContext();
ImGuiIO &io = ImGui::GetIO();
(void)io;
io.ConfigFlags |= ImGuiConfigFlags_NavEnableKeyboard;
io.ConfigFlags |= ImGuiConfigFlags_DockingEnable;
io.ConfigFlags |= ImGuiConfigFlags_ViewportsEnable;
ImGui::StyleColorsDark();
// When viewports are enabled we tweak WindowRounding/WindowBg so platform windows can look identical to regular ones.
ImGuiStyle &style = ImGui::GetStyle();
if (io.ConfigFlags & ImGuiConfigFlags_ViewportsEnable)
{
style.WindowRounding = 0.0f;
style.Colors[ImGuiCol_WindowBg].w = 1.0f;
}
{
VkDescriptorPoolSize pool_sizes[] =
{
{VK_DESCRIPTOR_TYPE_SAMPLER, 1000},
{VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, 1000},
{VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE, 1000},
{VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, 1000},
{VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER, 1000},
{VK_DESCRIPTOR_TYPE_STORAGE_TEXEL_BUFFER, 1000},
{VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER, 1000},
{VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1000},
{VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC, 1000},
{VK_DESCRIPTOR_TYPE_STORAGE_BUFFER_DYNAMIC, 1000},
{VK_DESCRIPTOR_TYPE_INPUT_ATTACHMENT, 1000}};
VkDescriptorPoolCreateInfo pool_info = {};
pool_info.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO;
pool_info.flags = VK_DESCRIPTOR_POOL_CREATE_FREE_DESCRIPTOR_SET_BIT;
pool_info.maxSets = 1000 * IM_ARRAYSIZE(pool_sizes);
pool_info.poolSizeCount = (uint32_t)IM_ARRAYSIZE(pool_sizes);
pool_info.pPoolSizes = pool_sizes;
if (vkCreateDescriptorPool(m_Device, &pool_info, nullptr, &m_ImGuiDescriptorPool) != VK_SUCCESS)
throw std::runtime_error("Create DescriptorPool for m_ImGuiDescriptorPool failed!");
}
// Create RenderPass for m_ImGuiRenderPass
{
createImGuiRenderPass();
}
// Create CommandPool for m_ImGuiCommandPool
{
createCommandPool(&m_ImGuiCommandPool);
}
// Create CommandBuffers for m_ImGuiCommandBuffers
{
createImGuiCommandBuffers();
}
{
createImGuiFramebuffers();
}
ImGui_ImplGlfw_InitForVulkan(m_Window, true);
ImGui_ImplVulkan_InitInfo init_info = {};
init_info.Instance = m_Instance;
init_info.PhysicalDevice = m_PhysicalDevice;
init_info.Device = m_Device;
init_info.QueueFamily = m_QueueFamilyIndices.graphicsFamily.value();
init_info.Queue = m_GraphicsQueue;
init_info.PipelineCache = VK_NULL_HANDLE;
init_info.DescriptorPool = m_ImGuiDescriptorPool;
init_info.Subpass = 0;
init_info.MinImageCount = m_ImageCount;
init_info.ImageCount = m_ImageCount;
init_info.MSAASamples = VK_SAMPLE_COUNT_1_BIT;
init_info.Allocator = nullptr;
init_info.CheckVkResultFn = nullptr;
ImGui_ImplVulkan_Init(&init_info, m_ImGuiRenderPass);
// Upload Fonts
{
VkCommandBuffer commandBuffer = beginSingleTimeCommands(m_ImGuiCommandPool);
ImGui_ImplVulkan_CreateFontsTexture(commandBuffer);
endSingleTimeCommands(commandBuffer, m_ImGuiCommandPool);
ImGui_ImplVulkan_DestroyFontUploadObjects();
}
// m_Dset = ImGui_ImplVulkan_AddTexture(m_TextureSampler, m_TextureImageView, VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL);
}
void drawFrame()
{
vkWaitForFences(m_Device, 1, &m_InFlightFences[currentFrame], VK_TRUE, UINT64_MAX);
uint32_t imageIndex;
VkResult result = vkAcquireNextImageKHR(m_Device, m_SwapChain, UINT64_MAX, m_ImageAvailableSemaphores[currentFrame], VK_NULL_HANDLE, &imageIndex);
if (result == VK_ERROR_OUT_OF_DATE_KHR)
{
recreateSwapChain();
return;
}
else if (result != VK_SUCCESS && result != VK_SUBOPTIMAL_KHR)
{
throw std::runtime_error("failed to acquire swap chain image!");
}
// Only reset the fence if we are submitting work
vkResetFences(m_Device, 1, &m_InFlightFences[currentFrame]);
// vkResetCommandBuffer(m_CommandBuffers[currentFrame], 0);
recordCommandBuffer(m_CommandBuffers[currentFrame], imageIndex, m_DescriptorSets[currentFrame]);
{
// vkResetCommandPool(m_Device, m_ImGuiCommandPool, 0);
VkCommandBufferBeginInfo info = {};
info.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;
info.flags |= VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT;
vkBeginCommandBuffer(m_ImGuiCommandBuffers[currentFrame], &info);
VkRenderPassBeginInfo renderPassInfo = {};
renderPassInfo.sType = VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO;
renderPassInfo.renderPass = m_ImGuiRenderPass;
renderPassInfo.framebuffer = m_ImGuiFramebuffers[imageIndex];
renderPassInfo.renderArea.offset = {0, 0};
renderPassInfo.renderArea.extent = m_SwapChainExtent;
VkClearValue clearColor = {0.0f, 0.0f, 0.0f, 1.0f};
renderPassInfo.clearValueCount = 1;
renderPassInfo.pClearValues = &clearColor;
vkCmdBeginRenderPass(m_ImGuiCommandBuffers[currentFrame], &renderPassInfo, VK_SUBPASS_CONTENTS_INLINE);
// Record dear imgui primitives into command buffer
ImGui_ImplVulkan_RenderDrawData(ImGui::GetDrawData(), m_ImGuiCommandBuffers[currentFrame]);
vkCmdEndRenderPass(m_ImGuiCommandBuffers[currentFrame]);
vkEndCommandBuffer(m_ImGuiCommandBuffers[currentFrame]);
}
updateUniformBuffer(currentFrame);
VkSubmitInfo submitInfo{};
submitInfo.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO;
std::array<VkCommandBuffer, 2> submitCommandBuffers =
{m_CommandBuffers[currentFrame], m_ImGuiCommandBuffers[currentFrame]};
VkSemaphore waitSemaphores[] = {m_ImageAvailableSemaphores[currentFrame]};
VkPipelineStageFlags waitStages[] = {VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT};
submitInfo.waitSemaphoreCount = 1;
submitInfo.pWaitSemaphores = waitSemaphores;
submitInfo.pWaitDstStageMask = waitStages;
submitInfo.commandBufferCount = static_cast<uint32_t>(submitCommandBuffers.size());
submitInfo.pCommandBuffers = submitCommandBuffers.data();
VkSemaphore signalSemaphores[] = {m_RenderFinishedSemaphores[currentFrame]};
submitInfo.signalSemaphoreCount = 1;
submitInfo.pSignalSemaphores = signalSemaphores;
if (vkQueueSubmit(m_GraphicsQueue, 1, &submitInfo, m_InFlightFences[currentFrame]) != VK_SUCCESS)
{
throw std::runtime_error("failed to submit draw command buffer!");
}
VkPresentInfoKHR presentInfo{};
presentInfo.sType = VK_STRUCTURE_TYPE_PRESENT_INFO_KHR;
presentInfo.waitSemaphoreCount = 1;
presentInfo.pWaitSemaphores = signalSemaphores;
VkSwapchainKHR swapChains[] = {m_SwapChain};
presentInfo.swapchainCount = 1;
presentInfo.pSwapchains = swapChains;
presentInfo.pImageIndices = &imageIndex;
presentInfo.pResults = nullptr; // Optional
result = vkQueuePresentKHR(m_PresentQueue, &presentInfo);
if (result == VK_ERROR_OUT_OF_DATE_KHR || result == VK_SUBOPTIMAL_KHR || m_FramebufferResized)
{
m_FramebufferResized = false;
recreateSwapChain();
}
else if (result != VK_SUCCESS)
{
throw std::runtime_error("failed to present swap chain image!");
}
currentFrame = (currentFrame + 1) % MAX_FRAMES_IN_FLIGHT;
}
void recordCommandBuffer(const VkCommandBuffer &commandBuffer, const uint32_t &imageIndex, const VkDescriptorSet &descriptorSet)
{
VkCommandBufferBeginInfo beginInfo{};
beginInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;
// beginInfo.flags = 0; // Optional
// beginInfo.pInheritanceInfo = nullptr; // Optional
if (vkBeginCommandBuffer(commandBuffer, &beginInfo) != VK_SUCCESS)
{
throw std::runtime_error("failed to begin recording command buffer!");
}
VkRenderPassBeginInfo renderPassInfo{};
renderPassInfo.sType = VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO;
renderPassInfo.renderPass = m_RenderPass;
renderPassInfo.framebuffer = m_SwapChainFramebuffers[imageIndex];
renderPassInfo.renderArea.offset = {0, 0};
renderPassInfo.renderArea.extent = m_SwapChainExtent;
std::array<VkClearValue, 2> clearValues{};
clearValues[0].color = {{0.0f, 0.0f, 0.0f, 1.0f}};
clearValues[1].depthStencil = {1.0f, 0};
renderPassInfo.clearValueCount = static_cast<uint32_t>(clearValues.size());
renderPassInfo.pClearValues = clearValues.data();
vkCmdBeginRenderPass(commandBuffer, &renderPassInfo, VK_SUBPASS_CONTENTS_INLINE);
vkCmdBindPipeline(commandBuffer, VK_PIPELINE_BIND_POINT_GRAPHICS, m_GraphicsPipeline);
VkBuffer vertexBuffers[] = {m_VertexBuffer};
VkDeviceSize offsets[] = {0};
vkCmdBindVertexBuffers(commandBuffer, 0, 1, vertexBuffers, offsets);
vkCmdBindIndexBuffer(commandBuffer, m_IndexBuffer, 0, VK_INDEX_TYPE_UINT16);
vkCmdBindDescriptorSets(commandBuffer, VK_PIPELINE_BIND_POINT_GRAPHICS, m_PipelineLayout, 0, 1, &descriptorSet, 0, nullptr);
vkCmdDrawIndexed(commandBuffer, static_cast<uint32_t>(indices.size()), 1, 0, 0, 0);
vkCmdEndRenderPass(commandBuffer);
if (vkEndCommandBuffer(commandBuffer) != VK_SUCCESS)
{
throw std::runtime_error("failed to record command buffer!");
}
}
For me image just not rendering in ImGui::Image and i don't know why. I saw my image in shader inputs, but in shader outputs i had just black color.
I guess in your case you should render your scene to another image (not swapchain image). And it means that you need to use another render pass for it. Use swapchain images only for imgui output.
I think when we createSwapChain(); set
VkSurfaceFormatKHR surfaceFormat;
surfaceFormat.format = VK_FORMAT_B8G8R8A8_UNORM;
surfaceFormat.colorSpace = VK_COLORSPACE_SRGB_NONLINEAR_KHR;
like this to avoid validation error!

Record rtsp stream with ffmpeg in iOS

I've followed iFrameExtractor to successfully stream rtsp in my swift project. In this project, it also has recording function. It basically use avformat_write_header
, av_interleaved_write_frame and av_write_trailer to save the rtsp source into mp4 file.
When I used this project in my device, the rtsp streaming works fine, but recording function will always generate a blank mp4 file with no image and sound.
Could anyone tell me what step that I miss?
I'm using iPhone5 with iOS 9.1 and XCode 7.1.1.
The ffmpeg is 2.8.3 version and followed the compile instruction by CompilationGuide – FFmpeg
Following is the sample code in this project
The function that generate every frame:
-(BOOL)stepFrame {
// AVPacket packet;
int frameFinished=0;
static bool bFirstIFrame=false;
static int64_t vPTS=0, vDTS=0, vAudioPTS=0, vAudioDTS=0;
while(!frameFinished && av_read_frame(pFormatCtx, &packet)>=0) {
// Is this a packet from the video stream?
if(packet.stream_index==videoStream) {
// 20130525 albert.liao modified start
// Initialize a new format context for writing file
if(veVideoRecordState!=eH264RecIdle)
{
switch(veVideoRecordState)
{
case eH264RecInit:
{
if ( !pFormatCtx_Record )
{
int bFlag = 0;
//NSString *videoPath = [NSHomeDirectory() stringByAppendingPathComponent:#"Documents/test.mp4"];
NSString *videoPath = #"/Users/liaokuohsun/iFrameTest.mp4";
const char *file = [videoPath UTF8String];
pFormatCtx_Record = avformat_alloc_context();
bFlag = h264_file_create(file, pFormatCtx_Record, pCodecCtx, pAudioCodecCtx,/*fps*/0.0, packet.data, packet.size );
if(bFlag==true)
{
veVideoRecordState = eH264RecActive;
fprintf(stderr, "h264_file_create success\n");
}
else
{
veVideoRecordState = eH264RecIdle;
fprintf(stderr, "h264_file_create error\n");
}
}
}
//break;
case eH264RecActive:
{
if((bFirstIFrame==false) &&(packet.flags&AV_PKT_FLAG_KEY)==AV_PKT_FLAG_KEY)
{
bFirstIFrame=true;
vPTS = packet.pts ;
vDTS = packet.dts ;
#if 0
NSRunLoop *pRunLoop = [NSRunLoop currentRunLoop];
[pRunLoop addTimer:RecordingTimer forMode:NSDefaultRunLoopMode];
#else
[NSTimer scheduledTimerWithTimeInterval:5.0//2.0
target:self
selector:#selector(StopRecording:)
userInfo:nil
repeats:NO];
#endif
}
// Record audio when 1st i-Frame is obtained
if(bFirstIFrame==true)
{
if ( pFormatCtx_Record )
{
#if PTS_DTS_IS_CORRECT==1
packet.pts = packet.pts - vPTS;
packet.dts = packet.dts - vDTS;
#endif
h264_file_write_frame( pFormatCtx_Record, packet.stream_index, packet.data, packet.size, packet.dts, packet.pts);
}
else
{
NSLog(#"pFormatCtx_Record no exist");
}
}
}
break;
case eH264RecClose:
{
if ( pFormatCtx_Record )
{
h264_file_close(pFormatCtx_Record);
#if 0
// 20130607 Test
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^(void)
{
ALAssetsLibrary *library = [[ALAssetsLibrary alloc]init];
NSString *filePathString = [NSHomeDirectory() stringByAppendingPathComponent:#"Documents/test.mp4"];
NSURL *filePathURL = [NSURL fileURLWithPath:filePathString isDirectory:NO];
if(1)// ([library videoAtPathIsCompatibleWithSavedPhotosAlbum:filePathURL])
{
[library writeVideoAtPathToSavedPhotosAlbum:filePathURL completionBlock:^(NSURL *assetURL, NSError *error){
if (error) {
// TODO: error handling
NSLog(#"writeVideoAtPathToSavedPhotosAlbum error");
} else {
// TODO: success handling
NSLog(#"writeVideoAtPathToSavedPhotosAlbum success");
}
}];
}
[library release];
});
#endif
vPTS = 0;
vDTS = 0;
vAudioPTS = 0;
vAudioDTS = 0;
pFormatCtx_Record = NULL;
NSLog(#"h264_file_close() is finished");
}
else
{
NSLog(#"fc no exist");
}
bFirstIFrame = false;
veVideoRecordState = eH264RecIdle;
}
break;
default:
if ( pFormatCtx_Record )
{
h264_file_close(pFormatCtx_Record);
pFormatCtx_Record = NULL;
}
NSLog(#"[ERROR] unexpected veVideoRecordState!!");
veVideoRecordState = eH264RecIdle;
break;
}
}
// Decode video frame
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
}
else if(packet.stream_index==audioStream)
{
// 20131024 albert.liao modfied start
static int vPktCount=0;
BOOL bIsAACADTS = FALSE;
int ret = 0;
if(aPlayer.vAACType == eAAC_UNDEFINED)
{
tAACADTSHeaderInfo vxAACADTSHeaderInfo = {0};
bIsAACADTS = [AudioUtilities parseAACADTSHeader:(uint8_t *)packet.data ToHeader:&vxAACADTSHeaderInfo];
}
#synchronized(aPlayer)
{
if(aPlayer==nil)
{
aPlayer = [[AudioPlayer alloc]initAudio:nil withCodecCtx:(AVCodecContext *) pAudioCodecCtx];
NSLog(#"aPlayer initAudio");
if(bIsAACADTS)
{
aPlayer.vAACType = eAAC_ADTS;
//NSLog(#"is ADTS AAC");
}
}
else
{
if(vPktCount<5) // The voice is listened once image is rendered
{
vPktCount++;
}
else
{
if([aPlayer getStatus]!=eAudioRunning)
{
dispatch_async(dispatch_get_main_queue(), ^(void) {
#synchronized(aPlayer)
{
NSLog(#"aPlayer start play");
[aPlayer Play];
}
});
}
}
}
};
#synchronized(aPlayer)
{
int ret = 0;
ret = [aPlayer putAVPacket:&packet];
if(ret <= 0)
NSLog(#"Put Audio Packet Error!!");
}
// 20131024 albert.liao modfied end
if(bFirstIFrame==true)
{
switch(veVideoRecordState)
{
case eH264RecActive:
{
if ( pFormatCtx_Record )
{
h264_file_write_audio_frame(pFormatCtx_Record, pAudioCodecCtx, packet.stream_index, packet.data, packet.size, packet.dts, packet.pts);
}
else
{
NSLog(#"pFormatCtx_Record no exist");
}
}
}
}
}
else
{
//fprintf(stderr, "packet len=%d, Byte=%02X%02X%02X%02X%02X\n",\
packet.size, packet.data[0],packet.data[1],packet.data[2],packet.data[3], packet.data[4]);
}
// 20130525 albert.liao modified end
}
return frameFinished!=0;
}
avformat_write_header:
int h264_file_create(const char *pFilePath, AVFormatContext *fc, AVCodecContext *pCodecCtx,AVCodecContext *pAudioCodecCtx, double fps, void *p, int len )
{
int vRet=0;
AVOutputFormat *of=NULL;
AVStream *pst=NULL;
AVCodecContext *pcc=NULL, *pAudioOutputCodecContext=NULL;
avcodec_register_all();
av_register_all();
av_log_set_level(AV_LOG_VERBOSE);
if(!pFilePath)
{
fprintf(stderr, "FilePath no exist");
return -1;
}
if(!fc)
{
fprintf(stderr, "AVFormatContext no exist");
return -1;
}
fprintf(stderr, "file=%s\n",pFilePath);
// Create container
of = av_guess_format( 0, pFilePath, 0 );
fc->oformat = of;
strcpy( fc->filename, pFilePath );
// Add video stream
pst = avformat_new_stream( fc, 0 );
vVideoStreamIdx = pst->index;
fprintf(stderr,"Video Stream:%d",vVideoStreamIdx);
pcc = pst->codec;
avcodec_get_context_defaults3( pcc, AVMEDIA_TYPE_VIDEO );
// Save the stream as origin setting without convert
pcc->codec_type = pCodecCtx->codec_type;
pcc->codec_id = pCodecCtx->codec_id;
pcc->bit_rate = pCodecCtx->bit_rate;
pcc->width = pCodecCtx->width;
pcc->height = pCodecCtx->height;
if(fps==0)
{
double fps=0.0;
AVRational pTimeBase;
pTimeBase.num = pCodecCtx->time_base.num;
pTimeBase.den = pCodecCtx->time_base.den;
fps = 1.0/ av_q2d(pCodecCtx->time_base)/ FFMAX(pCodecCtx->ticks_per_frame, 1);
fprintf(stderr,"fps_method(tbc): 1/av_q2d()=%g",fps);
pcc->time_base.num = 1;
pcc->time_base.den = fps;
}
else
{
pcc->time_base.num = 1;
pcc->time_base.den = fps;
}
// reference ffmpeg\libavformat\utils.c
// For SPS and PPS in avcC container
pcc->extradata = malloc(sizeof(uint8_t)*pCodecCtx->extradata_size);
memcpy(pcc->extradata, pCodecCtx->extradata, pCodecCtx->extradata_size);
pcc->extradata_size = pCodecCtx->extradata_size;
// For Audio stream
if(pAudioCodecCtx)
{
AVCodec *pAudioCodec=NULL;
AVStream *pst2=NULL;
pAudioCodec = avcodec_find_encoder(AV_CODEC_ID_AAC);
// Add audio stream
pst2 = avformat_new_stream( fc, pAudioCodec );
vAudioStreamIdx = pst2->index;
pAudioOutputCodecContext = pst2->codec;
avcodec_get_context_defaults3( pAudioOutputCodecContext, pAudioCodec );
fprintf(stderr,"Audio Stream:%d",vAudioStreamIdx);
fprintf(stderr,"pAudioCodecCtx->bits_per_coded_sample=%d",pAudioCodecCtx->bits_per_coded_sample);
pAudioOutputCodecContext->codec_type = AVMEDIA_TYPE_AUDIO;
pAudioOutputCodecContext->codec_id = AV_CODEC_ID_AAC;
// Copy the codec attributes
pAudioOutputCodecContext->channels = pAudioCodecCtx->channels;
pAudioOutputCodecContext->channel_layout = pAudioCodecCtx->channel_layout;
pAudioOutputCodecContext->sample_rate = pAudioCodecCtx->sample_rate;
pAudioOutputCodecContext->bit_rate = 12000;//pAudioCodecCtx->sample_rate * pAudioCodecCtx->bits_per_coded_sample;
pAudioOutputCodecContext->bits_per_coded_sample = pAudioCodecCtx->bits_per_coded_sample;
pAudioOutputCodecContext->profile = pAudioCodecCtx->profile;
//FF_PROFILE_AAC_LOW;
// pAudioCodecCtx->bit_rate;
// AV_SAMPLE_FMT_U8P, AV_SAMPLE_FMT_S16P
//pAudioOutputCodecContext->sample_fmt = AV_SAMPLE_FMT_FLTP;//pAudioCodecCtx->sample_fmt;
pAudioOutputCodecContext->sample_fmt = pAudioCodecCtx->sample_fmt;
//pAudioOutputCodecContext->sample_fmt = AV_SAMPLE_FMT_U8;
pAudioOutputCodecContext->sample_aspect_ratio = pAudioCodecCtx->sample_aspect_ratio;
pAudioOutputCodecContext->time_base.num = pAudioCodecCtx->time_base.num;
pAudioOutputCodecContext->time_base.den = pAudioCodecCtx->time_base.den;
pAudioOutputCodecContext->ticks_per_frame = pAudioCodecCtx->ticks_per_frame;
pAudioOutputCodecContext->frame_size = 1024;
fprintf(stderr,"profile:%d, sample_rate:%d, channles:%d", pAudioOutputCodecContext->profile, pAudioOutputCodecContext->sample_rate, pAudioOutputCodecContext->channels);
AVDictionary *opts = NULL;
av_dict_set(&opts, "strict", "experimental", 0);
if (avcodec_open2(pAudioOutputCodecContext, pAudioCodec, &opts) < 0) {
fprintf(stderr, "\ncould not open codec\n");
}
av_dict_free(&opts);
#if 0
// For Audio, this part is no need
if(pAudioCodecCtx->extradata_size!=0)
{
NSLog(#"extradata_size !=0");
pAudioOutputCodecContext->extradata = malloc(sizeof(uint8_t)*pAudioCodecCtx->extradata_size);
memcpy(pAudioOutputCodecContext->extradata, pAudioCodecCtx->extradata, pAudioCodecCtx->extradata_size);
pAudioOutputCodecContext->extradata_size = pAudioCodecCtx->extradata_size;
}
else
{
// For WMA test only
pAudioOutputCodecContext->extradata_size = 0;
NSLog(#"extradata_size ==0");
}
#endif
}
if(fc->oformat->flags & AVFMT_GLOBALHEADER)
{
pcc->flags |= CODEC_FLAG_GLOBAL_HEADER;
pAudioOutputCodecContext->flags |= CODEC_FLAG_GLOBAL_HEADER;
}
if ( !( fc->oformat->flags & AVFMT_NOFILE ) )
{
vRet = avio_open( &fc->pb, fc->filename, AVIO_FLAG_WRITE );
if(vRet!=0)
{
fprintf(stderr,"avio_open(%s) error", fc->filename);
}
}
// dump format in console
av_dump_format(fc, 0, pFilePath, 1);
vRet = avformat_write_header( fc, NULL );
if(vRet==0)
return 1;
else
return 0;
}
av_interleaved_write_frame:
void h264_file_write_frame(AVFormatContext *fc, int vStreamIdx, const void* p, int len, int64_t dts, int64_t pts )
{
AVStream *pst = NULL;
AVPacket pkt;
if ( 0 > vVideoStreamIdx )
return;
// may be audio or video
pst = fc->streams[ vStreamIdx ];
// Init packet
av_init_packet( &pkt );
if(vStreamIdx ==vVideoStreamIdx)
{
pkt.flags |= ( 0 >= getVopType( p, len ) ) ? AV_PKT_FLAG_KEY : 0;
//pkt.flags |= AV_PKT_FLAG_KEY;
pkt.stream_index = pst->index;
pkt.data = (uint8_t*)p;
pkt.size = len;
pkt.dts = AV_NOPTS_VALUE;
pkt.pts = AV_NOPTS_VALUE;
// TODO: mark or unmark the log
//fprintf(stderr, "dts=%lld, pts=%lld\n",dts,pts);
// av_write_frame( fc, &pkt );
}
av_interleaved_write_frame( fc, &pkt );
}
av_write_trailer:
void h264_file_close(AVFormatContext *fc)
{
if ( !fc )
return;
av_write_trailer( fc );
if ( fc->oformat && !( fc->oformat->flags & AVFMT_NOFILE ) && fc->pb )
avio_close( fc->pb );
av_free( fc );
}
Thanks.
It looks like you're using the same AVFormatContext for both the input and output?
In the line
pst = fc->streams[ vStreamIdx ];
You're assigning the AVStream* from your AVFormatContext connected with your input (RTSP stream). But then later on you're trying to write the packet back to the same context av_interleaved_write_frame( fc, &pkt );. I kind of think of a context as a file which has helped me navagate this type of thing better. I do something identicial to what you're doing (not iOS though) where I use a separate AVFormatContext for each of the input (RTSP stream) and output (mp4 file). If I'm correct, I think what you just need to do is initialize an AVFormatContext and properly.
The following code (without error checking everything) is what I do to take an AVFormatContext * output_format_context = NULL and the AVFormatContext * input_format_context that I had associated with the RTSP stream and write from one to the other. This is after I have fetched a packet, etc., which in your case it looks like you're populating (I just take the packet from av_read_frame and re-package it.
This is code that could be in your write frame function (but it also does include the writing of the header).
AVFormatContext * output_format_context;
AVStream * in_stream_2;
AVStream * out_stream_2;
// Allocate the context with the output file
avformat_alloc_output_context2(&output_format_context, NULL, NULL, out_filename.c_str());
// Point to AVOutputFormat * output_format for manipulation
output_format = output_format_context->oformat;
// Loop through all streams
for (i = 0; i < input_format_context->nb_streams; i++) {
// Create a pointer to the input stream that was allocated earlier in the code
AVStream *in_stream = input_format_context->streams[i];
// Create a pointer to a new stream that will be part of the output
AVStream *out_stream = avformat_new_stream(output_format_context, in_stream->codec->codec);
// Set time_base of the new output stream to equal the input stream one since I'm not changing anything (can avoid but get a deprecation warning)
out_stream->time_base = in_stream->time_base;
// This is the non-deprecated way of copying all the parameters from the input stream into the output stream since everything stays the same
avcodec_parameters_from_context(out_stream->codecpar, in_stream->codec);
// I don't remember what this is for :)
out_stream->codec->codec_tag = 0;
// This just sets a flag from the format context to the stream relating to the header
if (output_format_context->oformat->flags & AVFMT_GLOBALHEADER)
out_stream->codec->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
}
// Check NOFILE flag and open the output file context (previously the output file was associated with the format context, now it is actually opened.
if (!(output_format->flags & AVFMT_NOFILE))
avio_open(&output_format_context->pb, out_filename.c_str(), AVIO_FLAG_WRITE);
// Write the header (not sure if this is always needed but h264 I believe it is.
avformat_write_header(output_format_context,NULL);
// Re-getting the appropriate stream that was populated above (this should allow for both audio/video)
in_stream_2 = input_format_context->streams[packet.stream_index];
out_stream_2 = output_format_context->streams[packet.stream_index];
// Rescaling pts and dts, duration and pos - you would do as you need in your code.
packet.pts = av_rescale_q_rnd(packet.pts, in_stream_2->time_base, out_stream_2->time_base, (AVRounding) (AV_ROUND_NEAR_INF | AV_ROUND_PASS_MINMAX));
packet.dts = av_rescale_q_rnd(packet.dts, in_stream_2->time_base, out_stream_2->time_base, (AVRounding) (AV_ROUND_NEAR_INF | AV_ROUND_PASS_MINMAX));
packet.duration = av_rescale_q(packet.duration, in_stream_2->time_base, out_stream_2->time_base);
packet.pos = -1;
// The first packet of my stream always gives me negative dts/pts so this just protects that first one for my purposes. You probably don't need.
if (packet.dts < 0) packet.dts = 0;
if (packet.pts < 0) packet.pts = 0;
// Finally write the frame
av_interleaved_write_frame(output_format_context, &packet);
// ....
// Write header, close/cleanup... etc
// ....
This code is fairly bare bones and doesn't include the setup (which it sounds like you're doing correctly anyway). I would also imagine this code could be cleaned up and tweaked for your purposes, but this works for me to re-write the RTSP stream into a file (in my case many files but code not shown).
The code is C code, so you might need to do minor tweaks for making it Swift compatible (for some of the library function calls maybe). I think overall it should be compatible though.
Hopefully this helps point you to the right direction. This was cobbled together thanks to several sample code sources (I don't remember where), along with warning prompts from the libraries themselves.

Converting MKPolygon Objective-C code to Swift

I have the following Objective-C method:
- (void)addPolygonToMap {
NSInteger numberOfPoints = [self.coordinates count];
if (numberOfPoints > 4) {
CLLocationCoordinate2D points[numberOfPoints];
for (NSInteger i = 0; i < numberOfPoints; i++) {
points[i] = [self.coordinates[i] MKCoordinateValue];
}
self.polygon = [MKPolygon polygonWithCoordinates:points count:numberOfPoints];
[self.mapView addOverlay: self.polygon];
}
self.isDrawingPolygon = NO;
[self.drawPolygonButton setTitle:#"draw" forState:UIControlStateNormal];
self.canvasView.image = nil;
[self.canvasView removeFromSuperview];
}
My attempt at converting it to Swift:
func addPolygonToMap() {
var numberOfPoints: NSInteger = self.coordinates.count
if (numberOfPoints > 4) {
var points: [CLLocationCoordinate2D] = []
var coordsPointer = UnsafeMutablePointer<CLLocationCoordinate2D>.alloc(numberOfPoints)
for i in 0..<numberOfPoints {
points.append(coordsPointer[i])
}
self.polygon = MKPolygon(coordinates: &points, count: numberOfPoints)
self.mapView.addOverlay(self.polygon)
coordsPointer.dealloc(numberOfPoints)
}
self.isDrawingPolygon = false
self.drawPolygonButton.setTitle("Draw", forState: .Normal)
self.canvasView.image = nil
self.canvasView.removeFromSuperview()
}
Finally, when the delegate method is called it's not actually adding the overlay to the mapView. I can't see anything.
I'm assuming it's my self.addPolytonToMap() method, but I'm not 100% sure. The whole scenario works fine in my Objective-C project.
func mapView(mapView: MKMapView!, rendererForOverlay overlay: MKOverlay!) -> MKOverlayRenderer! {
if (overlay is MKPolygon) {
var overlayPathView = MKPolygonRenderer(overlay: overlay)
overlayPathView.fillColor = UIColor.cyanColor().colorWithAlphaComponent(0.2)
overlayPathView.strokeColor = UIColor.cyanColor().colorWithAlphaComponent(0.2)
overlayPathView.lineWidth = 5
return overlayPathView
} else if (overlay is MKPolyline) {
var overlayPathView = MKPolylineRenderer(overlay: overlay)
overlayPathView.strokeColor = UIColor.blueColor().colorWithAlphaComponent(0.7)
overlayPathView.lineWidth = 5
return overlayPathView
}
return nil
}
UPDATE:
I just noticed that in my Objective-C version, the points[i].latitude are coming through okay, however when I do the following I get a strange output:
println(coordsPointer[i].latitude)
Output:
1.63041663127611e-321
1.64523860065135e-321
1.65511991356818e-321
1.68970450877706e-321
1.7045264781523e-321
1.72922976044436e-321
This would explain why I don't see the overlay, however my experience with UnsafeMutablePointer<> is limited.
Fixed by modifying the addPolygonToMap() method:
func addPolygonToMap() {
var numberOfPoints: NSInteger = self.coordinates.count
if (numberOfPoints > 4) {
var points: [CLLocationCoordinate2D] = []
for i in 0..<numberOfPoints {
points.insert(self.coordinates[i].MKCoordinateValue, atIndex: i)
}
self.polygon = MKPolygon(coordinates: &points, count: numberOfPoints)
self.mapView.addOverlay(self.polygon)
}
self.isDrawingPolygon = false
self.drawPolygonButton.setTitle("Draw", forState: .Normal)
self.canvasView.image = nil
self.canvasView.removeFromSuperview()
}
Thanks to #Volker for the help.

sizeof crashing on iPad 1 but not > iPad 1

Consider this code:
CGFloat largerLineSpacing = kStreamCellParagraphSpacing;
CTParagraphStyleSetting paragraphSettings[1] = {
{ kCTParagraphStyleSpecifierParagraphSpacing, sizeof(CGFloat), &largerLineSpacing }
};
CTParagraphStyleRef paragraphStyle = CTParagraphStyleCreate(paragraphSettings, sizeof(*paragraphSettings));
This code crashes with an EXC_BAD_ACCESS when running on an iPad 1 (5.1), but not a 5.1 simulator or an iPad 3 (6.0). My C is weak - am I making a dumb mistake with sizeof?
The docs for CTParagraphStyleCreate suggest that its second argument gives the number of CTParagraphStyleSetting instances in the paragraphSettings array (1 in your case) rather than the size in bytes of the array.
If you change your code to
CTParagraphStyleRef paragraphStyle = CTParagraphStyleCreate(paragraphSettings, 1);
it should work. Or, if you want to cope with adding more settings in future, you could try
int numElems = sizeof(paragraphSettings)/sizeof(paragraphSettings[0]);
CTParagraphStyleRef paragraphStyle = CTParagraphStyleCreate(paragraphSettings,
numElems);
static CFIndex const settingCount = 1;
CTParagraphStyleSetting paragraphSettings[settingCount] = {
{ kCTParagraphStyleSpecifierParagraphSpacing, sizeof(CGFloat), &largerLineSpacing }
};
CTParagraphStyleRef paragraphStyle = CTParagraphStyleCreate(paragraphSettings, settingCount);

Example of using Audio Queue Services

I am seeking an example of using Audio Queue Services.
I would like to create a sound using a mathematical equation and then hear it.
Here's my code for generating sound from a function. I'm assuming you know how to use AudioQueue services, set up an AudioSession, and properly start and stop an audio output queue.
Here's a snippet for setting up and starting an output AudioQueue:
// Get the preferred sample rate (8,000 Hz on iPhone, 44,100 Hz on iPod touch)
size = sizeof(sampleRate);
err = AudioSessionGetProperty (kAudioSessionProperty_CurrentHardwareSampleRate, &size, &sampleRate);
if (err != noErr) NSLog(#"AudioSessionGetProperty(kAudioSessionProperty_CurrentHardwareSampleRate) error: %d", err);
//NSLog (#"Current hardware sample rate: %1.0f", sampleRate);
BOOL isHighSampleRate = (sampleRate > 16000);
int bufferByteSize;
AudioQueueBufferRef buffer;
// Set up stream format fields
AudioStreamBasicDescription streamFormat;
streamFormat.mSampleRate = sampleRate;
streamFormat.mFormatID = kAudioFormatLinearPCM;
streamFormat.mFormatFlags = kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked;
streamFormat.mBitsPerChannel = 16;
streamFormat.mChannelsPerFrame = 1;
streamFormat.mBytesPerPacket = 2 * streamFormat.mChannelsPerFrame;
streamFormat.mBytesPerFrame = 2 * streamFormat.mChannelsPerFrame;
streamFormat.mFramesPerPacket = 1;
streamFormat.mReserved = 0;
// New output queue ---- PLAYBACK ----
if (isPlaying == NO) {
err = AudioQueueNewOutput (&streamFormat, AudioEngineOutputBufferCallback, self, nil, nil, 0, &outputQueue);
if (err != noErr) NSLog(#"AudioQueueNewOutput() error: %d", err);
// Enqueue buffers
//outputFrequency = 0.0;
outputBuffersToRewrite = 3;
bufferByteSize = (sampleRate > 16000)? 2176 : 512; // 40.5 Hz : 31.25 Hz
for (i=0; i<3; i++) {
err = AudioQueueAllocateBuffer (outputQueue, bufferByteSize, &buffer);
if (err == noErr) {
[self generateTone: buffer];
err = AudioQueueEnqueueBuffer (outputQueue, buffer, 0, nil);
if (err != noErr) NSLog(#"AudioQueueEnqueueBuffer() error: %d", err);
} else {
NSLog(#"AudioQueueAllocateBuffer() error: %d", err);
return;
}
}
// Start playback
isPlaying = YES;
err = AudioQueueStart(outputQueue, nil);
if (err != noErr) { NSLog(#"AudioQueueStart() error: %d", err); isPlaying= NO; return; }
} else {
NSLog (#"Error: audio is already playing back.");
}
Here's the part that generates the tone:
// AudioQueue output queue callback.
void AudioEngineOutputBufferCallback (void *inUserData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer) {
AudioEngine *engine = (AudioEngine*) inUserData;
[engine processOutputBuffer:inBuffer queue:inAQ];
}
- (void) processOutputBuffer: (AudioQueueBufferRef) buffer queue:(AudioQueueRef) queue {
OSStatus err;
if (isPlaying == YES) {
[outputLock lock];
if (outputBuffersToRewrite > 0) {
outputBuffersToRewrite--;
[self generateTone:buffer];
}
err = AudioQueueEnqueueBuffer(queue, buffer, 0, NULL);
if (err == 560030580) { // Queue is not active due to Music being started or other reasons
isPlaying = NO;
} else if (err != noErr) {
NSLog(#"AudioQueueEnqueueBuffer() error %d", err);
}
[outputLock unlock];
} else {
err = AudioQueueStop (queue, NO);
if (err != noErr) NSLog(#"AudioQueueStop() error: %d", err);
}
}
-(void) generateTone: (AudioQueueBufferRef) buffer {
if (outputFrequency == 0.0) {
memset(buffer->mAudioData, 0, buffer->mAudioDataBytesCapacity);
buffer->mAudioDataByteSize = buffer->mAudioDataBytesCapacity;
} else {
// Make the buffer length a multiple of the wavelength for the output frequency.
int sampleCount = buffer->mAudioDataBytesCapacity / sizeof (SInt16);
double bufferLength = sampleCount;
double wavelength = sampleRate / outputFrequency;
double repetitions = floor (bufferLength / wavelength);
if (repetitions > 0.0) {
sampleCount = round (wavelength * repetitions);
}
double x, y;
double sd = 1.0 / sampleRate;
double amp = 0.9;
double max16bit = SHRT_MAX;
int i;
SInt16 *p = buffer->mAudioData;
for (i = 0; i < sampleCount; i++) {
x = i * sd * outputFrequency;
switch (outputWaveform) {
case kSine:
y = sin (x * 2.0 * M_PI);
break;
case kTriangle:
x = fmod (x, 1.0);
if (x < 0.25)
y = x * 4.0; // up 0.0 to 1.0
else if (x < 0.75)
y = (1.0 - x) * 4.0 - 2.0; // down 1.0 to -1.0
else
y = (x - 1.0) * 4.0; // up -1.0 to 0.0
break;
case kSawtooth:
y = 0.8 - fmod (x, 1.0) * 1.8;
break;
case kSquare:
y = (fmod(x, 1.0) < 0.5)? 0.7: -0.7;
break;
default: y = 0; break;
}
p[i] = y * max16bit * amp;
}
buffer->mAudioDataByteSize = sampleCount * sizeof (SInt16);
}
}
Something to watch out for is that your callback will be called on a non-main thread, so you have to practice thread safety with locks, mutexs, or other techniques.
This is a version using C# of the same sample from #lucius
void SetupAudio ()
{
AudioSession.Initialize ();
AudioSession.Category = AudioSessionCategory.MediaPlayback;
sampleRate = AudioSession.CurrentHardwareSampleRate;
var format = new AudioStreamBasicDescription () {
SampleRate = sampleRate,
Format = AudioFormatType.LinearPCM,
FormatFlags = AudioFormatFlags.LinearPCMIsSignedInteger | AudioFormatFlags.LinearPCMIsPacked,
BitsPerChannel = 16,
ChannelsPerFrame = 1,
BytesPerFrame = 2,
BytesPerPacket = 2,
FramesPerPacket = 1,
};
var queue = new OutputAudioQueue (format);
var bufferByteSize = (sampleRate > 16000)? 2176 : 512; // 40.5 Hz : 31.25 Hz
var buffers = new AudioQueueBuffer* [numBuffers];
for (int i = 0; i < numBuffers; i++){
queue.AllocateBuffer (bufferByteSize, out buffers [i]);
GenerateTone (buffers [i]);
queue.EnqueueBuffer (buffers [i], null);
}
queue.OutputCompleted += (object sender, OutputCompletedEventArgs e) => {
queue.EnqueueBuffer (e.UnsafeBuffer, null);
};
queue.Start ();
return true;
}
This is the tone generator:
void GenerateTone (AudioQueueBuffer *buffer)
{
// Make the buffer length a multiple of the wavelength for the output frequency.
uint sampleCount = buffer->AudioDataBytesCapacity / 2;
double bufferLength = sampleCount;
double wavelength = sampleRate / outputFrequency;
double repetitions = Math.Floor (bufferLength / wavelength);
if (repetitions > 0)
sampleCount = (uint)Math.Round (wavelength * repetitions);
double x, y;
double sd = 1.0 / sampleRate;
double amp = 0.9;
double max16bit = Int16.MaxValue;
int i;
short *p = (short *) buffer->AudioData;
for (i = 0; i < sampleCount; i++) {
x = i * sd * outputFrequency;
switch (outputWaveForm) {
case WaveForm.Sine:
y = Math.Sin (x * 2.0 * Math.PI);
break;
case WaveForm.Triangle:
x = x % 1.0;
if (x < 0.25)
y = x * 4.0; // up 0.0 to 1.0
else if (x < 0.75)
y = (1.0 - x) * 4.0 - 2.0; // down 1.0 to -1.0
else
y = (x - 1.0) * 4.0; // up -1.0 to 0.0
break;
case WaveForm.Sawtooth:
y = 0.8 - (x % 1.0) * 1.8;
break;
case WaveForm.Square:
y = ((x % 1.0) < 0.5)? 0.7: -0.7;
break;
default: y = 0; break;
}
p[i] = (short)(y * max16bit * amp);
}
buffer->AudioDataByteSize = sampleCount * 2;
}
}
You also want these definitions:
enum WaveForm {
Sine, Triangle, Sawtooth, Square
}
WaveForm outputWaveForm;
const float outputFrequency = 220;
High level: use AVAudioPlayer https://github.com/hollance/AVBufferPlayer
Med level: audio queues trailsinthesand.com/exploring-iphone-audio-part-1/ gets you going nicely. NOTE: I removed the http so the old link could be there, but it does direct to a bad site, so it apparently has changed.
Low level: alternatively, you can drop down a level and do it with audio units: http://cocoawithlove.com/2010/10/ios-tone-generator-introduction-to.html