I wanted to understand best practices/approaches for performing blocking CPU operations in production grade server side applications. (Note that I am talking about server side and not Android apps)
What I mean by blocking CPU operation?
Operation which runs on and consumes CPU, for example, huge matrix multiplication, huge data transformation done in while loop etc.
My Setup
We are building kotlin dsl powered by coroutines
We have created our own CoroutineScope with single thread for non blocking operations so that mutating variables from callback are thread safe because all of them runs on same thread
We are recommending users to use withContext(Dispatchers.IO) for doing all the blocking IO operations
Open Questions
We have fixed strategy to do non blocking ops and thread safe mutations as well as doing IO ops as mentioned above
We are exploring options to perform blocking CPU bound operations as mentioned in the beginning
Some of the Solutions
Use Dispatchers.Default for do blocking cpu operations
Does anyone foresee any issue with using Default dispatcher?
What if other part of code/transitive libs are also using Default dispatcher?
Create separate CoroutineScope with fixed threads (usually equals to no. CPU threads) and then ask our users to use that isolated scope for running any blocking CPU bound operations
Which approach we should go with? pros and cons? Is there any other alternative good approach?
Side Note
Coming from Scala, it is usually not preferred to use Global execution context and Dispatchers.Default somehow maps to global context
Related
tl;dr: How can I get a glimpse of basic state for a coroutine dispatcher?
Specifically:
How many total coroutines are active?
Is the dispatcher exceeding its maximum parallelism? (e.g. are there coroutines stuck waiting)
Context:
I'm running a game server where coroutines are used to listen on ports, handle asynchronous work, manage game state, etc. It is very easy to introduce subtle latency bugs where someone adds a new coroutine inside of a request and then the coroutine load substantially increases and request latency suffers (and in rare cases, deadlocks).
Optimal coroutine scheduling is integral to a good user experience for my service and is something I want to measure. Are there any available health metrics I can observe related to coroutines?
I am trying to implement a "penalty box" for unsuccessful attempts to use our API when it is dealing with security tokens. This extends my previous question Would delayElement be susceptible to a DoS attack? which is isn't.
Here's my handler at the moment.
.onErrorResume(
SecurityException.class,
ex -> {
ServerWebExchangeUtils.setResponseStatus(
exchange, HttpStatus.UNAUTHORIZED);
ServerWebExchangeUtils.setAlreadyRouted(exchange);
securityLog.warn("security error obtaining claims: {}", ex.getMessage());
return chain
.filter(exchange)
.delayElement(
Duration.ofMillis(authProperties.getPenaltyDelayInMillis()),
penaltyScheduler
)
.then(
respondWithUnauthorized(config, exchange, "invalid_token"));
})
I have my scheduler as Schedulers.newParallel("penalty") which allows for parallel threading. But I am not sure if parallel would be the better scheduler or should I just use boundedElastic?
i see no reason as to why you would use parallel.
A parallel scheduler guarantees that work will be scheduled over several cores. Running things in parallel is only beneficial if you need raw computing power, for instance if you need to do heavy mathematical calculations.
Using a parallel scheduler also has a setup time as we need to send instructions to the other cpus, they need to allocate threads etc.
What you want is to allocate threads that are going to most likely spend 99% of their time waiting, which means there is no reason whatsoever that you need the multiple cpu core guarantee.
boundedElastic will create a scheduler with an upper bounded limit which will suffice. Async work is more about orchestration, and not raw cpu-power so the framework will handle the threading and optimization for you as efficient as possible.
Thats the reason for using webflux, so we dont need to think about those things.
In Erlang/OTP, I've read how light weight processes, the actor model, and supervisors are important in creating reliable services. How would this compare to OpenResty (master/worker, async IO, embedded Lua)?
I am curious over a general architectural overview on the main concepts to better understand how OpenResty would be used alongside (or instead of) Erlang/OTP.
These two links partially answers the question:
https://github.com/openresty/lua-nginx-module/blob/master/README.markdown
The Lua interpreter or LuaJIT instance is shared across all the
requests in a single nginx worker process but request contexts are
segregated using lightweight Lua coroutines.
Loaded Lua modules persist in the nginx worker process level resulting
in a small memory footprint...
https://github.com/openresty/lua-nginx-module/wiki/Introduction
...for each incoming request, lua-nginx-module creates a coroutine to run user code to process the request, and the coroutine will be destroyed when the request handling process is done. Every coroutine has its own independent global environment, which inherits shared read-only common data.
...lua-nginx-module can handle tens of thousands of concurrent requests with very low memory overhead. According to our tests, the memory overhead for each request in lua-nginx-module is only 2 KB or even half smaller if LuaJIT is used.
I am currently pursuing an undergraduate level course in Operating Systems. I'm somewhat confused about the functions of dispatcher and scheduler in process scheduling. Based on what I've learnt, the medium term scheduler selects the process for swapping out and in , and once the processes are selected, the actual swap operation is performed by Dispatcher by context switching. Also the short term scheduler is responsible for scheduling the processes and allocate them CPU time, based on the scheduling algorithm followed.
Please correct me if I'm wrong. I'm really confused about the functions of medium term scheduler vs dispatcher, and differences between Swapping & context switching.
You describing things in system specific terms.
The scheduler and the dispatcher could be all the same thing. However, the frequently are divided so that the scheduler maintains a queue of processes and the dispatcher handles the actual context switch.
If you divide the scheduler into long term, medium term, and short term, that division (if it exists at all) is specific to the operating system.
Swapping in the process of removing a process from memory. A process can be made non-executable through a context switch but may not be swapped out. Swapping is generally independent of scheduling. However, a process must be swapped in to run and the memory management will try to avoid swapping out executing processes.
A scheduler evaluate the requirement of the request to be serviced and thus imposes ordering.
Basically,whatever you have known about scheduler and dispatcher is correct.Sometimes they are referred to as a same unit or scheduler(short time in this case) contains dispatcher as a single unit and together are responsible for allocating a process to CPU for execution.Sometimes they are referred as two separate units,the scheduler selects a process according to some algorithm and the dispatcher is a software that is responsible for actual context switching.
I searched a variety of sources but don't really understand the difference between using NSThreads and GCD. I'm completely new to the OS X platform so I might be completely misinterpreting this.
From what I read online, GCD seems to do the exact same thing as basic threads (POSIX, NSThreads etc.) while adding much more technical jargon ("blocks"). It seems to just overcomplicate the basic thread creation system (create thread, run function).
What exactly is GCD and why would it ever be preferred over traditional threading? When should traditional threads be used rather than GCD? And finally is there a reason for GCD's strange syntax? ("blocks" instead of simply calling functions).
I am on Mac OS X 10.6.8 Snow Leopard and I am not programming for iOS - I am programming for Macs. I am using Xcode 3.6.8 in Cocoa, creating a GUI application.
Advantages of Dispatch
The advantages of dispatch are mostly outlined here:
Migrating Away from Threads
The idea is that you eliminate work on your part, since the paradigm fits MOST code more easily.
It reduces the memory penalty your application pays for storing thread stacks in the application’s memory space.
It eliminates the code needed to create and configure your threads.
It eliminates the code needed to manage and schedule work on threads.
It simplifies the code you have to write.
Empirically, using GCD-type locking instead of #synchronized is about 80% faster or more, though micro-benchmarks may be deceiving. Read more here, though I think the advice to go async with writes does not apply in many cases, and it's slower (but it's asynchronous).
Advantages of Threads
Why would you continue to use Threads? From the same document:
It is important to remember that queues are not a panacea for
replacing threads. The asynchronous programming model offered by
queues is appropriate in situations where latency is not an issue.
Even though queues offer ways to configure the execution priority of
tasks in the queue, higher execution priorities do not guarantee the
execution of tasks at specific times. Therefore, threads are still a
more appropriate choice in cases where you need minimal latency, such
as in audio and video playback.
Another place where I haven't personally found an ideal solution using queues is daemon processes that need to be constantly rescheduled. Not that you cannot reschedule them, but looping within a NSThread method is simpler (I think). Edit: Now I'm convinced that even in this context, GCD-style locking would be faster, and you could also do a loop within a GCD-dispatched operation.
Blocks in Objective-C?
Blocks are really horrible in Objective-C due to the awful syntax (though Xcode can sometimes help with autocompletion, at least). If you look at blocks in Ruby (or any other language, pretty much) you'll see how simple and elegant they are for dispatching operations. I'd say that you'll get used to the Objective-C syntax, but I really think that you'll get used to copying from your examples a lot :)
You might find my examples from here to be helpful, or just distracting. Not sure.
While the answers so far are about the context of threads vs GCD inside the domain of a single application and the differences it has for programming, the reason you should always prefer GCD is because of multitasking environments (since you are on MacOSX and not iOS). Threads are ok if your application is running alone on your machine. Say, you have a video edition program and want to apply some effect to the video. The render is going to take 10 minutes on a machine with eight cores. Fine.
Now, while the video app is churning in the background, you open an image edition program and play with some high resolution image, decide to apply some special image filter and your image application being clever detects you have eight cores and starts eight threads to process the image. Nice isn't it? Except that's terrible for performance. The image edition app doesn't know anything about the video app (and vice versa) and therefore both will request their respectively optimum number of threads. And there will be pain and blood while the cores try to switch from one thread to another, because to avoid starvation the CPU will eventually let all threads run, even though in this situation it would be more optimal to run only 4 threads for the video app and 4 threads for the image app.
For a more detailed reference, take a look at http://deusty.blogspot.com/2010/11/introducing-gcd-based-cocoahttpserver.html where you can see a benchmark of an HTTP server using GCD versus thread, and see how it scales. Once you understand the problem threads have for multicore machines in multi-app environments, you will always want to use GCD, simply because threads are not always optimal, while GCD potentially can be since the OS can scale thread usage per app depending on load.
Please, remember we won't have more GHz in our machines any time soon. From now on we will only have more cores, so it's your duty to use the best tool for this environment, and that is GCD.
Blocks allow for passing a block of code to execute. Once you get past the "strange syntax", they are quite powerful.
GCD also uses queues which if used properly can help with lock free concurrency if the code executing in the separate queues are isolated. It's a simpler way to offer background and concurrency while minimizing the chance for deadlocks (if used right).
The "strange syntax" is because they chose the caret (^) because it was one of the few symbols that wasn't overloaded as an operator in C++
See:
https://developer.apple.com/library/ios/#documentation/General/Conceptual/ConcurrencyProgrammingGuide/OperationQueues/OperationQueues.html
When it comes to adding concurrency to an application, dispatch queues
provide several advantages over threads. The most direct advantage is
the simplicity of the work-queue programming model. With threads, you
have to write code both for the work you want to perform and for the
creation and management of the threads themselves. Dispatch queues let
you focus on the work you actually want to perform without having to
worry about the thread creation and management. Instead, the system
handles all of the thread creation and management for you. The
advantage is that the system is able to manage threads much more
efficiently than any single application ever could. The system can
scale the number of threads dynamically based on the available
resources and current system conditions. In addition, the system is
usually able to start running your task more quickly than you could if
you created the thread yourself.
Although you might think rewriting your code for dispatch queues would
be difficult, it is often easier to write code for dispatch queues
than it is to write code for threads. The key to writing your code is
to design tasks that are self-contained and able to run
asynchronously. (This is actually true for both threads and dispatch
queues.)
...
Although you would be right to point out that two tasks running in a
serial queue do not run concurrently, you have to remember that if two
threads take a lock at the same time, any concurrency offered by the
threads is lost or significantly reduced. More importantly, the
threaded model requires the creation of two threads, which take up
both kernel and user-space memory. Dispatch queues do not pay the same
memory penalty for their threads, and the threads they do use are kept
busy and not blocked.
GCD (Grand Central Dispatch): GCD provides and manages FIFO queues to which your application can submit tasks in the form of block objects. Work submitted to dispatch queues are executed on a pool of threads fully managed by the system. No guarantee is made as to the thread on which a task executes. Why GCD over threads :
How much work your CPU cores are doing
How many CPU cores you have.
How much threads should be spawned.
If GCD needs it can go down into the kernel and communicate about resources, thus better scheduling.
Less load on kernel and better sync with OS
GCD uses existing threads from thread pool instead of creating and then destroying.
Best advantage of the system’s hardware resources, while allowing the operating system to balance the load of all the programs currently running along with considerations like heating and battery life.
I have shared my experience with threads, operating system and GCD AT http://iosdose.com