I am trying to modify scheduling in minix 3.
Till now I am able locate:
-> boot image in kernel/table.c
-> process table in kernel/proc.h
-> priv table in kernel/priv.h
-> initial process table entries (boot image) in kernel/main.c
-> shed(), pick_proc(), enque(), deque() in kernel/proc.c
But i am not able to locate, where/how a new process table entry (for a new process) is created and is assiged its priority, quantum and other process table entries?
Also shed(), pick_proc(), enque(), deque() all are called with a pointer to current process process table. Who calls these function?
Help!
Related
I have one delta table name "ali" . I have read stream from that delta table :
from pyspark.sql.functions import col
streamDF = spark.readStream.format('delta').load('dbfs:/user/hive/warehouse/pdf/ali')
display(streamDF)
Now I want write my data stream into my silver delta table :
(streamDF
.writeStream
.format("delta")
.option("checkpointLocation" ,"dbfs:/user/hive/warehouse/pdf/ali") #allows us to pick up where we left off if we lose connectivity
.outputMode("append") # appends data to our table
.option("mergeSchema","true")
.start('dbfs:/tmp/hive/ali4') )
Spark job continuously in running stage :
What should I do ?
Thanks
These are streaming jobs and will be in running mode till you manually cancel it. It will stream the data from source delta (Say bronze) and it will write to destination delta table (Say Silver).
You will see a graph showing whats the input rate and whats the output rate.
This page has more info around streaming UI of spark services - https://www.databricks.com/blog/2020/07/29/a-look-at-the-new-structured-streaming-ui-in-apache-spark-3-0.html
If you want to validate, you can perform below.
-> Open a new notebook instead of adding a new cell below your write stream code and add a new cell like this - spark.read.format("delta").load("silver location")
-> ingest some sample data to your source delta table (In your case it would be bronze delta location)
-> Run the cell you created in first step where you are reading the data from silver and you should see the newly ingested data flowing into silver(destination) from bronze location(Source).
I have a piece of code that is essentially executing the following with Infinispan in embedded mode, using version 13.0.0 of the -core and -clustered-lock modules:
#Inject
lateinit var lockManager: ClusteredLockManager
private fun getLock(lockName: String): ClusteredLock {
lockManager.defineLock(lockName)
return lockManager.get(lockName)
}
fun createSession(sessionId: String) {
tryLockCounter.increment()
logger.debugf("Trying to start session %s. trying to acquire lock", sessionId)
Future.fromCompletionStage(getLock(sessionId).lock()).map {
acquiredLockCounter.increment()
logger.debugf("Starting session %s. Got lock", sessionId)
}.onFailure {
logger.errorf(it, "Failed to start session %s", sessionId)
}
}
I take this piece of code and deploy it to kubernetes. I then run it in six pods distributed over six nodes in the same region. The code exposes createSession with random Guids through an API. This API is called and creates sessions in chunks of 500, using a k8s service in front of the pods which means the load gets balanced over the pods. I notice that the execution time to acquire a lock grows linearly with the amount of sessions. In the beginning it's around 10ms, when there's about 20_000 sessions it takes about 100ms and the trend continues in a stable fashion.
I then take the same code and run it, but this time with twelve pods on twelve nodes. To my surprise I see that the performance characteristics are almost identical to when I had six pods. I've been digging in to the code but still haven't figured out why this is, I'm wondering if there's a good reason why infinispan here doesn't seem to perform better with more nodes?
For completeness the configuration of the locks are as follows:
val global = GlobalConfigurationBuilder.defaultClusteredBuilder()
global.addModule(ClusteredLockManagerConfigurationBuilder::class.java)
.reliability(Reliability.AVAILABLE)
.numOwner(1)
and looking at the code the clustered locks is using DIST_SYNC which should spread out the load of the cache onto the different nodes.
UPDATE:
The two counters in the code above are simply micrometer counters. It is through them and prometheus that I can see how the lock creation starts to slow down.
It's correctly observed that there's one lock created per session id, this is per design what we'd like. Our use case is that we want to ensure that a session is running in at least one place. Without going to deep into detail this can be achieved by ensuring that we at least have two pods that are trying to acquire the same lock. The Infinispan library is great in that it tells us directly when the lock holder dies without any additional extra chattiness between pods, which means that we have a "cheap" way of ensuring that execution of the session continues when one pod is removed.
After digging deeper into the code I found the following in CacheNotifierImpl in the core library:
private CompletionStage<Void> doNotifyModified(K key, V value, Metadata metadata, V previousValue,
Metadata previousMetadata, boolean pre, InvocationContext ctx, FlagAffectedCommand command) {
if (clusteringDependentLogic.running().commitType(command, ctx, extractSegment(command, key), false).isLocal()
&& (command == null || !command.hasAnyFlag(FlagBitSets.PUT_FOR_STATE_TRANSFER))) {
EventImpl<K, V> e = EventImpl.createEvent(cache.wired(), CACHE_ENTRY_MODIFIED);
boolean isLocalNodePrimaryOwner = isLocalNodePrimaryOwner(key);
Object batchIdentifier = ctx.isInTxScope() ? null : Thread.currentThread();
try {
AggregateCompletionStage<Void> aggregateCompletionStage = null;
for (CacheEntryListenerInvocation<K, V> listener : cacheEntryModifiedListeners) {
// Need a wrapper per invocation since converter could modify the entry in it
configureEvent(listener, e, key, value, metadata, pre, ctx, command, previousValue, previousMetadata);
aggregateCompletionStage = composeStageIfNeeded(aggregateCompletionStage,
listener.invoke(new EventWrapper<>(key, e), isLocalNodePrimaryOwner));
}
The lock library uses a clustered Listener on the entry modified event, and this one uses a filter to only notify when the key for the lock is modified. It seems to me the core library still has to check this condition on every registered listener, which of course becomes a very big list as the number of sessions grow. I suspect this to be the reason and if it is it would be really really awesome if the core library supported a kind of key filter so that it could use a hashmap for these listeners instead of going through a whole list with all listeners.
I believe you are creating a clustered lock per session id. Is this what you need ? what is the acquiredLockCounter? We are about to deprecate the "lock" method in favour of "tryLock" with timeout since the lock method will block forever if the clustered lock is never acquired. Do you ever unlock the clustered lock in another piece of code? If you shared a complete reproducer of the code will be very helpful for us. Thanks!
I'm fresher in oozie bundle. I want to run multiple coordinators one after another in bundle job.My requirement is after completion of one coordinator job _SUCCESS file will be generated, then by using that _SUCCESS file second coordinator should be triggered. I don't know how to do that.For that i used data dependency technique which will keep track for generated output files of previous coordinator. I'm sharing some code which i tried.
Lets say there are 2 coordinator jobs:A and B.and i want to trigger only A coordinator.and if _SUCCESS file for Coordinator A generated then only Coordinator B should get start.
A - coordinator.xml
<workflow>
<app-path>${aDir}/aWorkflow</app-path>
</workflow>
this will call respective workflow.and _SUCCESS file is generated at ${aDir}/aWorkflow/final_data/${date}/aDim location so i included this location in
B coordinator:
<dataset name="input1" frequency="${freq}" initial-instance="${START_TIME1}" timezone="UTC">
<uri-template>${aDir}/aWorkflow/final_data/${date}/aDim</uri-template>
</dataset>
<done-flag>_SUCCESS</done-flag>
<data-in name="coordInput1" dataset="input1">
<instance>${START_TIME1}</instance>
</data-in>
<workflow>
<app-path>${bDir}/bWorkflow</app-path>
</workflow>
but when i run it first coordinator gets KILLED itself, but if i run individually they are running successfully.i'm not getting why these are all getting KILLED.
help to sort out
I find out easy way to do that. I'm sharing solution.For coordinator B coordinator.xml I'm sharing.
1)For Data-set instance should be start time of second one but it should not be time instance of first coordinator.otherwise that particular coordinator will get KILLED.
2)If you want to run multiple coordinators one after another then you can also include controls in coordinator.xml. e.g. concurrency, timeout or throttle. Detailed information about these controls you can find out in "apache oozie" book's 6th chapter.
3)in "" i included latest(0) it will take latest generated folder in mentioned output path.
4)for "input-events" it is mandatory to include it's name as a input to ${coord:dataIn('coordInput1')}.otherwise oozie will not consider dataset.
30
1
${aimDir}/aDimWorkflow/final_data/${date}/aDim
_SUCCESS
${coord:latest(0)}
${bDir}/bWorkflow
input_files
${coord:dataIn('coordInput1')}
I am aware that you can preform simple message passing with the following:
self() ! hello.
and you can see the message by calling:
flush().
I can also create simple processes in functions with something like:
spawn(module, function, args).
However I am not clear how one can send messages to the processes with out registering the Pid.
I have seen examples showing that you can pattern match against this in the shell to get the Pid assigned to a var, so if i create a gen_server such as:
...
start_link() ->
gen_server:start_link(?MODULE, init, []).
init(Pid) ->
{ok, Pid}.
...
I can then call it with the following from the shell:
{ok, Pid} = test_sup:start_link().
{ok,<0.143.0>}
> Pid ! test.
test
So my question is, can you send messages to Pids in the form <0.0.0> with out registering them to an atom or variable in the shell? Experimenting and searching as proved fruitless...
If you happen to need to send a message to a Pid based on the textual representation of its Pid, you can do (assuming the string is "<0.42.0>"):
list_to_pid("<0.42.0>") ! Message
This is almost only useful in the shell (where you can see the output of log messages or monitor data from something like Observer); any spawned process should normally be a child of some form of parent process to which it is linked (or monitored).
As for sending a message to something you just spawned, spawn returns a Pid, so you can assign it directly to a variable (which is not the same as registering it):
Pid = spawn(M, F, A),
Pid ! Message.
If you have the string "" to identify a pid, it is
either because you are working in the shell, and you use the representation you see, and you forgot to store this pid in a variable. Then simply use pid(X,Y,Z) to get it;
either because you did something like io_lib:format("~p",[Val]) where Val is the pid or a an erlang term which contain this pid. Then simply assign the pid to a variable (directly or extracting it from the term). It can be stored in an ets, send to another process without transformation
You should avoid to use the shell (or string) representation. One reason is that this representation is different when you ask the pid of one process from 2 different nodes as shown in the next screen capture.
I have a common test suite that attempts to create an ets table for use in all suites and all test cases. It looks like so:
-module(an_example_SUITE).
-include_lib("common_test/include/ct.hrl").
-compile(export_all).
all() -> [ets_tests].
init_per_suite(Config) ->
TabId = ets:new(conns, [set]),
ets:insert(TabId, {foo, 2131}),
[{table,TabId} | Config].
end_per_suite(Config) ->
ets:delete(?config(table, Config)).
ets_tests(Config) ->
TabId = ?config(table, Config),
[{foo, 2131}] = ets:lookup(TabId, foo).
The ets_tests function failed with a badarg. Creating/destroying the ets table per testcase, which looks like so:
-module(an_example_SUITE).
-include_lib("common_test/include/ct.hrl").
-compile(export_all).
all() -> [ets_tests].
init_per_testcase(Config) ->
TabId = ets:new(conns, [set]),
ets:insert(TabId, {foo, 2131}),
[{table,TabId} | Config].
end_per_testcase(Config) ->
ets:delete(?config(table, Config)).
ets_tests(Config) ->
TabId = ?config(table, Config),
[{foo, 2131}] = ets:lookup(TabId, foo).
Running this, I find that it functions beautifully.
I'm confused by this behavior and unable to determine why this would happen, form the docs. Questions:
Why does this happen?
How can I have an ets table to share between a per suite and per testcase?
As was already mentioned in the answer by Pascal and as discussed in the User Guide only init_per_testcase and end_per_testcase run in the same process as the testcase. Since ETS tables are bound to a owner process your only way to have a ETS table persist during a whole suite or group is to give it away or define a heir process.
You can easily spawn a process in your init_per_suite or init_per_group functions, set it as heir for the ETS table and pass its pid along in the config.
To clean up all you need is to kill this process in your end_per_suite or end_per_group functions.
-module(an_example_SUITE).
-include_lib("common_test/include/ct.hrl").
-compile(export_all).
all() -> [ets_tests].
ets_owner() ->
receive
stop -> exit(normal);
Any -> ets_owner()
end.
init_per_suite(Config) ->
Pid = spawn(fun ets_owner/0),
TabId = ets:new(conns, [set, protected, {heir, Pid, []}]),
ets:insert(TabId, {foo, 2131}),
[{table,TabId},{table_owner, Pid} | Config].
end_per_suite(Config) ->
?config(table_owner, Config) ! stop.
ets_tests(Config) ->
TabId = ?config(table, Config),
[{foo, 2131}] = ets:lookup(TabId, foo).
You also need to make sure you can still access your table from the testcase process, by making it either protectedor public
An ets table is attached to a process and destroyed as soon as the process ends, unless you use the the give_away function (which is not feasible I fear in this case)
As state in the common tets doc, each test case and the init_per_suite and end_per_suite are run in separate processes, so the ets table is destroyed as soon as you leave the init_per_suite function.
fron common_test doc
init_per_suite and end_per_suite will execute on dedicated Erlang
processes, just like the test cases do. The result of these functions
is however not included in the test run statistics of successful,
failed and skipped cases.
from ets doc
The default owner is the process that created the table. Table
ownership can be transferred at process termination by using the heir
option or explicitly by calling give_away/3.