I am trying to run some parallel jobs through Python multiprocessing. Here is an example code:
import multiprocessing as mp
import os
def f(name, total):
print('process {:d} starting doing business in {:d}'.format(name, total))
#there will be some unix command to run external program
if __name__ == '__main__':
total_task_num = 100
mp.Queue()
all_processes = []
for i in range(total_task_num):
p = mp.Process(target=f, args=(i,total_task_num))
all_processes.append(p)
p.start()
for p in all_processes:
p.join()
I also set export OMP_NUM_THREADS=1 to make sure that only one thread for one process.
Now I have 20 cores in my desktop. For 100 parallel jobs, I want to let it run 5 cycles so that each core run one job (20*5=100).
I tried to do the same code in CentOS and ubuntu. It seems that CentOS will automatically do a job splitting. In other words, there will be only 20 parallel running jobs at the same time. However, ubuntu will start 100 jobs simultaneously. As such, each core will be occupied by 5 jobs. This will significantly increase the total run time due to high work load.
I wonder if there is an elegant solution to teach ubuntu to run only 1 job per core.
To enable a process run on a specific CPU, you use the command taskset in linux. Accordingly you can arrive on a logic based on "taskset -p [mask] [pid]" that assigns each process to a specific core in a loop.
Also , python helps in incorporation of affinity control via sched_setaffinity that can be checked for confining a process to specific cores. Accordingly , you can arrive on a logic for usage of "os.sched_setaffinity(pid, mask)" where pid is the process id of the process whose mask represents the group of CPUs to which the process shall be confined to.
In python, there are also other tools like https://pypi.org/project/affinity/ that can be explored for usage.
Related
I have 4 jobs (A, B, C, D), which I want to start using pyiron. All jobs need to run on a remote cluster using SLURM. Some of the jobs need results from other jobs as input.
Ideally, I would like to have a workflow like:
Job A is started by the user.
Jobs B and C start automatically and in parallel (!) as soon as job A is done.
Job D starts automatically as soon as the jobs B and C are finished.
I realize that I could implement this in Jupyter using some if-conditions and the sleep-command.
However, the jobs A, B, and C could run for multiple days and I don't want to keep my Jupyter notebook running for so long.
Is there a more convenient way to realize these job dependencies in pyiron?
I guess the easiest way would be to submit the whole Jupyter notebook to the queue using the script job class:
job = pr.create.job.ScriptJob("script")
job.script_path = 'workflow.ipynb'
job.server.queue = 'my_queue'
job.server.cores = 32
job.run()
Here workflow.ipynb would be your current notebook, my_queue your SLURM queue for remote submission and 32 the total number of cores for allocation.
I'm submitting jobs to a cluster via SLURM scheduler, and let's say I have access to 5 types of GPUs in my cluster. They are GPUs of type A,B,C,D,E. I would like to submit a job that requests the use of GPUs of type A or B or C but NOT of type D or E. So I need some type of OR logic with the --gres flag.
As a concrete example, here is what it looks like when I request a gpu of a single type (in this case, an RTX 2080):
qlogin -p gpu --gres=gpu:rtx2080:1 --mem=8g -c 2 I'd like to do this but allowing SLURM to pick from a list of allowed GPU types
Slurm does not have that option at this time.
One workaround is for the system administrator to setup features of the node with the GPU type to allow a request such as:
qlogin -p gpu --gres=gpu:1 --constraint="rtx2080|rtx3090" --mem=8g -c 2
(assuming qlogin uses the same options as sbatch)
If that is not possible, you can submit as many job as there are types of GPU that you want, all with the same --job-name=<SOME_NAME> and the --dependency=singleton option. Then you use whichever job starts first and cancel the other with
scancel --jobname <SOME_NAME> --state=PENDING
The --dependency option makes sure only one job is started at a time.
I followed gem5-tutorial to build a test config. It executes hello world, on se mode. But, now I want to run multiple processes, one by one. How to do that? So far i tried this
processes = []
processes.append([bzip2_benchmark, bzip2_input])
processes.append([mcf_benchmark, mcf_input])
processes.append([hmmer_benchmark, '--fixed=0', '--mean=325', '--num=45000', '--sd=200', '--seed=0', hmmer_input])
processes.append([sjeng_benchmark, sjeng_input])
processes.append([lbm_benchmark, 20, 'reference.dat', 0, 1, benchmark_dir+'470.lbm/data/100_100_130_cf_a.of'])
for p in processes:
process = Process()
process.cmd = p
system.cpu.workload = process
system.cpu.createThreads()
root = Root(full_system=False, system=system)
m5.instantiate()
print("Beginning simulation!")
exit_event = m5.simulate()
print('Exiting # tick {} because {}'
.format(m5.curTick(), exit_event.getCause()))
Assume that all imports are correct and, system is instantiated correctly. The above code gives "fatal: Attempt to allocate multiple instances of Root.", after running the first process. I understood why this happens, but I want to know how to run these benchmark programs one by one.
I am running over 40 spiders which are until now scheduled via cron and issued via scrapy crawl Due to several reasons I am now switching to scrapyd, one of them is to be able to see which jobs are running in case I need to do maintenance and reboot - so I can cancel a job.
Is it possible to cancel multiple jobs at once? I noticed that multiple jobs might be running at once with many waiting in quene with status "pending". Stopping the crawl might therefore require multiple calls of the cancel.json endpoint.
How to stop (or better pause) all jobs?
The scrapyd API (as of v1.3.0) does not support pausing. It does have stopping one job per call, however, so you have to loop the jobs yourself
I took #kolas's script from this question and update it to work with python 3.
import json, os
PROJECT_NAME = "MY_PROJECT"
cd = os.system('curl http://localhost:6800/listjobs.json?project={} > kill_job.text'.format(PROJECT_NAME))
with open('kill_job.text', 'r') as f:
a = json.loads(f.readlines()[0])
pending_jobs = list(a.values())[2]
for job in pending_jobs:
job_id = job['id']
kill = 'curl http://localhost:6800/cancel.json -d project={} -d job={}'.format(PROJECT_NAME, job_id)
os.system(kill)
I want to have a task that will execute every 5 minutes, but it will wait for last execution to finish and then start to count this 5 minutes. (This way I can also be sure that there is only one task running) The easiest way I found is to run django application manage.py shell and run this:
while True:
result = task.delay()
result.wait()
sleep(5)
but for each task that I want to execute this way I have to run it's own shell, is there an easy way to do it? May be some king custom ot django celery scheduler?
Wow it's amazing how no one understands this person's question. They are asking not about running tasks periodically, but how to ensure that Celery does not run two instances of the same task simultaneously. I don't think there's a way to do this with Celery directly, but what you can do is have one of the tasks acquire a lock right when it begins, and if it fails, to try again in a few seconds (using retry). The task would release the lock right before it returns; you can make the lock auto-expire after a few minutes if it ever crashes or times out.
For the lock you can probably just use your database or something like Redis.
You may be interested in this simpler method that requires no changes to a celery conf.
#celery.decorators.periodic_task(run_every=datetime.timedelta(minutes=5))
def my_task():
# Insert fun-stuff here
All you need is specify in celery conf witch task you want to run periodically and with which interval.
Example: Run the tasks.add task every 30 seconds
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
"runs-every-30-seconds": {
"task": "tasks.add",
"schedule": timedelta(seconds=30),
"args": (16, 16)
},
}
Remember that you have to run celery in beat mode with the -B option
manage celeryd -B
You can also use the crontab style instead of time interval, checkout this:
http://ask.github.com/celery/userguide/periodic-tasks.html
If you are using django-celery remember that you can also use tha django db as scheduler for periodic tasks, in this way you can easily add trough the django-celery admin panel new periodic tasks.
For do that you need to set the celerybeat scheduler in settings.py in this way
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
To expand on #MauroRocco's post, from http://docs.celeryproject.org/en/v2.2.4/userguide/periodic-tasks.html
Using a timedelta for the schedule means the task will be executed 30 seconds after celerybeat starts, and then every 30 seconds after the last run. A crontab like schedule also exists, see the section on Crontab schedules.
So this will indeed achieve the goal you want.
Because of celery.decorators deprecated, you can use periodic_task decorator like that:
from celery.task.base import periodic_task
from django.utils.timezone import timedelta
#periodic_task(run_every=timedelta(seconds=5))
def my_background_process():
# insert code
Add that task to a separate queue, and then use a separate worker for that queue with the concurrency option set to 1.