October 31, 2024
Job Dependency
For some users, it might be worth trying to setup dependency jobs in the LSF system.
For example, if you want to first run Tophat, then run Clifflink with the output file from the previous Tophat jobs. You do not need to wait for the first job to be finished before your submit the following jobs. Instead, you can check the dependency setting here.
1. You job is identified by either job-id or job name. Job id is easier, but you might need a bit more work to obtain the job-id because job-id is distributed by the LSF server, it could not set by yourself. Job name is the name set by bsub -J $job-name. The maximum length is 4K. If your job-name is too long, only the ending part will be displayed. Do not worry, the system still know your full job-name, even it was shown as *ABCDEF.
2. The second stage job could be set to start depending on the states of the first stage job. LSF offer 4 stages.
started/done/exit/ended.Started means your jobs has started running. If your job had finished, it is also taken as started. But if the job is "pend/psusp", then it is regarding as started.
Done means your jobs finished successfully.
Exit means your job terminated abnormally. It might be terminated by yourself before finishing. It could also terminated by the system automatically.
Ended means either Exit or Done.
3. You can also make logical operation of the dependency state: The logic operations include &&, ||
Example
Given one example here
#BSUB -w 'done (11111) && ( started (job1) || exit( job2) )'It means your current job will be submitted into LSF for pending, but it will only activate when 1) job with job-id 11111 had finished successfully AND [ (job with job-name "job1" started) or (job with job-name "job2" finished abnormally)]
In the beginning example, you can name the toptop job as job1-tophat , then add this line into your lsf script file for cufflinks.
#BSUB -w 'done(job1-tophat)'