==SIMPLEG-WBM coupling ==
=== Current status ===
  * Now you can run a series of Singularity containers automatically by submitting a job script from Terminal. 
  * The script runs four Singularity containers sequentially and uses previous stage's outputs
  * Current workflow assumes some convergence conditions. Workflow orchestration will be changed to alternative approach proposed by Iman. 
{{{
There are some minor changes but Jungha should be able to create the shocks for SIMPLEG using the code:
Spin-up:
We need an initial run of WBM for 2000-2005
Then we calculate the shocks for SIMPLEG using WBMtoSIMPLE_v3. The new R script takes the “sim.year1” (here 2004) and “sim.length” (here 4) as inputs. After the test run, the year will be updated at each run.
Then we solve SIMPLEG. It will create a new land area to be sent to WBM. for the test run, I like to label it with the “2005” run.
Consecutive WBM-SIMPLEG runs:
Here we need to run WBM only for the next year (the year 2005).
Then we calculate the shocks for SIMPLEG (for 2005). It creates a new land area to be sent to WBM for the next year, it will be labeled with the next year (here as 2006).
Then we repeat (a) and (b) for all the required years.
}}}

[[Image(coupling_flow.jpg, desc="Submit workflows and inputs/outputs. Red highlighted files are not generated correctly as of 10/19/2021. ")]]
Most recent status document is available at [https://docs.google.com/presentation/d/1xqOf9TrTKhif3ieuS-bc3gqlqx52jwFFL-zrzQdIv8g/edit?usp=sharing  WBM-SIMPLEG coupling status]

== Set up an environment to run coupled models ==
Singularity containers and their inputs are available at ''/depot/ssg/geoshare/data/GLASS/INFEWS/SIMPLEG_WBM_COUPLING'' directory.  [[BR]]
Please make sure you have READ privilege to that directory. Otherwise the script will fail. [[BR]]
The following script will  copy all necessary files to Brown's scratch space, namely ''/scratch/brown/$USER/SIMPLEG_WBM_COUPLING''. [[BR]]
Here ''$USER'' stands for your account. 

{{{ 
wooj@brown-fe02:~$ /depot/ssg/geoshare/data/GLASS/INFEWS/setup_wbm_simpleg_env.sh
}}} 

Please copy/paste this command to the terminal and execute it.[[BR]]
This may take long as it copies about 40GB zip files  and uncompress them. [[BR]]
Once the set up is done, you will see these directories under ''/scratch/brown/$USER/''[[BR]]

[[Image(SIMPLEG_WBM_COUPLING_directory.jpg, desc="Subdirectories under SIMPLEG_WBM_COUPLING folder")]]

== Submit a job ==
Log and error files are created at the directory where the submit executable is called. [[BR]]
Please look into "slurm-jobid.out" file for investigating errors. [[BR]]

{{{
wooj@brown-fe02:~$ sbatch /scratch/brown/wooj/SIMPLEG_WBM_COUPLING/submit/simpleg-wbm.sub
Submitted batch job 12837875
}}}

=== Checking status of a job ===

[[Image(jobstatus_check.jpg, desc="Job status is shown as 'Running' or 'R' at the ST (Status) column  ")]]


== Submit file ==

===  The sequence submit file executes containers ===
One iteration consist of four steps.  The script finishes after a single iteration. [[BR]]
Multiple iterations can be done by modifying ''for loop'' in the script later. [[BR]]
For example, to run three iterations, script can be updated to [[BR]]
{{{
    for i in 1 2 3
}}}


 # WBM Singularity container 
   * WBM container's output is written under wbm_output directory, not job-specific directory.
 # WBM to SIMPLEG Singularity container 
  * It reads daily files and aggregates them into yearly resolution. 
  * Start year and end year are currently hard-coded to 2000-2002. These will be configurable soon.
 # SIMPLEG Singularity container 
  * Each iteration's output is written to job specific directory such as Iteration_1_S3 or Iteration_2_S3 ... 
 # SIMPLEG to WBM container 
  * It is having some issues creating three netcdf files (s1_rice_irr_fr_av.nc and so on)
  * Its output is directly written to WBM's input directory 

[[Image(submitfile.jpg, , desc="A submit file executing four containers sequentially. If you want to run a specific step only, comment out the others with '#' at the beginning of the line "))]]

== Editing submit file ==
  * If you want to execute a certain Singularity only, comment out the other steps. 
  * DO NOT forget to replace the $SLURM_ID with the actual job id you want to refer to.
  * Please contact Jungha Woo (wooj@purdue.edu) for any questions.

== Container recipes ==
Four Singularity recipes are available at github [https://github.com/junghawoo/simpleg-wbm-coupling],
but they are private repositories at this moment.  Please contact Jungha Woo ( wooj@purdue.edu).

== Container 101 ==
 Slides from Yucheng Zhang@RCAC is available at [https://www.rcac.purdue.edu/files/training/containers101.pdf]
 * Why we should use containers
   * Build once, run almost anywhere 
   * Ensure reproducibility 
 *  Why we use Singularity containers 
   * Secure! 
   * Users do not need to ask their system admin to install software for them
   * Does not require root privilege