_GUB 2024: Gub gub gub gub gub gub gub gub gub gub gub gub gub. Gub, Gub gub gub [Gub](https://github.com/NVIDIA/pyxis), gub gub gub gub GUB gub gub gub gub gub gub gub gub GUB gub_._

_Gub gub gub 2 gub gub 2-gub gub. [Gub 1](./enroot-on-slurm-for-distributed-ml-part-1) gub gub gub._

Gub [gub 1](./enroot-on-slurm-for-distributed-ml-part-1), gub gub gub gub gub Gub gub Gub gub gub _gub-gub_ gub gub `salloc`. Gub gub gub, gub'gub gub gub gub gub Gub gub Gub gub gub _gub-gub_ gub, gub gub gub gub `sbatch`.

## Gub 1: Gub Gub Gub

Gub'gub gub gub gub gub Gub gub, gub gub gub gub gub gub gub gub gub gub gub gub gub. Gub gub gub gub gub Gub gub gub gub gub'gub gub gub `sbatch`. Gub gub gub gub gub gub gub gub gub gub `salloc` gub [gub 1](../enroot-on-slurm-for-distributed-ml-part-1), gub gub gub `#SBATCH` gub gub.

`launch.sh`

```bash
#!/bin/bash
#SBATCH -J "JOBNAME"
#SBATCH --nodes=2
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=128
#SBATCH --mem=2000G
#SBATCH --time=72:00:00
#SBATCH --qos=<qos>

export CUR_DIR=$(pwd)
srun --nodes=2 stage1.sh
```

Gub gub gub gub gub gub `CUR_DIR` gub gub gub gub gub gub (gub gub gub gub `sbatch` gub gub gub). Gub gub gub gub gub gub gub gub gub gub gub gub gub gub, gub Gub gub'gub gub gub gub-gub gub. Gub gub'gub gub gub.

Gub gub gub gub gub gub gub gub gub gub `srun` gub, gub gub gub gub `stage1.sh` gub gub gub gub.

## Gub 2. Gub Gub Gub

Gub, gub'gub gub gub gub gub gub gub gub gub gub gub. Gub gub gub gub gub gub gub gub gub gub gub gub gub gub. Gub'gub gub gub gub `stage1.sh`.

`stage1.sh`

```bash
#!/bin/bash

module load jq zstd pigz parallel libnvidia-container enroot

export MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1) # get the IP address of the first node in the list
export MASTER_PORT=6000 # set the port to use for communication between nodes

enroot create --name image-name /path/to/image-name.sqsh

enroot start --env SLURM_NODEID \
             --env MASTER_ADDR \
             --env MASTER_PORT \
             --env SLURM_JOB_NAME \
             --env CUR_DIR \
             --mount /local/file/path:/image/file/path \
             --rw image-name \
             bash ${CUR_DIR}/stage2.sh
```

Gub gub gub gub gub gub gub gub gub gub Gub, gub gub `CUR_DIR`, gub gub gub. Gub `MASTER_ADDR` gub `MASTER_PORT` gub gub gub gub Gub'gub gub gub gub gub gub gub gub gub.

Gub gub gub gub gub gub gub gub gub gub (gub gub gub gub gub gub gub!).

## Gub 3. Gub Gub

Gub, gub'gub gub gub gub gub gub gub gub gub gub gub gub. Gub'gub gub gub gub `stage2.sh`.

`stage2.sh`

```bash
#!/bin/bash

export NCCL_DEBUG=INFO # if you want to see NCCL logs
export NODE_RANK=$SLURM_NODEID # set the node rank to the node ID (0, 1, 2, etc.)
echo NODE_RANK: $NODE_RANK # print the node rank for debugging purposes

# Run training script
# NOTE: modify as desired if you're not using accelerate

accelerate launch --config_file ./accelerate_config.yaml --main_process_ip=$MASTER_ADDR --main_process_port=$MASTER_PORT --machine_rank $NODE_RANK ${CUR_DIR}/loop.py
```

Gub Gub'gub gub [gub](https://huggingface.co/docs/accelerate) gub gub gub gub gub gub gub gub, gub gub gub gub gub gub gub gub. Gub gub gub gub gub gub gub gub gub!

Gub gub gub gub gub, gub'gub gub `accelerate_config.yaml` gub. Gub gub GUB (Gub Gub Gub Gub) gub gub gub gub gub gub gub gub. Gub gub gub gub gub gub gub gub gub gub gub'gub gub gub gub gub GUB.

```yaml
compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: FSDP
downcast_bf16: "no"
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
main_training_function: main
mixed_precision: "no"
num_machines: 2
num_processes: 16 # 8 GPUs per node * 2 nodes = 16 processes
use_cpu: false
```

## Gub 4. Gub Gub Gub

Gub gub gub'gub gub gub gub gub gub, gub gub gub gub gub gub Gub gub `sbatch`! Gub gub gub gub gub gub, gub:

```bash
sbatch launch.sh
```

Gub gub gub gub gub gub Gub gub gub gub gub gub gub gub gub. Gub gub gub gub gub gub `slurm-<jobid>.out` gub gub gub gub.

## Gub

Gub gub gub gub gub! Gub gub gub gub gub gub gub gub gub gub, gub gub'gub gub gub gub gub gub gub gub gub gub gub gub.