Predicting Fine‑Tuning Time: Data‑Driven Scheduler Beats FLOP Estimates
THE SHIFT
When a fine‑tuning job sits idle on a shared GPU pool for hours, the wasted hardware and delayed deadlines become painfully obvious. Most teams still rely on a simple rule: divide the model’s total FLOPs by the peak throughput of a GPU and assume the result is the wall‑clock time. In practice that back‑of‑the‑envelope math often predicts a nine‑hour run that actually finishes the next afternoon, leaving schedules scrambled.
Imagine a scheduler that could forecast the exact duration before any GPU is allocated, letting you line up compatible jobs, fill gaps, and meet delivery dates with confidence. With such a prediction, idle GPUs become rare, queue backlogs shrink, and research teams can plan experiments rather than react to unexpected overruns, turning a chronic bottleneck into a manageable resource.
Source: Hugging Face: a]:hidden"> Why your fine-tuning job takes longer than the FLOPs say ↗
THE PROOF
The Dharma‑AI team collected 272 supervised fine‑tuning runs from a production cluster of NVIDIA H200 GPUs, totaling about 7,382 GPU‑hours. These jobs covered model families up to 15 billion parameters, using both LoRA adapters and full‑parameter fine‑tuning, and varied in wall‑clock time from 1.6 to 172.4 hours, giving a mean of 23.43 hours and a median of 15.29 hours.
Feeding only pre‑dispatch configuration data—such as model size, dataset token count, evaluation cadence, checkpoint frequency and GPU allocation—into a CatBoost regression model yielded an R² of 0.98 and a mean absolute error of 1.71 hours, with a median error of just 0.51 hours across five‑fold cross‑validation. By contrast, the traditional FLOP‑based estimate achieved only R² = 0.31 and MAE around 12 hours, exposing a massive gap.
Source: Hugging Face: a]:hidden"> Why your fine-tuning job takes longer than the FLOPs say ↗
THE CATCH
The model’s strong performance comes with clear limitations. It was trained exclusively on supervised fine‑tuning jobs, so it cannot predict durations for other training strategies such as RLHF, DPO or reinforcement‑learning‑based alignment, which have different compute and I/O patterns. Users must therefore treat its forecasts as inapplicable to those workloads.
All runs used the same H200 hardware, embedding that GPU’s performance profile into the learned relationships. Transferring the predictor to other GPU generations or mixed‑type clusters would require retraining with hardware‑specific descriptors. Moreover, prediction error grows for the longest jobs—runs over 60 hours showed median errors above five hours due to sparse data in that tail.
YOUR MOVE
If your team fine‑tunes on a homogeneous H200‑like pool, you can adopt the open‑source predictor or replicate its CatBoost pipeline to generate duration estimates before dispatch. Provide the same configuration fields—model parameters, dataset token count, evaluation and checkpoint settings, and GPU allocation—to obtain a realistic wall‑clock forecast that can be fed into your job‑queue manager.
Start by logging the same set of features for each run in your environment, then periodically retrain the model to keep it tuned to your hardware and workload patterns. Sharing the enriched dataset with the community can help broaden applicability beyond a single GPU generation. Integrating the predicted duration into scheduling software will let you batch compatible jobs, release GPUs promptly, and reduce costly idle time.
A data‑driven duration model dramatically outperforms FLOP estimates, making GPU scheduling far more reliable for fine‑tuning on homogeneous H200 clusters.
Sources & further reading
Published by The Breakdown. How we report · Our source desk.