Weights & Biases (wandb)

Related: How do you fine-tune a model using peft

Let's say you need to train or fine-tune a model. This can take a long time. And while you train, you need to be able to see how the training is going.

• Is the loss truly going down ?
• How does this run compare to another one with different hyperparameters ?
• How did the GPU use change as time went on ?

Well, Weights and Biases is here to help !

You can install it as a python package pip install wandb. Then, you need to create an account at https://wandb.ai, for example by using your google account. Then, you can login by copying the access token from your account to the console to connect the two.

Afterwards, whenever you train a transformer, wandb will detect this and create a new run. You can view all your runs in the webconsole and see parameters such as the loss overtime, the memory use, etc...

You can also see what hyper parameters you used for the run and how they compare with the ones from other runs.

An alternative to WandB is Tensorboard. If haven't tried it, so it might be good too, but Wandb fits my needs.