by rreyes1979on 11/24/2023, 11:32 AMwith 3 comments
What would be your suggestions on hardware to either rent or buy to get into LLM fine-tuning without going broke? Got a personal project and would like to start exploring fine-tuning (Llama2?) in the near future.
Fine tuning in what respect? With large datasets? Or little ones. Makes a huge difference as far as training time and costs.
Look into qLoRA or 8bit quantization. You won't need a lot of memory if you do it locally with a small model and don't merge the weights. You can also use Modal Labs, RunPod, or Replicate.com. They have Serverless and per second billing which is great for testing inference.
Amazon Sagemaker training jobs can also work. Just be careful not to leave an inference endpoint on if you use Amazon for that too. You can easily go broke.
Fine tuning in what respect? With large datasets? Or little ones. Makes a huge difference as far as training time and costs.
Look into qLoRA or 8bit quantization. You won't need a lot of memory if you do it locally with a small model and don't merge the weights. You can also use Modal Labs, RunPod, or Replicate.com. They have Serverless and per second billing which is great for testing inference.
Amazon Sagemaker training jobs can also work. Just be careful not to leave an inference endpoint on if you use Amazon for that too. You can easily go broke.