Data Strategy for Training a Customer Support Foundation Model
This brief describes the data, objective, starting point, and trade-offs for the three main training phases: pre-training, supervised fine-tuning (SFT), and preference fine-tuning.
1. Pre-training
- Data format: Large collections of text documents, tokenized and arranged into sequences. The text usually does not need question-and-answer labels.
- Data source: A licensed text corpus combined with the software company’s public product documentation and help articles.
- Objective: The model learns to predict the next token, building general language ability and knowledge.
- Starting point: Randomly initialized model weights, with the model architecture already defined.
- Risk if skipped: The model would lack broad language fluency and the general knowledge needed to understand varied customer questions.
2. Supervised Fine-Tuning (SFT)
- Data format: Labeled examples, often multi-turn chat records with roles such as
userandassistant, showing a question and a suitable support response. - Data source: Historical support conversations reviewed, anonymized, and curated by the company’s support team.
- Objective: The model learns to follow support instructions and produce useful answers in the expected style and format.
- Starting point: The pre-trained base model.
- Risk if skipped: The model may understand language but struggle to follow instructions or handle support conversations consistently.
3. Preference Fine-Tuning
- Data format: Comparisons between responses to the same prompt—for example, a preferred answer and a less helpful answer—with preference labels or rankings.
- Data source: Support specialists compare candidate answers using criteria such as accuracy, clarity, tone, and whether the answer should escalate to a human.
- Objective: The model learns which of several plausible responses people prefer.
- Starting point: The SFT model.
- Risk if skipped: Responses may be less helpful, clear, or appropriately cautious, even when they are relevant and well-formed.
Bonus: Which Phase Would Save the Least Compute to Skip?
Preference fine-tuning would typically save the least compute if skipped. It usually requires fewer training steps and less data than pre-training, which is generally the most compute-intensive phase.
It can still be worth keeping because it helps the assistant choose more helpful, clear, and appropriate responses among answers it already knows how to produce. Human review and annotation may also take time and money, even when the training compute is modest.
Data Handling Note
Before using support conversations for training, remove or mask personal and sensitive customer details, and confirm that the company has the rights and approvals needed to use the data.