Where to Find LLM Fine-Tuning Datasets
For instruction tuning, start with Databricks Dolly 15k or OpenAssistant OASST1; for preference tuning (RLHF or DPO), start with Anthropic HH-RLHF, UltraFeedback or Stanford SHP. Alpaca is useful for quick experiments but its non-commercial license blocks production use. Below is what each dataset actually contains, its license, and how to pick between them.
The short answer
Fine-tuning an LLM usually happens in two stages, and the datasets you need are different for each. Supervised fine-tuning (SFT), also called instruction tuning, needs prompt-response pairs; Databricks Dolly 15k and OpenAssistant OASST1 are commercially usable starting points. Preference tuning (RLHF reward modeling or direct preference optimization) needs ranked or chosen-versus-rejected response pairs; Anthropic HH-RLHF, UltraFeedback and Stanford SHP cover this. Alpaca remains a common reference dataset in papers and tutorials, but its CC BY-NC 4.0 license means you cannot ship it in a commercial product without a separate agreement.
Instruction-tuning (SFT) datasets
Databricks Dolly 15k
Dolly 15k is a set of 15,011 instruction-response pairs written by Databricks employees by hand, rather than generated by another model, which makes it a cleaner starting point when you want to avoid training on synthetic outputs from a third-party LLM. It spans eight task categories including brainstorming, classification, closed and open question answering, information extraction, summarization and creative writing. It is released under CC BY-SA 3.0, which permits commercial use but requires share-alike attribution. Get it from Hugging Face: databricks/databricks-dolly-15k.
OpenAssistant OASST1
OASST1 is a crowd-sourced conversation tree dataset with 88.8k rows (84.4k train / 4.4k validation), built from volunteer-written multi-turn conversations covering 24 or more languages, each message rated by other contributors. Because it is structured as branching conversation trees rather than flat pairs, it supports both SFT and preference-style training in the same release. It is Apache 2.0 licensed, which allows commercial use with attribution. Get it from Hugging Face: OpenAssistant/oasst1.
Stanford Alpaca
Alpaca popularized cheap instruction tuning: 52,002 instruction-output pairs generated by prompting OpenAI's text-davinci-003 with a self-instruct pipeline, released by Stanford in 2023. It is small enough to fine-tune quickly and is still widely referenced in tutorials and reproducibility studies. The license is CC BY-NC 4.0 — non-commercial only — so treat it as a research and evaluation dataset, not a production data source. Get it from Hugging Face: tatsu-lab/alpaca.
Preference and RLHF datasets
Anthropic HH-RLHF
HH-RLHF is Anthropic's original human-preference dataset from the paper "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback," containing 169,352 rows of chosen-versus-rejected response pairs labeled for helpfulness and harmlessness, plus a separate red-teaming split of adversarial conversations. It is one of the most cited preference datasets for reward-model training and is MIT licensed. The dataset card carries a content warning: some transcripts include offensive or harmful material by design, since the goal is to reduce those behaviors. Get it from Hugging Face: Anthropic/hh-rlhf.
UltraFeedback
UltraFeedback is a large, fine-grained preference dataset built from 64,000 prompts pooled from sources such as UltraChat, ShareGPT, Evol-Instruct, TruthfulQA and FLAN, with 256,000 responses collected across 17 different models and scored by GPT-4 on instruction-following, truthfulness, honesty and helpfulness. Because the scoring is broken out by dimension rather than a single chosen/rejected label, it is useful when you want a reward model that can explain why one response beat another. It is MIT licensed. Get it from Hugging Face: openbmb/UltraFeedback.
Stanford Human Preferences (SHP)
SHP takes a different approach: instead of paying annotators to rank model outputs, it mines naturally occurring preference signal from Reddit, using upvote and reply patterns across communities like r/askacademia to build about 386,000 pairwise comparisons (349k train / 18.4k validation / 18.4k test). That makes it a useful complement to model-graded datasets like UltraFeedback because the preference signal comes from real human behavior rather than another LLM's judgment. Confirm the current license on the dataset page before commercial use, since it is not stated in a single clear field. Get it from Hugging Face: stanfordnlp/SHP.
Datasets side by side
| Dataset | Stage | Scale | License |
|---|---|---|---|
| Databricks Dolly 15k | SFT / instruction tuning | 15,011 pairs | CC BY-SA 3.0 |
| OpenAssistant OASST1 | SFT + preference | 88.8k rows, 24+ languages | Apache 2.0 |
| Stanford Alpaca | SFT / instruction tuning | 52,002 pairs | CC BY-NC 4.0 (non-commercial) |
| Anthropic HH-RLHF | RLHF reward modeling | 169,352 rows | MIT |
| UltraFeedback | RLHF / DPO | 64k prompts, 256k responses | MIT |
| Stanford SHP | RLHF / DPO | ~386k pairs | Check dataset page |
How to choose
Start from the stage you are actually at. If the base model does not yet follow instructions reliably, begin with an SFT dataset — Dolly 15k if you need a commercially clean license, OASST1 if you need multilingual coverage. Once the model follows instructions but you want to shape tone, safety or helpfulness, move to a preference dataset — HH-RLHF for a well-established helpfulness/harmlessness baseline, UltraFeedback if you want dimension-level feedback for a more informative reward model, and SHP if you want a real-world, non-model-graded preference signal to sanity-check the others against. Whichever you use, mix in a slice of your own domain data if the target task is specialized — generic instruction data alone rarely captures the vocabulary and edge cases of a narrow professional domain.
FAQ
What is the difference between an instruction-tuning dataset and an RLHF dataset?
An instruction-tuning (SFT) dataset pairs a prompt with a single correct or preferred response, and the model is trained to reproduce that response directly. An RLHF or preference dataset pairs a prompt with two or more candidate responses that have been ranked or labeled chosen versus rejected, and that ranking signal trains a reward model or drives direct preference optimization. You typically need SFT data first and preference data second.
Can I use Alpaca or HH-RLHF data in a commercial product?
Check the license on each dataset before you decide. Alpaca is CC BY-NC 4.0, which restricts commercial use. Anthropic HH-RLHF and UltraFeedback are MIT licensed, and OASST1 is Apache 2.0, both of which generally permit commercial use with attribution. Dolly 15k is CC BY-SA 3.0, which allows commercial use but requires share-alike attribution. Always read the current license text on the dataset page rather than relying on a summary.
How much data do I actually need to fine-tune an LLM?
It depends on the base model and the task. Lightweight instruction-following behavior has been demonstrated with a few thousand well-written examples, as Alpaca and Dolly show. Domain adaptation for a narrow professional task often needs a few hundred to a few thousand high-quality, task-specific examples rather than a huge generic set. Preference tuning generally benefits from tens of thousands of ranked pairs because the signal per example is weaker than direct supervision.
Still cannot find the right training data? Leave it to us
If none of these fit your domain, or you need a dataset for a specialized vertical that is not well covered by the general releases above, tell us your task goal and the conditions it must meet. DeepSData can run a real availability search and verify sources for you — checking reputable data platforms and judging matches and gaps against each requirement, with every source link kept traceable.
New accounts get 3 free searches; every result comes with verifiable source links. See also curated datasets.