Note

Phi-1 is intended for research purposes, please use in accordance with license terms.

Introduction

The phi-1 model is a state-of-the-art code generation model designed for writing simple Python functions from docstrings. This model leverages high-quality data and a carefully crafted training process to achieve impressive performance despite its smaller size compared to competing models. The model was trained using a combination of high-quality data from the web (6 billion tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1 billion tokens). phi-1 is obtained by finetuning phi-1-base on the CodeExercises dataset

phi-1 Model Details

• Model Type: Transformer-based language model
• Number of Parameters: 1.3 billion
• Training Data: Combines "textbook quality" data from the web (6 billion tokens) and synthetic Python textbooks and exercises generated with GPT-3.5 (1 billion tokens).
• Training Process: Pretraining on "textbook quality" data followed by finetuning on "textbook-exercise-like" data.

Use Cases

The phi-1 model is primarily designed for the following use case:

• Writing simple Python functions from docstrings, making it a valuable tool for code generation tasks.

Prompt:
def valid_guessing_letters(word: str, guesses: List[str]) -> List[str]:
""" Returns a list of valid guessing letters, which are letters that have not been guessed yet and are present in the word. Parameters: word (str): The word to guess. guesses (List[str]): A list of letters that have already been guessed. Returns: List[str]: A list of valid guessing letters. """

Response:
def valid_guessing_letters(word: str, guesses: List[str]) -> List[str]:
""" Returns a list of valid guessing letters, which are letters that have not been guessed yet and are present in the word. Parameters: word (str): The word to guess. guesses (List[str]): A list of letters that have already been guessed. Returns: List[str]: A list of valid guessing letters. """
valid_letters = [] for letter in word: if letter not in guesses and letter not in valid_letters:

valid_letters.append(letter) return valid_letters

Dataset Information

The phi-1 model's key innovation lies in its curated training data, which significantly improves its performance. Phi-1 relies on three primary datasets for its training:

1. Filtered Code-Language Dataset: A subset of The Stack and StackOverflow, comprising approximately 6 billion tokens, obtained through a language model-based classifier.
2. Synthetic Textbook Dataset: Generated by GPT-3.5, it contains less than 1 billion tokens of synthetic Python textbooks.
3. Synthetic Exercises Dataset: Comprising around 180 million tokens, this dataset contains Python exercises and their corresponding solutions.

Evaluation

The phi-1 model has been evaluated on various benchmarks, with notable results:

• Pass@1 accuracy of 50.6% on HumanEval.
• Pass@1 accuracy of 55.5% on MBPP (Mostly Basic Python Programs).

These results indicate the model's effectiveness in code generation tasks.