Note

Phi-1.5 is intended for research purposes please use in accordance with license terms.

Model is not fine-tuned either for instruction following or through reinforcement learning from human feedback. Due to the lack of instruction finetuning, model typically does not stop properly, and sometimes generates more content in the style of its training data.

Introduction

Phi-1.5 is a 1.3 billion parameter language model that has been trained primarily on a specially curated "textbook-quality" synthetic dataset, which was designed to be of high quality and to provide a diverse range of examples for the model to learn from. The model has been found to perform at a level comparable to models that are 5x larger, and it has been shown to surpass most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding.

Phi-1.5 Model

Phi-1.5 is a Transformer-based language model with an architecture that is exactly the same as the phi-1 model.

• Architecture: a Transformer-based model with next-word prediction objective
• Dataset size: 30B tokens
• Training tokens: 150B tokens
• Precision: fp16

Note: Phi-1.5 has not been fine-tuned either for instruction following or through reinforcement learning from human feedback.

Usecases

Phi-1.5 can be used for a variety of natural language processing tasks, including text classification, sentiment analysis, and language generation. It is particularly well-suited for complex reasoning tasks such as grade-school mathematics and basic coding.

Being a base model, it often produces irrelevant text following the main answer. In the following example, answers have been truncated for illustrative purposes only.

• Language modeling: Phi-1.5 can be used to generate coherent and grammatically correct text, which can be useful for tasks such as text completion, summarization, and translation. 

Prompt: Write a detailed analogy between mathematics and a lighthouse.

Response: Mathematics is like a lighthouse, guiding us through the vast ocean of numbers and calculations. Just as a lighthouse illuminates the darkness, mathematics provides us with a clear path to navigate through complex problems. It helps us make sense of the world around us, just like a lighthouse helps ships find their way home.

• Question answering: Phi-1.5 can be used to answer questions based on a given context, which can be useful for tasks such as customer support, chatbots, and virtual assistants. 

Why are theoretical computer scientists working on deep learning now?

Response: Theoretical computer scientists are working on deep learning now because it has the potential to revolutionize the way we process and analyze data. By developing algorithms that can learn and make predictions based on complex patterns, deep learning can help us solve a wide range of problems, from image recognition to natural language processing.

• Reasoning tasks: Phi-1.5 has been found to perform well on complex reasoning tasks such as grade-school mathematics and basic coding, which can be useful for tasks such as automated theorem proving, logical reasoning, and problem solving.

Prompt: Suppose Alice originally had 3 apples, then Bob gave Alice 7 apples, then Alice gave Cook 5 apples, and then Tim gave Alice 3x the amount of apples Alice had. How many apples does Alice have now? Think step by step.

Response: 1. Alice originally had 3 apples. 2. Bob gave Alice 7 apples, so Alice now has 3 + 7 = 10 apples. 3. Alice gave Cook 5 apples, so Alice now has 10 - 5 = 5 apples. 4. Tim gave Alice 3x the amount of apples Alice had, so Tim gave Alice 3 * 5 = 15 apples. 5. Alice now has 5 + 15 = 20 apples.

Dataset 

Phi-1.5 was trained on 150 billion tokens, with 20% from phi-1's training data(7B tokens) and 80% from the newly created synthetic, “textbook-like” data (roughly 20B tokens) for the purpose of teaching common sense reasoning and general knowledge of the world (science, daily activities, theory of mind, etc.).

Phi-1 and phi-1.5 leads us to the conclusion that the creation of a robust and comprehensive dataset demands more than raw computational power: It requires intricate iterations, strategic topic selection, and a deep understanding of knowledge gaps to ensure quality and diversity of the data.

Evaluation

Phi-1.5 has been found to perform at a level comparable to models that are 10 times its size and have been trained on datasets more than 10 times larger. In fact, phi-1.5's performance on natural language tasks is comparable to models that are 5 times larger.

The model is evaluated on standard natural language benchmarks, including common sense reasoning, language understanding, mathematics and coding. Phi-1.5 achieves comparable results to Llama2-7B, Falcon-7B and Vicuna-13B on nearly all of the benchmarks.

• Generate Inaccurate Code and Facts: The model often produces incorrect code snippets and statements. Users should treat these outputs as suggestions or starting points, not as definitive or accurate solutions.
• Limited Scope for code: If the model generates Python scripts that utilize uncommon packages or scripts in other languages, we strongly recommend users manually verify all API uses.
• Unreliable Responses to Instruction: The model has not undergone instruction fine-tuning. As a result, it may struggle or fail to adhere to intricate or nuanced instructions provided by users.
• Language Limitations: The model is primarily designed to understand standard English. Informal English, slang, or any other language outside of English might pose challenges to its comprehension, leading to potential misinterpretations or errors in response.
• Potential Societal Biases: Regardless of the safe data used for its training, the model is not entirely free from societal biases. There's a possibility it may generate content that mirrors these societal biases, particularly if prompted or instructed to do so. We urge users to be aware of this and to exercise caution and critical thinking when interpreting model outputs.
• Toxicity: Despite that the model is trained with carefully selected data, the model can still produce harmful content if explicitly prompted or instructed to do so. We chose to release the model for research purposes only -- We hope to help the open-source community develop the most effective ways to reduce the toxicity of a model directly after pretraining.