Overview

Falcon is a new family of state-of-the-art language models created by the Technology Innovation Institute. The Falcon family is composed of two base models: Falcon-40B and Falcon-7B. The 40B parameter model currently tops the charts of the Open LLM Leaderboard, while the 7B parameter model is the best in its weight class. Instruct versions of the models, Falcon-7B-Instruct and Falcon-40B-Instruct have been finetuned on instructions and conversational data; they thus lend better to popular assistant-style tasks.

Falcon-7B-Instruct Model

Falcon-7B-Instruct is a 7B parameters causal decoder-only model based on Falcon-7B and finetuned on a mixture of chat/instruct datasets, making them particularly suitable for popular assistant-style tasks. Falcon-7B has been trained on 1.5 trillion tokens, in line with modern models optimising for inference. The architecture of Falcon was optimized for performance and efficiency. Combining high-quality data with these optimizations, Falcon significantly outperforms GPT-3 for only 75% of the training compute budget—and requires a fifth of the compute at inference time.

Use Cases

Here are some use cases of the Falcon-7B-Instruct model.

• Chatbot Assistance: Falcon-7B-Instruct can be deployed as a chatbot to provide real-time conversational support, answer user queries, and engage in interactive conversations across various industries such as e-commerce, customer service, and education.
• Instructional Guidance: The model's fine-tuning on instruct datasets makes it highly effective in providing clear and concise instructions, step-by-step guidance, and explanations for complex tasks. It can be utilized in applications like virtual tutors, DIY platforms, or online learning platforms.
• Content Generation: Falcon-7B-Instruct can generate high-quality content for various purposes, such as blog articles, social media posts, product descriptions, and marketing materials. Its language proficiency and creativity can assist businesses in producing engaging and informative content more efficiently.
• Customer Support Automation: The model can automate customer support processes by handling common queries, providing instant responses, and guiding users through troubleshooting steps. This reduces the need for human intervention, improves response time, and enhances customer satisfaction.
• Language Learning and Translation: Falcon-7B-Instruct's multilingual capabilities make it valuable for language learning applications. It can generate language exercises, provide explanations, and assist learners in practicing their skills. Additionally, the model can be utilized for language translation tasks, facilitating communication and understanding across different languages.

Dataset

Falcon models are their training data, predominantly based (>80%) on RefinedWeb — a novel massive web dataset based on CommonCrawl.

Falcon RefinedWeb was created to serve as an English large-scale dataset for the pretraining of large language models. It may be used on its own, or augmented with curated sources (e.g., Wikipedia, StackOverflow). It was built on top of CommonCrawl, leveraging stringent filtering and extensive deduplication.

RefinedWeb is focused on scaling and improving the quality of web data, leveraging large-scale deduplication and strict filtering to match the quality of other corpora. The Falcon models still include some curated sources in their training (such as conversational data from Reddit), but significantly less so than has been common for state-of-the-art LLMs like GPT-3 or PaLM. RefinedWeb is also "multimodal-friendly": it contains links and alt texts for images in processed samples.

Examples

Q&A

Prompt: Hey Falcon! Any recommendations for my holidays in Abu Dhabi?

Response: Abu Dhabi is a great destination for a holiday! Some recommendations would be to visit the Abu Dhabi Mall, the world's largest indoor theme park, the Abu Dhabi International Exhibition Centre, and the Sheikh Zayed Mosque. You could also take a desert safari tour, go shopping at the local markets, or try some of the local cuisine. Have a great trip!

Prompt: Who was president of the United States in 1955?

Response: The president of the United States in 1955 was Dwight D. Eisenhower.

Grammar correction

Prompt:
Correct this to standard English: She no went to the market.

Response:
She did not go to the market.

Summarize

Prompt:
Summarize this,

Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half timesthat of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history. It is named after the Roman god Jupiter.[19] When viewed from Earth, Jupiter can be bright enough for its reflected light to cast visible shadows,[20] and is on average the third-brightest
natural object in the night sky after the Moon and Venus.

Response:
Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half times that of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history.

Evaluation Results

HuggingFace has a leaderboard for open LLMs where comparison of quantitative and qualitative performance of open LLMs on 4 different benchmark datasets

The Falcon-40B base and instruct models are very strong, and currently top on the LLM leaderboard 🏆!

Falcon-7B and instruct models, we see that the base model is better than llama-7b and edges out MosaicML's mpt-7b to become the current best pretrained LLM at this scale.

What makes the Falcon-7B-Instruct Model special?

• Multiquery Attention: The Falcon models incorporate an interesting feature called multi-query attention. Unlike the traditional multi-head attention scheme, which has separate query, key, and value embeddings per head, multi-query attention shares a single key and value across all attention heads. This innovation enhances the scalability of inference, significantly reducing memory costs and enabling optimizations like statefulness. The smaller K, V-cache during autoregressive decoding contributes to these benefits.

• Multilingual Capabilities: Falcon models also have multilingual capabilities. It understands English, German, Spanish, and French and has limited capabilities in some European languages such as Dutch, Italian, Romanian, Portuguese, Czech, Polish, and Swedish.

• Consumer Hardware Accessibility: Falcon-7B Instruct requires less GPU memory, making it more accessible on consumer hardware. Despite its power, Falcon uses only 75 percent of GPT-3’s training compute.

• Reduced Compute at Inference Time: It requires one-fifth of the compute at inference time compare to GPT-3. 

• Fine-Tuned for Assistant Tasks: Falcon models, including Falcon-7B Instruct, have been fine-tuned on instructions and conversational data, which enhances their performance on assistant-style tasks.