Overview

Falcon is a new family of state-of-the-art language models created by the Technology Innovation Institute. The Falcon family is composed of two base models: Falcon-40B and Falcon-40B. The 40B parameter model currently tops the charts of the Open LLM Leaderboard, while the 40B parameter model is the best in its weight class. Instruct versions of the models, Falcon-40B-Instruct and Falcon-40B-Instruct have been finetuned on instructions and conversational data; they thus lend better to popular assistant-style tasks.

Falcon-40B-Instruct Model

Falcon-40B-Instruct is a 40B parameters causal decoder-only model based on Falcon-40B and finetuned on a mixture of chat/instruct datasets, making them particularly suitable for popular assistant-style tasks. Falcon-40B has been trained on 1 trillion tokens, in line with modern models optimising for inference. The architecture of Falcon was optimized for performance and efficiency. Combining high-quality data with these optimizations, Falcon significantly outperforms GPT-3 for only 75% of the training compute budget—and requires a fifth of the compute at inference time.

Use Cases

Here are some use cases of the Falcon-40B-Instruct model.

• Chatbot Assistance: Falcon-40B-Instruct can be deployed as a chatbot to provide real-time conversational support, answer user queries, and engage in interactive conversations across various industries such as e-commerce, customer service, and education.
• Instructional Guidance: The model's fine-tuning on instruct datasets makes it highly effective in providing clear and concise instructions, step-by-step guidance, and explanations for complex tasks. It can be utilized in applications like virtual tutors, DIY platforms, or online learning platforms.
• Content Generation: Falcon-40B-Instruct can generate high-quality content for various purposes, such as blog articles, social media posts, product descriptions, and marketing materials. Its language proficiency and creativity can assist businesses in producing engaging and informative content more efficiently.
• Customer Support Automation: The model can automate customer support processes by handling common queries, providing instant responses, and guiding users through troubleshooting steps. This reduces the need for human intervention, improves response time, and enhances customer satisfaction.
• Language Learning and Translation: Falcon-40B-Instruct's multilingual capabilities make it valuable for language learning applications. It can generate language exercises, provide explanations, and assist learners in practicing their skills. Additionally, the model can be utilized for language translation tasks, facilitating communication and understanding across different languages.

Dataset

Falcon models are their training data, predominantly based (>80%) on RefinedWeb — a novel massive web dataset based on CommonCrawl. Falcon RefinedWeb was created to serve as an English large-scale dataset for the pretraining of large language models. It may be used on its own, or augmented with curated sources (e.g., Wikipedia, StackOverflow). It was built on top of CommonCrawl, leveraging stringent filtering and extensive deduplication. RefinedWeb is focused on scaling and improving the quality of web data, leveraging large-scale deduplication and strict filtering to match the quality of other corpora. The Falcon models still include some curated sources in their training (such as conversational data from Reddit), but significantly less so than has been common for state-of-the-art LLMs like GPT-3 or PaLM. RefinedWeb is also "multimodal-friendly": it contains links and alt texts for images in processed samples.

Examples

Q&A

Prompt: Hey Falcon! Any recommendations for my holidays in Abu Dhabi?
Response: Abu Dhabi is a great destination for a holiday! Some recommendations would be to visit the Abu Dhabi Mall, the world's largest indoor theme park, the Abu Dhabi International Exhibition Centre, and the Sheikh Zayed Mosque. You could also take a desert safari tour, go shopping at the local markets, or try some of the local cuisine. Have a great trip!
Prompt: Who was president of the United States in 1955?
Response: The president of the United States in 1955 was Dwight D. Eisenhower.

Grammar correction

Prompt:
Correct this to standard English: She no went to the market.
Response:
She did not go to the market.

Summarize

Prompt:
Summarize this,
Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half timesthat of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history. It is named after the Roman god Jupiter.[19] When viewed from Earth, Jupiter can be bright enough for its reflected light to cast visible shadows,[20] and is on average the third-brightest
natural object in the night sky after the Moon and Venus.
Response:
Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half times that of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history.

Evaluation Results

HuggingFace has a leaderboard for open LLMs where comparison of quantitative and qualitative performance of open LLMs on 4 different benchmark datasets The Falcon-40B base and instruct models are very strong, and currently top on the LLM leaderboard :trophy:!

What makes the FalconModel special?

• Multiquery Attention: The Falcon models incorporate an interesting feature called multi-query attention. Unlike the traditional multi-head attention scheme, which has separate query, key, and value embeddings per head, multi-query attention shares a single key and value across all attention heads. This innovation enhances the scalability of inference, significantly reducing memory costs and enabling optimizations like statefulness. The smaller K, V-cache during autoregressive decoding contributes to these benefits.
• Multilingual Capabilities: Falcon models also have multilingual capabilities. It understands English, German, Spanish, and French and has limited capabilities in some European languages such as Dutch, Italian, Romanian, Portuguese, Czech, Polish, and Swedish.
• Consumer Hardware Accessibility: Falcon-40B Instruct requires less GPU memory, making it more accessible on consumer hardware. Despite its power, Falcon uses only 75 percent of GPT-3’s training compute.
• Reduced Compute at Inference Time: It requires one-fifth of the compute at inference time compare to GPT-3.
• Fine-Tuned for Assistant Tasks: Falcon models, have been fine-tuned on instructions and conversational data, which enhances their performance on assistant-style tasks.

Disclaimer

Please be advised that this model utilizes wrapped Artificial Intelligence (AI) provided by TogetherAI (the "Vendor"). These AI models may collect, process, and store data as part of their operations. By using our website and accessing these AI models, you hereby consent to the data practices of the Vendor. We do not have control over the data collection, processing, and storage practices of the Vendor. Therefore, we cannot be held responsible or liable for any data handling practices, data loss, or breaches that may occur. It is your responsibility to review the privacy policies and terms of service of the Vendor to understand their data practices. You can access the Vendor's privacy policy and terms of service at https://www.togetherai.com/legal/privacy-policy.

We disclaim all liability with respect to the actions or omissions of the Vendor, and we encourage you to exercise caution and to ensure that you are comfortable with these practices before utilizing the AI models hosted on our site.