What it does
This repository collects research from Microsoft on "foundation models", large AI models trained on huge amounts of data that can then be adapted to many tasks. It covers language, vision, speech, documents and models that combine them.
Projects include LayoutLM for understanding scanned documents and forms, TrOCR for reading text in images, BEiT for images, VALL-E for text-to-speech, the Kosmos multimodal models, and new model designs such as BitNet, RetNet and LongNet.
Many projects have their own folder with code and pretrained models. Most are aimed at researchers rather than everyday users.
Who it's for
AI researchers and engineers who want to study or build on Microsoft's published models.
Can I use it without coding?
Developers only. Research code; using it needs programming and machine-learning experience. New to this? Start here.
Can I run it?
- Cost
- FreeNothing to pay or sign up for.
- API key
- Not needed
Try it
Using Claude Code, Cursor or another AI coding assistant? Paste the prompt and it will walk you through the setup.New to this?
Before you install
- Only download from the official GitHub page. Copies with similar names exist for many popular projects.
- License (MIT): Very permissive: free to use, change and share, including for business.
- More on staying safe
Where people are talking about it
- BitNet: 1-bit Transformers for Large Language ModelsResearch paper · Oct 2023
- Kosmos-2.5: a multimodal literate modelResearch paper · Sept 2023
- LongNet: Scaling Transformers to 1,000,000,000 TokensResearch paper · Jul 2023
- RetNet: Retentive Network: A Successor to Transformer for Large Language ModelsResearch paper · Jul 2023
- Microsoft Kosmos-1: A Multimodal Large Language ModelHacker News · Mar 2023
- Kosmos-1: A Multimodal Large Language Model (MLLM)Research paper · Feb 2023
- VALL-E: a language modeling approach for text to speech synthesisResearch paper · Jan 2023
- Foundation Transformers (Magneto)Research paper · Oct 2022
- BEiT-3: a general-purpose multimodal foundation modelResearch paper · Aug 2022