Next Generation Infrastructure for Foundation Models

Purpose and goal:

The project aims to develop the next generation infrastructure to train and implement foundation models such as BERT, GPT-3, Dall-E and CLIP.

The planned infrastructure will be based on a decentralized learning architecture to work around many of the limitations found in the current centralized infrastructure. Expected results and effects: The proposed infrastructure will have higher performance, increase collaboration between

stakeholders, lead to a hardware-agnostic solution, have better privacy and security, as well as reduced costs and increased availability.

The project will have a positive impact on the global foundation model ecosystem and has high potential for commercialization. Approach and implementation:

The project will create a prototype and demonstrate the next generation infrastructure for foundation models, controlled by a central server at AI Sweden. The project has two work packages where WP1 will carry out benchmarks on the Swedish GPT-SW3 model and WP2 will develop the decentralized architecture. For dissemination there is also a work package, as spreading knowledge is also an objective.