Announcing Minari

Since the introduction of Gym (now Gymnasium) in 2016 and PettingZoo in 2020, these libraries have helped to provide a common API for training libraries and environments to build upon. In line with Farama’s long term goal, described in our Announcing The Farama Foundation blog post, we propose Minari, a dataset API for Offline Reinforcement Learning (Offline RL). Minari provides the capability to create your own environment-based datasets, to download open-source datasets and to upload your own datasets for others to use. This blog post is dedicated to outlining what Offline Reinforcement Learning is, the design philosophy of Minari, and our plan going forward.

You can start using Minari today; see our website, minari.farama.org, for example implementations and tutorials.

Why is Offline Reinforcement Learning important?

The majority of previous Reinforcement Learning research has focused on online learning where an agent/policy actively interacts with an existing environment to update and improve over time. This is commonly achieved through simulations, where the environment is a video game or physics engine, e.g. Atari games or MuJoCo. However, deploying online Reinforcement Learning in the real world, such as in robotics, autonomous driving, energy management or healthcare, is more challenging because the agent must learn through trial and, in particular, error. For example, training a self-driving car to navigate through a city with online reinforcement learning would require the agent to learn from scratch among pedestrians and other drivers, an unacceptable safety hazard.

An alternative method is to train the agent through the use of a large dataset of human driving experience such that the agent can learn safe driving with real data before ever being deployed to the real world. This approach is the foundation of offline reinforcement learning, which has seen an explosion in use and research in the past few years [1]. In comparison to online RL, where agents learn by directly interacting with the environment, agents in Offline RL learn by updating their policy from samples of a static dataset of previously collected data.

The collected dataset can be generated from humans, a suboptimal policy, or any sort of control system that gives actions for the agent. This approach has already shown promising results for robotics, video games, disease mitigation, autonomous driving, generalist agents as well as other industry applications like the recommendation system of Spotify and Amazon’s research on order fraud evaluation for e-commerce.

Within Offline RL there already exist a number of open-source datasets. However, they do not share a common API; for example, Bridge, RoboNet, and VAL for real-world robotic visual control, D4RL and RL Unplugged which contain benchmark datasets of different simulated environments, and CrowdPlay which provides a human-interaction interface to collect datasets from RL environments.

Minari

Today, the Farama Foundation is introducing Minari as one of its core API packages alongside Gymnasium and PettingZoo, to serve as an open-source standard API and reference collection of Offline RL datasets. We believe that by open-sourcing a big collection of standard datasets, researchers can advance the field more efficiently, effectively, and collaboratively.

We aim for Minari to become the de facto API for open-source offline RL datasets that will support the development of new algorithms and provide researchers a common benchmark to compare results. We’re uniquely poised to do this given that we’re a neutral nonprofit with a diverse board of directors, and we already maintain a number of open source RL environments, e.g., Minigrid, Miniworld and Gymnasium-Robotics. This release serves as our first push into the offline RL space, a space that we want to begin supporting at a comparable level to online RL going forward. We plan to integrate Minari into all of the environments that Farama maintains, as well as work with third party libraries to use it, similar to what we do for Gymnasium and PettingZoo.

Furthermore, several major open source RL projects have agreed to switch to using Minari as their standard, and we hope to see many releases along these lines in the future. We’re also planning to deprecate D4RL in favor of Minari. Several D4RL datasets (Adroit Hand, Point Maze and Kitchen) are already included in Minari, and we are actively working on adding the rest.

When designing Minari, we have worked to incorporate several key ideas:

• Interoperability - Our biggest goal for Minari is to enable datasets and tooling to be easily shared among different projects in the field without the need for conversion or reimplementation. We designed the dataset format to be general and extensible to this end, and this initial release has a set of datasets presented in a common standard format so that different learning libraries can integrate them effectively in their framework. Our documentation covers the API in more detail.
• Reproducibility - Beyond the reproducibility improvements from not requiring reimplementation or conversion, Minari also has dedicated tooling (e.g. storing episode seeds and recovering the original environment) so that any user can completely replicate experiments, which is key to the scientific method and has been historically difficult in RL.
• Accessibility - Our datasets, and Minari as a whole, are built to be easily accessible and well-documented. Minari provides a public Farama Google Cloud Platform (GCP) bucket where users will be able to share and download benchmark datasets, as well as a CLI tool to download and list existing datasets. Accessibility allows researchers to share their work with others, making it easier for the community to build on each other’s work.
• Meaningful Datasets - One of the challenges for Offline RL is creating datasets that contain meaningful data. Researchers have recognised that to promote progress in this field datasets need specific design properties such as sparse rewards, suboptimal data, multitask environments and more, see [2, (section VI, A, 1)] for more information. We intend to follow these recommended design factors for each dataset.

Our Plans Going Forward

We envision a bright future for Offline RL with many new applications in different disciplines, and hope that Minari will be an integral part of this process. To further this goal, we have the following high level roadmap:

• Grow the list of built in datasets - Currently, we only provide a small number of datasets, with a focus on porting over all D4RL datasets and soon the CrowdPlay datasets. In the future we anticipate generating more datasets for other Farama environments, and otherwise accepting more novel datasets (including multi-agent ones, as well as vision or language-based datasets).
• Make Minari faster and more efficient - A big priority of ours is to make Minari more efficient. Initial work for this has begun by creating an efficient built-in replay buffer to use with our datasets. For future upgrades we’re exploring supporting vectorized environment collection, distributed computing tasks, integrating more efficient dataset formats than HDF5, and integrating compression and data streaming features.
• Improve the reproducibility workflow - Datasets can be useful for reproducing and comparing different algorithms. However, it is also important to understand how those datasets were generated. We currently provide metadata in each dataset to recreate or generate more data such as episode seeds or retrieval of the original Gymnasium environment. We would like to keep adding more reproducibility features such as access to stored models of behavior policies if possible. We also plan to develop well-defined testing procedures to strengthen the reliability of the dataset standards. We want to make sure that the datasets created and shared with Minari follow the standards so that other projects can trust their usage.

A lot of work remains to be done, and we’re continuing to develop Minari. If you want to be part of this journey, we would love to hear from you and look forward to your contributions. If you have questions or want to get involved, the best way to get in touch with us is to join our discord server. We hope to see you there.

[1] Levine, Sergey, et al. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”. ArXiv [Cs.LG], 2020, http://arxiv.org/abs/2005.01643. arXiv.

[2] R. F. Prudencio, M. R. O. A. Maximo and E. L. Colombini, “A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems,” in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2023.3250269.