Make to Soong

1. What is the AOSP Make Build System?

The Make build system was the original system used by AOSP to build Android projects. It relies on GNU Make, a tool that controls the build process based on a set of rules defined in Makefiles. These files specify how to compile and link the program. For Android:

• Makefiles: These are text files used by make to determine how to build targets (such as executables, libraries, etc.). It contains instructions, dependencies, and recipes for how components should be built.

• The AOSP build system had a set of pre-written Makefiles to define how each module (apps, libraries, etc.) should be compiled and built.

2. Why did AOSP Move from Make to Soong?

The migration from the Make-based build system to the Soong build system happened because Make had several limitations:

• Scaling Issues: As Android grew larger and more complex, the Make build system showed performance bottlenecks when building massive codebases with thousands of modules and dependencies.

• Complex Dependency Management: The build dependencies in a Makefile can become very complex and hard to debug, leading to inaccurate or incomplete builds if not written carefully.

• Parallelism: While Make can run parallel jobs, it wasn’t designed with multi-core scalability in mind, which is essential for improving build times in modern software projects.

• Inflexibility: The Make system is less modular and harder to extend. Writing new build rules and handling edge cases in Makefiles was challenging.

Because of these issues, AOSP switched to the Soong build system, which is built on ninja, a fast and parallel build tool optimized for large projects.

3. The Soong Build System

Soong is the replacement for the Make build system in AOSP. Soong is designed specifically for Android and integrates deeply into the platform. It offers several advantages over the Make-based build system:

• Performance: Soong is faster than Make because it generates ninja files, which are very efficient and optimized for parallel builds. Ninja is known for its speed in handling builds, especially in multi-core systems.

• Modularity: Soong allows for a cleaner and more modular approach to defining build targets through .bp (Blueprint) files, which are more maintainable compared to Makefiles.

• Better Dependency Management: Soong manages dependencies more effectively, reducing build complexity and ensuring correctness.

• Tooling: Soong also has better tooling for developers to debug and understand the build system.

4. How does Kati Convert Makefiles to Ninja Files?

To ensure backward compatibility with legacy code, the AOSP project uses Kati, a tool that translates Makefiles into ninja build files. Let’s break down the process:

• Kati is essentially a reimplementation of GNU Make with the ability to output a ninja build file.

• Kati Workflow:

  1. It reads the existing Makefiles (just like GNU Make does).

  2. Instead of executing the build directly, Kati processes the Makefiles and converts them into a build.ninja file. This file is an intermediate build file that ninja uses to efficiently orchestrate the actual build.

  3. The ninja tool then takes over the build process by executing the commands defined in the build.ninja file. Ninja is highly efficient at orchestrating these builds, leveraging parallelism and reducing redundant builds.

5. How does Soong Convert .bp Files to Ninja Files?

Soong has its own file format: .bp (Blueprint) files. These files describe build configurations in a declarative, JSON-like format. They replace the functionality of Makefiles in Soong. The process of converting .bp files to ninja files is as follows:

• Blueprint to Soong: Soong parses .bp files using the Blueprint framework, which is a Python-based build configuration parser. These .bp files define all the build modules (libraries, binaries, APKs) and their properties.

• Soong Workflow:

  1. Soong reads the .bp files to understand the build modules, their dependencies, and configurations.

  2. It converts these .bp files into a corresponding set of ninja build files.

  3. The ninja build files are executed by the ninja build system, which builds the final Android system (APKs, libraries, binaries).

6. Ninja Build System

The ninja build system is a crucial component in both the Kati and Soong workflows. Ninja is a tool specifically designed to handle very large builds efficiently by focusing on parallelism and avoiding unnecessary builds. Its core advantages are:

• Speed: Ninja processes build files quickly and determines the minimal set of tasks to execute, avoiding unnecessary rebuilds.

• Parallelism: It scales well with multi-core processors, making it much faster for large-scale projects.

• Incremental Builds: It supports efficient incremental builds, meaning if only a few files have changed, ninja can rebuild only those parts of the project that are affected.

7. Components Involved in the Transition

The transition from Make to Soong involved several components:

• Kati: Maintained backward compatibility with old Make-based build configurations by translating Makefiles to ninja files.

• Soong: Became the modern build system for Android, using .bp files for configuration and producing ninja files for the actual build.

• Blueprint: A configuration parsing framework that Soong uses to interpret .bp files.

• Ninja: The core tool that executes the build process after Kati or Soong generates the build.ninja files.

8. Entire Process Summary

• Kati reads Makefiles, converts them into ninja files, and then hands over the process to ninja.

• Soong reads .bp files, converts them into ninja files, and then ninja executes the build.

• Ninja orchestrates the actual build by running commands to compile, link, and package the Android system components (APKs, libraries, binaries, etc.).

Why the Change Matters

The switch from Make to Soong is about improving scalability, performance, and modularity for Android development. Soong makes it easier to manage the growing complexity of Android and to optimize build times on modern hardware.

In conclusion, the AOSP build system has evolved to better handle the massive codebase that Android represents. Moving from Make to Soong (with ninja at its core) represents a shift towards faster, more scalable builds that are easier to manage and configure.

Feel free to ask for further clarification on any specific part!