Understanding TWRP

TWRP is the leading open-source and community developed Android recovery, a fallback system for manually applying updates, factory reset the device, or performing rescue operations. Custom recoveries may provide features such as backups, root shell, or advanced partition management. TWRP tries to make them accessible, but they require knowledge about how Android stores data. This article explains how the main recovery features work, starting with an overview of Android storage partitions.

§

Introduction

Before diving into any specific feature, the key to learn how the recovery works is understanding how Android manages data. This section presents the main storage partitions, along with their purpose. For more information, see AOSP architecture.

§

Storage partitions

Android works just like a regular Linux system. It has storage drives, partitions, and filesystems. LineageOS does not contain a lot of Unix tools, but TWRP provides BusyBox, including fdisk, to inspect drive partitions. With your device booted on TWRP, and connected to your computer, initiate a shell session through ADB:

$ adb shell
twrp:~# 

You can then list the partitions on the internal storage:

twrp:~# fdisk -l /dev/block/mmcblk0
Disk /dev/block/mmcblk0: 30535680 sectors, 2622M

Number  Start (sector)  End (sector)   Size   Name
    24          658528        724063  32.0M   boot
    25          724064       3869791  1536M   system
    26         3869792       3935327  32.0M   recovery
    29         4259840       4567039   150M   cache
    31         4568064      30535646  12.3G   userdata

And on the external storage:

twrp:~# fdisk -l /dev/block/mmcblk1
Disk /dev/block/mmcblk1: 63.8 GB, 63864569856 bytes

Device                Start   End    Blocks   System
/dev/block/mmcblk1p1      1  7765  62366703+  Win95 FAT32 (LBA)

The internal partition layout is fixed and device specific. Newer devices support the A/B partition scheme to allow seamless system updates, with automatic rollbacks on failure. In this mode, the system partitions are duplicated between two slots, and the data partitions are shared between them. The system installs updates in the background on the inactive slot, which becomes active at the next boot, without going through the recovery.

§

Storage locations

The partitions host a number of storage locations: some of them take a full partition, others are subfolders of the same partition. The following table lists the main storage locations in Android:

The Recovery partition is where TWRP is installed, the Boot and System partitions contain the OS. These partitions are reserved for the system and do not contain any user data.

The caches contain automatically generated data that you do not need to care about. When installing a new ROM, you can safely wipe them.

§

User and application data

The Data partitions contains user data, organized into multiple subfolders:

• /data/app stores APKs installed by the user.
• /data/data contains data generated by installed applications.
• /data/media is the Internal Storage, containing shared media files (when an application asks access to "photos, medias, and files," it corresponds to this location).

The private application storage is mounted at /data/data. This location is inaccessible to regular applications without root privileges, except for a single subdirectory dedicated to each application. It contains almost everything an application needs to save on a device. For instance, Android settings, phone calls, SMS history, Firefox profile and browsing data, are all stored there.

Assuming your phone is rooted, you can list the open Firefox tabs:

$ adb root
restarting adbd as root
$ adb exec-out \
	cat /data/data/org.mozilla.firefox/files/mozilla_components_session_storage_gecko.json |\
	jq . |\
	head
{
  "version": 2,
  "selectedTabId": "83570200-6413-48bf-8700-04c302c0a731",
  "sessionStateTuples": [
    {
      "session": {
        "url": "https://dzx.fr/",
        "uuid": "b59f6279-c450-42ac-b9fc-4a2680b3e7fb",
        "parentUuid": "",
        "title": "dzx.fr",

You can also select the last 3 rows in the call log database:

$ adb exec-out sqlite3 /data/data/com.android.providers.contacts/databases/calllog.db \
	'SELECT _id, number, DATETIME(ROUND(date / 1000), "unixepoch")
	 FROM calls
	 ORDER BY _id DESC
	 LIMIT 3'
3063|0123456789|2021-08-24 14:13:36
3062|0123456789|2021-08-23 09:16:25
3061|0123456789|2021-08-22 12:57:03

Internal Storage, mounted at /data/media, contains user files (downloads, pictures, music), but also some application data (such as shared media files in instant messaging application). The file manager on your phone can display and manage its content, and any application can request access permission to it.

Finally, MicroSD is the expansion storage, mounted under /storage. It has the same access permissions as Internal Storage, and you can optionally link them both (but that removes the ability to transfer data directly on the microSD card from an external device).

§

Main menu

The following table represents the main menu options (the links point to the relevant sections):

§

Install

There are two ways to install a package on an Android device:

• Push the file to the device beforehand.
• Sideload its content through ADB.

§

Local file

The Install command installs a package on a device. The command fastboot flash PARTITION IMAGE with a .img flashes a single partition with the content of the image file (akin to writing the raw bytes with dd to the partition). With a .zip installation package, the recovery can flash multiple partitions, but also mount them and extract files into it.

Custom ROMs contain an extension mechanism to automatically reinstall additional packages after an update. It relies on scripts placed in /system/addon.d, executed during the update process. For instance, the Magisk rooting utility patches the boot image and installs a manager application from its addon.d.sh script.

The LineageOS installation package populates the Boot and System partitions. The LineageOS recovery accesses only these two partitions, and as a consequence, it cannot run addon.d scripts that depend on Data such as Magisk's. The advantage of TWRP is that it also mounts the Data partition, and can even decrypt it on some devices.

§

Sideload

ADB Sideload installs packages right from a connected computer. It performs the same tasks as Install, without having to manually copy the installation packages beforehand:

1. Start the server in Advanced Features > ADB Sideload. You can safely check Wipe Dalvik Cache and Wipe Cache.

2. Send a package from your computer:

   $ adb sideload package.zip
   serving: 'package.zip'  (100%)
   

§

Backup and restore

The Backup and Restore features can save a partition to a file, and later restore it. They are not designed to transfer data from one device to another, as the partition layout is device-specific.

If you want to quickly restore a device to its previous state after an update, backup Boot, System, and Data (note that Internal Storage is excluded). If the new system does not work as expected, you will be able to restore these partitions from the backup file.

§

Data

For more granular application data backups, with the ability to restore on another device, you can try one of these free and open source solutions:

• SeedVault (has to be built into your ROM).
• OAndBackupX (requires a rooted device).

§

Internal Storage

The only builtin way to "backup" Internal Storage is to manually copy the files with adb pull. The alternative is to use a synchronization tool, such as the following free and open source solutions:

• Nextcloud (you need to have a server instance).
• Syncthing (peer-to-peer).

§

Wipe

Now that you can back up your device, we can get to the most dangerous part. The Wipe menu can be slightly confusing, but it is extremely important to understand its mode of operation. If you are unsure about the partition content, refer to the Android storage section.

§

Factory Reset

Factory Reset wipes Data (excluding Internal Storage), Cache, and Dalvik Cache. Although Internal Storage is excluded, it still removes important data from installed apps (application and device settings, call history, SMS, contacts, calendars, the user installed application, etc), but not shared user data (documents, downloads, music, photos, exported application data).

Android contains a Factory reset option, warning that it also wipes Internal Storage. Actually, its effects depend on the recovery image that performs the operation:

• With a stock recovery image, the operation formats all of Data.
• With TWRP, the operation deletes Data, Cache, and Dalvik Cache, excluding Internal Storage.

To stay on the safe side, always use the option directly from TWRP and make backups. For more information, see: "What is a data/media device?"

§

Format Data

Format Data formats all of Data, including Internal Storage! This option is useful for:

• Removing the device encryption (all plaintext data will be lost).
• Removing user data before disposing of the device (but that does not prevent recovery if its content was unencrypted).

§

Advanced Wipe

Finally, Advanced Wipe allows to selectively wipe partitions. You can use Repair or change filesystem to fix partitions that fail to mount properly.

§

Conclusion

To update your device to the next minor version:

• Wipe Cache and Dalvik Cache.
• Install or ADB Sideload the new image.

To update your device to the next major version or to another ROM:

• Factory Reset.
• (Optionally) Wipe System.
• Install the new image.

To erase your device before selling it or to remove encryption:

• Format Data.
• (Optionally) Reinstall the stock ROM.

Final recommendations:

• Backup application data with SeedVault or OAndBackupX.
• Synchronize shared user data with Nextcloud or Syncthing.
• Encrypt your device (assuming you do not need Magisk's addon.d persistence, or TWRP can decrypt Data).