Skip to content

Commit

Permalink
Split dm-verify hash tree logic out of avb module
Browse files Browse the repository at this point in the history
* Refactor hash tree computation to work on a preallocated hash tree
  buffer. This makes it possible to partially update a hash tree, which
  is now supported.

* Add new subcommands for working with hash trees. There's no standard
  header format for dm-verity information, so these commands write hash
  tree files with a custom header. The commands are not really useful
  outside of debugging avbroot's hash tree implementation.

  Using AVB was considered, but it has no support for the hash tree data
  living in a separate file from the input. If other parties agree on a
  standard header in the future, avbroot will switch to that format.

* Add tests for the hash tree implementation.

Signed-off-by: Andrew Gunnerson <[email protected]>
  • Loading branch information
chenxiaolong committed Dec 18, 2023
1 parent ce43b3c commit 7ea8e8f
Show file tree
Hide file tree
Showing 9 changed files with 1,020 additions and 203 deletions.
48 changes: 47 additions & 1 deletion README.extra.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ The number of parity bytes (between 2 and 24, inclusive) can be configured using
### Updating FEC data

```bash
avbroot fec update -i <input data file> -f <output FEC file> [-r <start> <end>]...
avbroot fec update -i <input data file> -f <FEC file> [-r <start> <end>]...
```

This will update the FEC data corresponding to the specified regions. This can be significantly faster than generating new FEC data from scratch for large files if the regions where data was modified are known.
Expand All @@ -202,3 +202,49 @@ avbroot fec repair -i <input/output data file> -f <input FEC file>
This will repair the file in place. As described above, in each column, up to `parity / 2` bytes can be corrected.

Note that FEC is **not** a replacement for checksums, like SHA-256. When there are too many errors, the file can potentially be "successfully repaired" to some incorrect data.

## `avbroot hash-tree`

This set of commands is for working with dm-verity hash tree data. They are not especially useful outside of debugging avbroot itself because the output format is custom. There is a custom header that sits in front of the standard dm-verity hash tree data.

| Offsets | Type | Description |
|------------|--------|--------------------------------|
| 0..16 | ASCII | `avbroot!hashtree` magic bytes |
| 16..18 | U16LE | Version (currently 1) |
| 18..26 | U64LE | Image size |
| 26..30 | U32LE | Block size |
| 30..46 | ASCII | Hash algorithm |
| 46..48 | U16LE | Salt size |
| 48..50 | U16LE | Root digest size |
| 50..54 | U32LE | Hash tree size |
| (Variable) | BINARY | Salt |
| (Variable) | BINARY | Root digest |
| (Variable) | BINARY | Hash tree |

For more information on the hash tree data, see the [Linux kernel documentation](https://docs.kernel.org/admin-guide/device-mapper/verity.html#hash-tree) or avbroot's implementation in [`hashtree.rs`](./avbroot/src/format/hashtree.rs).

### Generating hash tree

```bash
avbroot hash-tree generate -i <input data file> -H <output hash tree file>
```

The default behavior is to use a block size of 4096, the `sha256` algorithm, and an empty salt. These can be changed with the `-b`, `-a`, and `-s` options, respectively.

All parameters needed for verification are included in the hash tree file's header.

### Updating hash tree

```bash
avbroot hash-tree update -i <input data file> -H <hash tree file> [-r <start> <end>]...
```

This will update the hash tree data corresponding to the specified regions. This can be significantly faster than generating new hash tree data from scratch for large files if the regions where data was modified are known.

### Verifying a file

```bash
avbroot hash-tree verify -i <input data file> -H <input hash tree file>
```

This will check if the input file has any corrupted blocks. Currently, the command cannot report which specific blocks are corrupted, only whether the file is valid.
4 changes: 3 additions & 1 deletion avbroot/src/cli/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use std::sync::atomic::AtomicBool;
use anyhow::Result;
use clap::{Parser, Subcommand};

use crate::cli::{avb, boot, completion, cpio, fec, key, ota};
use crate::cli::{avb, boot, completion, cpio, fec, hashtree, key, ota};

#[allow(clippy::large_enum_variant)]
#[derive(Debug, Subcommand)]
Expand All @@ -18,6 +18,7 @@ pub enum Command {
Completion(completion::CompletionCli),
Cpio(cpio::CpioCli),
Fec(fec::FecCli),
HashTree(hashtree::HashTreeCli),
Key(key::KeyCli),
Ota(ota::OtaCli),
/// (Deprecated: Use `avbroot ota patch` instead.)
Expand All @@ -44,6 +45,7 @@ pub fn main(cancel_signal: &AtomicBool) -> Result<()> {
Command::Completion(c) => completion::completion_main(&c),
Command::Cpio(c) => cpio::cpio_main(&c, cancel_signal),
Command::Fec(c) => fec::fec_main(&c, cancel_signal),
Command::HashTree(c) => hashtree::hash_tree_main(&c, cancel_signal),
Command::Key(c) => key::key_main(&c),
Command::Ota(c) => ota::ota_main(&c, cancel_signal),
// Deprecated aliases.
Expand Down
5 changes: 1 addition & 4 deletions avbroot/src/cli/avb.rs
Original file line number Diff line number Diff line change
Expand Up @@ -512,10 +512,7 @@ fn verify_and_repair(
status!("Verifying hash tree descriptor{suffix}");

match d.verify(&file, cancel_signal) {
Err(
e @ avb::Error::InvalidRootDigest { .. }
| e @ avb::Error::InvalidHashTree { .. },
) if repair => {
Err(e @ avb::Error::HashTree(_)) if repair => {
warning!("Failed to verify hash tree descriptor{suffix}: {e}");
warning!("Attempting to repair using FEC data{suffix}");

Expand Down
176 changes: 176 additions & 0 deletions avbroot/src/cli/hashtree.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
/*
* SPDX-FileCopyrightText: 2023 Andrew Gunnerson
* SPDX-License-Identifier: GPL-3.0-only
*/

use std::{
fs::{File, OpenOptions},
io::{BufReader, BufWriter, Write},
path::{Path, PathBuf},
sync::atomic::AtomicBool,
};

use anyhow::{Context, Result};
use clap::{Parser, Subcommand};

use crate::{
format::hashtree::HashTreeImage,
stream::{FromReader, PSeekFile, ToWriter},
};

fn open_input(path: &Path, rw: bool) -> Result<PSeekFile> {
OpenOptions::new()
.read(true)
.write(rw)
.open(path)
.map(PSeekFile::new)
.with_context(|| format!("Failed to open file: {path:?}"))
}

fn read_hash_tree(path: &Path) -> Result<HashTreeImage> {
let reader = File::open(path)
.map(BufReader::new)
.with_context(|| format!("Failed to open for reading: {path:?}"))?;
let hash_tree = HashTreeImage::from_reader(reader)
.with_context(|| format!("Failed to read hash tree data: {path:?}"))?;

Ok(hash_tree)
}

fn write_hash_tree(path: &Path, hash_tree: &HashTreeImage) -> Result<()> {
let mut writer = File::create(path)
.map(BufWriter::new)
.with_context(|| format!("Failed to open for writing: {path:?}"))?;
hash_tree
.to_writer(&mut writer)
.with_context(|| format!("Failed to write hash tree data: {path:?}"))?;
writer
.flush()
.with_context(|| format!("Failed to flush hash tree data: {path:?}"))?;

Ok(())
}

fn generate_subcommand(cli: &GenerateCli, cancel_signal: &AtomicBool) -> Result<()> {
let salt = hex::decode(&cli.salt).context("Invalid salt")?;
let input = open_input(&cli.input, false)?;

let hash_tree =
HashTreeImage::generate(&input, cli.block_size, &cli.algorithm, &salt, cancel_signal)
.context("Failed to generate hash tree data")?;

write_hash_tree(&cli.hash_tree, &hash_tree)?;

Ok(())
}

fn update_subcommand(cli: &UpdateCli, cancel_signal: &AtomicBool) -> Result<()> {
let ranges = cli
.range
.chunks_exact(2)
.map(|w| w[0]..w[1])
.collect::<Vec<_>>();

let input = open_input(&cli.input, false)?;
let mut hash_tree = read_hash_tree(&cli.hash_tree)?;

hash_tree
.update(&input, &ranges, cancel_signal)
.context("Failed to update hash tree data")?;

write_hash_tree(&cli.hash_tree, &hash_tree)?;

Ok(())
}

fn verify_subcommand(cli: &VerifyCli, cancel_signal: &AtomicBool) -> Result<()> {
let input = open_input(&cli.input, false)?;
let hash_tree = read_hash_tree(&cli.hash_tree)?;

hash_tree
.verify(&input, cancel_signal)
.context("Failed to verify data")?;

Ok(())
}

pub fn hash_tree_main(cli: &HashTreeCli, cancel_signal: &AtomicBool) -> Result<()> {
match &cli.command {
HashTreeCommand::Generate(c) => generate_subcommand(c, cancel_signal),
HashTreeCommand::Update(c) => update_subcommand(c, cancel_signal),
HashTreeCommand::Verify(c) => verify_subcommand(c, cancel_signal),
}
}

/// Generate hash tree data for a file.
#[derive(Debug, Parser)]
struct GenerateCli {
/// Path to input data.
#[arg(short, long, value_name = "FILE", value_parser)]
input: PathBuf,

/// Path to output hash tree data.
#[arg(short = 'H', long, value_name = "FILE", value_parser)]
hash_tree: PathBuf,

/// Block size.
#[arg(short, long, value_name = "BYTES", default_value = "4096")]
block_size: u32,

/// Hash algorithm.
#[arg(short, long, value_name = "NAME", default_value = "sha256")]
algorithm: String,

/// Salt (in hex).
#[arg(short, long, value_name = "HEX", default_value = "")]
salt: String,
}

/// Update hash tree data after a file is modified.
#[derive(Debug, Parser)]
struct UpdateCli {
/// Path to input data.
#[arg(short, long, value_name = "FILE", value_parser)]
input: PathBuf,

/// Path to hash tree data.
///
/// The file will be modified in place.
#[arg(short = 'H', long, value_name = "FILE", value_parser)]
hash_tree: PathBuf,

/// Input file ranges that were updated.
///
/// This is a half-open range and can be specified multiple times.
#[arg(short, long, value_names = ["START", "END"], num_args = 2)]
range: Vec<u64>,
}

/// Verify that a file contains no errors.
#[derive(Debug, Parser)]
struct VerifyCli {
/// Path to input data.
#[arg(short, long, value_name = "FILE", value_parser)]
input: PathBuf,

/// Path to input hash tree data.
#[arg(short = 'H', long, value_name = "FILE", value_parser)]
hash_tree: PathBuf,
}

#[derive(Debug, Subcommand)]
enum HashTreeCommand {
Generate(GenerateCli),
Update(UpdateCli),
Verify(VerifyCli),
}

/// Generate dm-verity hash tree data and verify files.
///
/// These commands operate on a standard hash tree data prepended by a custom
/// header.
#[derive(Debug, Parser)]
pub struct HashTreeCli {
#[command(subcommand)]
command: HashTreeCommand,
}
1 change: 1 addition & 0 deletions avbroot/src/cli/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ pub mod boot;
pub mod completion;
pub mod cpio;
pub mod fec;
pub mod hashtree;
pub mod key;
pub mod ota;

Expand Down
Loading

0 comments on commit 7ea8e8f

Please sign in to comment.