pmetal-gguf

GGUF file format support for llama.cpp and Ollama compatibility.

Overview

This crate provides reading and writing support for the GGUF (GPT-Generated Unified Format) file format, enabling compatibility with llama.cpp, Ollama, and other GGUF-compatible inference engines.

Features

• GGUF Reading: Parse GGUF files and extract metadata/tensors
• GGUF Writing: Create GGUF files from SafeTensors/PyTorch models
• Tensor Dequantization: Convert quantized tensors to full precision
• Metadata Handling: Read/write model metadata and tokenizer info

Usage

Reading GGUF Files

use pmetal_gguf::GgufContent;

// Load GGUF file
let gguf = GgufContent::from_file("model.gguf")?;

// Access metadata
println!("Architecture: {}", gguf.metadata.get("general.architecture")?);
println!("Context length: {}", gguf.metadata.get("llama.context_length")?);

// Iterate tensors
for (name, tensor) in gguf.tensors() {
    println!("{}: {:?}", name, tensor.shape());
}

Dequantizing Tensors

use pmetal_gguf::{GgufContent, dequant};

let gguf = GgufContent::from_file("model-q4.gguf")?;

// Dequantize a specific tensor
let weights = gguf.get_tensor("model.layers.0.self_attn.q_proj.weight")?;
let fp32_weights = dequant::dequantize(&weights)?;

Converting to GGUF

use pmetal_gguf::{GgufWriter, Quantization};

let mut writer = GgufWriter::new("output.gguf")?;

// Set metadata
writer.set_metadata("general.architecture", "llama")?;
writer.set_metadata("general.name", "My Model")?;

// Add tensors with optional quantization
writer.add_tensor("model.embed_tokens.weight", &embeddings, Quantization::None)?;
writer.add_tensor("model.layers.0.self_attn.q_proj.weight", &weights, Quantization::Q4_K)?;

writer.finish()?;

Supported Quantization Types

Modules

License

MIT OR Apache-2.0