gpuMatMulQuant (Function)

Signature: gpuMatMulQuant(x: GpuBuffer<float32>, weights: GpuBuffer<uint8>, type: string, rows: int, inner: int, cols: int) -> Result<GpuBuffer<float32>, Error>

Description: The product of a rows x inner float32 matrix and cols rows of quantized weights in the named GGUF block format (Q4_K, Q8_0, Q6_K, Q5_K, Q3_K, IQ4_NL, IQ4_XS or IQ3_S), as a rows x cols float32 buffer. Each weight is dequantized, multiplied and added in ggml's block order with every step rounded, in block-range parts added in order (one part above 2^20 outputs, four above 2^16, sixteen otherwise), identically on the host and every device. A length, element type or inner dimension the shape disagrees with returns Error.

Parameters

• x (GpuBuffer<float32>): The rows x inner float32 buffer
• weights (GpuBuffer<uint8>): The uint8 buffer holding cols rows of blocks, as the GGUF tensor stores them
• type (string): The GGUF tensor type name
• rows (int): The row count of x
• inner (int): The shared dimension, a multiple of the block's weight count
• cols (int): The weight row count

Returns: Result<GpuBuffer<float32>, Error>

Example

fn project(xs) = match gpuMapFile("model.gguf", 10996640, 14745600) {
    Success { value } => { let weights = value
    match toGpu(xs) {
        Success { value } => match gpuMatMulQuant(value, weights, "Q4_K", 1, 5120, 5120) {
            Success { value } => print(gpuLength(value))
            Error { message } => print(message)
        }
        Error { message } => print(message)
    }
    }
    Error { message } => print(message)
}