GGML Shape & Strides question for custom kernel/hardware

[akapoor3518]

Hi,
I was looking GGML code it has ne & nb array under tensor structure. ne is used for shape & nb is used for stride.They have GGML_OP_RESHAPE operation where src tensor can be break in multiple tensors with shape & stride)
I have following questions:
1 For Llama.cpp application how we are tuning shape & stride for tensor for particular GGML operation. The reason i asked Base on this information we will general AOT(ahead of time) Kernel Binary for our custom hardware. Once kernel binary is created we cant change shape. Do we have any recommendation?
2 Do we have any recommendation for shape & stride for Matrix Multiplication , does it different for every model?
3 Can backend can control ne & nb value for tensor or it can be only controlled trough llama.cpp application?

Thanks,
Anoop

[akapoor3518]

Hi @ggerganov,
Can you please help me with above question.
Thanks,
Anoop

[ggerganov]

The questions are too broad to be able to give you a meaningful answer. Start from the examples in ggml and llama.cpp and work your way through the logic. If something is not clear in - ask specifically about it.

[akapoor3518]

Hey Georgi,
I am playing with example of ggml. Infact we created add & mut operation for our backend and validated. Now my question is once we bring GGML Code to Lamma.cpp to run tinylamma. Currently Tinylamma is using below operation(i did graph print to get all operation for tinylamma) . GGML_OP_ADD & GGML_OP_MUL​ will go through with our Custom Hardware backend rest will go through CPU(since those operation we not supported and implemented). Currently i have kernel for GGML_OP_ADD/GGML_OP_MUL for fixed size of 64 elements for F32 for our custom kernel. We can create any size of Custom kernel using AOT method but once kernel is created we cant change the size. Hence to handle different size Either i need to reshape during backend Compute or if we upfront know for lamma.cpp for size of tensor we can build AOT Kernel for those size. Please advise here. I also need to know functionality of below GGML operation , otherwise i need to trace the code to understand.

GGML_OP_ADD​

GGML_OP_MUL​
GGML_OP_RMS_NORM​
GGML_OP_MUL_MAT​
GGML_OP_CPY​
GGML_OP_RESHAPE​
GGML_OP_VIEW​
GGML_OP_PERMUTE​
GGML_OP_TRANSPOSE​
GGML_OP_GET_ROWS​
GGML_OP_SOFT_MAX​
GGML_OP_ROPE​
GGML_OP_MAP_BINARY​

[ggerganov]

For different models there will be different shapes. You can for example run llama-eval-callback and see all the operators and shapes being applied for a given model:

./bin/llama-eval-callback -m ../models/llama-3.1-8b/ggml-model-q8_0.gguf -n 1 | grep ggml_debug

ggml_debug:                 inp_embd = (f32)   GET_ROWS(token_embd.weight{4096, 128256, 1, 1}, leaf_2{1, 1, 1, 1}}) = {4096, 1, 1, 1}
ggml_debug:                   norm-0 = (f32)   RMS_NORM(Metal#inp_embd#0{4096, 1, 1, 1}, }) = {4096, 1, 1, 1}
ggml_debug:              attn_norm-0 = (f32)        MUL(norm-0{4096, 1, 1, 1}, blk.0.attn_norm.weight{4096, 1, 1, 1}}) = {4096, 1, 1, 1}
ggml_debug:                   Qcur-0 = (f32)    MUL_MAT(blk.0.attn_q.weight{4096, 4096, 1, 1}, attn_norm-0{4096, 1, 1, 1}}) = {4096, 1, 1, 1}
ggml_debug:        Qcur-0 (reshaped) = (f32)    RESHAPE(Qcur-0{4096, 1, 1, 1}, }) = {128, 32, 1, 1}
ggml_debug:                   Qcur-0 = (f32)       ROPE(Qcur-0 (reshaped){128, 32, 1, 1}, Metal#inp_pos#0{1, 1, 1, 1}}) = {128, 32, 1, 1}
ggml_debug:                   Kcur-0 = (f32)    MUL_MAT(blk.0.attn_k.weight{4096, 1024, 1, 1}, attn_norm-0{4096, 1, 1, 1}}) = {1024, 1, 1, 1}
ggml_debug:        Kcur-0 (reshaped) = (f32)    RESHAPE(Kcur-0{1024, 1, 1, 1}, }) = {128, 8, 1, 1}
ggml_debug:                   Kcur-0 = (f32)       ROPE(Kcur-0 (reshaped){128, 8, 1, 1}, Metal#inp_pos#0{1, 1, 1, 1}}) = {128, 8, 1, 1}
ggml_debug:                   Vcur-0 = (f32)    MUL_MAT(blk.0.attn_v.weight{4096, 1024, 1, 1}, attn_norm-0{4096, 1, 1, 1}}) = {1024, 1, 1, 1}
...

Note that some shapes depend on the context size, so they are basically not constant and will increase with more and more tokens added to the context. This is because some operations in the attention depend on the number of tokens that are being attended.

I am planning to add option for constant-graph inference where the compute graph would be calculated once and will always use the same shapes for all operations. This can be useful for small-context decoders or embedding models. But it will take a bit of time to get there, so currently your backend would have to deal with dynamic shapes.

[akapoor3518]

Thanks alot for for detail. To handle what changes is needed for dynamic shapes at my backend. Let me go over one simple example with GGML_OP_ADD there we have three tensors(two leaf tensor and one node tensor. lets Say src0 & src1 leaf and node is Node Tensor and size of element are 640 with F32 type . During Backend graph compute i traverse through whole graph list find all nodes and for each node we find leaf tensor. AT above example Node: node and leaf Node src0 & src1. Now we have Customer_kernel_X which take three parameter two input and one output, at backend compute i need to following
shape = 64;
total_elements = 640(calculated from a->ne). Currently i am considering Contiguous _GGML example(where no reshape done). But logic i will implement will handle any non_Contiguous stuff)

Below is very basic Pseudo code , i wanted to check if approach is correct.
shape = 64;
total_elements = 640;
for (int i = 0; i < cgraph->n_nodes; i++) {while (i < total_elements) {
node = cgraph->nodes[i];
if (node->src[0] && node->src[1]) {
src0 = node->src[0];
src1 = node->src[1];
}
int j =0;
while (j < total_elements) {

          // The custom header memory is allocated before beginning of data which out custom backend API will take care. All works well i have tested it
         // This Custom Header : MemRefDescriptor will have shape & stride and data pointer which i will be keep changing
        // and then call below API
            **Custom_kernel_X(src, src1, node)** 
           j+=shape;
     }

}