updated

docs/8. Quantized-Low Rank Adaptation(Qlora).md

@@ -19,16 +19,86 @@ In the case of **32-bit floating-point representation**, the number is broken do
 - **Exponent**: 8 bits to represent the magnitude of the number (how large or small it is).
 - **Mantissa (or significand)**: 23 bits to represent the precision of the number.
 
-#### Example of 32-bit Floating Point Representation:
 
-Consider the number **-3.75**:
+#### Example: Converting -3.75 to 32-bit Floating Point (Step by Step)
+
+Let's break down how -3.75 is stored in a computer's memory:
+
+1. **First, write -3.75 in binary:**
+   ```
+   Step 1: Convert 3 to binary
+   3 = 11 (in binary)
+   
+   Step 2: Convert 0.75 to binary
+   0.75 × 2 = 1.5  → Write down 1
+   0.5 × 2  = 1.0  → Write down 1
+   0.0      → Stop here
+   
+   So, 0.75 = 0.11 (in binary)
+   
+   Therefore, -3.75 = -11.11 (in binary)
+   ```
+
+2. **Normalize it (move decimal point to get 1.xxx format):**
+   ```
+   -11.11 = -1.111 × 2¹
+   Now we have:
+   • Sign: Negative (1)
+   • Number: 1.111
+   • Power of 2: 1
+   ```
+
+3. **Convert to 32-bit format:**
+
+| Part | Explanation | Calculation | Final Bits |
+|------|-------------|-------------|------------|
+| Sign | Negative = 1, Positive = 0 | Number is negative | 1 |
+| Exponent | Power of 2 (1) + Bias (127) | 1 + 127 = 128 = 10000000 | 10000000 |
+| Mantissa | Everything after 1. in 1.111 | Just the 111, then pad with zeros | 11100000000000000000000 |
+
+4. **Put it all together:**
+   ```
+   Sign     Exponent        Mantissa
+   1    |   10000000   |   11100000000000000000000
+   ```
+
+#### Example: Converting -3.75 to 4-bit (Quantization)
+
+Now, let's see how we convert this same number (-3.75) to a 4-bit number:
+
+1. **Understand the range:**
+   ```
+   4 bits can represent 16 numbers (0 to 15)
+   We need to represent both positive and negative numbers
+   So we map:
+   • -5.0 → 0
+   • 0.0  → 8
+   • +5.0 → 15
+   ```
+
+2. **Convert -3.75 using a simple proportion:**
+   ```
+   Original range: -5.0 to +5.0 (total range = 10.0)
+   4-bit range: 0 to 15 (16 different values)
+   
+   Formula:
+   New value = 8 + (old value × 8/5)
+   For -3.75:
+   = 8 + (-3.75 × 8/5)
+   = 8 + (-6)
+   = 2
+   ```
+
+3. **Visual representation of the conversion:**
+
+| Step | Value | Explanation |
+|------|-------|-------------|
+| Original number | -3.75 | Our starting value |
+| Scale factor | 8/5 | How much we multiply to fit in new range |
+| Scaled number | -6 | -3.75 × (8/5) |
+| Add offset | 2 | 8 + (-6) = 2 |
+| Final 4-bit value | 0010 | 2 in binary (4 bits) |
 
-| Component | Bits Used | Value | Binary Representation |
-|-----------|-----------|-------|---------------------|
-| Sign bit | 1 bit | Negative (1) | 1 |
-| Exponent | 8 bits | 1 (after bias) | 10000000 |
-| Mantissa | 23 bits | 1.111 binary | 11100000000000000000000 |
-| **Complete 32-bit** | **32 bits** | **-3.75** | **1 10000000 11100000000000000000000** |
 
 ### How Does Quantization Work (32-bit to 4-bit)?