how string decoder in nodejs knows that character is split into 2 chunks

The StringDecoder module in Node.js is specifically designed to handle multi-byte character encodings, such as UTF-8, which can span multiple bytes. When dealing with streams of binary data, it's common for characters to be split across multiple chunks. The StringDecoder class handles this by buffering incomplete characters until it has enough bytes to decode a complete character.

How StringDecoder Detects Incomplete Characters

The StringDecoder class works by keeping track of the number of bytes required to complete a character and buffering incomplete sequences until the necessary bytes are received. Here’s a step-by-step explanation of how it detects and handles split characters:

1. Initialization:

   • The StringDecoder is initialized with a specific character encoding, such as 'utf8', which defines how bytes should be interpreted as characters.

2. Reading Chunks:

   • When a chunk of data is passed to the StringDecoder's write() method, it processes the bytes in the chunk one by one (or in groups, depending on the encoding).

3. Checking Byte Lengths:

   • For UTF-8, the StringDecoder looks at the first byte of each character to determine the number of bytes that make up the character:
     • A single-byte character (0xxxxxxx) is complete.
     • A two-byte character (110xxxxx) requires one more byte (10xxxxxx).
     • A three-byte character (1110xxxx) requires two more bytes (10xxxxxx 10xxxxxx).
     • A four-byte character (11110xxx) requires three more bytes (10xxxxxx 10xxxxxx 10xxxxxx).

4. Buffering Incomplete Characters:

   • If the current chunk ends before all bytes of a multi-byte character are received, the StringDecoder stores the incomplete character bytes in an internal buffer.
   • When the next chunk arrives, the StringDecoder checks the buffer to see if it can complete the character using the new bytes.

5. Combining and Decoding:

   • Once all necessary bytes for a multi-byte character are available, the StringDecoder combines them and decodes the character, then returns the complete string.

Example with StringDecoder

Here’s a simple example to illustrate how the StringDecoder handles split multi-byte characters:

const { StringDecoder } = require('string_decoder');
const decoder = new StringDecoder('utf8');

// Example buffers containing parts of a multi-byte UTF-8 character (e.g., €)
const buffer1 = Buffer.from([0xE2, 0x82]);
const buffer2 = Buffer.from([0xAC]);

console.log(decoder.write(buffer1)); // Outputs: '' (incomplete character, stored internally)
console.log(decoder.write(buffer2)); // Outputs: '€' (complete character after combining)

In this example, the first buffer (buffer1) contains the initial bytes of the Euro sign character (€), but not enough to complete it. The StringDecoder buffers these bytes. When the second buffer (buffer2) arrives with the remaining byte, the StringDecoder can then decode and output the complete character.

Internal Mechanism

Internally, the StringDecoder uses a combination of logic to determine if the bytes represent a complete character or if more bytes are needed. Here’s a simplified look at how it handles UTF-8 encoding:

class StringDecoder {
  constructor(encoding) {
    this.encoding = encoding;
    this.charBuffer = Buffer.alloc(6); // buffer to hold incomplete characters
    this.charLength = 0;
    this.charReceived = 0;
  }

  write(buffer) {
    let charStr = '';
    while (this.charLength) {
      const available = (buffer.length >= this.charLength - this.charReceived)
        ? this.charLength - this.charReceived
        : buffer.length;

      buffer.copy(this.charBuffer, this.charReceived, 0, available);
      this.charReceived += available;

      if (this.charReceived < this.charLength) {
        return '';
      }

      buffer = buffer.slice(available, buffer.length);
      charStr = this.charBuffer.slice(0, this.charLength).toString(this.encoding);

      this.charReceived = this.charLength = 0;

      if (buffer.length === 0) {
        return charStr;
      }
    }

    let i = 0;
    while (i < buffer.length) {
      const byte = buffer[i];
      let charLen = 0;
      if (byte >> 5 === 0x06) {
        charLen = 2;
      } else if (byte >> 4 === 0x0E) {
        charLen = 3;
      } else if (byte >> 3 === 0x1E) {
        charLen = 4;
      } else {
        charStr += buffer.toString(this.encoding, i, i + 1);
        i++;
        continue;
      }

      if (i + charLen <= buffer.length) {
        charStr += buffer.toString(this.encoding, i, i + charLen);
        i += charLen;
      } else {
        buffer.copy(this.charBuffer, 0, i, buffer.length);
        this.charReceived = buffer.length - i;
        this.charLength = charLen;
        i = buffer.length;
      }
    }

    return charStr;
  }

  end(buffer) {
    let res = '';
    if (buffer && buffer.length) {
      res = this.write(buffer);
    }
    if (this.charReceived) {
      res += this.charBuffer.slice(0, this.charReceived).toString(this.encoding);
    }
    return res;
  }
}

This class buffers incomplete characters and attempts to decode them as complete characters arrive, ensuring proper handling of multi-byte sequences.