Funbit - Erlang/OTP Bit Syntax for Go

Funbit is a Go library providing Erlang/OTP bit syntax compatibility for bitstring and binary data manipulation. It includes interfaces for constructing and pattern matching bitstrings.

Features

Erlang Compatibility: 1:1 mapping of Erlang bit syntax to a Go API.
Bit-Level Operations: Operates on a bit stream, not restricted to byte-aligned segments.
Data Types: Integer, float (16/32/64-bit), binary, bitstring, UTF-8/16/32.
Dynamic Sizing: Supports variables and arithmetic expressions in size fields (e.g., total-6).
Unit Multipliers: Allows size multiplication with unit:N (e.g., 32/float-unit:2 for a 64-bit double).
Endianness Support: Big, little, and native byte ordering.
Compound Specifiers: Supports combinations of specifiers (e.g., 32/big-unsigned-integer-unit:8).
String Literals in Patterns: Allows constant string matching for protocol validation (e.g., "IHDR":4/binary).
Builder API Pattern: Operations are chained, with a single error check.
Type Semantics: Differentiates between integer and binary representations.
Protocol Parsing: Designed for parsing protocols such as IPv4, TCP, and PNG.
Performance: Optimized for construction and pattern matching.

Core Semantics

Binary vs. Integer Size

// CRITICAL DIFFERENCE:
funbit.Integer(m, &val, funbit.WithSize(32))  // 32 BITS
funbit.Binary(m, &data, funbit.WithSize(32))  // 32 BYTES = 256 BITS!

// For binary segments: WithSize(N) means N UNITS (default: bytes)
funbit.Binary(m, &data, funbit.WithSize(4))   // 4 bytes
funbit.Binary(m, &data, funbit.WithSize(32))  // 32 bytes

Unit Multipliers for Dynamic Sizing

// WITHOUT WithUnit(1): size*8 interpreted as BYTES, multiplied by 8!
funbit.Binary(m, &data, funbit.WithDynamicSizeExpression("size*8"))
// size=5 → 5*8=40, but binary interprets as 40*8=320 bits!

// WITH WithUnit(1): size*8 interpreted as exact BITS
funbit.Binary(m, &data, funbit.WithDynamicSizeExpression("size*8"), funbit.WithUnit(1))
// size=5 → 5*8=40 bits exactly

UTF Extraction Semantics

// Erlang UTF supports BOTH approaches:

// 1. String encoding (entire strings)
funbit.AddUTF8(builder, "Hello")  // Encodes full string

// 2. Code point extraction (individual characters)
funbit.UTF8(matcher, &codepoint)  // Single code point

// 3. Binary extraction (for full strings)
var bytes []byte
funbit.RestBinary(matcher, &bytes)
text := string(bytes)  // Full string

Quick Start

Builder API Pattern

Funbit uses a builder pattern for deferred error handling:

// Chain operations, check error once
builder := funbit.NewBuilder()
funbit.AddInteger(builder, 42, funbit.WithSize(8))
funbit.AddUTF8Codepoint(builder, 0x1F680) // rocket emoji
funbit.AddFloat(builder, 3.14, funbit.WithSize(32))

bitstring, err := funbit.Build(builder) // Error checked once
if err != nil {
    return err
}

// Traditional approach:
// if err := AddInteger(...); err != nil { return err }
// if err := AddUTF8Codepoint(...); err != nil { return err }
// if err := AddFloat(...); err != nil { return err }

Characteristics:

Chainable: Add* functions do not return errors.
Efficient: The first error halts processing; subsequent calls are ignored.
Clean: A single error check is performed at Build() time.
Consistent: The pattern is used throughout the API.

Installation

go get github.com/funvibe/funbit

Basic Construction and Matching

package main

import (
    "fmt"
    "math/big"
    "github.com/funvibe/funbit/pkg/funbit"
)

func main() {
    // Construction: <<42:8, "hello":5/binary>>
    builder := funbit.NewBuilder()
    funbit.AddInteger(builder, 42, funbit.WithSize(8))
    funbit.AddBinary(builder, []byte("hello"))
    
    bitstring, _ := funbit.Build(builder)
    
    // Pattern Matching: <<value:8, text:5/binary>>
    matcher := funbit.NewMatcher()
    var value int
    var text []byte
    
    funbit.Integer(matcher, &value, funbit.WithSize(8))
    // CRITICAL: Binary size in BYTES! WithSize(5) = 5 bytes = 40 bits
    funbit.Binary(matcher, &text, funbit.WithSize(5)) // 5 bytes
    
    results, err := funbit.Match(matcher, bitstring)
    if err == nil && len(results) > 0 {
        fmt.Printf("Value: %d, Text: %s\n", value, string(text))
        // Output: Value: 42, Text: hello
    }

    // Big.Int support for huge integers
    hugeInt := new(big.Int)
    hugeInt.SetString("999999999999999999999999999999", 10)
    
    builder2 := funbit.NewBuilder()
    funbit.AddInteger(builder2, hugeInt, funbit.WithSize(256)) // 256-bit integer
    
    bitstring2, _ := funbit.Build(builder2)
    fmt.Printf("Huge integer bitstring: %d bits\n", bitstring2.Length())
}

Core Concepts

Construction vs Pattern Matching

Construction builds bitstrings from values:

builder := funbit.NewBuilder()
funbit.AddInteger(builder, 1000, funbit.WithSize(16))
funbit.AddFloat(builder, 3.14, funbit.WithSize(32))
bitstring, _ := funbit.Build(builder)

Pattern Matching extracts values from bitstrings:

matcher := funbit.NewMatcher()
var num int
var pi float32
funbit.Integer(matcher, &num, funbit.WithSize(16))
funbit.Float(matcher, &pi, funbit.WithSize(32))
results, _ := funbit.Match(matcher, bitstring)

Type Semantics

Funbit follows Erlang semantics where the default type is integer:

// Construction
funbit.AddInteger(builder, 42, funbit.WithSize(8))     // Integer: displays as 42
funbit.AddBinary(builder, []byte("A"), funbit.WithSize(1))  // Binary: displays as 'A' (1 byte)

// Pattern Matching
funbit.Integer(matcher, &num, funbit.WithSize(8))      // Extract as number: 42
funbit.Binary(matcher, &char, funbit.WithSize(1))      // Extract as character: 'A' (1 byte)

The interpretation of a bit sequence depends on the type specifier.

Advanced Features

Non-byte-aligned Bitstrings

The library supports bit-level operations that are not restricted to byte boundaries:

// Build 7-bit value (not byte-aligned)
builder := funbit.NewBuilder()
funbit.AddInteger(builder, 0b101, funbit.WithSize(3))   // 3 bits
funbit.AddInteger(builder, 0b1111, funbit.WithSize(4))  // 4 bits
// Total: 7 bits (not a full byte)

bitstring, _ := funbit.Build(builder)

// Pattern matching bit-level
matcher := funbit.NewMatcher()
var part1, part2 int

funbit.Integer(matcher, &part1, funbit.WithSize(3))  // Extract 3 bits
funbit.Integer(matcher, &part2, funbit.WithSize(4))  // Extract 4 bits

results, _ := funbit.Match(matcher, bitstring)
// part1 = 5 (0b101), part2 = 15 (0b1111)

UTF Codepoint API

The API provides functions for handling single codepoints:

// Encoding single codepoints
builder := funbit.NewBuilder()
funbit.AddUTF8Codepoint(builder, 0x1F680)  // emoji
funbit.AddUTF16Codepoint(builder, 0x1F31F, funbit.WithEndianness("big"))
funbit.AddUTF32Codepoint(builder, 65)  // 'A'

bitstring, err := funbit.Build(builder)
if err != nil {
    // Handles invalid codepoints (e.g., surrogate pairs)
    fmt.Printf("Error: %v\n", err)
}

// Extract as INTEGER (Erlang spec!)
matcher := funbit.NewMatcher()
var codepoint int
funbit.UTF8(matcher, &codepoint)  // Returns 0x1F680 (integer)

results, _ := funbit.Match(matcher, bitstring)
// codepoint = 128640 (0x1F680)

Signed vs Unsigned Integers

// Signed interpretation
builder := funbit.NewBuilder()
funbit.AddInteger(builder, -50, funbit.WithSize(8), funbit.WithSigned(true))

// Unsigned interpretation (default)
funbit.AddInteger(builder, 200, funbit.WithSize(8), funbit.WithSigned(false))

bitstring, _ := funbit.Build(builder)

// Pattern matching with signedness
matcher := funbit.NewMatcher()
var signedVal, unsignedVal int

funbit.Integer(matcher, &signedVal, funbit.WithSize(8), funbit.WithSigned(true))
funbit.Integer(matcher, &unsignedVal, funbit.WithSize(8), funbit.WithSigned(false))

results, _ := funbit.Match(matcher, bitstring)
// signedVal = -50, unsignedVal = 200

Dynamic Sizing with Expressions

// Create matcher and register variables
matcher := funbit.NewMatcher()
var headerSize int = 32
var total int = 96
funbit.RegisterVariable(matcher, "headerSize", &headerSize)
funbit.RegisterVariable(matcher, "total", &total)

// Use in expressions
funbit.AddBinary(builder, data, funbit.WithDynamicSizeExpression("total-headerSize"))

String Literals in Patterns

String literals can be used in patterns for protocol field validation:

// Validate PNG header: expect exactly "IHDR"
matcher := funbit.NewMatcher()
var length int
expectedType := "IHDR"

funbit.Integer(matcher, &length, funbit.WithSize(32))
funbit.Binary(matcher, &expectedType, funbit.WithSize(4)) // Must match "IHDR" (4 bytes)

Endianness Control

// Big-endian (default)
funbit.AddInteger(builder, 0x1234, funbit.WithSize(16), funbit.WithEndianness("big"))

// Little-endian  
funbit.AddInteger(builder, 0x1234, funbit.WithSize(16), funbit.WithEndianness("little"))

Unit Multipliers

Unit multipliers allow size multiplication for precise control:

// 32-bit float with unit:2 = 64-bit IEEE 754 double precision
funbit.AddFloat(builder, 3.14159265359, funbit.WithSize(32), funbit.WithUnit(2))

// 8-bit size with unit:16 = 128 effective bits
funbit.AddInteger(builder, 8, funbit.WithSize(8), funbit.WithUnit(16))

// Pattern matching with unit multipliers
funbit.Float(matcher, &doubleValue, funbit.WithSize(32), funbit.WithUnit(2))

Compound Specifiers

Combine multiple specifiers for complex data layouts:

// 32/big-unsigned-integer-unit:8
funbit.AddInteger(builder, 0xDEADBEEF,
    funbit.WithSize(32),
    funbit.WithEndianness("big"),
    funbit.WithUnit(8))

// 16/little-unsigned-integer
funbit.AddInteger(builder, 0x1234,
    funbit.WithSize(16),
    funbit.WithEndianness("little"))

Bit-Level Precision

// Individual flag bits
funbit.AddInteger(builder, 1, funbit.WithSize(1))  // Single bit
funbit.AddInteger(builder, 0, funbit.WithSize(1))  // Another bit
funbit.AddInteger(builder, 3, funbit.WithSize(2))  // 2-bit value

Examples

TCP Header Parsing

builder := funbit.NewBuilder()
funbit.AddInteger(builder, 0x1234, funbit.WithSize(16))    // Source port
funbit.AddInteger(builder, 0x5678, funbit.WithSize(16))    // Dest port
funbit.AddInteger(builder, 1, funbit.WithSize(1))          // URG flag
funbit.AddInteger(builder, 0, funbit.WithSize(1))          // ACK flag
// ... more flags
funbit.AddBinary(builder, []byte("payload"))

// Pattern matching extracts all fields with proper types

PNG Header Validation

// Pattern: <<length:32, "IHDR":4/binary, width:32, height:32>>
matcher := funbit.NewMatcher()
var length, width, height int
expectedChunk := "IHDR"

funbit.Integer(matcher, &length, funbit.WithSize(32))
funbit.Binary(matcher, &expectedChunk, funbit.WithSize(4))  // Validates "IHDR" (4 bytes)
funbit.Integer(matcher, &width, funbit.WithSize(32))
funbit.Integer(matcher, &height, funbit.WithSize(32))

Usage Guidelines

1. Understand Type Semantics

Use funbit.Integer() for numeric values (displays as numbers)
Use funbit.Binary() for text/character data (displays as characters)
Default type is integer, not binary

2. Handle Dynamic Sizes Properly

// Register all variables before use
funbit.RegisterVariable("size", 32)
funbit.RegisterVariable("total", 128)

// Use expressions for complex sizing
funbit.WithDynamicSizeExpression("total-size-8")

3. Validate Protocol Constants

// Use string literals to validate protocol headers
expectedType := "IHDR"
funbit.Binary(matcher, &expectedType, funbit.WithSize(4))  // 4 bytes = "IHDR"

4. Use Unit Multipliers for Precision

// For IEEE 754 double precision floats
funbit.AddFloat(builder, value, funbit.WithSize(32), funbit.WithUnit(2))  // 64-bit

// For size fields that represent bit counts
funbit.AddInteger(builder, 8, funbit.WithSize(8), funbit.WithUnit(16))    // 8*16 bits

5. Combine Specifiers for Complex Layouts

// Full compound specifier
funbit.AddInteger(builder, value,
    funbit.WithSize(32),
    funbit.WithEndianness("big"),
    funbit.WithUnit(8))

6. Endianness Format

// Use short forms
funbit.WithEndianness("big")     // Supported
funbit.WithEndianness("little")  // Supported  
funbit.WithEndianness("native")  // Supported

7. Pattern Size Validation

Funbit automatically validates pattern sizes unless:

Pattern contains rest patterns (funbit.RestBinary())
Pattern contains dynamic sizes
Pattern contains string literals (for validation)
Pattern contains unit multipliers (calculated dynamically)

Erlang to Funbit Syntax Reference

Erlang Syntax	Funbit Equivalent	Description
`<<42:8>>`	`funbit.AddInteger(b, 42, funbit.WithSize(8))`	8-bit integer
`<<42:8/big>>`	`funbit.AddInteger(b, 42, funbit.WithSize(8), funbit.WithEndianness("big"))`	Big-endian integer
`<<999999999999999999999:256>>`	`hugeInt := new(big.Int); hugeInt.SetString("999999999999999999999", 10); funbit.AddInteger(b, hugeInt, funbit.WithSize(256))`	Arbitrary-precision integer
`<<3.14:32/float>>`	`funbit.AddFloat(b, 3.14, funbit.WithSize(32))`	32-bit float
`<<"hello world"/binary>>`	`funbit.AddBinary(b, []byte("hello world"))`	Binary data (full)
`<<"hello world":5/binary>>`	`funbit.AddBinary(b, []byte("hello world"), funbit.WithSize(5))`	Binary data (truncated to 5 bytes: "hello")
`<<Size:8, Data:Size/binary>>`	`funbit.Integer(m, &size, funbit.WithSize(8))` `funbit.Binary(m, &data, funbit.WithDynamicSizeExpression("size"))`	Dynamic sizing
`<<Value:16/unit:8>>`	`funbit.AddInteger(b, value, funbit.WithSize(16), funbit.WithUnit(8))`	Unit multiplier
`<<Codepoint/utf8>>`	`funbit.AddUTF8Codepoint(b, codepoint)`	UTF-8 codepoint
`<<"text"/utf8>>`	`funbit.AddUTF8(b, "text")`	UTF-8 string
`<<-50:8/signed>>`	`funbit.AddInteger(b, -50, funbit.WithSize(8), funbit.WithSigned(true))`	Signed integer
`<<Rest/binary>>`	`funbit.RestBinary(m, &rest)`	Rest pattern
`<<1:3, 15:4>>`	`funbit.AddInteger(b, 1, funbit.WithSize(3))` `funbit.AddInteger(b, 15, funbit.WithSize(4))`	Non-byte-aligned

API Differences from Erlang:

Funbit uses an explicit builder pattern vs Erlang's literal syntax.
Funbit requires variable registration for dynamic sizes.
Funbit supports method chaining and error accumulation.
Funbit provides stronger type safety with Go's type system.
Funbit supports arbitrary-precision integers via *big.Int for huge numbers

🔢 Arbitrary-Precision Integer Support

Funbit now supports *big.Int for handling arbitrarily large integers without precision loss:

import "math/big"

// Create a huge integer
hugeInt := new(big.Int)
hugeInt.SetString("999999999999999999999999999999", 10) // 30 digits

// Add to bitstring with appropriate size
builder := funbit.NewBuilder()
funbit.AddInteger(builder, hugeInt, funbit.WithSize(256)) // 256-bit integer

bitstring, err := funbit.Build(builder)
if err != nil {
    log.Fatalf("Failed to build: %v", err)
}

// Pattern matching extracts as *big.Int
matcher := funbit.NewMatcher()
var extracted *big.Int
funbit.Integer(matcher, &extracted, funbit.WithSize(256))

results, err := funbit.Match(matcher, bitstring)
if err == nil && len(results) > 0 {
    fmt.Printf("Extracted: %s\n", extracted.String()) // Full precision
}

Benefits:

No Precision Loss: Handle numbers larger than float64 can represent
Exact Arithmetic: Perfect for cryptographic applications, large IDs, etc.
Automatic Detection: Funbit automatically uses *big.Int when needed
Backward Compatible: Regular int values continue to work as before

Use Cases:

Cryptographic keys and hashes
Large database IDs
Financial calculations with high precision
Scientific computing with large numbers
Protocol fields with arbitrary size

Performance

Construction: O(n) where n is number of segments
Pattern Matching: O(n) where n is number of segments
Memory: Bitstrings are immutable and memory-efficient
Threading: Thread-safe for concurrent reads; builders are not thread-safe.

Integration with Runtimes

When integrating with language runtimes (like Lua, JavaScript):

// For integers extracted from patterns
if intValue >= 0 && intValue <= 255 {
    // In mixed binary context, might display as character
    // In pure integer context, display as number
}

// For binary data
string(binaryData) // Always display as characters

Additional Examples

Refer to funbit/examples/public_api_example.go for examples covering:

Basic construction and matching
Data types and specifiers
Endianness support
Dynamic sizing
String literals in patterns
Complex protocol parsing
Unit multipliers
Compound specifiers
Advanced float handling
Type semantics
Integration patterns

Contributing

Contributions are accepted.

License

MIT License. See the LICENSE file for details.

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
acceptancetests		acceptancetests
doc		doc
examples		examples
internal		internal
pkg/funbit		pkg/funbit
.gitignore		.gitignore
LICENSE.md		LICENSE.md
README.md		README.md
go.mod		go.mod

Folders and files

Latest commit

History

Repository files navigation

Funbit - Erlang/OTP Bit Syntax for Go

Features

Core Semantics

Binary vs. Integer Size

Unit Multipliers for Dynamic Sizing

UTF Extraction Semantics

Quick Start

Builder API Pattern

Installation

Basic Construction and Matching

Core Concepts

Construction vs Pattern Matching

Type Semantics

Advanced Features

Non-byte-aligned Bitstrings

UTF Codepoint API

Signed vs Unsigned Integers

Dynamic Sizing with Expressions

String Literals in Patterns

Endianness Control

Unit Multipliers

Compound Specifiers

Bit-Level Precision

Examples

TCP Header Parsing

PNG Header Validation

Usage Guidelines

1. Understand Type Semantics

2. Handle Dynamic Sizes Properly

3. Validate Protocol Constants

4. Use Unit Multipliers for Precision

5. Combine Specifiers for Complex Layouts

6. Endianness Format

7. Pattern Size Validation

Erlang to Funbit Syntax Reference

🔢 Arbitrary-Precision Integer Support

Performance

Integration with Runtimes

Additional Examples

Contributing

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 1

Contributors

Uh oh!

Languages