fast-io
A high-performance Common Lisp library for fast octet-vector and octet-stream I/O operations.
What is fast-io?
fast-io is a Common Lisp library designed to significantly improve the performance of octet-vector and octet-stream input/output operations. It provides a straightforward interface for reading and writing bytes and sequences, with a focus on reducing consing and execution time compared to standard approaches like vector-push-extend and flexi-streams.
Common Lisp developers working on performance-critical applications that involve extensive octet-level I/O, such as network programming, binary data processing, or foreign function interface (FFI) interactions.
Developers choose fast-io for its substantial performance gains in speed and reduced memory allocation, its support for multi-byte endian operations, and its integration with static-vectors for efficient FFI usage without extra copying.
Overview
Fast octet-vector/stream I/O for Common Lisp
Use Cases
Best For
- Optimizing octet-vector I/O operations where performance and low consing are critical, such as in high-throughput network message generation.
- Reading and writing multi-byte values with specific endianness requirements, like parsing binary file formats or network protocols.
- Integrating with foreign function interfaces (FFI) by outputting to static-vectors for direct pointer passing without additional data copying.
- Wrapping Lisp streams with fast-io buffers for compatibility, though with awareness of the performance trade-off compared to using buffers directly.
- Benchmarking and comparing I/O performance against standard Common Lisp methods like vector-push-extend and flexi-streams.
- Building applications that require efficient binary data serialization or deserialization with minimal memory overhead.
Not Ideal For
- Projects that rely heavily on standard Lisp stream APIs and cannot accept the 3-4x performance penalty of the gray-streams wrapper.
- Applications requiring automatic memory management for foreign data, as static-vectors necessitate manual freeing and risk leaks.
- Teams seeking a drop-in, high-level serialization library with built-in support for formats like JSON or protocol buffers.
- Simple binary I/O tasks where the overhead of learning and integrating fast-io isn't justified by marginal performance gains.
Pros & Cons
Pros
Benchmarks show fast-io writes 50-byte vectors 8.5x faster than vector-push-extend and 28x faster than flexi-streams, with execution time dropping from 0.767s to 0.090s.
Reduces bytes consed by over 80% compared to standard methods, from 104.8M to 18.4M in benchmarks, minimizing garbage collection overhead.
Provides a comprehensive set of functions like write32-be and readu16-le for reading/writing 8- to 128-bit values in both big- and little-endian formats.
Supports output to static-vectors via :static option, enabling direct pointer passing to foreign functions without extra copying, as highlighted in the README.
Cons
The gray-streams wrapper is 3-4x slower than using buffers directly, with benchmarks showing 1.7s vs 0.8s for writing 1M vectors, limiting its utility for stream-heavy code.
Static-vectors require explicit freeing with static-vectors:free-static-vector, adding complexity and potential for memory leaks if not handled correctly.
Focused solely on raw octet operations; lacks built-in support for serialization formats or error handling beyond basic EOF checks, requiring additional layers for complex data.
Frequently Asked Questions
Related Projects
babelBabel is a charset encoding/decoding library, not unlike GNU libiconv, written in pure Common Lisp.
named-readtablesNamed readtables for Common Lisp
trivial-garbagePortable GC-related APIs for Common Lisp.
simple-currencysimple-currency is a Common Lisp library which converts between currencies using foreign exchange information published daily by the ECB
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.