A Common Lisp library for distributing computational tasks across multiple machines using the lparallel API.
lfarm is a Common Lisp library that enables distributed computing by extending the lparallel API to work across multiple machines. It allows developers to parallelize tasks across a cluster of servers, making it possible to scale computations beyond a single machine. The library provides a familiar interface for those already using lparallel, with added support for remote task execution and security features.
Common Lisp developers who need to scale parallel computations beyond a single machine, such as those working on data processing, scientific computing, or high-performance applications.
Developers choose lfarm because it seamlessly extends the lparallel API to distributed environments, minimizing the learning curve. It offers robust fault tolerance, portable serialization across Lisp implementations, and built-in security options like Kerberos authentication.
Distribute work across machines using the lparallel API.
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Exports the lparallel kernel API, allowing existing parallel code to run distributed with minimal changes, as shown in the synopsis where functions like pmapcar and plet work identically.
Enables computations to scale across multiple machines by implementing remote servers, making it ideal for CPU-intensive tasks beyond single-machine limits, such as batch processing or scientific computing.
Client automatically reconnects on connection errors, permitting server restarts without kernel disruption, which enhances system reliability in unstable network environments.
Supports built-in Kerberos/GSSAPI authentication and provides guidance for SSH tunneling, offering secure remote execution, though setup can be complex as detailed in the security section.
Requires manual server management and configuration of security mechanisms like Kerberos keytabs or SSH tunnels, which can be time-consuming and error-prone compared to integrated solutions.
Closures with lexical functions are not fully supported on all Lisp implementations, restricting the types of tasks that can be easily distributed, as noted in the Tasks section, and serialization relies on cl-store with potential performance trade-offs.
Missing features like task-handler-bind and ineffective debug options (*debug-tasks-p*) reduce debugging capabilities, and the potential for duplicate task execution due to reconnection logic poses challenges for idempotent operations.