A Kotlin library for extracting path-based code representations and ASTs from multiple languages to prepare code for machine learning models.
astminer is a library for mining path-based representations of code and raw abstract syntax trees (ASTs) from source code. It processes code in multiple languages like Java, Python, and C++ to convert it into formats suitable for training machine learning models, addressing the need for structured code data in software engineering research.
Researchers and developers working on machine learning for code, software engineering tools, or code analysis pipelines who need to preprocess source code into machine-readable representations.
Developers choose astminer for its multi-language support, extensible design, and focus on path-based code representations, which are essential for modern code understanding models, and its integration flexibility as a Kotlin library or CLI tool.
A library for mining of path-based representations of code (and more)
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Supports Java, Python, C/C++, JavaScript, and PHP using parsers like ANTLR and TreeSitter, enabling cross-language code analysis for diverse research projects.
Extracts path-based representations as per the code2vec paper, which are crucial for training modern code understanding and similarity detection models.
Designed for easy extension to new languages and custom processing modules, allowing researchers to adapt the pipeline for specific mining tasks.
Offers configurable storage formats for processed data, facilitating integration with various machine learning frameworks and output requirements.
The README explicitly states it is no longer maintained, meaning no updates, bug fixes, or community support, which limits long-term viability.
Language support is inconsistent; for example, C/C++ parsing relies only on the Fuzzy parser, which may have limited features compared to more robust parsers for other languages.
Requires building from source or using Docker, and integration into Kotlin/Java pipelines can be cumbersome for developers not familiar with JVM ecosystems.