Is astminer still maintained?

No, the project is marked as no longer maintained on its GitHub README, so it's not receiving updates or active support from the JetBrains Research team.

How to extract ASTs from Python code using astminer?

You can use the ANTLR parser for Python by configuring the pipeline in astminer's CLI or integrating it as a Kotlin dependency, with examples provided in the documentation for setting up the extraction process.

astminer vs tree-sitter for code parsing?

astminer is focused on batch mining of path-based representations and raw ASTs for machine learning pipelines, while tree-sitter is a parser generator for incremental, real-time parsing; astminer is better for research data preprocessing, but tree-sitter excels in live editor integrations.

What output formats does astminer support?

astminer supports configurable storage formats, including path-based representations and raw ASTs, which can be outputted in ways suitable for training models, as detailed in the storages documentation.

Can astminer handle large codebases?

Yes, it includes filters to remove redundant samples and is designed for batch processing, making it capable of handling large datasets, but performance may depend on hardware and pipeline configuration.

How to extend astminer for a new language?

You can extend astminer by implementing new parsers and integrating them into the pipeline, leveraging its extensible architecture, but this requires Kotlin/Java development skills and familiarity with the codebase.

astminer — Code Representation Extraction Library

What is astminer?

astminer is a library for mining path-based representations of code and raw abstract syntax trees (ASTs) from source code. It processes code in multiple languages like Java, Python, and C++ to convert it into formats suitable for training machine learning models, addressing the need for structured code data in software engineering research.

Target Audience

Researchers and developers working on machine learning for code, software engineering tools, or code analysis pipelines who need to preprocess source code into machine-readable representations.

Value Proposition

Developers choose astminer for its multi-language support, extensible design, and focus on path-based code representations, which are essential for modern code understanding models, and its integration flexibility as a Kotlin library or CLI tool.

Overview

A library for mining of path-based representations of code (and more)

Use Cases

Best For

Extracting path-based code representations for training models like code2vec
Preprocessing source code for machine learning in software engineering research
Building custom code mining pipelines for multiple programming languages
Analyzing code style and patterns across large codebases
Converting raw source code into structured AST formats for analysis
Creating labeled datasets for code classification or generation tasks

Not Ideal For

Projects requiring real-time or live code analysis, as astminer is designed for batch processing in ML pipelines
Teams seeking an actively maintained tool with ongoing support and updates
Developers needing simple, out-of-the-box code parsing without Kotlin/Java integration overhead

Pros & Cons

Pros

Multi-language Parser Support

Supports Java, Python, C/C++, JavaScript, and PHP using parsers like ANTLR and TreeSitter, enabling cross-language code analysis for diverse research projects.

Path-based Code Mining

Extracts path-based representations as per the code2vec paper, which are crucial for training modern code understanding and similarity detection models.

Extensible Architecture

Designed for easy extension to new languages and custom processing modules, allowing researchers to adapt the pipeline for specific mining tasks.

Flexible Data Storage

Offers configurable storage formats for processed data, facilitating integration with various machine learning frameworks and output requirements.

Cons

Unmaintained Project

The README explicitly states it is no longer maintained, meaning no updates, bug fixes, or community support, which limits long-term viability.

Uneven Language Coverage

Language support is inconsistent; for example, C/C++ parsing relies only on the Fuzzy parser, which may have limited features compared to more robust parsers for other languages.

Complex Integration Setup

Requires building from source or using Docker, and integration into Kotlin/Java pipelines can be cumbersome for developers not familiar with JVM ecosystems.

astminer

What is astminer?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?

astminer

What is astminer?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?