How to install Perceval on Windows?

The README focuses on Linux/macOS with apt-get; Windows users may need to use Docker or WSL, as native pip installation might require additional setup not detailed.

Perceval vs custom scripts for GitHub data?

Perceval simplifies batch retrieval with built-in backends and authentication, whereas custom scripts offer more control but require more coding; Perceval is better for standardized, multi-platform collection.

How to fetch data from a private Git repository?

Include credentials directly in the URL, e.g., perceval git https:// : @github.com/repo, as shown in the Git backend examples for private access.

Does Perceval support real-time updates?

No, it performs batch retrieves based on parameters like --from-date; for near-real-time data, you'd need to schedule frequent runs or use other tools.

How to extend Perceval with a new backend?

Follow the modular architecture and community examples; backends are implemented in separate repositories, but the README lacks detailed documentation on creation.

What output format does Perceval use?

The README doesn't specify, but it likely outputs structured data (e.g., JSON) for integration with GrimoireLab analytics tools, requiring further processing for analysis.

Perceval

GPL-3.0Python1.4.7

A command-line tool to fetch and gather data from software repositories and development platforms using modular backends.

Visit Website

What is Perceval?

Perceval is a command-line tool that fetches and gathers data from software repositories and development platforms using modular backends. It solves the problem of collecting heterogeneous project data (e.g., commits, issues, pull requests) from sources like GitHub, GitLab, JIRA, and Slack for analysis and metrics. As part of the GrimoireLab ecosystem, it enables software development analytics and research.

Target Audience

Researchers, data scientists, and practitioners focused on software development analytics, metrics, and mining software repositories. It is also useful for teams implementing CHAOSS metrics or building custom dashboards for project insights.

Value Proposition

Developers choose Perceval for its extensive backend support, flexibility in data retrieval, and integration with the GrimoireLab stack. Its modular design and active community make it a robust open-source alternative for automated data collection from diverse development tools.

Overview

Send Sir Perceval on a quest to retrieve and gather data from software repositories.

Use Cases

Best For

Collecting commit history and code contributions from Git repositories

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

GitHub

319 stars185 forks0 contributors

Aggregating issue and pull request data from GitHub or GitLab for project analysis

Fetching discussion data from forums like Discourse, Slack, or Mattermost

Gathering build and CI data from Jenkins servers

Retrieving bug reports from Bugzilla or JIRA instances

Building datasets for research on software development processes

Not Ideal For

Teams needing real-time, streaming data ingestion for live monitoring dashboards
Users preferring graphical user interfaces or web-based tools for data extraction and management
Projects focused solely on data from a single common platform (e.g., only GitHub) where dedicated APIs or simpler scripts might suffice
Environments with legacy systems or restrictions requiring Python versions below 3.10

Pros & Cons

Pros

Extensive Backend Support

Supports over 30 backends for platforms like Git, GitHub, JIRA, and Slack, providing broad coverage for diverse data sources as listed in the README.

Flexible Data Retrieval

Allows date-range filtering and category-specific fetching (e.g., issues, pull requests), demonstrated in examples with --from-date flags for granular control.

Robust Authentication Options

Works with API tokens, SSH keys, and credentials, essential for accessing private repositories and authenticated services, as shown in backend-specific commands.

Multiple Deployment Methods

Available via PyPI, Docker, or source code with Poetry/pip, offering flexibility for different installation and integration scenarios.

Cons

Setup Overhead for Some Backends

Requires manual configuration like SSH key setup for Gerrit or token generation for Slack, adding complexity and potential points of failure.

Command-Line Only Interface

Lacks a graphical user interface, which can be a barrier for users unfamiliar with CLI tools or those seeking visual data exploration workflows.

Dependency on External Tools

Needs system packages like build-essentials and git for installation, complicating setup on minimal or restricted environments.

Frequently Asked Questions

Home

Empirical Software Engineering

PyDriller

Python Framework to analyse Git repositories

Stars955

Forks155

Last commit4 months ago

RefactoringMiner

RefactoringMiner is a Java library and API designed to automatically identify refactoring operations within code changes across multiple programming languages. It analyzes commits, pull requests, and commit ranges to detect over 100 refactoring types, from simple renames to complex structural changes. The tool also generates detailed Abstract Syntax Tree (AST) diffs, providing a deeper understanding of code evolution beyond traditional line-based diffs. ## Key Features - **Refactoring Detection** — Identifies 40+ classic refactorings from Fowler's catalog, 52 API-level changes, 8 migration patterns, and 5 test-specific refactorings. - **Multi-Language Support** — Works with Java, Python, and Kotlin codebases, with TypeScript support planned. - **AST Diff Generation** — Produces syntax-aware diffs at commit, pull request, and commit range levels. - **Visualization Tools** — Includes a Chrome extension for refactoring-aware commit reviews and interactive diff visualization in browsers. - **Advanced Diff Features** — Supports refactoring-aware tooltips, single-page views, embedded GitHub comments, and handling of code moved between files. ## Philosophy RefactoringMiner aims to make code evolution transparent and understandable by precisely tracking structural changes, helping developers and researchers analyze refactoring practices and improve code review processes.

A library for mining of path-based representations of code (and more)

Stars300

Forks78

Last commit6 months ago

DesigniteJava

Detects smells and computes metrics of Java code

Stars192

Forks68

Last commit1 year ago

Empirical Software Engineering475