web-codegen-scorer
A tool for evaluating the quality of web code generated by Large Language Models (LLMs) using configurable checks and automated repair.
What is web-codegen-scorer?
Web Codegen Scorer is a specialized evaluation tool designed to assess the quality of web code produced by Large Language Models. It enables developers to make evidence-based decisions by providing consistent, repeatable measurements across different models, prompts, and frameworks, moving beyond trial-and-error approaches.
Developers and teams using LLMs to generate web application code, particularly those who need to systematically compare models, optimize prompts, or monitor code quality over time.
It focuses specifically on web code and uses well-established quality metrics like build success, runtime errors, accessibility, and security, rather than relying on generic benchmarks. It also offers automated repair attempts and a visual report viewer for comparison.
Overview
Web Codegen Scorer is a tool for evaluating the quality of web code generated by LLMs.
Use Cases
Best For
- Iterating on system prompts to find the most effective instructions for a specific web project.
- Comparing the code quality output of different LLM models (e.g., Gemini vs. OpenAI vs. Anthropic) for web development.
- Monitoring how generated web code quality changes as AI models and agents evolve over time.
- Evaluating AI-generated code against specific quality checks like accessibility, security, and coding best practices.
- Automatically attempting to repair build or runtime issues in LLM-generated web code.
- Setting up configurable evaluations with different web frameworks, tools, and custom MCP servers.
Not Ideal For
- Developers needing quick, one-off code snippets without evaluation overhead
- Projects focused on non-web technologies or niche frameworks not supported in configurations
- Teams with strict budget constraints, as it incurs LLM API costs for both generation and rating
- Scenarios requiring real-time, interactive AI coding assistance rather than batch evaluations
Pros & Cons
Pros
Allows setting up evaluations with different LLM models, web frameworks, and tools, as detailed in the command-line flags and environment config reference.
Assesses generated code for build success, runtime errors, accessibility, security, and coding best practices, providing comprehensive, empirical metrics beyond generic benchmarks.
Can automatically fix issues detected during code generation, with configurable repair attempts via the --max-build-repair-attempts flag, reducing manual intervention.
Focuses specifically on web code with established quality metrics, as emphasized in the philosophy, making it more relevant than broad coding benchmarks.
Cons
Requires setting up multiple API keys as environment variables and configuring evaluations through files, which can be a barrier for quick adoption or casual use.
The README admits that more checks are coming soon, and key features like interaction testing are on the roadmap, indicating current gaps in assessment capabilities.
Relies on external LLM APIs for both code generation and rating, leading to potential costs and vendor lock-in, with no built-in cost controls or offline alternatives.
Frequently Asked Questions
Related Projects
context7Context7 Platform -- Up-to-date code documentation for LLMs and AI code editors
repomix📦 Repomix is a powerful tool that packs your entire repository into a single, AI-friendly file. Perfect for when you need to feed your codebase to Large Language Models (LLMs) or other AI tools like Claude, ChatGPT, DeepSeek, Perplexity, Gemini, Gemma, Llama, Grok, and more.
deep-chatFully customizable AI chatbot component for your website
hashbrownHashbrown is a framework for building agents that run the browser. Built for Angular and React.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.