We help companies hire & upskill developers. A customer recently asked: What % of HackerRank problems can LLMs solve? That got us thinking—how should hiring evolve when AI can translate natural language to code?
Our belief: AI will handle much of code generation, so developers will be assessed more on SDLC skills with AI assistants.
To explore this, we’re benchmarking LLMs on real-world software dev scenarios—starting with 65 unseen problems across 10 domains. Beyond correctness, we evaluated consistency—an often overlooked aspect of AI reliability. We’re open-sourcing the dataset on Huggingface and expanding it to cover more domains, ambiguous specs, and harder challenges.
Would love the HN community’s take on this!
Comments URL: https://news.ycombinator.com/item?id=43015631
Points: 4
# Comments: 0
Chcete-li přidat komentář, přihlaste se
Ostatní příspěvky v této skupině
Article URL: https://lapcatsoftware.com/articles/2025/2/5.html
Article URL: https://github.com/Hawzen/hdp
Comments URL: https://news.ycombinator.com/ite
Article URL: https://bugzilla.mozilla.org/show_bug.cgi?id=1950144
Article URL: https://www.shapr3d.com/blog/history-of-cad
Comments URL: http
Article URL: https://johnoliverwantsyourraterotica.com/
Comments URL: https:
Article URL: https://gamma.app/docs/a16z-AI-Voice-Update-2025--ttkorld8iy6wfnj?mode=doc
Comments URL
