A GitHub study has found that the developers who report the greatest productivity gains from using GitHub Copilot, the company’s AI-based programming assistant, were those who accepted the most Copilot code suggestions. The study combined a survey of users’ perceived productivity with analysis of their Copilot usage.
GitHub on July 14 said it had surveyed more than 2,000 developers who have been using Copilot, which suggests lines of code using an AI model that has been trained on billions of lines of open source code. Copilot users who reported “huge” productivity gains were those who had found nearly 30% of Copilot’s suggestions acceptable. At the other end of the spectrum, users who reported “modest” productivity gains had found only about 23% of Copilot’s suggestions acceptable. Users reporting “medium” and “high” productivity gains had accepted roughly 27% and 28% of Copilot’s suggestions, respectively.
GitHub also found that developers did not care if they needed to rework a suggestion, as long as Copilot gave them a suitable starting point. The company noted that “Copilot isn’t designed to build software by itself, but to offer helpful suggestions that make it easier for developers to stay in the flow.” It offers developers the parts but leaves it up to them to design and build the finished product.
GitHub has published an academic research paper with its findings, “Productivity Assessment of Neural Code Completion,” and plans more studies of Copilot usage.
GitHub Copilot, which is accessible from GitHub, became generally available last month. The tool has drawn criticism, notably from the Free Software Foundation, which raised questions about the fairness, legitimacy, and legality of its use of freely licensed source code.