GitHub developed Copilot, an AI-powered tool that helps developers by offering code suggestions and auto-completions while they write code. It uses multiple machine-learning models trained in a vast range of programming languages.
The goal of Copilot is to streamline the development process and reduce repetitive tasks by anticipating a developer’s coding needs. Copilot can also be used as a helper for a variety of development tasks, including code documentation, test writing, debugging, and more.
Tools like Copilot help streamline the development process and reduce potentially repetitive tasks. The primary metric for Copilot increasing productivity would be code completion time for X task.
As you may find out later in this post, AI tools like Copilot are by no means perfect and should not be a replacement for a developer’s decision-making when it comes to what to include in the code. As such, an additional metric to explore is the number of bugs introduced as a result of using Copilot. Monitor the frequency of bugs or issues reported in code generated with Copilot’s assistance, which could point to the quality and reliability of Copilot’s suggestions.
Nobody wants to use a tool that makes their lives more difficult. Collect qualitative feedback from your teams to understand Copilot’s usability and ease. Developer satisfaction is crucial when using tools like Copilot, as it directly influences productivity, collaboration, and overall morale, ultimately affecting the quality of the final product.
Let’s explore some of the common ways that developers use Copilot and the results of each. To gather insights, I held a fireside chat with my team of Ippon Technologies product engineers. They have varying enterprise experience with Copilot across multiple languages such as Java, Go, and Python.
Inline code competitions are suggestions that are akin to “auto-complete” for a line or snippet of code as a developer type in their IDE. Copilot takes into account what you’ve already typed and tries to predict the rest of the snippet by using the context that it has.
Multiple developers on the team have found that the inline completion feature of Copilot has not been as accurate when using Python compared to the results they have gotten when using Go. However, the quality of the suggestion is largely dependent on the Large Language Model (LLM) being used. If you don’t mind paying for a different LLM to use with Copilot, then it may give better results.
We found that the Copilot chat feature provides more accurate results when asking targeted questions rather than relying on autocompletion.
Tip: Using plugins helps Copilot’s LLM give smarter suggestions by providing Copilot with even more context about the desired outcome. Play around with the plugins in your IDE to see if you may achieve a better outcome that way.
Copilot offers the ability to create unit tests for your codebase, but it's essential to remember that we, as developers, are in the driver’s seat when it comes to crafting effective tests.
One common challenge arises with tests for functions that call other methods. To ensure that all methods are properly mocked, we need to be specific in our requests to Copilot. Since it operates based on our instructions, there will be times when we need to explicitly indicate how to mock the dependencies we expect.
While Copilot may not deliver a perfect solution every time, it serves as a valuable starting point for building out our tests. It can help generate a shell for our tests and produce boilerplate code. Ultimately, Copilot provides us with pieces that we can review and refine, allowing us to create robust unit tests tailored to our needs.
Tip: Double-check any tests that Copilot has written by making sure your code coverage is covering the lines of code that you’re expecting. This is especially true for methods that are supposed to be called within other methods.
Using Copilot for documentation can be quite helpful, especially when it comes to generating content efficiently. While it may not be inherently better than non-AI auto-completion tools like IntelliJ stubs, it provides valuable assistance by suggesting function parameters and formatting or helping to lay out the structure of a README.md file.
In many cases, Copilot can save us significant time by handling much of the documentation generation. It allows us to focus on refining and completing the content with small edits, making it a practical tool in our documentation workflow.
Copilot won’t know if there is a business decision behind why something might be coded a certain way, so be sure to include context for unique circumstances or peculiarities around implementation decisions.
Example: If you're using Copilot to generate a function as part of implementing user authentication, you might need to specify that your application has a unique requirement for two-factor authentication due to regulatory compliance. By providing that context, you ensure the generated code aligns with the necessary security protocols rather than relying solely on standard practices.
Copilot generally enhances the development process rather than adding friction. If you’re already familiar with your framework, it can help you pinpoint what might be missing in more complex scenarios. Additionally, Copilot aids with mundane tasks, such as deciphering code to understand its functionality.
Copilot streamlines the coding experience and reduces repetitive tasks. Offering intelligent code suggestions enables developers to focus on more complex and creative challenges, boosting their sense of accomplishment and perceived productivity. This added support can lead to greater job satisfaction as developers become more efficient in their work.
As a developer, it’s important to recognize that it’s not a perfect tool and to take suggestions with a grain of salt. Double-check the suggestions to ensure that Copilot is actually doing what you want it to do.
“It’s like a better Google.” - dev
“It’s like having an intern do the work, but at the end of the day, you still have to double-check it. Or perhaps it’s a research assistant who gives you suggestions.” -dev
When you ask it a question, Copilot’s learning model uses the files that are open in the project as context to provide you with the most accurate response. The more files you have open, the more context it has.
A trick to getting a different (but perhaps more accurate) answer from Copilot would be to open a new, blank project. When asked a question in a blank project, Copilot can give a very different answer compared to if you ask it a question in an existing project. Knowing this, you can play around with the files that you have open to see how Copilot changes its suggestions.
Prompt experimentation is a necessary strategy when using Copilot, as the quality of its suggestions often depends on how developers frame their requests. By varying the specificity and wording of prompts (whether through detailed comments or partial code snippets), developers can guide Copilot to generate code that is closer to the desired outcome.
For instance, trying different phrasing for the same functionality can lead to diverse code outputs. Leverage existing code snippets so Copilot has a pattern to follow. However, it's crucial to always review the code suggestions for accuracy, as Copilot may not always produce the correct solution. If an initial suggestion doesn’t seem right, express your concerns and ask for a re-evaluation.
Example: You might say, "This doesn't seem right; can you give it another look? The function signature doesn't seem to line up with the example I provided. My example returns two things, one of which is a custom error." This kind of feedback allows for iterative refinement, leading to better results.
If you’re a tech professional and you’re wondering where to start, the key to success with using Copilot (or any tool, really) is to start small and experiment with your approach in order to figure out what works.
While Copilot can enhance productivity, developers must critically evaluate the accuracy of AI-generated code and suggestions. By maintaining a healthy skepticism and employing best practices in validation, developers can use Copilot while minimizing risks associated with potential errors.
As AI continues to evolve, it’s crucial for developers to adapt to the changing tech landscape to remain competitive and efficient. Embracing tools such as Copilot can enhance productivity, streamline workflows, and improve code quality, allowing developers to focus on more complex tasks. As developers, we stay agile in tool adoption, and this is no different; when used correctly, it is simply another tool in our developer tool belt.
Understanding AI tools opens new opportunities for innovation, positioning developers to harness the full potential of emerging technologies. Staying informed and adaptable is key to thriving in an increasingly automated and AI-driven development environment. Be sure to check back regularly as we share insights and tips on effectively utilizing productivity tools to enhance your workflow and drive success.