Skip to main content

GitHub Copilot

Background​

AI code suggestions are the next step into helping developers to write code faster and better.

Analysis​

Pros and cons of using Copilot...

When doing something we don’t know how to do​

  • 🟢 Get faster to production (less need to learn via tutorial etc)
  • 🟢 Less steep learning curve
  • đź”´ More chance to write bad code → Is this a bad thing ? It means that the lack of knowledge will be detected in code review instead of staying unnoticed. On the other end a search on Google/StackOverflow might have taught the right way of doing things without the code review.

When doing something we already know how to do​

  • 🟢 Discovery of different solution
  • 🟢 Auto-completion of "dumb code"
  • 🟢 Forces better documentation to have better suggestions

⚠️ Warnings​

  • đź”´ Time to learn to use the tool
  • đź”´ Friction with other tools (Ex: code completion of VSCode)
  • đź”´ Almost good code but not perfect → Hard to see that there is an issue
Note

Note on cognitive load: It might seem that Copilot increases it as more things are displayed on the screen (and therefore needs to be processed) by the developer. However I find that after some time the brain process it in the background as it does for IDE suggestions.

See this research paper from Copilot for more details.

Bibliography​

Feedbacks​

  • Interesting to see if Copilot encourages us to seek/test new things.
    • Hypothesis: Steeper learning curve.
  • Is it sensible to test things for which we already have results from external studies? - Yes, because it depends on the context. - However, it can be used for lighter tests to confirm if it applies in our context. Request only a few inputs.

Next steps:

  • If working with Copilot, it would be helpful to see during a code review what code was generated by Copilot. - Goal: Identify any issues if the code was generated by Copilot. - Determine if there are more feedback or issues.
  • Review the schema “Doing something we don’t know how to do” and see how many research tasks are conducted with and without Copilot, including how many Stack Overflow searches are made.

Context switching: Code Time/Wakatime.

Survey analysis​

These reviews are from a Google Form Survey proposed to M33 employees. Quotes are in French or English to respect the original response.

Advantages​

  1. Autocomplete over doc check

    • The auto completion of code which would need documentation checks for use
    • helps me use some libraries
    • Less need to go and search for documentation

  2. Recommendation

    • Some very useful recommendations Specific functions (unit tests, algorithms) are
    • very easy to implement with copilot. Good DevX
    • propose new ideas to write functions
    • Copilot suggest me thinks I didn't know and thus enable me to code better and discover new things
    • It makes me learn new things
    • it often suggests a simpler approach than some habits I've had ingrained for years. I also really like writing pseudo code for a new feature in comments and seeing what it generates, because I often learn something new or at least a small optimization.

  3. Write boring easy code

    • Produce basic/classic/repetitive code.
    • When we write a repetitive code, it understands it and write it for you instead, big gain of time. It understands the code logic, will write efficiently small functions / types if you names your variables correctly. I would say it makes me code 20% faster.
    • Ecrire du code redondant devient tres facile (comme les tests, les mocks, ...)
    • Documentation autofill (for open source lib i dev) Json schema definition Simple functions implem (1 to 1 mapper) Repetitive pattern in the code base
    • Relieve the developer from writing any repetitive piece of code
    • Facile de rapidement Ă©crire du code rĂ©pĂ©titif comme insĂ©rer un spacer ou un bout de texte
    • Speeds up code I've written very often
    • Generate tests boilerplate, regular expressions, imports
    • Copilot is really powerful to write boilerplate codes (POJO class or basic enums for instance)
    • I save so much time not scrolling/clicking to double check other parts of my code to see what a variable name was or how I am supposed to use a certain library, and it really adds up.
    • Makes menial work like setting things up much easier. it is really good at doing repetitive work that you need to apply in a few places.
    • It definitely speeds up tedious processes, such as creating variations of the same code, and is smart enough to deduce/create new functions out of existing code

  4. Others

    • Time saving. For front end development, github copilot increased my productivity ~25% which is huge. For back end development, it is less impressive ~5%, but it still makes a significative difference on the long run.

Disadvantages​

  1. Distraction

    • Noise when writing documentation inline, or for some tests
    • It can distract you with bad auto-completion while coding.
    • Les suggestions dĂ©concentrent lorsque l'on rĂ©flĂ©chit Ă  quoi Ă©crire et que l'on fait quelque chose de complexe.
    • Can be annoying when the proposed code does not work and it keeps suggesting it

  2. Bug/Bad code

    • The solution is more often not helping than helping in my experience. The code generated is too long and sometimes way out of the expected response. The syntax is very often not handled properly (not closing brackets, etc.)
    • 1 bug introduced (in a config file, not caught by a dev)
    • The closing parenthesis or braces are often not matched with my existing code. Its results are not always on the spot
    • You need to know how to use it correctly (basically: can introduce bugs)
    • It is often slightly wrong, especially in a large existing codebase
    • It's really bad with closing brackets and quotes sometimes It's also really easy to introduce bugs into your code with it since it's easy to lose control over what it's doing
    • Copilot can sometimes hallucinate code snippets which don't work but look like they should
    • Might give a false sense of security, even though the developer should always check the code written by Copilot.

  3. Hurts learning

    • If you learn a new language, copilot is often writing things that you don't understand, which often kill an opportunity to learn the language.
    • When you're learning I think it's not a good tool because you just select what Copilot suggest you
    • Discourages from learning the basics of the language/framework the hard way

  4. Hard to debug

    • When copilot is wrong it's tougher to debug

  5. Others

    • Il a parfois du mal a cerner le contexte
      • notamment dans un mono repo web et app il va proposer des composants web a l'app et vice versa
      • et meme Ă  l'intĂ©rieur d'un meme fichier si on se met au dĂ©but/milieu il a du mal Ă  comprendre le code qui est en dessous
    • This is unclear what data GitHub collects
    • Using code without knowing his source, it seems both a security risk and a bypass on the right of using the work of someone else