1

GitHub Copilot solves tasks written in Chinese worse than tasks written in Japanese and English.

Koyanagi et al. used 756 questions of varying difficulty levels from AtCoder contests. The results reveal that Copilot performed the worst in the Chinese setting compared to the other two languages. Inspecting the situations when Copilot succeeds for English and Japanese but fails for Chinese the authors observed that Copilot tends ot suggest incorrect code in situations where the correct code should handle multiple conditions, complex conditions, or string outputs.

Preprint https://win.tue.nl/~aserebre/...

Comments
  • 3
    well china is also very locked down compared to the others. Theres probably less training data available.
  • 1
    Your conclusion of your whitepaper should include something like "MS could consider acquiring gitee or gitcafe"
  • 1
    @asgs Since Gitee was chosen by the Ministry of Industry and Information Technology of the Chinese government to make an "independent, open-source code hosting platform for China", I doubt that MS will be able to acquire it...
Add Comment