Skip to main content

Use openAI to generate the tests

Parent project Hub Test Generation​

Strategy​

Use the Completion API with the model 'text-davinci-003' to generate tests by giving examples of a few good tests in the codebase and the top level (page) component I want to test.

Control points​

  • Prompt size : What is got prompt size ? What number of files ? Files length ?
  • Content :
    • Is the test syntax ok ?
    • Is the test correct ?

Limitations​

  • Completion API's current models doesn't allow to send enough files (3-4 files currently)
  • The model doesn't grasp the meaning of the component with only the top-level/parent component

Example (by Matthieu, 28/08/23)​

Prompt

preprompt = "Test of a react-native component. \n\nFirst, here are some examples of good tests: \n"

examples_of_tests = ""
with open("ExamplesOfGoodTests.tsx", "r") as file:
	examples_of_tests += file.read()

component_to_test = ""
with open("ComponentToTest.tsx", "r") as file:
	component_to_test += file.read()

prompt = preprompt + "\n" \
+ "Examples of good tests : \n" + "```" + "\n" + examples_of_tests + "```" + "\n" \
+ "---\nComponent to test : " + "\n" \
+ "```" + "\n" + component_to_test + "\n" + "```" + "\n" \
+ "---\nTest : " + "\n"


response = openai.Completion.create(
	model="text-davinci-003",
	prompt=prompt,
	max_tokens=300,
)

Part of the response :

import { getNavigationMock } from "#shared/services/testing/getNavigationMock";
import { renderWithProviders } from "#shared/services/testing/renderWithProviders";
import { fireEvent, screen, waitFor } from "@testing-library/react-native";
import { HomePage } from "./HomePage.page";

it("matches to snapshot", () => {
  renderWithProviders(<HomePage navigation={getNavigationMock()} />);
  expect(screen.getByText("Chiffre d'affaires estimé")).toBeOnTheScreen();
  expect(screen).toMatchComponentSnapshot();
});

Observations​

  • The prompt is too long (too many tokens) => I had to make examples_of_tests smaller
  • The "style" of the test is good :
    • good imports
    • name of the component to test
    • the getByText is plausible (but wrong text)
  • It doesn't know what to test, expect
  • It doesn't know what to mock

Considerations​

If the prompt contains the subcomponent and functions, is the result better ? Can a vscode extension allows us to "command + click" to get all the sub-components ?