Detect bug with ChatGPT from a commit

Question

Given a piece of code, can ChatGPT tell us if there is a bug without any training or example files, and if there is, what is the bug?

Short answer

No.

How?

1. Get a dataset of bug examples: see Creating a dataset of bug examples
2. Create a prompt:

def get_prompt(commit):
	prompt = """
	You are a senior react native developer with 20 years of experience.
	Here is a piece of code from a react native project, in typescript.
	"""
	for filename in random_commit.filesBefore:
		prompt += f"\nFile: {filename}\n"
		prompt += f"{random_commit.filesBefore[filename]}\n"

	prompt += """
	Your role is to find any bug or potential bug that could happens in this code.
	It could be any programmatic problem but also potential undesired end-user behavior.
	Your answer must be a JSON file, with the following structure:
	{
		"is_bugged": bool // true if there is a bug, else false,
		"explanation": str // your explanation of why there is a bug or not
	}
	"""
	return prompt

3. Ask ChatGPT:

def ask_gpt(prompt):
	response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{ 'role': 'user', 'content': prompt}],
        temperature=1,
        max_tokens=200,
        top_p=1.0,
        frequency_penalty=0.0,
        presence_penalty=0.0,
    )
    return response.choices[0]['message']['content'], response.usage

Example

Example with a commit:

The commit itself:

Random Commit Hash: 9bdaed3cb80f4bbb56a1eb079af2ea21ba0375b0
Author: anoukhello
Commit Message:
	 (PC-15100) fix missing navigation object crash (#2996)
Modified Files (State Before Modification):
File: src/features/errors/pages/AsyncErrorBoundary.tsx
File: src/ui/components/GenericErrorPage.tsx

The prompt sent to OpenAI: You are a senior react native developer with 20 years of experience. Here is a piece of code from a react native project, in typescript. File: src/features/errors/pages/AsyncErrorBoundary.tsx

import { t } from "@lingui/macro";
import React, { useEffect } from "react";
import { FallbackProps } from "react-error-boundary";
import { useQueryErrorResetBoundary } from "react-query";
import styled from "styled-components/native";

import { AsyncError, MonitoringError, eventMonitoring } from "libs/monitoring";
import { ScreenError } from "libs/monitoring/errors";
import { Helmet } from "libs/react-helmet/Helmet";
import { ButtonPrimaryWhite } from "ui/components/buttons/ButtonPrimaryWhite";
import { GenericErrorPage } from "ui/components/GenericErrorPage";
import { BrokenConnection } from "ui/svg/BrokenConnection";
import { Typo } from "ui/theme";

interface AsyncFallbackProps extends FallbackProps {
  resetErrorBoundary: (...args: Array<unknown>) => void;
  error: AsyncError | ScreenError | Error;
  headerGoBack?: boolean;
}

export const AsyncErrorBoundaryWithoutNavigation = ({
  resetErrorBoundary,
  error,
}: AsyncFallbackProps) => {
  const { reset } = useQueryErrorResetBoundary();

  useEffect(() => {
    if (
      // we already captures MonitoringError exceptions (in AsyncError constructor)
      !(error instanceof MonitoringError) ||
      // we don't need to capture those errors
      !(error instanceof ScreenError)
    ) {
      eventMonitoring.captureException(error);
    }
  }, [error]);

  const handleRetry = async () => {
    reset();
    resetErrorBoundary();
    if (error instanceof AsyncError && error.retry) {
      await error.retry();
    }
  };

  if (error instanceof ScreenError) {
    return (
      <error.Screen
        resetErrorBoundary={resetErrorBoundary}
        callback={error.retry}
        error={error}
      />
    );
  }

  return (
    <React.Fragment>
      <Helmet>
        <title>
          {t`Page erreur\u00a0: Erreur pendant le chargement` +
            " | pass Culture"}
        </title>
      </Helmet>
      <GenericErrorPage
        title={t`Oups\u00a0!`}
        icon={BrokenConnection}
        headerGoBack
        buttons={[
          <ButtonPrimaryWhite
            key={1}
            wording={t`Réessayer`}
            onPress={handleRetry}
            buttonHeight="tall"
          />,
        ]}
      >
        <StyledBody>{t`Une erreur s'est produite pendant le chargement.`}</StyledBody>
      </GenericErrorPage>
    </React.Fragment>
  );
};

export const AsyncErrorBoundary = (props: AsyncFallbackProps) => {
  return <AsyncErrorBoundaryWithoutNavigation {...props} />;
};

const StyledBody = styled(Typo.Body)(({ theme }) => ({
  color: theme.colors.white,
  textAlign: "center",
}));

File: src/ui/components/GenericErrorPage.tsx

import { t } from "@lingui/macro";
import React, { ReactNode, FunctionComponent } from "react";
import styled, { useTheme } from "styled-components/native";

import { homeNavConfig } from "features/navigation/TabBar/helpers";
import { useGoBack } from "features/navigation/useGoBack";
import { Helmet } from "libs/react-helmet/Helmet";
import { styledButton } from "ui/components/buttons/styledButton";
import { Touchable } from "ui/components/touchable/Touchable";
import { Background } from "ui/svg/Background";
import { ArrowPrevious } from "ui/svg/icons/ArrowPrevious";
import { IconInterface } from "ui/svg/icons/types";
import { getSpacing, Spacer, Typo } from "ui/theme";
import { getHeadingAttrs } from "ui/theme/typography";
import { useCustomSafeInsets } from "ui/theme/useCustomSafeInsets";

type Props = {
  headerGoBack?: boolean;
  noIndex?: boolean;
  icon?: FunctionComponent<IconInterface>;
  title: string;
  buttons?: Array<ReactNode>;
};

export const GenericErrorPage: FunctionComponent<Props> = ({
  children,
  headerGoBack,
  noIndex = true,
  icon,
  title,
  buttons,
}) => {
  const { isTouch } = useTheme();
  const { top } = useCustomSafeInsets();
  const { canGoBack, goBack } = useGoBack(...homeNavConfig);
  const Icon =
    !!icon &&
    styled(icon).attrs(({ theme }) => ({
      size: theme.illustrations.sizes.fullPage,
      color: theme.colors.white,
    }))``;

  return (
    <Container>
      {!!noIndex && (
        <Helmet>
          <meta name="robots" content="noindex" />
        </Helmet>
      )}
      <Background />
      {headerGoBack && canGoBack() ? (
        <HeaderContainer
          onPress={goBack}
          top={top + getSpacing(3.5)}
          testID="Revenir en arrière"
        >
          <StyledArrowPrevious />
        </HeaderContainer>
      ) : null}
      <Content>
        <Spacer.TopScreen />
        <Spacer.Flex />
        {!!isTouch && <Spacer.Column numberOfSpaces={spacingMatrix.top} />}
        {!!Icon && (
          <React.Fragment>
            <Icon />
            <Spacer.Column numberOfSpaces={spacingMatrix.afterIcon} />
          </React.Fragment>
        )}
        <StyledTitle>{title}</StyledTitle>
        <Spacer.Column numberOfSpaces={spacingMatrix.afterTitle} />
        {children}
        <Spacer.Column numberOfSpaces={spacingMatrix.afterChildren} />
        {!!buttons && (
          <BottomContainer>
            {buttons.map((button, index) => (
              <React.Fragment key={index}>
                {index !== 0 && <Spacer.Column numberOfSpaces={4} />}
                {button}
              </React.Fragment>
            ))}
          </BottomContainer>
        )}
        {!!isTouch && <Spacer.Column numberOfSpaces={spacingMatrix.bottom} />}
        <Spacer.Flex />
        <Spacer.BottomScreen />
      </Content>
    </Container>
  );
};

const Container = styled.View({
  flex: 1,
  alignItems: "center",
});

const spacingMatrix = {
  top: 10,
  afterIcon: 5,
  afterTitle: 5,
  afterChildren: 10,
  bottom: 10,
};

const StyledArrowPrevious = styled(ArrowPrevious).attrs(({ theme }) => ({
  color: theme.colors.white,
  size: theme.icons.sizes.small,
  accessibilityLabel: t`Revenir en arrière`,
}))``;

const HeaderContainer = styledButton(Touchable)<{ top: number }>(
  ({ theme, top }) => ({
    position: "absolute",
    top,
    left: getSpacing(6),
    zIndex: theme.zIndex.floatingButton,
  }),
);

const StyledTitle = styled(Typo.Title2).attrs(() => getHeadingAttrs(1))(
  ({ theme }) => ({
    color: theme.colors.white,
    textAlign: "center",
  }),
);

const Content = styled.View({
  flexDirection: "column",
  flex: 1,
  justifyContent: "center",
  alignItems: "center",
  paddingHorizontal: getSpacing(4),
  maxWidth: getSpacing(90),
});

const BottomContainer = styled.View({
  flex: 1,
  alignSelf: "stretch",
});

Your role is to find any bug or potential bug that could happens in this code. It could be any programmatic problem but also potential undesired end-user behavior. Your answer must be a JSON file, with the following structure:

{
    "is_bugged": bool // true if there is a bug, else false,
    "explanation": str // your explanation of why there is a bug or not
}

The answer from OpenAI:

{
  "is_bugged": true,
  "explanation": "There is a potential bug in the code. In the AsyncErrorBoundary component, the condition for capturing exceptions is written as !(error instanceof MonitoringError) || !(error instanceof ScreenError), which means that if error is an instance of either MonitoringError or ScreenError, the exception will not be captured. However, the intention might have been to capture exceptions only if error is not an instance of either MonitoringError or ScreenError. The correct condition should be if (!(error instanceof MonitoringError) && !(error instanceof ScreenError)). This bug could result in certain exceptions not being captured and handled properly."
}

The actual fix that was done:

=====================
src/features/errors/pages/AsyncErrorBoundary.tsx
---------------------
- import React, { ReactNode, useEffect } from 'react'
+ import React, { useEffect } from 'react'
- import { homeNavConfig } from 'features/navigation/TabBar/helpers'
- import { useGoBack } from 'features/navigation/useGoBack'
- import { styledButton } from 'ui/components/buttons/styledButton'
- import { Touchable } from 'ui/components/touchable/Touchable'
- import { ArrowPrevious } from 'ui/svg/icons/ArrowPrevious'
- import { getSpacing, Typo } from 'ui/theme'
+ import { Typo } from 'ui/theme'
- import { useCustomSafeInsets } from 'ui/theme/useCustomSafeInsets'
-   header?: ReactNode
+   headerGoBack?: boolean
-   header,
-         header={header}
+         headerGoBack
-   const { goBack, canGoBack } = useGoBack(...homeNavConfig)
-   const { top } = useCustomSafeInsets()
-
-   return (
-     <AsyncErrorBoundaryWithoutNavigation
+   return <AsyncErrorBoundaryWithoutNavigation {...props} />
-       {...props}
-       header={
-         !!canGoBack() && (
-           <HeaderContainer onPress={goBack} top={top + getSpacing(3.5)} testID="Revenir en arrière">
-             <StyledArrowPrevious />
-           </HeaderContainer>
-         )
-       }
-     />
-   )
-
- const StyledArrowPrevious = styled(ArrowPrevious).attrs(({ theme }) => ({
-   color: theme.colors.white,
-   size: theme.icons.sizes.small,
-   accessibilityLabel: t`Revenir en arrière`,
- }))``
-
- const HeaderContainer = styledButton(Touchable)<{ top: number }>(({ theme, top }) => ({
-   position: 'absolute',
-   top,
-   left: getSpacing(6),
-   zIndex: theme.zIndex.floatingButton,
- }))

=====================
src/ui/components/GenericErrorPage.tsx
---------------------
+ import { t } from '@lingui/macro'
+ import { homeNavConfig } from 'features/navigation/TabBar/helpers'
+ import { useGoBack } from 'features/navigation/useGoBack'
+ import { styledButton } from 'ui/components/buttons/styledButton'
+ import { Touchable } from 'ui/components/touchable/Touchable'
+ import { ArrowPrevious } from 'ui/svg/icons/ArrowPrevious'
+ import { useCustomSafeInsets } from 'ui/theme/useCustomSafeInsets'
-   header?: ReactNode
+   headerGoBack?: boolean
-   header,
+   headerGoBack,
+   const { top } = useCustomSafeInsets()
+   const { canGoBack, goBack } = useGoBack(...homeNavConfig)
-       {header}
+       {headerGoBack && canGoBack() ? (
+         <HeaderContainer onPress={goBack} top={top + getSpacing(3.5)} testID="Revenir en arrière">
+           <StyledArrowPrevious />
+         </HeaderContainer>
+       ) : null}
+ const StyledArrowPrevious = styled(ArrowPrevious).attrs(({ theme }) => ({
+   color: theme.colors.white,
+   size: theme.icons.sizes.small,
+   accessibilityLabel: t`Revenir en arrière`,
+ }))``
+
+ const HeaderContainer = styledButton(Touchable)<{ top: number }>(({ theme, top }) => ({
+   position: 'absolute',
+   top,
+   left: getSpacing(6),
+   zIndex: theme.zIndex.floatingButton,
+ }))

Analysis: OpenAI found a potential bug, but it is not the expected bug. Note that this example is one of the rare cases where it found a bug.

Tests

We can test it with by sending different fix commits (in their pre-state). Results on a test on 20 different commits:

• The 20 commits were fix commits, but we only give GPT the files before the commit diffs
• Expected: 20 files with bugs, 0 without
• Results: 1 file with bug, 19 without

Conclusion

GPT does not find any bug in the given prompt. It only checks for syntax errors or basic errors, but basically does not do anymore than eslint and so does not find anything for the types of bug introduced in our code and fixed by the fix commits.

Note that on the online version, it has a tendency to find more bugs, but it never finds the expected bug (it tends to invent some).

Note also that it is very difficult for a human to find the bug with the same input: the bugs from the fix commits seem to be more "behavior" bugs than "code" bugs.

Next steps

• Try to fine-tune the model with other fix commits?
• Try to give more inputs (e.g. the ticket or the figma)?