feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

Randall-Jiang · 2025-02-20T02:27:27Z

Types of changes

Bug fix (non-breaking change which fixes an issue)
[ X] New feature (non-breaking change which adds functionality)

Description

This PR introduces a build-exec loop for internal Amazon users, allowing them to run a customized build command after Amazon Q generates unit tests. The goal is to ensure that the newly generated unit tests do not break the initial build.

Solution

Adding back build and execute logic for internal amazon users to fix the errors when generating unit tests. Allow users to run 3 iterations to fix the codes if there are any issues cause build failure.

TO-DO

Add telemetry for build & execute loop
Fixing the UI base on the PM's requirement

Checklist

[ x] My code follows the code style of this project
I have added tests to cover my changes
[ x] A short description of the change has been added to the CHANGELOG if the change is customer-facing in the IDE.
I have added metrics for my changes (if required)

License

I confirm that my contribution is made under the terms of the Apache 2.0 license.

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -89,6 +88,10 @@
        val session = codeTestChatHelper.getActiveSession()
        session.isGeneratingTests = true
        session.iteration++
+        if (session.testGenerationJobGroupName.isNullOrEmpty()) {


.../software/aws/toolkits/jetbrains/services/amazonqCodeTest/model/BuildAndExecuteStatusIcon.kt

@@ -25,15 +28,19 @@
        BuildAndExecuteStatusIcon.WAIT.icon
    } else if (progressStatus == BuildAndExecuteProgressStatus.RUN_EXECUTION_TESTS) {
        BuildAndExecuteStatusIcon.CURRENT.icon
+    } else if (progressStatus == BuildAndExecuteProgressStatus.BUILD_FAILED || progressStatus == BuildAndExecuteProgressStatus.FIXING_TEST_CASES) {


laileni-aws · 2025-02-20T18:12:16Z

...ws/toolkits/jetbrains/services/codewhisperer/codetest/sessionconfig/CodeTestSessionConfig.kt

+            val buildLogFilePath = Path.of(utgDir, buildAndExecuteLogDir, "buildAndExecuteLog")
+                .toString().replace("\\", "/") // Ensures consistent path format
+
+            LOG.debug { "Adding build logs to ZIP at: $buildLogFilePath" }


Do we need this log?

I added this log to make sure the build log is added to the currect folder.

Yeah but we should not print this out for the customers. So, we can remove this.

laileni-aws · 2025-02-20T18:14:33Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

+
+                    return
+                }
+//                taskContext.progressStatus = BuildAndExecuteProgressStatus.RUN_EXECUTION_TESTS


Do we need this commented code? as this is being added as part of this PR

I think this commented code is for execution loop.

seems like a bad practice to leave commented out code as future TODO. Can you put it in a different branch and PR when ready? lets not add clutter

I think we can remove this code, anyway we will have git history to track.

laileni-aws · 2025-02-20T18:15:35Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

@@ -821,13 +828,12 @@ class CodeTestChatController(
                    "Successfully sent test generation telemetry. RequestId: ${
                        testGenerationEventResponse.responseMetadata().requestId()}"
                }
-                sessionCleanUp(session.tabId)
+                // sessionCleanUp(session.tabId)


Remove commented code.

laileni-aws · 2025-02-20T18:17:07Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

@@ -691,24 +695,27 @@ class CodeTestChatController(
                        canBeVoted = false
                    )
                )
-                sessionCleanUp(session.tabId)
+                if (!isInternalUser) {


Did you test this by login with Builder Id vs internal user?

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

laileni-aws · 2025-02-20T18:20:14Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -175,7 +179,7 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
        var shortAnswer = ShortAnswer()
        LOG.debug {
            "Q TestGen session: ${codeTestChatHelper.getActiveCodeTestTabId()}: " +
-                "polling result for id: ${job.testGenerationJobId()}, group name: ${job.testGenerationJobGroupName()}, " +
+                "polling result for id: ${job.testGenerationJobId()}, group name: ${session.testGenerationJobGroupName}, " +


nit: not required as both job.testGenerationJobGroupName() and session.testGenerationJobGroupName will be same

laileni-aws · 2025-02-20T18:21:29Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -250,10 +255,9 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
                    if (shortAnswer.stopIteration == "true") {
                        throw CodeTestException("TestGenFailedError: ${shortAnswer.planSummary}", "TestGenFailedError", shortAnswer.planSummary)
                    }
-                    val fileName = shortAnswer.sourceFilePath?.let { Path.of(it).fileName.toString() } ?: path.fileName.toString()


Why did we remove this check? Any reason?

laileni-aws · 2025-02-20T18:22:38Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

                    codeTestChatHelper.updateAnswer(
                        CodeTestChatMessageContent(
-                            message = generateSummaryMessage(fileName) + shortAnswer.planSummary,
+                            message = generateSummaryMessage(path.fileName.toString()) + shortAnswer.planSummary,


What if user opens test file and run /test. This says Generating unit test for Test file but this should be Source file.

when user opens test file and run /test there, if this says Generating unit test for Test file, that's expected right? Because utg is generating unit test for opening files?

I don't think this code change is from me, I rebased from main and the code change is there. Maybe @ashishrp-aws or @chungjac has more info about this.

Can you add this back?
Looks like it was removed as part of this PR

laileni-aws · 2025-02-20T18:24:35Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -297,7 +301,7 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
            // TODO: Modify text according to FnF
            codeTestChatHelper.addAnswer(
                CodeTestChatMessageContent(
-                    message = message("testgen.error.generic_technical_error_message"),
+                    message = message("testgen.message.failed"),


Was this necessary?
We should show users
I am experiencing technical difficulties at the moment. Please try again … instead of Test generation failed

laileni-aws · 2025-02-20T18:25:58Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

-            previousIterationContext?.buildLogFile,
-            previousIterationContext?.testLogFile
-        )
+        val combinedBuildAndExecuteLogFile =


variable declaration is not needed here, if required can we change variable name to buildLogFile etc..

I keeped this in case we need excute in the future, will do it in the next pr.

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

-                taskContext.progressStatus = BuildAndExecuteProgressStatus.FIXING_TEST_CASES
-                val buildAndExecuteMessageId = updateBuildAndExecuteProgressCard(taskContext.progressStatus, messageId, session.iteration)
+                // TODO: only go to future iterations when buildExitCode or testExitCode > 0, right now iterate regardless
+                if (taskContext.buildExitCode > 0) {


…& add telemetry for build&execute

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                        requestId = session.startTestGenerationRequestId
+                    )
+                } else {
+                    AmazonqTelemetry.unitTestGeneration(


...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

@@ -114,6 +116,7 @@
    private val authController: AuthController = AuthController(),
    private val cs: CoroutineScope,
 ) : InboundAppMessagesHandler {
+    var buildResult = false


...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

-                    requestId = session.startTestGenerationRequestId
-                )
+                if (session.iteration == 1) {
+                    AmazonqTelemetry.utgGenerateTests(


...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+
+    // Find the nearest Gradle root directory
+    var packageRoot: File? = testFileAbsolutePath.parentFile
+    while (packageRoot != null && packageRoot != projectRoot) {


laileni-aws · 2025-02-20T18:48:05Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -469,10 +473,8 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
                previousIterationContext.selectedFile
            }

-        val combinedBuildAndExecuteLogFile = combineBuildAndExecuteLogFiles(


Can we remove combineBuildAndExecuteLogFiles function as this is not being used

laileni-aws · 2025-02-22T02:32:29Z

...ws/toolkits/jetbrains/services/codewhisperer/codetest/sessionconfig/CodeTestSessionConfig.kt

+            val buildLogFilePath = Path.of(utgDir, buildAndExecuteLogDir, "buildAndExecuteLog")
+                .toString().replace("\\", "/") // Ensures consistent path format
+
+            LOG.debug { "Adding build logs to ZIP at: $buildLogFilePath" }


Yeah but we should not print this out for the customers. So, we can remove this.

laileni-aws · 2025-02-22T02:33:18Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

+
+                    return
+                }
+//                taskContext.progressStatus = BuildAndExecuteProgressStatus.RUN_EXECUTION_TESTS


I think we can remove this code, anyway we will have git history to track.

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

laileni-aws · 2025-02-25T20:09:25Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

                    codeTestChatHelper.updateAnswer(
                        CodeTestChatMessageContent(
-                            message = generateSummaryMessage(fileName) + shortAnswer.planSummary,
+                            message = generateSummaryMessage(path.fileName.toString()) + shortAnswer.planSummary,


Can you add this back?
Looks like it was removed as part of this PR

laileni-aws · 2025-02-25T20:11:14Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                        promptInputDisabledState = true,
+                        promptInputProgress = buildAndExecuteProgrogressField,
+                    )
+                }


You can optimize this!

codeTestChatHelper.updateUI( promptInputDisabledState = true, promptInputProgress = if (session.iteration == 1) { testGenProgressField(0) } else { buildAndExecuteProgrogressField } )

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

laileni-aws · 2025-02-25T20:13:54Z

...oftware/aws/toolkits/jetbrains/services/amazonqCodeTest/controller/CodeTestChatController.kt

-            /*
-            //TODO: this is for unit test regeneration build iteration loop
+
+            // TODO: this is for unit test regeneration build iteration loop


You can remove TODO here

laileni-aws · 2025-02-25T20:16:34Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+        }
+        val final = session.testGenerationJobGroupName
+
+        if (session.iteration == 1) {


You can optimize this!

codeTestChatHelper.updateUI( promptInputDisabledState = true, promptInputProgress = if (session.iteration == 1) { testGenProgressField(0) } else { buildAndExecuteProgrogressField } )

rli · 2025-02-25T21:44:04Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+    // Find the nearest Gradle root directory
+    var packageRoot: File? = testFileAbsolutePath.parentFile
+    while (packageRoot != null && packageRoot != projectRoot) {
+        if (File(packageRoot, "settings.gradle.kts").exists() || File(packageRoot, "build.gradle.kts").exists()) {


what if you're using the groovy version of the gradle build script?

rli · 2025-02-25T21:44:25Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+
+    // If no valid Gradle directory is found, fallback to the project root
+    val workingDir = packageRoot ?: projectRoot
+    println("Running command in directory: ${workingDir.absolutePath}")


use logger instead of print

rli · 2025-02-25T21:45:16Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+        packageRoot = packageRoot.parentFile
+    }
+
+    // If no valid Gradle directory is found, fallback to the project root


? does that actually make sense?

rli · 2025-02-25T21:45:33Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

@@ -109,10 +130,14 @@ fun runBuildOrTestCommand(
    }

    val processBuilder = ProcessBuilder()
-        .command(listOf(command) + args)
-        .directory(File(repositoryPath))
+        .command(listOf("zsh", "-c", "source ~/.zshrc && $command ${args.joinToString(" ")}"))


what if they are on windows? or don't use zsh?

rli · 2025-02-25T21:46:41Z

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt

+    val env = processBuilder.environment()
+
+    env["PATH"] = System.getenv("PATH")


processes should be started using GeneralCommandLine. see its doc string for an explanation on how the environment variable logic works

Randall-Jiang requested a review from a team as a code owner February 20, 2025 02:27

github-advanced-security bot found potential problems Feb 20, 2025

View reviewed changes

laileni-aws reviewed Feb 20, 2025

View reviewed changes

Randall-Jiang requested a review from a team as a code owner February 20, 2025 18:12

laileni-aws reviewed Feb 20, 2025

View reviewed changes

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt Show resolved Hide resolved

laileni-aws reviewed Feb 20, 2025

View reviewed changes

...-community/src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/utils/UTGChatUtil.kt Show resolved Hide resolved

add build-execute code

24a0832

Randall-Jiang force-pushed the build-execute branch 2 times, most recently from bfac8e6 to 24a0832 Compare February 21, 2025 21:50

address comment

4168e78

github-advanced-security bot found potential problems Feb 21, 2025

View reviewed changes

create progress bar for fixing build failure

b5de261

Randall-Jiang force-pushed the build-execute branch from 83095f6 to b5de261 Compare February 21, 2025 22:35

Randall-Jiang added 3 commits February 24, 2025 15:28

fix the bug which might cause user can not execute the build command …

9cf1786

…& add telemetry for build&execute

fix format error

ca2c128

correct the logic to send the telemetry

9151dce

github-advanced-security bot found potential problems Feb 25, 2025

View reviewed changes

Randall-Jiang force-pushed the build-execute branch from c14981b to 84062da Compare February 25, 2025 20:37

addressed comment

b5792bc

Randall-Jiang force-pushed the build-execute branch from 84062da to b5792bc Compare February 25, 2025 20:39

nit

5231789

laileni-aws reviewed Feb 25, 2025

View reviewed changes

rli reviewed Feb 25, 2025

View reviewed changes

addressing comments

9e323fe

Randall-Jiang force-pushed the build-execute branch from 78d4edf to 9e323fe Compare February 25, 2025 22:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

Randall-Jiang commented Feb 20, 2025 •

edited

Loading

laileni-aws Feb 20, 2025

Randall-Jiang Feb 21, 2025

laileni-aws Feb 22, 2025

laileni-aws Feb 20, 2025

Randall-Jiang Feb 21, 2025

samgst-amazon Feb 21, 2025

laileni-aws Feb 22, 2025

laileni-aws Feb 20, 2025 •

edited

Loading

laileni-aws Feb 20, 2025

Randall-Jiang Feb 21, 2025

laileni-aws Feb 20, 2025

laileni-aws Feb 20, 2025

laileni-aws Feb 20, 2025

Randall-Jiang Feb 21, 2025

Randall-Jiang Feb 21, 2025

laileni-aws Feb 25, 2025

laileni-aws Feb 20, 2025

laileni-aws Feb 20, 2025

Randall-Jiang Feb 21, 2025

laileni-aws Feb 20, 2025

laileni-aws Feb 22, 2025

laileni-aws Feb 22, 2025

laileni-aws Feb 25, 2025

laileni-aws Feb 25, 2025

laileni-aws Feb 25, 2025

laileni-aws Feb 25, 2025

rli Feb 25, 2025

rli Feb 25, 2025

rli Feb 25, 2025

rli Feb 25, 2025

rli Feb 25, 2025

		val env = processBuilder.environment()

		env["PATH"] = System.getenv("PATH")

feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

Are you sure you want to change the base?

feat(amazonq): Adding UTG build and execute logic back for internal Amazon customers. #5396

Conversation

Randall-Jiang commented Feb 20, 2025 • edited Loading

Types of changes

Description

Solution

TO-DO

Checklist

License

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laileni-aws Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Randall-Jiang commented Feb 20, 2025 •

edited

Loading

laileni-aws Feb 20, 2025 •

edited

Loading