using `runIdeForUiTests` in GitHub Actions on Windows & MacOs

Hello, I've tried to run a test on my real MacOs and it also works fine. Unfortunately we have no experience in launching tests on GitHub Actions machines, but I can try to help to figure out what is the problem exactly. My suggest would be to catch a snapshot of UI hierarchy when `remote-robot` fails to find a second heavyWeightWindow(where 'Download JDK...' text is located). 

object HierarchyDownloader {
    private val client = OkHttpClient()
    private const val baseUrl = "http://127.0.0.1:8082"

    fun catchHierarchy(code: () -> Unit) {
        try {
            code()
        } finally {
            HierarchyDownloader.saveHierarchy()
        }
    }

    private fun saveHierarchy() {
        val hierarchySnapshot = saveFile(baseUrl, "build/hierarchy-reports", "hierarchy-${System.currentTimeMillis()}.html")
        if (File("build/hierarchy-reports/styles.css").exists().not()) {
            saveFile("$baseUrl/styles.css", "build/hierarchy-reports", "styles.css")
        }
        println("Hierarchy snapshot: ${hierarchySnapshot.absolutePath}")
    }
    private fun saveFile(url: String, folder: String, name: String): File {
        val response = client.newCall(Request.Builder().url(url).build()).execute()
        return File(folder).apply {
            mkdirs()
        }.resolve(name).apply {
            writeText(response.body()?.string() ?: "")
        }
    }
}

with(heavyWeightWindow()) {
    findText("Add SDK").click()
    HierarchyDownloader.catchHierarchy {
        heavyWeightWindow().findText("Download JDK...").click()
    }
}

Hope it will help to understand how the UI looks in the moment of test fail.

As for Windows, it seems that IntelliJ Idea was not started yet when test was launched. I can't find `./gradlew runIdeForUiTests &` in `Setup UI env` step logs which you have in Mac and Linux jobs.

hey Eugene,

thanks for a quick response.

tried the approach, (for that particular `macos-latest`) look like this now:
https://github.com/Aalto-LeTech/intellij-plugin/actions/runs/595544930
and the hierarchy report is here: https://transfer.sh/bSgo2/hierarchy-reports.zip

What comes to the `windows-latest`, the `runIdeForUi...` task is there, a bit hidden behind the netstat output, thus: https://github.com/Aalto-LeTech/intellij-plugin/actions/runs/595544930/workflow#L83

Yep, in the hierarchy snapshot there is no second HeavyWeightWindow after click at 'Add SDK'. I've tried Github Actions and now understand better how to deal with it. I've tried to add sleep, to wait longer for the HeavyWeightWindow, but the only thing that helped was a second click. 

customComboBox("\u001BProject SDK:").dropdown()
with(heavyWeightWindow()) {
    attempt(2) {
        findText("Add SDK").click()
        heavyWeightWindow().findText("Download JDK...").click()
    }
}

https://github.com/nizienko/intellij-plugin/runs/1981615836?check_suite_focus=true

I also configured logging and added screenshot capturing on error in my fork. That might help to investigate test fails. Please check https://github.com/nizienko/intellij-plugin/blob/automate-ui-testing-win-mac/src/e2e/kotlin/fi/aalto/cs/apluscourses/e2e/utils/Utils.kt

As for windows issue, hope to check it tomorrow

Hello Eugene Nizienko,

superb :-) !

And the way you tackle this is far more mature (logging, screenshots) than the ininital approach. The tricks, additionally to **tmate** will hopefully help us in the future too, thanks!

I've now merging your changes into own fork and then into the upstream. Looking forward for your discoveries with **Windows**. I think there are at least coupple more teams (RH Quarkus, AWS Toolkit), that would be interested to see the solution multiple-environment UI testing either as a template or at least on some (e.g. ours) example.

In addition, letting you know, that I've earlier made a GH issue for a plugin template repo as Jakub Chrzanowski requested.

Thanks again !/ Огромное спасибо! :wink:

Eugene,

this is slightly puzzling, but it seems, that Windows build worked as well:

https://github.com/Aalto-LeTech/intellij-plugin/actions/runs/602220172 :open_mouth:

however, on the average, things still fail:
https://github.com/Aalto-LeTech/intellij-plugin/runs/1987047131?check_suite_focus=true

Hello Seal, 

I believe I found out how to run tests on windows. We tried to run Idea in background wrong way, windows doesn't have '&' and don't throw any error for some reason, thats confusing.

- name: Run Idea
  run: start gradlew.bat :ui-test-example:runIdeForUiTests

I also found useful action:

- name: Wait for Idea started
  uses: jtalk/url-health-check-action@v1.4
  with:
    url: http://localhost:8082
    max-attempts: 12 
    retry-delay: 30s 

That helped to wait less for Idea started.

Here is my config working for all three OSs https://github.com/nizienko/intellij-ui-test-robot/actions/runs/614263211/workflow

Thanks for the issue in PluginTemplateRepo, I will add ready workflows there. 

Eugene Nizienko it worked, thank you for the help!