using `runIdeForUiTests` in GitHub Actions on Windows & MacOs

Answered

(also posted in slack/intellij-platform)

hey,

we've been configuring UI-tests (runIdeForUItests) in GitHub actions. And whereas on Linux things work fine, there are some challenges related to the Mac and Windowshttps://github.com/Aalto-LeTech/intellij-plugin/runs/1916759334

On Mac, tests start execution, thus figuring out what is the problem is complicated. The same test works nicely on the real MacOS of the same version. Could it be some issue with MacOS screen resolution inside "the box"?
Any advice appreciated! :pray:

A+ Courses plugin: https://plugins.jetbrains.com/plugin/13634-a-courses

5 comments
Comment actions Permalink

Hello, I've tried to run a test on my real MacOs and it also works fine. Unfortunately we have no experience in launching tests on GitHub Actions machines, but I can try to help to figure out what is the problem exactly. My suggest would be to catch a snapshot of UI hierarchy when `remote-robot` fails to find a second heavyWeightWindow(where 'Download JDK...' text is located). 

object HierarchyDownloader {
private val client = OkHttpClient()
private const val baseUrl = "http://127.0.0.1:8082"

fun catchHierarchy(code: () -> Unit) {
try {
code()
} finally {
HierarchyDownloader.saveHierarchy()
}
}

private fun saveHierarchy() {
val hierarchySnapshot = saveFile(baseUrl, "build/hierarchy-reports", "hierarchy-${System.currentTimeMillis()}.html")
if (File("build/hierarchy-reports/styles.css").exists().not()) {
saveFile("$baseUrl/styles.css", "build/hierarchy-reports", "styles.css")
}
println("Hierarchy snapshot: ${hierarchySnapshot.absolutePath}")
}
private fun saveFile(url: String, folder: String, name: String): File {
val response = client.newCall(Request.Builder().url(url).build()).execute()
return File(folder).apply {
mkdirs()
}.resolve(name).apply {
writeText(response.body()?.string() ?: "")
}
}
}
with(heavyWeightWindow()) {
findText("Add SDK").click()
HierarchyDownloader.catchHierarchy {
heavyWeightWindow().findText("Download JDK...").click()
}
}

Hope it will help to understand how the UI looks in the moment of test fail.

As for Windows, it seems that IntelliJ Idea was not started yet when test was launched. I can't find `./gradlew runIdeForUiTests &` in `Setup UI env` step logs which you have in Mac and Linux jobs.

0
Comment actions Permalink

hey Eugene,

thanks for a quick response.

tried the approach, (for that particular `macos-latest`) look like this now:
https://github.com/Aalto-LeTech/intellij-plugin/actions/runs/595544930
and the hierarchy report is here: https://transfer.sh/bSgo2/hierarchy-reports.zip

What comes to the `windows-latest`, the `runIdeForUi...` task is there, a bit hidden behind the netstat output, thus: https://github.com/Aalto-LeTech/intellij-plugin/actions/runs/595544930/workflow#L83

0
Comment actions Permalink

Yep, in the hierarchy snapshot there is no second HeavyWeightWindow after click at 'Add SDK'. I've tried Github Actions and now understand better how to deal with it. I've tried to add sleep, to wait longer for the HeavyWeightWindow, but the only thing that helped was a second click. 

customComboBox("\u001BProject SDK:").dropdown()
with(heavyWeightWindow()) {
attempt(2) {
findText("Add SDK").click()
heavyWeightWindow().findText("Download JDK...").click()
}
}

https://github.com/nizienko/intellij-plugin/runs/1981615836?check_suite_focus=true

I also configured logging and added screenshot capturing on error in my fork. That might help to investigate test fails. Please check https://github.com/nizienko/intellij-plugin/blob/automate-ui-testing-win-mac/src/e2e/kotlin/fi/aalto/cs/apluscourses/e2e/utils/Utils.kt

As for windows issue, hope to check it tomorrow

0
Comment actions Permalink

Hello Eugene Nizienko,

superb :-) !

And the way you tackle this is far more mature (logging, screenshots) than the ininital approach. The tricks, additionally to **tmate** will hopefully help us in the future too, thanks!

I've now merging your changes into own fork and then into the upstream. Looking forward for your discoveries with **Windows**. I think there are at least coupple more teams (RH Quarkus, AWS Toolkit), that would be interested to see the solution multiple-environment UI testing either as a template or at least on some (e.g. ours) example.

In addition, letting you know, that I've earlier made a GH issue for a plugin template repo as Jakub Chrzanowski requested.

Thanks again !/ Огромное спасибо! :wink:

0
Comment actions Permalink

Eugene,

this is slightly puzzling, but it seems, that Windows build worked as well:

https://github.com/Aalto-LeTech/intellij-plugin/actions/runs/602220172 :open_mouth:

however, on the average, things still fail:
https://github.com/Aalto-LeTech/intellij-plugin/runs/1987047131?check_suite_focus=true

0

Please sign in to leave a comment.