These changes are restricted to the Buildkite CI that gets triggered on arc diff. These changes aren't necessary for the GitHub Actions CI since every build happens in a brand new "clean" virtual environment... and because it has been working without issue (barring unavoidable JCenter/Maven repo 500s).

1. Include ./gradlew clean step before ./gradlew bundleRelease. This step is included in /native/package.json: yarn clean-all, but doesn't get included in normal yarn cleaninstall.
2. Specify --no-daemon flag. From the Gradle docs:

Since Gradle 3.0, we enable Daemon by default and recommend using it for both developers' machines and Continuous Integration servers. However, if you suspect that Daemon makes your CI builds unstable, you can disable it to use a fresh runtime for each build since the runtime is completely isolated from any previous builds. (https://docs.gradle.org/current/userguide/gradle_daemon.html)

There's varying opinions online about whether this is necessary, but anecdotally I've noticed quite a few error messages involving the Gradle daemon when scanning through the logs of failed builds. It doesn't seem to add much overhead to the build time, and it seems like there's a possibility that it improves reliability. I haven't tested this in any meaningful/robust way...

3. Pass in jvmargs from the command-line. Was previously doing this via environment variable (GRADLE_OPTS) and $HOME/gradle.properties... but looks like passing from command-line has highest precedence... and is the only option that has higher precedence than project-level gradle.properties. I was previously mistaken in thinking GRADLE_OPTS and $HOME/gradle.properties were being taken into account when I previously tried to resolve the Out of Memory issue and claimed I'd tried adjusting jvmargs.

Here is the discussion on precedence from the docs (https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties):

Here are the arguments I'm passing:

"-Dorg.gradle.jvmargs=-Xmx32g -XX:MaxPermSize=32g -XX:+HeapDumpOnOutOfMemoryError"

All of the Android CI machines currently have >= 64GB of RAM, may have to adjust this up/down depending on what machine-types we go with in the future. In the future we could be smart about this and get information about available RAM from /proc/meminfo?

Ran the builds a few times with/without these changes and observed that they took marginally longer with the changes.

I think we'll just need to see over time whether this improves reliability?