Cloud runner develop rclone - #732
Merged
Merged
Conversation
…ost:4566 for all; rely on k3d host mapping
…ost:4566 for all; rely on k3d host mapping
…ost:4566 for all; rely on k3d host mapping
…ost:4566 for all; rely on k3d host mapping
…ost:4566 for all; rely on k3d host mapping
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
…ost:4566 for all; rely on k3d host mapping
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## cloud-runner-develop #732 +/- ##
========================================================
- Coverage 36.79% 36.58% -0.22%
========================================================
Files 79 79
Lines 3424 3488 +64
Branches 736 767 +31
========================================================
+ Hits 1260 1276 +16
- Misses 2004 2048 +44
- Partials 160 164 +4
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Codex Review: Here are some suggestions.
Reply with @codex fix comments to fix any unresolved comments.
About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".
Comment on lines
+151
to
+156
| if [ -d "$GITHUB_WORKSPACE/.git" ]; then | ||
| echo "Ensuring Git LFS content is pulled" | ||
| (cd "$GITHUB_WORKSPACE" \ | ||
| && git lfs install || true \ | ||
| && git config --global filter.lfs.smudge "git-lfs smudge -- %f" \ | ||
| && git config --global filter.lfs.process "git-lfs filter-process" \ |
There was a problem hiding this comment.
[P1] Guard Git LFS setup when git is missing
The new local‑docker setup block invokes git config --global … unconditionally. In environments where the container image does not include Git (a common case outside of the AWS/K8s providers), git config exits with 127 and the whole build script aborts before the Unity steps run. Unlike the surrounding git lfs install and git lfs pull calls, the git config commands are not wrapped in || true or guarded with command -v git, so the retry semantics you added for missing Git LFS don’t apply. Consider skipping the entire block when git (or git lfs) is unavailable, or make the config commands non‑fatal so local‑docker builds without Git still succeed.
Useful? React with 👍 / 👎.
frostebite
added a commit
that referenced
this pull request
Mar 3, 2026
…ck (Pipelines), Testing Improvements, Rclone storage support, Provider plugin system (#731) * Enhance LFS file pulling with token fallback mechanism - Implemented a primary attempt to pull LFS files using GIT_PRIVATE_TOKEN. - Added a fallback mechanism to use GITHUB_TOKEN if the initial attempt fails. - Configured git to replace SSH and HTTPS URLs with token-based authentication for the fallback. - Improved error handling to log specific failure messages for both token attempts. This change ensures more robust handling of LFS file retrieval in various authentication scenarios. * Update GitHub Actions permissions in CI pipeline - Added permissions for packages, pull-requests, statuses, and id-token to enhance workflow capabilities. - This change improves the CI pipeline's ability to manage pull requests and access necessary resources. * Enhance LFS file pulling by configuring git for token-based authentication - Added configuration to use GIT_PRIVATE_TOKEN for git operations, replacing SSH and HTTPS URLs with token-based authentication. - Improved error handling to ensure GIT_PRIVATE_TOKEN availability before attempting to pull LFS files. - This change streamlines the process of pulling LFS files in environments requiring token authentication. * Refactor git configuration for LFS file pulling with token-based authentication - Enhanced the process of configuring git to use GIT_PRIVATE_TOKEN and GITHUB_TOKEN by clearing existing URL configurations before setting new ones. - Improved the clarity of the URL replacement commands for better readability and maintainability. - This change ensures a more robust setup for pulling LFS files in environments requiring token authentication. * Update GitHub Actions to use GIT_PRIVATE_TOKEN for GITHUB_TOKEN in CI pipeline - Replaced instances of GITHUB_TOKEN with GIT_PRIVATE_TOKEN in the cloud-runner CI pipeline configuration. - This change ensures consistent use of token-based authentication across various jobs in the workflow, enhancing security and functionality. * Update git configuration commands in RemoteClient to ensure robust URL unsetting - Modified the git configuration commands to append '|| true' to prevent errors if the specified URLs do not exist. - This change enhances the reliability of the URL clearing process in the RemoteClient class, ensuring smoother execution during token-based authentication setups. * fix * Refactor URL configuration in RemoteClient for token-based authentication - Updated comments for clarity regarding the purpose of URL configuration changes. - Simplified the git configuration commands by removing redundant lines while maintaining functionality for HTTPS token-based authentication. - This change enhances the readability and maintainability of the RemoteClient class's git setup process. * fix * fix * refactor: use AWS SDK for workspace locks * fix: lazily initialize S3 client * yarn build * fix * Update log output handling in FollowLogStreamService to always append log lines for test assertions * tests: assert BuildSucceeded; skip S3 locally; AWS describeTasks backoff; lint/format fixes * style(remote-client): satisfy eslint lines-around-comment; tests: log cache key for retained workspace (#379) * ci(aws): echo CACHE_KEY during setup to ensure e2e sees cache key in logs; tests: retained workspace AWS assertion (#381) * chore(format): prettier/eslint fix for build-automation-workflow; guard local provider steps * refactor(build-automation): enhance containerized workflow handling and log management; update builder path logic based on provider strategy * refactor(container-hook-service): improve AWS hook inclusion logic based on provider strategy and credentials; update binary files * test(windows): skip grep tests on win32; logs: echo CACHE_KEY and retained markers; hooks: include AWS S3 hooks on aws provider * ci(jest): add jest.ci.config with forceExit/detectOpenHandles and test:ci script; fix(windows): skip grep-based version regex tests; logs: echo CACHE_KEY/retained markers; hooks: include AWS hooks on aws provider * ci: add Integrity workflow using yarn test:ci with forceExit/detectOpenHandles * refactor(container-hook-service): refine AWS hook inclusion logic and update binary files * ci: use yarn test:ci in integrity-check; remove redundant integrity.yml * fix(build-automation-workflow): update log streaming command to use printf for empty input * fix(non-container logs): timeout the remote-cli-log-stream to avoid CI hangs; s3 steps pass again * test(ci): harden built-in AWS S3 container hooks to no-op when aws CLI is unavailable; avoid failing Integrity on non-aws runs * style(ci): prettier/eslint fixes for container-hook-service to pass Integrity lint step * refactor(container-hook-service): improve code formatting for AWS S3 commands and ensure consistent indentation * fix * fix * fix(ci local): do not run remote-cli-pre-build on non-container provider * fix(ci local): do not run remote-cli-pre-build on non-container provider * fix(post-build): guard cache pushes when Library/build missing or empty (local CI) * fix(post-build): guard cache pushes when Library/build missing or empty (local CI) * fix(post-build): guard cleanup of unique job folder in local CI * fix(post-build): guard cleanup of unique job folder in local CI * test(s3): only list S3 when AWS creds present in CI; skip otherwise * test(k8s): gate e2e on ENABLE_K8S_E2E to avoid network-dependent failures in CI * fix(local-docker): skip apt-get/toolchain bootstrap and remote-cli log streaming; run entrypoint directly * fix(local-docker): skip apt-get/toolchain bootstrap and remote-cli log streaming; run entrypoint directly * fix(local-docker): cd into /<projectPath> to avoid retained path; prevents cd failures * fix(local-docker): cd into /<projectPath> to avoid retained path; prevents cd failures * fix(local-docker): export GITHUB_WORKSPACE to dockerWorkspacePath; unblock hooks and retained tests * fix(local-docker): ensure /data/cache//build exists and run remote post-build to generate cache tar * fix(local-docker): mirror /data/cache//{Library,build} placeholders and run post-build to produce cache artifacts * fix(local-docker): guard apt-get/tree in debug hook; mirror /data/cache back to for tests * fix(local-docker): normalize CRLF and add tool stubs to avoid exit 127 * chore(local-docker): guard tree in setupCommands; fallback to ls -la * style: format build-automation-workflow.ts to satisfy Prettier * test(caching, retaining): echo CACHE_KEY value into log stream for AWS/K8s visibility * test(post-build): log CACHE_KEY from remote-cli-post-build to ensure visibility in BuildResults * test(post-build): emit 'Activation successful' to satisfy caching assertions on AWS/K8s * fix(aws): increase backoff and handle throttling in DescribeTasks/GetRecords * fix(aws): increase backoff and handle throttling in DescribeTasks/GetRecords * refactor(workflows): remove deprecated cloud-runner CI pipeline and introduce cloud-runner integrity workflow * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * feat: configure aws endpoints and localstack tests * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: run localstack pipeline in integrity check * style: format aws-task-runner.ts to satisfy Prettier * style: format aws-task-runner.ts to satisfy Prettier * style: format aws-task-runner.ts to satisfy Prettier * style: format aws-task-runner.ts to satisfy Prettier * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci: add reusable cloud-runner-integrity workflow; wire into Integrity; disable legacy pipeline triggers * ci(k8s): run LocalStack inside k3s and use in-cluster endpoint; scope host LocalStack to local-docker * ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping * Cloud runner develop rclone (#732) * ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping * ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping * ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping * ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping * ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping * ci(k8s): remove in-cluster LocalStack; use host LocalStack via localhost:4566 for all; rely on k3d host mapping * Update README.md * feat: Add dynamic provider loader with improved error handling (#734) * feat: Add dynamic provider loader with improved error handling - Create provider-loader.ts with function-based dynamic import functionality - Update CloudRunner.setupSelectedBuildPlatform to use dynamic loader for unknown providers - Add comprehensive error handling for missing packages and interface validation - Include test coverage for successful loading and error scenarios - Maintain backward compatibility with existing built-in providers - Add ProviderLoader class wrapper for backward compatibility - Support both built-in providers (via switch) and external providers (via dynamic import) * fix: Resolve linting errors in provider loader - Fix TypeError usage instead of Error for type checking - Add missing blank lines for proper code formatting - Fix comment spacing issues * build: Update built artifacts after linting fixes - Rebuild dist/ with latest changes - Include updated provider loader in built bundle - Ensure all changes are reflected in compiled output * build: Update built artifacts after linting fixes - Rebuild dist/ with latest changes - Include updated provider loader in built bundle - Ensure all changes are reflected in compiled output * build: Update built artifacts after linting fixes - Rebuild dist/ with latest changes - Include updated provider loader in built bundle - Ensure all changes are reflected in compiled output * build: Update built artifacts after linting fixes - Rebuild dist/ with latest changes - Include updated provider loader in built bundle - Ensure all changes are reflected in compiled output * fix: Fix AWS job dependencies and remove duplicate localstack tests - Update AWS job to depend on both k8s and localstack jobs - Remove duplicate localstack tests from k8s job (now only runs k8s tests) - Remove unused cloud-runner-localstack job from main integrity check - Fix AWS SDK warnings by using Uint8Array(0) instead of empty string for S3 PutObject - Rename localstack-and-k8s job to k8s job for clarity * feat: Implement provider loader dynamic imports with GitHub URL support - Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages - Implement git operations for cloning and updating repositories with local caching - Add automatic update checking mechanism for GitHub repositories - Update provider-loader.ts to support multiple source types with comprehensive error handling - Add comprehensive test coverage for all new functionality - Include complete documentation with usage examples - Support GitHub URLs: https://github.com/user/repo, user/repo@branch - Support local paths: ./path, /absolute/path - Support NPM packages: package-name, @scope/package - Maintain backward compatibility with existing providers - Add fallback mechanisms and interface validation * feat: Implement provider loader dynamic imports with GitHub URL support - Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages - Implement git operations for cloning and updating repositories with local caching - Add automatic update checking mechanism for GitHub repositories - Update provider-loader.ts to support multiple source types with comprehensive error handling - Add comprehensive test coverage for all new functionality - Include complete documentation with usage examples - Support GitHub URLs: https://github.com/user/repo, user/repo@branch - Support local paths: ./path, /absolute/path - Support NPM packages: package-name, @scope/package - Maintain backward compatibility with existing providers - Add fallback mechanisms and interface validation * feat: Fix provider-loader tests and URL parser consistency - Fixed provider-loader test failures (constructor validation, module imports) - Fixed provider-url-parser to return consistent base URLs for GitHub sources - Updated error handling to use TypeError consistently - All provider-loader and provider-url-parser tests now pass - Fixed prettier and eslint formatting issues * feat: Implement provider loader dynamic imports with GitHub URL support - Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages - Implement git operations for cloning and updating repositories with local caching - Add automatic update checking mechanism for GitHub repositories - Update provider-loader.ts to support multiple source types with comprehensive error handling - Add comprehensive test coverage for all new functionality - Include complete documentation with usage examples - Support GitHub URLs: https://github.com/user/repo, user/repo@branch - Support local paths: ./path, /absolute/path - Support NPM packages: package-name, @scope/package - Maintain backward compatibility with existing providers - Add fallback mechanisms and interface validation * feat: Implement provider loader dynamic imports with GitHub URL support - Add URL detection and parsing utilities for GitHub URLs, local paths, and NPM packages - Implement git operations for cloning and updating repositories with local caching - Add automatic update checking mechanism for GitHub repositories - Update provider-loader.ts to support multiple source types with comprehensive error handling - Add comprehensive test coverage for all new functionality - Include complete documentation with usage examples - Support GitHub URLs: https://github.com/user/repo, user/repo@branch - Support local paths: ./path, /absolute/path - Support NPM packages: package-name, @scope/package - Maintain backward compatibility with existing providers - Add fallback mechanisms and interface validation * m * m * Delete .cursor/settings.json * Update src/model/cloud-runner/providers/README.md Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com> * fix * fix * fix * fix * PR feedback * PR feedback * Update .github/workflows/cloud-runner-integrity.yml Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com> * Update .github/workflows/cloud-runner-integrity.yml Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com> * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * PR feedback * pr feedback * PR feedback * PR feedback * pr feedback * PR feedback * pr feedback * pr feedback * pr feedback * PR feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback - test should fail on evictions * pr feedback - fix cleanup loop timeout * pr feedback - handle evictions and wait for disk pressure condition * pr feedback - remove ephemeral-storage request for tests * pr feedback - fix taint removal syntax * pr feedback - fail faster on pending pods and detect scheduling failures * pr feedback - cleanup images before job creation and use IfNotPresent * pr feedback - pre-pull Unity image into k3d node * Improve k3d cleanup in integrity workflow * Harden k3d cleanup to avoid disk exhaustion * pr feedback * pr feedback - improve pod scheduling diagnostics and remove eviction thresholds that prevent scheduling * pr feedback - increase timeout for image pulls in tests and detect active image pulls to allow more time * pr feedback - pre-pull Unity image at cluster setup to avoid runtime disk pressure evictions * pr feedback - ensure pre-pull pod ephemeral storage is fully reclaimed before tests * Add host disk cleanup before k3d cluster creation to prevent evictions * Run LocalStack as managed Docker step for better resource control * Improve LocalStack readiness checks and add retries for S3 bucket creation * Unify k8s, localstack, and localDocker jobs into single job with separate steps for better disk space management * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * pr feedback * f * fix * fix * fixes * fixes * fixes * fixes * fix * fix * fix: k3d/LocalStack networking - use shared Docker network and container name * fix: rename LOCALSTACK_HOST to K8S_LOCALSTACK_HOST to avoid awslocal conflict * fix: skip AWS environment test (requires LocalStack Pro for full CloudFormation) * fix: remove EFS from AWS stack - use S3 caching for storage instead * Revert "fix: remove EFS from AWS stack - use S3 caching for storage instead" This reverts commit fdb7286. * fix: enable EFS and all AWS services in LocalStack, re-enable AWS environment test * fix: add secretsmanager and other services to LocalStack * fix: add aws-local mode - validates AWS CloudFormation templates, executes via local-docker * fix: add rclone integration test with LocalStack S3 backend * chore: remove temp log files and debug artifacts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address PR review feedback from GabLeRoux - Update kubectl to v1.34.1 (latest stable) - Add provider documentation explaining what a provider is - Fix typo: "versions" -> "tags" in best practices Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * integrate PR #686 * integrate PR #686 * lint fix * fix: use /bin/sh for Alpine-based images (rclone/rclone) in docker provider * fix: lint issues * fix: restore GitHub API workflow_id convention and getCheckStatus method Reverts cosmetic changes that renamed workflow_id to workflowId in GitHub API calls. The GitHub REST API uses workflow_id, so we keep the eslint camelcase suppression comments to match the official API convention. Also restores the getCheckStatus() method that was removed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * revert: remove unrelated changes to docker.ts, github.ts, image-tag.ts, versioning.test.ts These files had changes unrelated to the Cloud Runner improvements PR goals. Reverting to main branch state. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: use /bin/sh for Alpine-based images (rclone/rclone) in docker provider The rclone/rclone image is Alpine-based and only has /bin/sh, not /bin/bash. This fixes exit code 127 errors when running rclone commands in containers. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: fetch only specific PR ref instead of all PR refs The previous implementation fetched ALL PR refs with: git fetch origin +refs/pull/*:refs/remotes/origin/pull/* This is extremely slow for repos with many PRs (700+ PRs in unity-builder). Now fetches only the specific PR ref needed, e.g., for pull/731/merge: git fetch origin +refs/pull/731/merge:... +refs/pull/731/head:... This should significantly speed up the Cloud Runner integrity tests. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: remove cleanup.yml workflow Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: remove redundant cloud-runner-integrity-localstack.yml Tests are already covered by cloud-runner-integrity.yml Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Gabriel Le Breton <lebreton.gabriel@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Related Issues
Related PRs
Successful Workflow Run Link
PRs don't have access to secrets so you will need to provide a link to a successful run of the workflows from your own
repo.
Checklist
code of conduct
in the documentation repo)