ci: migrate docs + vrt websites from netlify to azure by Rajdeepc · Pull Request #5450 · adobe/spectrum-web-components

Rajdeepc · 2025-05-07T11:44:21Z

Description

This PR migrates all preview deployments (documentation site, Storybook, and Visual Regression Tests) from Netlify to Azure Blob Storage for cost efficiency, and unified deployment strategy.
.
(JK, THE MAIN GOAL IS TO AVOID THE $20 BILL THAT JOSH IS PAYING OUT OF HIS OWN POCKET)
.
A quick overview of all the stuff I did in this PR

Key Changes:

Preview Deployments Now on Azure Blob Storage

All documentation, Storybook, and VRT previews for PRs and beta builds are now deployed to Azure Blob Storage.
- PR previews use the format:
- https://swcpreviews.z13.web.core.windows.net/pr-<PR_NUMBER>/docs/
- Beta documentation:
- https://swcpreviews.z13.web.core.windows.net/beta/docs/
- Visual regression previews and high-contrast mode are available at similar URLs under each PR.
All preview URLs posted on PRs and in documentation have been updated to point to Azure.

CI/CD Pipeline Modernization

CircleCI config (.circleci/config.yml):
- Adds an install-azcopy command for installing and authenticating AzCopy CLI.
- Replaces all Netlify deployment steps with AzCopy uploads to Azure.
- Updates the logic for determining the PR number and deployment path, closely mirroring GitHub Actions logic.
- Removes Netlify and related deployment retries.
- Lint and documentation jobs updated to streamline and target Azure deployment.
GitHub Actions:
- Adds .github/actions/setup-azcopy composite action to set up AzCopy for workflows.
- Adds .github/workflows/preview-docs.yml to build, deploy, and clean up PR documentation and Storybook on Azure.
- Adds .github/workflows/trigger-ci-on-pr.yml to trigger CircleCI on PR open.
- Updates .github/workflows/urls-smoke-test.yml to check Azure preview URLs and run smoke tests against them.

Documentation and User-Facing Changes

Preview URLs in documentation updated to point to Azure Blob Storage, e.g. the beta documentation link.
Automated PR comments now use Azure URLs for all previews (docs, Storybook, VRTs, and High Contrast Mode).
Playwright E2E tests updated to handle Azure URL structure (e.g., SWC_DIR path prefix).
Rollup config and asset URLs are now prefix-aware for Azure subdirectory deployments.

Dependency and Cleanup

Removes Netlify-specific scripts and dependencies (e.g., netlify-cli, Netlify deploy scripts, related code).
Cleans up preview quick-link assets and removes unused files.

Motivation and context

Operational Benefits:

Unified hosting on Azure for all preview artifacts.
Improved reliability and scalability, especially for PR and beta builds.
THE $20 BILL

Related issue(s)

SWC-900

Author's checklist

I have read the CONTRIBUTING and PULL_REQUESTS documents.
I have reviewed at the Accessibility Practices for this feature, see: Aria Practices
I have added automated tests to cover my changes.
I have included a well-written changeset if my change needs to be published.
I have included updated documentation if my change required it.

Reviewer's checklist

Includes a Github Issue with appropriate flag or Jira ticket number without a link
Includes thoughtfully written changeset if changes suggested include patch, minor, or major features
Automated tests cover all use cases and follow best practices for writing
Validated on all supported browsers
All VRTs are approved before the author can update Golden Hash

Manual review test cases

Open a PR and verify that:
- Documentation and Storybook previews are posted as comments and accessible at the new Azure URLs.
- Visual Regression Test (VRT) previews are available and updated for each PR.
- Beta documentation is accessible at the new Azure location.
Ensure all CI jobs (CircleCI and GitHub Actions) succeed and the preview links are correct.
Check that all Netlify links/references are gone from docs and config.
Reviewers: See the posted PR comment for all preview links.

Device review

Did it pass in Desktop?
Did it pass in (emulated) Mobile?
Did it pass in (emulated) iPad?

changeset-bot · 2025-05-07T11:44:25Z

⚠️ No Changeset found

Latest commit: a6b7ed9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

github-actions · 2025-05-07T11:49:39Z

Tachometer results

Currently, no packages are changed by this PR...

.circleci/config.yml

Rajdeepc

Good job! I am happy that you have battle tested it. Feel free to get more reviews and feedback.

Rajdeepc · 2025-06-20T10:54:42Z

.circleci/config.yml

+                      wget -O azcopy.tar.gz https://github.com/Azure/azure-storage-azcopy/releases/download/v10.29.1/azcopy_linux_amd64_10.29.1.tar.gz
+                      tar -xf azcopy.tar.gz --strip-components=1
+                      chmod +x azcopy
+                      export PATH=$PATH:$(pwd)
+                      azcopy --version


Can we add this in a retry loop to mitigate any flakiness?

rubencarvalho

Only really blocking thing is the auth method - great work here! 🚀

rubencarvalho · 2025-06-20T10:57:51Z

.github/workflows/preview-docs.yml

+                  path: |
+                      .cache/yarn
+                      node_modules
+                  key: ubuntu-latest-node20-${{ hashFiles('yarn.lock') }}


We probably want to also hash the package.json, in case it changes:

Suggested change

key: ubuntu-latest-node20-${{ hashFiles('yarn.lock') }}

key: ubuntu-latest-node20-${{ hashFiles('yarn.lock', 'package.json') }}

rubencarvalho · 2025-06-20T11:02:11Z

.github/workflows/preview-docs.yml

+            - name: Install AzCopy
+              run: |
+                  # Download and install AzCopy
+                  wget -O azcopy.tar.gz https://github.com/Azure/azure-storage-azcopy/releases/download/v10.29.1/azcopy_linux_amd64_10.29.1.tar.gz


We're hard-coding the AzCopy version here, but we have access to the latest releases, could we curl it and grep the platform we want?

I only suggested this just seeing some cons which might introduced variability into our builds due to potential breakage if a release is pulled or changes format

rubencarvalho · 2025-06-20T11:05:20Z

.github/workflows/preview-docs.yml

+                  pr_hash="pr-${{ github.event.pull_request.number }}"
+                  echo "hash=${pr_hash}" >> $GITHUB_OUTPUT
+
+            - name: Install AzCopy


I see we're also installing AzCopy for this job, could we make this centralized and re-use in both places?

rubencarvalho · 2025-06-20T11:06:56Z

.github/workflows/trigger-ci-on-pr.yml

Why do we need this? 🤔

In our current CircleCI workflow, jobs are triggered on every commit. However, the pull request (PR) number is only available in the environment after a PR is opened. This means that jobs initiated by pre-PR commits lack access to the PR context. As a result, we need to manually re-run the job after the PR is created to retrieve and utilize the PR number correctly.

rubencarvalho · 2025-06-20T11:17:03Z

.github/workflows/preview-docs.yml

+                  # Upload Storybook
+                  echo "Uploading Storybook to ${PR_HASH}/docs/storybook/"
+                  azcopy copy "storybook-static/*" \
+                    "https://swcpreviews.blob.core.windows.net/\$web/${PR_HASH}/docs/storybook/?${CLEAN_SAS_TOKEN}" \


Let's not include the token here, please.
There are other ways of authenticating that do not require having the token potentially exposed like this, please take a look at using azcopy login with Azure AD credentials.

@rubencarvalho given that we've done a few runs in this PR, do you think there's a need to preemptively rotate that token now?

I might be being a little unnecessarily cautious, so happy to take a "yes" or a "no" here.

I have migrated our scripts to authorize using a service principal by using a client secret instead of a SAS tokens so I can simply remove/delete that token to be safe!

rubencarvalho · 2025-06-20T11:33:35Z

yarn.lock

So many things went away 😮
Nice!

Feels lighter. netli-bye

rubencarvalho · 2025-06-20T11:37:12Z

.github/scripts/build-preview-urls-comment.js

+        ({ system, color, scale, direction, urlPath }) => {
+            const vrtUrl = `${baseUrl}/${prHash}/${urlPath}/review/`;
+
+            previewLinks.push(`


super nitpicky, but can we push these links inline and then in the end we add the line breaks (${previewLinks.join('\n')})?

rubencarvalho · 2025-06-20T11:45:34Z

.circleci/config.yml

+                      set +x
+                      PR_NUMBER=""
+                      if [ -n "$CIRCLE_PULL_REQUEST" ]; then
+                          PR_NUMBER=$(echo $CIRCLE_PULL_REQUEST | sed 's/.*\/pull\///')


Under what circumstance does this not exist? CIRCLE_PR_NUMBER shows to always be there. 🤔
I'd love to get rid of the complexity below 😛

its flaky. i tested by making many prs and there were a total of 2 instances out of 7-8 where this didn't exist.
https://discuss.circleci.com/t/ci-pull-reuqest-and-other-pr-environment-variables-are-missing/38535
this has happened before and to a lot of many people so i wanted to avoid flakiness here

Thanks for the context! I’m OK with keeping the fallback logic for now, but could we try to pinpoint when it is missing and follow up with CircleCI support?
For example, we know the PR-related env vars are only set when the build is part of a PR - could those failures have happened on a regular branch commit?

rubencarvalho · 2025-06-20T11:47:56Z

.circleci/config.yml

-                      echo "Deployment failed after $max_attempts attempts."
-                      exit 1
+                      # Install AzCopy
+                      wget -O azcopy.tar.gz https://github.com/Azure/azure-storage-azcopy/releases/download/v10.29.1/azcopy_linux_amd64_10.29.1.tar.gz


If we end up fetching the latest version, we can also use it here.

rubencarvalho · 2025-06-20T11:51:10Z

.circleci/config.yml

+                      set +x
+
+                      # Clean the SAS token (remove any whitespace/newlines)
+                      CLEAN_SAS_TOKEN=$(echo "${AZURE_STORAGE_SAS_TOKEN}" | tr -d '\n\r\t ')


Like the comment below, let’s please switch to Azure AD-based auth here as well. SAS tokens are bearer tokens - anyone with the URL and token can access the blob, so they’re more risky to use in CI environments.

Not sure everyone can use it since circle CI will mask these tokens during build time. but its definitely a good practice to avoid leaks.

'

rubencarvalho · 2025-06-23T10:06:06Z

.circleci/config.yml

+                      # Set up Azure authentication
+                      echo 'export AZCOPY_AUTO_LOGIN_TYPE="SPN"' >> $BASH_ENV
+                      echo 'export AZCOPY_SPA_APPLICATION_ID="$AZURE_CLIENT_ID"' >> $BASH_ENV
+                      echo 'export AZCOPY_SPA_CLIENT_SECRET="$AZURE_CLIENT_SECRET"' >> $BASH_ENV
+                      echo 'export AZCOPY_TENANT_ID="$AZURE_TENANT_ID"' >> $BASH_ENV
+                      source $BASH_ENV


Is this necessary? Looks like we’re already setting these in the node environment.

Also, single quotes here prevent variable expansion, so we’d just be writing the literal strings (e.g. $AZURE_CLIENT_ID) into $BASH_ENV.

rubencarvalho

Nice!

TarunAdobe

APPROVAL GRANTED! IN FACT, APPROVAL ASCENDED TO MYTHICAL STATUS!

Rajdeepc · 2025-06-24T11:56:08Z

Superb work @TarunAdobe ! This needs a celebration.

pfulton

@TarunAdobe would you mind taking checking on the beta URL? I tried hitting the URL in your original PR description (and then also tried putting the pr number in there and moving it around to different parts of the URL structure) and either I have the wrong URL or it didn't correctly deploy.

TarunAdobe · 2025-06-25T02:12:08Z

@TarunAdobe would you mind taking checking on the beta URL? I tried hitting the URL in your original PR description (and then also tried putting the pr number in there and moving it around to different parts of the URL structure) and either I have the wrong URL or it didn't correctly deploy.

@pfulton Beta docs deploy only happens on main branch. so that link won't be accessible until this pr gets merged.

Rajdeepc changed the title ~~ci: upgraded to azure from netlify~~ ci: migrate netlify to azure May 9, 2025

cdransf reviewed May 12, 2025

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

TarunAdobe force-pushed the rajdeep/azure-circle-ci branch 11 times, most recently from 26c248d to 17651df Compare May 22, 2025 06:06

adobe deleted a comment from github-actions bot May 22, 2025

TarunAdobe force-pushed the rajdeep/azure-circle-ci branch 5 times, most recently from 98e4da9 to 982025f Compare May 22, 2025 10:53

adobe deleted a comment from github-actions bot Jun 10, 2025

TarunAdobe force-pushed the rajdeep/azure-circle-ci branch 3 times, most recently from 8f6eed3 to ade5424 Compare June 10, 2025 07:12

ci: upgraded to azure from netlify

7deed4e

TarunAdobe added 2 commits June 18, 2025 14:42

chore: rerun ci on opening a pr

8f8c78f

chore: migrate beta-docs to azure blob storage

31896de

TarunAdobe mentioned this pull request Jun 18, 2025

chore: remove netlify instances #5548

Merged

14 tasks

TarunAdobe added 4 commits June 19, 2025 08:14

chore: remove all netlify instances (#5548)

2eb38a5

Merge branch 'main' into rajdeep/azure-circle-ci

439f6c4

chore: remove netlify from gitignore

ad33e5f

Merge branch 'main' into rajdeep/azure-circle-ci

d66d6fe

Rajdeepc commented Jun 20, 2025

View reviewed changes

rubencarvalho requested changes Jun 20, 2025

View reviewed changes

TarunAdobe added 5 commits June 23, 2025 10:33

chore: hash packagejson in docs workflow

01283a8

'

chore: use spn vars instead of sas token for azure

799439a

chore: minor code review changes

5e8e27e

chore: use a secure way for azure auth in ci

6e42f20

Merge branch 'main' into rajdeep/azure-circle-ci

32b3d2d

rubencarvalho reviewed Jun 23, 2025

View reviewed changes

TarunAdobe and others added 3 commits June 23, 2025 18:48

chore: remove single quoted echo

00799ac

Merge branch 'main' into rajdeep/azure-circle-ci

46f0f0d

chore: remove debugging echos

c90e863

pfulton mentioned this pull request Jun 23, 2025

fix(azure-preview-workflow): update resource url to remove + use same token pattern adobe/spectrum-css#3979

Merged

8 tasks

TarunAdobe added 4 commits June 24, 2025 10:28

chore: add debugging logs

b42fd37

chore: add env vars to bash env

7ca2a12

chore: remove custom trigger to ci workflow

b44ac75

chore: run install azcopy always

91374a1

rubencarvalho approved these changes Jun 24, 2025

View reviewed changes

TarunAdobe approved these changes Jun 24, 2025

View reviewed changes

pfulton reviewed Jun 24, 2025

View reviewed changes

Merge branch 'main' into rajdeep/azure-circle-ci

a6b7ed9

	key: ubuntu-latest-node20-${{ hashFiles('yarn.lock') }}
	key: ubuntu-latest-node20-${{ hashFiles('yarn.lock', 'package.json') }}

Conversation

Rajdeepc commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes:

Preview Deployments Now on Azure Blob Storage

CI/CD Pipeline Modernization

Documentation and User-Facing Changes

Dependency and Cleanup

Motivation and context

Related issue(s)

Author's checklist

Reviewer's checklist

Manual review test cases

Device review

Uh oh!

changeset-bot bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

github-actions bot commented May 7, 2025

Tachometer results

Uh oh!

Uh oh!

Rajdeepc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rubencarvalho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rajdeepc Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rubencarvalho left a comment

Choose a reason for hiding this comment

Uh oh!

TarunAdobe left a comment

Choose a reason for hiding this comment

Rajdeepc commented May 7, 2025 •

edited

Loading

changeset-bot bot commented May 7, 2025 •

edited

Loading

Rajdeepc left a comment •

edited

Loading

Rajdeepc Jun 20, 2025 •

edited

Loading