fix(nx-dev): cross site link checks working as expected

Author: barbados-clemens

astro-docs/project.json

@@ -86,9 +86,18 @@
       "command": "playwright show-report dist/astro-docs/playwright-report"
     },
     "validate-links": {
-      "dependsOn": ["build"],
+      "dependsOn": [
+        "build",
+        {
+          "projects": ["nx-dev"],
+          "target": "sitemap"
+        }
+      ],
       "cache": true,
       "inputs": [
+        "{projectRoot}/validate-links.ts",
+        "{workspaceRoot}/docs/**/*.md",
+        { "dependentTasksOutputFiles": "**/sitemap*.xml" },
         "{projectRoot}/src/**/*",
         "{projectRoot}/astro.config.mjs",
         "{projectRoot}/sidebar.mts",

astro-docs/src/content/docs/concepts/executors-and-configurations.mdoc

@@ -69,7 +69,7 @@ If defining a new target that needs to run a single shell command, there is a sh
 }
 ```
 
-For more info, see the [run-commands documentation](/docs/reference/nx/executors/run-commands)
+For more info, see the [run-commands documentation](/docs/guides/tasks--caching/run-commands-executor)
 
 ## Build your own Executor
 

astro-docs/src/content/docs/guides/Tips-n-Tricks/define-environment-variables.mdoc

@@ -130,7 +130,7 @@ If you want to load variables from `env` files other than the ones listed above:
 
 1. Use the [env-cmd](https://www.npmjs.com/package/env-cmd) package: `env-cmd -f .qa.env nx serve`
 2. Use [dotenvx](https://github.com/dotenvx/dotenvx): `dotenvx run --env-file=.qa.env -- nx serve`
-3. Use the `envFile` option of the [run-commands](/docs/reference/nx/executors/run-commands#envfile) builder and execute your command inside of the builder
+3. Use the `envFile` option of the [run-commands](/docs/guides/tasks--caching/run-commands-executor#envfile) builder and execute your command inside of the builder
 
 ### Ad-hoc Variables
 

astro-docs/src/content/docs/guides/Tips-n-Tricks/feature-based-testing.mdoc

@@ -152,7 +152,7 @@ When running tests from multiple features in parallel, be mindful of shared reso
 
 - **Using unique test data**: Don't rely on specific database records or application state
 - **Managing ports**: Configure each test to use different ports, or let the test framework find free ports automatically
-  - For Cypress, use the [`--port` flag](/nx-api/cypress/executors/cypress#port) to specify or auto-detect ports
+  - For Cypress, use the [`--port` flag](/docs/technologies/test-tools/cypress/executors) to specify or auto-detect ports
   - For Playwright, the `webServerAddress` can be dynamically assigned
 - **Isolating state**: Use test-specific user accounts, temporary data, or cleanup between tests
 

astro-docs/src/content/docs/technologies/angular/Guides/nx-and-angular.mdoc

@@ -398,7 +398,7 @@ Learn more about the [graph features here](/docs/features/explore-graph).
 
 ### Extensible and Customizable: Make it fit your own needs
 
-Nx is [built to be extensible](/docs/getting-started/intro#start-small-extend-as-you-grow). Just like the [packages published by the Nx core team](/docs/plugin-registry) you can create your own Nx plugins by [extending Nx](/docs/extending-nx/intro). This can be as simple as using [run-commands](/docs/reference/nx/executors/run-commands) to integrate custom commands into the project configuration or as complex as [creating your own local executor](/docs/extending-nx/local-executors).
+Nx is [built to be extensible](/docs/getting-started/intro#start-small-extend-as-you-grow). Just like the [packages published by the Nx core team](/docs/plugin-registry) you can create your own Nx plugins by [extending Nx](/docs/extending-nx/intro). This can be as simple as using [run-commands](/docs/guides/tasks--caching/run-commands-executor) to integrate custom commands into the project configuration or as complex as [creating your own local executor](/docs/extending-nx/local-executors).
 
 And if you ever need to expand beyond Angular or diversify your stack, you can still keep using Nx, which is [battle-tested with many different technologies](/docs/technologies).
 

astro-docs/src/content/docs/technologies/module-federation/concepts/nx-module-federation-technical-overview.mdoc

@@ -5,7 +5,7 @@ filter: 'type:Concepts'
 ---
 
 Nx's Module Federation support is provided through a mixture of `executors` and the `withModuleFederation()` util that is used in your `webpack.config` or `rspack.config` file. Understanding what is happening under the hood can help when developing applications that use Module Federation as well as debugging any potential issues you run into.
-With Rspack, Module Federation support can also be provided through the [`NxModuleFederationPlugin`](nx-api/module-federation/documents/nx-module-federation-plugin) and [`NxModuleFederationDevServerPlugin`](nx-api/module-federation/documents/nx-module-federation-dev-server-plugin) plugins that can be used in the `rspack.config` file when utilizing [Inferred Tasks](/docs/concepts/inferred-tasks).
+With Rspack, Module Federation support can also be provided through the [`NxModuleFederationPlugin`](/docs/technologies/module-federation/guides/nx-module-federation-plugin) and [`NxModuleFederationDevServerPlugin`](/docs/technologies/module-federation/guides/nx-module-federation-dev-server-plugin) plugins that can be used in the `rspack.config` file when utilizing [Inferred Tasks](/docs/concepts/inferred-tasks).
 
 ## What happens when you serve your host?
 

astro-docs/validate-links.ts

@@ -1,18 +1,10 @@
 import fs from 'node:fs';
 import path from 'node:path';
 import { workspaceRoot } from '@nx/devkit';
+import glob from 'glob';
 
-const ignoredLinks = [
-  // '/reference/devkit',
-  // these are typically source from the plugin specific examples and can't change until we're pushing to canary
-  '/nx-api/',
-  '/reference/core-api',
-  // TODO: caleb make this nx api reference page
-  '/docs/reference/nx/executors',
-  'NxPowerpack-Trial-v1.1.pdf',
-  '/blog/',
-  '/contact',
-];
+// Links to pages hosted outside of both astro-docs and nx-dev sites
+const ignoredLinks = ['/contact'];
 
 // These are more so until we cut over and can modify production file links
 const filesToIgnore = [
@@ -23,7 +15,16 @@ const filesToIgnore = [
 ];
 
 const distDir = path.join(workspaceRoot, 'astro-docs', 'dist');
-const sitemapPath = path.join(distDir, 'sitemap-0.xml');
+const sitemapIndexPath = path.join(distDir, 'sitemap-index.xml');
+const sitemapFallbackPath = path.join(distDir, 'sitemap-0.xml');
+const nxDevSitemapPath = path.join(
+  workspaceRoot,
+  'dist',
+  'nx-dev',
+  'nx-dev',
+  'public',
+  'sitemap-0.xml'
+);
 
 if (!fs.existsSync(distDir)) {
   console.error(
@@ -32,9 +33,9 @@ if (!fs.existsSync(distDir)) {
   process.exit(1);
 }
 
-if (!fs.existsSync(sitemapPath)) {
+if (!fs.existsSync(sitemapIndexPath) && !fs.existsSync(sitemapFallbackPath)) {
   console.error(
-    `Sitemap does not exist at path. Have you ran the build?: ${sitemapPath}`
+    `No sitemap found at ${sitemapIndexPath} or ${sitemapFallbackPath}. Have you ran the build?`
   );
   process.exit(1);
 }
@@ -112,11 +113,169 @@ function parseSitemap(sitemapContent: string) {
   return routes;
 }
 
+/**
+ * Parses sitemap-index.xml to discover all sitemap files,
+ * then merges routes from all of them.
+ * Falls back to sitemap-0.xml if sitemap-index.xml doesn't exist.
+ */
+function loadAllSitemapRoutes(): Set<string> {
+  const allRoutes = new Set<string>();
+
+  // Load astro-docs sitemap routes
+  if (fs.existsSync(sitemapIndexPath)) {
+    const indexContent = fs.readFileSync(sitemapIndexPath, 'utf-8');
+    const locRegex = /<loc>([^<]+)<\/loc>/gi;
+    const sitemapUrls: string[] = [];
+    let match;
+
+    while ((match = locRegex.exec(indexContent)) !== null) {
+      sitemapUrls.push(match[1]);
+    }
+
+    for (const sitemapUrl of sitemapUrls) {
+      const parsedUrl = new URL(sitemapUrl);
+      const filename = path.basename(parsedUrl.pathname);
+      const sitemapFilePath = path.join(distDir, filename);
+
+      if (!fs.existsSync(sitemapFilePath)) {
+        console.warn(
+          `Sitemap file referenced in index not found: ${sitemapFilePath}`
+        );
+        continue;
+      }
+
+      const sitemapContent = fs.readFileSync(sitemapFilePath, 'utf-8');
+      const routes = parseSitemap(sitemapContent);
+      for (const route of routes) {
+        allRoutes.add(route);
+      }
+    }
+  } else {
+    // Fallback to sitemap-0.xml
+    const sitemapContent = fs.readFileSync(sitemapFallbackPath, 'utf-8');
+    for (const route of parseSitemap(sitemapContent)) {
+      allRoutes.add(route);
+    }
+  }
+
+  // Load nx-dev (Next.js) sitemap routes for cross-site validation
+  if (fs.existsSync(nxDevSitemapPath)) {
+    const nxDevSitemapContent = fs.readFileSync(nxDevSitemapPath, 'utf-8');
+    const nxDevRoutes = parseSitemap(nxDevSitemapContent);
+    console.log(
+      `Found ${nxDevRoutes.size} routes in nx-dev sitemap (${nxDevSitemapPath})`
+    );
+    for (const route of nxDevRoutes) {
+      allRoutes.add(route);
+    }
+  } else {
+    console.warn(
+      `nx-dev sitemap not found at ${nxDevSitemapPath}. Run "nx build nx-dev" to enable cross-site link validation against nx-dev routes.`
+    );
+  }
+
+  return allRoutes;
+}
+
 function toFriendlyName(file: string) {
   // TODO: resolve to the actual markdown file if possile?
   return path.relative(workspaceRoot, file);
 }
 
+/**
+ * Extracts links pointing to /docs/ paths from markdown file content.
+ * Handles both markdown links like [text](/docs/...) and card url attributes like url="/docs/..."
+ */
+function extractDocsLinksFromMarkdown(content: string): Set<string> {
+  const links = new Set<string>();
+
+  // Match markdown links: [text](/docs/...)
+  const markdownLinkRegex = /\]\(\/docs\/[^)]*\)/g;
+  let match;
+  while ((match = markdownLinkRegex.exec(content)) !== null) {
+    // Extract the path from ](/docs/...)
+    const linkPath = match[0].slice(2, -1);
+    // Strip anchors and query params
+    const clean = linkPath.split('#')[0].split('?')[0];
+    links.add(clean);
+  }
+
+  // Match card url attributes: url="/docs/..."
+  const urlAttrRegex = /url="(\/docs\/[^"]*)"/g;
+  while ((match = urlAttrRegex.exec(content)) !== null) {
+    const linkPath = match[1];
+    const clean = linkPath.split('#')[0].split('?')[0];
+    links.add(clean);
+  }
+
+  // Match reference-style links: [ref]: /docs/...
+  const refStyleRegex = /^\s*\[[^\]]+\]:\s*(\/docs\/\S*)/gm;
+  while ((match = refStyleRegex.exec(content)) !== null) {
+    const linkPath = match[1];
+    const clean = linkPath.split('#')[0].split('?')[0];
+    links.add(clean);
+  }
+
+  // Match card url attributes without leading slash: url="docs/..."
+  const urlAttrNoSlashRegex = /url="(docs\/[^"]*)"/g;
+  while ((match = urlAttrNoSlashRegex.exec(content)) !== null) {
+    const linkPath = '/' + match[1]; // normalize by prepending /
+    const clean = linkPath.split('#')[0].split('?')[0];
+    links.add(clean);
+  }
+
+  return links;
+}
+
+/**
+ * Scans docs/ markdown files for links to /docs/ paths (astro-docs pages)
+ * and validates them against the astro sitemap.
+ * Returns broken links grouped by source file.
+ */
+function validateCrossSiteLinks(
+  availableInternalRoutes: Set<string>
+): Map<string, string[]> {
+  const docsDir = path.join(workspaceRoot, 'docs');
+  const crossSiteBrokenLinks = new Map<string, string[]>();
+
+  if (!fs.existsSync(docsDir)) {
+    console.warn(
+      `docs/ directory not found at ${docsDir}, skipping cross-site link check`
+    );
+    return crossSiteBrokenLinks;
+  }
+
+  const mdFiles = glob.sync('**/*.md', { cwd: docsDir });
+  console.log(`Found ${mdFiles.length} markdown files in docs/\n`);
+
+  let totalLinks = 0;
+
+  for (const relPath of mdFiles) {
+    const fullPath = path.join(docsDir, relPath);
+    const content = fs.readFileSync(fullPath, 'utf-8');
+    const docsLinks = extractDocsLinksFromMarkdown(content);
+
+    for (const link of docsLinks) {
+      totalLinks++;
+
+      if (!availableInternalRoutes.has(link)) {
+        const existing = crossSiteBrokenLinks.get(relPath);
+        if (existing) {
+          existing.push(link);
+        } else {
+          crossSiteBrokenLinks.set(relPath, [link]);
+        }
+      }
+    }
+  }
+
+  console.log(
+    `Checked ${totalLinks} cross-site links from docs/ to /docs/ astro pages\n`
+  );
+
+  return crossSiteBrokenLinks;
+}
+
 function validateLinks() {
   const linksToFiles = new Map<string, string[]>();
 
@@ -151,16 +310,9 @@ function validateLinks() {
     }
   }
 
-  const filteredLinks = Array.from(linksToFiles.keys()).filter((href) => {
-    const includesIgnoredLink = ignoredLinks.some((il) => href.includes(il));
-
-    if (includesIgnoredLink) {
-      console.warn(`Skipping link since matching manual ignore list: ${href}`);
-    }
-
-    // filter out any manually ingored links
-    return !includesIgnoredLink;
-  });
+  const filteredLinks = Array.from(linksToFiles.keys()).filter(
+    (href) => !ignoredLinks.some((il) => href.includes(il))
+  );
 
   const actualLinksUsed = new Set(filteredLinks);
   console.log(
@@ -169,20 +321,22 @@ function validateLinks() {
 
   console.log('📍 Parsing sitemap for valid routes...');
 
-  const sitemapContent = fs.readFileSync(sitemapPath, 'utf-8');
-  const availableInternalRoutes = parseSitemap(sitemapContent);
+  const availableInternalRoutes = loadAllSitemapRoutes();
   console.log(
     `Found ${availableInternalRoutes.size} unique routes in sitemap\n`
   );
 
-  console.log('✅ Validating links...\n');
+  console.log('✅ Validating astro internal links...\n');
 
   // Find links that exist in actualLinksUsed but not in availableInternalRoutes
   const brokenLinks: Set<string> = new Set(
     [...actualLinksUsed].filter((link) => !availableInternalRoutes.has(link))
   );
 
+  let hasBrokenLinks = false;
+
   if (brokenLinks.size > 0) {
+    hasBrokenLinks = true;
     console.log(`Found ${brokenLinks.size} broken links:\n`);
 
     const filesWithErrors = new Map<string, string[]>();
@@ -217,7 +371,31 @@ function validateLinks() {
     console.log(
       `\n🔎 Check the above output to resolve the ${brokenLinks.size} broken links in each respecitve source (.mdoc, .astro, and/or content collection generation`
     );
+  }
+
+  // Cross-site validation: check docs/ markdown links to /docs/ astro pages
+  console.log(
+    '\n🔗 Validating cross-site links from docs/ markdown to astro pages...\n'
+  );
+  const crossSiteBrokenLinks = validateCrossSiteLinks(availableInternalRoutes);
+
+  if (crossSiteBrokenLinks.size > 0) {
+    hasBrokenLinks = true;
+    let totalCrossSiteBroken = 0;
+    for (const [file, badLinks] of crossSiteBrokenLinks) {
+      totalCrossSiteBroken += badLinks.length;
+      console.log(
+        `\n❌ docs/${file} has ${badLinks.length} broken cross-site links:`
+      );
+      badLinks.forEach((link) => console.log(`\t- ${link}`));
+    }
+
+    console.log(
+      `\n🔎 Found ${totalCrossSiteBroken} broken cross-site links from docs/ markdown pointing to non-existent astro pages`
+    );
+  }
 
+  if (hasBrokenLinks) {
     process.exit(1);
   }
 

nx-dev/nx-dev/project.json

@@ -36,8 +36,7 @@
       },
       "inputs": [
         "{workspaceRoot}/docs/**/*",
-        "{workspaceRoot}/dist/nx-dev/nx-dev/public/sitemap*.xml",
-        "{workspaceRoot}/astro-docs/dist/sitemap*.xml",
+        { "dependentTasksOutputFiles": "**/sitemap*.xml" },
         "{workspaceRoot}/scripts/tsconfig.scripts.json",
         "{workspaceRoot}/scripts/documentation/internal-link-checker.ts"
       ]
@@ -46,6 +45,7 @@
       "dependsOn": ["check-links"]
     },
     "sitemap": {
+      "dependsOn": ["build-base"],
       "executor": "nx:run-commands",
       "inputs": [
         "{workspaceRoot}/docs/**/*",