Add machinery for MacOS signing and notarizing

This adds two things. The first is extending ExtraFileSigner to support signing files locally, rather than shipping them off to a signing server to be signed and shipped back.

However, we are no longer using this functionality, and it remains there in case we need it later. With MacOS 15, we now have to sign every single binary, dylib, and bundle in the entire package, which is hundreds of files. Also, the previous method of signing was pretty awkward, where we'd make the .pkg and .dmg, then mount the dmg, sign the pkg, then recreate the dmg.

Finally, the VANAGON_FORCE_SIGNING env var was intended to allow you to build the package without signing for dev purposes. However, the only test it would use to determine if it should proceed with trying to sign or not was SSHing to the remote signing host. It did not test if signing actually worked on that host, and would fail even a dev build if signing failed and VANAGON_FORCE_SIGNING was unset.

Now, the flow looks like the following. Note the paths are openvox-agent specific, because that is the only Mac package we make right now. We'll make this more flexible in the future.
- If VANAGON_FORCE_SIGNING is not set, don't do any of the signing/notarizing at all. If you want to sign, you must set VANAGON_FORCE_SIGNING.
- Sign every binary, dylib, and bundle.
- Verify the signature on every binary, dylib, and bundle.
- Sign the .pkg file.
- Verify the signature on the .pkg file.
- Sign the .dmg file.
- Verify the signature on the .dmg file.
- Submit the .dmg for notarization.
- Staple the approved notarization to the .dmg file.
- Test that Gatekeeper is happy with the .dmg file.

When you have VANAGON_FORCE_SIGNING set, you must also have the following environment variables set.
 SIGNING_KEYCHAIN - the name of the keychain containing the code/installer signing identities
 SIGNING_KEYCHAIN_PW - the password to unlock the keychain
 APPLICATION_SIGNING_CERT - the identity description used for application signing
 INSTALLER_SIGNING_CERT - the identity description used for installer .pkg signing
 NOTARY_PROFILE - The name of the notary profile stored in the keychain

You must do this on a VM that has the appropriate application and installer identities (certs + private key) as well as a valid notary profile.

Author: nmburgan

.rubocop.yml

@@ -618,9 +618,6 @@ Style/MultilineIfThen:
 Style/MultilineInPatternThen:
   Enabled: true
 
-Style/MultilineTernaryOperator:
-  Enabled: true
-
 Style/NegatedIf:
   Enabled: true
 
@@ -809,3 +806,9 @@ Style/WhileUntilModifier:
 
 Style/WordArray:
   Enabled: false
+
+Style/FormatStringToken:
+  Enabled: false
+
+Style/MultilineTernaryOperator:
+  Enabled: false

lib/vanagon/platform/osx.rb

@@ -18,7 +18,7 @@ def install_build_dependencies(list_build_dependencies)
       #
       # @param project [Vanagon::Project] project to build a osx package of
       # @return [Array] list of commands required to build a osx package for the given project from a tarball
-      def generate_package(project) # rubocop:disable Metrics/AbcSize
+      def generate_package(project) # rubocop:disable Metrics/AbcSize,Metrics/PerceivedComplexity
         target_dir = project.repo ? output_dir(project.repo) : output_dir
 
         # Here we maintain backward compatibility with older vanagon versions
@@ -35,55 +35,130 @@ def generate_package(project) # rubocop:disable Metrics/AbcSize
           bom_install = []
         end
 
-        if project.extra_files_to_sign.any?
-          sign_commands = Vanagon::Utilities::ExtraFilesSigner.commands(project, @mktemp, "/osx/build/root/#{project.name}-#{project.version}")
-        else
-          sign_commands = []
+        # Previously, the "commands" method would test if it could SSH to the signer node and just skip
+        # all the signing stuff if it couldn't and VANAGON_FORCE_SIGNING was not set. It never really tested
+        # that signing actually worked and skipped it if it didn't. Now with the local commands, we really
+        # can't even do that test. So just don't even try signing unless VANAGON_FORCE_SIGNING is set.
+        unlock = 'security unlock-keychain -p $$SIGNING_KEYCHAIN_PW $$SIGNING_KEYCHAIN'
+        extra_sign_commands = []
+        sign_files_commands = []
+        # If we're not signing, move the pkg to the right place
+        sign_package_commands = ["mv $(tempdir)/osx/build/#{project.name}-#{project.version}-#{project.release}-installer.pkg $(tempdir)/osx/build/pkg/"]
+        sign_dmg_commands = []
+        notarize_dmg_commands = []
+        if ENV['VANAGON_FORCE_SIGNING']
+          # You should no longer really need to do this, but it's here just in case.
+          if project.extra_files_to_sign.any?
+            method = project.use_local_signing ? 'local_commands' : 'commands'
+            extra_sign_commands = Vanagon::Utilities::ExtraFilesSigner.send(method, project, @mktemp, "/osx/build/root/#{project.name}-#{project.version}")
+          end
+
+          # As of MacOS 15, we have to notarize the dmg. In order to get notarization, we have to
+          # code sign every single binary, .bundle, and .dylib file in the package. So instead of
+          # only signing a few files we specify, sign everything we can find that needs to be signed.
+          # We then need to notarize the resulting dmg.
+          #
+          # This requires the VM to have the following env vars set in advance.
+          #   SIGNING_KEYCHAIN - the name of the keychain containing the code/installer signing identities
+          #   SIGNING_KEYCHAIN_PW - the password to unlock the keychain
+          #   APPLICATION_SIGNING_CERT - the identity description used for application signing
+          #   INSTALLER_SIGNING_CERT - the identity description used for installer .pkg signing
+          #   NOTARY_PROFILE - The name of the notary profile stored in the keychain
+
+          paths_with_binaries = {
+            "root/#{project.name}-#{project.version}/opt/puppetlabs/bin/" => '*',
+            "root/#{project.name}-#{project.version}/opt/puppetlabs/puppet/bin/" => '*',
+            "root/#{project.name}-#{project.version}/opt/puppetlabs/puppet/lib/ruby/vendor_gems/bin" => '*',
+            "root/#{project.name}-#{project.version}/opt/puppetlabs/puppet/lib/" => '*.dylib',
+            "root/#{project.name}-#{project.version}/opt/puppetlabs/puppet/lib" => '*.bundle',
+            'plugins' => 'puppet-agent-installer-plugin',
+          }
+
+          sign_files_commands = [unlock]
+          sign_files_commands += paths_with_binaries.map do |path, name|
+            "find $(tempdir)/osx/build/#{path} -name '#{name}' -type f -exec codesign --timestamp --options runtime --keychain $$SIGNING_KEYCHAIN -vfs \"$$APPLICATION_SIGNING_CERT\" {} \\;"
+          end
+          sign_files_commands += paths_with_binaries.map do |path, name|
+            "find $(tempdir)/osx/build/#{path} -name '#{name}' -type f -exec codesign --verify --strict --verbose=2 {} \\;"
+          end
+
+          sign_package_commands = [
+            unlock,
+            "productsign --keychain $$SIGNING_KEYCHAIN --sign \"$$INSTALLER_SIGNING_CERT\" $(tempdir)/osx/build/#{project.name}-#{project.version}-#{project.release}-installer.pkg $(tempdir)/osx/build/pkg/#{project.name}-#{project.version}-#{project.release}-installer.pkg",
+            "rm $(tempdir)/osx/build/#{project.name}-#{project.version}-#{project.release}-installer.pkg",
+          ]
+
+          dmg = "$(tempdir)/osx/build/dmg/#{project.package_name}"
+          sign_dmg_commands = [
+            unlock,
+            'cd $(tempdir)/osx/build',
+            "codesign --timestamp --keychain $$SIGNING_KEYCHAIN --sign \"$$APPLICATION_SIGNING_CERT\" #{dmg}",
+            "codesign --verify --strict --verbose=2 #{dmg}",
+          ]
+
+          notarize_dmg_commands = ENV['NO_NOTARIZE'] ? [] : [
+            unlock,
+            "xcrun notarytool submit #{dmg} --keychain-profile \"$$NOTARY_PROFILE\" --wait",
+            "xcrun stapler staple #{dmg}",
+            "spctl --assess --type install --verbose #{dmg}"
+          ]
         end
 
          # Setup build directories
-        ["bash -c 'mkdir -p $(tempdir)/osx/build/{dmg,pkg,scripts,resources,root,payload,plugins}'",
-         "mkdir -p $(tempdir)/osx/build/root/#{project.name}-#{project.version}",
-         "mkdir -p $(tempdir)/osx/build/pkg",
-         # Grab distribution xml, scripts and other external resources
-         "cp #{project.name}-installer.xml $(tempdir)/osx/build/",
-         #copy the uninstaller to the pkg dir, where eventually the installer will go too
-         "cp #{project.name}-uninstaller.tool $(tempdir)/osx/build/pkg/",
-         "cp scripts/* $(tempdir)/osx/build/scripts/",
-         "if [ -d resources/osx/productbuild ] ; then cp -r resources/osx/productbuild/* $(tempdir)/osx/build/; fi",
-         # Unpack the project
-         "gunzip -c #{project.name}-#{project.version}.tar.gz | '#{@tar}' -C '$(tempdir)/osx/build/root/#{project.name}-#{project.version}' --strip-components 1 -xf -",
-
-         bom_install,
-
-         # Sign extra files
-         sign_commands,
-
-         # Package the project
-         "(cd $(tempdir)/osx/build/; #{@pkgbuild} --root root/#{project.name}-#{project.version} \
-          --scripts $(tempdir)/osx/build/scripts \
-          --identifier #{project.identifier}.#{project.name} \
-          --version #{project.version} \
-          --preserve-xattr \
-          --install-location / \
-          payload/#{project.name}-#{project.version}-#{project.release}.pkg)",
-         # Create a custom installer using the pkg above
-         "(cd $(tempdir)/osx/build/; #{@productbuild} --distribution #{project.name}-installer.xml \
-          --identifier #{project.identifier}.#{project.name}-installer \
-          --package-path payload/ \
-          --resources $(tempdir)/osx/build/resources  \
-          --plugins $(tempdir)/osx/build/plugins  \
-          pkg/#{project.name}-#{project.version}-#{project.release}-installer.pkg)",
-         # Create a dmg and ship it to the output directory
-         "(cd $(tempdir)/osx/build; \
-           #{@hdiutil} create \
-            -volname #{project.name}-#{project.version} \
-            -fs JHFS+ \
-            -format UDBZ \
-            -srcfolder pkg \
-            dmg/#{project.package_name})",
-         "mkdir -p output/#{target_dir}",
-         "cp $(tempdir)/osx/build/dmg/#{project.package_name} ./output/#{target_dir}"].flatten.compact
+        [
+          "bash -c 'mkdir -p $(tempdir)/osx/build/{dmg,pkg,scripts,resources,root,payload,plugins}'",
+          "mkdir -p $(tempdir)/osx/build/root/#{project.name}-#{project.version}",
+          "mkdir -p $(tempdir)/osx/build/pkg",
+          # Grab distribution xml, scripts and other external resources
+          "cp #{project.name}-installer.xml $(tempdir)/osx/build/",
+          #copy the uninstaller to the pkg dir, where eventually the installer will go too
+          "cp #{project.name}-uninstaller.tool $(tempdir)/osx/build/pkg/",
+          "cp scripts/* $(tempdir)/osx/build/scripts/",
+          "if [ -d resources/osx/productbuild ] ; then cp -r resources/osx/productbuild/* $(tempdir)/osx/build/; fi",
+          # Unpack the project
+          "gunzip -c #{project.name}-#{project.version}.tar.gz | '#{@tar}' -C '$(tempdir)/osx/build/root/#{project.name}-#{project.version}' --strip-components 1 -xf -",
+
+          bom_install,
+
+          # Sign extra files
+          extra_sign_commands,
+
+          # Sign all binaries
+          sign_files_commands,
+
+          # Package the project
+          "(cd $(tempdir)/osx/build/; #{@pkgbuild} --root root/#{project.name}-#{project.version} \
+            --scripts $(tempdir)/osx/build/scripts \
+            --identifier #{project.identifier}.#{project.name} \
+            --version #{project.version} \
+            --preserve-xattr \
+            --install-location / \
+            payload/#{project.name}-#{project.version}-#{project.release}.pkg)",
+
+          # Create a custom installer using the pkg above
+          "(cd $(tempdir)/osx/build/; #{@productbuild} --distribution #{project.name}-installer.xml \
+            --identifier #{project.identifier}.#{project.name}-installer \
+            --package-path payload/ \
+            --resources $(tempdir)/osx/build/resources  \
+            --plugins $(tempdir)/osx/build/plugins  \
+            #{project.name}-#{project.version}-#{project.release}-installer.pkg)",
+
+          sign_package_commands,
+
+          # Create a dmg and ship it to the output directory
+          "(cd $(tempdir)/osx/build; \
+            #{@hdiutil} create \
+              -volname #{project.name}-#{project.version} \
+              -fs JHFS+ \
+              -format UDBZ \
+              -srcfolder pkg \
+              dmg/#{project.package_name})",
+
+          sign_dmg_commands,
+          notarize_dmg_commands,
+          "mkdir -p output/#{target_dir}",
+          "cp $(tempdir)/osx/build/dmg/#{project.package_name} ./output/#{target_dir}"
+        ].flatten.compact
       end
 
       # Method to generate the files required to build a osx package for the project

lib/vanagon/project.rb

@@ -111,12 +111,11 @@ class Project
     attr_accessor :no_packaging
 
     # Extra files to sign
-    # Right now just supported on windows, useful for signing powershell scripts
-    # that need to be signed between build and MSI creation
     attr_accessor :extra_files_to_sign
     attr_accessor :signing_hostname
     attr_accessor :signing_username
-    attr_accessor :signing_command
+    attr_accessor :signing_commands
+    attr_accessor :use_local_signing
 
     # For creating reproducible builds
     attr_accessor :source_date_epoch
@@ -172,7 +171,8 @@ def initialize(name, platform) # rubocop:disable Metrics/AbcSize
       @extra_files_to_sign = []
       @signing_hostname = ''
       @signing_username = ''
-      @signing_command = ''
+      @signing_commands = []
+      @use_local_signing = false
       @source_date_epoch = (ENV['SOURCE_DATE_EPOCH'] || Time.now.utc).to_i
     end
 

lib/vanagon/project/dsl.rb

@@ -395,7 +395,15 @@ def signing_username(username)
       #
       # @param [String] the command to sign additional files
       def signing_command(command)
-        @project.signing_command = command
+        @project.signing_commands << command
+      end
+
+      # When true, run the signing commands locally rather than SSHing to a
+      # signing host.
+      #
+      # @param [Boolean] Whether to use local signing
+      def use_local_signing(var)
+        @project.use_local_signing = var
       end
     end
   end

lib/vanagon/utilities/extra_files_signer.rb

@@ -1,7 +1,52 @@
+require 'open3'
+
 class Vanagon
   module Utilities
     module ExtraFilesSigner
       class << self
+        RED = "\033[31m".freeze
+        GREEN = "\033[32m".freeze
+        RESET = "\033[0m".freeze
+
+        def run_command(cmd, silent: true, print_command: false, report_status: false)
+          puts "#{GREEN}Running #{cmd}#{RESET}" if print_command
+          output = ''
+          Open3.popen2e(cmd) do |_stdin, stdout_stderr, thread|
+            stdout_stderr.each do |line|
+              puts line unless silent
+              output += line
+            end
+            exitcode = thread.value.exitstatus
+            unless exitcode.zero?
+              err = "#{RED}Command failed! Command: #{cmd}, Exit code: #{exitcode}"
+              # Print details if we were running silent
+              err += "\nOutput:\n#{output}" if silent
+              err += RESET
+              abort err
+            end
+            puts "#{GREEN}Command finished with status #{exitcode}#{RESET}" if report_status
+          end
+          output.chomp
+        end
+
+        def local_commands(project, mktmp, source_dir)
+          commands = []
+          signing_script_path = File.join(run_command("#{mktmp} 2>/dev/null"), File.basename('sign_extra_file'))
+
+          project.extra_files_to_sign.each do |file|
+            commands += ["echo > #{signing_script_path}"]
+            commands += project.signing_commands.map { |c| "echo '#{c.gsub('%{file}', file)}' >> #{signing_script_path}" }
+            commands += ["/bin/bash #{signing_script_path}"]
+          end
+
+          commands
+        rescue RuntimeError
+          require 'vanagon/logger'
+          VanagonLogger.error "Error signing extra files: #{project.extra_files_to_sign.join(',')}"
+          raise if ENV['VANAGON_FORCE_SIGNING']
+          []
+        end
+
         def commands(project, mktemp, source_dir) # rubocop:disable Metrics/AbcSize
           tempdir = nil
           commands = []
@@ -25,8 +70,9 @@ def commands(project, mktemp, source_dir) # rubocop:disable Metrics/AbcSize
             remote_source_path = "#{remote_host}:#{remote_file_to_sign_path}"
             local_destination_path = local_source_path
 
+            commands << "#{Vanagon::Utilities.ssh_command} #{remote_host} \"echo > #{remote_signing_script_path}\""
+            commands += project.signing_commands.map { |c| "#{Vanagon::Utilities.ssh_command} #{remote_host} \"echo '#{c.gsub('%{file}', remote_file_to_sign_path)}' >> #{remote_signing_script_path}\"" }
             commands += [
-              "#{Vanagon::Utilities.ssh_command} #{remote_host} \"echo '#{project.signing_command} #{remote_file_to_sign_path}' > #{remote_signing_script_path}\"",
               "rsync -e '#{Vanagon::Utilities.ssh_command}' --verbose --recursive --hard-links --links --no-perms --no-owner --no-group #{extra_flags} #{local_source_path} #{remote_destination_directory}",
               "#{Vanagon::Utilities.ssh_command} #{remote_host} /bin/bash #{remote_signing_script_path}",
               "rsync -e '#{Vanagon::Utilities.ssh_command}' --verbose --recursive --hard-links --links --no-perms --no-owner --no-group #{extra_flags} #{remote_source_path} #{local_destination_path}"