Use containers for local development

[djberg96]

Description

You have database requirements in your Gemfile. I don't want to install mysql2 locally.

No, you can't make me, you're not my dad.

Let's use containers!

We'll prefer podman, and fall back to docker because that's the way, uh-huh, uh-huh, I like it.

The gemspec has been updated to ignore these files, since I assume you don't want them bundled in an actual release.

Before

Sadness. Storm clouds hovered nearby, and I could barely get out of bed.

A local bundle install fails because I do not have mysql2 installed. It's only a coincidence that I have sqlite3 and postgresql installed, otherwise those would've failed too.

After

Joy. The skies are clear and I hear birds singing outside. I am ready to take on the world.

I can run bundle install test:compose and everything builds in the container, and all tests pass.

Type of Change

• Feature

Checklist

• My code follows the style guidelines (e.g., RuboCop)
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally (bundle exec rake test)

No new code really, other than some Rakefile and Gemfile updates. This is just infrastructure.

How Has This Been Tested?

The proof is in the pudding. Does bundle exec rake test:compose work locally?

Yes, on both Fedora Linux (podman) and MacOS (Docker).

Le proof!

dberger:~/Dev/ancestry [podman_docker]>bundle exec rake test:compose                                                         [16:08]
docker build -t ancestry-dev:latest .
[+] Building 72.9s (16/16) FINISHED                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                            0.0s
 => => transferring dockerfile: 798B                                                                                            0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                                       3.6s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                0.0s
 => docker-image://docker.io/docker/dockerfile:1@sha256:4a43a54dd1fedceb30ba47e76cfcf2b47304f4161c0caeac2db1c61804ea3c91        0.5s
 => => resolve docker.io/docker/dockerfile:1@sha256:4a43a54dd1fedceb30ba47e76cfcf2b47304f4161c0caeac2db1c61804ea3c91            0.0s
 => => sha256:3b88f5232a559cd96d2d4d212d697742a4c3bf960fd09c8168bb95cf9cd5694b 12.71MB / 12.71MB                                0.4s
 => => extracting sha256:3b88f5232a559cd96d2d4d212d697742a4c3bf960fd09c8168bb95cf9cd5694b                                       0.1s
 => [internal] load metadata for docker.io/library/ruby:3.3-slim                                                                0.9s
 => [auth] library/ruby:pull token for registry-1.docker.io                                                                     0.0s
 => [internal] load .dockerignore                                                                                               0.0s
 => => transferring context: 144B                                                                                               0.0s
 => [1/7] FROM docker.io/library/ruby:3.3-slim@sha256:b3e7a216b8f33f22ad4859f26f945b70fcb25f87f50160bd6bedfc84434ee0c7          2.2s
 => => resolve docker.io/library/ruby:3.3-slim@sha256:b3e7a216b8f33f22ad4859f26f945b70fcb25f87f50160bd6bedfc84434ee0c7          0.0s
 => => sha256:4c16c920acfe472959dac40751f3c35068986701827a1de3d2ed742282902b03 145B / 145B                                      0.2s
 => => sha256:a4e41d356bbad2ce60f7e8412ac6420041c0368205bd9566972c931f33305e8d 190B / 190B                                      0.3s
 => => sha256:e3741f426cf9cd5f1f9bc834f629206e927d04f36a4a2541e87e96b298b13f32 38.33MB / 38.33MB                                1.9s
 => => sha256:7babf668b2898d0ccdde79f303e82e6b4cfc2d3c1f13a9564ce96476a80caeac 1.26MB / 1.26MB                                  0.4s
 => => sha256:f4badedbec24858ef2dc51256f6418328e305e9c3c5a5e093581f83425618bd5 30.14MB / 30.14MB                                0.9s
 => => extracting sha256:f4badedbec24858ef2dc51256f6418328e305e9c3c5a5e093581f83425618bd5                                       0.3s
 => => extracting sha256:7babf668b2898d0ccdde79f303e82e6b4cfc2d3c1f13a9564ce96476a80caeac                                       0.0s
 => => extracting sha256:a4e41d356bbad2ce60f7e8412ac6420041c0368205bd9566972c931f33305e8d                                       0.0s
 => => extracting sha256:e3741f426cf9cd5f1f9bc834f629206e927d04f36a4a2541e87e96b298b13f32                                       0.2s
 => => extracting sha256:4c16c920acfe472959dac40751f3c35068986701827a1de3d2ed742282902b03                                       0.0s
 => [internal] load build context                                                                                               0.0s
 => => transferring context: 496.69kB                                                                                           0.0s
 => [2/7] WORKDIR /app                                                                                                          0.1s
 => [3/7] RUN apt-get update -y   && apt-get install -y --no-install-recommends     build-essential     git     libpq-dev      10.1s
 => [4/7] COPY lib/ancestry/version.rb ./lib/ancestry/version.rb                                                                0.0s
 => [5/7] COPY Gemfile* *.gemspec Appraisals ./                                                                                 0.0s
 => [6/7] RUN bundle config set --local path "/usr/local/bundle"   && bundle install --jobs 4 --retry 3                        46.7s
 => [7/7] COPY . .                                                                                                              0.0s
 => exporting to image                                                                                                          8.7s
 => => exporting layers                                                                                                         7.2s
 => => exporting manifest sha256:9430ed8e7ddc8cca7c7362ff227c07b15586989959f1d21030d8b6bcbd49eb67                               0.0s
 => => exporting config sha256:c9289d45fa487a27c7edf4afe67602aa9a04dbcea0a0ea8cff3b845ca173d5fb                                 0.0s
 => => exporting attestation manifest sha256:3a18d43123efb20ba1b6455d46961b5ecd4d7a1835cd9ae37795ef74337a8fc4                   0.0s
 => => exporting manifest list sha256:99d8ffc1ebca5e0d6e4d470e119753b7f5ad4e331ed3395610d9b4f13d41fd6d                          0.0s
 => => naming to docker.io/library/ancestry-dev:latest                                                                          0.0s
 => => unpacking to docker.io/library/ancestry-dev:latest                                                                       1.5s
docker run --rm -it -v /Users/dberger/Dev/ancestry:/app -w /app ancestry-dev:latest bundle exec rake test
/usr/local/bin/ruby -w -I"lib:test" /usr/local/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/rake_test_loader.rb "test/concerns/arrangement_test.rb" "test/concerns/array_test.rb" "test/concerns/associations_test.rb" "test/concerns/build_ancestry_test.rb" "test/concerns/counter_cache_test.rb" "test/concerns/db_test.rb" "test/concerns/default_scopes_test.rb" "test/concerns/depth_caching_test.rb" "test/concerns/depth_constraints_test.rb" "test/concerns/depth_virtual_test.rb" "test/concerns/has_ancestry_test.rb" "test/concerns/hooks_test.rb" "test/concerns/integrity_checking_and_restoration_test.rb" "test/concerns/leaves_test.rb" "test/concerns/ltree_test.rb" "test/concerns/materialized_path2_test.rb" "test/concerns/materialized_path3_test.rb" "test/concerns/materialized_path_test.rb" "test/concerns/orphan_strategies_test.rb" "test/concerns/parent_caching_test.rb" "test/concerns/parent_virtual_test.rb" "test/concerns/relations_test.rb" "test/concerns/root_caching_test.rb" "test/concerns/root_virtual_test.rb" "test/concerns/scopes_test.rb" "test/concerns/setter_test.rb" "test/concerns/sort_by_ancestry_test.rb" "test/concerns/stale_ancestry_test.rb" "test/concerns/sti_support_test.rb" "test/concerns/touching_test.rb" "test/concerns/tree_navigation_test.rb" "test/concerns/tree_predicate_test.rb" "test/concerns/uuid_test.rb"
testing sqlite3 (with string ancestry)
column format: materialized_path options: {:collation=>"binary", :null=>true}

Loaded Ancestry test suite environment:
  Ruby: 3.3.11
  ActiveRecord: 7.2.3
  Database: SQLite

Run options: --seed 14357

# Running:

.............................................................................SSS.S......................................................................................SSS.SS............................................................................

Finished in 1.635645s, 152.8449 runs/s, 1942.3529 assertions/s.

250 runs, 3177 assertions, 0 failures, 0 errors, 9 skips

You have skipped tests. Run with --verbose for details.

[kbrock]

Hey Dan!

Wondering if that PR ballooned a little bit

• So you want to modify ancestry, but don't have a database installed?
• Is your plan to run tests on your local Linux/Mac, or only run tests in Podman?
• If you are not running tests locally, then is it a possibility to not run bundle install locally, since the only goal of bundle install is so you can run the tests, and you can only run tests if you have a database.
• Or maybe your only beef with mysql2 and you are being gracious and allowing someone to have a beef with anything?

I had on my TODO list to test the trilogy driver. I'll get that in tonight.

I will make mysql2 optional. It is a pain to install. The mysql driver was also a pain.

Since CI runs tests, I'm hesitant to add a second testing environment.
Especially if the only reason is that you don't want to install mysql.

If I misunderstood, please let me know. This does look like cool stuff.

[kbrock]

ok. I fixed a merge conflict. (you may have to pull down)
I think Docker file needs to change to address new driver.

But I'm hoping we can maybe just gate the mysql2 driver?

[djberg96]

@kbrock Hola!

You wouldn't use this in CI, this would only be for local development. It may feel a bit redundant if you already have a github workflow, but it makes local development easier.

While mysql2 was the main motivation for this PR, really it would be beneficial to anyone who doesn't want any of the database engines installed locally, but still wants to test local changes against all of them.

[djberg96]

@kbrock ok, rebased master and updated the Gemfile. I'm not familiar with trilogy, but based on what you did I guessed as to the proper handling.

[kbrock]

If you are running bundle install, then you are running tests locally?
Or are you thinking you'll only test in a container?

Based on the rakefile, it looks like you plan to run tests only in Podman.

If you are testing in a container, do you need to bundle install?

If you are developing locally, it seems a pain to have to remember to:

BUNDLE_INSTALL_POSTGRES=1 BUNDLE_INSTALL_SQLITE3=1 BUNDLE_INSTALL_MYSQL=1 bundle install

DB=pg BUNDLE_INSTALL_POSTGRES=1 bundle exec rake

Also, what database are you running?

Do we really need to guard anything other than MySQL?

I'm leaning heavily towards only guarding the mysql2 driver. Don't really see the need to guard against SQLite3, PG, or even Trilogy.

[kbrock]

We only have a wild character in the lib directory.
Does What does this filter out?

[kbrock]

Going to reiterate, The original s.files has wildcharacters and does not touch docker_artifacts.

Remove this unless you see a need that I'm missing

[kbrock]

When using appraisals, we generate gemspecs for each Rails version.
We can't carry the if condition across. So we'll need to ensure the gemspec gets all the necessary gems.

Please add appraisal condition to SQLite and PG. blocker.

[kbrock]

ok I ran locally.
the ergonomics are tricky.

I will try see if I can get something running off podman with a local Container/Docker file.

I think adding a few .gitignore rules and a custom Rakefile is all you really need.

[djberg96]

I would say any library that has any sort of external dependency, especially if it requires configuration (like a database), should at least have the option to run in a containerized environment. You, personally, may have everything setup locally so that you don't need to do it, and that's fine, but it makes contributing easier for other people. So yes, I would say it should be done for postgres and sqlite as well.

I've done this for a couple of my libraries now, like rkerberos, since I have absolutely no desire to install a kerberos server on my personal computer. ;)

[djberg96]

I'll see what I can do about cleaning things up a bit with the Rakefile and gemspec.

[djberg96]

Ok, I guess I should actually add a proper compose file so we can run tests with PG or MySQL instead of sqlite3.

[kbrock]

Please understand my primary thought:

If you are not running tests locally, then you don't need to bundle install locally, because you don't need to develop the gem locally.

This is orthogonal to running tests in a container.

Because we could add Podman support without implementing any of the bundler environmental variables.

Thought I might be missing something about why you need to bundle install.

Second though
Can we leverage the act gem instead of rebuilding the GitHub pipeline locally?

[kbrock]

@djberg96 miq slack?

[djberg96]

Ok, hopefully this PR makes more sense now and is more useful, as it was what I had originally intended but apparently never actually implemented.

Anyway, so now you can do the following, which will spin up the appropriate container with the database engine of your choice and run the tests:

rake compose:test_pg
rake compose:test_mysql
rake compose:test_sqlite

And you'll see some output like this:

Loaded Ancestry test suite environment:
  Ruby: 3.3.10
  ActiveRecord: 7.2.3
  Database: PostgreSQL

Run options: --seed 62408

# Running:

..........................................................................................................................................................................................................................................................

Finished in 6.293306s, 39.7248 runs/s, 510.7014 assertions/s.

I guessed a bit on the gemspec, let me know if it needs updating.

[djberg96]

Alright, updated so that it will default to sqlite3, so you can just run bundle exec rake locally and it will still work.

As discussed, this is a lighter approach than the act-cli approach, since that will take longer and runs through more variations that are currently setup in the github workflow.

I personally prefer this, but if you decide you'd rather go with the act-cli approach I will understand and you can close this.

[kbrock]

we are in 2 modes:

1. DB=pg rake test
2. rake compose:build_sqlite

Why not DB=pg rake compose:build?

Dependencies probably work better for the build_sqlite long version but this adds quite a few cookie-cutter methods.

• How do you typically run local tests?
• How do you typically run container tests?
• I don't care if you call this docker or container for calling podman/docker.

Would prefer to remove many of these aliases. Just pick one way and if it is wrong, we can pick another.

DB=pg rake test
DB=pg rake container:test # docker:test test:docker

---

I agree act is overboard, but it is nice to maintain only 1 set of container commands.

I keep going back to thinking your changes could be an add on gem or a local file and not part of this project.

Sure, it may need some modifications, and I'm good with that. But the duplication of the github action has me concerned.

[djberg96]

Ok, I've updated it so that rake container:test honors the DB environment variable, if present.