Have a discussion on architecture

[ngunia]

Blocked by #4 and probably requires a larger discussion.

Generally, I propose that we have a frontend app, a backend app, and a database, all running as separate services/VMs, depending on the selected cloud. No rush on this, but probably something that should be decided before the next puzzle.

Outcomes:

• A rough architecture diagram

• Stretch: Codified infrastructure (terraform?)

[robot-o]

so the current development setup works like this

• frontend is hosted and deployed separately on github and heroku
• ingress, backend and database are orchestrated with docker-compose on a single node
• node is provisioned on terraform cloud by the code in ops/terraform
• gets deployed via github ci runner in .github/workflows/deploy_develop.yml
• gh ci runs ops/ansible/playbooks/compose/configure.yml and ops/ansible/playbooks/compose/deploy.yml playbooks

however we also have a fully distributed configuration ready:

• code for docker-swarm/stack ready in ops/ansible/playbooks/stack/deploy.yml
• would be trivial to extend the playbooks to configure / deploy fully distributed and add to ci
• would require more terraform code for spinning up production worker nodes

so in conjunction with the switch to git flow we have very solid footing for continuous delivery, one more thing that i want is automated database backup but that can be done with ansible very easily.

[robot-o]

oh and we probably also need to figure out how to share secrets within the organization because the decentral approach right now only works in a marathon setting where you always have people passing the proverbial discord-plaintext-password-sharing-torch

[robot-o]

cc @Sannemen

[ngunia]

Agreed on the secrets management, but I think that's something to discuss once we have selected a cloud provider. If AWS, their secrets manager probably is the least effort for the apps to pull secrets, and we can store references to the secrets as config.

If something else, we could spin up a tiny vault cluster/single node instance. This is something I'm extremely familiar with and could have ready to go in TF/Ansible.

I think overall I'm a little more concerned with the overall flow/fitting things together (10,000 foot view), and the individual services/app technologies can/should be another discussion. For example, I really would like to see the database run in our selected cloud's higher level solution or at least it's own VM, separate from the backend.

[robot-o]

agreed on getting a saas db for production BUT this is just for solving the puzzle, so i don't know if it's worth the money. feel free to hmu on discord so we can discuss this in more detail, i'll relay the results back here for transparency

[Sannemen]

Opinions based on my personal experience, not claiming to own the truth here:

For the infrastructure:

• Build components in docker, pointing to each other. Gets us a stable runtime for each environment (no need to worry about how to get each component running in a different flavour of the OS), easy to transpose the dev enviroment to production, and easy to swap components around
• Deploy using docker compose files on docker swarm cluster. Low overhead, low barrier of entry, comes with built-in secret manager, easily scalable from one node to multiple, easy to do rolling deployments and rollbacks (where appropriate). Each component can be easily replaced as wished, and devs get to run the same stack as prod.
• Orchestrate with ansible, maybe some of with with terraform.
• Don't worry about CI/CD in the beginning, getting a minimum product working first is priority, then we automate it. Otherwise we risk automating the action of making a mistake.

For the backend:

• I'm way more used to python than node, so that's my vote. Golang as a second pick. No special reason, just what i'm familiar with.
• flask over django. lower barrier of entry while still being fully-featured (ORM with migrations, task dispatcher, login with oauth, etc). Way more lightweight to write an API to be consumed by, say, a node frontend.

For the frontend:

• The only sane thing I can say about the frontend is that I should have no say on it xD

[kevrom]

Flask or Node (Express) would allow me to be productive immediately, Go would require some ramp-up time for me (though I'm currently under pressure for my job to learn Go because that's what we use for our APIs). My preference will always be Node, simply because I am absurdly competent with it and its versatility is off the charts. All that being said, I'm a frontend dev by trade, and will deal with whatever we choose. :)

For frontend I think React is a no-brainer. I'd likely avoid using Next.js, unless we intend to use Node as the backend as well (and even then I think it's worth discussing).