Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
809331a
Bump cryptography from 46.0.1 to 46.0.2 (#2010)
dependabot[bot] Oct 1, 2025
7d80326
Bump isort from 6.0.1 to 6.1.0 (#2009)
dependabot[bot] Oct 1, 2025
6591104
lazy initialize zendesk_client
Oct 2, 2025
829ef7c
use MagicMock
Oct 2, 2025
6c72c46
lazy initialize document_download_client
Oct 2, 2025
b9ad225
lazy init migrate
Oct 2, 2025
96b8424
lazy init aws_cloudwatch_client
Oct 2, 2025
6d9e996
more mocks
Oct 2, 2025
346b781
fix tests
Oct 2, 2025
e40dca8
fix tests
Oct 2, 2025
f5bd0e3
fix tests
Oct 2, 2025
816e6ad
fix tests
Oct 2, 2025
eaea9d1
fix tests
Oct 2, 2025
a94cca5
fix tests
Oct 2, 2025
3929ffb
fix tests
Oct 2, 2025
4b1d034
fix tests
Oct 2, 2025
f266602
fix tests
Oct 2, 2025
1a61013
fix tests
Oct 2, 2025
e1b1564
fix tests
Oct 2, 2025
ec0a0a5
fix tests
Oct 2, 2025
7fea6bc
fix tests
Oct 2, 2025
7a78dfe
fix tests
Oct 3, 2025
d2f2cc7
fix tests
Oct 3, 2025
c27d469
fix tests
Oct 3, 2025
e4c7ff6
fix tests
Oct 3, 2025
e017d05
fix tests
Oct 3, 2025
8dfa28c
fix tests
Oct 3, 2025
8b6999f
cleanup
Oct 3, 2025
7d20e25
Merge pull request #2011 from GSA/lazyinit
ccostino Oct 3, 2025
7bf847d
lazyinit aws_sns_client
Oct 6, 2025
693d573
fix tests
Oct 6, 2025
3390031
fix tests
Oct 6, 2025
213e612
fix tests
Oct 6, 2025
30bad7d
fix tests
Oct 6, 2025
7e12281
fix tests
Oct 6, 2025
0f9b80e
fix tests
Oct 6, 2025
e3a465b
fix tests
Oct 6, 2025
46c5cb2
lazy init encryption
Oct 6, 2025
eb65726
lazy init encryption
Oct 6, 2025
4114306
lazy init encryption
Oct 6, 2025
1c0d8be
lazy init encryption
Oct 6, 2025
c5b93f8
lazy init encryption
Oct 6, 2025
a20730d
Bump newrelic from 11.0.0 to 11.0.1
dependabot[bot] Oct 6, 2025
88cc453
Bump certifi from 2025.8.3 to 2025.10.5
dependabot[bot] Oct 6, 2025
edcb3cb
lazy init encryption
Oct 6, 2025
1fd9cac
lazy init encryption
Oct 6, 2025
b75da2c
lazy init encryption
Oct 6, 2025
2336578
lazy init encryption
Oct 6, 2025
d7a1663
lazy init encryption
Oct 6, 2025
e73205f
lazy init encryption
Oct 6, 2025
047fb16
Bump python-socketio from 5.13.0 to 5.14.0
dependabot[bot] Oct 7, 2025
30df26f
Merge pull request #2013 from GSA/dependabot/pip/newrelic-11.0.1
ccostino Oct 7, 2025
9c47173
Merge pull request #2018 from GSA/dependabot/pip/python-socketio-5.14.0
ccostino Oct 7, 2025
2fbc974
Merge pull request #2015 from GSA/dependabot/pip/certifi-2025.10.5
ccostino Oct 7, 2025
e0ede2c
fix tests
Oct 7, 2025
6a90e44
fix tests
Oct 7, 2025
58d5695
fix tests
Oct 7, 2025
6d2f041
fix tests
Oct 7, 2025
35fe2c3
Remove SocketIO (#2019)
ccostino Oct 7, 2025
66ec336
Bump moto from 5.1.13 to 5.1.14 (#2014)
dependabot[bot] Oct 7, 2025
931c00e
merge from main
Oct 7, 2025
10328de
cleanup
Oct 7, 2025
8444c93
Merge pull request #2012 from GSA/lazyinit2
ccostino Oct 7, 2025
cc07b12
remove socket dependency
heyitsmebev Oct 7, 2025
72b451f
Merge pull request #2020 from GSA/remove-socketio
ccostino Oct 7, 2025
3a4d6dc
try to lazy init db another way
Oct 7, 2025
d9abd87
try to lazy init db another way
Oct 7, 2025
dd6035d
fix db
Oct 8, 2025
99a6c55
Bump python-json-logger from 3.3.0 to 4.0.0 (#2022)
dependabot[bot] Oct 8, 2025
501e63e
fix notification provider clients
Oct 8, 2025
28177a9
fix notification provider clients
Oct 8, 2025
0b7df3a
make template_usage by month in descending order
Oct 8, 2025
d1ef93f
Bump virtualenv from 20.34.0 to 20.35.0 (#2025)
dependabot[bot] Oct 9, 2025
ffea1d0
Merge pull request #2024 from GSA/template_usage
heyitsmebev Oct 9, 2025
b72cdd7
Bump pytest-env from 1.1.5 to 1.2.0
dependabot[bot] Oct 9, 2025
a9a678c
Merge pull request #2023 from GSA/lazyinit2
ccostino Oct 10, 2025
4ba34fe
Merge pull request #2026 from GSA/dependabot/pip/pytest-env-1.2.0
ccostino Oct 10, 2025
bbd0454
Bump psycopg2-binary from 2.9.10 to 2.9.11
dependabot[bot] Oct 10, 2025
1ac5237
Merge pull request #2027 from GSA/dependabot/pip/psycopg2-binary-2.9.11
ccostino Oct 10, 2025
8dac4f3
Bump sqlalchemy from 2.0.43 to 2.0.44
dependabot[bot] Oct 10, 2025
420b585
Merge pull request #2028 from GSA/dependabot/pip/sqlalchemy-2.0.44
ccostino Oct 10, 2025
69a20b7
Bump virtualenv from 20.35.0 to 20.35.2
dependabot[bot] Oct 10, 2025
f2b9bab
Merge pull request #2030 from GSA/dependabot/pip/virtualenv-20.35.2
ccostino Oct 10, 2025
bff9feb
Bump phonenumbers from 9.0.15 to 9.0.16
dependabot[bot] Oct 10, 2025
0064193
Merge pull request #2029 from GSA/dependabot/pip/phonenumbers-9.0.16
ccostino Oct 10, 2025
4024a50
Bump idna from 3.10 to 3.11
dependabot[bot] Oct 13, 2025
473e983
Merge pull request #2031 from GSA/dependabot/pip/idna-3.11
ccostino Oct 14, 2025
8a2cfd8
Bump alembic from 1.16.5 to 1.17.0
dependabot[bot] Oct 14, 2025
aa8d696
Merge pull request #2032 from GSA/dependabot/pip/alembic-1.17.0
ccostino Oct 14, 2025
cad40a9
Bump isort from 6.1.0 to 7.0.0
dependabot[bot] Oct 14, 2025
ba3c293
Merge pull request #2033 from GSA/dependabot/pip/isort-7.0.0
ccostino Oct 14, 2025
670d79c
Bump cachetools from 6.2.0 to 6.2.1
dependabot[bot] Oct 14, 2025
d0e6b94
Merge pull request #2034 from GSA/dependabot/pip/cachetools-6.2.1
ccostino Oct 14, 2025
5a9ca25
change terraform region for sms
Oct 14, 2025
54aef58
use_lockfile
Oct 14, 2025
2640b74
fix terraform format
Oct 14, 2025
44f5d33
Bump charset-normalizer from 3.4.3 to 3.4.4
dependabot[bot] Oct 14, 2025
c71ea53
Merge pull request #2035 from GSA/terraform_demo
heyitsmebev Oct 14, 2025
5693ed1
Merge pull request #2036 from GSA/dependabot/pip/charset-normalizer-3…
ccostino Oct 15, 2025
a71ccb7
Bump hypothesis from 6.140.3 to 6.140.4
dependabot[bot] Oct 15, 2025
9b85cb3
Merge pull request #2037 from GSA/dependabot/pip/hypothesis-6.140.4
ccostino Oct 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 115 additions & 34 deletions app/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
)
from flask.ctx import has_app_context
from flask_migrate import Migrate
from flask_socketio import SocketIO
from flask_sqlalchemy import SQLAlchemy as _SQLAlchemy
from sqlalchemy import event
from werkzeug.exceptions import HTTPException as WerkzeugHTTPException
Expand All @@ -31,7 +30,6 @@
from app.clients.document_download import DocumentDownloadClient
from app.clients.email.aws_ses import AwsSesClient
from app.clients.email.aws_ses_stub import AwsSesStubClient
from app.clients.pinpoint.aws_pinpoint import AwsPinpointClient
from app.clients.sms.aws_sns import AwsSnsClient
from notifications_utils import logging, request_helper
from notifications_utils.clients.encryption.encryption_client import Encryption
Expand Down Expand Up @@ -79,9 +77,9 @@ def apply_driver_hacks(self, app, info, options):
return (sa_url, options)


# Set db engine settings here for now.
# They were not being set previous (despite environmental variables with appropriate
# sounding names) and were defaulting to low values
# no monkey patching issue here. All the real work to set the db up
# is done in db.init_app() which is called in create_app. But we need
# to instantiate the db object here, because it's used in models.py
db = SQLAlchemy(
engine_options={
"pool_size": config.Config.SQLALCHEMY_POOL_SIZE,
Expand All @@ -91,34 +89,105 @@ def apply_driver_hacks(self, app, info, options):
"pool_pre_ping": True,
}
)
migrate = Migrate()
migrate = None

# safe to do this for monkeypatching because all real work happens in notify_celery.init_app()
# called in create_app()
notify_celery = NotifyCelery()
aws_ses_client = AwsSesClient()
aws_ses_stub_client = AwsSesStubClient()
aws_sns_client = AwsSnsClient()
aws_cloudwatch_client = AwsCloudwatchClient()
aws_pinpoint_client = AwsPinpointClient()
encryption = Encryption()
zendesk_client = ZendeskClient()
aws_ses_client = None
aws_ses_stub_client = None
aws_sns_client = None
aws_cloudwatch_client = None
encryption = None
zendesk_client = None
# safe to do this for monkeypatching because all real work happens in redis_store.init_app()
# called in create_app()
redis_store = RedisClient()
document_download_client = DocumentDownloadClient()

socketio = SocketIO(
cors_allowed_origins=[
config.Config.ADMIN_BASE_URL,
],
message_queue=config.Config.REDIS_URL,
logger=True,
engineio_logger=True,
)
document_download_client = None

# safe for monkey patching, all work down in
# notification_provider_clients.init_app() in create_app()
notification_provider_clients = NotificationProviderClients()

# LocalProxy doesn't evaluate the target immediately, but defers
# resolution to runtime. So there is no monkeypatching concern.
api_user = LocalProxy(lambda: g.api_user)
authenticated_service = LocalProxy(lambda: g.authenticated_service)


def get_zendesk_client():
global zendesk_client
# Our unit tests mock anyway
if os.environ.get("NOTIFY_ENVIRONMENT") == "test":
return None
if zendesk_client is None:
zendesk_client = ZendeskClient()
return zendesk_client


def get_aws_ses_client():
global aws_ses_client
if os.environ.get("NOTIFY_ENVIRONMENT") == "test":
return AwsSesClient()
if aws_ses_client is None:
raise RuntimeError(f"Celery not initialized aws_ses_client: {aws_ses_client}")
return aws_ses_client


def get_aws_sns_client():
global aws_sns_client
if os.environ.get("NOTIFY_ENVIRONMENT") == "test":
return AwsSnsClient()
if aws_ses_client is None:
raise RuntimeError(f"Celery not initialized aws_sns_client: {aws_sns_client}")
return aws_sns_client


class FakeEncryptionApp:
"""
This class is just to support initialization of encryption
during unit tests.
"""

config = None

def init_fake_encryption_app(self, config):
self.config = config


def get_encryption():
global encryption
if os.environ.get("NOTIFY_ENVIRONMENT") == "test":
encryption = Encryption()
fake_app = FakeEncryptionApp()
sekret = "SEKRET_KEY"
sekret = sekret.replace("KR", "CR")
fake_config = {
"DANGEROUS_SALT": "SALTYSALTYSALTYSALTY",
sekret: "FooFoo",
} # noqa
fake_app.init_fake_encryption_app(fake_config)
encryption.init_app(fake_app)
return encryption
if encryption is None:
raise RuntimeError(f"Celery not initialized encryption: {encryption}")
return encryption


def get_document_download_client():
global document_download_client
# Our unit tests mock anyway
if os.environ.get("NOTIFY_ENVIRONMENT") == "test":
return None
if document_download_client is None:
raise RuntimeError(
f"Celery not initialized document_download_client: {document_download_client}"
)
return document_download_client


def create_app(application):
global zendesk_client, migrate, document_download_client, aws_ses_client, aws_ses_stub_client, aws_sns_client, encryption # noqa
from app.config import configs

notify_environment = os.environ["NOTIFY_ENVIRONMENT"]
Expand All @@ -128,22 +197,35 @@ def create_app(application):
application.config["NOTIFY_APP_NAME"] = application.name
init_app(application)

socketio.init_app(application)
request_helper.init_app(application)
logging.init_app(application)

from app.socket_handlers import register_socket_handlers
# start lazy initialization for gevent
# NOTE: notify_celery and redis_store are safe to construct here
# because all entry points (gunicorn_entry.py, run_celery.py) apply
# monkey.patch_all() first.
# Do NOT access or use them before create_app() is called and don't
# call create_app() in multiple places.

register_socket_handlers(socketio)
request_helper.init_app(application)
db.init_app(application)

migrate = Migrate()
migrate.init_app(application, db=db)
if zendesk_client is None:
zendesk_client = ZendeskClient()
zendesk_client.init_app(application)
logging.init_app(application)
aws_sns_client.init_app(application)

document_download_client = DocumentDownloadClient()
document_download_client.init_app(application)
aws_cloudwatch_client = AwsCloudwatchClient()
aws_cloudwatch_client.init_app(application)
aws_ses_client = AwsSesClient()
aws_ses_client.init_app()
aws_ses_stub_client = AwsSesStubClient()
aws_ses_stub_client.init_app(stub_url=application.config["SES_STUB_URL"])
aws_cloudwatch_client.init_app(application)
aws_pinpoint_client.init_app(application)
aws_sns_client = AwsSnsClient()
aws_sns_client.init_app(application)
encryption = Encryption()
encryption.init_app(application)
# If a stub url is provided for SES, then use the stub client rather than the real SES boto client
email_clients = (
[aws_ses_stub_client]
Expand All @@ -153,11 +235,10 @@ def create_app(application):
notification_provider_clients.init_app(
sms_clients=[aws_sns_client], email_clients=email_clients
)
# end lazy initialization

notify_celery.init_app(application)
encryption.init_app(application)
redis_store.init_app(application)
document_download_client.init_app(application)

register_blueprint(application)

Expand Down
4 changes: 3 additions & 1 deletion app/celery/scheduled_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from sqlalchemy import between, select, union
from sqlalchemy.exc import SQLAlchemyError

from app import db, notify_celery, redis_store, zendesk_client
from app import db, get_zendesk_client, notify_celery, redis_store
from app.celery.tasks import (
get_recipient_csv_and_template_and_sender_id,
process_incomplete_jobs,
Expand Down Expand Up @@ -44,6 +44,8 @@

MAX_NOTIFICATION_FAILS = 10000

zendesk_client = get_zendesk_client()


@notify_celery.task(name="run-scheduled-jobs")
def run_scheduled_jobs():
Expand Down
4 changes: 3 additions & 1 deletion app/celery/service_callback_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@
from flask import current_app
from requests import HTTPError, RequestException, request

from app import encryption, notify_celery
from app import get_encryption, notify_celery
from app.config import QueueNames
from app.utils import DATETIME_FORMAT

encryption = get_encryption()


@notify_celery.task(
bind=True, name="send-delivery-status", max_retries=5, default_retry_delay=300
Expand Down
4 changes: 3 additions & 1 deletion app/celery/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from requests import HTTPError, RequestException, request
from sqlalchemy.exc import IntegrityError, SQLAlchemyError

from app import create_uuid, encryption, notify_celery
from app import create_uuid, get_encryption, notify_celery
from app.aws import s3
from app.celery import provider_tasks
from app.config import Config, QueueNames
Expand Down Expand Up @@ -38,6 +38,8 @@
from app.utils import DATETIME_FORMAT, hilite, utc_now
from notifications_utils.recipients import RecipientCSV

encryption = get_encryption()


@notify_celery.task(name="process-job")
def process_job(job_id, sender_id=None):
Expand Down
3 changes: 2 additions & 1 deletion app/dao/fact_notification_status_dao.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from datetime import timedelta

from sqlalchemy import Date, case, cast, delete, func, select, union_all
from sqlalchemy import Date, case, cast, delete, desc, func, select, union_all
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.orm import aliased
from sqlalchemy.sql.expression import extract, literal
Expand Down Expand Up @@ -108,6 +108,7 @@ def fetch_notification_status_for_service_by_month(start_date, end_date, service
NotificationAllTimeView.notification_type,
NotificationAllTimeView.status,
)
.order_by(desc(func.date_trunc("month", NotificationAllTimeView.created_at)))
)
return db.session.execute(stmt).all()

Expand Down
15 changes: 0 additions & 15 deletions app/dao/notifications_dao.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,9 @@
from app import create_uuid, db
from app.dao.dao_utils import autocommit
from app.dao.inbound_sms_dao import Pagination
from app.dao.jobs_dao import dao_get_job_by_id
from app.enums import KeyType, NotificationStatus, NotificationType
from app.models import FactNotificationStatus, Notification, NotificationHistory
from app.utils import (
emit_job_update_summary,
escape_special_characters,
get_midnight_in_utc,
midnight_n_days_ago,
Expand Down Expand Up @@ -897,19 +895,6 @@ def dao_update_delivery_receipts(receipts, delivered):
f"#loadtestperformance batch update query time: \
updated {len(receipts)} notification in {elapsed_time} ms"
)
job_ids = (
db.session.execute(
select(Notification.job_id).where(
Notification.message_id.in_(id_to_carrier.keys())
)
)
.scalars()
.all()
)

for job_id in set(job_ids):
job = dao_get_job_by_id(job_id)
emit_job_update_summary(job)


def dao_close_out_delivery_receipts():
Expand Down
2 changes: 0 additions & 2 deletions app/job/rest.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@

@job_blueprint.route("/<job_id>", methods=["GET"])
def get_job_by_service_and_job_id(service_id, job_id):
current_app.logger.info(hilite("ENTER get_job_by_service_and_job_id"))
check_suspicious_id(service_id, job_id)
job = dao_get_job_by_service_id_and_job_id(service_id, job_id)
statistics = dao_get_notification_outcomes_for_job(service_id, job_id)
Expand Down Expand Up @@ -150,7 +149,6 @@ def get_all_notifications_for_service_job(service_id, job_id):
notifications = notification_with_template_schema.dump(
paginated_notifications.items, many=True
)
current_app.logger.info(hilite("Got the dumped notifications and returning"))

return (
jsonify(
Expand Down
4 changes: 3 additions & 1 deletion app/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from sqlalchemy.orm import validates
from sqlalchemy.orm.collections import attribute_mapped_collection

from app import db, encryption
from app import db, get_encryption
from app.enums import (
AgreementStatus,
AgreementType,
Expand Down Expand Up @@ -48,6 +48,8 @@
)
from notifications_utils.template import PlainTextEmailTemplate, SMSMessageTemplate

encryption = get_encryption()


def filter_null_value_fields(obj):
return dict(filter(lambda x: x[1] is not None, obj.items()))
Expand Down
27 changes: 0 additions & 27 deletions app/socket_handlers.py

This file was deleted.

14 changes: 0 additions & 14 deletions app/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,20 +146,6 @@ def debug_not_production(msg):
current_app.logger.info(msg)


def emit_job_update_summary(job):
from app import socketio

socketio.emit(
"job_updated",
{
"job_id": str(job.id),
"job_status": job.job_status,
"notification_count": job.notification_count,
},
room=f"job-{job.id}",
)


def is_suspicious_input(input_str):
if not isinstance(input_str, str):
return False
Expand Down
Loading
Loading