Eliminate N+1 queries in team timesheet compact view API

[Copilot]

Fixes #770

The get_compact_view_data() endpoint was making 5 database queries per employee in a loop, resulting in ~50 queries for a page of 10 employees.

Changes

Replaced per-employee queries with batch operations:

• Employee working hours: Single get_all() for all employees instead of cached individual calls
• Leave applications: Single SQL query with proper date overlap logic for all employees
• Holidays: Batch fetch unique holiday lists, then batch fetch holidays for those lists
• Timesheet states: New batch_get_timesheet_states() helper replacing individual lookups

Before:

for employee in employees:
    working_hours = get_employee_working_hours(employee.name)  # Query
    leaves = get_employee_leaves(employee.name, ...)  # Query
    holidays = get_holidays(employee.name, ...)  # Query
    status = get_timesheet_state(employee.name, ...)  # Query

After:

# Batch fetch all data upfront
emp_work_map = {...}  # Single query
leave_map = {...}  # Single query
holidays_by_list = {...}  # Two queries
all_states = batch_get_timesheet_states(...)  # Single query

for employee in employees:
    # Dictionary lookups only

Additional improvements:

• Early return for empty employee list
• Dictionary-based holiday lookup (O(1) vs O(n))
• Extracted WORK_DAYS_PER_WEEK constant

Impact

• Queries: 50 → 7 (86% reduction)
• Expected response time: 500-2000ms → 50-100ms

Original prompt

---

This section details on the original issue you should resolve

<issue_title>N+1 in get_compact_view_data API</issue_title>
<issue_description>

Severity: High

Category: Database/API

Location

• File: next_pms/timesheet/api/team.py
• Lines: 106-164
• Function: get_compact_view_data()

Description

The get_compact_view_data() API endpoint, which powers the team timesheet compact view, loops through employees and makes multiple database queries per employee. This is called frequently as users navigate the timesheet interface.

Problematic Code

for employee in employees:
    working_hours = get_employee_working_hours(employee.name)  # DB query (cached)
    daily_working_hours = get_employee_daily_working_norm(employee.name)  # DB query (cached)
    local_data = {**employee, **working_hours}
    employee_timesheets = timesheet_map.get(employee.name, [])

    status = get_timesheet_state(  # Potentially expensive
        employee=employee.name,
        start_date=dates[0].get("start_date"),
        end_date=dates[-1].get("end_date"),
    )
    local_data["status"] = status
    local_data["data"] = []

    leaves = get_employee_leaves(  # DB query per employee
        start_date=add_days(dates[0].get("start_date"), -max_week * 7),
        end_date=add_days(dates[-1].get("end_date"), max_week * 7),
        employee=employee.name,
    )
    holidays = get_holidays(employee.name, dates[0].get("start_date"), dates[-1].get("end_date"))  # DB query per employee

    for date_info in dates:
        for date in date_info.get("dates"):
            # O(n) scans for leaves and holidays per date
            for leave in leaves:
                if leave["from_date"] <= date <= leave["to_date"]:
                    # ...
            for holiday in holidays:
                if date == holiday.holiday_date:
                    # ...

Impact

• For a page showing 10 employees over 2 weeks:
  • 10 * get_employee_working_hours() = 10 queries (may be cached)
  • 10 * get_employee_daily_working_norm() = 10 queries (may be cached)
  • 10 * get_timesheet_state() = 10 queries
  • 10 * get_employee_leaves() = 10 queries
  • 10 * get_holidays() = 10 queries
• Total: 50 queries per page load
• API response time: 500ms-2s when it should be <100ms

Recommended Fix

@whitelist()
@error_logger
def get_compact_view_data(
    date: str,
    max_week: int = 2,
    employee_name=None,
    employee_ids: list[str] | str | None = None,
    department=None,
    project=None,
    user_group=None,
    page_length=10,
    start=0,
    status_filter=None,
    status=None,
    reports_to: str | None = None,
    by_pass_access_check=False,
):
    if not by_pass_access_check:
        only_for(["Timesheet Manager", "Timesheet User", "Projects Manager"], message=True)

    dates = []
    data = {}
    if status_filter and isinstance(status_filter, str):
        status_filter = json.loads(status_filter)

    for i in range(max_week):
        week = get_week_dates(date=date)
        dates.append(week)
        date = add_days(getdate(week["start_date"]), -1)

    dates.reverse()
    res = {"dates": dates}
    
    start_date = dates[0].get("start_date")
    end_date = dates[-1].get("end_date")
    extended_start = add_days(start_date, -max_week * 7)
    extended_end = add_days(end_date, max_week * 7)

    employees, total_count = filter_employee_by_timesheet_status(
        employee_name=employee_name,
        department=department,
        project=project,
        user_group=user_group,
        page_length=page_length,
        start=start,
        reports_to=reports_to,
        status=status,
        timesheet_status=status_filter,
        employee_ids=employee_ids,
        start_date=start_date,
        end_date=end_date,
    )
    
    if not employees:
        return {"dates": dates, "data": {}, "total_count": 0, "has_more": False}
    
    employee_names = [employee.name for employee in employees]

    # Batch fetch employee working hours
    emp_work_data = frappe.get_all(
        "Employee",
        filters={"name": ["in", employee_names]},
        fields=["name", "custom_working_hours", "custom_work_schedule"]
    )
    emp_work_map = {e.name: e for e in emp_work_data}
    
    # Default working hours from settings
    default_hours = frappe.db.get_single_value("HR Settings", "standard_working_hours") or 8

    # Batch fetch all timesheets
    timesheet_data = get_all(
        "Timesheet",
        filters={
            "employee": ["in", employee_names],
            "start_date": [">=", start_date],
            "end_date": ["<=", end_date],
            "docstatus": ["!=", 2],
        },
        fields=["employee", "start_date", "end_date", "total_hours", "note", "custom_approval_status"],
    )
    
    timesheet_map = {}
    for ts in timesheet_data:
        if ts.employee not in timesheet_map:
            timesheet_map[ts.employee] = []
        timesheet_map[ts.employe...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes rtCamp/next-pms#770

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

[wreckage0907]

In our code we get one value per employee ,, here we get the first value which is not None , hence there is a mismatch in "status" field of the API Endpoint

def get_timesheet_state(employee: str, start_date: str, end_date: str):
    status = frappe.db.get_value(
        "Timesheet",
        {
            "employee": employee,
            "start_date": [">=", getdate(start_date)],
            "end_date": ["<=", getdate(end_date)],
            "docstatus": ["!=", 2],
        },
        "custom_weekly_approval_status",
    )
    if status:
        return status
    return "Not Submitted"