fix: Improve edge detection for polygons

[hoxbro]

Description

Fixes #1494

The following test has been changed because of this change, see my comments below for my reasoning to why this is okay. I don't think we can expect non horizontal and vertical lines to not have overlap when we are working in finite space (somewhat like the staircase paradox). The changes also make the output of the test 90 deg rotational symmetric.

datashader/tests/test_polygons.py::test_no_overlap[GeoDataFrame]
datashader/tests/test_polygons.py::test_no_overlap[dask_GeoDataFrame]
datashader/tests/test_polygons.py::test_no_overlap_agg[GeoDataFrame]
datashader/tests/test_polygons.py::test_no_overlap_agg[dask_GeoDataFrame]

How Has This Been Tested?

CI + Reasoning about failing test locally

AI Disclosure

• This PR contains AI-generated content.
  • I have tested all AI-generated content in my PR.
  • I take responsibility for all AI-generated content in my PR.

Tools and Models: Claude Code + Opus 4.6. Redirected it to the function, asked for it to find an off by one error in draw_polygon

AI Summary

Change: yincreasing[ei] → yincreasing[ei] > 0 at line 224.

Why: Original tiebreak count on-edge case for any non-zero yincreasing (both +1 and -1 truthy). Cause asymmetry — left edge (yinc=-1) and right edge (yinc=+1) of CCW polygon both "count" when point on edge. At shared-vertex between two touching polygons, two edges meet at same pixel column, contributions cancel → winding=0 → NaN column.

How to apply: Convention now "include right boundary, exclude left" (CCW: yinc=+1 = right edge). On-edge pixel claimed exactly once by polygon to its left. Touching polygons at shared vertex → left polygon claim the column.

Quadmesh.py use stricter bxa * yincreasing < 0 (no on-edge at all) — polygon.py need on-edge inclusion for pixel-grid-aligned vertex (e.g. unit-diamond lower tip), so need tiebreak, but now direction-consistent.

Checklist

• Tests added and are passing

[codspeed-hq]

Merging this PR will degrade performance by 23.46%

❌ 2 (👁 2) regressed benchmarks
✅ 50 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
👁	Simulation	test_quadmesh_raster[512]	14.9 ms	16.6 ms	-10.33%
👁	Simulation	test_layout[forceatlas2_layout]	45.2 ms	59.1 ms	-23.46%

---

_{Comparing fix_polygons (2ff83af) with main (93925f8)}

[hoxbro]

Looking at one of the failing tests example

import holoviews as hv
import spatialpandas as spd
import pandas as pd
from holoviews.operation.datashader import rasterize

hv.extension("bokeh")

df = spd.GeoDataFrame(
    {
        "polygons": pd.Series(
            [
                [[1, 1, 2, 2, 1, 3, 0, 2, 1, 1],[0.5, 1.5, 0.5, 2.5, 1.5, 2.5, 1.5, 1.5, 0.5, 1.5]],
                # [[0.5, 1.5, 1.5, 1.5, 1.5, 2.5, 0.5, 2.5, 0.5, 1.5]],
                # [[0, 1, 2, 1, 2, 3, 0, 3, 0, 1, 1, 1, 0, 2, 1, 3, 2, 2, 1, 1]],
            ],
            dtype="Polygon[float64]",
        ),
    }
)

poly = hv.Polygons(df).opts(alpha=0.4)
rpoly = rasterize(poly, dynamic=False, width=16, height=16).opts(alpha=0.4)
(poly * rpoly).opts(width=800, height=800)

This is how it looks on main:

And on this branch:

[hoxbro]

The other failing test shows similar inconsistency. My take is this is the branch has the correct behavior and overlapping is expected when working with finite precision and lines which has angle which is not horizontal or vertical.

import holoviews as hv
import spatialpandas as spd
import pandas as pd
import datashader as ds
from holoviews.operation.datashader import rasterize

hv.extension("bokeh")

df = spd.GeoDataFrame({
    'polygons': pd.Series([
       [[1, 1, 2, 2, 1, 3, 0, 2, 1, 1], [0.5, 1.5, 0.5, 2.5, 1.5, 2.5, 1.5, 1.5, 0.5, 1.5]],
       [[0.5, 1.5, 1.5, 1.5, 1.5, 2.5, 0.5, 2.5, 0.5, 1.5]],
       [[0, 1, 2, 1, 2, 3, 0, 3, 0, 1, 1, 1, 0, 2, 1, 3, 2, 2, 1, 1]]
    ], dtype='Polygon[float64]'),
    'v': range(3)
})

poly = hv.Polygons(df, vdims="v").opts(alpha=0.4)
rpoly = rasterize(poly, dynamic=False, width=16, height=16, aggregator=ds.sum("v")).opts(alpha=0.4)
(poly * rpoly).opts(width=800, height=800)

main

this branch

[hoxbro]

Same data as before.

[hoxbro]

Same data as before

[dcherian]

Certainly looks better to me!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.46%. Comparing base (93925f8) to head (2ff83af).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1495   +/-   ##
=======================================
  Coverage   88.46%   88.46%           
=======================================
  Files          96       96           
  Lines       19452    19461    +9     
=======================================
+ Hits        17208    17217    +9     
  Misses       2244     2244           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.