Very large or long living aggregate sets

[JosephCottam]

As you zoom, dense aggregates become less practical (lots of zeros; at 8 tasks, 1GB has problems with 5000x5000 grids). Also, its sort of silly to recalculate the aggregate values EVERY time a dataset is used.

Working with very large aggregate sets will be more important as zoom is explored further.

[JosephCottam]

Idea: Sparse-tile and lazy allocation aggregates (addresses large tile sets, many of which are probably empty).
On creation, an array for tiles is created, but all items are null.
On write to a null tile, the tile is allocated and initialized.
On write to an existing tile, just updates the tile.
On read from a null tile, returns defaultValue.
On read from an existing tile, returns the found value.

[JosephCottam]

Idea: Avro tiles
Addresses persistence and possibly working with HUGE tile sets that are too large for memory.
Perhaps using weak references and lazy loading, the data structure described above could do transparent loading of such tiles....writing them is another story.