Create Collaborative Labeler for Massachusetts Orthoimagery

[DWGodwin]

This repository will allow collaborative labeling of orthoimagery using a jupyter lab environment.

[DWGodwin]

Over the last week, I wrote a script to download orthoimagery based on input bounding boxes and store it on the disk in an accessible format.

Today, we talked about the best way to store and access data during labeling and the best format to publish and provide it in afterwards.

Labeling

• I will test different ways of loading imagery into the labeling interface to make sure that it is as fast as possible. I will start out with storing and accessing the data in a COG, then try loading specific chips given the chosen labeling bounding box if this is not fast enough. In general, we want data to be cached as much as possible so that the next chosen chip has imagery already loaded.
• Clear instructions should be provided to labelers about labeling outside the bounding box, in order to avoid edge effects.

Publishing

• Images should be provided as a COG, with labels as parquet.

I will have a demo of labeling by our meeting next week.

[DWGodwin]

Over the last week, I revised the chip generation script to exclude chips for which buildings do not exist using the MS Open Buildings dataset, generated chips and downloaded their equivalent imagery, and started building the labeling interface.

Today, I demoed the labeling interface, which, once some issues with calling zoom_to_layer were fixed, loads quickly for arbitrary image chips and labels, at about 2 seconds each time new chip is called.

For improvements, we talked about:

• Ensuring cached imagery is used by assigning blocks of sequential chips which share imagery to labelers.
• Possibly running a tile server that all users connect to. This can be accomplished using QGIS or with localtileterver (https://github.com/banesullivan/localtileserver) running in a background thread (From the documentation: "The TileClient class can be used to to launch a tile server in a background thread which will serve raster imagery to a viewer (usually ipyleaflet or folium in Jupyter notebooks").
• Combining all rasters for a year into a single COG by creating a template raster and copying values from each individual raster to the 'canvas' of the template at their appropriate location rather than using merge_rasters(). This should overcome the slow processing that merge_rasters() was causing.

My next steps are to combine rasters into COGs, refine the labeling flow, and create visualizations of the desired labeling. Possible follow ups are to use one localtileserver instance instead of having each user create one using add_cog_layer and to make the labeling interface accessible through a dedicated portal through something like Solara rather than having it in a jupyter notebook.

[DWGodwin]

Progress as of today:

• The COG creation script functions. It uses gdal.translate and a VRT to write all rasters into one COG. This ends up being 60GB for the Worcester dataset.
• I have added a checking function to check if a tileserver exists for the data already and have leafmap connect to this server if it does. This is done by explicitly specifying the server port in the leafmap add raster call. This means that one tileserver can serve to all kernels using leafmap without having separate scripts to run the server.
• I have confirmed that the cache for this server is consistent across all kernels - when a user in one notebook zooms into an area, it takes several seconds to load, but when another notebook zooms into the same area, it loads instantly.

My next steps are to create visualizations of desired labeling and modify the chip selection loop to get sequential chips, making the labeling loop faster as adjacent tiles will already be cached.

Another thought - we have enough RAM to load everything into memory at the highest zoom level. I believe this is possible using localtileserver if I call get_tile on all tiles in the client at a single zoom level. If we do labeling on a day where there is not another RAM-intensive task on blade, this could make imagery loading much faster. On machines without so much RAM, this could still be used to load large chunks of a dataset, have everyone label that chunk, and then move on.

Another thought - a more elegant and less RAM-intensive way to do this would be to have a list of 6 or so future tiles for each user. These tiles are taken out of the pool of available tiles, so nobody else labels them. The tile client is called appropriately for all of these tiles. Then, when the user finishes a tile, another is added to their future stack. That way, the tile client is always loading tiles ahead of time, so when the user goes to the next tile, the imagery is already there. The challenge here is calling the tile client based on a bounding box, but I expect that a method to do this is already in leafmap, as the Map object must call the tileserver based on the bounding box and zoom level of the leaflet map.

[DWGodwin]

The main hurdle with using Solara to deploy the app as created in the notebook is that I don't believe jupyter-server-proxy works with the Solara server, and so running add_raster tries to serve the tiles using localtileserver but fails silently as jupyter-server-proxy is incompatable. I think that serving the tiles separately from Solara within the Docker container could be a solution, and then connecting to it as a tile server using add_tile_layer. Failing that, I will try to deploy using Voila

[DWGodwin]

The first solution worked - starting a TileClient instance and connecting to it with Solara is functional and does not require a jupyter server or any proxying. This approach is generic to any local raster. I will now work on adapting the user interface for the labeler to Solara.

[DWGodwin]

I have added the next chip button to Solara and learned a lot about how React works and how to use widgets. Now I will finish the interface and work on improving the speed of the app by labeling adjacent chips as discussed earlier, as this is the simplest way to speed up the interface.

[DWGodwin]

My original idea for pre-loading chips before the user navigates to them was flawed - there is no way to tell the leaflet-based widget to request a different set of chips from the server, as leaflet does not support extensions or any injection of Javascript code. However, I found a workaround - I can create a Solara component that passes links to the browser for download when the current chip changes. Using this, I plan to keep a running list of future chips for each user's instance which will cause their browser to cache the images for future use, making the process much faster - especially in skipping past empty chips.

[DWGodwin]

In order to work around the limitations of rendering chips on the fly, I added a script to pre-generate tiles which can then be served from local static files by Solara natively.

I also added a button to the Solara server to save user input geometries. Next steps are to organize these files and add buttons to correctly flag chips according to their status. Then, I will make a user guide and do a small test to improve user experience and flow.

[DWGodwin]

I am focusing on this task as we approach the deadline for FARR-RCN.

I have updated the Solara platform to only load one year of imagery at a time, which improves its speed to the point where I feel confident with a dozen people using it at the same time.

One issue I have come across is that the imagery has nodata values for some years which do not exist for others along the fringes of the images. I will determine why this is and will potentially need to re-generate the data.

The priority as we approach the deadline is to make the platform presentable and user-ready. The most important thing is to add a user guide to the platform. We also discussed in the last research meeting the creation of a checking function to put unlabeled chips back into the pool when someone is detected to have abandoned the labeling platform - after 10 minutes or so.

[DWGodwin]

After rebuilding the COGs with overviews this time (before, they had none, making visualization at smaller scales take much longer than it should), I can see that there is something wrong with the data pipeline - 2019 and 2021 have missing data on the periphery that should not be missing. I will troubleshoot this while also building out the labeling platform, since it will involve longer processes of downloading and combining the data.