Error Converting BiRefNet Model to Core ML: Handling DeformConv2d

[pierre1618]

I'm trying to convert the BiRefNet (model) to Core ML but encountering some issues, particularly with the DeformConv2d layer. Here's the detailed setup and error I’m facing:

Setup

1. Model Loading:
   I am using the following code to load and prepare the BiRefNet model:

   from models.birefnet import BiRefNet
   import torch
   import torchvision.transforms as transforms
   import coremltools as ct
   import DeformConv2dConvert
   
   # Load the model
   birefnet = BiRefNet.from_pretrained('zhengpeng7/BiRefNet')
   device = 'cuda' if torch.cuda.is_available() else 'cpu'
   birefnet.to(device)
   birefnet.eval()
   print('BiRefNet is ready to use.')
   
   # Define input transformation
   transform_image = transforms.Compose([
       transforms.Resize((1024, 1024)),
       transforms.ToTensor(),
       transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
   ])

2. Conversion Code:
   I am using the following code to convert the model to Core ML:

   import coremltools as ct
   from torchvision.ops.deform_conv import DeformConv2d
   import DeformConv2dConvert
   
   # Register DeformConv2dConvert
   DeformConv2dConvert.register_op()
   
   # Move model to CPU and trace
   birefnet.cpu()
   example_input = torch.randn(1, 3, 1024, 1024)
   traced_model = torch.jit.trace(birefnet, example_input)
   
   # Convert to Core ML
   coreml_model = ct.convert(
       traced_model,
       inputs=[ct.TensorType(name="input", shape=example_input.shape)],
       convert_to="neuralnetwork"
   )

Error Message

During the conversion process, I encounter the following error:

Saving value type of int64 into a builtin type of int32, might lose precision!
Saving value type of int64 into a builtin type of int32, might lose precision!

ERROR - converting 'torchvision::deform_conv2d' op (located at: 'squeeze_module/0/dec_att/aspp1/atrous_conv'):

Converting PyTorch Frontend ==> MIL Ops:  76%|█████████████████▍     | 6929/9165 [00:01<00:00, 3651.18 ops/s]

AssertionError: the `offset` param should be stored in the weights

Additional Details

• Environment: Python 3.12, CoreMLTools v6.x, PyTorch v2.x.
• DeformConv2dConvert: I have installed and registered the DeformConv2dConvert library, which handles the conversion of DeformConv2d layers.

Questions

1. AssertionError: The conversion fails with an AssertionError related to the offset parameter. What could be causing this issue, and how can I resolve it?

Any help or suggestions on how to resolve these issues would be greatly appreciated. Thanks in advance!

[mrlzla]

I tried to do this too. It assumes that offset and mask are weight in the model, but they are supposed to be inputs as well.
This is the commit of fixing the issue. But in swift demo results are different. It seems like something is wrong with texture reading.

[dneprDroid]

Currently this custom layer only supports offset and mask tensors in the weights. The weights tensors are packed into 1-channel MTLTextures, but inputs are 4-channel MTLTextures.

I'll add support for offset and mask as 4-channel MTLTextures later.

[Zeleni9]

Same issue with same model. Thanks for implementing custom ops.

[wtesler]

I'm interested in the progress of this for the support of Birefnet conversation as well

[mht1421]

I tried to do this too. It assumes that offset and mask are weight in the model, but they are supposed to be inputs as well. This is the commit of fixing the issue. But in swift demo results are different. It seems like something is wrong with texture reading.

Hello, I tried to make a conversion of aliked model mentioned in the github https://github.com/ajuric/aliked-tensorrt, but I get the error ValueError: Invalid target shape in reshape op ([1, 1] to [1, 1, 1, 9, 60, 80]). Is there someone who can help?

[namanjaswani27]

Is the support for deform_conv2d added ?

I am getting RuntimeError: PyTorch convert function for op 'torchvision::deform_conv2d' not implemented.

[mrlzla]

Check out #4

[czkoko]

The following error appeared in the latest code:

converter/ops.py", line 41, in type_inference
    assert self.p1.shape[-1] == self.p2.shape[-2]
AssertionError

When I use DeformConv2dConvert.register_op(isMobile=False)
The end of the conversion appeared:

RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Error reading protobuf spec. validator error: LoadConstantNDLayer 'reshape_91' has empty constants."

[pierre1618]

I got the same error (@czkoko) when i tried the convert the model with the develop branch (that should include the correction patch of @mrlzla) .
p1 shape is (1, 1, 1, 16384) and p2 shape is (1, 1, 64, 1024).
Could it be that BiRefNet architecture got updated since the correction?

[RAE-CO]

Any news on this?
I have the ONNX version of BiRefNet_512x512 working well on windows, but I'm currently looking to convert to CoreML for hardware acceleration on mac. After running into this error, I first tried ONNX->CoreML (with all kinds of version combinations of onnxruntime and coremltools), but without luck. I also tried implementing the custom DeformConv2d operation, but also kept running into various problems.
I'm wondering if, given the popularity of model and the time since the last comments, it has been solved somehow?