Bug: Generating datapackage from schemas does not include changes from profile

[mrshll1001]

In the Profiles mechanism, the schema tools will take all the changes defined in a profile directory and create full profile schemas via a JSON merge patch, into the schema directory. It will also then generate a datapackage.json file based on these schemas.

This is not behaving as expected, since some changes made to the schema via the Profile are not reflected in the datapackage.json. This has a knock-on effect for the ERD diagrams for profiles.

This problem also persists when generating a datapackage manually e.g via hsds_schema.py schemas-to-datapackage schemas/ > test.json, so the root cause may be there.

1. New properties are not reflected in datapackage.json

E.g. adding test to funding.json, and running profile-all --clean with the repo URI will not add test as a field inside the funding entry in datapackage.json. This prevents it from being rendered in the ERD. test is present in the schema/funding.json file, so should be reflected in the datapackage.

2. Removed properties are not removed from the foreign keys in datapackage.json

Conversely, removed fields are actually removed from the datapackage. The UK profile removes organization_id from funding.json. After running profile-all --clean and opening datapackage.json, it's clear that organization_id is not present inside of resources['funding']/fields. However, it is still referenced inside of resources['funding']/foreignKeys. This causes errors when generating an ERD from the datapackage.

[Bjwebb]

1)

I'm not entirely sure if it should, but _schemas_to_datapackage‎ expects a constraints attribute in the schema file.

Something like this should work:

"test": {
  "name": "test",
  "type": "string",
  "title": "Test",
  "description": "A test.",
  "example": "test",
  "constraints": {
      "unique": false
  }
}

2)

A foreign key can be detected from either direction. In this case it's still being picked up from organization.json. Adding "funding": null to profile/organization.json should get you the behaviour you want.

[mrshll1001]

Thanks.

In light of this, is my current understanding correct?

• This behaviour is intended and not representative of a bug. There may be good reason for the requirement of the constraints key, and Profiles should take care to null out foreign keys from both directions
• Therefore the issue is with documentation, and this behaviour should be documented appropriately both on the Schema Tools and in the UK Profile Documentation

[Bjwebb]

Yes. Although I'm not sure about constraints