configurable value of zingg properties in spark session provider in junits #1275
Description
current junits rely on SparkSessionProvider which reads properties from zingg.properties. TestSparkBase calls this class and all our Spark junits are annotated with it.
The zingg.property file has 8gb set for driver and executor. 8gb is only needed for some complex tests like e2e integration tests.
So, we should have three classes - SparkSessionProviderBase which takes a properties file and creates the sessions. Then SparkSessionProviderLite where the default value in the file is set to 512mb or 1gb. And SparkSessionProviderHeavy where the value is 8gb.
Also need two classes TestSparkBaseLite and TestSparkBaseHeavy which call the respsective spark session providers.
All annotations of TestSparkBase should be moved to TestSparkBaseLite. Only the integration test should invoke TestSparkBaseHeavy