lakehouse.spark.utils.settings

lakehouse.spark.utils.settings.get_databricks_connect_session() → SparkSession

Creates a Databricks Connect Spark Session.

Requires DATABRICKS_HOST, DATABRICKS_TOKEN and DATABRICKS_CLUSTER_ID as env variables.

Returns:

Databricks Connect Spark Session

lakehouse.spark.utils.settings.get_spark_opensource_session() → SparkSession

Creates a Spark Open Source session.

Returns:

Spark Open Source Session