Upload the compiled artifacts to the Databricks cluster and synchronize as part of session initialization.

This is required when the program requires external dependencies to evaluate the Spark query. For any Spark query using a typed Dataset API such as filter(), map(), foreach(), etc., or if the query is using a UDF, the enclosing classes should be synchronized with the cluster.

For example: the following code snippet includes the compiled artifact of class Foo. The uri in this example will either point to a JAR file or to a folder that contains the compiled class files.

val uri = Foo.getClass.getProtectionDomain.getCodeSource.getLocation.toURI
val spark = DatabricksSession.builder()
    .addCompiledArtifacts(uri)
    ...

The Databricks cluster id to connect.

Get a spark session if one was already created with the given configuration. Otherwise, create a new one.

Connection parameters for connecting to Databricks Connect is collected in the following order. When one of the configuration parameters is collected, the lookup for that parameter stops; collection for other parameters continues down the chain.

• Parameters configured directly in code via host(), token(), etc.
• Parameters configured in the Databricks SDK's DatabricksConfig specified via sdkConfig()
• The SPARK_REMOTE environment variable. For details on the connection string, see https://github.com/apache/spark/blob/master/connector/connect/docs/client-connection-string.md
• Use the "default" configuration profile from the Databricks config file. For details on Databricks configuration profiles, see https://docs.databricks.com/dev-tools/auth.html

By default, validates the used configuration by retrieving the used Databricks Runtime version with the Databricks SDK. Logs a warning if the authentication fails or if there is an unsupported combination of Databricks Runtime & Databricks Connect versions. Change this behaviour with DatabricksSession.builder().validateSession().

Add a header to the Spark Connect GRPC requests. This method is cumulative (can be called repeatedly to add more headers).

The URL of the Databricks workspace to connect.

API returns the same instance of the builder unchanged. This API is here for backwards compatibility.

Provide a custom SDK Config object to use for connection configuration. Can be used to override the default SDK Config that will be used.

The access token to use while for connecting. The user associated with this token should have access to the workspace and cluster. Subsequent queries will be executed on behalf of this user.

A user agent string identifying the application using the Databricks Connect module. Databricks Connect sends a set of standard information, such as, OS, Python version and the version of Databricks Connect included as a user agent to the service. The value provided here will be included along with the rest of the information. It is recommended to provide this value in the format "<product-name>/<product-version>" as described in https://datatracker.ietf.org/doc/html/rfc7231#section-5.5.3, but is not required.

Setting this option will run validations and throw an error if any fail. Validations are run only when connecting to a Databricks cluster.

• the specified connection parameters are valid and can communicate with the cluster
• the Databricks Runtime version of the cluster is greater than or equal to the Databricks Connect version. By default, these validations are run and a warning is logged. Unsetting this option will turn off these validations. Skipping validations for connections strings.