Add a single artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.

Supported target file extensions are .jar and .class.

Example

addArtifact("/Users/dummyUser/files/foo/bar.class", "foo/bar.class")
addArtifact("/Users/dummyUser/files/flat.class", "flat.class")
// Directory structure of the session's working directory for class files would look like:
// ${WORKING_DIR_FOR_CLASS_FILES}/flat.class
// ${WORKING_DIR_FOR_CLASS_FILES}/foo/bar.class

Add a single in-memory artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.

Supported target file extensions are .jar and .class.

Example

addArtifact(bytesBar, "foo/bar.class")
addArtifact(bytesFlat, "flat.class")
// Directory structure of the session's working directory for class files would look like:
// ${WORKING_DIR_FOR_CLASS_FILES}/flat.class
// ${WORKING_DIR_FOR_CLASS_FILES}/foo/bar.class

Add a single artifact to the client session.

Currently it supports local files with extensions .jar and .class and Apache Ivy URIs

Add a single artifact to the client session.

Currently only local files with extensions .jar and .class are supported.

Add one or more artifacts to the session.

Currently it supports local files with extensions .jar and .class and Apache Ivy URIs

Add a tag to be assigned to all the operations started by this thread in this session.

Often, a unit of execution in an application consists of multiple Spark executions. Application programmers can use this method to group all those jobs together and give a group tag. The application can use org.apache.spark.sql.SparkSession.interruptTag to cancel all running running executions with this tag. For example:

// In the main thread:
spark.addTag("myjobs")
spark.range(10).map(i => { Thread.sleep(10); i }).collect()

// In a separate thread:
spark.interruptTag("myjobs")

There may be multiple tags present at the same time, so different parts of application may use different tags to perform cancellation at different levels of granularity.

Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.

Clear the current thread's operation tags.

Close the SparkSession.

Release the current session and close the GRPC connection to the server. The API will not error if any of these operations fail. Closing a closed session is a no-op.

Close the allocator. Fail if there are still open SparkResults.

Runtime configuration interface for Spark.

This is the interface through which the user can get and set all Spark configurations that are relevant to Spark SQL. When getting the value of a config, his defaults to the value set in server, if any.

Applies a schema to a List of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

Creates a DataFrame from a local Seq of Product.

Creates a Dataset from a java.util.List of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Java Example

List<String> data = Arrays.asList("hello", "world");
Dataset<String> ds = spark.createDataset(data, Encoders.STRING());

Creates a Dataset from a local Seq of data of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Example

import spark.implicits._
case class Person(name: String, age: Long)
val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19))
val ds = spark.createDataset(data)

ds.show()
// +-------+---+
// |   name|age|
// +-------+---+
// |Michael| 29|
// |   Andy| 30|
// | Justin| 19|
// +-------+---+

Returns a DataFrame with no rows or columns.

Creates a new Dataset of type T containing zero elements.

Get the tags that are currently set to be assigned to all the operations started by this thread.

Interrupt all operations of this session currently running on the connected server.

Interrupt an operation of this session with the given operationId.

Interrupt all operations of this session with the given operation tag.

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.

Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.

Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.

sparkSession.read.parquet("/path/to/file.parquet")
sparkSession.read.schema(schema).json("/path/to/file.json")

Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.

sparkSession.readStream.parquet("/path/to/directory/of/parquet/files")
sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")

Register a ClassFinder for dynamically generated classes.

Remove a tag previously added to be assigned to all the operations started by this thread in this session. Noop if such a tag was not added earlier.

Executes a SQL query using Spark, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Executes a SQL query substituting positional parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Synonym for close().

Returns the specified table/view as a DataFrame. If it's a table, it must support batch reading and the returned DataFrame is the batch scan query plan of this table. If it's a view, the returned DataFrame is simply the query plan of the view, which can either be a batch or streaming query plan.

Executes some code block and prints to stdout the time taken to execute the block. This is available in Scala only and is used primarily for interactive testing and debugging.

Returns a TableValuedFunction that can be used to call a table-valued function (TVF).

A collection of methods for registering user-defined functions (UDF).

The following example registers a Scala closure as UDF:

sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

The following example registers a UDF in Java:

sparkSession.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1,
    DataTypes.StringType);

(Scala-specific) Implicit methods available in Scala for converting common names and Symbols into Columns, and for converting common Scala objects into DataFrames.

val sparkSession = SparkSession.builder.getOrCreate()
import sparkSession.implicits._