object functions
- Alphabetic
- By Inheritance
- functions
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @IntrinsicCandidate()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
from_avro(data: Column, subject: String, schemaRegistryAddress: String, options: Map[String, String]): Column
Converts a binary column of Schema-Registry avro format into its corresponding catalyst value.
Converts a binary column of Schema-Registry avro format into its corresponding catalyst value. The schema of the given subject in Schema-Registry should not change in an incompatible way, otherwise exception will be thrown at runtime when Spark consumes data with new schema.
- data
the binary column.
- subject
the subject in Schema-Registry that these data belong to.
- schemaRegistryAddress
the address(host and port) of Schema-Registry.
- options
options to control how the Avro record is parsed and configs for schema registry client.
- Annotations
- @Experimental()
-
def
from_avro(data: Column, subject: String, schemaRegistryAddress: String): Column
Converts a binary column of Schema-Registry avro format into its corresponding catalyst value.
Converts a binary column of Schema-Registry avro format into its corresponding catalyst value. The schema of the given subject in Schema-Registry should not change in an incompatible way, otherwise exception will be thrown at runtime when Spark consumes data with new schema.
- data
the binary column.
- subject
the subject in Schema-Registry that these data belong to.
- schemaRegistryAddress
the address(host and port) of Schema-Registry.
- Annotations
- @Experimental()
-
def
from_avro(data: Column, jsonFormatSchema: String, options: Map[String, String]): Column
Converts a binary column of Avro format into its corresponding catalyst value.
Converts a binary column of Avro format into its corresponding catalyst value. The specified schema must match actual schema of the read data, otherwise the behavior is undefined: it may fail or return arbitrary result. To deserialize the data with a compatible and evolved schema, the expected Avro schema can be set via the option avroSchema.
- data
the binary column.
- jsonFormatSchema
the avro schema in JSON string format.
- options
options to control how the Avro record is parsed.
- Annotations
- @Experimental()
- Since
3.5.0
-
def
from_avro(data: Column, jsonFormatSchema: String): Column
Converts a binary column of avro format into its corresponding catalyst value.
Converts a binary column of avro format into its corresponding catalyst value. The specified schema must match the read data, otherwise the behavior is undefined: it may fail or return arbitrary result.
- data
the binary column.
- jsonFormatSchema
the avro schema in JSON string format.
- Annotations
- @Experimental()
- Since
3.5.0
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @IntrinsicCandidate()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @IntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @IntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @IntrinsicCandidate()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
def
to_avro(data: Column, subject: Column, schemaRegistryAddress: String, options: Map[String, String], jsonFormatSchema: String): Column
Converts a column into binary of Schema-Registry avro format.
Converts a column into binary of Schema-Registry avro format. The input data schema must have been registered to the given subject in Schema-Registry, or the query will fail at runtime.
- data
the data column.
- subject
the subject in Schema-Registry that these data belong to.
- schemaRegistryAddress
the address(host and port) of Schema-Registry.
- options
options to control how the Avro record is parsed and configs for schema registry client.
- jsonFormatSchema
user-specified output avro schema in JSON string format.
-
def
to_avro(data: Column, subject: Column, schemaRegistryAddress: String, jsonFormatSchema: String): Column
Converts a column into binary of Schema-Registry avro format.
Converts a column into binary of Schema-Registry avro format. The input data schema must have been registered to the given subject in Schema-Registry, or the query will fail at runtime.
- data
the data column.
- subject
the subject in Schema-Registry that these data belong to.
- schemaRegistryAddress
the address(host and port) of Schema-Registry.
- jsonFormatSchema
user-specified output avro schema in JSON string format.
-
def
to_avro(data: Column, subject: Column, schemaRegistryAddress: String, options: Map[String, String]): Column
Converts a column into binary of Schema-Registry avro format.
Converts a column into binary of Schema-Registry avro format. The input data schema must have been registered to the given subject in Schema-Registry, or the query will fail at runtime.
- data
the data column.
- subject
the subject in Schema-Registry that these data belong to.
- schemaRegistryAddress
the address(host and port) of Schema-Registry.
- options
options to control how the Avro record is parsed and configs for schema registry client.
- Annotations
- @Experimental()
-
def
to_avro(data: Column, subject: Column, schemaRegistryAddress: String): Column
Converts a column into binary of Schema-Registry avro format.
Converts a column into binary of Schema-Registry avro format. The input data schema must have been registered to the given subject in Schema-Registry, or the query will fail at runtime.
- data
the data column.
- subject
the subject in Schema-Registry that these data belong to.
- schemaRegistryAddress
the address(host and port) of Schema-Registry.
- Annotations
- @Experimental()
-
def
to_avro(data: Column, jsonFormatSchema: String): Column
Converts a column into binary of avro format.
Converts a column into binary of avro format.
- data
the data column.
- jsonFormatSchema
user-specified output avro schema in JSON string format.
- Annotations
- @Experimental()
- Since
3.5.0
-
def
to_avro(data: Column): Column
Converts a column into binary of avro format.
Converts a column into binary of avro format.
- data
the data column.
- Annotations
- @Experimental()
- Since
3.5.0
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
Deprecated Value Members
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] ) @Deprecated
- Deprecated