Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • object functions extends DeprecatedFunctionLogging

    Commonly used functions available for DataFrame operations.

    Commonly used functions available for DataFrame operations. Using functions defined here provides a little bit more compile-time safety to make sure the function exists.

    Spark also includes more built-in functions that are less common and are not defined here. You can still access them (and all the functions defined here) using the functions.expr() API and calling them through a SQL expression string. You can find the entire list of functions at SQL API documentation of your Spark version, see also <a href="https://spark.apache.org/docs/latest/api/sql/index.html">the latest list

    As an example, isnan is a function that is defined here. You can use isnan(col("myCol")) to invoke the isnan function. This way the programming language's compiler ensures isnan exists and is of the proper form. You can also use expr("isnan(myCol)") function to invoke the same function. In this case, Spark itself will ensure isnan exists when it analyzes the query.

    regr_count is an example of a function that is built-in but not defined here, because it is less commonly used. To invoke it, use expr("regr_count(yCol, xCol)").

    This function APIs usually have methods with Column signature only because it can support not only Column but also other types such as a native string. The other variants currently exist for historical reasons.

    Definition Classes
    sql
    Since

    3.4.0

  • partitioning

object partitioning

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. partitioning
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def bucket(numBuckets: Int, e: Column): Column

    (Scala-specific) A transform for any type that partitions by a hash of the input column.

    (Scala-specific) A transform for any type that partitions by a hash of the input column.

    Since

    4.0.0

  6. def bucket(numBuckets: Column, e: Column): Column

    (Scala-specific) A transform for any type that partitions by a hash of the input column.

    (Scala-specific) A transform for any type that partitions by a hash of the input column.

    Since

    4.0.0

  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @IntrinsicCandidate()
  8. def days(e: Column): Column

    (Scala-specific) A transform for timestamps and dates to partition data into days.

    (Scala-specific) A transform for timestamps and dates to partition data into days.

    Since

    4.0.0

  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  13. def hours(e: Column): Column

    (Scala-specific) A transform for timestamps to partition data into hours.

    (Scala-specific) A transform for timestamps to partition data into hours.

    Since

    4.0.0

  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. def months(e: Column): Column

    (Scala-specific) A transform for timestamps and dates to partition data into months.

    (Scala-specific) A transform for timestamps and dates to partition data into months.

    Since

    4.0.0

  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. def years(e: Column): Column

    (Scala-specific) A transform for timestamps and dates to partition data into years.

    (Scala-specific) A transform for timestamps and dates to partition data into years.

    Since

    4.0.0

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Partition transform functions

Support functions for DataFrames