com.aliyun.odps.storage.write

Class TableArrowWriter

java.lang.Object

com.aliyun.odps.storage.write.TableArrowWriter

All Implemented Interfaces:

ArrowWriter, Closeable, AutoCloseable

Direct Known Subclasses:

TableArrowBatchBlobWriter

public class TableArrowWriter
extends Object
implements ArrowWriter

A buffered ArrowWriter implementation.

It serializes and caches the VectorSchemaRoot batches written by the user in memory. When the flush() method is called, it packages all cached batch data into a single Arrow IPC Stream format request and sends it over the network at once.

This design ensures the safety of reusable VectorSchemaRoot objects for users, but consumes more memory to cache the data.

Note: This class is not thread-safe for concurrent write operations.

Method Summary

Modifier and Type	Method and Description
List<Blob>	batchUploadBlob(long columnId, List<byte[]> dataList) Uploads multiple blobs in a single batch request.
long	bytesWritten()
void	close()
org.apache.arrow.vector.VectorSchemaRoot	createVectorSchemaRoot() Creates a new VectorSchemaRoot instance that is compatible with this writer.
void	flush()
org.apache.arrow.memory.BufferAllocator	getAllocator()
RecordWriter	getAsRecordWriter(long rowCountPerBatch)
long	getCachedSize()
String	getLastRequestId() Returns the request ID of the last successful write (flush).
WriteSchema	getWriteSchema()
Blob	uploadBlob(long columnId, InputStream data)
void	writeBatch(org.apache.arrow.vector.VectorSchemaRoot root)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

writeBatch

public void writeBatch(org.apache.arrow.vector.VectorSchemaRoot root)

Specified by:

writeBatch in interface ArrowWriter

createVectorSchemaRoot

public org.apache.arrow.vector.VectorSchemaRoot createVectorSchemaRoot()

Creates a new VectorSchemaRoot instance that is compatible with this writer.

The caller is responsible for managing the lifecycle of the returned VectorSchemaRoot, specifically by calling VectorSchemaRoot.close() in a try-with-resources block to prevent memory leaks.

Returns:

A new, empty VectorSchemaRoot ready to be populated.

bytesWritten

public long bytesWritten()

Specified by:

bytesWritten in interface ArrowWriter

flush

public void flush()

uploadBlob

public Blob uploadBlob(long columnId,
                       InputStream data)

batchUploadBlob

public List<Blob> batchUploadBlob(long columnId,
                                  List<byte[]> dataList)

Uploads multiple blobs in a single batch request.

This is significantly more efficient than calling uploadBlob(long, InputStream) repeatedly, as all blobs are uploaded in a single HTTP request instead of N individual requests.

Note: This method is not supported for tables with primary keys (Delta Tables).

Parameters:

columnId - the column ID of the BLOB column

dataList - a list of byte arrays, each containing the raw data for one blob

Returns:

a list of Blob references in the same order as the input list

Throws:

ClientException - if called on a Delta Table or if the server response is inconsistent

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

getWriteSchema

public WriteSchema getWriteSchema()

getLastRequestId

public String getLastRequestId()

Returns the request ID of the last successful write (flush). For client-side logging.

getAllocator

public org.apache.arrow.memory.BufferAllocator getAllocator()

getAsRecordWriter

public RecordWriter getAsRecordWriter(long rowCountPerBatch)

Specified by:

getAsRecordWriter in interface ArrowWriter

getCachedSize

public long getCachedSize()