spark_read()

Read(1770) Label: read, file,

Description:

Read a file stored in Spark database.

Syntax:

spark_read(con,sfile,k:v,...)

Note:

The external library function (See External Library Guide) reads content of a file stored in Spark database and returns result as a table sequence.

Parameter:

con

Local or HDFS connection string

sfile

File name

k:v

Parameters that can be omitted. When sfile is a text file separated by “”, set the pair as ”sep”:”;”; and when the text file is separated by “#”, set the pair as ”sep”:”#”

Option:

@c

Read content of a file and return result as a cursor

@t

Read the first row of a text file as field names, which is equivalent to "header":"true"; by default, use the automatically generated field names c0, _c1…

@x

Close Spark database connection

Return value:

A table sequence or a cursor

Example:

 

A

 

1

=spark_open()

 

2

=spark_read@t(A1,"D:/people.txt","sep":",")

Read a semicolon-separated txt file and set reading the first row as field names.

3

=spark_read(A1,"D:/people.txt","header":"true","sep":",")

Same as A2.

4

=spark_read(A1,"D:/people.json")

Read content of people.json.

5

=spark_read@c(A1,"D:/people.csv")

Read content of people.csv and return result as a cursor.

6

=A5.fetch()

 

7

=spark_read@x(A1,"D:/people.csv")

Close Spark database connection.