APL Wiki|
Size: 7910
Comment: Added APLX examples
|
Size: 8218
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
| Line 9: | Line 8: |
| * Fields are separated by commas | * Fields are separated by commas. Well, mostly. |
| Line 13: | Line 12: |
| * If a field contains either a comma or one of the end line chars, either the char(s) or the whole contents needs to be escaped. Excel escapes these values by embedding the field inside a set of double quotes. For example, a single cell with the text apples, carrots, and oranges becomes "apples, carrots, and oranges" | * If a field contains either a comma or a double quote or one of the end line chars, either the char(s) or the whole contents needs to be escaped. Excel escapes these values by embedding the field inside a set of double quotes. For example, a single cell with the text `apples, carrots, and oranges` becomes `"apples, carrots, and oranges"`. |
| Line 15: | Line 14: |
| * Strictly speaking, the delimiter is not defined in the specs. Some banks offer downloads where a semicolon is used instead of a comma. You might wonder why the name of this format is '''Comma''' Seperated Values, so, but anyway. | * Strictly speaking, the delimiter is not defined in the specs. Some banks offer downloads where a semicolon is used instead of a comma. You might wonder why the name of this format is '''Comma''' Seperated Values, so, but anyway. Several versions of Excel do not recognize a semicolon as a separator. |
| Line 17: | Line 16: |
| For details and background information see http://www.csvreader.com/csv_format.php | For details and background information see: . http://www.csvreader.com/csv_format.php |
| Line 22: | Line 23: |
| Reading and writing CSV files in APLX is very straightforward. You can just use the {{{⎕IMPORT}}} and {{{⎕EXPORT}}} system commands, specifying CSV as the format to use: | Reading and writing CSV files in APLX is straightforward; You can just use the {{{⎕IMPORT}}} and {{{⎕EXPORT}}} system commands, specifying CSV as the format to use: |
| Line 26: | Line 27: |
| myVariable←⎕IMPORT 'C:\Users\simon\Desktop\spreadsheet_data.csv' | myVariable←⎕IMPORT 'C:\Users\simon\Desktop\spreadsheet_data.csv' 'csv' |
| Line 28: | Line 29: |
| ⍝ Writing a CSV file | ⍝ Writing a CSV file |
| Line 30: | Line 31: |
| some_data ⎕EXPORT 'C:\Users\simon\Desktop\new_data.csv' | some_data ⎕EXPORT 'C:\Users\simon\Desktop\new_data.csv' 'csv' |
| Line 32: | Line 33: |
| For another example of reading a CSV file of Google finance data and charting it, see [[CSVandChartingAplx|here]] | |
| Line 34: | Line 36: |
| Line 49: | Line 50: |
| === First Step: Partition The String Being Read From File === |
=== First step: partition the string being read from file === |
| Line 54: | Line 54: |
| r←{ignoreBetween}PartitionRecordsWithDyalog string;masked;cr;lf;bool | r←PartitionRecordsWithDyalog string;masked;cr;lf;bool |
| Line 59: | Line 59: |
| ⍝ Note that everything between "ignoreBetween" is ignored. ⍝ This can be used to masked stuff between "" (CSV files), for example. |
|
| Line 62: | Line 60: |
| (cr lf)←⎕TC[2 3] ⍝ <CarriageReturn> and <LineFeed> :If 0=⎕NC'ignoreBetween' ignoreBetween←'' ⍝ establish default |
(cr lf)←⎕TC[2 3] ⍝ <CarriageReturn> and <LineFeed> :If 0<+/bool←(cr,lf)⍷string ⍝ are there any cr+lf in "string"? string←(~bool)/string ⍝ Let only the cr survive |
| Line 66: | Line 64: |
| :If ~masked←0∊⍴ignoreBetween masked←~{⍵∨≠\⍵}'"'=string ⍝ what is not escaped (between "") |
:If 0<+/bool←cr=string ⍝ Are there still any cr's? (bool/string)←lf ⍝ Convert them to lf |
| Line 69: | Line 67: |
| :If 0∊bool←~(cr,lf)⍷masked/string ⍝ are there any unmasked cr/lf in "string"? bool←(~masked)∨masked\bool ⍝ "insert" the masked string[1+{⍵/⍳⍴⍵}~bool]←cr ⍝ convert lf into cr string←bool/string ⍝ remove original cr masked←bool/masked :ElseIf 1∊bool←lf=masked/string ⍝ Are there any unmasked lf in "string"? ((masked\bool)/string)←cr ⍝ change them to cr |
⍝ In the remaining string, there might be lf's inside text, Those ⍝ need to be masked before we decide where recors really starts. masked←~{⍵∨≠\⍵}'"'=string ⍝ what is not escaped (between "") :If 1∊bool←lf=masked/string ⍝ are there any unmasked lf in "string"? r←(~masked\bool)⊂string :Else ⍝ so it's a single record r←⊂string |
| Line 77: | Line 75: |
| r←¯1↓(+\1,1↓masked\cr=masked/string)⊂string r←(0,1↓1⍴⍨⍴r)↓¨r |
|
| Line 80: | Line 76: |
=== Second Step: Extract The Real Data === |
=== Second step: extract the real data === |
| Line 84: | Line 78: |
| r←{sep}Csv2MatrixWithDyalog csv;bool;⎕IO;buffer;isNum | r←{sep}Csv2MatrixWithDyalog csv;bool;⎕IO;buffer;isNum |
| Line 98: | Line 92: |
| r←(~'""'∘⍷¨r)/¨r ⍝ Convert double-" into a single one | |
| Line 99: | Line 94: |
| === The Final Step === Put it all together: |
|
| Line 100: | Line 97: |
| === The Final Step === Put it all together: |
|
| Line 105: | Line 99: |
| ⍝ Read "filename" which is assumed to be a *.csv file | ⍝ Read "filename" which is assumed to be a *.csv file |
| Line 111: | Line 105: |
| Line 119: | Line 112: |
| Line 124: | Line 116: |
| APL is fine, very fine 1 2.2 ¯3 |
APL is fine, very fine 1 2.2 ¯3 |
| Line 127: | Line 119: |
The following functions take such an array as right argument and convert it into a string that can be written to a file with the extension ".csv". The left argument defaults to "windows" and can be "unix" or "mac" as well. Note that the left argument is case sensitive. The left argument is used to determine the appropriate record separator. |
The following function takes such an array as right argument and converts it into a string that can be written to a file with the extension ".csv". The left argument defaults to "windows" and can be "unix" or "mac" as well. Note that the left argument is case sensitive. The left argument is used to determine the appropriate record separator. |
| Line 131: | Line 122: |
| r←{os}Array2CsvWithDyalog array;cr;lf;sep;bool;IsChar | r←{os}Array2CsvWithDyalog array;cr;lf;sep;bool;IsChar;dq |
| Line 144: | Line 135: |
| bool←,(lf∊¨array)∨','∊¨array ⍝ where are special chars used? | dq←,'"'∊¨array ⍝ Where are double quotes in the text? (dq/,array)←{b←'"'=w←⍵ ⋄ (b/w)←⊂'""' ⋄ ⊃,/w}¨dq/,array ⍝ Double the double quotes bool←dq∨,(lf∊¨array)∨','∊¨array ⍝ where are special chars used? |
| Line 146: | Line 139: |
| bool←'"'∊¨array ⍝ where are special chars used? | |
| Line 148: | Line 142: |
| ((r='¯')/r)←'-' ⍝ Handle ¯ | ((r='¯')/r)←'-' |
| Line 150: | Line 144: |
| Line 153: | Line 146: |
| #.Array2Csv 2 3⍴'APL' 'is' 'fine, very fine' 1 2.2 ¯3 APL,is,"fine, very fine" 1,2.2,-3 |
#.Array2CsvWithDyalog 2 3⍴'APL' 'really "really" is' 'fine, very fine' 1 2.2 ¯3 APL,"really ""really"" is","fine, very fine" 1,2.2,-3 |
| Line 159: | Line 150: |
CSV to APL
Contents
CSV stands for comma separated values. Those files are still used to transport tabular data between applications that are not directly connected. The files can be edited with any spreadsheet application like Microsoft Excel.
There are some things one need to know about CSV file in order to deal with them:
- Fields are separated by commas. Well, mostly.
- Records are separated with system end of line characters, CRLF (ASCII 13 Dec or 0D Hex and ASCII 10 Dec or 0A Hex respectively) for Windows, LF for Unix, and CR for Mac
If a field contains either a comma or a double quote or one of the end line chars, either the char(s) or the whole contents needs to be escaped. Excel escapes these values by embedding the field inside a set of double quotes. For example, a single cell with the text apples, carrots, and oranges becomes "apples, carrots, and oranges".
Strictly speaking, the delimiter is not defined in the specs. Some banks offer downloads where a semicolon is used instead of a comma. You might wonder why the name of this format is Comma Seperated Values, so, but anyway. Several versions of Excel do not recognize a semicolon as a separator.
For details and background information see:
Note that the format comes with a nasty built-in-problem: there is no way to recognize a cell as being numeric. Converting cells which only contain a proper number does not help because if you enter a digit with a leading quote, Excel handles this as text but again this cannot be recognized as text in the csv file. The only solution is therefore to make an informed guess.
Reading and writing CSV files using APLX
Reading and writing CSV files in APLX is straightforward; You can just use the ⎕IMPORT and ⎕EXPORT system commands, specifying CSV as the format to use:
⍝ Reading a CSV file... myVariable←⎕IMPORT 'C:\Users\simon\Desktop\spreadsheet_data.csv' 'csv' ⍝ Writing a CSV file some_data←2 3⍴'APL' 'is' 'fine, very fine' 1 2.2 ¯3 some_data ⎕EXPORT 'C:\Users\simon\Desktop\new_data.csv' 'csv'
For another example of reading a CSV file of Google finance data and charting it, see here
Reading a CSV file using Dyalog APL
Given an Excel spreadsheet that looks like this:
Saving this into a csv file, the file can be read into APL. The variable would look like this:
To convert this into an APL matrix is a two-step-process:
- partition the simple string from file
- extract the data and build up the APL matrix
First step: partition the string being read from file
With the following two functions this variable can be transformed into an APL array where every item represents a record. Data masked by " remain unchanged. The functions can deal with files from Unix, Mac and Windows.
r←PartitionRecordsWithDyalog string;masked;cr;lf;bool
⍝ Takes a string and partitions records.
⍝ Can deal with Mac/Unix/Windows files.
⍝ For that, CR+LF as well as single LFs are converted into CR.
⍝ CR is then used to partition "string".
⎕IO←1 ⋄ ⎕ML←3
(cr lf)←⎕TC[2 3] ⍝ <CarriageReturn> and <LineFeed>
:If 0<+/bool←(cr,lf)⍷string ⍝ are there any cr+lf in "string"?
string←(~bool)/string ⍝ Let only the cr survive
:EndIf
:If 0<+/bool←cr=string ⍝ Are there still any cr's?
(bool/string)←lf ⍝ Convert them to lf
:EndIf
⍝ In the remaining string, there might be lf's inside text, Those
⍝ need to be masked before we decide where recors really starts.
masked←~{⍵∨≠\⍵}'"'=string ⍝ what is not escaped (between "")
:If 1∊bool←lf=masked/string ⍝ are there any unmasked lf in "string"?
r←(~masked\bool)⊂string
:Else ⍝ so it's a single record
r←⊂string
:EndIf
Second step: extract the real data
r←{sep}Csv2MatrixWithDyalog csv;bool;⎕IO;buffer;isNum
⍝ Convert vector-of-text-vectors "csv" that is assumed to
⍝ come from a *.csv file and which got already partinioned
⍝ into an APL matrix. Takes care of escaped stuff.
⍝ "sep" defaults to a comma but that can be changed by specifying a left argument.
⎕IO←1 ⋄ ⎕ML←3
sep←{2=⎕NC ⍵:⍎⍵ ⋄ ','}'sep'
r←(⌽∨\0≠⌽↑∘⍴¨csv)/csv ⍝ remove empty stuff from the end if any
bool←{~{⍵∨≠\⍵}'"'=⍵}¨r ⍝ prepare booleans useful to mask escaped stuff
r←⊃r{⍺⊂⍨⍵≠sep}¨bool{⍺\⍺/⍵}¨r ⍝ partition fields by unmasked commas
r←{'"'≠1⍴⍵:⍵ ⋄ ¯1↓1↓⍵}¨r ⍝ remove leading and trailing "
buffer←{0=+/bool←'-'=w←⍵:⍵ ⋄ (bool/w)←'¯' ⋄ w}¨r ⍝ "buffer" is a copy of r with "¯" for "-"
buffer←{0=+/bool←','=w←⍵:⍵ ⋄ (bool/w)←'.' ⋄ w}¨buffer ⍝ "," gets "."
r←buffer{↑1⊃v←⎕VFI ⍺:↑2⊃v ⋄ ⍵}¨r ⍝ make fields whith appropriate content numeric scalars
r←(~'""'∘⍷¨r)/¨r ⍝ Convert double-" into a single one
The Final Step
Put it all together:
r←DealWithCsv filename;data ⍝ Read "filename" which is assumed to be a *.csv file ⍝ and converts it into a matrix data←FileRead filename data←'"'PartitionRecordsWithDyalog data r←Csv2MatrixWithDyalog data
The resulting variable in APL would look like this:
Note that the 1 in the second row/second column got converted into the number because the contents of the cell remained of digits only. However, in the original Excel spreadsheet that cell is text; this is indicated by the small green triangle. This information is not contained in the CSV file.
Writing a CSV file using Dyalog APL
Given an APL array like:
⎕←2 3⍴'APL' 'is' 'fine, very fine' 1 2.2 ¯3 APL is fine, very fine 1 2.2 ¯3
The following function takes such an array as right argument and converts it into a string that can be written to a file with the extension ".csv". The left argument defaults to "windows" and can be "unix" or "mac" as well. Note that the left argument is case sensitive. The left argument is used to determine the appropriate record separator.
r←{os}Array2CsvWithDyalog array;cr;lf;sep;bool;IsChar;dq
⎕IO←1 ⋄ ⎕ML←3
:If 0=⎕NC'os'
os←'windows'
:EndIf
'Invalid left argument; must be one of: windows, unix, mac'⎕SIGNAL 11/⍨~(⊂os)∊'windows' 'unix' 'mac'
'Right argument must have a depth of 2'⎕SIGNAL 11/⍨2≠≡array
'Right argument must be either a matrix or a vector'⎕SIGNAL 11/⍨~(⍴⍴array)∊1 2
(cr lf)←⎕TC[2 3] ⍝ <CarriageReturn> and <LineFeed>
sep←('windows' 'unix' 'mac'⍳⊂os)⊃(cr,lf)lf cr ⍝ select proper record separator
IsChar←{0 2∊⍨10|⎕DR ⍵} ⍝ Version 12 compatible
bool←,~IsChar¨array ⍝ locate number
(bool/,array)←⍕¨bool/,array ⍝ make numbers text
dq←,'"'∊¨array ⍝ Where are double quotes in the text?
(dq/,array)←{b←'"'=w←⍵ ⋄ (b/w)←⊂'""' ⋄ ⊃,/w}¨dq/,array ⍝ Double the double quotes
bool←dq∨,(lf∊¨array)∨','∊¨array ⍝ where are special chars used?
(bool/,array)←{'"',⍵,'"'}¨bool/,array ⍝ escape field with special chars
bool←'"'∊¨array ⍝ where are special chars used?
array←{⊃{⍺,',',⍵}/⍵}¨↓array ⍝ separate fields by comma
r←⊃,/array,¨⊂sep ⍝ make it simpel
((r='¯')/r)←'-'⍝ Example: #.Array2CsvWithDyalog 2 3⍴'APL' 'really "really" is' 'fine, very fine' 1 2.2 ¯3 APL,"really ""really"" is","fine, very fine" 1,2.2,-3
Author: KaiJaeger