iFit: Loaders import sub-library

1. iLoad the main file importer
   1. Distant files (http://, ...)
   2. Compressed files (.zip, ...)
   3. Internal anchor references (file#anchor)
2. Supported data formats (default)
   1. Legacy data formats (text, NetCDF, HDF, FITS, images, XML, CSV, ...)
   2. More specific data formats (neutrons and X-ray)
   3. Unsupported data formats: how to import them anyway ?
3. The import filters: customizing data formats (iLoad_ini)
   1. Reading and Writing the iLoad configuration
   2. iLoad configuration structure
   3. In short: procedure to add a new data format

iLoad: the main file importer

>> a = iData([ ifitpath 'Data/ILL_IN6.dat' ]);
>> a = load(iData, [ ifitpath 'Data/ILL_IN6.dat' ]);

1. resolves the file name, and identifies directory importation, wildcards (*, ?), distant files with URLs (file://, ftp://, http://, https://), compressed files (zip, gzip, tar, compress) ;
2. selects a list of possible file types from e.g. file extension ;
   
3. searches for patterns in text headers to better determine file type ;
   
4. requests importation by all possible file types one after the other until one succeeds ;
   
5. formats the file content so that it is returned as a structure containing standard fields ;
   
6. executes remaining post-process scripts.
   

>> a = iLoad('')

>> a = iLoad(filename); % this is a structure
>> a = iData(a);        % convert to iData object

>> iLoad('formats')
...
HDF   read_hdf5/load_psi_RITA  HDF5
...
>> a = iLoad(filename, 'HDF5'); % explicitly use HDF5 loader

Distant files (http://, ...)

• file://filename
• ftp://filename
• http://filename
• https://filename
  

Compressed files (.zip, ...)

Internal anchor references (file#anchor)

Supported data formats (default)

>> a = iLoad('formats');

Legacy data formats (text, NetCDF, HDF, FITS, images, XML, CSV, ...)

More specific data formats (neutrons and X-ray, ...)

Unsupported data formats: how to import them anyway ?

>> iData('filename','text format with fast import method')
>> iData('filename','Data (text format)')

>> a=iLoad(filename, 'text', '--catenate','--fast','--headers','--wrapped', ... other options);

>> iLoad(' ','text','--help')

The import filters: customizing data formats (iLoad_ini)

Reading and Writing the iLoad configuration

>> iLoad('formats');	% display the list of supported formats
>> iLoad('force');	% force re-read of the configuration file and check importers

[ ifitpath 'Libraries' filesep 'Loaders' filesep 'iLoad_ini' ]

prefdir

>> iLoad('save');
% Saved iLoad configuration into /home/farhi/.matlab/R2015a/iLoad.ini <- this is prefdir

>> delete([ prefdir filesep 'iLoad_ini.m' ]);

>> config = iLoad('config'); % retrieve the iLoad configuration and file loaders (from cache)
>> iLoad('save', config);

iLoad configuration structure

• loader.name: a description of the format
  
• loader.method: the function used to read the file and return a structure or numerical array: data=method(filename)
  
• loader.extension: a single file extension, or a cell string of file extensions
  
• loader.patterns: a cell string of patterns to search in text headers (less than 10% of binary data in the first 10 kb) [optional]
• loader.options: additional options to be passed to the loader.method. A single string is catenated after the file name, a cell array is sent as additional arguments to the method. [optional]
• loader.postprocess: a script which handles iData commands to shape the final object, id=loader.postprocess(id) [optional]
  

format14.name       ='INX tof data';
format14.extension  ='inx';
format14.patterns   ={'INX'};
format14.method     ='looktxt';
format14.options    ='--headers --fortran  --catenate --fast --comment=NULL';
format14.postprocess='load_ill_inx';


(...)

config.loaders =  { format1, format2, format3, format4, format5, format6, ...
	       format7, format8, format9, format10, format11, format12, format13, format14, format15 };

struct = looktxt(filename)

struct = looktxt([ filename options ])

struct=method(filename, options{1}, options{2}, ...)

object = postprocess( iData( method(filename, options...) ) )

>> iLoad('force load config'); % force to re-read the configuration files

In short: procedure to add a new data format

1. Start by getting the documentation about the new data format you wish to implement. This gives the format name and the extension.
   
2. Then write a function (locally or in ifitpath/Loaders) that takes as input a filename and returns the file content as a structure. This is where you will open the file, access its guts, and return something (or empty/error if this fails). This is your method.
   
3. Edit the iLoad_ini file from prefdir or ifitpath/Loaders.
4. Create a new structure (which you may name as you wish) and set the name (description), method, and extension fields.
5. Optionally define some patterns to search, to complement the extension (as a list of words).
6. Optionally write a post-process script and preferably place it in the ifitpath/Loaders/postprocess.
7. Add the new structure to the config.loaders list (cell) at the end of the iLoad_ini file.
8. Save the file, and optionally re-read it with iLoad('force load config').
9. Send me [farhi (at) ill.fr] the method/postprocess and structure information so that I can contribute your work to the package.
   

E. Farhi - iFit/Loaders - Aug. 22, 2017 1.10 - back to Main iFit Page