KnowledgeHub
Questions
Tags
Users
Search
Alex Rivera
|
Logout
Edit Question
Title
Body
I am processing data from a set of files which contain a date stamp as part of the filename. The data within the file does not contain the date stamp. I would like to process the filename and add it to one of the data structures within the script. Is there a way to do that within Pig Latin (an extension to PigStorage maybe?) or do I need to preprocess all of the files using Perl or the like beforehand? I envision something like the following: -- Load two fields from file, then generate a third from the filename rawdata = LOAD '/directory/of/files/' USING PigStorage AS (field1:chararray, field2:int, field3:filename); -- Reformat the filename into a datestamp annotated = FOREACH rawdata GENERATE REGEX_EXTRACT(field3,'*-(20\d{6})-*',1) AS datestamp, field1, field2; Note the special "filename" datatype in the LOAD statement. Seems like it would have to happen there as once the data has been loaded it's too late to get back to the source filename.
Tags (comma-separated)
Save Edits
Cancel