SSIS - Handling Different Types Of Record In Same Flat File Source

Hi,

I've just started looking at SSIS and have encountered what should hopefully be a simple problem to solve. I have a pipe-separated source file that looks like this (I've added Line numbers for simplicity):

Ln 01: HDR|FEED_CODE|31-MAR-2006
Ln 02: Tom|100|Jones|ZZ1 1ZZ|USA
Ln 03: Tom|200|Singer|
Ln 04: Tom|305||Red|Porche ||Lanzarote |Apple|Carrot| | |
Ln 05: Dick|100|Van Dyke|ZZ1 1ZZ|USA
Ln 06: Dick|200|Actor|
Ln 07: Dick|305||Blue|Ford||California |Tomato | |||Beef
Ln 08: Harry|100|Houdini|ZZ1 1ZZ|GBR
Ln 09: Harry|200|Escapologist|
Ln 10: Harryk|305| |Green ||Triumph |Poland|Banana|Sprout| | |
Ln 11: TRL|9


In addition to a header and footer records, this file contains three record types for each person.

Record types are identified by the second column.

Each record type has a different number of columns:

Type 100 has 5 columns
Type 200 has 4 columns
Type 305 has 12 columns

The Row delimiter for all records is the {CR}{LF} character

I've set up a flat file input source and specified {CR}{LF} as the row delimiter for both header and data rows and the "|" character as the field delimiter.

It appears that SSIS is assuming that because the first data row has 5 columns, then everything must fit that format too. So the {CR}{LF} character that separates lines 02 and 03 is interpreted as text rather than a separation character and all remaining | field separators after 305 are interpreted as text containing in the fifth column. SSIS is also complaining that the last row is incomplete.

A bit like this (I've used tildes to indicate column separation):

Tom~100~Jones~ZZ1 1ZZ~USA
Tom~200~Singer~{CR}{LF}Tom~305||Red|Porche ||Lanzarote |Apple|Carrot| | |

I've seen one other reference to this behaviour but the response seemed to be SSIS doesn't know which columns are missing. In this scenario, we don't have missing columns, rather, we have different types of record in a single file. in DTS I would effectively parse the file once for each record type thus:

if cStr(DTSSource("Col002")) = "100" then

DTSDestination("in_Name") = trim(DTSSource("Col001"))
...

Main = DTSTransformStat_OK
else
Main = DTSTransformStat_SkipInsert
end if


...not the most efficient solution I know but the load only runs once a month so this was an acceptable workaround.

DTS was never this fussy but I'm sure this is user error rather than an SSIS limitiion. Can someone please put me straight?

Many thanks,

Greg