Multiple Record Types In One File
May 14, 2007
I have a text file to import where there are three file types: a header which has info about who sent the file and begins with 'H', detail records that begins with D and a trailer record that begins with T and just has the record count following that. The fields are delimited by '*'. H, D and T records each contain a different number of fields.
I suspect that what I should do is to split this file into three separate files. I tried to do this with SSIS but ran into problems. If I make the output a file destination, it won't let me use that output as input for the next process. There are no arrows I can grab onto to link to the next transform.
This is my first SSIS package although I made hundreds of DTS packages a few years ago. I can't figure this out in DTS either.
This sounds like it should be an EASY thing to do.
View 3 Replies
ADVERTISEMENT
Sep 27, 2006
A flat file I must parse has multiple record types in the same file that must be processed together. For instance, in the following example:
01TestStuff888
02TestStuff2
03TestStuff3
01TestStuff4
02TestStuff5
Each 01 record type has the records after it associated to it until the next 01 appears, so TestStuff would have TestStuff 2,3 related to it while TestStuff 4,5 belong together. In the example the 888 in the 01 record is the key to the group, but it does not appear in the following lines.
The problem is that each record type has different line formats, columns, etc, so they must be parsed differently. I have created a conditional branch on the first two characters, and written each record type out to a seperate flat file for that type, so that they can be imported again and parsed with the Flat File Source, but I am unsure how to relate them again. I tried appending the 888 to the other lines before they were written out, but I can't find a way to share the variable across the conditional split branches using a script component.
Does anyone have an idea how I could parse these files and keep the relationship intact?
Is there a way to tell the flat file wizard to use a different map based on certain characters?
Is there a way to share a variable across the different braches of a conditional split.
Strange question I know, but thanks for any help.
Travis
View 4 Replies
View Related
May 1, 2008
I'm using SSIS to import seven flat files (each containing a different record type) into a staging database. This part was easy.
Now I need to export the records from all seven tables into a single flat file structured in a nested hierarchy using common keys. (This format is required by the vendor for loading data into a new system).
I could use some ideas on the data transformations needed to combine all seven record types into an hierarchical record set which can then be written to my Flat File Destination. I'm currently looking at an article on SLQIS.com ("Handling Different Row Types In The Same File") which seems close to what I need, but they are importing (ref: www.sqlis.com/54.aspx ). I'm not sure if I should just reverse this for export or use something different. Any comments are appreciated.
Diagram of Record Hierarchy
typeA (parent key, ...)
typeB1 (parent key, childSet key, date, ...)
typeB2 (parent key, childSet key, ...)
typeC (parent key, childSet key, ...)
typeD (parent key, childSet key, ...)
typeE1 (parent key, childSet key, date, ...)
typeE2 (parent key, childSet key, ...)
The record types B1 through E2 form a complete set. Each set has it's own unique child-set key. There may be one or more sets for each typeA record (although it's possible that typeE records don't exist in the most recent set).
View 3 Replies
View Related
Jul 20, 2007
Howdy all,
I've seen several posts about reading and writing files that have different record types with varying column metadata. My particular file has 11 record types plus several header types and looks something like:
<Header1>
<Header2>
<Detail01-#1>
<Subdetail02>
<Subdetail03>
...
<Detail01-#2>
<Subdetail02>
<Subdetail03>
...
...
Since i need to get different detail and subdetail records, i can't really use the technique of 3 dest file connection managers found in http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=87269&SiteID=1
I've tried using an exec sql to get the main detail records and then a forech ADO en umerator that would get the subdetails, but it all seems so kludgy. I'm starting to think that I should just write the bulk of the file creation code in a c# app instead of trying to smush this into SSIS. Opinions? Am I missing some trick in SSIS?
TIA,
-Peter
View 7 Replies
View Related
Feb 26, 2008
I am attempting to create a multi-record file (as described in my last thread) and have found the following set of instructions very helpful:
http://vsteamsystemcentral.com/cs21/blogs/steve_fibich/archive/2007/09/25/multi-record-formated-flat-file-with-ssis.aspx
I have been able to create a sample file with two of my record types.
I now need to build on this further, because I have 9 record types in total that need to be extracted to a single flat file.
does anyone have any ideas how I might extend the example above to include more record types or know of another means of achieving this?
Thanks in advance for any help you might be able to provide.
View 3 Replies
View Related
Jan 15, 2015
I want to return Order records which are one type and don't have the other type.
The issue is I have Orders with which has 2 distriubtion types .
Example
Order 12345 has Type S and Type X.
Order 67891 has Type S
I only want to return Order 67891 that are s Type and does not have type X
View 1 Replies
View Related
May 2, 2007
I want to enumerate all *.xls and *.csv file. How to fill the Files box? I tried
*.xls, *.csv
*.xls *.csv
*.(xls|cvs)
all doesn't work
View 4 Replies
View Related
May 10, 2006
Hi,
I've just started looking at SSIS and have encountered what should hopefully be a simple problem to solve. I have a pipe-separated source file that looks like this (I've added Line numbers for simplicity):
Ln 01: HDR|FEED_CODE|31-MAR-2006
Ln 02: Tom|100|Jones|ZZ1 1ZZ|USA
Ln 03: Tom|200|Singer|
Ln 04: Tom|305||Red|Porche ||Lanzarote |Apple|Carrot| | |
Ln 05: Dick|100|Van Dyke|ZZ1 1ZZ|USA
Ln 06: Dick|200|Actor|
Ln 07: Dick|305||Blue|Ford||California |Tomato | |||Beef
Ln 08: Harry|100|Houdini|ZZ1 1ZZ|GBR
Ln 09: Harry|200|Escapologist|
Ln 10: Harryk|305| |Green ||Triumph |Poland|Banana|Sprout| | |
Ln 11: TRL|9
In addition to a header and footer records, this file contains three record types for each person.
Record types are identified by the second column.
Each record type has a different number of columns:
Type 100 has 5 columns
Type 200 has 4 columns
Type 305 has 12 columns
The Row delimiter for all records is the {CR}{LF} character
I've set up a flat file input source and specified {CR}{LF} as the row delimiter for both header and data rows and the "|" character as the field delimiter.
It appears that SSIS is assuming that because the first data row has 5 columns, then everything must fit that format too. So the {CR}{LF} character that separates lines 02 and 03 is interpreted as text rather than a separation character and all remaining | field separators after 305 are interpreted as text containing in the fifth column. SSIS is also complaining that the last row is incomplete.
A bit like this (I've used tildes to indicate column separation):
Tom~100~Jones~ZZ1 1ZZ~USA
Tom~200~Singer~{CR}{LF}Tom~305||Red|Porche ||Lanzarote |Apple|Carrot| | |
I've seen one other reference to this behaviour but the response seemed to be SSIS doesn't know which columns are missing. In this scenario, we don't have missing columns, rather, we have different types of record in a single file. in DTS I would effectively parse the file once for each record type thus:
if cStr(DTSSource("Col002")) = "100" then
DTSDestination("in_Name") = trim(DTSSource("Col001"))
...
Main = DTSTransformStat_OK
else
Main = DTSTransformStat_SkipInsert
end if
...not the most efficient solution I know but the load only runs once a month so this was an acceptable workaround.
DTS was never this fussy but I'm sure this is user error rather than an SSIS limitiion. Can someone please put me straight?
Many thanks,
Greg
View 7 Replies
View Related
Sep 18, 2006
We have a flat file format generated from a vendor. It contains a "mainframe" view of the data with a header record, batch header record, detailed records, batch trailer record and trailer record. It arrives as a .dat file. What is the best approach to extract the necessary columns out of this file to populate the corresponding SQL server tables and rows?
View 3 Replies
View Related
Apr 4, 2007
I am trying to load a file using SSIS that contains records with two different layouts in one data file but in the flat file connection I can only specify one layout and this is causing the records with the second layout to be loaded incorrectly.
The different record layouts can be identified by the first character of the record. Example: If Field begins with "A" then assign one layout; "B" assign second layout.
Has anybody come accross this issue, if so some guidence would be appreciated.
Thanks,
ire_ssis
View 10 Replies
View Related
Jan 10, 2008
Hi,
How can we insert multiple records in a OLEDB destination table for each entry from the source table.
To be more clear, for every record from source we need to insert some 'n' number of records into the destination table. this 'n' changes depending on the record.
how is this achieved.
thanks.
View 4 Replies
View Related
Jul 8, 2005
Since most use SQL server, I thought I would post the question here.Is it possible, or is there a product or DBMS that enforces referential integrity across multiple databases and database types? Such as SQL Server, Oracle, etc...Thanks,Zath
View 2 Replies
View Related
Sep 21, 2006
Hi,
I am trying to display a report from a bug tracker that shows the number of issues logged per month, per year and how many issues have been closed in that month. I have no difficulty displaying the number of issues that have been logged within a particular months, but when I need to display the number of issues closed - I get stuck. I am calling from one table 'bugs', which contains date_reported, status_id. I can do a count on the number of bugs between certain dates but I cannot list the number of closed issues in that exact time. I get the feeling I need a select statement within a select statement? If anyone can help. Thanks
View 3 Replies
View Related
Sep 25, 2013
Our database stores vehicle data in one table, but 3 different types of data are stored in the one table. The table contains all the columns for all 3 types so when you query the table you get at least 3 rows back with null values for all the columns that don't apply to that record. The data is imported to the table when it's updates so there's a possibility that they're updated at different times so they have a different BATCH like:
BATCH TYPE ID RATING INSURANCE SAFETY
300 SAFE 123 NULL NULL A
300 INS 123 NULL YES NULL
250 RATE 123 A NULL NULL
What I'd like returned is:
ID, RATING, INSURANCE, SAFETY
123 A YES A
I'm trying to do a case statement to pull the data down, but I keep ending up with multiple rows because of all the nulls. I tried doing a SUM of the case statement with an ISNULL(SAFETY,0) but I can't SUM char values. I can probably do this with 3 temp tables to load the data that I want for each TYPE into them and then select and join them together, but is there a better way to do this?
View 2 Replies
View Related
Mar 22, 2007
Hello All,
We are developing customer support application. We will have so many customers after launching this product. But my problem is how will i store data of all these customers in SQL Server. Please suggest me.
View 1 Replies
View Related
Jan 24, 2008
Does anyone know if it is possible to have text in a single table cell where the first field is formatted in italics and the second is in normal?
eg: = Fields!firstname.Value(as italic) & " " & Fields!lastname.Value(as normal)?
shidot
View 5 Replies
View Related
Apr 29, 2007
hello,I am building a web page in asp.net 2.0 and i'm looking for a way to save various files uploaded by users (such as doc, pdf, cs, txt .... whatever). Obviously the normal way would be to store it on the filesystem of the web server (or any other middleware server).But I am asking if there is a way to store these files in the database, and not affect too much of the performance of the web page?I know a picture can be stored easily, but what about all the other file types? I would appreciate if someone can spead light of the subject.Thanks,Tranquil.
View 1 Replies
View Related
Feb 6, 2008
Hi,
I have a flat file that contains 2 types of records - Dev and production. The Dev will be noted with an D and the Production with a P. These records are different - The dev records are in a different order and contain different info then the Production. I need to use SSIS to import the data into 2 different SQL Tables. How to do this?
Can any one help me
Thanks in advance
View 3 Replies
View Related
Mar 8, 2006
I'm getting a bit lost in SSIS. I've got an Excel source file that I'm trying to load into a table. I keep getting validation errors that warn about not being able to convert between unicode and non-unicode string data types.
I'm trying figure out where I have to change this and am frankly confused. It seems SSIS is selecting various columns as unicode/WSTR data types, but I want them to import as regular string types.
On the Data Flow tab in SSIS, I right-click on the source Data Flow component (the Excel file) and select Show Advanced Editor. Then on the last tab, Input and Output Properties, there's a tree view for the Excel output. There are "External Columns" and "Output Columns" containers in the tree view.
I tried setting some of these but they don't seem to "take". Do I need to change the data type for each column under both the External and Output columns?
That seems like a lot of work! And, as I say, I tried setting some, but I still got the same validation errors. So, then I go back to this spot (Advanced Editor -> Input and Output Properties tab) and my changes seem to have been lost.
Any help would be appreciated!
View 5 Replies
View Related
Jul 17, 2015
Is it possible to categorize a table by "type"; for example "Master file" or "Transaction file"?
If yes, then:
Where is this specification made on the table, and how can I interrogate the table to determine the "type" ....
View 8 Replies
View Related
May 25, 2007
Hello everybody.
I have a problem with "Flat file"-connection, which I cannot understand at the present. Here is the issue: I've got an ASCII-file containing 233898 lines. I try to read this file in two different packages using two different connections. In the first connection I used default data type - DT_STR of length 50, in the second one I used "Propose types..." feature (with 2000 samples) to detect types which are better matching the reality. And, when I try to load my data, the first connection reads exactly 233898 lines from the file, the second one 203898. Somehow it skips 30000 lines unloaded.
I tried to observe the error output for the second connection - everything goes smoothly and problemless. But somehow those 30000 lines are missed.
Has anybody experienced such a situation? Is the issue known?
Thanks in advance,
A.G.
View 9 Replies
View Related
Jul 3, 2007
This question is around how we can get the data types and lengths populated into the flat file source columns.
In Connection Manager, you have your flat file defined. You can choose "Suggest Types...", and the minimum lengths and correct data types will be returned from within the data in the flat file.
Is there some way to automate this data type definition, but coming from the other direction (coming from the destination table that we are loading)?
For example, you have mapped the columns that will be loaded. Can you then reverse engineer the data types and lengths for the columns in the flat file from the destination table?
TIA
View 5 Replies
View Related
Jul 31, 2014
I need to write a process to get file size in kb and record count in a file. I was planning on writing a c# console app that takes the file path and name as a param however should i use a CLR?
I cant put a script in the ssis when it's bringing the file down because it has been deemed that we only use ssis for file consumption.
View 1 Replies
View Related
May 26, 2005
SQL Server 2000, ASP.Net 1.1
I've been writing this stuff for a while, and can't seem to come to the
conclusion of how I should be retrieving data and assigning this data
to variables.
Since i'm using SQL Server, I'm convinced that I should be using the
datareaders GetSqlDouble (or whatever) function, but this would mean i
need my local variables to be one of the SQL types. The problem
with that is, that there will have to be lots of conversions done by me
to be able to use a SQL type in my application.
For instance, I have a class where i'm retrieving dates. In order
to retrieve them correctly (Null values included), I need to retrieve
them with GetSqlDateTime(), then when it comes time to display the date
in a table, i must first check for nulls, then convert to a
string. This seems to be very cumbersome. Would I be better
off just using GetDateTime(), and the .ToString method, and ignoring
Sql Types all together?
so, basically, how are you guys using your sql server data? with
the supplied sql types, and doing all of the post-processing work
manually? I feel like i'm having trouble conveying my
issue...hopefully someone knows what i mean....i'd just like some
direction to save trouble in the long run, since i feel like there's
got to be a better way...
Confused!
Thanks,
JJ
View 1 Replies
View Related
Jun 11, 2008
SOURCE TABLE
ID SEQ COM VARCHAR(10)
1 1 I
1 2 am
1 3 doctor
2 1 I
2 2 am nurse
DESTINATION TABLE --Each comm field is varchar(20)
ID_COM1___COM2______COM3
1__I______am_________doctor
2__I______am nurse____NULL
Please Help
View 6 Replies
View Related
Oct 13, 2007
I have two table and there are releationsheep between them.
Topic:
topicId, topicName, topicDate
Reply:
replyId,topicId,replyName,replyDate
I want to insert two table at the same time.
Topic: 1,example,10/21/2007
Reply:1,1,example, 10/21/2007
View 1 Replies
View Related
Dec 9, 2002
I am trying to do an update to a database with the code below, the code will work in generating a single record (if i exclude the IF statement) but i need it to create multiple new records based on the IF statement (or something similar) i have in the code. It needs to create updates for every record that has a parentguid that matches the one specified in the code.
the idea is, the hierachy structure in the tables are:
Location1
--> Part1
--> part2 etc
Location2
--> part1 etc
i want it so that when i input a new part it updates to all the locations. The locations all have the same parentguid (the one specified in the code)
With regards to the stored procedure it executes, this stored procedure creates the entry. I don't really want to change the stored procedure because other functions reference and use it.
My code:
DECLARE @PREFIX VARCHAR(6)
DECLARE @CODE VARCHAR(8)
DECLARE @GUID UNIQUEIDENTIFIER
DECLARE @PARENTGUID UNIQUEIDENTIFIER
DECLARE @PARENTGUID2 VARCHAR(50)
select @parentguid2 = parentguid from dsdba.itemgroups
if @PARENTGUID2 = '8CF850AD-2026-411B-AABE-BF1584624EB3'
BEGIN
SET @PREFIX = 'JUST'
SELECT @PARENTGUID = GUID FROM DSDBA.iTEMGROUPS WHERE CODE = @PREFIX
SET @CODE = @PREFIX + '02'
SELECT @GUID = NEWID()
EXEC DSDBA.usp_ItemsDB_InsertGroup @GUID, @CODE, 'JMB', @PARENTGUID, 1
END
:confused:
View 1 Replies
View Related
Mar 3, 2004
can anyone help to explain how to insert multiple record into one/two table?
ex:lets say when user specify start date and end date, then we need to created and insert the record on that duration.
and how to do with insert the record weekly or monthly?until the end of the date!
View 1 Replies
View Related
Jul 19, 2013
I have following situation -
DECLARE @table1 TABLE
(
Stuid INT NOT NULL,
schoolcd varchar(10),
enterDate Datetime,
entercd char(10)
)
INSERT @Student VALUES
(4648, ‘20130515’, ‘892’,’30’),
(4648, ‘20130103’, ‘764,’10’)
SELECT STATEMENT will give me following -
Stuid schoolcd entercd enterDate
46488923005/15/2013
46487641001/03/2013
Desired output is one row for two or more records for same stuid
464889230 05/15/20137641001/03/2013xxxxxxx/xx/xxxx
How can I accomplish that?
View 5 Replies
View Related
Jan 26, 2007
Hi there,
I've a complex stored procedure, which I'm trying to get a list of one or more users based the list of records that are being display. If I mentioned the overall store procedure here it'll turn you off, so I'll put the questions in bitesize chunks.
Example would be if Bob and Luke work for Acrm Corp and John for Maxwellarms Ltd, and there was an intimediate table which linked both. With my stored procedure I want to display both names in the same row as Acrm Corp. Is there a way to do this?
User tableUserid Username
1 Bob2 Luke3 John
Store procedure resultCompanyname Username
Acrm Corp NullMaxwellarm Ltd Null
Regards,
View 1 Replies
View Related
Apr 18, 2005
It's come up more than once for me, where I need to DECLARE and SET several SQL variables in a Stored Procedure where many of these values come from the same table record - what is the best method for doing this, where I don't have to resort to making a separate query for each value.
Currently I'll do something like this:
DECLARE @var1 intSET @var1 = (SELECT TOP 1 field1 FROM table1 WHERE recordkey = @somekey)DECLARE @var2 nvarchar(20)SET @var2 = (SELECT TOP 1 field2 FROM table1 WHERE recordkey = @somekey)
Of course, I'd rather just have to query "table1" just once to assign my variables.
What obvious bit of T-SQL am I missing?
Thank you in advance.
View 2 Replies
View Related
Feb 25, 2008
Hi,
I have a table with a large number of records that I need to delete, before attempt to perform the delete I also archived the records to another table.
So I need to delete all of these selected records stored in the archive table from the main table. I can now reference all the records that qualify for the delete in the main table by performing a join on the archive table like so:
select * from my_main_table a
join my_archive_table b
on a.distinct_id=b.distinct_id
and a.surrogate_key=b.surrogate_key
and a.identifier=b.indentifier
So all the records check out to be the ones I'd like to perform a delete on but I just can't figure out how to perform a delete of the records with little or no change to the existing query.
Obviously something like this won't work:
delete from my_main_table a
join my_archive_table b
on a.distinct_id=b.distinct_id
and a.surrogate_key=b.surrogate_key
and a.identifier=b.indentifier
Though it would be nice if it did.:D
So my question is how would I use the existing query with some modification to delete only the records that this query returns. I've tried selection of records in the main table based on the existing records in the archive table but it can return a higher number of records than what I know is expected. I actually need the join specified to be in place to do it.
Can anyone render any assistance on this one???
I would certainly appreciate it.
Thanks.
View 1 Replies
View Related
Jun 16, 2006
I am pretty new to the DB part of this but have built an asp.net web appplication with 2 tables:
FORMS and UNITS
I have created a web page that will allow users to add forms and associate a unit with that form. I now need to be able to allow users to associate the form with multiple units.
I can change the web page list box to allow multiple selections but that doesn't solve the problem. This seems like a pretty simple task but I can't seem to find anything on it.
any help???
below is the stored procedure I was using:
CREATE PROCEDURE dbo.USP_AddForm
@UNIT_IDint,
@F_TYPE varchar(20),
@Titlevarchar(100),
@Keywords ntext = NULL,
@Descriptionntext = NULL,
@FileNamevarchar(50)
AS
BEGIN
SET NOCOUNT ON
DECLARE @NewFOrmID INT
INSERT FORMS
(
UNIT_ID,
F_TYPE,
Title,
Keywords,
Description,
FileName,
RevDate
)
VALUES
(
@UNIT_ID,
@F_TYPE,
@Title,
@Keywords,
@Description,
@FileName, GetDate()
)
SELECT @newFormID = @@IDENTITY
SELECT newFormID = @newFormID
END
GO
View 1 Replies
View Related
