Dynamic Selection Of Flat File

Oct 6, 2006

I am writing a package where the user uploads a flat file to a web folder. I need to automate this package to run everytime it sees a new file.

How can I implement this?

Can I make a call to a package or a sql server job to run from .net 2.0?

Do I need to use a service broker to look for a new file and run the package or a stored proc....I am looking for an async process where user doesnt have to wait for the package to run as it involves data validation of flat file and its huge...

Please help!!

View 2 Replies


ADVERTISEMENT

Integration Services :: Get FileName Fo Each File Created Via Dynamic Flat File Destination

Jul 24, 2015

Need to know how I can get the dynamic filename created in the FlatFile destination for insert into a package audit table?

Scenario: Have created a package that successfully outputs Dynamiclly named flat files { Format: C:Test’Comms_File_’ + ‘User::FileNumber’+’_’+Date +’.txt’

E.g.: Comms_File_1_20150724.txt, Comms_File_2_20150724.txt  etc} using Foreach Loop Container  :

* Enumerator Set to: “Foreach ADO Enumerator” with the ADO object source variable selected to identify how many total loop iterations there are i.e. Let’s say 4 thus 4 files to be created

*Variable Mappings : added the User::FileNumber – indicates which file number current loop iteration is i.e. 1,2,3,4

For the DataFlow task have a OLDBSource and a FlatFile Destination where Flat File ConnectionString is set up as:

@[User::Output_Path] + "Comms_File"+ @[User:: FileNumber] +"_" + replace((DT_WSTR, 10) (DT_DBDATE) GETDATE(),"-","")+ ".txt"

All this successfully creates these 4 files:

Comms_File_1_20150724.txt, Comms_File_2_20150724.txt, Comms_File_3_20150724.txt, Comms_File_4_20150724.txt

Now the QUESTION is how do I get these filenames as I need to insert them into a DB Audittable. The audit table looks like this:

CREATE TABLE dbo.MMMAudit
  (
     AuditID      INT IDENTITY(1, 1) NOT NULL,
     PackageName     VARCHAR(100) NULL,
  
FileName           VARCHAR(100) NULL,
     LoadTime        DATETIME NULL,
     NumberofRecords INT NULL
  ) 

To save the Filename & how many records in each file in our Audit Table, am using an Execute SQL Task and configuring it as this:

Execute SQL Task

Parameter mapping - Mapped the User Variable (RecordsInserted) and System Variable( PackageName) to Insert statement as shown below

SQLStatement: INSERT INTO [dbo].[MMMAudit] ( 
PackageName,NumerofRecords,LoadTime)
 (?,?.GETDATE)

Again this all works terrific & populates the dbo.MMMAudit table as shown below BUT I also need to insert the respsctive file name – How do I do that?

AuditID PackageName FileName  NumberOfRecords
1           MMM       NULL                      12
2          MMM  NULL                23
3          MMM  NULL      14
4          MMM  NULL              1                     

View 2 Replies View Related

Regarding The Dynamic File Change Within Flat File Source.

Mar 28, 2006

Hi,

we have one requirement to run the package daily basis.

The package should run at specific time on that day.

we are using windows schedular for that.

we will have one new flatfile everyday.

Is there any process to attach this file to flat file source dynamically?

The requirement is,

The flat file should be able to read the new flatfile everyday.

we have no option change it manually, the flatfile source should have to take the file automatically at that time.

So that it can take that flatfile and load it into database table.

View 1 Replies View Related

Dynamic Flat File Creation!

May 14, 2008

I'm trying to read the folder which has flat files and transform it and put it another folder with the SAME original name.
I created a For Eachloop containter and assign it to variable infile. In the data flow I added some transform and flat file destination. currently its reading 3 files and output is only one file.
I would like to create dynamic flat file connection so that it'll create separate files (same name) in different folder.
Thanks,

View 5 Replies View Related

Dynamic Flat File Destination Name!

May 4, 2007

Hi,

I am trying to access from OLE DB source. And based on one of the columns, I need to write the data to a Flat File Destination.

For Example,

CustID, ProductID, Product Name, Product Description

Say I am going to write to a different Flat File for every product. So if there are 10 products in the data. There should be 10 Flat Files. Also the file name should include the Product Name And Product ID.

It is being done in a single Data Flow Task.

Right now the Property Expression for the File Name is which is not working)




Code Snippet@DestFolder + [Data Conversion].ProductID + @TodaysDate + ".txt"





The ProductIDs are in the ascending order. Any help or guidance?

Thanks

-Leo







View 7 Replies View Related

Dynamic Flat File Creation

Mar 19, 2008



Hi All,


I'm new to SSIS. I have a small problem. In our system we are trying to create flat files based on data from returned from different SQL stored procedures. The configuration settings for creating the flat files are stored in a SQL table.

For example, the config table will look like below :

Feed Id Feed Name Feed File Name Feed Stored Procedure Name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 Biz C:Biz.txt dvx_GetBiz
2 PG C:PG.csv dvx_GetPG
3 NexT C:NexT.txt dvx_GetNexT
4 Amez C:Amez.csv dvx_GetAmez

The problem is dvx_GetBiz, dvx_GetPG,... all have different number and types of columns. So, I'm not able to create a dynamic Flat File destination depending on the columns returned from the corresponding stored procedure.

Is there a way to accomplish this ?

Pl. help.

Awaiting someones reply ASAP.

Thanks & Regards,
Vidya.




View 4 Replies View Related

Dynamic Name For Flat File Destination - WEEK NO

Apr 17, 2008

Anyone knows how to give the name of a flat file specifying today's week number??

filename_ww
for today it should be filename_16 (since week number is 16)

View 1 Replies View Related

Merging, And Loading Dynamic Flat File.

Mar 14, 2007

I have number of csv files in a folder, all of them with same columns, need to be merged into one table and imported to sql server.

-The first row of the csv file is a header.
-The csv files are updated everyday
-The destination table is replace by new table with new info in the csv.
-The new csv files can be created and old csv files may no longer exist, but we are only interested in information contain in current csv files in the folder.
-I need SSIS to combine all the csv files in the folder and merge into one table.
-Other issue is that the field names may change in csv, so can the SSIS package recognize the change in field name and made necessary change in destination table as well.

Any insight on this issues will be greatly appreciated?

View 1 Replies View Related

Dynamic Columns For Flat File Destination?

Jul 10, 2006

I have a database app, and we're implementing various data export features using SSIS.

Basically, it's a couple of straight extracts of various recordsets/views, etc. to CSV (flat files) from our SQL Server 2005 database, so I'm creating an SSIS package for each of these datasets.

So far, so good, but my problem comes here: My requirements call for users to select from a list of available columns the fields that they want to include in their exported file. Then, the package should run, but only output the columns specified by the user.

Does anyone have any idea as to the best way to accomplish this? To recap, at design time, I know which columns the users will have to choose from, but at run time, they will specify the columns to export to the flat file.

Any help or guidance here is greatly appreciated

View 7 Replies View Related

Dynamic Destination Flat File Loading

Jan 4, 2006

Here's what I want to do -

Dynamically load a flat file from a dynamic source table-

The source table metadata is known via the SYSOBJECTS and SYSCOLUMNS tables- I can pull the column names, type and lengths from these tables based on the table name. (my goal is pretty simple- pull data from a table in our database, and save it down to a flat file)

Would this be enough to dynamically create the destination flat file?  If so, how do I do it?

Thanks

-rob

View 3 Replies View Related

Integration Services :: Flat File With Dynamic Columns

Aug 31, 2015

I will be receiving a CSV daily where columns within the file will change. The column order and number of columns can change daily. I need a way to read in the header from the csv and create a flat file connection that reflects the columns listed in the header.  

Is there an easy way to do this using a script task? I have already read the header into a table but I have been unable to create the dynamic file connection.

View 4 Replies View Related

SSIS-problems Making A Dynamic Flat File Connection String

May 8, 2007

Hello.

I have packages that must generate log errors dynamically including time of execution and the name of the task.

I make it changing the properties of the connection inserting two expressions.

1.-I alter the File Usage Type to 1 to generate this files.
2.- I alter the connection string as: @[User::myvariable] +"constant_description"+ time description+ +".txt"

The time description is :
(DT_STR,40,1252) DAY (GETDATE())+"-"+(DT_STR,40,1252) MONTH( GETDATE())+"-"+(DT_STR,40,1252) YEAR( GETDATE())+" + REPLACE( (DT_STR,10,1252) (DT_DBTIME) GETDATE(),":","_")
But it is not the problem

In those packages I have one connector for all the tasks and in execution time it creates one file for each of the tasks.

The problem is the way I insert in the filename the task name.

I have a pre-execute event handler in each task that modifies a string variable( myvariable) appending the task name. When I execute de package it works great but when I only execute a task, the program do not enter in the event and do not put the tasks name.

How can I put that name without using that handler? There is another handler can I use to do it that happens before the system generates the new file name and after pre-execute? Anyone knows another way to do this kind of things?

View 3 Replies View Related

How To Pass Filename As A Dynamic Variable To Flat File Source In SSIS ?

Sep 3, 2007

Hi,

I am migrating one of my DTS package to SSIS.

My task is to read the filename from a database table and transfer the flat file data in to a table.
In SSIS,I am able to fetch the file name using a Data Reader Source; but how to pass this fileName parameter to Flat File Source ?

In DTS I have used ActiveX script to pass filename variable as flatfilecon.Source.
Any help ?

Thanks,
Ravi

View 4 Replies View Related

Unable To Edit Pre-defined Flat File Connection Manager Properties In The Flat File Destination Editor

Aug 24, 2007

Hi,

I am testing SSIS and have created a Flat File Destination. I defined the Flat File Connection as New for the first time and it worked fine. Now, I would like to go back and modify the Flat File Connection in the Flat File Destination Editor, but it allows only to create a New connection rather allowing me to edit the existing one. For testing, I can go back and create a new connection, but if my connection had 50-100 columns then it would be an issue to re-create it from scratch.

Did someone else faced this issue?


Thanks,
AQ

View 1 Replies View Related

Flat File Connection Manager Throws Error When A Column Gets Added To The Flat File

Dec 27, 2006

Hi,

I have a situation where a tab limited text file is used to populate a sql server table.

The tab limited text file comes from a third party vendor. There are fixed number of columns we need to export to the sql server table. However the third party may add colums in the text file. Whenenver the text file has an added column (which we dont need to import) the build fails since the flat file connection manager does not create the metadata for it again. The problem goes away  where I press the button "Reset Columns" since it builds the metadata then. Since we need to build the tables everyday we cannot automate it using SSIS because the metadata does not change automatically. Is there a way out in SSIS?

View 5 Replies View Related

Output Column Width Not Refected In The Flat File That Is Created Using A Flat File Destination?

May 11, 2006

I am transferring data from an OLEDB source to a Flat File Destination and I want the column width for all of the output columns to 30 (max width amongst the columns selected), but that is not refected in the Fixed Width Flat File that got created. The outputcolumnwidth seems to be the same as the inputcolumnwidth. Is there any other setting that I am possibly missing or is this a possible defect?

Any inputs will be appreciated.

M.Shah

View 3 Replies View Related

How To Redirect The Error Of A Source Flat File To The Destination Flat File?

Nov 10, 2006

Hi all,

I m using SSIS and i am transfering the data from Flat File Source to the OLE DB destination File. The source file contain some corrupt data which i am transfering to the other Flat file destination file.

Debugging is succesful but i am not getting any error output in the Flat file destination file.

i had done exactly which is written in the msdn tutorial of SSIS.

Plz tell me why i am not getting the error output in the destination flat file?

thanx

View 1 Replies View Related

Dynamic Database Selection?

Jun 15, 2007

I'm working on a script to convert data from one software packageto another. Greatly simplified, it looks something likecreate procedure import_widget asbegininsert into our_widget (foo, bar)select baz, quux from their_db.dbo.their_widgetendgoThe problem is that the name of the source database varies fromone system to another, so I want to pass the database name as aparameter. I think I could do the following, but is there abetter way to go about it?create procedure import_widget (@db_name sysname) asbeginexec 'create view their_widget as select * from '+ @db_name + '.dbo.their_widget'insert into our_widget (foo, bar)select baz, quux from their_widgetdrop view their_widgetendgo

View 3 Replies View Related

Dynamic Selection Of DataSource

Feb 6, 2008



Hi,
I need to write a report which will run based on the user's input for Server Group. Meaning, if the user wants the report to be run for one say, UAT group, the report (and therefore the stored procedure) should be executed against UAT server. Or, if the user wants it to be run on the Production machine, the stored procedure should be executed on the Production Server. Assume that both UAT and Production servers are in the same domain to rule out any authentication issue.

I think one way of doing it is to create a linked server on one server which we use to connect via the Report Manager. So, whatever Server Group is selected based on that the SP will be executed on that respective Server Group's Server, since in the query we will have given full names of the tables that we use.

But, I strongly suspect, that having linked servers will not be accepted since it requires that the server owners maintain that. So, I was thinking if there is any other way of achieving this. If anyone knows, please help me with this.

Thanks a lot...

Manoj.

View 3 Replies View Related

Dynamic Member Selection In MDX?

May 28, 2008

I've got a cube in SSAS 2205 with some financial measures that I want to appear conditionally in an MDX query. Here's what I mean...

The cube measures I'm interested in are actual revenue, forecast revenue and budget revenue. The dimension I'm reporting across is the [Time].[month] member. The report I'm trying to create with this query has each month across the top. But the headers show actual revenue if the current reporting month is on or before the month in question. For future months, we show the forcast revenue if there is any. For any other future revenue, we show the budget measures. So a sample report for April would have columns like...



Code SnippetJanuary February March April May June July August September
Actual Actual Actual Actual Forecast Forecast Forecast Budget Budget

I need to conditionally bring back the right measure and have the measure name (e.g., column header) indicate if it's an actual, forecast or budget value.

Thanks.



View 2 Replies View Related

Dynamic Selection Of Column

Nov 19, 2007



Hi,
I have to choose a particular column from a table whenever a condition met. That means i have to use the column in the select query whenever the condition met otherwise i have to create a dummy column with the same column name.
I tried to use the following sample query.
Eg:
Create table temp
(test_id int,
test varchar(100))


Declare @var varchar(100)
Set @Var='Hi'
Select
case when @Var='HI' then "None" else Test_new End as Test
from Temp

But it throw error as "Invalid column 'Test_new' "

As per my situation the Test_new column will be available in temp table whenever the condition is not met.So when @var<>'HI' i have to get the details of the test_new column otherwise i have to create a dummy column with the same name.

View 5 Replies View Related

[SQLDatasource]Dynamic Selection Of Column

Mar 29, 2006

Hey, I have a search form with a selectbox. This selectbox contains the columnnames.I want when I put a text in a textbox and select a value in the selectbox and click submit that it search database.The Columnnames I put in a session.
If you see I have put in the querystring as columnname @sescolumn which I have initialised as asp:sessionparameter.But it gives no results. When I put @sescolumn between [] like normal columnnames are it doesn't work also.
Can someon put my on the right path?

 
<asp:SqlDataSource ID="Database_ecars" runat="server" ConnectionString="<%$ ConnectionStrings:connectionstring %>"
SelectCommand="SELECT [AutoID], [Merk], [Kleur], [Type], [Autotype], [prijs], [Zitplaatsen], [Afbeelding1], [Afbeelding2], [Afbeelding3], [Afbeelding4] FROM [Auto] where @sescolumn like @seskeyword and [AutoID] not in (select [AutoID] from [verhuring] where [StartVerhuur] >= @sesdatefrom and [Eindeverhuur] <= @sesdatetill)" >
<SelectParameters>
<asp:SessionParameter Name="sesdatefrom" SessionField="datefrom" Type="Decimal" />
<asp:SessionParameter Name="sesdatetill" SessionField="datetill" Type="Decimal" />
<asp:SessionParameter Name="seskeyword" SessionField="keyword" Type="string" />
<asp:SessionParameter Name="sescolumn" SessionField="columnname" Type="string" />
</SelectParameters>
</asp:SqlDataSource> 

View 3 Replies View Related

Dynamic Selection Of Multivalued Parameters

May 4, 2007

Hi All,

I Hope someone can help me out with this problem.. its pretty NB.

I have a pair of multivalued parameters. So, parent and child... the child needs to get its selection based on the selection made by the user on the parent parameter ( the child parameter is hidden).

Fictional values:
Parent Parameter: SelectAll, ALL , 1 , 2
Child Parameter: Home, Work, Play

If user selects "ALL" from the parent parameter I need Home and Play selected.

My idea was this:
=iif(Parameters!Parent.Value(1) = "All","Home,Play") <-- in Available values.

However, the dataset that reads this value does not like the coma delimited string.
This dataset is a cube created dataset, reading the child parameter.

Any help is greatly appreciated.
Kind Regards,
Neil

View 1 Replies View Related

Dynamic Column Selection In Stored Procedure

Feb 10, 2005

Hey Guys,

Here is the issue I'm having. I am writing a stored procedure that takes a couple of parameters. Each one is the value within a specific column in one table (i.e., @part = 'o-ring' or @sub_assembly = 'hydraulic ram'). Needless to say, the columns form a hierarchy. What I am trying to achieve is to allow the user to specify one of the parameters and get a count for all records where the specified value is in the corresponding column. So, if the user puts in the parameter @part = 'o-ring', I want it to know that the where clause for the select statement should look for o-ring in the part column and not the sub_assembly column. Here is what I am trying to do, which isn't working.

DECLARE @querycolumn varchar(20),
@queryvalue varchar(35)

SET @querycolumn = ''
SET @queryvalue = ''

IF(@sub_assembly = NULL)
BEGIN
IF(@part = NULL)
BEGIN
PRINT 'This is an error. You must have at least a part'
END
ELSE
BEGIN
SET @querycolumn = 'Part'
SET @queryvalue = @part
END
END
ELSE
BEGIN
SET @querycolumn = 'SubAssembly'
SET @queryvalue = @sub_assembly
END

SELECT SubAssembly, Part, COUNT(RecordID)
FROM Table
WHERE @querycolumn = @queryvalue
GROUP BY SubAssembly, Part
ORDER BY SubAssembly, Part

The problem is that I'm getting an error when I try to use @querycolumn to supply the column name to the WHERE clause. Any ideas or suggestions?

View 14 Replies View Related

SSRS Dynamic Chart Type Selection?

Apr 16, 2008

Does anyone know if there is a way to dynamically set the chart type (i.e. pie, line, column, etc.) for your report in SSRS? I want the customer to be able to chose which type of chart they'd like to see.

I've looked all over the Internet and can't find anything.

Any help would be appreciated.
Alan

Clarification:
I guess I should clarify a little. I am using an ASP.NET 2.0 page written in C# and using a ReportViewer control.

More Clarification:
The ReportViewer displays an rdlc.

View 10 Replies View Related

Converting Flat File To SQL2005 Table (Flat File From H***)

Feb 11, 2008

First, a couple of important bits of information. Until last week, I had never touched SISS, and therefore, I know very little about it. I just never had the need to use it...until now. I was able to convert my first 3 flat files to SQL2005 tables by right clicking on "SISS Package" and choosing "SISS Import and Export Wizard". That is the extent of my knowledge! So please, please, please be patient with me and be as descriptive as possible.

I thought I could attach some sample files to this post, but it doesn't look like I can. I'll just paste the information below in two separate code boxes. The first code box is the flat file specifications and the second one is a sample single line flat file similar to what I'm dealing with (the real flat file is over 2 gigs).

My questions are below the sample files.


Code Snippet
Record Length 400

Positions Length FieldName

Record Type 01
1,2 L=2 Record Type (Always "01")
3,12 L=10 Site Name
13,19 L=7 Account Number
20,29 L=10 Sub Account
30,35 L=6 Balance
36,37 L=1 Active
37,41 L=5 Filler
Record Type 02
1,2 L=2 Record Type (Always "02")
3,4 L=2 State
5,30 L=26 Address
31,41 L=11 Filler
Record Type 03
1,2 L=2 Record Type (Always "03")
3,6 L=4 Coder
7,20 L=14 Locator ID
21,22 L=2 Age
23,41 L=19 Filler
Record Type 04
1,2 L=2 Record Type (Always "04")
3,9 L=7 Process
10,19 L=10 Client
20,26 L=6 DOB
26,41 L=16 Filler
Record Type 05
1,2 L=2 Record Type (Always "05")
3,16 L=14 Guarantor
17,22 L=6 Guar Account
23,23 L=1 Active Guar
**There can be multiple 05 records, one for each Guarantor on the account**


and the single line flat file...



Code Snippet
01Site1 12345 0000098765 Y 02NY1155 12th Street 03ELL 0522071678 29 04TestingSmith,Paul071678 05Smith, Jane 445978N 05Smith, Julie 445989N 05Smith, Jenny 445915N 01Site2 12346 0000098766 N 02MN615 Woodland Ct 04InfoJones,Chris 012001 01Site3 12347 0000098767 Y 02IN89 Jade Street 03OWB 6429051282 25 04Screen New,Katie 879500





As you can see, each entry could have any number of records and multiples of some of the record types, with one exception, every entry must have a "01" record and can only have one "01" record. Oh, and each record has a length of 400.

I need to get this information into a SQL 2005 database so I can create a front end for accessing the data. Originally, I wanted one line for each account and have null values listed for entries that don't have a specific record. Now that I've looked at the data again, that doesn't look like a good idea. I think a better way to do it would be to create 5 different tables, one for each record type. However, records 2 through 5 don't have anything I can make a primary key. So here are my questions...


Is it possible to make 5 tables from this one file, one table for each of the record types?

If so, can I copy the Account number in record 01, position 13-19 in each of the subsequent record types (that way I could link the tables as needed)?

Can this be done using the SISS Import and Export Wizard to create the package? If not, I'm going to need some very basic step by step instructions on how to create the package.

Is SISS the best way to do this conversion or is there another program that would be better to use?
I know this is a huge question and I appreciate the help of anyone who boldly decides to help me! Thank you in advance and I welcome anyone's suggestions!

View 13 Replies View Related

SQL 2012 :: Creating Dynamic SSIS File Format - Dynamic CSV File As Output

Mar 2, 2014

I am trying to create an ssis package with dynamic csv file as output. and out format contains query output.

sample file name:

Unique identifier + query output + systemdate();

The expression is looking like this.

@[User::FilePath] + @[User::FileName] + ".CSV"

-- user filepath is a variable from ssis package. File name is the output from SQL query. using script task i have assigned the values to @[User::FileName] .

When I debugged the script task the value getting properly but same variable am using for Flafile destination. but its not working.

View 3 Replies View Related

How Can I Take This Example Flat File And Parse Out Each Section To A New Flat File? Each Section Starts With HD (header Row)

Mar 13, 2006

How can I take this example Flat file and parse out each section to a new flat file?  Each section starts with HD (header row)

http://www.webfound.net/flat_file_example.txt

e.g. an example output file based on above (cutting out the first section) would be:

http://www.webfound.net/flatfile_output.txt

Also, I'll need to grab a certain value in each header row (certain position in the 100 byte header row) to use that as part of the filename that's outputed.  I assume it would be better to insert these rows into a temp table then somehow do a search on a specific position in the row...but that's impossible?  The other route is to insert each row into a temp table separated out by fields but that is going to be too combursome because we have several formats to determine separation of fields based on the row type so I'd have to create many temp tables and many components in SSIS when all we want to do is again:

1) output each group (broken by each header row) into it's own txt file

2) use a field in  the  header row as part of the name of the output txt file (e.g. look at the first row, whcih is a header row in flat_file_example. txt.  I want to grab the text 'AR10' and use that as part of the filename that I create

Any suggestions on how to approach this whole process in SSIS...the simplest approach that will work ?

View 1 Replies View Related

How To Load Dynamic Multiple Flat Files

Apr 23, 2008



Hi Evry one,

I Have Multiple Flat Files in Source Folder(They have Naming Conventions With Todays Date ex: Flatfile_20082204_1,Flatfile_20082204_2,Flatfile_20082204_3 ),
I need to Extract Each and Evry file by Dynamically, and Transform the Flat File then load that Flat file into the Destination Folder with Standard Prefix and Todays Date with a Sequence No ex:Flatfile_20082304_A,Flatfile_20082304_B, Flatfile_20082304_C


Please HELP Me
Thanks In Advance.

View 20 Replies View Related

Depending On The Drop Down Selection I Need To Show Another RDL File.

Dec 12, 2007

Hi all,

i have a requirement that user only sees drop down list and a button. when he selects an item from drop down and click button then i need to load rdl dynamically depending on the selection from the DDL.

i have created 3 rdl files, each one contains different content. I need to show the user, the selected rdl from the drop down.

is it possible in SSRS?

if you didn't understand well, feel free to ask.

please help me out.

thanks
-Praveen.

View 5 Replies View Related

Dynamic Transformation To Upload Flat Files Into Database

Feb 6, 2007

Hi,

My scenario:

I have 4 different flat files types each having different no. of column, order of columns etc. I want to upload all the 4 types into the same destination table in the SQL database. Before uploading I need to apply transformation to each column in the flat files. The transformations could be like

1) Multipying the source column by 100

2) Put an if condition for 2 source columns and then select one column to be copied into the destination.

I have the flat files schema with me and also all the transformations that are required.

Question:

Can SSIS provide me with a component that can read the flat file schema and the transformations from the database and apply them to the source data and then upload it to the constant destination table? Can derived column transformation be provided with the input columns list and the transformation to be done on each dynamically?

Why I want this way?

In future there can be an addition of extra flat file formats and we want to keep the changes to the SSIS packages to he mininum. Just entereing the addiional schema and transformation details in the database should run the package on the new flat file successfully.

Thanks for your time.

Regards,

$wapnil

View 15 Replies View Related

How To Load SSAS Cube Partitions With Dynamic Name Directly From Flat Files

Feb 1, 2008

HI,

As the source data of the cube is from MySQL, and the source data volume is more than 80M row per month, I have to build multiple partition in the cube, each partition contain only one month data, even though, the time to load data directly from MySQL is still too long, and because the mysql .Net provider and is not mature enough, the connection often break while loading, so I have to try to load from flat file which was exported from MySQL.
The Partition Processing Destination seems support this way, but, even it shows the partition name in component editor, it actually process the partition with partition ID, and there's not any way to change destination partition name for this component.
Unless I have to change the SSIS package every month, looks like it is impossible to make a smart ETL program that can dynamic create new partition with the ID and name as YYYYMM every month and load data directly from the flat csv file exported from big MySQL table.


Does anyone know how to load data from flat text, and also support dynamic destination partition name?

And I also find a bug in Partition Processing Destination, even 2005 SP2, if there's more than one cube in a SSAS database, and if two partition name in different cube is same, in component Editor, you can not set mapping to the second one with the same name, even you point to the second one, and click ok, the next time you open the editor, you will find it high light at the first one. And even it shows name in editor, it actually process with partition ID instead of partition name, this make it is not possible by change the partition name which need to process to a constant name, say currentMonth to force the component process it.

Thanks.

Jun

View 4 Replies View Related

Integration Services :: Flat File Error File Being Created In-spite Of No Errors

Jun 23, 2015

I have a package in which there are only one Data flow Task and it has only three components. 1) Source , which is a SQL db 2) destination and 3) OLE DB Destination flat file Error output file.   I want the error file to be created ONLY if there is any error while dumping the data into destination DB. But , the issue is, the error flat file is being created inspite of No error while dumping the  data from Source to Destination.

View 5 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved