What I am trying to do is move data from a staging table into a live environment and then update the staging table AFTER the row has move (and not errored). There does not seem to be a reliable method for doing this.
Any assistance would be appreciated.
Hi,
A long long time ago I asked for the ability to define an order in which you insert data into multiple destinations in a single data-flow. This was to get around the problem of loading tables in the same data-flow that have FK relationships between them.
My chosen alternative at the moment is to load the table with the unique key first and drop the data for the table with the FK into a raw file and load it in a seperate data-flow. Another alternative is to disable FK constraints and reenable after but I've chosen the raw file method.
Now that I'm using SSIS on a real project this is becoming a much bigger problem that I ever envisaged it might be. Its rare that I'm building stuff where I don't have to use raw files for this very reason and this means that I have god knows how many raw files hanging around all over the place.
I suppose the reason for this post is to flag the importance of this requested feature. I really hope that this makes it into Katmai. Or into a service pack would be even better!
Its a bit of a failing at the moment.
Are there any chances that this will make it into Katmai? its one of my biggest bug-bears about SSIS v1.
-Jamie
[SSIS Team Followup]
Hi All,
I wanna know if we can have more than one "OLEDB Destination" within a Data Flow Task, I want to use the same data flow and write to two different tables in a database with some changes. If we cannot do this within the same data flow what is the best way to do this.
Thanks
Is there a way to programatically set (using expressions, variables) the FastLoadMaxInsertCommitSize property of an OLEDB destination in a data flow for Fast Load Operations.
Basically, what I want to do is based on the # of records which are going to be inserted want to set the FastLoadMaxInsertCommit size.
Hi,
The further i get with doing my current SSIS package the more i am starting to wonder about best practices and performance.
My current package loops through CSV files in a specified location and extracts events from these files. Each file contains multiple events which are a mixture of different types. Depending on the event there are a different number of comma seperated values. In the package i firstly set each event to one column seperated by a comma delimeter. I then create an array for the event which is split by the delimeter. In a script i weed out all elements of the array that are common to all events and set the remaining events to another array. After some processing i come to my conditional split transformation which splits the processing of each event based on the EventID. This is where i'm having doubts on whether i have approched the package correctly. There are approximately 60 different events so each one of these has a seperate pipeline to process the remaining parameters in the array and output them to the destination table. The destination table is differnet for each ID. Is it viable to have this amount conditions and paths when creating the pacakge and is this likely to have any detrimental effect on performance. Is there possibly another way that i could approach this problem?
Many thanks, i hope that made sense.
Grant
When I set up a Flat File, Excel, or XML source, I have to specify the complete file name, in particular the folder where the file exists. I would like to specify the location dynamically, via a variable or property -- but how?
View 4 Replies View RelatedIt looks like you can only extended connection managers for data flow sources.
Is there anyway to develop a custom connection manager for the SQL destination in the data flow destinations?
I can€™t use hardcoded connection strings. I can€™t use configurations because they are not encrypted. I already have managed code that will give the corrected connection string. I already have a custom connection manager that I use from the data flow sources. I just need one for data flow destinations but I can€™t see a way to extend into the OLE DB?
I have a package that used to work fine, but after I imported it into a different existing solution and tried to run it, it always stalls during the data flow task at the very end.
It will run through and process all the rows and insert them into the destination except for the last chunk. The task blocks eventually all turn green as well, but then it never proceeds to the next task after the data flow. Looking at the Progress tab for the data flow task, I get the following:
Progress: Pre-Execute - 100 percent complete
[DTS.Pipeline] Information: Execute phase is beginning.
[DTS.Pipeline] Information: Post Execute phase is beginning.
Progress: Post Execute - 0 percent complete
Then it just stays at 0 percent for Post Execute. The program doesn't hang or anything, it just doesn't progress at all. I also just checked and I run the package from its original solution and it is now exhibiting the same behavior as well.
Any ideas what might be causing this or something to do to try and figure it out or at least get more information about what might be causing it?
Thanks.
I have a data flow that reads multiple rows from a table and then inserts to another table for each row. I use an ole destination for my inserts. However, after that insert I need to do other table inserts and I can't figure out how to continue the data flow with the fields in the pipeline. Out of the destination is only the Error flow - Is there a way to do this ?
thx
Does anyone have a helpful link for using the partition processing data
flow task in SSIS? I am trying to process a monthly partition
from within my package and am getting the following error:
Error: 0xC113000A Errors in the high-level relational engine. Pipeline
processing can only reference a single table in the data source view.
If anyone has used this before and could point me in the right direction, I would appreciate it.
Thanks,
Nick
Hello,
Is it possible to use existing data flow components (Merge Join, aggregation,...) in a custom data flow component?
Thanks,
Yoann
During the execution of SSIS Package which is populating huge data into OLEDB Destination from OLEDB Source, then some of the records are getting rejected. Again if we are executing it twice or thrice, the rejected records are getting inserted.
Wants to know, why the records are getting rejected? Target table does not contain any constraints.
Hope to hear soon!!!!!!!!!!!!!!!!!!
Hi, All,
I need to pass a parameter from control flow to data flow. The data flow will use this parameter to get data from a Oracle source.
I have an Execute SQL task in control flow to assign value to the Parameter, next step is a data flow which will need take a parameter in the SQL statement to query the Oracle source,
The SQL Looks like this:
select * from ccst_acctsys_account
where to_char(LAST_MODIFIED_DATE, 'YYYYMMDD') >?
THe problem is the OLE DB source Edit doesn€™t have anything for mapping parameter.
Thanks in Advance
I have an Execute SQL Task that returns a Full Rowset from a SQL Server table and assigns it to a variable objRecs. I connect that to a foreach container with an ADO enumerator using objRecs variable and Rows in first table mode. I defined variables and mapped them to the columns.
I tested this by placing a Script task inside the foreach container and displaying the variables in a messagebox.
Now, for each row, I want to write a record to an MS Access table and then update a column back in the original SQL Server table where I retreived data in the Execute SQL task (i have the primary key). If I drop a Data Flow Task inside my foreach container, how do I pass the variables as input to an OLE DB Destination on the Data Flow?
Also, how would I update the original source table where source.id = objRects.id?
Thank you for your assistance. I have spent the day trying to figure this out (and thought it would be simple), but I am just not getting SSIS. Sorry if this has been covered.
Thanks,
Steve
Dear All!
My package has a Data Flow Task. In Data Flow Task, I use a Script Component and a OLE BD Destination to transform data from txt file to database.
Within Data Flow Task, I want to call File System Task to move file to a folder or any Task of "Control Flow" Tab. So, Does SSIS support this task? Please show me if any
Thanks
Hi everyone,
Primary platform is 64 bit cluster.
How to move information allocated in SSIS variables from Data Flow to Control Flow layers??
We've got a SSIS package which load a value into a variable inside a Data Flow. Going back to Control Flow how could we retrive that value again????
Thanks in advance and regards,
I'm currently setting variables at the package level with an ExecuteSQL task. This works fine. However, I'm now starting to think about restartability midway through a package. It would be nice to have the variable(s) needed in a data flow set within the data flow so that I only have to restart that task.
Is there a way to do that using an SQL statement as the source of the value in a data flow?
OR, when using checkpoints will it save variable settings so that they are available when the package is restarted? This would make my issue a moot point.
Hi all! I recently started working with SSIS and one of the things that is puzzling me the most is what's the best way to go:
A small control flow, with large data flow tasks
A control flow with more, but smaller, data flow tasksAny help will be greatly appreciated.
Thanks,
Ricardo
Hi,
I'm trying to implement an incremental data pull (Oracle to SQL) based on Andy's blog:
http://sqlblog.com/blogs/andy_leonard/archive/2007/07/09/ssis-design-pattern-incremental-loads.aspx
My development machine is decent: 1.86 GHz, Intel core 2 CPU, 3 GB of RAM.
However it seems the data flow task gets hung whenever I test the package against the ~6 million row source, as can be seen from these screenshots. I have no memory limitations on the lookup transformation. After the rows have been cached nothing happens. Memory for the dtsdebug process hovers around 1.8 GB and it uses 1-6 percent of CPU resources continuously. I am not using fast load to insert new records into my sql target table. (I am right clicking Sequence Container 3 and executing this container NOT the entire package in the screenshots)
http://i248.photobucket.com/albums/gg168/boston_sql92/1.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/2.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/3.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/4.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/5.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/6.jpg
The same package works fine against a similar test table with 150k rows.
http://i248.photobucket.com/albums/gg168/boston_sql92/7.jpg
http://i248.photobucket.com/albums/gg168/boston_sql92/8.jpg
The weird thing is it only takes 24 minutes for a full refresh of the entire source table from Oracle to the SQL target table.
Any hints,advice would be appreciated.
---------------------------------------------------------------
My Original Post
I have to query n table(NLRImports) using the Distinct keyword, to retrieve a set of ID numbers. ( "Select DISTINCT id_nbr from NLRImport" ).
Now i want to use those values i retrieved, to process the records in the table(NLRImports) 1 by 1. How do i use those ID no's i retrieved as Variables or parameters for my next query?? If this makes sense?
----------------------------------------------------------------
First, thanks for the response.... now here is what im trying to do.
I created a simple application in delphi to import information to a table in MSSql2005. This is some of the resulting columns...
date | id_nbr | account_nbr | sub_account_nbr | ... etc
-------------------------------------------------------------
Now there will be several entries with the same id no but on different dates, so i take it dates would rather be my pkey.
Then i need to take one person's entries(i work on id_nbr) and go thru all the entries taking the earliest date and comparing all the other entries for that person to the first date and select all the dates more than 19 days after the first date and less than 91 days from first date and place it in a new table.
I used cursor s and while loops to kind of get it going but i know that cursors are not really recommended use but the performance implications dont bother with this particular job.
What other ways should i be using to accomplish this?
thanks, i hope this is clear...
Hi, all here,
Thank you very much for your kind attention.
I am wondering if it is possible to use SSIS to sample data set to training set and test set directly to my data mining models without saving them somewhere as occupying too much space? Really need guidance for that.
Thank you very much in advance for any help.
With best regards,
Yours sincerely,
Good morning, all,
I am working on importing an Excel workbook, saved as multiple CSV flat files, that has both group level data and related detail row on the same sheet. I have been able to import the group data into a table. As part of the Data Flow task, I want to be able to save the key value for the group, which I will use when I insert the detail rows.
My Data Flow has the following components: The flat file with the data, which goes to a derived column transformation to strip out extraneous dashes, which leads to the OLEDB Destination component.
I want to save the value as a package level variable, so that I can reference it in another dataflow.
Is this possible, and if so, at what point do I save the value?
Thanks,
Kathryn
Hi,I'm new at asp .net and have a problem. I have a page where the logged in user can add another user to his list of friends. To do this, I have a page where all the users are listed by using a datalist, and I have a form with an invisible field which passes the value of <%#DataBinder.Eval(Container.DataItem, "UserName")%> in post when the user clicks on "add as a friend" so that in the next page I can get that value and add it as well as the username of the logged in user to the table friendships. This is not working, can anyone help please? The code of the page with the datalist is:<script language="VB" runat="server"> Protected Sub Page_PreInit(ByVal sender As Object, ByVal e As System.EventArgs) ' Session("username") = User.Identity.Name Dim CurrentUser As String CurrentUser = Membership.GetUser.ProviderUserKey.ToString() End Sub Sub Page_Load(ByVal Sender As Object, ByVal E As EventArgs) Dim DS As DataSet Dim MyConnection As SqlConnection Dim MyCommand As SqlDataAdapter Dim CurrentUser As String CurrentUser = Membership.GetUser.ProviderUserKey.ToString() MyConnection = New SqlConnection("Data Source=.SQLEXPRESS;AttachDbFilename=|DataDirectory|ASPNETDB.MDF;Integrated Security=True;User Instance=True") MyCommand = New SqlDataAdapter("select u.*,f.buddyId as userNameIfFriend from aspnet_Users u left outer join aspnet_friendship f ON u.UserId=f.buddyId AND f.userId=@Param1 where IsAnonymous='False' and u.UserId<>@Param1", MyConnection) 'Data Source=.SQLEXPRESS;AttachDbFilename=|DataDirectory|;Integrated Security=True;User Instance=True MyCommand.SelectCommand.Parameters.AddWithValue("@Param1", CurrentUser) DS = New DataSet() MyCommand.Fill(DS, "aspnet_Users") MyDataList.DataSource = DS.Tables("aspnet_Users").DefaultView MyDataList.DataBind() End Sub</script><body> <br /> <ASP:DataList id="MyDataList" RepeatColumns="1" runat="server"> <ItemTemplate> <table cellpadding="10" style="font: 10pt verdana" cellspacing="0"> <tr> <td width="1" bgcolor="BD8672"/> <td valign="top"> </td> <td valign="top"> <div id="hey"><div id="dados"><b>Name: </b><%#DataBinder.Eval(Container.DataItem, "UserName")%><br><b>city: </b><%#DataBinder.Eval(Container.DataItem, "City")%><br></div> <div id="photo"><b>Photo: </b><%#DataBinder.Eval(Container.DataItem, "UserName")%><br>status: <form id="teste" name="teste" method="post" action="add_buddy.aspx"> <input name="UserId" type="hidden" id="UserId" value="<%#DataBinder.Eval(Container.DataItem, "UserName")%>" /> </form> <%#IIf(Container.DataItem("userNameIfFriend") Is DBNull.Value, "<a href='add_buddy.aspx'>Add as buddy</a> ", "Already buddy")%> </div> <p></div> <a href='<%# DataBinder.Eval(Container.DataItem, "UserName", "purchase.aspx?titleid={0}") %>' > </a> </td> </tr> </table> </ItemTemplate> </ASP:DataList></body></asp:Content>The code of the page where the user is redirected to add a friend and where I want to make the insert is: <script runat="server"> Protected Sub Page_PreInit(ByVal sender As Object, ByVal e As System.EventArgs) Dim CurrentUser As String CurrentUser = Membership.GetUser.ProviderUserKey.ToString() End Sub Sub Page_Load(ByVal Sender As Object, ByVal E As EventArgs) Dim CurrentUser As String CurrentUser = Membership.GetUser.ProviderUserKey.ToString() ' Declaring variables Dim UserId As String ' A Function to check if some field entered by user is empty ' Receiving values from Form UserId = (Request.Form("UserId")) Dim MyConnection As SqlConnection Dim MyCommand As SqlDataAdapter MyConnection = New SqlConnection("Data Source=.SQLEXPRESS;AttachDbFilename=|DataDirectory|ASPNETDB.MDF;Integrated Security=True;User Instance=True") MyCommand = New SqlDataAdapter("insert @Param2, @Param1 into aspnet_friendship.userId, aspnet_friendship.buddyId", MyConnection) MyCommand.SelectCommand.Parameters.AddWithValue("@Param1", CurrentUser) MyCommand.SelectCommand.Parameters.AddWithValue("@Param2", UserId) ' Creating Connection Object and opening the database Dim DS As DataSet DS = New DataSet() MyCommand.Fill(DS, "aspnet_friendship") ' Done. Close the connection End Sub </script> I am not sure if sending the information in a form in post to another page and executing the query there is the best option or if I can do it all within
the if statement. I appreciate any help, since I am having difficulties in solving this.Thank you.
I have 3x6 textbox lay out, QTY, Descp, Cost, with submit button with followin code...., insert function has too many function, anyone know how to work thiis type of data entery?? Thanks in advanc.Dim Quntityx() As Integer = {txtQ1.Text, txtQ2.Text, txtQ3.Text, txtQ4.Text, txtQ5.Text, txtQ6.Text}
Dim xDex() As String = {txtD1.Text, txtD2.Text, txtD3.Text, txtD4.Text, txtD5.Text, txtD6.Text}Dim xcost() As Decimal = {txtC1.Text, txtC2.Text, txtC3.Text, txtC4.Text, txtC5.Text, txtC6.Text}
Dim i As IntegerFor i = 0 To 5
itemInput.InsertParameters.Add("RequestN", RequestN)itemInput.InsertParameters.Add("Description", xDex(i))
itemInput.InsertParameters.Add("Cost", xcost(i))itemInput.InsertParameters.Add("Quantity", Quntityx(i))
itemInput.Insert(i)
Next i
I need to update the string in a longtext field. NewString = "This is a 'test'"
I can't figure out how to use an escape character to identify the string with " ' " and then also include the " ' " character inside the string.
Can anybody help me?
Thanks,
Alex
Hi,
Yesterday I had posted a query of How to transfer the Data from AS400 files to Sql 2005 tables with DTS and the error I received for the same. I am not able to view the replies since it has been deleted. Pls repost the same. I posted the query expecting for a solution but instead my question was deleted..... the very purpose of forum is defeated.
Regds,
Anu
I have a table that is returning rows from a table query. It seems I have done it before but I cannot seem to get the right procedure to obtain the values. I will paste in the code below in which you will see my bad attempts at accomplishing what I need.
Dim uid As String
Dim pw As String
Dim em As String, fn, ln, mi As String
Dim par As String
Dim Field, n, j As Integer
Dim JJ As Integer
[code]...
Hello everyone. I need help regarding the following:Given the following table:CREATE TABLE T1 (C1 nvarchar(10), C2 money)INSERT INTO T1 VALUES ('A',1)INSERT INTO T1 VALUES ('B',2)INSERT INTO T1 VALUES ('C',3)let's say that i have this table in a local server and i want to uploadit to a remote server and in the remote server upload it to a databasethat contains the same table.the uploading part can be done by another application in the remoteserver, but i want i need is a way to transfer the data at the fastestpossible way.what steps do i need to follow?tia,Rey Guerrero
View 10 Replies View RelatedHi everyone,
I have to extract, dayly a list of contacts on a exchange server in a table on our EDW on sql server 2005. Is it possible to get the information directly from a dataflow or i will have to developpe a script task ?
Need help desperatly !!!
Hello,
I have noticed that for one of my data-flows, the process is really long during the phase "the final commit data insertion has started".
To be accurate, the process is fast until it reaches this phase. It happens often when I load millions of lines.
The extraction is done from a database SQL Server 2005 to a database SQL Server 2005, on the same server (with the SQL Server native provider).
I used a SQL Server destination but I have tried with an OLE DB destination and it is the same situation.
Why the process could be so long during this phase?
There is a way to optimised my package to avoid that?
Any idea is welcome.
Thanks.
Guillaume
Hi All,
I want to export data from SQL Server2005 to an Excel spreadsheet thru "Data Flow Task". I am using OLE DB for SQL Server for the source connection and a Connection To Excel as my destination source. The Excel spreadsheet (2003) exists and has the first row with column names. I don't have any warnings before trying to execute.
The SQL datable fileds are
i) ID - Int
ii) RefID
iii) txtRemarks - nvarchar(MAX)
iv) ddlWaterLevel - nvarchar(50)
While executing the tasks, I got the error
Error: 0xC0202025 at Data Flow Task, Excel Destination [427]: Cannot create an OLE DB accessor. Verify that the column metadata is valid.
Error: 0xC004701A at Data Flow Task, DTS.Pipeline: component "Excel Destination" (427) failed the pre-execute phase and returned error code 0xC0202025.
After analysing I found in the DataFlow --> Excel destination --> Advanced Editor for Excel Destination, the default data type for txtRemarks shows as "Unicode string [DT_WSTR]". But this is supposed to be "Unicode text stream [DT_NTEXT]". Even if I change the data type in the design time, It doesn't accept.
Please do help me out.
thanks
Sanra
I need to see inside a SSIS 2012 project a new SSIS installed component, but in the SSDT 2010 I cannot see the SSIS Data Flow Items tab for adding data source/data destination respect to the choose toolbox items pane.
View 4 Replies View RelatedHi, all experts here,
Do we always have to use SCD component for the loading of data into data warehouse to handle changes of rows?
I am looking forward to hearing from you and thank you very much in advance for your help.
With best regards,