Passing A Table Programmatically To Datareader Source Component In Ssis
Feb 26, 2007
Hi There,
I am loading/executing packages from c# and I need to populate a temp table from user input and pass this table as a variable to the datareader source components sql command. I am using expression to build this query, but I am getting design time error when I have this command..
"select id, (SysDate + 28) as ExpiresDate from Table1 where id in (Select Id from" +@[User::Table2]+")"..
I have declared Table2 as a variable of type Object and I am creating Table2 in C# and I am assigning that Table to the user Table. But in the design mode, I am getting an error...expression cannot be evaluated.
I have a package that has a data lfow task. this task imports data from a db2 database (using the IBM Ole DB provider fro db2) and adds it to sql server database table. This package was created on the server. then though version control (using TFS source control) I check out the package on my local machine. and when I open the package I get the foll 3 errors.
Error 1 Validation error. Import Account Num from BMGP_BDR: DTS.Pipeline: The component metadata for "component "DataReader Source" (1113)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
Error 2 Error loading BMAG Download Xref Tables - bmag.dtsx: Microsoft.SqlServer.Dts.Pipeline.ComponentVersionMismatchException: The version of component "DataReader Source" (1113) is not compatible with this version of the DataFlow. [[The version or pipeline version or both for the specified component is higher than the current version. This package was probably created on a new version of DTS or the component than is installed on the current PC.]] at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostCheckAndPerformUpgrade(IDTSManagedComponentWrapper90 wrapper, Int32 lPipelineVersion)
Error 3 Error loading BMAG Download Xref Tables - bmag.dtsx: The component metadata for "component "DataReader Source" (1113)" could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
I have a package which reads an Access file from a folder. My connection manager to this file is .NET providers for OledbMicrosoft Jet 4.0 OLE DB Provider.
Package works from my computer. But when I execute it on the server as a SQL Agent job, I get
The component metadata for "component "DataReader Source" (1) could not be upgraded to the newer version of the component. The PerformUpgrade method failed.
I copied the mdb file to a folder on the server which my packages have no problem reading data from.
My packages run under the same domain account as defined in proxies.
Attempting to create a data flow task to copy data from AS/400 (DB2) to SQL2005, using an existing System DSN ODBC connection defined on the SQL2005 host.
Problem:
When adding the DataReader Source component to the package, I cannot assign the Connection Manager. Designer issues the error message:
"The runtime connection manager with the ID "" cannot be found. Verify that the connection manager collection has a connection manager with that ID."
Editing the DataReaderSrc component shows only one row under the Connection Managers tab:
The datareadersrc component editor displays the warning message: "Not all connection managers have been set. Set all connection managers.". Clicking the Refresh button causes the error message to be displayed "The runtime connection manager with the ID "" cannot be found. Verify that the connection manager collection has a connection manager with that ID."
I am prevented from assigning my Connection Manager object the DataReaderSrc.
The package already contains one Connect Manager object:
Provider: .Net Providers/Odbc Data Provider System DSN
I have created an SSIS On my PC, That extracts data out of Lotusnotes. When I run it manaully from Visual studion-execute package, it works just fine. However, when I go tO Management studio, chose the servername, then go to Stored Package-MSDB- And chose Packagename, and hit Run PKG from there, it gives me the error below. Server has 2005 AND IS WINDOWS 2003 OS.
The product level is insufficient for component "datareader source"
Can you tell me why the same pkg will work fine when executed from Visual studio-But when I go the run it from Storedpackages-MSDB-it fails.
Does anyone know how to create a Source Script Component programmatically. I can only seem to create a Transformation Script Component. I have this:
PipeLineWrapper.IDTSComponentMetaData90 sourceComponent = ((dataflowTask as TaskHost).InnerObject as PipeLineWrapper.MainPipe).ComponentMetaDataCollection.New();
dear experts, i'm trying to build a package programmatically from client c# application. I'm working to create three dataflow components: - OleDB Source - Derived Column Transformations - OleDb Destination.
My package works good, but i have several problems when insert an expression as Value of an IDTSCustomPropriety90 object, like this one: [LEN](#firsname.lineageID) > 5 ? <<if true>> : <<if false>>
Simple expressions like concatenate strings, for example "#firstname.lineageID" + "#lastname.lineageID" seem to work good.
I have an integration services script that was working ok until a recent database upgrade. When I run the script in gui debug mode it behaves as follows. The data flow reads in data from an oracle server table VALNREQ & then uses it to populate a sqlserver table (with a little manipulation of the fields in between). For some reason although it populates the destination table with the correct number of rows (& the data looks ok) it errors out with the messages listed below. The DataReader Source dataflow source box turns red & all the other boxes turn green including the destination one. I have run out of ideas ... any suggestions? Have I inadvertently changed a property without noticing? I can't see anything obvious & the input data itself looks clean enough.
Any pointers in the right direction would be great, thanks. I've been thrown in at the deep end with this s/w so I imagine there are a number of large gaps in my knowledge so apologies if there is a simple solution to this.
B
[DataReader Source VALNREQ [1]] Error: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper90 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer90[] buffers, IntPtr ppBufferWirePacket)
[DTS.Pipeline] Error: The PrimeOutput method on component "DataReader Source VALNREQ" (1) returned error code 0x80004003. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.
I need to create an ODBC source script component that outputs into SQL Server. When I debug I get the following error message:
Error at Data Flow Task [Script Component [1]]: System.InvalidCastException: Unable to cast object of type 'System.Data.Odbc.OdbcConnection' to type 'System.Data.SqlClient.SqlConnection'. at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.HandleUserException(Exception e) at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.AcquireConnections(Object transaction) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostAcquireConnections(IDTSManagedComponentWrapper90 wrapper, Object transaction)Error at Data Flow Task [DTS.Pipeline]: component "Script Component" (1) failed validation and returned error code 0x80004002.
Public Class ScriptMain Inherits UserComponent Dim connMgr As IDTSConnectionManager90 Dim sqlConn As SqlConnection Dim sqlReader As SqlDataReader
Public Overrides Sub AcquireConnections(ByVal Transaction As Object) connMgr = Me.Connections.PP sqlConn = CType(connMgr.AcquireConnection(Nothing), SqlConnection) End Sub
Public Overrides Sub PreExecute() Dim cmd As New SqlCommand("SELECT Solution_Code_From, Solution_Code_To FROM Solconv", sqlConn) sqlReader = cmd.ExecuteReader End Sub
Public Overrides Sub CreateNewOutputRows() Do While sqlReader.Read With SolutionOutputBuffer .AddRow() .solcodefr = sqlReader.GetString(1) .solcodeto = sqlReader.GetString(0) End With Loop End Sub
Public Overrides Sub PostExecute() sqlReader.Close() End Sub
Public Overrides Sub ReleaseConnections() connMgr.ReleaseConnection(sqlConn) End Sub
as far as I know from docs and forum datareader is for .NET data in memory. So if a use a complex dataflow to build up some data and want to use this in other dataflow componens - could i use data datareader source in the fist dataflow and then use a datareader souce in the second dataflow do read the inmemoty data from fist transform to do fursther cals ?
how to pass in memory data from one dataflow to the next one (i do not want to rebuild the logic in each dataflow to build up data data ?
Is there a way to do this ? and is the datareader the proper component ? (because its the one and only inmemory i guess, utherwise i need to write to temp table and read from temp table in next step) (I have only found examples fro .NET VB or C# programms to read a datareader, but how to do this in SSIS directly in the next dataflow?
At our business we are getting a lot of PDF documents that are being hand keyed into a database. Has anyone heard ior know of a SSIS Data Flow Source component that I coud use to read thos documents into a data stream (?) and process?
we need download files from FTP location , so we are using script component as source and when we are using the fallowing code// read a file and write out as columns in rows
WebClient myWebClient = new WebClient(); myWebClient.Credentials = new NetworkCredential(Variables.ftpLogin, Variables.ftpPassword);
// Concatenate the domain with the Web resource filename. string myStringWebResource = Variables.fileURL + Variables.fileName;
string s = myWebClient.DownloadString(myStringWebResource); StringReader reader = new StringReader(s);
we are getting the fallowing error The server returned an error: (404) Not Found. are we missing anything in the code.
I have implemented a custom source component that can be used as the data source in the Data Flow task.
I have also created a custom UI for this component by using the IDtsComponentUI .
But my component does not have the capability of setting the custom properties via the DTS Variables using the Expression Builder.
I have looked around for samples on how to do this, but I can only find samples of how to do this for custom Control Tasks, i.e. IDtsTaskUI.
My question is, How can implement the Expression Builder in my custom Source component + custom Source UI. Or do you know of any samples which I can look at.
I am getting the following exception when attempting to read from a DataReaderDestination:
System.Exception was unhandled Message="Could not obtain a DataReader object from the specified data flow component." Source="Microsoft.SqlServer.Dts.DtsClient" StackTrace: at Microsoft.SqlServer.Dts.DtsClient.DtsCommand.internalPrepare(Boolean fReaderRequired) at Microsoft.SqlServer.Dts.DtsClient.DtsCommand.ExecuteReaderInThread() at Microsoft.SqlServer.Dts.DtsClient.DtsCommand.ExecuteReader(CommandBehavior behavior) at CA3DataImportTool.ViewSSISOutput.btnRun_Click(Object sender, EventArgs e) in C:Documents and Settings heinMy DocumentsVisual Studio 2005ProjectsCA3DataImportToolCA3DataImportToolViewSSISOutput.cs:line 35 at System.Windows.Forms.Control.OnClick(EventArgs e) at System.Windows.Forms.Button.OnClick(EventArgs e) at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent) at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks) at System.Windows.Forms.Control.WndProc(Message& m) at System.Windows.Forms.ButtonBase.WndProc(Message& m) at System.Windows.Forms.Button.WndProc(Message& m) at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m) at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m) at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam) at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg) at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32 dwComponentID, Int32 reason, Int32 pvLoopData) at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context) at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context) at System.Windows.Forms.Application.Run(Form mainForm) at CA3DataImportTool.Program.Main() in C:Documents and Settings heinMy DocumentsVisual Studio 2005ProjectsCA3DataImportToolCA3DataImportToolProgram.cs:line 18 at System.AppDomain.nExecuteAssembly(Assembly assembly, String[] args) at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args) at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()
If I use the example from SQL Server BOL (http://msdn2.microsoft.com/en-us/library/ms135917.aspx), and make a new package for the sample in the same project, the sample works. The only thing that I can see that is significantly different between my code and the sample is that my DataReaderDestination has a lot more data in it, but here's the relevant code:
string dtexecArgs;
string dataReaderName;
DtsConnection dtsConnection;
DtsCommand dtsCommand; //IDbCommand
IDataReader dtsDataReader;
DataTable dtsTable;
dtexecArgs = @"/FILE ""C:Documents and Settings heinMy DocumentsVisual Studio 2005ProjectsCA3DataImportToolML3000_IntegrationProjectPackage.dtsx"" ";
dataReaderName = "DataReaderDest";
dtsConnection = new DtsConnection();
dtsConnection.ConnectionString = dtexecArgs;
dtsConnection.Open();
dtsCommand = new DtsCommand(dtsConnection);
dtsCommand.CommandText = dataReaderName;
dtsDataReader = dtsCommand.ExecuteReader(CommandBehavior.Default); // EXCEPTION HERE
64 bit SQL 2005 running on Windows Server 2003 X64
I have an exececute SQL task (in the control flow obviously)
SELECT MAX(last_update) AS OrdersLastUpdateFROM orders
This task executes successfully and I can see that my user variable called "User:tmOrdersLastUpdate" populates correctly in the "variables" pane.. ALL GOOD.
The next step of the Control flow is a dataflow task
Details
DataFlow Source = DataReader Source (MySQL .NET connector) DataFlow Dest = local SQL Server OLE DB.
In the DataFlow Source the DataReader SQLCommand property is Select * from orders where last_update >= @User:tmOrdersLastUpdate
I've tried every conceivable permutation and I can't get SSIS to itnerpret the variable as such...it always gets passed to the server as a literal.
How do I pass a user-defined global variable to the WHERE clause in a DataRader object?
Hi, I have a datareader component of which i am dynamically setting its sqlcommand statement with expression (click the background of dataflow > properties > expressions). Now my sql select statement has about 600 fields so that makes my expression statment "select field1, field2, .....from table1 where field2 >=" + @[User::dateforfield2] but when i evalute the expresssion (which is right), i get the error: A string literal in the expression exceeds the maximum allowed length of 4000 character and i think its because of the fields in my select statment causing my string literal to grow more than 4000 characters. Is there any way to increase the max string literal for expressions. Please help.
I have created a package which simply imports data from a flat file to a SQL Server table. But I need to incorporate a data conversion component by which I may change the source-destination column mapping programmatically. So what I thought that I need to add a data conversion component into the dataflow task. After adding this component (I found its component id as {C3BF62C8-7C5C-4F85-83C3-E0B6F6BE267C}) I have created a path which establishes the mapping between output columns of source component and the input columns of data conversion component. Now I am not sure how to establish the mapping between the data conversion component€™s input column collection and output column collection.
I am giving my code snippet here, IDTSComponentMetaData90 sourceDataFlowComponent = dataFlowTask.ComponentMetaDataCollection.New(); sourceDataFlowComponent.ComponentClassID = "{90C7770B-DE7C-435E-880E-E718C92C0573}"; €¦ €¦ €¦. // Code for configuring the source data flow component
IDTSComponentMetaData90 conversionDataFlowComponent = dataFlowTask.ComponentMetaDataCollection.New();// creating data conversion conversionDataFlowComponent.ComponentClassID = "{C3BF62C8-7C5C-4F85-83C3-E0B6F6BE267C}";// This is the GUID for data conversion component CManagedComponentWrapper conversionInstance = conversionDataFlowComponent.Instantiate();//Instantiate conversionInstance.ProvideComponentProperties(); // Now creating a path to connet the source and conversion IDTSPath90 fPath = dataFlowTask.PathCollection.New(); fPath.AttachPathAndPropagateNotifications( sourceDataFlowComponent.OutputCollection[0], conversionDataFlowComponent.InputCollection[0]); // Sould I need to accuire connect for data conversion? Im not sure conversionInstance.AcquireConnections(null); conversionInstance.ReinitializeMetaData(); // Get the input collection IDTSInput90 input = conversionDataFlowComponent.InputCollection[0]; IDTSVirtualInput90 vInput = input.GetVirtualInput(); foreach (IDTSVirtualInputColumn90 vColumn in vInput.VirtualInputColumnCollection){ conversionInstance.SetUsageType( input.ID, vInput, vColumn.LineageID, DTSUsageType.UT_READONLY); }
.. . // Well here I am stucked. What I need to do here to establish a map // between conversionDataFlowComponent.InputCollection[0] and // conversionDataFlowComponent.OutputCollection[0]?
As you can see I am just away from creating the mapping between input and output collection. Can anybody give me an idea how can I achieve this?
I will appreciate all kind of suggestions and comments.
Does anyone have any examples of programmatically creating a Transformation Script Component (or Source/Destination) in the dataflow? I have been able to create other Transforms for the dataflow like Derived Column, Sort, etc. but for some reason the Script Component doesn't seem to work the same way.
I have done it as below trying many ways to get the componentClassId including the AssemblyQualifiedname & the GUID as well. No matter, what I do, when it hits the ProvideComponentProperties, it get Exception from HRESULT: 0xC0048021
HI everyone; i have a problem , is there a possible to pass a dataset to a ssrs report programmatically ? or at runtime without specifying it at design time ...
when I configurate the datareader source using the ODBC connection manager. it show the follow error message: "Error at Data Flow Task[DataReader Source [562]]: Cannot acquire a managed connection from the run-time connection manager" this ODBC is connect to IBM DB2.
Hello, I have a DataReader souce configured to an ADO.NET connection manager that uses a .NET ODBC Data Provider. The DSN configured in the connection manager uses the IBM DB2 ODBC Driver.
I have a package variable, the value of which I want to use in the WHERE clause of the query contained in the Data Reader source. The query is as follows:
SELECT T1."IRK_ACCT_CD", T1."IRK_COMPANY_NBR", T1."IRK_ACCT_NAME", T1."IRK_ADDRESS_1", T1."IRK_ADDRESS_2", T1."IRK_ADDRESS_3", T1."IRK_STATE_CD", T1."IRK_POSTAL_CD", T1."IRK_AR_TYPE", T2."RK502_ITEM_RE_NO", T2."RK502_TRANS_CD", T2."RK502_TRANS_TYPE", T2."RK502_ITEM_AMT", T2."RK502_ITEM_DT", T2."RK502_PURCH_ORD_NO", T2."RK502_GL_EFF" FROM "CBDBOW"."IRK_RECORD" T1, "CBDBOW"."RK502_OPEN_ITEMS" T2 WHERE T1."IRK_COMPANY_NBR" = T2."RK502_COMP_NO" AND T1."IRK_ACCT_NUMBER" = T2."RK502_ACCT_NO" AND T2."RK502_ITEM_DT" < ?
I'm not very sure as to how to map the package variable this way. I think for ODBC parameters, I need to represent the parameter with a question mark.
I have configured my DataReader to use an ADO.net (ODBC) connectivity (entered Select * from AMPFM) in Sqlcommand and can see my database columns listed in the Advanced Editor / Column mappings window. My process needs to perform a straight column to column population from AMPFM table into my dbo.visitfinancials table. How do I point the output to the above table?
I have a problem with DataReaderSource. I'm trying to get data from Notes table. I created a Connection manager and the connection was successful. The SQLCommand in "Component properties" tab is a simple "select * from <table_name>". When I switch to the "Column mappings" tab, only the first column from the table is displayed. Pressing the "Reftesh" button resulst in the following error: Error at Data Flow Task {DTS.Pipeline]: The "output column <column_name> has a length that is not valid. The length must be between 0 and 4000. When I go to the "Input and Output Properties" tab, the DataType for the output column is not populated and the error message "Error in Data Flow Task [DTS.Pipeline]: The output column <column_name> had an invalid datatype (0) set." The DataType property is not populated at all. Changing the data type to DT_STR results in error: "Property value is not valid". Details: Error at Data Flow Task [DataReader Source]: The data type of output columns on the component "DataReader Source" cannot be changed".
I read on a previous post to explicitly convert field , and tried to explicitly covnert the dataype on the field in my query (ex. select convert(varchar(50) from fieldname)
It then gives foll err:
ERROR [42000] [Lotus][ODBC Lotus Notes]Incorect syntax near ',' [Lotus][ODBC Lotus Notes]Name, constant or expression expected.
I am trying to have a DataReader Source that can run a variable which I used to store the SQL statement. For example, I have:
Variable #1
Variable name: tablename
Data Type: string
Value: "name_of_table"
Variable #2
Variable name: sql_stmt
Data Type: string
Value: "SELECT * FROM " + @tablename
I want to use DataReader Source to run Variable #2 in the SqlCommand that connects to an ODBC connection. If it is possible by any way, please let me know. Thanks in advance.
I am running an MDX query in SSIS but I don't know what is the best way of doing this, performance wise. I know I can run the MDX query through an openquery in the OLEDB, and also run it through a Datareader, no openquery needed.
I know the datareader is slower in a normal basis due to .Net, but in this case the OLEDB is running an open query to a linked server which won't be fast like running a regular SQL.
If anyone knows which of this two run faster in this scenario I'll appretiate if you let me know.
Apologies for asking a similar question again but I am still non the wiser with this problem!
Let me explain to you my situation and the method I've adopted to try and solve it.
I have some source data residing in a SQL Server 6.5 database. The source data consists of a single table. For this example I will assume that my table contains only 2 columns, an ID column called result_ID and a Result_Name.
The idea is to retrieve new data each time the package is run. We will know this because the result_IDs in the source table will be greater than the maximum result_ID in my destination table . The way the package should work is like this :
1) Retrieve maximum result_ID from destination table
2) retrieve data from source table where result_ID > maximum result_ID from destination table.
My package consists of a
1) SQL Query Task which retrieves the maximum result_ID and places it in a user variable (type Int32).
2) A Data flow task with a Datareader source adapter which uses an expression to retrieve the data. My expression looks like this : "select * from result where result_id > " + (dt_str, 10, 1252) @[User::max_result_id]
When I run my package the first time all the rows are retrieved (as my destination table is empty to begin with). BUT when I run it the second time the same thing happens again!! All rows are retrieved.
I placed a breakpoint at the point where the variable gets populated with the maximum result_ID and true enough, the variable gets populated with the correct result_ID BUT then that variable gets reset to 0 in my expression!
This problem is driving me crazy! Has anybody out there experienced this kind of problem before?! What are the ways to solve it?!
I get an error at the end of a 47 million row job when I use the datareader source. It goes through all the records and then the package fails. The error (DataReader Source [1]] Error: System.NullReferenceException: Object reference not set to an instance of an object. ) occurs at the datareader source. I suspect it's because my record set returns a null value at some point. Any ideas?
Is there a way to control the types for output columns of a DataReader Source? It appears that any System.String will always come out as DT_WSTR. As I have my own managed provider, and I know what went in, I can say that really it should be DT_STR. The GetSchemaTable call from my provider will always say System.String as it does not have much choice, but GetSchemaTable does contain a ProviderType which is different for my DT_STR vs DT_WSTR, or rather when I want each. I think something like MappingFiles as used by the Wizard would work, but can I do anything today?
