Data Problems With Xml Source In Integration Services

Mar 10, 2008

Hi everyone,

I've got a problem to retrieve data from a Xml Source.
Basically, I call a method from a Web Service which gives me a Xml file.

The problem is that the XML structure is not really good. But we can't touch it.

Here is the Xml File :





Code Snippet

<?xml version="1.0" encoding="utf-16"?>
<ArrayOfWSTargetVO xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<WSTargetVO>
<ProjectId>
<Value>131</Value>
</ProjectId>
<Id>
<Value>Toto</Value>
</Id>
<Name>
<Value>bateau</Value>
</Name>
</WSTargetVO>
<WSTargetVO>
<ProjectId>
<Value>131</Value>
</ProjectId>
<Id>
<Value>Tata</Value>
</Id>
<Name>
<Value>F35</Value>
</Name>
</WSTargetVO>
...
</ArrayOfWSTargetVO>
As you can see, for each WSTargetVO, we have a projectid, an id and a name. But the value is not directly put into these nodes but in a new one : <value>

That causes my problem because here is the xsd file generated by visual studio :





Code Snippet

<?xml version="1.0"?>
<xsd:schema xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsd="http://www.w3.org/2001/XMLSchema" attributeFormDefault="unqualified" elementFormDefault="qualified">
<xs:element name="ArrayOfWSTargetVO">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" name="WSTargetVO">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" name="ProjectId">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" name="Value" type="xs:unsignedByte" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" name="Id">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" name="Value" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" name="Name">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" name="Value" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xsd:schema>
And when I try to use the outpul results from the Xml file, I can't see how I can get a datatable with three columns corresponding to projectid, id and name.

Integration Services only asks me to choose between WSTargetVO or ProjectID or Id or Name and give me the <value> value.

I don't know if it is possible to modifiy the contents of the XmlFile or something else using XPath.

Of course, if I try to modifiy the XSD file and delete the value node to have a simple structure, I see my three columns but i can't get any data.

I'm aware that the XML file is pretty bad but it is impossible for me to change it.

If somebody has an idea, I would be happy to hear it :-)

(I'm a beginner in Integration Services)

Thank you,
Radik

View 3 Replies


ADVERTISEMENT

Integration Services :: How To Do Data Profiling On 3 Source Tables

Sep 29, 2015

need to do data profiling on 3source tables .Can I use the data profiling task for it.

I am mapping the xml output to excel file using dataflow task.

View 2 Replies View Related

Integration Services :: SSIS Not Pulling All Data From Source?

May 6, 2015

I have ssis package that pull data from SAP (Using ADO.net connection) to SQL server every night but i have noticed that all data from source is not getting pulled by package . package losing some amount of row.

View 7 Replies View Related

Integration Services :: Adding Oracle Data Source In SSIS

Nov 21, 2011

I am trying to create new data source. I already tried these data sources

Oracle Provider for OLE DB
Oracle Client Data Provider
Microsoft OLE DB Provider for Oracle.

After configuring when i test the connection, it tells connection succeeded but if i click on then giving the error "The given path is not support".

View 8 Replies View Related

Integration Services :: Collect Data From Multiple ODBC Source

Jun 11, 2015

I am able to collect data from Progress DB, using ODBC Connectivity. The problem I am facing is, i have to iterate thru multiple servers. How do i configure ODBC source dynamically. It creates problem. Using expression, i tried to set the connectionstring dynamically, but it fails.

View 2 Replies View Related

SSIS - Data Source Password Lost When Imported Into Integration Services

Apr 17, 2007

After designing a SSIS package in Visual Studio 2005 that had two connection manager defined to keep the password. After I deployed the package to a file system. I then Imported the .dtsx file after making a Integration Services connection in Sql Server Management Studio. When I tried to run the package it failed when it tried to make the connection. When I edited the connection manager connection string and added the password and the package ran fine but it does not retain the password!. I need to have this package scheduled to run daily so I need to know how to have the package keep the password in the connection string. I have seen other posts on this issue but not seen a good solution. Could someone point me to the proper MSDN article that would explain how to implement this ? Is it a SQL Server configuration issue or a property in Visual Studio SSIS design time ?



thanks.

View 4 Replies View Related

Integration Services :: Get Data From Source By Executing Set Of Queries That Have Temp Tables

Jul 29, 2015

I need to grab data from teradata(using odbc connection).. i have no issues if its just bunch of joins and wheres conditions.. but now i have a challenge. simple scenario, i have to create volatile table, dump data into this and then grab data from this volatile table. (Don't want to modify the query in such a way i don't have to use this volatile table.. its a pretty big query and i have no choice but create bunch of volatile tables, above scenarios is just mentioned on simple 1 volatile table ).

So i created a proc and trying to pass this string into teradata, not sure if it works.. what options i have.. (I dont have a leisure to create proc in terdata and get it executed when ever i want and then grab data from the table. )

View 2 Replies View Related

Integration Services :: Compare Source And Target Data Using Conditional Split

Aug 12, 2015

I'm encountering a very peculiar situation when I'm trying to compare source and target data using conditional split. Following is the Data Flow and how I'm trying to achieve this.

Source Data : Col_A (PK)      Col_2
                       1                    100
                       8                    500
Target Data : Col_A (PK)      Col_2
                       1                    100
                       3                    700
                       8                    500
Look-up Target on Col_A to check for existing records. Now we have four columns in Look-up match output: Col_A, Col_B, Lkp_Col_A (Target Col), Lkp_Col_B (Target Col).

Conditional Split: Compare Col_B with Lkp_Col_B

Update target if there is any change in the existing value of Col_B.When I'm running the package for every record in source, the conditional split fails and even when there is no change in Col_B, some of the records (Not all and quite randomly) get updated with the same value. If I run the package for few records, it works absolutely fine.

View 8 Replies View Related

Integration Services :: Package Failed After Changing Password In Shared Data Source

Jun 19, 2015

I'm using a shared data source to connect an Oracle server in my packages.  After changing the database user password in the shared data source, I noticed the package concerned would fail with the following description.

Source: "OraOLEDB"  Hresult: 0x80004005  Description: "ORA-01017: invalid username/password; logon denied".

Is there a way to ensure the packages will use the latest information in the shared data source?  I did do a Rebuild before executing the packages.

View 5 Replies View Related

Integration Services :: Source Script Component To Read Data From Sharepoint List?

Apr 29, 2015

All examples I found refer to classes under Microsoft.SharePoint namespace. However, I have the SharePoint CSOM that only gives me the Microsoft.Sharepoint.Client namespace.

I need to read the selected values of a multichoice field, but not sure how to do it with classes in the namespace above.

everthing works, exept the TSQL_x0020_Reference_x0020_Numbe field.

my code looks like this:

Webweb = cont.Web;
cont.Load(web);
cont.ExecuteQuery();
Listsstest = web.Lists.GetByTitle("T-SQL
Code Review Tracking");
//CamlQuery query = CamlQuery.CreateAllItemsQuery();

[code]....

View 2 Replies View Related

Integration Services :: SSIS OLEDB Data Source - Flat File Generation

Apr 20, 2015

I am working to archive some old data from a data warehouse using SQL server and SSIS.  The data will be read and denormalized, then shipped out to a delimited text file.

The rowcount of the incoming data is significant, call it 10M+ rows per unit of work (one text file).

There are development advantages of using a stored proc for the data source - mainly ease of changing the denormalization logic as required.  Wondering if there are performance advantages of an embeded query for the data source instead?

It was mentioned by one developer that when using a stored procedure, the output stream from the proc and subsequent SSIS steps cannot start until the full procedure processing is complete; i.e. the proc churns out its' result set in one big chunk. 

He hinted that an embedded query does not have this same effect, but I am not sure that is accurate.

View 4 Replies View Related

Integration Services :: Can't Preview Data In Excel Source Editor Unless Sheet Is Open?

Nov 10, 2010

I have a package which has an Excel source with the 'Data access mode' set to SQL command and then a sql select statement.  When I try and hit the 'Preview...' button below the 'SQL command text' window I get the following error:

 "Error at Standard Data Flow Tasks [source tasks name]: No column information was returned by the SQL command"
 
Ordinarily this would be down to the fact that my SQL is shocking, I hit the 'Preview...' button whilst the workbook the source is pointing at was open and it works fine??
 
I can't figure this out, but needless to say the package errors with a NEEDSNEWMETADATA when I try and run it.

View 17 Replies View Related

Integration Services :: SSIS Excel Data Source Numeric Values Returned As Null

May 8, 2009

I'm using SSIS 2005 Enterprise edition,  I'm creating a package that reads an excel (xls) file using the "excel source" component, and it dumps the data into an OLEDB destination (a sql server). When I drag the excel source component and create the excel connection to my file the component automatically reads the columns and their datatypes.

The problem is that I have a column which has numeric data and the package uploads as NULL every number that starts with a zero. (note: in excel this column is formatted as "text", despite it has only numbers, because it's the only way excel maintains the left sided zeros).

So I checked the data types by right clicking the excel source component -> show advanced editor and my surprise is that this column's data type is detected as double-precision float, and it doesn't let me change it. URL... but it only works when the first row of data has a number beginning with zero on this column. How to get the data imported correctly?

View 15 Replies View Related

Integration Services :: DefaultBufferMaxRows - Is It Determined By Row Length Of Data Flow Task Source Or Destination

Oct 18, 2015

We have a single generic SSIS package that is used to import several hundred iSeries tables into SQL. I am not looking to rewrite the process. But I am looking for ways to improve performance.

I have tried retain same connection, maximum insert commit size, lock table (tablock), removed some large columns, played with the log file location and size, and now I am working to tweak the defaultbuffermaxrows.

To describe the data flow task - there are six data flows tasks (dft)  working at the same time. Each dtf has their own list of iSeries tables and columns and the corresponding generic SQL table names. Each dtf determines their list of tables based on the number of columns to import. So there is dft30 (iSeries table has 1-30 columns to import), dtf60 (iSeries table has 31-60 columns to import), etc. The destination SQL tables are generically called Staging30, Staging60, etc. Each column in the generic Staging tables are varchar(100). The dtfs are comprised of an OLE DB Source and an OLE DB Destination.

The OLE DB Source uses a SQL Command from Variable to build a SELECT statement. The OLE DB Source uses a connection manager that uses an IBM iAccess IBMDA400 provider.  The SQL Command ends up looking like this for the dtf30. This specific example is importing from the iSeries table TDACLR and it only has two columns so it will be copied to the Staging30 table.

select TCREAS AS C1,TCDESC AS C2,0 AS C3,0 AS C4,0 AS C5,0 AS C6,0 AS C7,0 AS C8,0 AS C9,0 AS C10,0 AS C11,0 AS C12,0 AS C13,0 AS C14,0 AS C15,0 AS C16,0 AS C17,0 AS C18,0 AS C19,0 AS C20,0 AS C21,0 AS C22,0 AS C23,0 AS C24,0 AS C25,0 AS C26,0 AS C27,0 AS
C28,0 AS C29,0 AS C30,''TDACLR'' AS T0 from Store01.TDACLR

The OLD DB Source variable value looks like the following, but I am not showing the full 30 columns

select cast(0 AS varchar(100)) AS C1,cast(0 AS varchar(100)) AS C2,cast(0 AS varchar(100)) AS C3,cast(0 AS varchar(100)) AS C4,cast(0 AS varchar(100)) AS C5, ... cast(0 AS varchar(100)) AS C30.

The OLE DB Destination uses OpenRowSet Using FastLoad From Variable. The insert into Staging30 ends up looking like this.

insert bulk STAGE30([C1] varchar(100) ,[C2] varchar(100) ,[C3] varchar(100) ,[C4] varchar(100) ,[C5] varchar(100) , ...  ,[C30] varchar(100) ,[T0] varchar(20)

Of course we then copy and transform the Staging30 data to the SQL table that equals T0.

But back to defaultbuffermaxrows. Previously the dtfs had default values of 10000 for DefaultBufferMaxRows and 10485760 for DefaultBufferSize. I added a SQL task to SUM the iSeries column sizes, TCREAS and TCDESC in this example, and set the DefaultBufferMaxRows by dividing the SUM of the columns max_length into 10485760. But I did not see a performance improvement. Do you think that redefining the columns as varchar(100) for the insert is significant? Should I possibly SUM the actual number of columns (2) as 2x100 or SUM the 30x100?

View 4 Replies View Related

Integration Services :: Package Development For Pulling Data Into Excel Destination File From OLEDB Source

Sep 2, 2015

1 How to get the desired output colums into Excel file without having 'copy of column/unwanted columns' in destination file.

2. How to override the existing file in excel destination.

View 2 Replies View Related

Integration Services :: Metadata Change In Source Disturbs All Configuration Of Source

Jun 18, 2015

I have a got a package with source as sql table which has got 50 columns. We are using only 10 columns out of this. Recently one column name has changed and thus throws error invalid mapping. When I open the source to do the changes noticed that all the colums are prselected now and also the datatypes got changed to default ( I had changed the datatypes as per my requirement while i developed). So now I had to select required columns from source and redo the datatype changes in advanced editor.Is there any option which doesnt disturb this settings and we just need to correct the mapping alone. 

View 4 Replies View Related

Integration Services :: Load Incremental Data Into Fact Table When Source Table Not Have Timestamp And Unique Key

Sep 24, 2015

I have a transaction table having about 40 crore rows in source. It don't have timestamp and unique key columns. It have only Bill_month and Bill_Year columns. Actually for loading this table into staging I have added a new datetime column by adding default bill_date as 01. Then

* First we delete last 3 month data from staging tables.
* Get last 3 months data from source table.
* Load that 3 months data from source to staging table. 

We do this because we only get update for last three months data. Now I have to include this transaction table as Fact table in DW. What will be the best practice for loading the fact table by picking data form staging table. Also we have to look up with dimensions for Foreign Keys. 

* Should I implement the same method of deleting last 3 months records and loading them again. 

View 3 Replies View Related

Integration Services And Teradata As Source

Nov 17, 2006

Hi All,

Please let me know whether we can use Teradata as Source in SSIS (Target is SQL Server 2005). ie. Do we have oledb driver for NCR Teradata to connect to it. Our SSIS will be hosted in 64 bit SQL, but for development we use 32 bit.

Any inputs is really appreciated.

Regards, kart

View 7 Replies View Related

Integration Services :: Progress Database As A Source

Aug 29, 2005

I'm having trouble using a Progress database as a source.  I have an OpenLink driver installed and a System DSN set up.  I can successfully test the connection. I added this DSN to the connection manager and added it to a DataReader Source.  I then added the SQLCommand property.  I was able to map columns and such, so I believe the SQLCommand was successfully parsed.  However, when I try to save the DataReader Source, I get an error:      

Error at Data Flow Task [DataReader Source [2266]]: System.Data.Odbc.OdbcException: ERROR [HY010] [OpenLink][ODBC][Driver]Function sequence error   at System.Data.Odbc.OdbcConnection.HandleError(OdbcHandle hrHandle, RetCode retcode)   at System.Data.Odbc.OdbcDataReader.NextResult()   at System.Data.Odbc.OdbcDataReader.Close()   at Microsoft.SqlServer.Dts.Pipleline.DataReaderSourceAdapter.ReinitializeMetaData()   at Microsoft.SqlServer.Dts.Pipleline.ManagedComponentHost.HostReinitialieMetaData(IDTSManagedComponentWrapper90 wrapper)

View 6 Replies View Related

Integration Services :: Component OLE DB Source Has No Inputs

Nov 12, 2015

I have a Data Flow Task. I have one "OLE DB Source" which gets my data from a SQL Server Database. I have a second "OLE DB Source" which uses DATEADD to derive a date qualifier that I would like to use as a date qualifier in my subsequent Excel spreadsheet...opting to use SQL Server and DATEADD rather than messing around with VB syntax to get the previous week date qualifier.I am trying to connect the flow from one OLE DB Source to the next OLE DB Source and get the error..Component OLE DB Source has no inputs, or all of its inputs are already connected to other outputs. You may be able to edit the component to add new inputs to it.Can't I connect two completely different and independent SQL Server queries using "OLE DB Source" within my Data Flow?

Is there any way to store my derived date from my second "OLE DB Source" to a variable so that I cana then use that as my date qualifier within my Excel destination?

View 6 Replies View Related

Integration Services :: Cannot Connect To Source Server

Aug 17, 2015

I have two servers (TESTSERVER, PRODSERVER). On the TESTSERVER i have a package that i'm using for data load from a source database on a SOURCESERVER. Everything works great until this point. Now from the PRODSERVER  in the SSIS package i'm trying to create a connection in the connection manager to the SOURCESERVER and i'm getting the following error

Test connection failed because of an error in initializing provider. Login timeout expired.A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections.

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 53)

View 10 Replies View Related

Integration Services :: Load Different Source Files Into Tables

Oct 12, 2015

I have one small requirement.. I want to load the different types of files(.txt, .csv, .tsv, .xlsx).

Using one forearch loop container how can I load the files to database and I shouldn't use the script task to split the filenames. Is there any other way to load all the files using forearch loop container, exesql task..

View 2 Replies View Related

Integration Services :: How To Change Sequence Of Source Column

Nov 3, 2015

I am working on 1 POC project.I have 2 customer having source file in txt format, but the column sequence of both customer are diffrent.Number of columns in all files are like below.

CustA

ID   NAME   AGE
1     VIPIN    29

CustB

ID   AGE   NAME
2     29      jayesh

As per source file you can see that CustA have column sequence ID,NAME,AGE and CustB Have ID,AGE,NAME sequence .I have target table #Temp with ID,NAME,AGE sequence.Like that I have many files from both customer, I have to load in ID,NAME,AGE sequence from all source file to target table.How can we change the sequence of source column before loading to target table.

View 5 Replies View Related

Integration Services :: CDC Source Timeout And Transaction Integrity?

Jul 31, 2015

I have created a SSIS Package which does the incremental update using CDC Controls.

The design is similar to any standard CDC incremental package.

It has a CDC Start which sets the Mark Processing Range, a data flow and a Mark Processed Range.

The issue that i'm facing is that the CDC Source control time-out but i can still see rows moving from CDC Source to Splitter and target table. After the rows are transferred, the Data Flow task Fails which leads to package failure.

This results in Mark Processed Range not being executed.

So my query is

1. Why is CDC Source being time-out?

2. What can i do so that all three i.e Mark Processing Range, data flow and Mark Processed Range execute successfully or nothing does.

View 2 Replies View Related

Integration Services :: How To Use Lookup Transformation Using Excel As A Source

Aug 27, 2015

i want to use lookup transformation using Excel as a source.i am having two excel files .

file1 one of the column contains 'Andhrapradesh'
file2 one of the column contains 'ap'

here want to match these using lookup.

View 5 Replies View Related

Integration Services :: For Each Loop Container Is Not Picking Up CSV Source File

Jul 2, 2015

My package is having .csv file as a source and I kept OLEDB destination to load it.

Stored the .csv file in a shared folder and the exact path is given in Enumerator configuration of the foreach loop container. When I execute my package, it is giving the warning as below:

It is saying that file is not there in the specified path and directory is empty. I am running the SSIS package from TFS. I am sure that I have read and write access for the shared folder for my userID. Is there any access there to pick up this file from path.

View 4 Replies View Related

Integration Services :: Cannot Find SharePoint List Source And Destination

Nov 6, 2015

I have installed the SharePoint adapters from codeplex and they show OK in SSIS 2008R2. But in SSIS 2012, I can't find them and their is no SSIS component tab to pick it and add it to the toolbox.

View 2 Replies View Related

Integration Services :: Using Database Name As Parameter In OLE DB Source Command Text?

Aug 6, 2015

I'm using following command to populate my OLE DB Source. I have two of those in each Data Flow in my package. One of the OLE DB Sources points to my source database the other to the destination. In order to limit the number of rows I use the WHERE clause below. The [EnergyMiserFSRLive] being the the name of the source database. The Connection manager points to the destination database.

I would like an elegant way to replace  [EnergyMiserFSRLive] with a parameter which I can reuse in each of my many data flows rather than use this hard coded value [EnergyMiserFSRLive].In particular I'm after the syntax of the below query that uses the parameter for [EnergyMiserFSRLive].

SELECT [SitesId]
      ,[SiteName]
      ,[FilePrefix]
      ,[City]
      ,[StateProv]
      ,[Country]
  FROM [Sites]
WHERE SitesId >= (SELECT MIN(SitesId) FROM [DBNameFSRLive].[dbo].[Sites])
  ORDER BY [SitesId]

View 5 Replies View Related

Integration Services :: Share Point Source RSSBus Adapter

Nov 16, 2015

I want to use the excel sheets located on a share point site as source, for this i downloaded the adapter from the following link URL....I got the connection manager to work, but when coming to load the excel files to the destination location its not doing what is intended. For example i want the destination location my local drive for the excel files, how can i do that which destination should i use?

View 2 Replies View Related

Integration Services :: How To Handle Dynamically Changing Source Columns

Jun 29, 2015

I have a scenario where we have to handle dynamically changing source columns.

For example , some times in the source files the number of columns will be increased or decreased, new columns can be added in the middle or in the end of the source file.

How to handle this kind of scenario in the SSIS ?

View 9 Replies View Related

Integration Services :: Automating SSIS Package For Different Source File

Sep 22, 2015

I want to design an SSIS package that loads data from files into SQL Server and I want to automate the process. My major issue is that the source file doesnt come in the same format. Some times I comes in either .csv , .xls , .txt or even .rpt file format. Is there a way I can write a code that checks through my folder and based on the available format on the folder it loads the value in ssis.

View 2 Replies View Related

Integration Services :: Dynamic Mapping From XML Source To Destination Table

Jun 1, 2015

I have a requirement to take xml file, in case the number of column changes, it should not fail the package, rather it should load the data in destination table. Destination table could be altered separately depending on xml schema by the DB team in production.

View 3 Replies View Related

Integration Services :: Source Excel File Causing Failure In Agent

Aug 13, 2015

I have a package from SQL Server 2008 R2, that loads data from .xlsx file to database table.There are total 15 columns and 14000 rows in the .xlsx. The package runs fine in BIDS. But the same package in SQL Agent fails with error "omponent "Excel Source" (1)" failed validation and returned validation status "VS_ISBROKEN".

When I tried to run the package by deleting the half of the records for first 7000 rows it ran successfully in agent. Then the second half (last 7000 rows) also succeed from agent job. So, there is no issue with the data/datatypes.The agent job is able to run with record upyo 11000 rows in .xlsx. When I am running for 12000 rows it is failing.Is there any problem with the number of records in .xlsx or size through SQL Agent?

I am running the package from a Proxy account in sql agent job.

ERROR:
Error: Executed as user: PROXY_ID. Microsoft (R) SQL Server Execute Package Utility Version 10.50.6000.34 for 32-bit Copyright (C) Microsoft Corporation 2010. All rights reserved. Started: 10:36:09 AM Error: 2015-08-10 10:36:10.87 Code: 0xC0202009 Source:
XX Connection manager "Excel Connection Manager 1"

[code]....

View 5 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved