BLOB Or VARBINARY(MAX) As SSIS Source.
Jan 4, 2007Hi all,
Can a SSIS package treat a file from a table ( VARBINARY(MAX) or BLOB) as a source for migration ?
Thanks in advance,
DBAnalyst
ADVERTISEMENT
Transact SQL :: Reading Text From BLOB / Varbinary Objects
Aug 26, 2015We have an interface where we receive data from an external supplier. One of the fields in the interface is of type BLOB (the source is an Oracle database), which would be read into our MSSQL database as image. This can also be converted to varbinary, and a typical field value looks something like:
0x70697A5F8F000000789C0DCCBD0DC2301....etc. etc.
However, we know that the origin only contains text, and we even know the text from the GUI they supply us with. The text could typically be "Delayed by 3 minutes because of water damage" or something like that.
What I want to do, is to extract that text from the field.
First, I have stored the incoming data stream in a table, where one column is of type varbinary(max). It looks like this goes swell. But I don't know which command to use in order to get the text extracted.
I have tried these:
1) select master.dbo.fn_varbintohexstr(Myfield) from Mytable
-> Returns just the text "0x70697A5F8...." which I have no interest in
2) select cast(Myfield as varchar(max)) from Mytable
-> Returns just Chinese signs.
3) select cast(Myfield as nvarchar(max)) from Mytable
-> Returns just Chinese signs.
4) declare @ptrval varbinary(16)
select @ptrval = TEXTPTR(MyField) from MyTable  -- with MyField defined as image
READTEXT MyTable.MyField @ptrval 1 30
-> Returns just the text "0x697A5F8...." which I have no interest in
Of course, since only text is stored in this field, the field should never have been defined as BLOB in the first place. But the source system is external, and it's a standard system, so we may not alter it in any way.
Transact SQL :: Export BLOB (varbinary Max) Column To Excel Or CSV File?
May 20, 2015I have a table in one of my databases that stores files in one of its columns. I need to be able to export this BLOB column into either a CSV or Excel file. I am forbidden from using xp_Cmdshell so I was wondering if there was a way to do this in the Cmd prompt.
View 5 Replies View RelatedSQL 2005: SSIS: Script Component: Working With BLOB
Jan 3, 2008I have few tables that I need to export to an MS Access database each day to send off to the customer. In these tables I have a few columns that I need to strip all the HTML out of before they are put into Access since Access does not display the formatting correctly. Some of the fields are varchar and some are text, but all of the fields that are varchar are over 255 in size, so this forces me to use Access' Memo type for both.
In SQL 2000 I used the ActiveX Script to modify these fields, stripping out the HTML, and it gave me no complaints. In SQL 2005, I am adding a Script Component into the Data Flow before the Destination and adding the script in there. So far so good. The problem arises when I try to retrieve and update the data in these fields because they are treated as BLOB data. I believe what I need to do is to retrieve the Byte array from the row using the GetBlobData(), then convert that to a string, strip out the HTML, convert back to a Byte array, then clear the original value using the ResetBlobData(), and then update the field using the AddBlobData. I think?
I am not even sure that my code below is converting the byte array into a string correctly. If I throw a Messagebox in there to see what it has for the string, it just gives me the first character in the field. But even ignoring that, the package will not execute and I am stumped.
The error I now get is:
Array cannot be null.
Parameter name: bytes
''-----------------------------------------------------
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim b As Byte()
If (Row.ProjectDescription.Length > 0) And (Not
(Row.ProjectDescription_IsNull)) Then
b = Row.ProjectDescription.GetBlobData(0,
CInt(Row.ProjectDescription.Length))
End If
Dim str As String
Dim enc As New System.Text.ASCIIEncoding()
str = enc.GetString(b)
str = RemoveHTML(str)
b = enc.GetBytes(str)
Row.ProjectDescription.ResetBlobData()
If b.Length > 0 Then
Row.ProjectDescription.AddBlobData(b)
End If
End Sub
''-----------------------------------------------------
SQL 2005: SSIS: Script Component: Working With BLOB
Apr 8, 2008
I have a blob which is essentially an email and I need to replace all carriage returns in the blob with a special character ("€°"). But it does not work. Please advice.
Dim intBlobLen As Integer = Convert.ToInt32(Row.body.Length)
Dim intFinish As Integer = intBlobLen - 1
Dim bytBlob(intFinish) As Byte
bytBlob = Row.body.GetBlobData(0, intFinish)
Dim strBlob As String = System.Text.Encoding.Unicode.GetString(bytBlob)
strBlob = Replace (strBlob,"/n","€°")
If strBlob.Length > 0 Then
Row.body.ResetBlobData()
Row.body.AddBlobData(System.Text.Encoding.Unicode.GetBytes(strBlob))
End If
Please help. Thanks in advance
Store Varbinary Data Result Into SSIS Variable Through Execute SQL Task
Feb 13, 2008I cannot find the data type for parameter mapping from Execute SQL Task Editor to make this works.
1. Execute SQL Task 1 - select max(columnA) from tableA. ColumnA is varbinary(8); set result to variable which data type is Object.
2. Execute SQL Task 2 - update tableB set columnB = ?
What data type should I use to map the parameter? I tried different data types, none working except GUI but it returned wrong result.
Does SSIS variable support varbinary data type? I know there's a bug issue with bigint data type and there's a work-around. Is it same situation with varbinary?
Thanks,
-Ash
Using XML Source In SSIS Vs Script Source To Parse Xml Files.
Sep 14, 2007
Hi
I am having a huge xml file with nested section.
i also have a xsd file for that xml.
i have a destination table where the data from the xml should be loaded into.
i am using the xml source transformation. But o get all the data i need to use multiple merje joins to get the data in a single row which i can insert into the destination.i was not quiet convinced with using so many joins.
so i tried using the script source transformation where i am using xml objects to get the node and dynamically construction the data row. and the output is then inserted into the destination.
on comparing the two approach the one using the script source is working much faster than the xml source transformation.
i wanted to know is there any limitaion using the script source to parse through xml files.
also i would like to know any other better way of getting the data from xml source without using the joins.
Hari
To BLOB Or Not To BLOB, That Is The Question.
Aug 1, 2006We are debating what is industry “best practice� for serving huge numbers of images in an industrial scale website. More directly, which approach produces the best performance and the best scalability? For example, how do sites like ebay, Amazon, and other large sites handle the millions or billions of images they must deal with?
Store as BLOB in sql server?
Store in /images folder and store url text into sql server?
We always assumed that the second approach is what most sites must do. But do they?
One developer on our team maintains that storing one million or more image files in a directory will most certainly result in poor performance, because the server must scan the directory, searching for the correct file, each time a web request is made. The directory is not indexed (?) so performance must eventually suffer.
Other developer counters that storing millions of images as BLOBs into sql server will result in poor performance and HUGE database. An additional layer of access (webserver to sql server, back to webserver, then to client) causes a delay and performance hit.
Who is right? What do the gurus as the world class sites do?
Source Code For The Standard SSIS Components That Come With SSIS?
Jun 2, 2006Hi,
I'm finding that the standard components often just don't quite meet my
needs, but would only need some fairly minor changes to save me and my team a lot of
work (and produce more elegant solutions). So I was just wondering whether the source code was available for the standard components that come with SSIS, or if there is anyway to extend their functionality?
Or do you just have to start form scratch?
Thanks,
Lawrie
Using SSIS As Source In .NET Application
May 13, 2008Hi,
I want to use output of SSIS package in Windows application, any suggestions?
For this I have created SSIS package and loaded data into DataReader destination, but now how to use this data in my application.
Thanks in advance.
Regards,
Omkar.
SSIS And Data Source
Aug 10, 2007Hello,
I have SSIS solution which has multiple packages and one Main package which starts all others.
I also have 2 Data Sources which I use on each package.
I need to do my testing on different Data Sources. When I try to change Data Source from one database to another - SSIS is getting confused and fails on the connections or starting to insert to the previous database.
Could you please explain the steps for doing it. What is the problem here?
Can I use Windows Attentications in data source or use special account?
Also some confusion here with Config file. Should I create Config file just from Main package or from each package. For now I am working with a Source code, but later I would need to change connections string before running the executable. If I change the string in my Main package - would it be understandable in all others?
Please give me details here.
Thanks.
V.
Working With Xl Source In SSIS
Jun 11, 2007Hi All,
I want to use an excel sheet as a source and pick up the data starting from 4th row.
can anybody suggest me a solution for this.
Cheers,
Prash
SSIS - Excel Source
Dec 28, 2007Dear Friends,
I need to import data from several excel files. How can I configure excel source object to dinamically import each file?
The name of the file will be in a parameter of ssis package and this name change frequently, and ach time the filename change I dont want to change the configuration on the excel source?
What you sugest?
Shoul I use a script component as source?!
Regards!
SSIS OLE DB Source Error
Oct 3, 2006I am creating an OLE DB Data Source in an SSIS Data Flow - it is calling a SQL Server 2005 stored procedure, which accepts 3 parameters and returns a result set, but I get the following error:
"There is a data source column with no name. All data source columns must have a name."
I have have tested the output of this proc a million times, and every column has a name! I am also able to call a different proc in the same way, without error:
EXEC pProc @DT = '20060928', @Id = 1015, @Debug = 0
Any ideas what could possibly be causing this error - assuming all of my columns DO have names??
Thanks : )
SQL 2012 :: Source As Query In CDC In SSIS
Aug 13, 2014I am working on task where I need to load data into DWH from OLTP. Load needs to b incremental load. For this I am planning to use CDC, to track changes and only load data which has been new inserts/updates/deletes. But to load data to destination, there are couple of joins, which I need to do. Is there a way in CDC, where I can use query as source?
View 0 Replies View RelatedSSIS SQL OLE DB Source - SQL Command Vs Mapping
Oct 10, 2007I created a Stored Proc to compute all the data I need to export to a CSV file.
I use the provider MS OLE DB Provider for SQL Server.
It's a very simple package with a single Data Flow Task.
This flow task is using an OLE DB Source to call a simple SQL Command : Exec MyStoredProc.
There are no params.
This Stored Proc is using table variables to compute the data.
It takes about 10 seconds to return anything.
The problem is that the mapping doesn't work with the OLE DB Source.
There are no fields at all shown in the mapping screen.
I tried to replace the Stored Proc by a version which only returns fields and no data.
Then the mapping would work just fine.
The package is then compiling and working fine.
But everytime I put back the real stored proc, even without changing the SQL Command, the SSIS execution breaks at execution.
It keeps saying :
"Error: 0xC0202005 at Data Flow Task, OLE DB Source [477]: Column "RecordType" cannot be found at the datasource."
My guess is that SSIS doesn't wait the 10 secs and thinks the Stored Proc is not returning anything.
What can I do to make this work ?
Thanks,
Vincent
Using .xlsx As Excel Source In SSIS
Oct 17, 2007
Hi,
I am using Excel datasource and Excel destination in a simple SSIS package which i created. I have got two queries regarding this
How does excel datasouce/destination accept .xlsx files, since i observed that they accept only .xsl file
Does excel destination ascept .xlsm files, because i have some macros enabled in destination
Thanks in advance for advice
Warm Regards,
gchanduu.
SSIS Project And Source Control
Aug 26, 2007Our team has the following problem:
I create a SSIS project in VS and add it to Team Source Control. When the project and all it's files exist on a machine and I want to open the project from Source Control, everithing works fine, but when another developer, who doesn't have the files on his machine, wants to open the project from Source Control he is required to check out the project file.
I've investigated the problem and found out that each project has a database file (with .database extension) and project file contains a reference to it. Once the user opens from Team Source Control a project that isn't presented on a target machine, the VS creates a new database file on a local machine and insists to change the reference in the project file.
Does anyone have a workaround for the problem?
Thanks
SSIS XML Source With Random Child ID??
Jan 17, 2006Hi,
I am trying to load a XML document using SSIS into relational tables and pulling my hair out.
In it's simpliest form I have:
<Batch>
<POSBasket>
<Header>
<SomeDetail/>
</Header>
</POSBasket>
<POSBasket>
<Header>
<SomeDetail/>
</Header>
</POSBasket>
</Batch>
Now, when you have a parent child situation, you get a column in the output called xxx_Id, in my case, POSBasket_Id. This allows you to relate your header back to your root and SomeDetail back to header etc etc.
Loading all of this in parrallel does not work because there is nothing unique once the load has happened more than once. i.e. You will have repeating POSBasket_Id's as well as Header_Id's.
I ended up loading POSBasket into a table with an identity column. Once done, I would load the header details and do a lookup (eek) on the POSBasket table by using POSbasket_Id "where isComplete = 0" (A flag to tell me current import).
Great, this works find with 1 item in my document. Add a second (2 POSBaskets) and it randomly works. (???) Add more and it hardley works. The look ups are failing with "no match found".
On using Data Views, I see that each XML source creates different POSBasket_Id's. Some matching, some not. Also seem to randomly match.
My question is why? Why would there be the ID if they differ when loading them from the same document? They not even unique across sources.
Bug or "by design"?
Anyone got a better method for doing this?
Cheers,
Crispin
Validating Source Schema Using SSIS
Sep 23, 2007Greetings!!
I have a MsAccess db containing a table called Employees which i am transforming to a staging table in Sql server 2005. Everything is working fine. I am using Foreach File enumerator and uploading the files one by one.However I now plan to validate the schema of MsAccess before uploading it. For eg: My employee table in msaccess is as follows :
Code Snippet
Employees
empId int,
empName varchar(60),
empAge int
Since the files come from different vendor, while looping, i want to perform a check if the empid or empAge are not of type long or string etc. If they are of type smallint,i have no problem.
However if they are larger datatypes than the the ones kept in Sql server, then the file needs to be logged in the db with the reason and moved to the error folder. In short, if the datatypes in access tables are smaller than those in Sqlserver, allow it, otherwise reject it. THe schema of Sqlserver table is same as of that of Employees in msaccess.
I want to compare the schema of the incoming access tbl fields with my desired schema and all mdb's having data types that are higher or incompatible with the desired schema should be moved to the error.
How do I do it.
Thanks ,
Ron
BUG In SSIS OLE DB Source Build Query
Jun 28, 2007Can anyone help me with this ?
I am trying to extract data from oracle 9i server and
pushing it onto the SQL Server 2005 using Data Flow Task.
Details for OLE DB Source are :
OLE DB Source Editor Details:
OLE DB Connection Manager - Oracle Source
Data Access Mode - SQL Command
SQL Command Text -
SELECT EMPNO, EMPNAME, JOB, HIREDATE
FROM EMP_DETAILS
WHERE (HIREDATE > TRUNC(SYSDATE) - 1)
SSIS parses this query succesfully but when i build the query it shows query
SELECT EMPNO, EMPNAME, JOB, HIREDATE
FROM EMP_DETAILS
WHERE (HIREDATE > TRUNC("SYSDATE") - 1)
Please note :- SYSDATE IN " "
This query returns no Result Set.
Report With Ssis Package As Source
Sep 3, 2007 The report having SSIS package as source works fine on my local 32 bit windows xp machine from Report Manager. But same doesn't work from 64bit windows 2003 machine... fails with below error... what could be an issue?
An error has occurred during report processing.
Cannot create a connection to data source 'DataSource1'.
is not a valid Win32 application. (Exception from HRESULT: 0x800700C1)
The same works fine when previewed from visual studio on 64 bit machine.
Thanks
Mahesh
Large XML File Source In SSIS????
Jan 19, 2007Hi,
I have a problem where I want to import a 1.6 GB XML file with SSIS into a SQL Server database. My hunch is that SSIS is not very good with handling such large amount of XML data. My test shows that SSIS tries to read all of the file into memory.
Does anyone know if there is any solution of solving this memory problem. My problem is that I want to take this source XML file import it into a database, make some transformations on it (eliminate duplicates etc) then produce a NEW XML file as output in a different XSD-format.
Is really SSIS the right tool for this operation?
The source XML file also have mixed content on Complex Types which seems to be a problem for SSIS as well.
Best regs,
//Patrick
SSIS Error On DataReader Source
Feb 29, 2008I have a package that runs every hour. It runs fine most of the time but once in a while
I get this not very helpful error message "the component 'xyz' is unable to process data" on DataReader Source component.
when i try to run it again on the same data it works fine. So i dont think it has to do with data.
Is there a way i can get more information on this error?
SSIS - Excel Source Dynamic
Jan 2, 2008Dear Friends,
I currently have my excel source dynamic for the connectionstring. Using a global variable with the filename and using the expressions of this control to dynamically have the connecttionstring. My problem is that i need tto read always the first workseet and the name changes frequently and generates an erro in SSIS.
how can I set the excel source to goes only for the first sheet independently of the worksheet name?!
Regards!!
SSIS ODBC Source Component
Apr 24, 2006I need to create an ODBC source script component that outputs into SQL Server. When I debug I get the following error message:
Error at Data Flow Task [Script Component [1]]: System.InvalidCastException: Unable to cast object of type 'System.Data.Odbc.OdbcConnection' to type 'System.Data.SqlClient.SqlConnection'. at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.HandleUserException(Exception e) at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.AcquireConnections(Object transaction) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostAcquireConnections(IDTSManagedComponentWrapper90 wrapper, Object transaction)Error at Data Flow Task [DTS.Pipeline]: component "Script Component" (1) failed validation and returned error code 0x80004002.
Here the problem code:
Imports System
Imports System.Data
Imports System.Data.SqlClient
Imports System.Math
Imports System.IO
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
Public Class ScriptMain
Inherits UserComponent
Dim connMgr As IDTSConnectionManager90
Dim sqlConn As SqlConnection
Dim sqlReader As SqlDataReader
Public Overrides Sub AcquireConnections(ByVal Transaction As Object)
connMgr = Me.Connections.PP
sqlConn = CType(connMgr.AcquireConnection(Nothing), SqlConnection)
End Sub
Public Overrides Sub PreExecute()
Dim cmd As New SqlCommand("SELECT Solution_Code_From, Solution_Code_To FROM Solconv", sqlConn)
sqlReader = cmd.ExecuteReader
End Sub
Public Overrides Sub CreateNewOutputRows()
Do While sqlReader.Read
With SolutionOutputBuffer
.AddRow()
.solcodefr = sqlReader.GetString(1)
.solcodeto = sqlReader.GetString(0)
End With
Loop
End Sub
Public Overrides Sub PostExecute()
sqlReader.Close()
End Sub
Public Overrides Sub ReleaseConnections()
connMgr.ReleaseConnection(sqlConn)
End Sub
End Class
Would appreciate any advice.
Thanks in advance,
Pozzled
SSIS Source Code Library For Others To Use
May 12, 2006In SSIS can one create €œuser defined functions€? €œsubroutines€? and put them in the SSIS designer for other developers to use?
TIA,
Bill
SSIS Source Table/Row Locking
Feb 8, 2008
Hi all,
We have an SSIS package which runs for around 30 min and it does a lot of things, now whenever this SSIS Package is run it is suspected that it is locing the table or records from the DB which it is reading the records, and indeed that system is having some fault that it is never able to remove that lock unless the DB restarts. In this source we only have a SQL Command and not a Table selected. Have anyone come across this problem? Any answers will be of great help.
SSIS VSS Source Control Structure
Apr 22, 2007I have all the config files for my various packages in a folder call configs. It seems that I cannot add folders to the SSIS solution. I can add config files, but they get copied into the root folder, which breaks the directory structure that I have.
Am I missing something? Is there a recommended to manage config files, and other external files used by the SSIS package, in SourceSafe and in the SSIS solution? Thank you.
Reading From SSIS Flat File Source?
Feb 9, 2012I have a simple SSIS package -> It reads a local text file which has 10 rows of data ( id, name, telephone # ) and puts it into a table.
It uses the "SSIS Flat File source" to read and a "SQL Command" to insert into the table. I can see that it reads line by line and puts each line into one row in my table.
Now, my production data is over 5 GiG of mainframe data and it seems their data is arranged in some hierarchical form.. so the position or arrangement of data in that file is important.
I pulled the data using my package and as far as I can see , my SSIS package pulled one line at a time ( from the flat file) and pushed it into my table. For each row, I also created an identity column in my table to be able to identify the positional arrangement of the hierarchical data and then use relational mappings to suit our business needs.
In all of this, my assumption is -
"SSIS reads one line at a time, inserts to my table and goes down to the next line .
It does NOT read a snapshot of rows from the flat file so as to write them into the table using internal ordering methods based on that particular snapshot "
My question is .. is my assumption correct ?
Using SSIS Package To Delete Source Files
Jan 31, 2013is it possible to delete a source file(*.txt) when a SSIS package is done with it?
View 3 Replies View RelatedDynamic File Name For Flatfile Source In SSIS
Apr 11, 2008I have CSV file as source for SSIS package every time the filename will be changing like trd_1990M1_1990M12.csv,trd_1991M1_1991M12.csv , trd_1992M1_1992M12.csv etc.,
so it will vary as per user selection . i need to run the same SSIS package to execute the different file name with the same structure.
Please let me know the solution for that how to pass the file name dynamically to SSIS package.
SSIS Datareader ODBC Source Error
Feb 24, 2006I am trying to import data from an ODBC source using the Datareader but have come accross a problem, can anyone help?
Whenever I try to do a 'SELECT * FROM sometable' I get the following error:
Error at Data Flow Task [DTS.Pipeline]: The output column "notes" (521) has a length that is not valid. The length must be betewwn 0 and 4000
Most of the tables we require have this field.
Is there any way to by pass this error and import the field as DT_WSTR type and convert to DT_NTEXT?
Our original DTS package (2000) worked well but we would like to move forward.
Thanks in advance.
John