Integration Services :: Transactions Handling Manually Without MSDTC
Aug 28, 2015
I created an SSIS packages on VS 2013 using SQL 2014. I have to use transactions and without MSDTC as we don't have that environment.
Ok my issue when I load a stage table during the process and then I select from that same stage table my records on another table are not being inserted. why? I have done sql profiler an all sql statements are being ran under same SPID.
Package design
1. For each container - this is to process files on a folder
2. Execute sql task - begin tran
3. sequence container - to hold all and organize containers
4. execute sql task - truncate ImpTableA
5. Data Flow task - Import data into ImpTableA from file
6. execute sql task - insert into DimProducts and new products found in ImpTableA that are not in DimProducts so I'm joining the 2 tables together as a outter join and getting the new ones.
7. Execute sql task - commit tran
8. Execute sql task - rollback tran
On Step 6 I don't find any new rows on DimProducts even though I know there is new products. I have tested the sql statement in SSMS and it works and I find new rows. For some reason SSIS is not finding new products, I'm assuming that the rows are not committed yet and therefore can query them but I thought if I was under the same transaction and SPID I would be able to read uncommitted rows.
Need to be able to read from table that was just inserted in a transaction incase of a failure roll everything back.
I've created a SSIS package that calls the access dll and fires off 2003 access reports, saves them as PDF's and emails them off.
Now this works fine when I run it manually, but when I schedule and fire off a job I get a very vague error "exception has been thrown by the target of an invocation".
I have copied the access dll to the GAC and .net framework v2.0.50727 but still no luck.
I'm using Bull zip PDF printer and those DLL's are also in the GAC
I have a class that use a TransactionScope object for make make two operations into a database. The problem is that in a computer works well but in another I receive a error message: "MSDTC is not running in .SQLEXPRESS". I had watched the services in the computer and the "Microsoft Distributed Transaction Coordinator" is running. Does somebody know what is happening? Thankyou and sorry for my English.
I have one ssis package moving the data from staging to destination. In stating table we have the duplicate data. But in destination table 4 columns have primary key. How to handle the duplicate records in oldedb source.
I want to caputure all error records with rowid and error code and Error description in SSIS 2012.We want to do this in Dataflow level... I am using error out option(Redirect Row). But it is not giving detailed information of the error records.
I receive a data feed from a third party in a pipe delimited file. From time to time, they add a column at the end. I would like my ssis package to continue to process the data even if they add a column with out it breaking. How best do I handle this situation?
I would like to have my SSIS tasks to be transacted, but due to infrastructure issues in our network, cannot utilize DTC, because our SQL Boxes are situated where the DTC cannot communicate. I know we can use execute SQL task with Bein Trans, but I dont think dataflow and all will not be in scope of this and cannot be rollback in case of any error in the underlying tasks.
HelloI had a question reagrding MSDTC usage. I know that MSDTC will enlisteach connection transactions as it's own (take ownership) andcommit/abort accordingly.However I'm not sure what would happen where I have a stored proc withinput parameters and 2 Delete commands on 2 different tables. In thiscase, would MSDTC also enforce ACID rules when 1 cmd succeeds and 2ndaborts ? In the script for the Stored Proc, there is no checking for@@ERROR value. There are just DELETE statement and a plain RETURN.thanksSunit
High level question: Is there a way to rollback a transaction without failing a task?
Here's the scenario:
A single foreach loop contains a sequence container. The sequence container has its TransactionOption set to "Required." All activity happens within the sequence container. This is so that each iteration of the foreach loop is in its own transaction space. Now the catch. In the sequence container I run a few tasks, but more importantly, at the end (final task before the next foreach iteration) I run a Script Task to compare the values of two variables to see if they match. If they do, then all is well; move onto the next iteration. If they don't (and this is where my problem lies), then I need to roll back the current transaction without halting the foreach loop's iterations.
Is the best idea/approach to increase the value of MaximumErrorCount for the parent/package and set the Dts.TaskResult to Dts.Results.Failure, or is there a way in the script task that I can roll back the transaction and set Dts.TaskResult = Dts.Results.Success??
Here is the situation. We have a .Net application that some consultants developed several years ago. They made use of DAO for database access, using some inbuilt classes like ContextUtil to get a transation environment.
Until last weekend, the envorinment was:
Web application on IIS5 / Windows 2000 Advanced Server
Database on SQL2000 on Windows 2000 Advanced Server
We switched over to a new database server environment - a cluster of two Windows 2003 servers wth SQL Server 2000. This required Active Directory to be set up on the two database server machines.
Everything worked but this particular application - where it goes to update the database. It appears (but since I haven't yet been able to rebuild the .Net solution) that the error - "Object reference not set to an instance of object." is in fact a spurious error caused when the transation tries to roll back.
I have spent about 10 hours on this, and have come to the conclusion that the fact I have MSDTC involved on a database server that is in an AD domain, while the web server that is issuing the transaction is outside that domain is the cause of the problem. Now, I don't want to put AD on the IIS 5 web server - we are migrating this to a new Win2k3/IIS6 server in 5 or 6 weeks time.
Any suggestions? If I have to, I can move the database for this particular app back to the non-clustered Windows 2000 server (which is also outside the AD domain).
We have created SSIS package to load a text file into a table. Source system shares 10 text files and recently they stopped generating data for one of the text file (comping empty), after few months they will start generating the data for the empty file batch processing.
The Issue here is Data Flow task is getting failed while loading empty text file into table. How to handle this empty file load issue in SSIS package.
Hi guys, Does any one know how to detect when a SQL transaction has been rolled back in either a windows application or ASP.NET. My Transactions always run but when they are rolled back Visual Basic does not pick up any errors in the 'Try Catch SqlException'. Does any one know a way round this. Sorry for the lack of code. Im writing this post on a friends PC but i wil put up my code as soon as possible. Thanks in advance Matt
I am creating a SSRS report which would be executed by User manually through ReportServer URL.User would be generating the SSRS report for different Customer ID based on ad-hoc basis.
I am passing CustomerID as input parameter to the report. Is there any way to get the manually generated SSRS report name as 'Report_CustomerID_TodayDate.xls'.
E.g.If User is generating report for Customer ID 123 today then report name should be 'Report_123_07092013.xls'
Hello friends. I managed to design an Integration service package,but the desired level of performance has not been achieved(i.e it is performing slow). So I want to know what are the best practices for optimized solution . In my package I'm exreacting data from XML file and Storing it in SQL server database with some processing dring data flow.
I'm using 1) Two Script Task Control -In these control,I m opening the connection to XML file through VB.net code and iterating each record at a time. 2)Two OLE DB Command -Each fetched record from script task component is processed in OLEDB command through stored procedure and then inseted into database. 3)One for Loop -This loop contains two script Task control and two OLEDB Command control, (mentioned above),for fetching single record and inserting it in database. 4)One derived Column 5)One Multicast 6)One Character Map 7)One OlEDB Source
As with my current performance I'm able to insert one record in every .5 second (Which is much below to acceptable limits) Is control lying disabled on SSIS designer pane also affect the performance of execution.
I am using a web service to access SQL Server to return data to the client as well as to update, insert and delete.
I am passing an Sql Transaction ByRef to the web service when I do updating...etc.
The problem is because I am using transactions, I cannot share the web service....I get the error: System.Data.SqlClient.SqlTransaction cannot be serialized because it does not have a parameterless constructor when I try to create a web reference.
Is there a way around this aside from having the transaction in the web service itself (which I am not sure is a good idea)?
I have an Analysis Services Processing Task in my SSIS package. I run the SSIS package using SQL Server job, the running of the package is a job step.
When I process manually the analysis services objects (in practise cubes) using dtexec utility I get a lot of log. In case the processing fails I get error messages that quite well describe the error. But when I run the job the only information I get in the job log is that the job step failed. I know the failure happens in the Analysis Services Processing Task.
Is there any way in SSIS to get a) the log of the Analysis Services processing or b) the error messages of the Analysis Services processing? Or should the processing be done some other way than I've been doing?
Hi, I have just install SQL 2005 SP2 and trying to get Window SharePoint Services V3 integrated with SQL 2005 SP2 reporting services. In SharePoint Central Administration, I select the Reporting Services Integration page and have setup the Report Server Web Service URL and Authentication Mode. I then goto Grant database access, specify the SQL server name, get promted for a username and password that has access SQL Reportserver and get the following error "The group name could not be found" Does anyone have any ideas? Thanks
Hello, I have a problem when trying to fully process an SSAS database using Integration Services "Analysis Services Processing Task" task. I have 2 of these tasks which are responsible for processing the Dimensions then the Cubes. When I run the package either via the BIDS environment or on the local server from the Integration Services engine, I will get an error after about 20 minutes stating:
"Error: Memory Error: Allocation failure. Not enough storage is available to process this command""Error: Errors in the metadata manager. An error occurred when loading the <cube name> cube from the file \?D:Program FilesMicrosoft SQL ServerMSSQL.2OLAPDataMyWarehouse<cube file>.xml"
The cube name is not specific, it will fail and any of my cubes could be in the error log
If I fully process the AS database using the AS engine (logon to local AS server, right-click AS database and click Process), I get no errors at all, it processes and completes fine. The processing options are identical when I run in AS or via the SSIS "Analysis Services Processing Task" task.
I've searched quite a lot online but no joy, the information I have gleaned from various sites does not directly link SSIS with SSAS processing problems.
When either the AS processing starts via SSAS or SSIS the memory usage of MSMDSRV.exe increases to around 1.4 / 1.5 GB but never goes to 2GB ever, even when the error appears.
I've done the following with no effect.
" Have run via AS and works fine " No specific cube it fails on " Have created a Dimension only package, same problem " Changed the maxmemorylimit " Changed the connections to localhost " Memory DOES NOT max out on server
Server Specs: Windows Server 2003 Standard + Service Pack 2 4GM ram, 2GB paging file
Hi there, I have decided to move all my transaction handling from asp.net to stored procedures in a SQL Server 2000 database. I know the database is capable of rolling back the transactions just like myTransaction.Rollback() in asp.net. But what about exceptions? In asp.net, I am used to doing the following: <code>Try 'execute commands myTransaction.Commit()Catch ex As Exception Response.Write(ex.Message) myTransaction.Rollback()End Try</code>Will the database inform me of any exceptions (and their messages)? Do I need to put anything explicit in my stored procedure other than rollback transaction? Any help is greatly appreciated
Is there a way to give customers access to SSIS? They need to be able to create their own SSIS packages. Of course we have more then one customer so it would be nice to have modular security in place where they don't get to see customers abc and customer xyz packages. Only their own.
I have created an integration services project (attached is a screenshot) that workes against the flat file (.DAT extension) and it does some manipluation in the data and then load it into the table. Everything works fine. Now I want to get the name of my flat file source fille (which is a .DAT file) and then insert it in the table. I am running th integration services against different .DAT files (only one file at one time) which are located on different locations.....so what I want is that, whenever I run the package it do the usuall processing and then while loading the data in to the destination table, it also load the name of the file into the destination table (lets called a field "FileName" of nvarchar type in the table "Comphistory")
Problem When you have a SSIS package that contains a connection from a data source, this connection is not updated when the data source changes based on a configuration change.
Situation : A SSIS solution contains 3 configurations : Development, Test, Production. You can create those configurations in configuration manager of the solution.
The SSIS project contains one Data source. It doesn't really matter what type but I take SQL Server. The database server in development is SQL_DEV, in test is SQL_TEST and in production is SQL_PROD. Initially they are for all configurations the same. You can specify those values by changing the active configuration and then editing the Data source.
In the SSIS package (DTSX), you can create a connection manager based on a Data source. If you change the Data source, the connection manager is also changed. If you change the Data source by changing the active configuration, the connection manager is not being updated.
If you think this isn't a big issue think big. We have 4 configuration, 10 shared Data sources and 25 DTSX packages. That would give a maximum of 1000 settings (4 x 10 x 25). Using this method it can be reduced to 40 (4 x 10). Of course this is a theoretical but it is very common to have the destination data source re-used on all packages, which still would be 100 settings (4 x 25)
Steps to reproduce - create a new SSIS project - In the solution explorer, create a new Data source named TestSource. - In the connection managers window of Package.dtsx, create a new connection from a Data source. - Make some changes in to TestSource.ds under the Data Sources. For example change the server or the database. - Verify that those changes are also in the package.
- in the solution explorer, right click the solution and select configuration manager - under active solution configuration, create a new configuration named test. - Set the copy settings from : development - Verify that Create new project configuration is checked. - click OK and close. - Notice that the active configuration is now Test - Make some changes the Testsource.ds like a different server. - Verify that those changes are also in the package. - Make the development configuration as active. - Notice that the Testsource.ds contains now the original settings. - You will notice that the connection manager still contains the "test" settings and not the development settings. - If you create a deployment utility it will still contains the wrong values.
Can someone explain to me why I am getting this kind of error though I am able to integrate all the data succeesfully to the next destination.
Ronald
SSIS package "Prescription.dtsx" starting.
Information: 0x4004300A at Data Flow Task, DTS.Pipeline: Validation phase is beginning.
Information: 0x4004300A at Data Flow Task, DTS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Data Flow Task, DTS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Data Flow Task, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x4004300C at Data Flow Task, DTS.Pipeline: Execute phase is beginning.
Error: 0xC0202009 at Data Flow Task, SQL Server Destination [521]: An OLE DB error has occurred. Error code: 0x80040E14.
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 58, column 1. The destination column (PatientId) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 27, column 12. The destination column (ServiceId) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 26, column 12. The destination column (ServiceId) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 25, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 24, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 23, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 22, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 21, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 20, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
Information: 0x40043008 at Data Flow Task, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x40043009 at Data Flow Task, DTS.Pipeline: Cleanup phase is beginning.
Can someone explain to me why I am getting this kind of error though I am able to integrate all the data succeesfully to the next destination.
I am trying to get the Prescription table from Access to SQL SERVER 2005 database
Ronald
SSIS package "Prescription.dtsx" starting.
Information: 0x4004300A at Data Flow Task, DTS.Pipeline: Validation phase is beginning.
Information: 0x4004300A at Data Flow Task, DTS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Data Flow Task, DTS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Data Flow Task, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x4004300C at Data Flow Task, DTS.Pipeline: Execute phase is beginning.
Error: 0xC0202009 at Data Flow Task, SQL Server Destination [521]: An OLE DB error has occurred. Error code: 0x80040E14.
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 58, column 1. The destination column (PatientId) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 27, column 12. The destination column (ServiceId) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 26, column 12. The destination column (ServiceId) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 25, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 24, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 23, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 22, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 21, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The bulk load failed. Unexpected NULL value in data file row 20, column 7. The destination column (AllergyCode) is defined as NOT NULL.".
Information: 0x40043008 at Data Flow Task, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x40043009 at Data Flow Task, DTS.Pipeline: Cleanup phase is beginning.
I'm trying to use the new integration services and have found all kinds of help for it, but where is it? I can't find it. Is it in the SQL Server Management Studio?
I am trying to know how to read many XML files within one data flowtask. I have many many files that I would like to apply to with thesame data flow so I need to know how to read them in and what taskshould I use under SQL 2005 DTS/Integration services.Thanks for your help!JON
I've to do a mining project and I intend to use the SSIS.
I've done a clustering plugin last year on analysis services and I also want to use it.
Let me try to explain the architecture of the process:
1) Receive data (read data from the database - these data are texts, actually)
2) Pre-process the data (transform the texts in a sparse matrix) using a new plugin
3) Call my clustering plugin and assign it to read the table created on the previous step
4) Call my KNN plugin to classify other pre-processed texts using the clusters found on the previous step as classes.
5) Show results.
Alright... It all running as a workflow on integration services
Here are my doubts:
A) How to view and use my plugin made for Analysis Services on Integration Services ? (is it possible or will I have to create another plugins from zero just to run on Integration Services ?)
B) Assuming the previous step is possible, how to modify my plugins to define inputs and outputs to do the correct communications between each plugin ? I think this is the most important question. Is it simple to do ? Is there any documented examples ?
