Run SSIS Package With Alternate Configuration Via DTEXEC?
Jan 31, 2008
It looks like you can run an SSIS package via the DTEXEC command. You can also specifiy a configuraiton file flag on the command. Is there anyway to specify a configuraiton filter if you stored it into a table vs and xml file?
I have a serious problem with my SSIS Package while executing using 32-bit DTExec and 64-bit DTExec.
Here are the details:
Environment:
Windows Server 2003 64-bit (Build 3790: Service Pack 2) SSIS 32-bit & 64-bit installed SQL Server 2005 (Microsoft SQL Server 2005 - 9.00.1399.06 (X64) - RTM)
SSIS Package details (compiled in 64 bit)
Script tasks only Microsoft Visual Basic .NET (using TRY...CATCH block) PreCompileScriptIntoBinaryCode = TRUE Run64BitRunTime = TRUE
Execution
Batch file that uses DTExec to execute the Package.
SCENARIO I am trying to exeucte the above SSIS package using both 32-bit and 64-bit DTExec to make it failure by providing invalid connection string. Here are the details,
Wrong connection String using 32-bit Execution
While establishing the connection the error message has been nicely captured in to my Exception block and writes into the log file.
Wrong connection String using 64-bit Execution
While establishing the connection the error has not been catpured anywhere (although I have TRY CATCH block) and it haults there itself with the message "Process is terminated due to StackOverflowException". Later I found that the error is due to the connection string along with the unhandled exception.
Please suggest any one of the following my findings, also if you have any other advice would be very much appreciated.
1. Shall I go ahead and fix the issue by handling those unhandled errors? (e.g Appdomain, application). I tried several but still not working using 64-bit DTExec.
2. Shall I go ahead and use 32-bit DTExec to execute the package? If so, is there any other major issue...like performance or anyother bug?
P.S: We cannot apply any service pack for SQL Server 2005 at the moment. Sorry abt it. If you have any specific hotfix for DTExec (without affecting SQL Server) then we can decide.
Sorry for the lengthy one and Thanks very much for you help in advance .
I developed my package in BI studio and tested runnning it through BI. Now when i have to deploy the package to main server it will be executed via batch job. This package requires an input file name too. Now when i am trying to run the package on my loacal box using following command on command prormpt i get error given below.
COMMAND USED BY ME: DTExec /FILE 'C:ProjectIntegrationServicesSFRIntegrationServicesinpkgSFRLocation.dtsx' /SET Package.Variables[InputParamFileName].Value;'C:DataDumpInputLOC00022'
ERROR MESSAGE: Microsoft (R) SQL Server Execute Package Utility Version 9.00.1399.06 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved.
Started: 1:29:54 PM Error: 2008-02-22 13:29:54.96 Code: 0xC0011007 Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Description: Unable to load the package as XML because of package does not ha ve a valid XML format. A specific XML parser error will be posted. End Error Error: 2008-02-22 13:29:54.96 Code: 0xC0011002 Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Description: Failed to open package file "'C:ProjectIntegrationServicesSFR IntegrationServicesinpkgSFRLocation.dtsx'" due to error 0x800700A1 "The speci fied path is invalid.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of either providing an incorrect file name was specified when calling LoadPackage o r the XML file was specified and has an incorrect format. End Error Could not load package "'C:ProjectIntegrationServicesSFRIntegrationServices inpkgSFRLocation.dtsx'" because of error 0xC0011002. Description: Failed to open package file "'C:ProjectIntegrationServicesSFRInt egrationServicesinpkgSFRLocation.dtsx'" due to error 0x800700A1 "The specifie d path is invalid.". This happens when loading a package and the file cannot be opened or loaded correctly into the XML document. This can be the result of eit her providing an incorrect file name was specified when calling LoadPackage or t he XML file was specified and has an incorrect format. Source: {9B1FAA88-A6FA-44CF-A2E2-DB3F737FDA65} Started: 1:29:54 PM Finished: 1:29:54 PM Elapsed: 0.047 seconds
PLease let me know solution to creating batch job for executing this pkg with input param.
Deployed Report having SSIS package as source do not work when Indirect Package configuration is used in ETL package. It seems ETL package when called/executed from Report manager does not recognize environment variable to pick up the dtsconfig file.
The Report works when Direct package configuration is used to same dtsconfig file.
What could be the reason? Any solution for this? This will cause our build/deployment to QA and Prod very difficult.
Hi, I'm facing a problem that I'm trying to solve. I report it in the following hoping somone can help me!
the context:
- I have a solution containing almost 20 packages - I have a main package containing the sequences (Sequence Container) calling all others packages (Execute Package Task) - every package has the connections to its own DBsource ; so the package X has the its own connections to DBsourceA and DBsourceB, the package Y has the its own connections to DBsourceB, DBsourceC, DBsourceD and the destination DB is unique for all the packages.
My problem is: I have the necessity to define in the best way a config (one for all the dtsx, possibly) that allow me to manage easily the switch from the developing env to the deployment env, basically for the DB connections.
Which is the best way to do this and can anyone tell me the steps to follow?
I'm facing a little problem concerning package configuration that is using a SQL table.
I have 4 exactly the same Db's on the same SQL server, each with another name of course (Dbipa, DbIpB, DbIpC & DbIpD).
My developers created 4 exactly the same SSIS packages with 4 exactly the same jobs, each for his own DB only the connection settings are different.
Now I was trying to manage this with only one SSIS package by using an SSIS Package configuration table in each DB.
I thought it would be possible to start the package with the following parameter
/SET Package.Connections[SQL].InitialCatalog;DBIPA OR
/SET Package.Connections[SQL].InitialCatalog;DBIPB OR
/SET Package.Connections[SQL].InitialCatalog;DBIPC OR
/SET Package.Connections[SQL].InitialCatalog;DBIPD But this doesn't work because before he changes the initial catalog in the connection named SQL, he already did the loading of the SSIS_Configuration table. So he point always to the same config table. Is there a way to change the connection before the package configuration is being executed. Ludo Bernaerts
1) We are doing data migration from SQL Server 2000 OLTP design to SQL Server 2005 OLAP design.
2) We have used SSIS packages and data flow tasks in which we mentioned connection strings for source and target containers.
3) We have a master execute package which contains series of execute packages in relational order.
4) When we execute this master package, we should be able to dynamically specify different source and connection strings for all packages.
5) In master execute package, we generate an XML configuration file using SSIS -> Package Configurations... using Connection Managers -> Particular connection
6) Now, we change connection string to point to another database after adding this new OLE DB connection in Connection Manager of each and every package.
7) When we save, build and execute master package it is still executing migration packages against the old database. Please let me know what I am missing.
I am new to SSIS. I have an SSIS package which we are storing it in the application server ( not in SQL Server ). We are storing the database name etc.. in package configuration. Where should I store the configuration file? In app server or in the database server. Also please tell how to execute the package stored as file system.
We have more than 100 packages which referenced many of the same data sources. In ideal situation, we would like to create three master filters based on the dev, test, and prod servers and all packages can just reference the same filters. However since one package normally only use a subset of master connections defined in the configuration files, if only those subset of connections are set up in the package, when the package is first open in BIDS, lots of errors will pop up saying that it could not find the connections defined in the configuration files. Even though this will not cost runtime error, it is still very annoying. Does anyone have better ideas on how to handle this?
I've been working with SSIS (SQL 2005 SP1) and I have a problem adding a package configuration of the SQL Server type. Upon completion of all steps in the wizard, the "Target Object" and "Target Property" attributes of this configuration remain empty. Hence, the package cannot set this configuration when being executed.
Does anyone know if this is a bug or have I missed something?
I have an issue with my SSIS package not using the package configuration specified in my SQL server table. This is a simple scenario where I need to move data from a csv file to a SQL server table.
I have 2 connection Managers - one for the location of CSV file and another for the SQL database connection string. I chose to Enable Package configuration. Two configurations -
a. Environment variable to store the database connection string.
b. SQL server table configuration for flat file connection manager.
The location of the CSV file is dynamic..so the idea is to change the path as and when needed in the SQL table.
No errors..everything works fine except that the properties of the flat file connection manager is not being pulled from the SQL server configuration table. It uses the location I had specified during design time.
How do I force the SSIS package to use the configuration specified in the SQL table always.
I've been spending a lot of time on this with no success. Please help..
Started: 10:48:56 AM Info: 2008-01-24 10:49:02.92 Code: 0x40016041 Source: SsisEndBatch Description: The package is attempting to configure from the XML file "C:Program Files (x86)Microsoft SQL Server90DTSPackagesBridgeNet.BI.SSIS.EndBatc ENDBATCH TEST ENV.dtsConfig". End Info Warning: 2008-01-24 10:49:03.15 Code: 0x80012011 Source: SsisEndBatch Description: Cannot load the XML configuration file. The XML configuration file may be malformed or not valid. End Warning
I have been asked by the powers that be to make sure that my configuration database gathers some auditing information. I have looked over these and have no idea where to find the information in SQL Server. If you could tell me the table and database that the informations are located in, I could write the T-SQL to find them. IMPORTANT you do not have to answer all of them, I am grateful to those that pick and choose even one to help me with.
SQL Server Auditing
General/Access Auditing Items
Software Install (DB): Verify that the sample and demonstration databases are not installed and remove the temporary setup files created by SQL Server setup process
Authentication and Access Control
Default user accounts & passwords (DB): Default user account passwords will be changed and will follow the corporate password standard for frequency of change, length and complexity User authentication (DB): Windows authentication is preferred. Mixed mode authentication can be used. User passwords (DB): All user passwords (mixed mode, SQL authentication, Windows authentication) will follow the corporate password standard for frequency of change, length and complexity. Sharing database user accounts (DB): Sharing of database user accounts is not permitted without an explicit exception documented and granted by IS Information Security. Normal users submitting jobs (DB): Normal user jobs should not be run as SA. Create a role-based policy for normal users that must be allowed to run jobs. They should be included in the role created for job submission.
Monitoring and Reporting
Auditing (DB): For all database installs after XX/XX/XXXX, Auditing should be turned on and at minimum set to log failed connections. Logs should be saved to a different hard drive than the one which data files are stored. SQL error log (DB): Processes should be in place to protect error log data so it can be reviewed for seven days prior to being overwritten or deleted from the system.
Networking
Cross-database ownership chaining (DB): Cross-database ownership chaining is disabled. Code of stored procedures, triggers and views (DB): Who has access Public access to SYSXLOGINS and SYSDATABASES tables: The public role will not have access to the SYSXLOGINS or the SYSDATABASES tables. Public access to SYSOBJECTS and SYSCOLUMNS tables: The public role will not have access to the SYSOBJECTS and SYSCOLUMNS tables. Public access to stored and extended stored procedures: The public role will not have access to stored or extended stored procedures. Public access to xp_regread and xp_instance_regread: Restrict public role access to xp_regread and xp_instance_regread. Public group access to mswebtasks table: The public group will not have insert, update, delete or select permissions to the msdb.dbo.mswebtasks table. Temporary directories: For all database installs after xx/xx/xxxx, all temporary directories will be cleaned on a periodic basis. Auditing II: Auditing should be turned on and at minimum set to log failed connections. Logs should be saved to a different hard drive than the one which data files are stored. SQL Server instances visibility: All SQL Server instances should not be visible across the network. They should be set up with the hidden option activated. Default ports: SQL Server should be configured to not use the default ports.
I have a package where I store connection managers in a configuration file (XML file). I updated manually the XML file to add the passwords.
When I launch the command line generated by dtexecui, everything is good: "dtexecui /FILE "E:mssql90DTSPackagesBVOMDWRefreshStage Refresh EUR 1.dtsx" /CONFIGFILE "E:mssql90DTSPackagesBVOMDWRefreshEnv_Prod.dtsconfig" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EW "
When I launch the command line with dtexec: "dtexec /FILE "E:mssql90DTSPackagesBVOMDWRefreshStage Refresh EUR 1.dtsx" /CONFIGFILE "E:mssql90DTSPackagesBVOMDWRefreshEnv_Prod.dtsconfig" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING EW ", I have the familiar error: "Failed to decrypt protected XML node "DTSassword""
I have the package level set to "EncryptSensitiveWithUserKey", but I think I don't need to update it to "EncryptSensitiveWithPassword" because it works with dtexecui.
I have a SSIS package called "MyExport" stored on the SQL server 2005 standard SP1. I have created an SSIS package configuration stored in a SQL table msdb.dbo.SSIS Configurations, with the configurationFilter = "Export2" with some configuration values which I can change programmatically as needed.
How do I execute this package with the configuration in the SQL table? I don't seem to have any problem when the package configuration is stored in an .xml file. The documentation is very poor or non-existant on trying to do execute the package with configurations stored in SQL server. I just can't seem to get the proper syntax.
Can someone give me an example of a dtexec command for the above or maybe some c# code?
In Execute Package Utility, when you select the Configurations option, it pops up a dialog box for a FILE based configuration file (*.dtsconfig). There is NO WAY to access a configuration stored in the [dbo].[SSIS Configurations] table - which is where the SQL based configurations are stored. If you could pick a SQL based configuration, you would then need to pick the ConfigurationFilter for the configuration you want to use.
It's the same when you try to create a new job in SQL Server Agent - you can't select the package configuration stored in SQL server. When you get to the configurations tab, you can only add a file based configuration.
I've checked BOL as well, and there are no examples or discussion of this that I can find.
If anyone can point me in the proper direction, I'd appreciate it.
How do I create, say an "update statistic" maintenace package that works on multiple servers and databases?
For example, how do I get the package to operate on the following servers and tables: ServerA, tableX, tableY, tableZ ServerB, tableM, tableN
Also, I would like to save the package configuration and store it on SQL Server. So if I want to add more servers or tables to the list, the SSIS package will pick it up at run time.
We have 6 SSIS packages which populates different sets of table by reading different set of excel file.We need to have a master SSIS package which will have the configuration (say xml) which consists of database connection details and file path details of child packages.what will the best way to achieve the desire results.
Package 1 use File x package 2 use File y package 3 use File z .... package 6 use File a
The parent ssis package will have xml file as configuration which will store the all the six different file details for child packages along with database connection string.Is the above option feasible . or what approach will be the best possible way to achieve the results.Since the Triggering of SSIS package (Master SSIS package) will be from SQL Job
I am new to SQL server 2005 and have a config question:
I am controlling database connection info using XML indirect config - no problems there.
Essentially I am going to have a number of packages that need to use a common file path, that might change from one server to the next, e.g. Server 1: C:sourceFiles versus Server 2: D:sourceFiles. Within this directory the filenames will remain static. So in the flat file connection manager I'd like to use a variable to reflect the folder - but I don't want to have to create this for each package.
So, I thought I would create a system environment variable and create expressions for the connection managers - something like %SOURCE_DIR% + "file.csv" - but this does not evaluate correctly.
So then i though I could use the SQL server configurations table with a configurationFilter SOURCE_DIR and appropriate configuration value - but then how do I access this in the flat file connection manager to create a dynamic file name?
So essentially I want a variable/property available globally to all my packages and potential flat file connection managers that help me to centrally control file path locations.
Hi --I was wondering if this is a bug when I add new data in my table SSIS Confiurations and give wizard a new Configuration filter the package configuration wizard can not see the new values --the old values from the previous configuration are still showing---is there any known workaround or forced refresh I can do
thanks in advance Dave
Background:
SQL Package Configurations are most important because they provide the possibility of a central configuration store for your entire enterprise!!!!!!!! and is in my mind the only way to go
USE [ETLConfiguration] GO /****** Object: Table [dbo].[SSIS Configurations] Script Date: 05/23/2006 13:34:35 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO CREATE TABLE [dbo].[SSIS Configurations]( [ConfigurationFilter] [nvarchar](255) COLLATE Latin1_General_CI_AS NOT NULL, [ConfiguredValue] [nvarchar](255) COLLATE Latin1_General_CI_AS NULL, [PackagePath] [nvarchar](255) COLLATE Latin1_General_CI_AS NOT NULL, [ConfiguredValueType] [nvarchar](20) COLLATE Latin1_General_CI_AS NOT NULL ) ON [PRIMARY]
Hi, I have a package which has 2 file system tasks and 2 data flow tasks all in a for each loop container. if i execute the package from the development studio its executes fine but when i try to run it from the command prompt using the dtexec utility..it just runs upto the first data flow task and then it hangs unexpectedly. Any help on this would be really appreciated.
both variables as you can see are in the top level. the problem is that ChannelCode is string but Status is Integer. if I take out the /SET Package.Variables[Status].Value;1 part from the command line it will work fine. I think th eproblem is down to diffenrt types of variable as there are two types of String and Interger.
any ideas on what the problem migth be. p.s. I am runnning SQL server 2005 Dev edition (without any SP) Cheers
I created a package and stored in SQL server. I am running it as follows
master.dbo.xp_cmdshell dtexec /sq "ABC DEF GHI" /ser "Prod" ABC DEF GHI --> is my package name. When i run this from a job, it runs without error but when i run the above sql i am getting error as
Msg 102, Level 15, State 1, Line 2
Incorrect syntax near '/'. Am i missing some thing, please advice. Is that the problem between spaces in the package name?
I would like to have my SSIS tasks to be transacted, but due to infrastructure issues in our network, cannot utilize DTC, because our SQL Boxes are situated where the DTC cannot communicate. I know we can use execute SQL task with Bein Trans, but I dont think dataflow and all will not be in scope of this and cannot be rollback in case of any error in the underlying tasks.
Hi everybody. I'm trying to run a package, stored in MSDB, using dtexec utility from cmd. I do this using "dtexec /sq Main_Package /ser local /user sa /Password Oracle"
But all I got is: Microsoft (R) SQL Server Execute Package Utility Version 9.00.3042.00 for 32-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved. Started: 12:14:15 Could not load package "Main_Package" because of error 0xC0014062. Description: The LoadFromSQLServer method has encountered OLE DB error code 0x80004005 (Login timeout expired). The SQL statement that was issued has failed. Source: Started: 12:14:15 Finished: 12:14:34 Elapsed: 18.297 seconds
It runs perfectly from "Integration Services" directly(SQL Management Studio), but not from cmd. I tried to run it withuot login and password, the same result
I have a file-deployed package which I need to call N times at once using DTEXEC. When I attempt to execute N instances of the package, the second instance doesn't execute. I can execute different packages at the same time, but never the same package more than once. Does anyone know how to get this scenario working? Just to clarify a bit more - I can call the package as many times as I like as long as I wait until the previous run of DTEXEC is finished. Please help!
I am executing a package via vb.net with package.execute using the IDTSEvents interface inheriting DefaultEvents. I am trying to emulate DTExec functionality. The package runs fine and I am catching events, but I don't get nearly as many info and progress events as are put out by DTExec. From my understanding DTExec also used the managedDTS application object, is that not the case? If it uses this does anyone have an ideas on how it is grabbing more events? For example I don't get these events that DTExec shows ..
Info: 2006-09-26 14:22:27.97 Code: 0x40016041 Source: DWB02130 Description: The package is attempting to configure from the XML file "D:SSISConfigurationsDatabase.dtsConfig". End Info Info: 2006-09-26 14:22:28.05 Code: 0x40016040 Source: DWB02130 Description: The package is attempting to configure from SQL Server using the configuration string ""Configurations Database";"[dbo].[SSISConfigurations]";"DWB02130";". End Info
I also don't get as many progress updates during validation, I basically get 0 and 100% while DTExec shows several 1%, 3%, 10%, etc.