ReUse Common Surrogate Key Pipeline
I have several stage to star (i.e. moving data from a staging table through the key lookups into a fact table) ETL transformations in a single SSIS package. Each fact table has a different set of measures but the identical foreign key set, e.g. ConsultantKey, SubsidiaryKey, ContestKey, ContestParamKey and MonthKey.
Currently I have to replicate the key lookup (Surrogate Key Pipeline, or SKP) for each data flow. If I could cache each dimension one time in the package and reuse it for each stage to fact it would be much more efficient.
Is there a way for me to reuse a common data flow?
View Complete Forum Thread with Replies
Related Forum Messages:
How Do I Write Multiple Pipeline Buffer To Multiple Targets Based On A Calculated Value In The Pipeline Buffer
The scenario is as follows: I have a source with many rows. Each row has a column called max_qty_value. I need to perform a calculation using another column called qty. This calculation is something similar to dividing qty/(ceiling) max_qty_value. Once I have that number I need to write an additional duplicate row for each value from the prior calculation performed. For example, 15/4 = 4. I need to write 4 rows to the same target table as in line information for a purchase order. The multicast transform appears to only support fixed and/or predetermined outputs. How do I design this logic in SSIS to write out dynamic number of rows to a target table. Any ideas would be greatly appreciated. thanks John
View Replies !
What's Surrogate Key?
hi! when i read some reference books about the SQL7.0, i often met 'surrogate key'. what's the surrogate key? what's its funtion? could you give me a good example? thanks very much!
View Replies !
Surrogate Key
Hi gurus can any one tell me what is the best way to use surrogate key (except uniqueidentifier datatype)? how can I use with TSQL?
View Replies !
Log Reuse Help
Dear Folks, I'm getting the error with application like this.............. "the transaction log for database "mydatabase" is full.....to find out why space in the log can not be reused, see the log_reuse_wait_desc column in sys databases. " I've checked with the column, there are totally 3 options.(I've checked with all other databases) 1)checkpoint, 2)log_backup, 3)nothing what can i choose for this error to be suppressed? actually there was log_backup option. please guide me in this regard......... thanks in advace. Vinod
View Replies !
Surrogate Or Composite Key?
The orininal design of my db (part of it...) is the following A JOB has a Number and a Description. Each JOB can have one or two TASKS (min one, max two). Each TASK is identified by the JOB it belongs to and an Index (unique only for the same JOB). Each TASK has one an only one set of INFO1, one and only one set of INFO2, one and only one set of INFO3 etc. A: JOB (JobNum [PK], JobDescription, ...) B: TASK (JobNum [PK] [FKa], Index [PK], TaskDescription, ...) C: INFO1 (JobNum [PK] [FKb], Index [PK] [FKb], ...) D: INFO2 (JobNum [PK] [FKb], Index [PK] [FKb], ...) (There is a reason to keep INFO1, 2 and 3 separate, because eachof them will be linked to different table. This might influence the answer to my real question.) First of all, I wouldn't add any surrogate key for TASK, not to loose the logic behind; plus I'd put an ined on JonMum only, being Index equal to 1 or 2 only, so not selective. The real question is about INFO1 (and 2, 3 etc.) table: should I leave JobNum and Index as PK (consider that the PK of INFo1 will be used as FK for another table), or should I use a surrogate key, like for eaxmple C: INFO1 (Info1ID [PK], JobNum [FKb], Index [FKb], ...) I don't really like this solution. Actually I'd prefer the following C: INFO1 (Info1ID [PK], ...) where Info1ID = JobNum + Index (+ = string concatenation). Any suggestion? Thanks
View Replies !
Surrogate Key Generation
Hi, I'm trying to use the SK script from Donald Farmers book but the code isn't accepted Imports System Imports System.Data Imports System.Math Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper Imports Microsoft.SqlServer.Dts.Runtime.Wrapper Public Class ScriptMain Inherits UserComponent Dim CurrentKey As Integer Public Overrides Sub PreExecute() CurrentKey = CInt(Me.Variables.FILCodesSK) End Sub Public Overrides Sub Input_ProcessInputRow(ByVal Row As Input0Buffer) CurrentKey += 1 Row.SurrogateKey = CurrentKey End Sub End Class There is a problem with the use of the overrides on the Input_ProcessInputRow sub should this be renamed? Cheers, Al
View Replies !
How Do I Reuse A SqlDataAdapter
It's a pretty basic question but I haven't been able to find any examples out there. I dimmed a dataadapter and would like to reuse later in my code (line 3 in the code below). What is the correct syntax to do this? Dim da As New SqlDataAdapter("SELECT * FROM myTable", conn)da.Fill(myDataTable)da.______ ("SELECT * FROM myTable2", conn)da.Fill(myDataTable2)
View Replies !
How Can I / Should I Reuse SqlCacheDependency? Thanks
I can create a SqlCacheDependency, and link it to a cached item in httpcontext cache. When something change, it will remove the cached item from the cache. I think I have to redo the process when that happens - prepare sql command, create SqlCacheDependency and insert the item into cache. Now I only need a notification from my SQL when something changes in one of my table, I don;t need read anything from db, and I think I should find a way to not recreate the SqlCacheDependency object everytime? any suggestion?
View Replies !
Can We Reuse SqlParameter's?
Hi, I am getting the following error: The SqlParameter with ParameterName '@pk' is already contained by another SqlParameterCollection If I could work out what code to post I would, but I can say that I am managing my Sql data in my code by caching small arrays of SqlParameter objects as pulled from the database. If I need to update the DB I change the cached SqlParameter and re-insert it. If I perform a SELECT I check the cache first to see if I already have the SqlParameter. However, currently, I am experiencing the above error when performing a select, then later an update, followed by another update. Would I be correct in saying that a SqlParameter object can only be used once for a database operation and should be discarded? Would I be correct if I said that the SqlCommand object should be discarded? I am barking up the wrong tree entirely? Distressed in DBLand, Matt.
View Replies !
Normal Vs Surrogate/artificial Key?
Hey All, I'm trying to decide what's the 'best' to use. I've been designing and creating database for a while and have pretty much always used a surrogate key and not a normal one. I've finally had some free time to start studying more so in my spare time and read up and come accross a lot of guides, articles and stories that tout that normal keys should be used whenever possible as they're a better identifier and that surrogate keys should only be used when there is not a readily available normal key. Now perhaps I'd be open to accepting that but absolutely every database I come across tends to only use surrogate keys. For example I'm doing an authentication system from scratch and am looking at the User table. Now of course the user name has to be unique, should that be the primary key or should I have a seperate column with a guid or an incrementing int or the like as the primary key? I can certainly see that username could be used. I can also see how it may be easier when looking through the data tables to identify who/what a table is refering to with a surrogate key. However it still seems sort of sloppy, for lack of a better word, to me. Where now I could have somebody's username (or any other piece of data used for this purpose) spread accross a lot of other tables. And while writting this I just thought of the scenario that perhaps somebody needs their username changed, with this method now the ids need to be changed on all the related rows of all the other tables whereas with a surrogate key it wouldn't matter. Anyways I'm mostly looking for opinions on which way to go (not just with the user sample, but more in general).Thanks.
View Replies !
INNER JOIN Using Surrogate ID, Or [Date] BETWEEN?
{CREATE TABLEs and INSERTs follow...}Gents,I have a main table that is in ONE-MANY with many other tables. For example, ifthe main table is named A, there are these realtionships:A-->BA-->CA-->DA-->EWith one field in Common (Person). The tables B, C, D and E are History tables,with Start and End dates. Each person has a Program history (table B, ie), anExperience history (table C, ie), and so on...many differernt types ofhistories, and it may grow from here....table F, G, etc.The included CREATE TABLEs and INSERTs contain tables A, B and C.The problem: Each tblCase (table A) record has a date. When joining all of thehistory tables to tblCase on Person, obviously you get a cross-product of eachhistory unless you specify a WHERE clause that extracts one single record fromeach of the histories (duh...that's the point...to extract a single record fromeach history, because there can only be one value in effect at the time of theCase.)QUESTION: From a performance standpoint, would it behoove me to maintain thesurrogate ***HistoryID from each history table in tblCase, or, assuming theindexes are set up properly, would a WHERE condition for each history besufficient? For example, the following select works as expected:SELECT CasePerson, CaseDate, ProCode, ExpYearFROM tblExperienceHistory INNER JOIN (tblCase INNER JOIN tblProgramHistory ONtblCase.CasePerson = tblProgramHistory.ProPerson) ON tblCase.CasePerson =tblExperienceHistory.ExpPersonWHERE CaseDate BETWEEN ProStartDate and ProEndDateAND CaseDate BETWEEN ExpStartDate and ExpEndDateIt extracts the single record from each history for each person for each case.But I'm afraid of performace with such a scenario.Instead, I could store each ***HistoryID in the table tblCase, and then justjoin on that...no WHERE needed. But the trade-off is that I'd have to buildprocesses to maintain that. ("Hey, when you insert a record into tblCase, makesure to go get each HistoryID from the History tables!" or "If the user changesthe date ranges in one of histories, make sure to update tblCase to match thenew historyID!")Maybe a clustered index on each ***History table on Person/StartDate combinedwith the WHERE clause should perform as well as a real JOIN on surrogateintegers.It seems cheesey to have to resort to surrogate IDs...but the performanceincrease might be worth it. Also, if I go that route, whenever I add a newhistory table, I'd have to change the design of tblCase AND any SPs thatreference it. With the WHERE solution, I'd only have to change the SPs.Comments are welcome! (tblCase grows at 250,000 records per year; the historytables will increase about 1000 records per year)DCMFANCREATE TABLE [dbo].[tblCase] ([CaseID] [char] (5) CONSTRAINT [PK_tblCase] PRIMARY KEY CLUSTERED NOT NULL ,[CaseDate] [smalldatetime] NOT NULL ,[CasePerson] [char] (5) NOT NULL) ON [PRIMARY]GOCREATE TABLE [dbo].[tblExperienceHistory] ([ExperienceHistID] [int] IDENTITY (1, 1) NOT NULL ,[ExpPerson] [char] (5) NOT NULL ,[ExpStartDate] [smalldatetime] NOT NULL ,[ExpEndDate] [smalldatetime] NOT NULL ,[ExpYear] [int] NOT NULL) ON [PRIMARY]GOCREATE TABLE [dbo].[tblProgramHistory] ([ProgramHistID] [int] IDENTITY (1, 1) NOT NULL ,[ProPerson] [char] (5) NOT NULL ,[ProStartDate] [smalldatetime] NOT NULL ,[ProEndDate] [smalldatetime] NOT NULL ,[ProCode] [int] NOT NULL) ON [PRIMARY]GOINSERT INTO [tblCase]([CaseID], [CaseDate], [CasePerson])VALUES('12345', '3/1/03', '00000')INSERT INTO [tblCase]([CaseID], [CaseDate], [CasePerson])VALUES('A1G34', '4/23/03', '00001')INSERT INTO [tblExperienceHistory]([ExpPerson], [ExpStartDate], [ExpEndDate],[ExpYear])VALUES('00000', '1/1/03', '5/19/03', 1)INSERT INTO [tblExperienceHistory]([ExpPerson], [ExpStartDate], [ExpEndDate],[ExpYear])VALUES('00000', '5/20/03', '12/31/03', 2)INSERT INTO [tblExperienceHistory]([ExpPerson], [ExpStartDate], [ExpEndDate],[ExpYear])VALUES('00001', '4/20/03', '11/1/03', 0)INSERT INTO [tblProgramHistory]([ProPerson], [ProStartDate], [ProEndDate],[ProCode])VALUES( '00000', '2/1/03', '9/30/03', '55555')INSERT INTO [tblProgramHistory]([ProPerson], [ProStartDate], [ProEndDate],[ProCode])VALUES( '00000', '10/1/03', '5/1/04', '55555')INSERT INTO [tblProgramHistory]([ProPerson], [ProStartDate], [ProEndDate],[ProCode])VALUES( '00001', '1/1/03', '12/31/03', '55555')
View Replies !
Surrogate Or Composite Primary Key?
My previous post was not really clear, so I'll try again with a (hopefully) better (even if longer) example... Consider the following... A JOB describes the processment of a document. Each document can exist in two versions: English and French. A JOB can have 1 or 2 TASK, each describing the processement of either the English or French version. So we have the following: A: JOB (JobNum [PK], DocReference, StartDate, EndDate, ...) B: TASK (JobNum [PK] [FKa], Version [PK], Priority, ...) that is there is an identifying 1:M (where maxium allowed for M is 2) relationship between JOB and TASK; TASK being identified by JobNum and Version (where the domain for Version is {E, F}). Each TASK may require a TRANSLATION sub_task. Each TASK may require a TYPING sub_task. Each TASK may require a DISTRIBUTION sub_task. For example, for a given doc, the English TASK requires TRANSLATION and DISTRIBUTION, while the French only DISTRIBUTION. That is, there is a 1:1 not-required relationship between TASK and TRANSLATION, TYPING and DISTRIBUTION. So we have the following: A: JOB (JobNum [PK], DocReference, StartDate, EndDate, ...) B: TASK (JobNum [PK] [FKa], Version [PK], Priority, ...) C: TRANSLATION (JobNum [PK] [FKb], Version [PK] [FKb], DueDate, ...) D: TYPING (JobNum [PK] [FKb], Version [PK] [FKb], DueDate, ...) E: DISTRIBUTION (JobNum [PK] [FKb], Version [PK] [FKb], Copies, ...) As you can see I am using the PK of TASK as FK and PK for each of the three SUB_TASKs. To complicate things, each SUB_TASK has one or more assignments. The assignments for each SUB_TASK records different information from the others. So we have... A: JOB (JobNum [PK], DocReference, StartDate, EndDate, ...) B: TASK (JobNum [PK] [FKa], Version [PK], Priority, ...) C: TRANSLATION (JobNum [PK] [FKb], Version [PK] [FKb], DueDate, ...) D: TYPING (JobNum [PK] [FKb], Version [PK] [FKb], DueDate, ...) E: DISTRIBUTION (JobNum [PK] [FKb], Version [PK] [FKb], Copies, ...) F: TRA_ASSIGN (JobNum [PK] [FKc], Version [PK] [FKc], Index [PK], Translator, ...) G: TYP_ASSIGN (JobNum [PK] [FKd], Version [PK] [FKd], Index [PK], Typyst, ...) H: REP_ASSIGN (JobNum [PK] [FKe], Version [PK] [FKe], Index [PK], Pages, ...) that is there is an identifying 1:M relationship between each SUB_TASK and its ASSIGNMENTs, each ASSIGNMENT being identified by the SUB_TASK it belongs to and an Index. I wish I could send a pic of the ER diagram... Maybe there is another and better way to model this: if so, any suggestion? Given this model, should I use for TRANSLATION, TYPING and DISTRIBUTION a surrogate key, instead of using the composite key, like for example: C: TRANSLATION (TranslationID [PK], JobNum [FKb], Version [FKb], DueDate, ...) D: TYPING (TypingID [PK], JobNum [FKb], Version [FKb], DueDate, ...) E: DISTRIBUTION (DistributionID [PK], JobNum [FKb], Version [FKb], Copies, ...) this will "improve" the ASSIGNMENTs tables: F: TRA_ASSIGN (TranslationID [PK] [FKc], Index [PK], Translator, ...) G: TYP_ASSIGN (TypingID [PK] [FKd], Index [PK], Typyst, ...) H: REP_ASSIGN (DistributionID [PK] [FKe], Index [PK], Pages, ...) I could even go further using a surrogate key even for TASK, which leads me to the following: A: JOB (JobNum [PK], DocReference, StartDate, EndDate, ...) B: TASK (TaskID [PK], JobNum [FKa], Version , Priority, ...) C: TRANSLATION (TaskID [PK] [FKb], DueDate, ...) D: TYPING (TaskID [PK] [FKb], DueDate, ...) E: DISTRIBUTION (TaskID [PK] [FKb], Copies, ...) F: TRA_ASSIGN (TaskID [PK] [FKc], Index [PK], Translator, ...) G: TYP_ASSIGN (TaskID [PK] [FKd], Index [PK], Typyst, ...) H: REP_ASSIGN (TaskID [PK] [FKe], Index [PK], Pages, ...) I don't really like this second solution, but I'm still not sure about the first solution, the one with the surrogate key only in the SUB_TASks tables.
View Replies !
Generating Surrogate Key Without IDENTITY
Hello I'm looking for a way of generating the next key value that works in MS and Sybase SQL Servers. Sybase identity columns are a bit dodgy, so... If I have a separate table NextKey (NextKey int) with one row that I update as follows... declare @NextKey int update NextKey set NextKey = NextKey + 1, @NextKey = NextKey + 1 insert into myTable (PrimaryKeyCol, ....) values (@NextKey, ....) are there any problems with concurrency ? As I see it the update will lock the row so different connections will always come up with a different @NextKey value.... Thanks John
View Replies !
Surrogate Key Population On A Select Into
I am performing a Select Into from a #table into a real table that has a surrogate key. If this is in a transaction (or not in one) am I guaranteed that the records inserted will be sequential surrogate key ids? Select * into REALTABLE from MYPOUNDTABLE --40 rows Can I assume that if the first one inserted is id 32 that the last one is 72?
View Replies !
Surrogate Key Population On A Select Into
I am performing a Select Into from a #table into a real table that has a surrogate key. If this is in a transaction (or not in one) am I guaranteed that the records inserted will be sequential surrogate key ids? Select * into REALTABLE from MYPOUNDTABLE --40 rows Can I assume that if the first one inserted is id 32 that the last one is 72?
View Replies !
Stored Procedure Reuse
I wonder if there is a solution for this in SQL 2000 (or do I have towait for SQL 2005)?I am currently in the middle of developing a 'Yahoo' style portal whichwill be rolled out in about 20 or so countries. I have set up in SQLServer one database per country. All the portals have the samefunctionality - but show different data.Is it possible to have a single database which holds storedproceedures, functions and views and have the individual countrydatabases use these?Note: I want to avoid using EXEC sp_executesql.I look forward to some good news on this! Thank you in advance.Dadou.
View Replies !
Reuse Of Conversation Handles
I have read the articles posted online concerning different dialog reuse strategies. Most of them create a new table in the sender to hold dialog ids. I was wondering what is wrong, if anything, with the following approach: Code Block declare @dlg uniqueidentifier select top 1 @dlg = conversation_handle from sys.conversation_endpoints where state IN ('CO') if @dlg is null begin begin dialog conversation @dlg from service [tcp://SFT3DEVSQL01:4022/TyMetrix360Audit/DataSender] to service '//TyMetrix360Audit/DataWriter','386DDD04-7E55-466A-BE83-37EFC20910B9' on contract [//TyMetrix360Audit/Contract] with encryption = off; end ;send on conversation @dlg message type [//TyMetrix360Audit/Message] (@msg) Here I simply select a conversation handle directly from the sys.conversation_endpoints table. Can anyone see any issues with this approach? Thanks in advance....
View Replies !
Reuse Of Report Elements
I am working on a project developing a fairly large number of reports with a team of developers. Many of these reports have common elements and code, such as common headers with user-selectable colors. Additionally, many of the common parts of the reports are at mockup stage currently, and many features will have to be added to the reports as time goes on. We're attempting to create a generalized framework that will minimize the duplication of effort as we develop these reports, and as we go back and modify or fix them later. What is the best way to approach this? My first attempt was to create a report template and base all the reports off of the same template. That was fine for the first pass, but as we need to make changes later, they will not be propagated to the already existing reports. My second attempt was to have each componant of that template reference a subreport, so that changes to the actual report template will be minimized as we go forward. This works great for minimizeing work, but it appears that you lose many features with the use of subreports, and there seems to be a pretty serious performance impact as well. I have posted about one such issue here: Pagination If anyone has pointers about how to go about this, and where I should start, they would be greatly apreciated!
View Replies !
Cannot Reuse Database Name After Deleting Old One
Hi. I'm starting to try out LINQ to SQL, and so I'm using SQL Express (on Vista) to experiment. My problem is that once I create and delete a database, I can never use the name again. If, after deleting the old database, I try to create another one with the same name (say Acct1), I get "Create failed for Database 'Acct1'. An exception occurred while executing a Transact-SQL statement or batch. The logical file name "Acct1" is already in use. Choose a different name. Error: 1828" I am trying to create the database in SQL Server Management Studio Express. That database name does not appear in the list of databases: there is only AdventureWorks and the system databases. The .MDF and .LDF files have been deleted. Not just sent to the recycle bin, but permanently deleted. I have already used up Acct1, Acct2, and Acct3, just to try out different scenarios. Each time, I delete the old database before trying to create a new one with the same name, but I am forced to always supply a new, different name. I have checked the directory with hidden files and system files showing to be sure there is no old file lurking there somewhere. Is there a way to delete these old "logical file names"? I can't even find any reference to their existence except for the message that says they are already in use.
View Replies !
Reuse Of Field Aliases
I have been working with SQL for quiet a while but think this perhaps is a very basic question that has always escaped me: At my work I was exposed to both, MS SQL Server 2000 and Sybase Adaptive Server Anywhere/Sybase SQL Anywhere. Under Sybase I was able to use aliases in other calculations and filters but i have never been able to do the same with SQL. Example: In Sybase I can write this: Select Price * Units as Cost Cost * SalesTax as TotalTaxFrom Invoice Where TotalTax > 3.5 However if i want to do this in MS SQL 2000 i have to go trough Select Price * Units as Cost Price * Units * SalesTax as TotalTax From Invoice Where (Price * Units * SalesTax) > 3.5 In the long run this is costing me a lot of code redundancy, not to mention a debugging nightmare. Is there a way to replicate this alias usage in MS SQL Server?
View Replies !
Reuse Calculated Values
What is the best code pratice to use do the following code, SELECT fo.no as LNum, fo.name as LName, sum(CASE fo.docnome WHEN "In" THEN fo.etotal ELSE 0 END) as In1, sum(CASE fo.docnome WHEN "In2" THEN fo.etotal ELSE 0 END) as In2, sum(In1+In2)/10 as inDec, from fo group by fo.no,fo.name order by fo.name instead of SELECT fo.no as LNum, fo.name as LName, sum(CASE fo.docnome WHEN "In" THEN fo.etotal ELSE 0 END) as In1, sum(CASE fo.docnome WHEN "In2" THEN fo.etotal ELSE 0 END) as In2, ((sum(CASE fo.docnome WHEN "In" THEN fo.etotal ELSE 0 END))+sum(CASE fo.docnome WHEN "In2" THEN fo.etotal ELSE 0 END)))/10 as inDec, from fo group by fo.no,fo.name order by fo.name I cant use functions and procedures. Is there any better and cleaner way to code this, reusing the calculated values?
View Replies !
Surrogate Key As Parameter In Stored Procedure?
I have two tables: countries(country_id integer, country_name string) authors(auth_id integer, country_id integer, auth_name string) ...Where "country_id" in the authors table refers to the same country_id in the countries table. I want a stored procedure to handle the insertion of new rows in the authors table. There are two methods of doing it: 1) CREATE PROCEDURE addAuthor( authorName, countryId ) And 2) CREATE PROCEDURE addAuthor( authorName, countryName ) Now, I like #1 because the implementation is simple -- the calling code simply passes an author name, and a country id and an INSERT INTO statement is called with those parameters INSERT INTO authors( @authorName, @countryId ) I like #1, because it hides the surrogate "id" key from the application calling code. But on the downside, it has more overhead work, because you have to first a) verify a country with that name exists, and b) select that id into a variable. DECLARE id INT; IF EXISTS (select * from countries where country_id = @countryId ) THEN SELECT country_id INTO id FROM countries WHERE country_name = @countryName; END IF; (Sorry I may have the SQL syntax wrong up there, but I was just trying to demonstrate the extra overhead involved). Which approach do you guys think is better?
View Replies !
Scrpting To Genrate Surrogate Keys
This is the code iam using to get the incremental surrogate keys: Imports System Imports System.Data Imports System.Math Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper Imports Microsoft.SqlServer.Dts.Runtime.Wrapper Public Class ScriptMain Inherits UserComponent 'Declare a variable scoped to class ScriptMain Dim counter As Integer Public Sub New() 'This method gets called only once per execution 'Initialise the variable counter = 1093 End Sub 'This method gets called for each row in the InputBuffer Public Overrides Sub Input_ProcessInputRow(ByVal Row As InputBuffer) 'Increment the variable counter += 1 'Output the value of the variable Row.instance = counter End Sub End Class --'Instance' is my surrogate feild name but iam getting an error saying that InputBuffer is not defined ..Any idea? If I want to add two more incrementive fileds ,where i have to add it? Sorry if it sounds silly ,iam very new to this scripting. Thanks Niru
View Replies !
Updating Dimension With Foreign Surrogate Key
Hi, I have a dimension called 'Caller Type' with the following attributes: CallerTypeKey ---- surrogate key CallerTypeID CallerTypeDesc CreatedByKey ---- foreign surrogate key from User Dimension I used Script Task to get the last used key and increment it so i can use it for new records in my dimension. however, my dimension is linked to a User Dimension and I need the surrogate key of that once I insert the new record to CallerType Dimension. How would I do that? cherriesh
View Replies !
Need Help Updating Foreign Surrogate Keys
I am in the process of building a fact table in a staging area. The data in the host system has numerous composite keys, so I have replaced all the composite keys in the dimensions with surrogate keys (integer) which are generated using an identity at load time. When I load the staging (fact) table, I have set the default value of all the foreign keys to 0. What I must do now is update all the foreign key values with the surrogate key values from the dimensions. I'm using an update command and the original gid values from the source system in the where clause...i.e. UPDATE X SET x.key_1 = y.key_1 FROM TableA X WITH (NOLOCK) INNER JOIN TableB Y WITH (NOLOCK) ON x.org_id = y.org_id AND x.bus_id = y.bus_id AND x.prov_gid = y.prov_gid AND x.log_gid = y.loc_gid; This seems to work fine for most tables. However, I am now trying to update a table that has over 10 million rows and approximately 30 foreign keys. The script runs for hours. I ususally stop it after about 8 hours when it still hasn't completed. Since the keys are dynamic and they could possibly change during each load process, I can't add them during the load process. Is there a better way to update these keys. I need to regenerate the fact tables every night and taking this much time to reload a fact table is just not practicle. I've indexed the alternate keys on all the dimensions and have also indexed the gids on the target fact table. Am I doing something wrong? Have I over indexed the target table? Please help! Thanks Jerry
View Replies !
Pipeline
Hi, I want to incorporate this code but I dont know how to import Microsoft.SqlServer.Dts.Pipeline in an Integration Services Project template. I was thinking of putting this code in the script task but still, I cant import Pipeline. Add reference list does not have it as well. Please let me know how to incorporate this code. Thanks! Code: if (ComponentMetaData.RuntimeConnectionCollection["SourceFileConnection"].ConnectionManager != null) { cm = DtsConvert.ToConnectionManager(ComponentMetaData.RuntimeConnectionCollection["SourceFileConnection"].ConnectionManager); if (cm.CreationName == "FILE") { fileUsage = (Microsoft.SqlServer.Dts.Runtime.DTSFileConnectionUsageType)cm.Properties["FileUsageType"].GetValue(cm); if (fileUsage == Microsoft.SqlServer.Dts.Runtime.DTSFileConnectionUsageType.FileExists) { connectionString = ComponentMetaData.RuntimeConnectionCollection["SourceFileConnection"].ConnectionManager.AcquireConnection(transaction).ToString(); if (connectionString == null || connectionString.Length == 0) { throw new Exception("No file name specfiy"); } } else throw new Exception("Incorrect file connection usage type, should be set to exiting file type"); } else throw new Exception("Connection is not a file connection"); } else throw new Exception("Connection is not as assign");
View Replies !
Reuse Numbers In Identity Column
Hello, I have an ID column which is an Identity column, and it counts from 1 upwards. If I have 10 records, I have ID's number 1 to 10. If I then add another record, it will give the ID number 11. Imagine I then delete ID number 9 Is it possible to get the next record I create to take number 9 instead of 12 ?
View Replies !
Help With Sample Code For Ssis Surrogate Key Transform
I am trying to write a ssis surrogate key data transform, my problem is I can't find an example how to add a column to the incoming columns and add some data to it. If anyone has a sample, can you please post it. I found a script option that works but I would like an actual transform. Thanks
View Replies !
Split Pipeline
This is probably obvious, but how do I split a pipeline. I.e. I've got a data source with 200 columns - I need to split this into 20 pipelines each containing 10 of the original columns.
View Replies !
Reuse Existing Connection String For TableAdapter?
I've added a typed DataSet and dragged a table across from the server explorer. When I click configure on the table adapter, then click previous back to the "Choose Your Data Connection" dialog. The only option is the new connection that was just created when I added the sql server to the server explorer. Is there anyway to reuse my existing web.config connection string? My goal is to have a single connection string in my web.config. Thanks. -David
View Replies !
Connection Manager Reuse For Oracle And SQL Databases
I'm trying to implement a configurable way of executing a group of SQL statements using either an Oracle or SQL database as the source for the data. I'm currently building a connection string with a Script Task and then assigning it to a package variable so that it can be used by a connection manager but this only works with a SQL OLEDB connection. I'm holding all the connection details in a database table, hence the need for the scripting but would it be possible to extend the task to reuse a single connection manager by changing it's properties?
View Replies !
The Client Was Unable To Reuse A Session With SPID XX
Hello, We have a recurring problem with one of our MS SQL Servers (MS SQL Server 2005 SP 1). The SQL Server serves as a database server for a Web application with approximately 1500 users, the server is dedicated for the DataBase server. The scenario is that the database servers cpu peaks at 100% and stays there until restart of the SQL Server service. The error message from the applicatoion log: Event Type: Error Event Source: MSSQLSERVER Event Category: (2) Event ID: 18056 The client was unable to reuse a session with SPID 94, which had been reset for conection pooling. This error may have been caused by an earlier operation failing. Check the error logs for failed operations immediately before this error message. SPID varies. Anybody that knows anything why this error occurs? Regards John
View Replies !
Populating A Surrogate Key Inside Data Flow Task
Hi, I have tables like the one below for my Stage and dimension tables: Stage Table accountid name address Dimension Table accountkey ---- surrogate key (DW key) accountid ---- business key (transaction's primary key) name address I used slowly changing dimension to detect the changes for the records inside my Dimension table. But I had a problem when a new record exists in the stage table. The accountkey is set as the primary key and it gets its value from a different table which stores the last account key that was created. I cannot load all the changes unless i have a business key. Is there a way that i can get the "last key" from a different table in the data flow area and then supply it together with the other fields in the new output branch of the slowly changing dimension? cherriesh thanks!
View Replies !
SQL 2K5 SSIS DTS.Pipeline Errors
We have deployed an SSIS package successfully to production. We needed to apply SP1 to fix a different issue and now have encountered a new problem. We have numerous Data Reader Sources in different Data Flow Tasks that connect to a IBM iSeries (DB2) source. Pretty simple extracts that have worked fine in the past. They pump the data into staging tables on the SQL2K5 instance running the package (64-bit). After we applied SP1 however, all of the Data Reader tasks fail AFTER they successfully copy the records with the following error. [iSeries Invoice Details [1]] Error: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper90 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer90[] buffers, IntPtr ppBufferWirePacket) If I delete the source and destination and recreate identical transforms, they work fine, but I don't feel like rebuilding all of the extracts. Any ideas! The problem occurs in all environments that we've tried. TIA, Michael Shugarman P.S. I just tried the SP2 CTP, but that doesn't fix the problem.
View Replies !
Remove Duplicates Within Pipeline
I have a situation where we get XML files sent daily that need uploading into SQL Server tables, but the source system producing these files sometimes generates duplicate records in the file. The tricky part is, that the record isn't entirely duplicated. What I mean, is that if I look for duplicates by grouping the key columns, having count(*) > 1, I find which ones are duplicates, but when I inspect the data on these duplicates, the other details in the remaining columns may differ. So our rule is: pick the first record, toss the rest of the duplicates. Because we don't sort on any columns during the import, the first record kept of the duplicates is arbitrary. Again, we can't tell at this point which of the duplicated records is more correct. Someday down the road, we will do this research. Now, I need to know the most efficient way to accomplish this in SSIS. If it makes it easier, I could just discard all the duplicates, since the number of them is so small. If the source were a relational table, I could use a SQL statement to filter the records to remove the duplicates, but since the source is an XML file, I don't know how to filter these out in the pipeline, since the file has to be aggregated to search for dups. Thanks Kory
View Replies !
DTS.Pipeline.1 In SQL Server 2008
Hi I have an existing application that programmatically builds SSIS 2005 packages. I'm trying to get to working with the February CTP of SQL Server 2008. Having changed all the 2005 references to 2008 references and things like IDTSComponentMetaData90 to IDTSComponentMetaData100, my application compiles okay now, but hits a problem when it tries to create a Data Flow task. The code which worked fine before (and seems to still be the recommended way in Books Online is): Code Snippet Dts.TaskHost myMainPipe = (Dts.TaskHost)container.Add("DTS.Pipeline.1"); However, this now produces the exception: Cannot create a task with the name "DTS.Pipeline.1". Verify that the name is correct. Should I be using a different moniker now? I took a stab at "DTS.Pipeline.2", but that didn't make a difference. Thanks, Andrew
View Replies !
DTS.Pipeline Information - Can I Access This?
Is there any way I can capture the below information? I want to capture this to get the no of rows processed by each transformation. [DTS.Pipeline] Information: "component "abc" (3798)" wrote 2142 rows. [DTS.Pipeline] Information: "component "xyz" (4223)" wrote 1026 rows. [DTS.Pipeline] Information: "component "abc2" (4324)" wrote 7875 rows. Thanks
View Replies !
SSIS [DTS.Pipeline] Error
Hi I have created a simple SSIS project on my client that carries out 4 Data Flow tasks, each one copying a few hundred rows from an Oracle 10.0.2 database. This works OK and will also run in debug mode fine. I have copied the package to the file system on our development server and get the following error when in debug mode:- [DTS.Pipeline] Information: Validation phase is beginning. Progress: Validating - 0 percent complete [OLE DB Source [1]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Server.user" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed. [DTS.Pipeline] Error: component "OLE DB Source" (1) failed validation and returned error code 0xC020801C. Progress: Validating - 50 percent complete [DTS.Pipeline] Error: One or more component failed validation. Error: There were errors during task validation. Validation is completed [Connection manager "Server.user"] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Oracle" Hresult: 0x80004005 Description: "Error while trying to retrieve text for error ORA-01019 ". Validation is completed If you go to the source of each flow task and select preview you can retreive the data. Thanks Paul
View Replies !
Microsoft.SqlServer.Dts.Pipeline.BlobColumn
I am using Component Script to do - Transforming Comma-delimited list row data to column and I want to use MessageBox to see the value Dim DataPnts As String DataPnts = Row.DataPnts.ToString() -- this is my input column (data type = text in Source table and I put as Unicode string [DT_WSTR] in Output column) MessageBox.Show(DataPnts, "DataPoints1", MessageBoxButtons.OK) ---and why can't I see it. It gives me some message with Microsoft.SqlServer.Dts.Pipeline.BlobColumn. Why? Values = DataPnts.Split(CChar(",")) Please point me to more info on how to do transform Comma-delimited list row data to column. Thanks.
View Replies !
Understanding What This Dts.Pipeline ERROR Means
Im am pulling down table called PRV from another server throught an ODBC connection in my SSIS package. I have the source and destination task all set up. I get this error when i run the packag. Most of the time, the error is pretty self explanatory but this one is .....beyond me. Any ideas. Error: 0xC02090F5 at PRV TABLE FROM CYPRESS, PRV SOURCE [1]: The component "PRV SOURCE" (1) was unable to process the data. Error: 0xC0047038 at PRV TABLE FROM CYPRESS, DTS.Pipeline: The PrimeOutput method on component "PRV SOURCE" (1) returned error code 0xC02090F5. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. Error: 0xC0047021 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Thread "SourceThread0" has exited with error code 0xC0047038. Error: 0xC0047039 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown. Error: 0xC0047021 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Thread "WorkThread0" has exited with error code 0xC0047039. Information: 0x40043008 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Post Execute phase is beginning. Information: 0x402090DF at PRV TABLE FROM CYPRESS, PRV Destination [4076]: The final commit for the data insertion has started. Error: 0xC0202009 at PRV TABLE FROM CYPRESS, PRV Destination [4076]: An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Arithmetic overflow occurred.". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Arithmetic overflow error converting IDENTITY to data type smallint.". Information: 0x402090E0 at PRV TABLE FROM CYPRESS, PRV Destination [4076]: The final commit for the data insertion has ended. Error: 0xC0047018 at PRV TABLE FROM CYPRESS, DTS.Pipeline: component "PRV Destination" (4076) failed the post-execute phase and returned error code 0xC0202009. Information: 0x40043009 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Cleanup phase is beginning. Information: 0x4004300B at PRV TABLE FROM CYPRESS, DTS.Pipeline: "component "PRV Destination" (4076)" wrote 113136 rows. Task failed: PRV TABLE FROM CYPRESS
View Replies !
Microsoft.SqlServer.Dts.Pipeline.DoesNotFitBufferException
Hi I have a SSIS project that has one parent package and three child packages. When I run the project on my development machine in debug mode it works fine. Also if i run the packages using dtexec on my development machine it still works fine. However the problem comes in when I try and run the project using dtexec on the staging server i get the following error: Microsoft.SqlServer.Dts.Pipeline.DoesNotFitBufferException: The value is too large to fit in the column data area of the buffer. does anyone have any idea how to fix this please? thanks G
View Replies !
Would You Like The Ability To Hide Columns In The Pipeline?
Alot of people complain, legitamately, that they wish to remove columns from the SSIS pipeline that they know are not going to be used again. This would help to avoid the "clutter" that can exist when there are alot of columns in the pipeline. If you are one of those people then click-through below, vote and (most importantly) add a comment. The more people that do that - the more likely we are to get this functionality in a future version. SSIS: Hide columns in the pipeline https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=252462 -Jamie
View Replies !
Intercept Pipeline Events Programmatically
Hello, I'm wish to receive pipeline events fired by a SSIS package. I execute the package successufully with the following code (c#): MyEventListener eventListener = new XplorerEventListener(); DtsApplication app = new DtsApplication(); Package pkg = app.LoadPackage("c: est.dstx", null); pkg.Execute(null, null, eventListener, null, null); MyEventListener is inherited from DefaultEvents, overriding all OnXXX methods. It works perfectly, however I cannot intercept the following events: - PipelineExecutionTrees - PipelineExecutionPlan - PipelineExecutionInitialization - BufferSizeTuning - PipelineInitialization Anyone knows how to catch those pipeline events? TIA, Paolo.
View Replies !
|
TRACKER
