Remove Duplicates Within Pipeline
I have a situation where we get XML files sent daily that need uploading into SQL Server tables, but the source system producing these files sometimes generates duplicate records in the file. The tricky part is, that the record isn't entirely duplicated. What I mean, is that if I look for duplicates by grouping the key columns, having count(*) > 1, I find which ones are duplicates, but when I inspect the data on these duplicates, the other details in the remaining columns may differ. So our rule is: pick the first record, toss the rest of the duplicates.
Because we don't sort on any columns during the import, the first record kept of the duplicates is arbitrary. Again, we can't tell at this point which of the duplicated records is more correct. Someday down the road, we will do this research.
Now, I need to know the most efficient way to accomplish this in SSIS. If it makes it easier, I could just discard all the duplicates, since the number of them is so small.
If the source were a relational table, I could use a SQL statement to filter the records to remove the duplicates, but since the source is an XML file, I don't know how to filter these out in the pipeline, since the file has to be aggregated to search for dups.
Thanks
Kory
View Complete Forum Thread with Replies
Related Forum Messages:
Remove Duplicates
I have a query which gives the following output, How can i get a output like this: QUERY COL1COL2COL3 A1AAGG A1BBHH A1CCJJ B1DDKK B1EELL B1FFMM OUTPUT COL1COL2COL3 A1AAGG BBHH CCJJ B1DDKK EELL FFMM
View Replies !
Remove Neighbouring Duplicates
Welcome,how can I alter following table in order to reduce neighbouringduplicates (symbol, position, quantity, price).Nr Symbol Position QuantityPrice Date1. wz9999b 1 1.02500.0 2007-05-09 08:09:42.6532. wz9999b 2 12.02500.0 2007-05-09 08:09:42.6533. wz9999b 1 100.02590.0 2007-05-10 15:47:04.1404. PZ0008VX 1 2280.8842090.55000000000022007-05-1612:43:12.4035. PZ0008VX 1 2280.8842102.05000000000022007-05-1612:45:27.4206. wz9999b 1 0.0012500.0 2007-05-18 09:47:16.0337. wz9999b 1 0.0012500.0 2007-05-18 09:47:53.2708. wz9999b 1 1.01.0 2007-05-22 12:35:07.8939. PZ0008VX 1 2280.8842102.05000000000022007-05-2409:38:26.16010. PZ0008VX 1 2280.8842102.05000000000022007-05-2409:38:38.80011. wz9999b 1 0.001 2500.02007-05-24 12:35:07.20712 wz9999b 1 0.002 2500.02007-05-24 12:35:14.98713. wz9999b 1 0.001 2500.02007-05-24 12:38:07.207In the result set I would like to get the rows number 6 and 10.Any suggestions??
View Replies !
Remove Duplicates Using Trigger?
I want to create a trigger on a table to do the following: Using an ID value, if any rows with the ID to be inserted exists, then insert those rows into a dupe table, then remove those rows from the original table, than complete the insert of the row to be inserted: PSUEDO-code (test tables) t_data_orig id as unique number Primary data1 as varchar data2 as varchar updDate as date t_data_dupes id as number (dupes allowed) data1 as varchar data2 as varchar updDate as date set a trigger on t_data_orig /*If a record exists for this id, move the orignal record to the dupe table*/ select into t_data_dupes id, data1,data2,updDate from t_data_orig where inserted.id = id /*Delete the duplicates from the orignal table*/ delete t_data_orig where id = inserted.id I am coming from Oracle so am unfamiliar with SQL Server syntax. In Oracle, this type of trigger would cause a mutating table data error. Can this be done in SQL Server?
View Replies !
Remove Duplicates On A Table
Hi All I have the dbo.OperatingHour It has many duplicates and I want to remove duplicates permanently The statement below works but when I open the table there are no changes Insert into OperatingHour(Weekdays, Wednesdays, Fridays,Saturdays, [Sundays/Public Holidays]) (SELECT DISTINCT Weekdays, Wednesdays, Fridays,Saturdays, [Sundays/Public Holidays] FROM OperatingHour)
View Replies !
Can Someone Proofread My 'remove Duplicates' Script?
DELETE FROM tblContacts WHERE tblContacts.ID IN( SELECT F.ID FROM tblContacts AS F WHERE Exists ( SELECT email, Count(ID) FROM tblContacts WHERE tblContacts.email = F.email GROUP BY tblContacts.email HAVING Count(tblContacts.ID) > 1 ) ) AND tblContacts.ID NOT IN( SELECT Min(ID) FROM tblContacts AS F WHERE Exists ( SELECT email, Count(ID) FROM tblContacts WHERE tblContacts.email = F.email GROUP BY tblContacts.email HAVING Count(tblContacts.ID) > 1 ) GROUP BY email ) I readily admit that I've shamelessly copied 'n pasted this from a tutorial and then taken a stab at tweaking it for my own ends. But I really don't understand what it's doing. Really, all I want to know is that it will remove records with duplicate email fields. But I could also do with confirming - looking at the "SELECT Min(ID)" bit - does that mean that if it finds a duplicate, it'll delete the latest-added one? And if so, that changing it to remove the earliest-added one is simply a case of changing MIN to MAX? Thanks :)
View Replies !
Custom Property For Remove Duplicates Transform Input Row
Im working through the MS example of "removeDuplicates". I cant seem to figure out how to add custom property for input column. I added the helper method: private static void AddIsKeyCustomPropertyToInput(IDTSInput90 input, object value) { IDTSCustomProperty90 isKey = input.CustomPropertyCollection.New(); isKey.Name = "IsKey"; isKey.Value = value; } I call it from: public override void ProvideComponentProperties() { //... AddIsKeyCustomPropertyToInput(input, false); //... } public override void ReinitializeMetaData() { IDTSInput90 input = ComponentMetaData.InputCollection[0]; if (input.CustomPropertyCollection.Count == 0) { AddIsKeyCustomPropertyToInput(input, false); } // ... } However when I deployed it and added the component to SSIS package - I cant see the Custom Column "IsKey" in the input column properties window. What am I missing - please help
View Replies !
Duplicates Again! UNION Join - Remove Records With Column Diff.
Hello All, We all were new at one point.... any help is appreciated. Objective: Combining two 49,000 row tables and remove records where there is only 1 column difference. (keeping the specified column value removing the one with a blank.) Reason: I have 2 people going through a list, coding a specific column with a single letter value. They both have different progress on each sheet. Hence I am trying to UNION them and have a result of their combined efforts without duplicates. My progress/where I'm stuck: Here is my first query/union: SELECT * FROM [Eds table] UNION SELECT * FROM [Vickis table]; As shown above, I have unioned these 2 tables and my results removed th obvious whole record duplicates, but since 1 column is different on these, a union without criteria considers them unique..... an example of duplicates that I must remove are as follows: 142301 - Product 5000 - 150# - S (Keep) 142031 - Product 5000 - 150# - "" <--- Blank (Remove) I am trying to run another query on my first query results so I don't mess my first query up. Here it is: SELECT DISTINCT [Prod #], [Prod Name], [Prod Description], [Product Type] FROM [Combined Tables] WHERE [Product Type]<>" "; Please Help! Thank you in advance. -------------------- 5 minutes away from pulling my last one! BaldNAskewed
View Replies !
Sort Component, Remove Duplicates, Comparison Flags - Ignore Case
What happens when you add the Ignore Case flag into the mix? I'm having a hell of a time - I'm dealing with an SCD situation using TableDifference component and I have both existing dimensions and new data coming in, each go through identical Case-Insensitive/Sort with remove duplicates, but I'm getting identical new and deleted records detected - I think because of ordering issues. I'm still trying to whittle the test case down, but I think data from all around the records I'm investigating seems to get sorted in between them, so I'm having trouble getting a small test case built. I think the mixed case data is the root of the problem, and I think the design is bad, but before I go back to the technical lead, I need to understand enough to show that you cannot take two pipelines sorted and de-duped case-insensitively and then do a case-sensitive table difference operation.
View Replies !
How Do I Write Multiple Pipeline Buffer To Multiple Targets Based On A Calculated Value In The Pipeline Buffer
The scenario is as follows: I have a source with many rows. Each row has a column called max_qty_value. I need to perform a calculation using another column called qty. This calculation is something similar to dividing qty/(ceiling) max_qty_value. Once I have that number I need to write an additional duplicate row for each value from the prior calculation performed. For example, 15/4 = 4. I need to write 4 rows to the same target table as in line information for a purchase order. The multicast transform appears to only support fixed and/or predetermined outputs. How do I design this logic in SSIS to write out dynamic number of rows to a target table. Any ideas would be greatly appreciated. thanks John
View Replies !
How Do I Clean Up The SQL Server (ctp) From ADD/REMOVE Program Without The Change/remove Button
I have uninstalled the CTP version of the SQL Server express so that I can install the released version but CTP version is still listed in the add/remove program list but without the change/remove button. I have been to different sites to find information on cleaning this up and I have ran all the uninstall tool I can find but the problem still prevails. I cannot install the released version without completely getting rid of the CTP version. Please help anyone. Thanks deebeez1
View Replies !
Pipeline
Hi, I want to incorporate this code but I dont know how to import Microsoft.SqlServer.Dts.Pipeline in an Integration Services Project template. I was thinking of putting this code in the script task but still, I cant import Pipeline. Add reference list does not have it as well. Please let me know how to incorporate this code. Thanks! Code: if (ComponentMetaData.RuntimeConnectionCollection["SourceFileConnection"].ConnectionManager != null) { cm = DtsConvert.ToConnectionManager(ComponentMetaData.RuntimeConnectionCollection["SourceFileConnection"].ConnectionManager); if (cm.CreationName == "FILE") { fileUsage = (Microsoft.SqlServer.Dts.Runtime.DTSFileConnectionUsageType)cm.Properties["FileUsageType"].GetValue(cm); if (fileUsage == Microsoft.SqlServer.Dts.Runtime.DTSFileConnectionUsageType.FileExists) { connectionString = ComponentMetaData.RuntimeConnectionCollection["SourceFileConnection"].ConnectionManager.AcquireConnection(transaction).ToString(); if (connectionString == null || connectionString.Length == 0) { throw new Exception("No file name specfiy"); } } else throw new Exception("Incorrect file connection usage type, should be set to exiting file type"); } else throw new Exception("Connection is not a file connection"); } else throw new Exception("Connection is not as assign");
View Replies !
Split Pipeline
This is probably obvious, but how do I split a pipeline. I.e. I've got a data source with 200 columns - I need to split this into 20 pipelines each containing 10 of the original columns.
View Replies !
Unable To Remove SQL Instance In Add/Remove Program
I need help, I am having a hard time removing my SQL instance inside the Add/Remove program. After i select the SQL Instance name and then I tried to remove it but it won't allow me to delete it. There isn't any error message or whatsoever. Actually, when i try to log it in my SQL Management studio, that certain sql instance name is not existing according to the message box. Is there any way to remove the Sql Instance in my system? I appreciate your help, Thanks IS Support
View Replies !
SQL 2K5 SSIS DTS.Pipeline Errors
We have deployed an SSIS package successfully to production. We needed to apply SP1 to fix a different issue and now have encountered a new problem. We have numerous Data Reader Sources in different Data Flow Tasks that connect to a IBM iSeries (DB2) source. Pretty simple extracts that have worked fine in the past. They pump the data into staging tables on the SQL2K5 instance running the package (64-bit). After we applied SP1 however, all of the Data Reader tasks fail AFTER they successfully copy the records with the following error. [iSeries Invoice Details [1]] Error: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers) at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper90 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer90[] buffers, IntPtr ppBufferWirePacket) If I delete the source and destination and recreate identical transforms, they work fine, but I don't feel like rebuilding all of the extracts. Any ideas! The problem occurs in all environments that we've tried. TIA, Michael Shugarman P.S. I just tried the SP2 CTP, but that doesn't fix the problem.
View Replies !
DTS.Pipeline.1 In SQL Server 2008
Hi I have an existing application that programmatically builds SSIS 2005 packages. I'm trying to get to working with the February CTP of SQL Server 2008. Having changed all the 2005 references to 2008 references and things like IDTSComponentMetaData90 to IDTSComponentMetaData100, my application compiles okay now, but hits a problem when it tries to create a Data Flow task. The code which worked fine before (and seems to still be the recommended way in Books Online is): Code Snippet Dts.TaskHost myMainPipe = (Dts.TaskHost)container.Add("DTS.Pipeline.1"); However, this now produces the exception: Cannot create a task with the name "DTS.Pipeline.1". Verify that the name is correct. Should I be using a different moniker now? I took a stab at "DTS.Pipeline.2", but that didn't make a difference. Thanks, Andrew
View Replies !
DTS.Pipeline Information - Can I Access This?
Is there any way I can capture the below information? I want to capture this to get the no of rows processed by each transformation. [DTS.Pipeline] Information: "component "abc" (3798)" wrote 2142 rows. [DTS.Pipeline] Information: "component "xyz" (4223)" wrote 1026 rows. [DTS.Pipeline] Information: "component "abc2" (4324)" wrote 7875 rows. Thanks
View Replies !
SSIS [DTS.Pipeline] Error
Hi I have created a simple SSIS project on my client that carries out 4 Data Flow tasks, each one copying a few hundred rows from an Oracle 10.0.2 database. This works OK and will also run in debug mode fine. I have copied the package to the file system on our development server and get the following error when in debug mode:- [DTS.Pipeline] Information: Validation phase is beginning. Progress: Validating - 0 percent complete [OLE DB Source [1]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Server.user" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed. [DTS.Pipeline] Error: component "OLE DB Source" (1) failed validation and returned error code 0xC020801C. Progress: Validating - 50 percent complete [DTS.Pipeline] Error: One or more component failed validation. Error: There were errors during task validation. Validation is completed [Connection manager "Server.user"] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for Oracle" Hresult: 0x80004005 Description: "Error while trying to retrieve text for error ORA-01019 ". Validation is completed If you go to the source of each flow task and select preview you can retreive the data. Thanks Paul
View Replies !
Microsoft.SqlServer.Dts.Pipeline.BlobColumn
I am using Component Script to do - Transforming Comma-delimited list row data to column and I want to use MessageBox to see the value Dim DataPnts As String DataPnts = Row.DataPnts.ToString() -- this is my input column (data type = text in Source table and I put as Unicode string [DT_WSTR] in Output column) MessageBox.Show(DataPnts, "DataPoints1", MessageBoxButtons.OK) ---and why can't I see it. It gives me some message with Microsoft.SqlServer.Dts.Pipeline.BlobColumn. Why? Values = DataPnts.Split(CChar(",")) Please point me to more info on how to do transform Comma-delimited list row data to column. Thanks.
View Replies !
Understanding What This Dts.Pipeline ERROR Means
Im am pulling down table called PRV from another server throught an ODBC connection in my SSIS package. I have the source and destination task all set up. I get this error when i run the packag. Most of the time, the error is pretty self explanatory but this one is .....beyond me. Any ideas. Error: 0xC02090F5 at PRV TABLE FROM CYPRESS, PRV SOURCE [1]: The component "PRV SOURCE" (1) was unable to process the data. Error: 0xC0047038 at PRV TABLE FROM CYPRESS, DTS.Pipeline: The PrimeOutput method on component "PRV SOURCE" (1) returned error code 0xC02090F5. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. Error: 0xC0047021 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Thread "SourceThread0" has exited with error code 0xC0047038. Error: 0xC0047039 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown. Error: 0xC0047021 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Thread "WorkThread0" has exited with error code 0xC0047039. Information: 0x40043008 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Post Execute phase is beginning. Information: 0x402090DF at PRV TABLE FROM CYPRESS, PRV Destination [4076]: The final commit for the data insertion has started. Error: 0xC0202009 at PRV TABLE FROM CYPRESS, PRV Destination [4076]: An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Arithmetic overflow occurred.". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80004005 Description: "Arithmetic overflow error converting IDENTITY to data type smallint.". Information: 0x402090E0 at PRV TABLE FROM CYPRESS, PRV Destination [4076]: The final commit for the data insertion has ended. Error: 0xC0047018 at PRV TABLE FROM CYPRESS, DTS.Pipeline: component "PRV Destination" (4076) failed the post-execute phase and returned error code 0xC0202009. Information: 0x40043009 at PRV TABLE FROM CYPRESS, DTS.Pipeline: Cleanup phase is beginning. Information: 0x4004300B at PRV TABLE FROM CYPRESS, DTS.Pipeline: "component "PRV Destination" (4076)" wrote 113136 rows. Task failed: PRV TABLE FROM CYPRESS
View Replies !
Microsoft.SqlServer.Dts.Pipeline.DoesNotFitBufferException
Hi I have a SSIS project that has one parent package and three child packages. When I run the project on my development machine in debug mode it works fine. Also if i run the packages using dtexec on my development machine it still works fine. However the problem comes in when I try and run the project using dtexec on the staging server i get the following error: Microsoft.SqlServer.Dts.Pipeline.DoesNotFitBufferException: The value is too large to fit in the column data area of the buffer. does anyone have any idea how to fix this please? thanks G
View Replies !
Would You Like The Ability To Hide Columns In The Pipeline?
Alot of people complain, legitamately, that they wish to remove columns from the SSIS pipeline that they know are not going to be used again. This would help to avoid the "clutter" that can exist when there are alot of columns in the pipeline. If you are one of those people then click-through below, vote and (most importantly) add a comment. The more people that do that - the more likely we are to get this functionality in a future version. SSIS: Hide columns in the pipeline https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=252462 -Jamie
View Replies !
Intercept Pipeline Events Programmatically
Hello, I'm wish to receive pipeline events fired by a SSIS package. I execute the package successufully with the following code (c#): MyEventListener eventListener = new XplorerEventListener(); DtsApplication app = new DtsApplication(); Package pkg = app.LoadPackage("c: est.dstx", null); pkg.Execute(null, null, eventListener, null, null); MyEventListener is inherited from DefaultEvents, overriding all OnXXX methods. It works perfectly, however I cannot intercept the following events: - PipelineExecutionTrees - PipelineExecutionPlan - PipelineExecutionInitialization - BufferSizeTuning - PipelineInitialization Anyone knows how to catch those pipeline events? TIA, Paolo.
View Replies !
Retrieves The Information About The Pipeline Components
Dear Experts, I can look the values of the proprieties in each PipelineComponentInfo, for example: ComponentType: Transform CreationName: DTSTransform.Merge.1 Description: Merge Transformation FileName: C:Program FilesMicrosoft SQL Server90DTSPipelineComponentsTxMerge.dll FileNameVersionString: 2000.90.1049.0 IconFile: C:Program FilesMicrosoft SQL Server90DTSPipelineComponentsTxMerge.dll IconResource: -201 ID: {08AE886A-4124-499C-B332-16E3299D225A} Name: Merge NoEditor: False ShapeProgID: UITypeName: Microsoft.DataTransformationServices..... but I don't know what means the proprieties: FileName, FileNameVersionString, IconFile, IconResource, NoEdit, ShapeProgID and UITypeName... Can anyone helps Me? Thanks Francesco
View Replies !
SSIS DTS.Pipeline To MSAccess Databse
I cannot get a simple package to execute a data pump to an access database from SQL2005. I have tried it in both SSIS and by running the Export data function. I have been able to write to this database in the past using dtp in SQL2000 but I am not able to write to it using SQL2005. What is the deal with the new SSIS? Does anybody have any ideas I can try to get my export to work. I have many more to do and I have to migrate over all of my SQL 2000 DTS packages to SQL2005 and some export to MSAccess. This is the only error message I can find: [DTS.Pipeline] Information: "component "OLE DB Destination 1" (2196)" wrote 0 rows. Edit: I found more errors in the debug section and a post here that discussed the problem as they had run into it. I was able to use part of that and some more research in order to tackle my problem. I would still be interested in finding out why I suddenly had this problem arise after I upgraded to SQL2005. This is going to be a real pain as apparently SQL2005 treats NULL as Zero Length and now all of my databsaes that had that set in access will have to be modified to deal with this in the export.
View Replies !
ReUse Common Surrogate Key Pipeline
I have several stage to star (i.e. moving data from a staging table through the key lookups into a fact table) ETL transformations in a single SSIS package. Each fact table has a different set of measures but the identical foreign key set, e.g. ConsultantKey, SubsidiaryKey, ContestKey, ContestParamKey and MonthKey. Currently I have to replicate the key lookup (Surrogate Key Pipeline, or SKP) for each data flow. If I could cache each dimension one time in the package and reuse it for each stage to fact it would be much more efficient. Is there a way for me to reuse a common data flow?
View Replies !
Small Necessary Changes Missing In SP2. Any Idea If This Is In Pipeline
1. i find it hard to believe that I cannot use stylesheets for textboxes or for that matter anything. I have to manually go and edit every textbox. 2. I have to define at t global level if a link opens in a new window or same window. Anyone has ideas if this is even in the pipeline to be done?. I had asked the same questions on the release of SQL Server RS and that time it was thought that it would already be there in SP1/SP2. But i do not see it. Or anyone has ideas how to achieve the above?
View Replies !
Transformation Object To Filter Data In Pipeline
Hi, I have some data coming through pipeline and I wanna add some component at some point to pass on only selected rows based on conditions to the objects onwards. My opinion is I should use conditional split object, but Please suggest me something if you know better. Thanks, Fahad
View Replies !
How To Update A Dimension Column With The Pipeline Tasks
I have been working with DTS and ETL in data warehousing projects for several years and my question is this. You can only update a dimension column with SSIS by using TSQL-update statements. There is no way to do this except issuing TSQL from the control flow or the data flow? This subject is not mentioned in Wrox SSIS book nore in Kirk Haseldens book. When you run the SCD task in the data flow you will get an OLEDB command that actually do this, issue a TSQL-statement. Is this correct? Regards Thomas Ivarsson
View Replies !
[DTS.Pipeline] Information: Pre-Execute Phase Is Beginning
Hi, I have a SSIS package which pumps data from one server to other without any additional steps. There are 11 tables for which data is transferred. And this packages runs fine on two different environments but fails in one environment i.e. on SIT. It doesn't throw any error and every time stops at the below step [DTS.Pipeline] Information: Pre-Execute phase is beginning. Progress: Pre-Execute - 0 percent complete Progress: Pre-Execute - 1 percent complete Progress: Pre-Execute - 2 percent complete Progress: Pre-Execute - 3 percent complete Progress: Pre-Execute - 4 percent complete Progress: Pre-Execute - 5 percent complete Progress: Pre-Execute - 6 percent complete Progress: Pre-Execute - 7 percent complete It doesn't complete neither throws an error. Any pointers on what the problem could be Thanks
View Replies !
Imports Microsoft.sqlserver.dts.pipeline Does Not Work
I have been trying to follow/implement the examples in the following help topics (thanks to Jamie for these links). Building Packages Programmatically (http://msdn2.microsoft.com/en-us/library/ms345167.aspx) Connecting Data Flow Components Programmatically (http://msdn2.microsoft.com/en-us/library/ms136086.aspx) The problem I am having is that MainPipe is not recognized as a valid type in my Script task, even though I have the imports statements that are listed in the example. I get the message "Error 30002: Type 'MainPipe' is not defined". The other and related problem is that when I type "imports microsoft.sqlserver.dts", the intellisense offers only two choices: {}Runtime and {}Tasks. I don't see any choice for Pipeline. Can anyone tell what I am missing? It seems to be some kind of configuration/installation issue, but I have no idea how to resolve it. I have tried this on 3 different machines, with both the RTM SQL 2005 standard edition, and with SP2 installed, all with the same result. Any help is appreciated Here is my code: ' Microsoft SQL Server Integration Services Script Task ' Write scripts using Microsoft Visual Basic ' The ScriptMain class is the entry point of the Script Task. Imports System Imports System.Data Imports System.Math Imports Microsoft.SqlServer.Dts.Runtime Imports Microsoft.SqlServer.Dts.Pipeline Imports Microsoft.SqlServer.Dts.Pipeline.wrapper Imports Microsoft.SqlServer.Dts. Public Class ScriptMain Public Sub Main() ' Dim package As Microsoft.SqlServer.Dts.Runtime.Package = _ New Microsoft.SqlServer.Dts.Runtime.Package() Dim e As Executable = package.Executables.Add("DTS.Pipeline.1") Dim thMainPipe As Microsoft.SqlServer.Dts.Runtime.TaskHost = _ CType(e, Microsoft.SqlServer.Dts.Runtime.TaskHost) Dim dataFlowTask As MainPipe = CType(thMainPipe.InnerObject, MainPipe) Dts.TaskResult = Dts.Results.Success End Sub End Class
View Replies !
Adding Column Attributes For Custom Pipeline Component
I'm building a custom transform component. I want to mark some input columns as keys for deduplicating. In a similar way to the provided Sort component, I want to check those columns and allow pass-throughs (or not) for the others - so next to each input column name I need two checkboxes (1:use for dedupe; 2:include in output if 1 not checked). If a column is checked for use in the dedupe, I want some other attributes to be shown indicating how it will be used. How do I display the checkboxes to let users select which columns to include for deduplication, and then how do I add further attributes underneath (copying the Sort component's look) for selection? Thanks in advance for guidance and pointers on this.
View Replies !
Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer Column Ordinal From Name?
Hi, I need to access columns from a data flow by ordinal position in a script transformation (I'm parsing an excel file which has several rowsets across the page). The first problem I encountered is the generated BufferWrapper does not expose the columns collection (i.e. Input0Buffer(0) does not work) but I got around that by implementing my own ProcessInputs(InputId, Buffer) method instead of using the wrapper. My problem now is that the column ordinals are in some random order (i.e. Column "F1" is ordinal 1 but Column "F2" is 243). Where in the object model can I map between the name and the ordinal - it's not jumping out at me? Dave PS Why is the script editor modal, it's frustrating having to switch between the Visual Studio environment and the VSA one.
View Replies !
MS SQL Server 2005: Collect Procedure For &&"dts Pipeline Generate Error
Dear experts, My MS SQL Server 2005 is generating the following error. may i know what's wrong with it? " The Collect Procedure for the "DTSPipeline" service in DLL "XXX:Program FilesMicrosoft SQL Server (x86)90DTSBinnDTSPipelinePerf.dll" generated an exception or returned an invalid status. Performance data returned by counter DLL will be not be returned in Perf Data Block. The exception or status code returned is the first DWORD in the attached data. " Thanks in advance for any assistance rendered. pat
View Replies !
Pipeline Error-excel Source-data Reader Does Not Read In Meta Data
Hi all, i got this error: [DTS.Pipeline] Error: "component "Excel Source" (1)" failed validation and returned validation status "VS_NEEDSNEWMETADATA". and also this: [Excel Source [1]] Warning: The external metadata column collection is out of synchronization with the data source columns. The column "Fiscal Week" needs to be updated in the external metadata column collection. The column "Fiscal Year" needs to be updated in the external metadata column collection. The column "1st level" needs to be added to the external metadata column collection. The column "2nd level" needs to be added to the external metadata column collection. The column "3rd level" needs to be added to the external metadata column collection. The "external metadata column "1st Level" (16745)" needs to be removed from the external metadata column collection. The "external metadata column "3rd Level" (16609)" needs to be removed from the external metadata column collection. The "external metadata column "2nd Level" (16272)" needs to be removed from the external metadata column collection. I tried going data flow->excel connection->advanced editor for excel source-> input and output properties and tried to refresh the columns affected. It seems that somehow the 3 columns are not read in from the source file? ans alslo fiscal year, fiscal week is not set up up properly in my data destination? anyone faced such errors before? Thanks
View Replies !
&&<Data&&>Microsoft.SqlServer.Dts.Pipeline.BlobColumn&&</Data&&>
I have a custom (dataset) destination component from ms samples and it has an input holds DT_NTEXT value. Whenever I try to retrieve data from this it returns "Microsoft.SqlServer.Dts.Pipeline.BlobColumn" as value. I try this but didn't work: String sValue = System.Text.Encoding.Default.GetString(Convert.FromBase64String(this.dataSet.Tables[0].Rows["Data"].ToString())); It throws an execption "invalid character in.." Please help how I can convert this? Thanks in advance
View Replies !
SQL Duplicates
I need some help. I have created a database that looks like the following: FirstName Table link to Main Table. I have created a Stored procedure that looks like this: Create procedure dbo.StoredProcedure ( @FirstName varchar(20) ) Declare FirstNameID int Insert Into Main Table ( FirstName ) Values ( @FirstName ) Select @FirstNameID = Scope_Identity() How could I redesign this to check if a value exists and if it exists then simply use that value instead of creating a new duplicate value? Thanks!!
View Replies !
Msg 512 But No Duplicates!?!
I am attempting to execute the Stored Procedure at the foot of thismessage. The Stored Procedure runs correctly about 1550 times, butreceive the following error three times:Server: Msg 512, Level 16, State 1, Procedure BackFillNetworkHours,Line 68Subquery returned more than 1 value. This is not permitted when thesubquery follows =, !=, <, <= , >, >= or when the subquery is used asan expression.I've done some digging, and the error message is moderatelyself-explanatory.The problem is that there is no Line 68 in the Stored Procedure. It'sthe comment line:-- Need to find out how many hours the employee is scheduled etc.Also, there are no duplicate records in the Employee table nor theWeeklyProfile table. At least I assume so - if the following SQL todetect duplicates is correct!SELECT E.*FROMEmployee Ejoin(select EmployeeIDfromEmployeeGroup by EmployeeIDhaving count(*) > 1) as E2On(E.EmployeeID = E2.EmployeeID)SELECTW.*FROMWeekProfile Wjoin(SelectWeekProfileIDFROMWeekProfileGROUP BYEmployeeID, MondayHours, WeekProfileIDHAVING COUNT(*) > 1) AS W2ONW.WeekProfileID = W2.WeekProfileIDNOTE: In the second statement, I have tried for MondayHours thruFridayHours.Anyone got any ideas? The TableDefs are set up in this thread:<http://groups-beta.google.com/group/comp.databases.ms-sqlserver/browse_frm/thread/fff4ef21e9964ab8/f5ce136923ebffc3?q=teddysnips&rnum=1&hl=en#f5ce136923ebffc3>The Stored Procedure that causes the error is here:--************************************************** ***********CREATE PROCEDURE BackFillNetworkHoursASDECLARE @EmployeeID intDECLARE @TimesheetDate DateTimeDECLARE @NumMinutes intDECLARE @NetworkCode int-- Get the WorkID corresponding to Project Code 2002SELECT@NetworkCode = WorkIDFROM[Work]WHERE(WorkCode = '2002')-- Open a cursor on a SELECT for all Network Support Employees whereany single workday comprises fewer than 7.5 hoursDECLARE TooFewHours CURSOR FORSELECTEmployeeID,CONVERT(CHAR(8), Start, 112) AS TimesheetDate,SUM(NumMins) AS TotalMinsFROM(SELECTTI.EmployeeID,W.WorkCode,TI.Start AS Start,SUM(TI.DurationMins) AS NumMinsFROMTimesheetItem TI LEFT JOIN[Work] W ON TI.WorkID = W.WorkIDWHERE EXISTS(SELECT*FROMEmployee EWHERE((TI.EmployeeID = E.EmployeeID) AND(E.DepartmentID = 2)))GROUP BY TI.EmployeeID, TI.Start, W.WorkCode) AS xGROUP BYEmployeeID,CONVERT(char(8), Start, 112)HAVINGSUM(NumMins) < 450ORDER BYEmployeeID,CONVERT(CHAR(8), Start, 112)-- Get the EmployeeID, Date and Number of Minutes from the cursorOPEN TooFewHoursFETCH NEXT FROM TooFewHours INTO @EmployeeID, @TimesheetDate,@NumMinutesWHILE (@@FETCH_STATUS=0)BEGINDECLARE @NewWorkTime datetimeDECLARE @TimesheetString varchar(50)DECLARE @Duration intDECLARE @RequiredDuration int-- Set the correct date to 08:30 - by default the cast from thecursor's select statement is middaySET @TimesheetString = @TimesheetDate + ' 08:30'SET @NewWorkTime = CAST(@TimesheetString AS Datetime)-- Need to find out how many hours the employee is scheduled to workthat day.SET @RequiredDuration = CASE (DATEPART(dw, @NewWorkTime))WHEN 1 THEN(SELECT CAST((60 * SundayHours) AS int) FROM WeekProfile WHERE(EmployeeID = @EmployeeID))WHEN 2 THEN(SELECT CAST((60 * MondayHours) AS int) FROM WeekProfile WHERE(EmployeeID = @EmployeeID))WHEN 3 THEN(SELECT CAST((60 * TuesdayHours) AS int) FROM WeekProfile WHERE(EmployeeID = @EmployeeID))WHEN 4 THEN(SELECT CAST((60 * WednesdayHours) AS int) FROM WeekProfile WHERE(EmployeeID = @EmployeeID))WHEN 5 THEN(SELECT CAST((60 * ThursdayHours) AS int) FROM WeekProfile WHERE(EmployeeID = @EmployeeID))WHEN 6 THEN(SELECT CAST((60 * FridayHours) AS int) FROM WeekProfile WHERE(EmployeeID = @EmployeeID))WHEN 7 THEN(SELECT CAST((60 * SaturdayHours) AS int) FROM WeekProfile WHERE(EmployeeID = @EmployeeID))ENDIF @NumMinutes < @RequiredDurationBEGIN-- Set the Start for the dummy work block to 08:30 + the number ofminutes the employee has already worked that daySET @NewWorkTime = DateAdd(minute, @NumMinutes, @NewWorkTime)-- Set the duration for the dummy work block to be required durationless the amount they've already workedSET @Duration = @RequiredDuration - @NumMinutes-- Now we have the correct data - insert into table.INSERT INTO TimesheetItem(EmployeeID,Start,DurationMins,WorkID)VALUES(@EmployeeID,@NewWorkTime,@Duration,@NetworkCode)ENDFETCH NEXT FROM TooFewHours INTO @EmployeeID, @TimesheetDate,@NumMinutesENDCLOSE TooFewHoursDEALLOCATE TooFewHoursGO--************************************************** ***********ThanksEdward
View Replies !
Getting Rid Of Duplicates
I have a table, TEST_TABLE, with 6 columns (COL1, COL2, COL3, COL4,COL5, COL6).... I need to be able to select all columns/rows whereCOL3, COL4, and COL5 are unique....I have tried using DISTINCT and GROUP BY, but both will only allow meto access columns COL3, COL4, and COL5..... i need access to allcolumns...I just want to get rid of duplicate rows (duplicates ofCOL3, COL4, and COL5)...Thanks in advance.Joe
View Replies !
Duplicates
i've imported a number of spreadsheets into a database the schema is directoryid firstname, lastname, extension 703168 Andrew Lim 78094 703154 Joseph Egan 78888 704548 Andrew Lim 78094 I realized the spreadsheet had a large number of duplicates. How can i delete the duplicates ? like deleting directoryid 703168 only.
View Replies !
Duplicates
i'm trying to get duplicates out of the my database SELECT COUNT(*) AS Amount, Firstname, surname, Internalextension FROM iac.dbo.sf_profil GROUP BY FirstName, surname, internalextension HAVING COUNT(*) > 1 order by firstname, surname How do i alter the query just retrieve records which have firstname and lastname which are similar but different extension numbers ?
View Replies !
Duplicates
I am having an issue trying to flag duplicate rows on a single table. For example, if there are two rows on the table which are identical, I want to update only one of the rows to mark it as a duplicate but I cannot find a way around it. I am using SQL 6.5 If you have any suggestions they would be appreciated.
View Replies !
Duplicates
have two tables. Table A contains client_no and accountno and table B contains the clientno,accountno,name,address,etc l'm trying to select the data from table B that has both the clientno and accountno from table a matching the clientno and accountno in table b. Table a has 20 records but when l do my selection l get more than 20records? l want to pupulate table a with the missing data that matches the two fields? Here is the query that l'm running. It returns duplicates when l run it.It should give me back the same number fo records as Table A Table a already has a client and account number just need to populate the rest of the data into the table insert into test(,,,,,,,etc) SELECT ltrim(rtrim(left(a.Title,5))) As Title, ltrim(rtrim(left(a.Surname,28))) As Surname, ltrim(rtrim(left(a.First_Name1,28))) As First_Name1, ltrim(rtrim(left(a.First_Name2,28))) First_Name2, ltrim(rtrim(left(a.Address1,28))) As Address1, ltrim(rtrim(left(a.Address2,28))) As Address2, ltrim(rtrim(left(a.Address3,28)))+ ' ' + ltrim(rtrim(left(Address4,28))) As Address4, ltrim(rtrim(left(a.PCode,4))) As PCode, convert(numeric(8),ltrim(rtrim(left(a.Order_Date_G ranted,8)))) AS Date_Of_Action, ltrim(rtrim(left(a.Court_Venue,8))) AS Court_Name, ltrim(rtrim(left(a.Admin_Order_Type,2))) AS AO_Type, ltrim(rtrim(left(a.Administrator_Name,35))) AS Administrator_Name, ltrim(rtrim(left(a.Administrator_Tel,22))) As Administrator_Tel, convert(numeric(8),ltrim(rtrim(left(a.Outstanding_ Bal,8)))) AS Amount FROM TableA a, TableB b where a.clientno = b.clientno and a.accountno = b.accountno
View Replies !
Getting Rid Of Duplicates
I have a dilema..... I have a databas eof about 60,000 users and i need to get rid of those users where there is a duplicate email address. I have written an asp utilty that works but is far too taxing on our little server and i thinkk itwill kill it. what it does is for each email address it compares it against all the others.... so for each address it checks against 60,000 other records 60,000 times.... you know what i mean. its pretty phucked.... i tested it on just one record and took about 5mins. anyway ive been trying to do it in SQL with no luck here is a simplified version of my dilema ***** NAMEEMAILAGE tomtom@mail23 tombomb tom@mail23 petepete@email23 davecool@mail21 stevesteve@mail17 marycool@mail89 thomas tom@mail13 richrich@mail65 richdick@mail65 tomtom@mail23 tomtom@mail23 so what i want to do is cut it down so there are no duplicate email addresses. I want the table looking like this: ***** NAMEEMAILAGE tomtom@mail23 petepete@email23 davecool@mail21 stevesteve@mail17 richrich@mail65 richdick@mail65 Can you think of a way to do this? i tried the following but it gets rid of duplicates of other fields which is not what i want to do CREATE table distinct_records SELECT name, Email, Age FROM duplicate_records GROUP BY name, Email, Age; any ideas guys? thanks tom.harrow@netpoll.net
View Replies !
Duplicates Only
I have the following data. CustID RoleID RelatID StartDate EndDate 20 RIX AGQ 5/05/2007 31/12/9999 20 RIX AGR 10/06/2007 31/12/9999 18 LRS AGQ 3/09/2004 31/12/9999 22 SRT AGP 5/03/2007 31/12/9999 22 SRT AGP 10/03/2007 31/12/9999 I wish to SELECT only the records that have duplicates. So, in the above data example, RoleID of LRS should be excluded. How to do?
View Replies !
|
TRACKER
