Fuzzy Grouping Seemingly Corrupting Data
Jan 10, 2007
I've seen one other post on this topic from October 2005 and I thought I'd bring it up again. I've a Fuzzy Grouping component in my data flow. The output data from it appears to be the result of records spliced into other records. This includes pass-through columns, not merely "clean" or similarity columns. For example (I've added the suffixes for illustrative purposes):
AddressLine1_in: 162 OAKMONT
AddressLine1_out: 162 OAKMONTLAMINATION INC
CityStateZip_in: Alexander, AR 72002-8539
CityStateZip_out: Alexander, AR 72002-8539116-7066
These are just pass-through columns, although "used" columns are seeing something similar (below.) Any others with this experience?
City_in: Alexander
City_out: Alexandertle Rock
View 1 Replies
ADVERTISEMENT
Aug 14, 2007
Dear Friends,
i think fuzzy lookup
COMPARES WHAT WE ARE MAPING THE COLUMNS WITH SPELLING (IT WILL REJECT ATLEAST 1 LETTER IS DIFFRENT IN ANY RECORD MAPPED COLUMN) EX: RAVI != REVI
what is fuzzy grouping ???? please explain
regards
koti
View 3 Replies
View Related
Aug 2, 2005
We are working with a client and are using Fuzzy Group transform for de-duping, and hierarchy creation for a national account list.
View 4 Replies
View Related
Oct 16, 2005
I am using the Sept CTP, I am doing a fuzzy grouping on 1.5Mil records.
View 7 Replies
View Related
Nov 14, 2007
I managed to get fuzzy grouping working. The relevant output (_key_in and _key_out) are stored in a new table that is a copy of the old table + fuzzy grouping columns.
How do i get SSIS to store the _key_in and _key_out in the original table?
The new matching column _key_out refers to the new key: _key_in. How could i get SSIS translate that to a matching column that refers to my original key?
View 1 Replies
View Related
Aug 2, 2007
hi focks,
WHAT IS THE USE OF Fuzzy Grouping IN SSIS
and please give me the example
regards
koti
View 1 Replies
View Related
May 15, 2006
Hi - we have been evaluating using Fuzzy Grouping and Lookup for maintaining our large list of customer records. Initial testing with Grouping on about 300K records went great but now with a larger sample of 7.3 million records we are running into problems. It doesn't appear to be system limitation - the index is built reasonably quickly and without errors but when it starts the matching we get these errors:
[Fuzzy Grouping Inner Data Flow : DTS.Pipeline] Error: The ProcessInput method on component "Fuzzy Lookup" (86) failed with error code 0x8000FFFF. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.
[Fuzzy Grouping Inner Data Flow : DTS.Pipeline] Error: Thread "WorkThread0" has exited with error code 0x8000FFFF.
[Fuzzy Grouping Inner Data Flow : DTS.Pipeline] Error: Thread "WorkThread1" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown.
[Fuzzy Grouping Inner Data Flow : OLE DB Source [1]] Error: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.
[Fuzzy Grouping Inner Data Flow : DTS.Pipeline] Error: Thread "WorkThread1" has exited with error code 0xC0047039.
[Fuzzy Grouping Inner Data Flow : DTS.Pipeline] Error: The PrimeOutput method on component "OLE DB Source" (1) returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.
[Fuzzy Grouping Inner Data Flow : DTS.Pipeline] Error: Thread "SourceThread0" has exited with error code 0xC0047038.
One thing we did find is that our test server didn't have SP1 installed and that seemed to help a lot (we were getting buffer errors prior to SP1). One other note - the desination table is populated with all the data but no scoring has been applied to it.
Does anyone have any ideas what could be causing this?
Thanks!
Keith Doyle
View 5 Replies
View Related
May 11, 2007
Hello,
I have created a project to do de-dupification of addresses.
I understand that Fuzzy Grouping will take less time if it has lesser data volume to process.
My source feed file is sometimes huge. So I am splitting the input into multiple branches based on
the first letter of the city. There are 7 branches in the process.
Source File Feed
|
Split data into 7 groups
|
------------------------------------------------------------------------------------------------------------------------------------------
| | | | | | |
FzGrpg FzGrpg FzGrpg FzGrpg FzGrpg FzGrpg FzGrpg
| | | | | | |
Split Split Split Split Split Split Split
| | | | | | |
------------- -------------- -------------- -------------- -------------- -------------- --------------
| | | | | | | | | | | | | |
<- - - - - - - Write the Canonicals and Dupes from each of these splits into database - - - - - - - - ->
When I designed this I was hoping that each of the Fuzzy Grouping tasks will execute in parallel.
But in reality they are processing one after the other.
Is there anyway to make them execute in parallel?
Appreciate your help.
Thanks
KM
View 12 Replies
View Related
Jul 29, 2006
Hi,
I have an Oracle table called "Party" which contains Party_Id as primary key and have Party_Name, Party_Addr etc., as fields. We have lot more duplicate party details such as (party_name and party_addr) in this table. We are trying to aviod duplicates using FUZZY logic of SSIS.
1. Is any body suggest me how to create package to avoid duplicates using Fuzzy logic for this scenario(Step by step instructions are good for me to understand SSIS).
2. Could you please provide me some samples for FUZZY(Please send me a sample to my email)
View 1 Replies
View Related
Oct 18, 2007
I was running a Fuzzy Grouping task on SQL Server Enterprise Edition SP1 without any issues. I then applied SP2 and now that same Fuzzy Grouping is causing a minidump and terminating the process.
First, does anybody know anything about this kind of issue?
Second, I tried to run the minidump file in Visual Studio but I cannot actually run the dump file in Visual Studio as I keep getting the following exception:
Debugging information for 'DtsDebugHost.exe' cannot be found or does not match. No symbols loaded.
Finally, I did obtain a random error on the server itself that displayed the GUID: 58FC39EB-9DBD-4EA7-B7B4-9404CC6ACFAB.
This GUID appears to be tied to a Dr. Watson error but, again, I cannot figure out what process is breaking.
Can somebody please help?
View 1 Replies
View Related
Oct 4, 2007
Hi,
We do not have any Address Cleansing tools and the requirement is we have to cleanse the data, finding the best possible record which has all info and update other records accordingly.
I am Not sure we can do this Fuzzy Grouping Transformation.
Example:
I have Source table with following info.
Customer_id
Location_Address
Location_City
Location_State
Location_Zip
Location_County
TT101
252 HARVARD RD
ATLANTA
GA
30340
FULTON
TT101
30340
TT101
252 HARVARD RD
ATLANTA
TT102
125TEST
CUMMING
TT102
125 TEST DR
CUMMING
GA
30040
FORSYTH
TT102
GA
30040
Please let me know the solution
Thanks in advance.
View 4 Replies
View Related
Jun 7, 2007
Hello,
I have been struggling with this for quite awhile so any help would be appreciated.
I need to know if there is away to populate the fuzzy grouping control dynamically. I know you programmatically design a package and customize it in C# but for our purposes we would like to control the SSIS package via database settings. When the settings change the package would then act different. Its a simple a package consisting of an Input - fuzzy grouping - conditional split - output. The connections are setup dynamically using parameters, expressions and a script task. Is there anyway I could do a similar thing for Fuzzy Grouping?
View 13 Replies
View Related
May 30, 2007
Hello All,
We have a SSIS package which includes Fuzzy Grouping in Data Flow. It takes two columns from source table and saves outputs in different table with match score etc. Following is the way we are doing it:
1. Load required data from table using OLEDB connection (source)
2. Sort the data
3. Apply Fuzzy grouping (using dedicated database instead tempdb and MinSimilarity = 0.6)
4. Send to destination table using OLEDB connection (destination)
In input table we have millions of records. It takes too long to execute and even sometime it fails after running 12 hours. Any suggestions for performance improvement are welcomed.
Appreciate your help.
Thanks and regards,
Ashish Basran
View 1 Replies
View Related
Nov 21, 2007
I have a few questions about the amounts of resources used by the fuzzy grouping transformation. I am running a little less than 5mil records through a fuzzy grouping that exact matches one column and fuzzy matches one. The server executing the package is a dual-core xeon with 2gb ram, running a default instance of sql 2005 enterprise.
I have been attempting to execute this package for a while now but it keeps erroring out for various reasons. At first, it was from a lack of available memory. I limited the memory usage of sql server to 256mb and set the buffer temp storage path, which alleviated those errors. However, now, my tempdb transaction log is growing significantly. It failed once for not being able to grow and reallocate quickly enough, but enlarging the auto-growth factor fixed that. Then, it filled up the volume the tempdb log was on, so now I have moved it to the san and am about to try again.
I was wondering, does anyone have a general idea on approximate resource usage by fuzzy grouping? Specifically, is there an approximate relation between the number of records grouped and the amount of ram/pagefile required? Also, on the database backend, how big can I expect the tempdb data/log files to get?
View 5 Replies
View Related
Apr 7, 2008
Hi,
I need some advice on fuzzy lookup / grouping design.
I have a requirement that, I think, is between lookup and grouping transformations.
In one of our applications, users can enter manually a label for some information in the database.
Every month, I will store all the new data in our OLAP DB, and I want to group these labels with a fuzzy logic.
Historical data (already loaded) have to be grouped, as well as new data coming every month.
I have no predefined canonical data, so Fuzzy Lookup seems not adapted to my pb.
Fuzzy Grouping seems ok, but it would require to put historical data as well as new data as an input of the Fuzzy Grouping Transfo to constitute groups. This seems not efficient to me.
Any clue ?
M.D
View 1 Replies
View Related
Apr 30, 2008
Hi all,
My question is how to calculate the similarity by using SQL query, example LIKE % , order by.....? Now i'm doing a function same like fuzzy grouping but i do not know how to get the answer, mean how they get match with those selected row of data.
Hope my question is clear. How to write the correct query? What should i do? I 'm newbie in Integration Services, so i need ur explaination in step by step if there hv correction.
I am looking forward to hearing from you shortly and thanks a lot in advance.
Thanks!
rgds,
xuenly
View 3 Replies
View Related
Jan 22, 2012
I have recently decided to dedupe my data but i am having a problem after running fuzzy grouping with the query on updating which duplicate to keep
_key_in is unique, _key_out is the duplicates so for example:
_key_in , _key_out , name , score , dedupe
1 , 1 , ron , 10 , purge
2 , 1 , ronn , 15 , keep
3 , 3 , john , 5 , keep
4 , 4 , matt , 15 , keep
5 , 4 , mat , 10 , purge
6 , 4 , matt , 15 , purge
I want to keep the _key_out with the higher score by setting the field de_dupe to 'keep' and the remainder to 'purge'. The score can also be the same within a duplicate so in the case it is the same i just need to keep one it doesnt matter which one. The query i have below nearly works but it marks duplicates with the same score as keep.
Code:
UPDATE b
SET b.dedupe_result = 'keep'
FROM
[BusinessListings].[dbo].[MongoOrganisationACTM1Destination] b
INNER JOIN
[Code] ....
View 2 Replies
View Related
Aug 14, 2007
Hello,
I was wondering how Fuzzy Grouping deals with and handles first name similarities. Is there a way to configure it so that Anthony = Tony, Bill = William, etc€¦? I created a simple package with several rows containing similar first names and ran the fuzzy grouping on the first name column. I received only one possible duplicate of Will = William which was at 56%. I lowered the threshold down to 1% and still only one match.
Now I understand and appreciate the reasons for this but was wondering if this type of situation was considered and a way of dealing with it is available.
Thanks,
Beac
View 3 Replies
View Related
Mar 2, 2008
Hi All,
Is there a way the fuzzy lookup or grouping can be trained so that similarities and confidence values rely on previously matched strong links?
For example: I can link 80% of my two datasets using one strong identifier (say phone #) which I trust. My goal then, is to use the probability of matching of the rest of my linking fields (say Name,Address,Gender,DOB) in a "matched by phone number" pair to train a fuzzy lookup task to be done on the unlinked 20% of the datasets.
This "training set" would in theory influence the similarity and confidence values of the fuzzy output since each linking column would carry a different weight or contribution towards a confident match.
Does anyone out there knows how to do this in practice in SSIS?
View 1 Replies
View Related
May 18, 2006
I have tried to process > 3 million Fuzzy grouping records on two different servers with no success. 3 mill works but anything above 4 mill doesn't. Some background:
We are trying to de-dup our customer table on: name (.5 min), address1 (.5 min), city (.5 min), state (exact). .8 overall record min score.
Output includes additional fields: customerid, sourceid, address2, country, phonenumber
Without SP1 installed I couldn't even get a few hundred thousand records to process
Two different servers - same problems. Note that SSIS and SQL Server are running locally on both
The higher end server has 4GB RAM, the other 2.5 GB RAM. Plenty of free disk space on both
SQL Server is configured to use 2 GB of RAM max
The page file is currently at 15GB
After running a number of test on both servers trying different batch sizes etc. the one thing I noticed is that it seems to always error out when SSIS takes over and starts chewing up all the available RAM. This happens after the index is created and SSIS starts "warming caches". On both servers SQL Server uses up about 1.6GB of RAM at this point while SSIS keeps taking over RAM until all physical RAM is used up.
Some questions:
Has anyone been able to process more then 3 million records and if so what is your hardware configuration?
Should we try running SSIS from a different server so it has access to the full amount of physical RAM? (so it doesn't have to fight for RAM with SQL Server)
Should we install Win 2003 Enterprise Server so we can add more RAM?
Any ideas why switching to the page file might be causing errors?
Thanks!!
Keith Doyle
View 17 Replies
View Related
May 30, 2007
Will the fuzzy grouping task match a null value to an empty string (or spaces)? I've got 5 columns I'm matching on, and one of them may be null for certain rows but an empty string for others. Given the 4 other columns may match, will this difference stop similar columns being grouped together?
(Someone's modified my grouped data since it was deduped, which takes a while, and I'm hoping for a quick answer on this).
Thanks in advance.
Ben
View 3 Replies
View Related
May 26, 2015
I have a table that I need to identify similarities so I'm running a Fuzzy Grouping Process. I'm getting the follow errors and I can't identify the problema since all the fields are varchar, except for the first that is int but not use in the fuzzy.
select
MSSEndCustomerTPID
, orgname
, address1
, cityname
, statename
, countryname
from [sales].[vw_Fact_VolumeSales] a
inner join [GMOFBI].[dbo].[vw_Dim_MSS_Organization] b
on a.EndCustomerOrganizationKey=b.MSSOrganizationKey
[code]...
View 3 Replies
View Related
Jul 20, 2005
Hello,I have this peculiar problem concerning MS SQL Server.My company works with an mailing application (ASP) which uses SQLServer as it's repository. What I want to do is send data directlyfrom my own application to this SQL Server in order to feed themailing application.To test if this was possible I linked the tables from SQL Server in MSAccess and entered the data. This worked fine and the data was pickedup correctly by the mailing application.The problem occurs when I send the data from my application (Javaapplication with JDBC connection). The data is in this case no longerpicked up by the application. The strange thing is that the data whichis entered through Access and the data from the Application lookidentical in de database view. The problem also occurs when the datais send with the tool winSQL and when I view the data in here it stilllooks identical.Even more strange is when I select the record which is not working inAccess and copy it into a new record (only changing the key) itsuddenly works!Has anyone have an idea how this can be?Thanks in advance,Sander Janssen.
View 3 Replies
View Related
Jun 17, 1999
Does anyone have a script that can force a table/database corruption. We are in process of evaluating MS Clustering on Compaq platform using SQL Enterprise. We are told by Compaq and MS that the MS clustering with Compaq hardware detects SQL table corruptions and does a fail over to the backup box. I need to be able to simulate the corruption to test this claim. Can someone out there help me.
View 1 Replies
View Related
May 12, 2006
Using an SQLDataSource control to populate a two column table...
I have not found a simple solid means of looking at the table to determine if a record exists and if it isn't there insert it. I've tried talking directly to the DataSource control and using dependent controls such as formview or gridviews. Anyone have a simple solid method to do this? I have a hunch I'm missing the obvious here.
Next, we (our dev team) is seeing an issue when switching modes of formview from Item, to Edit, Insert, etc. That the values of the TextBoxes and Labels become null. I'm guessing it is a cache issue but we need to get around it. Has anyone seen this sort of behavior and what did you do to correct/work around it?
Thanks,
View 1 Replies
View Related
Jul 20, 2005
We are having quite a time since moving a large database to a newserver (actually built new server, renamed as old to make seamless forusers, etc.)Import 104 million row database (5 column) into table (CD_Assets_bad2)from existing (CD_Assets):Account(varchar(8))TransactionDate(datetime(8)Flow(varchar(1))Category(varchar(7))TotalValue(decimal(8))Run DBCC CheckTable - no issues.Create 4 non clustered indexes (3 single column, 1 two-column). Allindexes create fine.Run DBCC CheckTable again and receive the following:Server: Msg 8951, Level 16, State 1, Line 1Table error: Table 'CD_Assets_bad2' (ID 244195920). Missing or invalidkey in index 'idx_totalvalue' (ID 7) for the row:Server: Msg 8955, Level 16, State 1, Line 1Data row (1:11154499:98) identified by (RID = (1:11154499:98) ) hasindex values (TotalValue = -10).Server: Msg 8952, Level 16, State 1, Line 1Table error: Database 'CD', index 'CD_Assets_bad2.idx_totalvalue' (ID244195920) (index ID 7). Extra or invalid key for the keys:Server: Msg 8956, Level 16, State 1, Line 1Index row (1:20855652:338) with values (TotalValue = -0¤4?) points to the data row identified by (RID = (1:11154499:98)).DBCC results for 'CD_Assets_bad2'.There are 104397173 rows in 677904 pages for object 'CD_Assets_bad2'.CHECKTABLE found 0 allocation errors and 2 consistency errors in table'CD_Assets_bad2' (object ID 244195920).repair_fast is the minimum repair level for the errors found by DBCCCHECKTABLE (CD.dbo.CD_Assets_bad2 ).Any ideas? It seems like some sort of corruption, but the indexcreates fine. If anyone can help please let me know. If I can provideany addtional information that might help, please let me know.Thanks,DavidJoin Bytes!
View 2 Replies
View Related
Jan 10, 2007
Am running ipwsitch's ws_ftp 2007 pro software on windows 2000 platform on a network. . Trying to download a 17.4mb zipped file via FTP. File comes across to my environment but is corrupt - when attempting to unzip it.
Transfer mode being used is BIN and not TXT.
Any suggestions?
View 1 Replies
View Related
Aug 16, 2006
I've been trying to find out why a simple query containing a couple of top-level order bys produces different results when I introduce TOP into the query. I've found nothing so far, other than the results of both queries (TOP and non-TOP) are different again if I add an index.
All the DDL and DML is below, along with the results. The database uses the BIN2 collator.
I guess there's a simple explanation for this...
DROP table employeeCREATE TABLE employee( id INTEGER IDENTITY (1, 1) NOT NULL, givenname NVARCHAR (20), familyname NVARCHAR (20), CONSTRAINT pk_employee_id PRIMARY KEY (id));insert into employee values('John', 'Smith');insert into employee values('John', 'SMITH');insert into employee values('John', 'Smyth');insert into employee values('John', 'SMYTH');goselect top 10 givenname, familynamefrom employee order by lower(familyname) asc, lower(givenname) ascgo-- Dropping top produces results in different order!select givenname, familynamefrom employee order by lower(familyname) asc, lower(givenname) ascgo-- Creating an index results in different order on both queries!CREATE INDEX idx_employee_names ON employee (familyname, givenname);goselect top 10 givenname, familynamefrom employee order by lower(familyname) asc, lower(givenname) ascgoselect givenname, familynamefrom employee order by lower(familyname) asc, lower(givenname) ascgo
This produces the following results:
givenname familyname
-------------------- --------------------
John SMITH
John Smith
John SMYTH
John Smyth
(4 rows affected)
givenname familyname
-------------------- --------------------
John Smith
John SMITH
John Smyth
John SMYTH
(4 rows affected)
givenname familyname
-------------------- --------------------
John Smith
John SMITH
John Smyth
John SMYTH
(4 rows affected)
givenname familyname
-------------------- --------------------
John SMITH
John Smith
John Smyth
John SMYTH
View 3 Replies
View Related
Jun 2, 2006
Hello-
I've an SSIS packeage that genrates offline cube (.cub) files via an MDX query run against an Analysis Services cube. When I run the package through the Management Studio all runs fine and the cube files are generated and can be used by excel.
However when I run this package through a SQL Agent job the files are generated and the job completes successfully , but when I try to open them in Excel they appear to be corrupted and I the Data Link properties screen comes up. In looking at the files the ones generated by the SQL Agent are also a different size than those generated from running the package directly
Has anyone else seen this kind of behavior? As an FYI I'm adding the files to the SQL Agent step from the SSIS Package Store.
Thanks,
Tristan
View 3 Replies
View Related
Dec 5, 2006
I have a fuzzy lookup in a data flow but this data flow never gets pass pre-execute phase. What is the problem?
Thanks
View 2 Replies
View Related
Oct 16, 2006
Upon restarting user defined function took seemingly forever to run
I am learning about nested while loops being used in some interdependent user defined functions. They seemed to work OK for a while.
Later, remembering how I lost the database due to hard disk reformatting, I backed up the database and copied it to a rewritable CD.
As the data is not really significant I deleted the database and practiced restoring the database from the CD.
This morning I restarted the user defined function and ran it. After more than half an hour with no result I gave up. Normally it took much less time to run such a user defined function.
I re-ran some other UDF and they worked. However, after I made some minor amendments to the TSQL scripts, saved the UDF and re-ran it, and it again seemed to take forever to run, even when I had set the counter in to while loop to 2.
I don€™t know what had gone wrong.
I went to register my copy of SQL 2005 EXPRESS. It didn€™t seem to help.
Suggestion and advice are much appreciated.
View 3 Replies
View Related
Oct 4, 2007
Just wondering.
View 1 Replies
View Related
Apr 15, 2008
I've been looking into ways to accomplish a fuzzy search and SSIS makes that possible if I want to do a bulk import or something like it. But what it I just want to look stuff up at any given time not haveing to run the package?
Is it possible to expose the fuzzy lookup outside of SSIS to for example t-sql?
Here's an example:
I want to lookup the music artist "Notorious BIG" but in the database it is "Notorious B.I.G." if I use the SSIS fuzzy lookup I basically get what I'm looking for. But how would I call this from a web application? So then I tried Full text search but this doesn't really work out as well.
Will I have to re-write the logic that the fuzzy lookup uses to enable it to work? i.e. using Full Text Indexes and FreeTextTable, ContainsTable, SoundEx and the like to somewhat even come close to what the Fuzzy Lookup has?
View 6 Replies
View Related
