Remove Duplicate Lines

Remove Duplicate Lines
how would i go about removing duplicate lines from a text file. (a big ish text file around 10,000 lines)

The duplicate lines are always next to each other so i suppose that makes it a lot easier.

Example:
1
2
2
3
4
2


It would only be required to remove the 2 in italics the 2 later on is ok. Speed isn't really needed either as long as it doesn't take like an hour or something silly.

---

RichTextBox - Remove Duplicate Lines
I tried to do this using some code for removing duplicates in a listgbox, but I couldnt get it to work.

Anyone here have some code offhand that takes care of this?


Thanks ahead of time.

---

How Can I Remove Duplicate Lines In A Text File?
I have made a function that removes duplicate lines in a text file but it does not always work here it is:


option Explicit

public Function RemoveDuplicateLines(FileName as string) as Integer

Open FileName for input Access Read as #1
Open App.Path & "integrity.tmp" for Output as #2

Dim LineIn(1 to 2) as string
Dim Impurities as Integer
Dim FirstPass as Boolean
Dim Last as Integer

FirstPass = true

Do
If EOF(1) = true then GoTo endof
'If it's the first time input 2 lines and compare them
'otherwise compare the last line with the next line
If FirstPass = true then

Line input #1, LineIn(1)
Last = 1

If EOF(1) = true then print #2, LineIn(1): GoTo endof

Line input #1, LineIn(2)
Last = 2

If EOF(1) = true then print #2, LineIn(2): GoTo endof

'Compare the lines
If LineIn(1) = LineIn(2) then

print #2, LineIn(1)

'Increment the defects
Impurities = Impurities + 1

else

print #2, LineIn(1)
print #2, LineIn(2)

End If

else 'It's not the first pass

If Last = 1 then

Line input #1, LineIn(2)
If EOF(1) = true then print #2, LineIn(2): GoTo endof

'Compare Line(1) with Line(2)

If LineIn(2) = LineIn(1) then
Impurities = Impurities + 1
print #2, LineIn(1)
else

print #2, LineIn(1)
print #2, LineIn(2)

End If

Last = 2

ElseIf Last = 2 then

Line input #1, LineIn(1)
If EOF(1) = true then print #2, LineIn(1): GoTo endof
'Compare Line(2) with Line(1)

If LineIn(1) = LineIn(2) then
Impurities = Impurities + 1
print #2, LineIn(2)
else

print #2, LineIn(2)
print #2, LineIn(1)

End If

Last = 1

End If

End If

Loop

endof:

Close #1
Close #2

Kill FileName
FileCopy App.Path & "integrity.tmp", FileName
Kill App.Path & "integrity.tmp"

RemoveDuplicateLines = Impurities

End Function





The problem with it is that it does two lines at a time so if my file is like this:

a
b
c
d
d
e
f

Then my output is like this:

a
b
c
d
d
e
f

Because 'd' was not reconized as a duplicate line. However if the file looked like this:

a
b
c
c
d
e
f

Then it would work because 'c' is an odd number.

My code looks complex because I though that it would work like this:

a <-- check this line with next line
b <-- check this line with next line
b <-- check this line with next line
c <-- etc..
d
e
f

But my code does this:

a <---\___ Check these two lines
b <---/
b <---\___ Check these two lines
c <---/
d
e
f


Can someone help me out??? Please!!

---

Duplicate Lines
How can I keep a line from being added to a listbox if it's already there?
30,388 lines had been added and a lot of them were duplicates.

---

Duplicate Remove
hi
i have a duplicate remove but what i want to do is if i have to listbox with listbox 1 with 1234 and 123 in it and also listbox 2 with 1234 and 342 in it how can i get it so it will find the 1234 in listbox 2 and remove it?

---

Remove Duplicate Value...
does any one knows how to remove duplicate values from combobox...???

Need it urgent...???

cheers...

---

Removing Duplicate Lines
hello,
i need the visual basic code to remove duplicate lines from a text file .

thoma

---

Remove Duplicate From A Grid
I have a MSFlexGrid with 4 columns. I would like to delete duplicates (4 same columns). Is there a fast way or do I have to do a loop, check each one and delete them?
Thanks!
LD

---

Duplicate Remove Function
Heres My Dup Remove Function, It Can Only Handle About, 1000 itmes, without freezing, any ideas making it handle more?? id like to get it to 5000+


Code:

Private Sub cmdDup_Click()
Dim Search1 As Long
Dim Search2 As Long
Dim KillDupe As Long
KillDupe = 0
For Search1& = 0 To List1.ListCount - 1
For Search2& = Search1& + 1 To List1.ListCount - 1
KillDupe = KillDupe + 1
If List1.List(Search1&) = List1.List(Search2&) Then
List1.RemoveItem Search2&
Search2& = Search2& - 1
End If
Next Search2&
Next Search1&
End Sub

---

Remove Duplicate Records
Hi,
Can somebody help with SQL Server query?

I have a SQL Server table with only 2 fields (Field1 & Field2). There are some duplicate records in the table and I try to delete all duplicates.

1) Here is what I use in Query Analyzer:
    BEGIN TRANSACTION
    delete
    from TableIndicators
    where EXISTS
    (
        select Field1, Field2,count(Field1),count(Field2)
        from TableIndicators
        group by Field1, Field2
        having count(Field1) > 1 and count(Field2) > 1
    )
    ROLLBACK TRANSACTION
This statement would delete the whole table.

2) When I use only sub query
    select Field1, Field2,count(Field1),count(Field2)
    from TableIndicators
    group by Field1, Field2
    having count(Field1) > 1 and count(Field2) > 1
it selects only duplicate records that is what I need.

Are there are any ideas why the whole statement (#1) does not work?

Thanks,
Romma

---

Find Duplicate Lines In File
i have a file hold approx 150 sentences. some of them are doubles i.e the same (exactly) is it possible to find the same repeated lines and delete the duplicate line? i will be needing a text box as i will need to save the changed data in the text box to a file.

---

Get Rid Off Duplicate Lines In Text File
hi,

i have a text file contains something like this:

as123
n9090
jkl7jy lokiue
n9090
as123

does anyone know how to get rid of the duplicate lines in the file?so i'll be just the items that are different.

as123
n9090
jkl7jy lokiue


thanks in advance

---

Remove Duplicate Listbox Entries
Hi,

I have searched but cannot find anything that works

I am wanting to remove any duplicate text from the list box....

heres an example of my code (does not work)


Quote:




Private Sub Command8_Click()

Dim itext As String

For i = 0 To List1.ListCount - 1
itext = List1.List(i)
Do
If List1.List(i) = List1.List(i) Then
MsgBox "2x song", , "ERROR"
End If
Loop
Next

End Sub




All help much appreciated!

Matt

---

Remove Duplicate Entries From A Collection
I'm trying to delete any duplicate entries from a collection thats been passed to a function. The idea i have in mind is to first add all the items in the collection to an array, then compare the array against the collection for duplicate values, but i'm not sure how to do this. I have:

Function DeDupe (Col as Collection) as Integer

For x = 1 To Col.count
ReDim Preserve checkarray(1 To x)
checkarray(x) = Col.item(x)
Next

For x = LBound(checkarray) To UBound(checkarray)

For y = 1 To Col.count
If checkarray(x) = Col.item(y) Then
MsgBox "Value: " & Col.item(y) & " removed", vbDefaultButton1, ""
Col.Remove (y)
End If
Next

Next

End Function

but all this does is remove the first value in the collection then throw a subscript out of range error.

Any ideas?

---

Remove Duplicate (or Triplicate Or More) Records
This thread is sort of a continuance of thread:
http://www.xtremevbtalk.com/showthread.php?t=187160
How do I remove duplicate, triplicate records (some are 4 or more the same)
I want to delete the entire records that are repeated.
Note that I mean delete from the db all together( using update I think is the method)

---

Find Lines Duplicate In A Text File
Hi at all..
i don't understand where i wrong. I would like fin a line duplicates in a text file. My code i wrong. Where? Thank's a lot.
Bye from Italy.
------------------------------ CODE START -------------------------------------
Const ForReading = 1, ForWriting = 2, ForAppending = 3
Const TristateUseDefault = -2, TristateTrue = -1, TristateFalse = 0
Dim fs, f, ts, s, filename, oldline, max
filename= "pippo.TXT"
Set fs = CreateObject("Scripting.FileSystemObject")
Set f = fs.GetFile(filename)

Set tsmax = f.OpenAsTextStream(ForReading, Tristatefalse)
while not tsmax.AtEndOfStream
max=max+1
text=tsmax.ReadLine
wend
msgbox (max&"="&text)
tsmax.Close

Set tsorig = f.OpenAsTextStream(ForReading, Tristatefalse)
for i = 1 to max
orig=tsorig.ReadLine
start = tsorig.line
wscript.echo orig&start
Set tscomp = f.OpenAsTextStream(ForReading, TristateFalse)
for a = start to max
comp=tscomp.ReadLine
if orig=comp then
wscript.echo "Record :"& orig & " is duplicated "
end if
next
tscomp.Close
next
tsorig.Close
----------------------------------- CODE END -----------------------------

---

Remove Non Duplicate Rows From An Excel Column
Hi guys, I hope somebody can help me, How through visual basic or a formula can I remove from a column all the non duplicate data in an excel file?

I have a range of data and what I need is to find the duplicate data cells to work with it removing the non duplicated lines.

Any idea?

Thanks so much

---

Listbox And Textbox. Remove Duplicate Entries
All,

I have a form that has a textbox and a listbox. These are filled dynamically from user input. I need to be able to look at each item the listbox and determin if it exits anywhere in the textbox (This textbox has multipul entries). And if it does, I need to remove it from the textbox. I do not want to clear the textbox, just remove the duplicated item.

Any Ideas how to do this?

---

Remove Duplicate Entries From Text File....
hi, say i have a textbox with numbers like this:


345
645
878
8978
585
64
3
56


how would i make a program which removes duplicate entries from the textbox? i've more or less got a similar bit of code working which works for a listbox, but i'm kinda stuck with how to do it on a textbox. can anyone help?

---

Remove Duplicate Strings From Array Improvement?
I wrote a function to replace duplicate strings in an array with "". Since I'm going to have to convert it over to VBScript for ASP use, can this be improved for faster performance?


VB Code:
Private Function ChopArray(ByRef Arr() As String)    Dim i As Integer 'outer loop counter    Dim j As Integer 'inner loop counter    Dim s As String 'temp store for current value        'loop through the array    For i = LBound(Arr) To UBound(Arr) Step 1        s = Arr(i)        'if its not an empty string, and we're not at the last item in the array,        'then loop through the rest of the array to remove the duplicated strings        If (Len(s) > 0) And (i <> UBound(Arr)) Then            For j = (i + 1) To UBound(Arr) Step 1                If Arr(j) = s Then Arr(j) = ""            Next        End If    Next End Function

---

Using Listview.finditem To Remove Duplicate Files....
I was wondering how to use the listview.finditem before loading items into the list, then form an if statement on either not adding if it is allready there and adding if it isn't..... can anyone tell me how this is done? or if there is an easier way to prevent duplicate items plz let me know, thanks

---

Crystal Reports: How Do I Remove Duplicate Records?
Hello,

How can I remove duplicate records?

I have are report as such:

Emp No Name     Hire Date Status Company/Subsidiary

7791     Anna Jackson     07/01/2002 Active Access
7997     Irene Hagedorn     08/01/2002 Active Access
7798     Alice Smith     08/01/2002 Active Access
7798     Alice Smith     01/01/2003 Active iSmart


Record Number 3: Displays a hire date of employee to 'Access' Company. Record 4: shows employee was employed in iSmart, a subsidiary of the main company.

The requirement of the report needs to display only record 4 and discard record 3 in the list.

What procedures should I do?

Thank you in advance,

Cristina.

---

Deletetion Of Duplicate Lines With Matching Part Of Line
Hello everyone, need some help with a text file. I need to check the first 40 characters of each line and if they match delete the previous line.  The problem is that the lines with matching first 40 characters are not one after another. Is this possible?  Any help will be greatly appreciated...

---

Script To Remove Duplicate Values From Sql Server Table
i know this isn't really a vb question -

anybody got a procedure in sql server to check for any duplicate numeric values in a table and update them to make them unique?

table was created with no primary key -

i was thinking something along the lines of select distinct field from table ordered by - set variable = to max value + 1 of field in question.

loop through the table and set field = variable value + 1 where field value = variable value.

obviously variable would need to be incremented by one through each loop.

i just can't figure the exact script!

thanks in advance and apologies for sidetracking a bit but i figured there would be guys on here who could do this easily enough.

---

Remove Lines Without Numbers
I have been trying to remove lines with less then 10 lines (done) and remove them if they don't contain numbers but am tearing my hair out, I can't get do it. Anyone have some pointers which could help me?

Thanks

---

ListBox Remove Lines
How do I remove the last line of a ListBox?

Thanks in advance

---

Remove Text Box Lines
I got text loaded into a text box, but need to remove the first 2 lines. Can anyone please tell me?

Thanks in advance

---

Remove Empty Lines Just B4 EOF
I have a Text file which I need to remove multiple blank lines which are just before the end of my text file.

How can I go about coding this:

Goto the End of the File then,
Do until line is not empty (Can you reverse Line Input of a File?)
> Remove blank line.



Thanks.

Did this forum use to email you if someone had replied to your post, has that stopped?

---

Add-in To Remove Blank Lines ?
Anybody know of an add-in to remove blank lines, also indent and remove comments would be helpful as well ?

---

How Do You Remove Lines From A ListView?
If this is the code to add a line to a ListView

Code:Dim itmx As ListItem

With ListView1
    Set itmx = ListView1.ListItems.Add(, , "Item1")
End With

then what is the code to delete a specified/selected line?



Edited by - dodgerslim on 9/2/2003 12:58:37 AM

---

Macro To Remove Unwanted Lines
I am trying to write a macro to remove unwanted lines from a text document.

The document includes around 300 pages, and I want to remove all the lines except for around 100 of them. The lines that I want to keep begin:

"Tree length = "

Following this statement is an integer (which I also want to retain).

I have tried numerous permutations of macros which seem to delete all lines except for the 1 occurence.

Any help, please?

Charlie Butterworth

---

Remove Text File Lines
What's the best way to keep the 50 first lines of a text file and drop the rest of the text file ?

---

Remove Empty Lines From Text
Hi all,

I'm getting a data from a WebBrowser control, The data might look something like this (E.g. For Google.com):


Quote:





Web Images Video News Maps more »Blogs
Books
Products
Groups
Patents
Scholar
even more »

Advanced Search
Preferences
Language Tools



Advertising Programs - Business Solutions - About Google - Go to Google Jordan
©2007 Google


Get a Google-enhanced search box






My Question is how can I remove all these empty lines and make it as a paragraph (As a string) like this for example:


Quote:




Web Images Video News Maps more »Blogs Books Products Groups Patents Scholar even more » Advanced Search Preferences Language Tools Advertising Programs - Business Solutions - About Google - Go to Google Jordan ©2007 Google Get a Google-enhanced search box






Thanks very much,
zeid

---

Proramatically Remove Lines Of Code?
In this $#@$$ project I have "inherited," there are some lines of code that repeat themselves in every single function, subroutine and module that I want to get rid of.... actually, they're code for an old fashioned way of doing some error trapping. I want to get rid of them so I can put in my own error trapping code.

My question: is there a way, in code, of going through all parts of this project and wiping out the lines I want to wipe out? Each piece of candidate code starts with '******** Start trapping ******* and ends with
'****** End Trapping *******

so what I'm thinking is some kind of code that goes through all my code and loops through this start to stop code, wiping it out.

If anyone has a thought or suggestion, would you send it my way? Thanks so much.....looking forward to your replies!!!

---

Remove All The Blank Lines From A File
how would i open a file and remove all the blank lines ie. "" and save it again?


thanks :-)

---

RichText : Remove Empty Lines
Anyone here ever remove blank lines from a rich text control? I have attempted to remove them, but it dosent work.

I can get it to mark the blank lines with a var, but I dont know how to remove the whole line it is on.

I may just be in need of a code which removes a line or the occourances of enter or chr13

---

Remove Lines From Text File.
i have a text file with the following...
---------------------------------------------
Email=gmatteson@myemail.com
Account=my email account
Username=gmatteson
Password=mypassword
Server=mail.myemail.com
Port=110
Query=10
---------------------------------------------

the email - gmatteson@myemail.com gets populated in a listview box.

so when the user clicks on gmatteson@myemail.com in the listview and click REMOVE I want it to delete everything from the
email=gmatteson@myemail.com to query=10.

The code below deletes the email=gmatteson@myemail.com, what i need is for it to delete the next 6 lines..... so that whenever a user deletes an email address it also deletes the information pertaining to that address..., does anyone know how to do this?

---------------------------------------------
Dim objFSO As New FileSystemObject
Dim objFile As File
Dim objTS As TextStream
Dim strReadLine As String
Dim strFind As String
Dim strData As String
Dim strEmail As String
Dim blnFound As Boolean

Dim strReadAll As String
Dim strELen As Integer
Dim strSLen As String
Dim strNewReadAll As String
Dim strWriteNString As String

Set objFile = objFSO.GetFile(App.Path & "pop3a.txt")

Set objTS = objFile.OpenAsTextStream(ForReading)
Do Until objTS.AtEndOfStream Or blnFound = True
'reads line in file
strReadLine = objTS.ReadLine
strFind = "="
strData = (InStr(1, strReadLine, strFind) + 1)
strEmail = Mid(strReadLine, strData)

If strEmail = lvwAccounts.ListItems(strSelected).Text Then
blnFound = True
End If
Loop
objTS.Close

Set objTS = objFile.OpenAsTextStream(ForReading)
Do Until objTS.AtEndOfStream
strReadAll = objTS.ReadAll
strELen = Len(strReadLine)
strSLen = InStr(1, strReadAll, strReadLine)
strNewReadAll = Mid(strReadAll, strSLen, strELen)
Loop

strWriteNString = Replace(strReadAll, strNewReadAll, "")
Debug.Print strWriteNString
objTS.Close
End If
-----------------------------------------

the strWriteNString is the string of the whole file that doesn't include the email that the user deleted, i would also like the user to be able to delete the info regarding that email, as explained above...
Thanks for any help,
- gabe

---

Remove Lines Be-4 Saving*Resolve*
How do I remove some lines like first 16 lines of text before i save in from the Richtextbox to a text file?

---

Remove Blank Lines In RichTextBox
I'm loading a file in to a RichTextBox that has blank lines scattered throughout. Anyone know a way to remove them all?
Thanks

---

Open For Binary - Remove A Few Lines?
CAn I open a file for Binary....then remove the first 5 lines and the last 4 or 5 lines?

(They are blank or text that I dont need...I am parsing the rest of the file for a report)

Thanks!

---

Remove Line Breaks Every Two Lines
I have a file that look something like:

1
2
3
4
5
6

I would like to open the file and resort it so it ever two line are on the same line like

1 2
3 4
5 6

---

How To Remove Extra Lines From Textbox Or RichTextBox
How To Remove Extra Lines From Textbox Or RichTextBox?

my TXTBOX look's like
----------------------------------------
NAME


















NAME
--------------------------------------


How can i make it look like

------------------
NAME

NAME
------------------


i am pissed @ this plz help !!

---

Remove Empty Lines From Word Document...
i want to remove empty lines from the end of the MSword Document through VB6. any one guide me about that????

---

Free VB6 Add-in To Remove Blank Lines From Code
Hi,

Does anyone know a free VB6 add-in that removes blank lines from code, or preferably from a selected piece of code? I'm currently using VB Assistant Add-In, which does the indenting, but doesn't remove blank lines.

George

---

Remove Empty Lines From Text File
How can I remove all the empty lines from a text file?



Edited by - TankMaster on 2/21/2004 7:38:04 AM

---

Remove All The Lines Of Text Wich Doesn’t Contain A Word .
Hello .

For my second thread ,I need to remove all the lines of text wich doesn’t contain a word .
Or I want to keep the lines wich contains a definite word and to remove all the other lines
How to do ? Would you be able to give me any ideas ?

Example I want to keep the lines wich contains AABB
1/You can download all the tuts off this page
2/On this page, we got the tuts about the basics of what we should know.
3/first, get your tools....AABB(Just Links)
4/Then info on some tools and how to set them up and use them:
Only this line will stay
3/first, get your tools....AABB(Just Links)
because she contains the occurrence AABB
1/2/4/ this lines will have to be remove

---

Remove Blank Lines In Address Labels : Crystal Reports 9
I'm using crystal reports 9 for address labels with VB6 / access

The problem is I have 4 address fields and a postcode. If only two lines of address are used when I come to print my address labels there is a gap between the address and postcode.
As a go between stage I did

strSQL = strSQL & " SELECT tblStudent.FirstName + ' ' + tblStudent.Surname as FullName, tblStudent.Address1 + chr(10) + chr(13) + tblStudent.Address2 + chr(10) + chr(13) + tblStudent.Address3 + chr(10) + chr(13) + tblStudent.Address4 + chr(10) + chr(13) + tblStudent.Postcode AS address"

But still when I iterated through the records to remove the blank lines it wasnt all that effective.

Is there an easy way to do this without iterating? In a word mail merge you just select toremove blank lines but i coudnt see anything in CR.

----------------------------------------------------------------------------
Dan Bayley
Affordable website design UK
Free Google & Yahoo sitemap generator

Edited by - DanInManchester on 8/27/2003 1:51:25 AM

---

: Display Non Duplicate Data &amp; Add Duplicate Data In Msflexgrid
My problem is when I retrieve the record from the database, the item name is duplicate, and what I want is to display non duplicate record. Then, the unit sold (qty sold) with same item name can added automatically.

I can display non duplicate record but I failed to add the unit sold with the same item name together.

So, how can I solve this kind of problems? Can you show me the example of the coding based on this situation. Thanks.

---

Reading Certain Lines Of Files And Outputting Those Lines
This is what I am trying to do and I think I am going about it all wrong.

I am trying to read a log file (ascii) and only want to read in certain pieces of information (e.g. a failed request for a web page) and then I want that information written to a new file.

I wont even attempt to post thw code I have trying to work with, if someone can give me some pointers of post some example, I would really appreciate it.

---