Full Text File Search With Indexing Service On Windows

MySQL 5.0, FULL-TEXT Indexing And Search Arabic Data, Unicode
MySQL 4.0, FULL-TEXT Indexing and Search Arabic Data, Unicode

[This version has a couple subtle edits from the orginial I posted
on mailing.database.myodbc - I'm cross posting here on this
topic/subject related newsgroup]

I was wondering if anybody has experienced the same issues
challenges I'm experiencing I'll describe shortly. Once
resolved some fascinating and powerful multi-lingual
apps incorporating non-English/latin character sets can be
realized by many developers.

I have a Unicode utf8 English - Arabic - Hebrew - Greek (and
several other languages) database in Microsoft Excel. I KNOW
that it is Unicode utf8 data because MySQL tells me it
recognizes the encoding as such but not in the context I want.

Allow me to explain ...

I can search the Unicode utf8 encoding with no problem in
Excel. While in Excel I highlight a complete word or a
partial string of an Arabic word copy it to the clipboard
(i.e. memory). I then do a find and the process is the
same successful result as if it was an English string.

MySQL 5.0 is supposed to handle Unicode utf8

I created a MySQL database I named: languages

CREATE DATABASE languages ;

and I implemented the following command on a MySQL
command prompt:

ALTER DATABASE languages DEFAULT CHARACTER SET utf8;

No problem (so far) MySQL seemingly recognized utf8 and
accepted it. My understanding is with the ALTER command
the tables I create against languages will be utf8.

I now created a table I named mainlang which denotes it
will be the main table for my languages.

mysql>CREATE TABLE mainlang
->(
->langNumID varchar(30),
->colB varchar(30),
->colC varchar(30),
->primary key (langNumID, colB)
->);

Again so far no problem: Table successfully created.
My third column 'colC' is where the Unicode data
will be stored.

I now attempt to import the database from my
Excel file into my MySQL database as follows:

mysql>load data infile 'c:arabicdictionary.csv'
->into table mainlang
->fields terminated by ','
->lines terminated by '
'
->(langNumID, colB, colC);
ERROR 1406 (22001): Data too long for 'colC' at row 1

So what to do? I did a search and found other
people seemingly had the same problem and someone
suggested:

ALTER DATABASE languages DEFAULT CHARACTER SET cp1250;

I dropped mainlang, recreated it, redid the load and
Lo and behold ... it seemed to work. No Data too long
error occurred and when I did the following query:

mysql>select langNumID, colB, colC
->from mainlang
->where colB = ��'

I see colA have a correct numeric value, colB a
correct numeric value (4994) and for colC a string of
unintelligible characters with diacritical marks,
oomlats etc. which I know is the cp1250 encoding
interpretation of the Unicode utf8 data which is
similarly unintelligible in its own regard.

Now what I try is: do a copy of the obscure colC
cp1250 character string into the clipboard/memory
and then do the following tweak on the original
select statement to see if I can search on the
(now) cp1250 character string:

mysql>select langNumID, colB, colC
->from mainlang
->where colc = 'paste of the cp1250 character string'

The computer would not allow a paste unless I pressed
the escape key. On initiating this select command
I got an empty set (no match)

My questions are:

Has anyone been successful creating a Unicode utf8
MySQL database that accepts Arabic?

If yes, how did you get around or not encounter the
Data too long issue?

Have you tried the cp1250 (or cp1251 - same mechanics
same results) work around as I have? Are you
able to search the cp1250 character string (my colC)?
If yes, how did you successfully manage to do it?

Lastly, if I take the cp1250 encoded string and paste
it into Excel ... I can string search the cp1250
encoding with no problem.

Also, here's how I know my Unicode utf-8 data is
correct apart from my own manual cross-referencing
and being recognized by MySQL in some respect:

When I copy the Unicode utf8 encoding and try to
paste it into the select command to see what would
happen I get the following error:

ERROR 1257 (HY000): Illegal mix of collations
(cp1250_general_ci, IMPLICIT) and
(utf8_general_ci, COERCIBLE) for operation '='

So what I have here is a situation where MySQL
is recognizing Unicode utf8 encoding but not
from the respect of packing a table!

Go Figure ...

---

Full-text Indexing Of Pdf, Rtf, Txt, Html
I have a big pile of files on my local machine that are in a variety of
formats - txt, rtf, pdf, html, etc. What I'm looking for is a script
that will crawl through the files and perform simple full-text indexing
on them, and will allow for queries to be executed on the index.

I gave PHPDig a shot but it prefers web-based setups - my files are all
stored locally and will only be accessed locally.

---

Full Text Search ( Mysql )
I was searching the sql man , for some good ways to search a mysql db. so i found that Full text search the best way is to go becease LIKE just gives to mutch output.PHP Code:

---

Full Text Search In PDF And Word Files ?
I need to perform full text searches on a batch of PDF and Word files.
What is the best way to go?

After some research, I'm thinking of extracting the plain text from the
files with "pdftotext" and "catdoc", hamonizing the various possible
encodings to UTF-8, storing the text in a MySQL database, and then
using the full text search capabilities of MySQL.
Do you think that would work well? I am told that the files are mostly
text and won't be longer than 30 pages.

---

Full Text Search With Pagination Class
I've used the Full Text search example from this site and added the pagination class from http://www.goodphptutorials.com/article/show/simple-php-mysql-pagination/1 but I'm stumbling where it actually breaks the page up into the prescribed number of page. Right now I have script set for 5 results per page and while I can get the total number of records in the search result set, I can't get the results to span over say 3 pages if there's 15 results. Code:

---

Give User A Choice Of Fields To Search With Full Text/Boolean?
I want to give the user the option of searching a combination of the fields $Topic, $Subtopic, $Theswords in Boolean/Full Text. The problem is if I make any of the fields empty prior to the query, I get a MySql syntax error message. The user has the option of selecting fields via checkboxes, but when the field is not selected the user will get the error message because the variable is empty in the query statemtnt.

Are there any code suggestions for avoiding the error message when a field is not selected by the user to be searched? Code:

---

PDF Indexing/Search
I was wondering if anyone has any information about any function and or 3rd party software that can work with PHP to index and search PDF, Word, Excel documents? I am a Cold Fusion programmer starting to go PHP and Cold Fusion has a verity search feature that does this. There has to be something in PHP or products that PHP can use to do the same thing right?

---

How To Build An Indexing Search
I know there are many pre-built indexing search engines, some pretty good like ht://dig. However, I'm running Win NT 4 and consequently can't use it .

What I've found so far isn't bad, but to meet my needs I'd have to revamp most of them. I know I've seen how-to's on creating an indexing search engine in the past, but all I can seem to find now are pre-built engines. I realize that it's probably going to be more difficult to "re-invent" the wheel, but I want to exhaust the possibility that it's not.

---

Session Id In Url And Search Engine Indexing
I would like searchengines to index pages without the session id appended in the url.

When a visitor has cookies disabled the urls look like :

http://www.site.com/main.php?SESSID...7325099c755b25b

This is what a searchengine would see also.


However I would like the searchengine to index :

www.site.com/main.php

It's important for this site that sessions also work for people who have cookies disabled, so simply not appending the session id in the url is not an option in this case.

---

Windows Service
I need to stop (sometimes start) a Windows seevice (in XP / Vista). Does
anyone know how to do such a thing in php?

---

Session_start() Ruining Search Engine Indexing
Google is trying to index my pages with session id's attached to them.

Ex. =   show_cat.php?catid=P70&PHPSESSID=aeg3klbcaeoirjerfodfifs

And I don't know if this is right but I put these to lines in my robots.txt but it still didn't work.
Disallow: /*?PHPSESSID=
Disallow: /*?*PHPSESSID=
I am using session_start();

at the beginning of all my pages. What do I need to do? I will be toast if I don't change something.

---

Gaining Access To How MySql Parses Text For Full Text
I want to gain access to the function or process MySql uses to parse words and phrases for Full Text searching. Here is an example.

If the user inputs...

Milan in history

MySql will search for milan, history, and milan history. Is there a way to extract just the combination of terms MySql uses to search the db without the stop words? Stop words are automatically eliminated from the search request unless the user encloses a phrase in quotes.

What I am trying to do is develop a script to highlight found search terms and phrases. I can explode a phrase into single words but if I do that the stop words would be included in the array. If there is some way of getting into the parsed words or phrases MySql Full Text actually uses to search, I can use each of those combinations as a keyword in my highlighting script.

This request is about searching for the code or any code related to the questions involved.

---

PHP CLI As Windows Service & Daemon
Does anyone have a good solution to run a Command Line Script as a
Windows Service, and be able to monitor it via Telnet (Or better yet
SSH)?

I have a small maintanance script that I would like to run as a service
on a Windows Server. It cleans up parts of the database on an hourly
basis, sends some e-mails etc. It would be great to monitor the
scripts progress without
checking a log file.

---

Mod_rewrite - Changing Dynamic Urls For Search Indexing
is it worth it? or will search engines other than google soon come around and start indexing past the '?' symbol ?

---

Php Service On Windows 2003 IIS6
this is the setup:
i have 2 computers at home. one of them has a registered domain name,
and is acting as a web server (runs under windows 2003 iis6). my
second machine has apache configured with php, perl, and mysql
installed. i wanted to make my 2nd machine accessible from the
internet, so i have created a virtual directory on my 1st machine,
pointing to my htdocs directory on the 2nd machine. when i try to
access a .html or .txt document (on my 2nd machine) through my 1st
machine everything works. but when im trying to access a .php file, i
get a 404 error. why does this happen? can anyone please help me?

p.s.
an example of urls are as follows:
1st machine -> http://www.machine1.com
2nd machine -> http://www.machine1.com/machine2
(machine2 being a virtual directory on the 1st one, pointing to htdocs
on the 2nd)

---

Indexing Text Files In A Folder
I'm newbie on Php. I cannot write codes but only modify.
I need a code doing this:
I'll put text files in a folder like file1.txt,file2.txt ...
I want to search this files, list most recent 10 and if files

call, script will put into the site design with <pre> tags.

---

UNIX/Windows Text File Compatibility
I guess it's a bit off topic but my script is written in PHP so I
guess somebody out there could help me.

I'm facing the well known issue between windows and UNIX related OS.

The file format is crucial. The file is made on a linux box and as to
be parsed on a windows box by a commercial software (so I cannot play
with the code on that side).

I'm having a problem to mark the end of lines.
"
" as you all know doesn't pass well on Windows.

How could I replace this one:
fwrite($fichier, "this is a new line
");

to make it works with windows?

---

Is It Possible To Restart Windows Apache (service) On A PHP Script?
Is it possible to restart Windows Apache (service) on a PHP script?
i have installed PHP/Apache on a Windows machine. Added the ext windows32
service. But could not find any how to or information online.

---

Search A Text File
I have a text file that each line is an array, I think. I looks like this

name | id | # | # | # | total
name | id | # | # | # | total

and so on...

I would like to seach the file by "id" and only display the line with the particular "id".
I also think that the bit size of each line will change from line to line, so I don't think that I can put in a static number to search to the end of. I thought about using fseek(), is that the best way to go.

---

Search And Replace In Text File
I have a text file with 30 or so lines. Each line has this format:

sometext somevalue. They are separated by a single space.

I want to search in the file for "sometext" and replace "somevalue" with a particular string.

---

Download Database Search Results As Text File?
Can someone please tell me how to query a database and then download results as a text file.

---

Search For A Specific Word Inside A Text File?
Basically what i'm trying to do is when a user inputs login/password information at a login page, I want PHP to search inside verify.txt and if it finds the login/password combination then allows the user to proceed. Is this possible? And if so, which functions would I use to get the job done?

Also, how can I save the login name so that it can be passed to/included in a url?

---

File Indexing
I would like a script that can do the following.

1. Check a given directory I.e "files"
2. Dump the Name of the File and the Location of the file (relative to the script) into a MySQL Database Table called for example "files" under the headings Name and Location I need the script to loop until it has indexed all the files in the given directory.

---

Creating Temporary Table From Full-text Query
i'm using a full-text search on my db which works fine, but now i'm working on filtering the results further, like products in range of prices. my logic is to simply SELECT them out of the temporary table, only if it would be created. i don't think i need to paste any code since i think you get the idea.

---

Delete Entire Line In Text File If Anything In The Line Matches Search String
how would I code a php script that would open a text file, look for a string i specified, and if it finds that string, it will delete the entire line in the text file that contained that text. I'm new to php, so sorry if this is really obvious..

---

Trouble Getting Full Path To Client File
using php 4.2.0 on xp and writing a simple uploader. using $_FILES["name"] was expecting the full path to stick through basename(), but its only returning the already trimmed path. How d'ya get the full path using $_FILES...?

---

How To Find Full Path Of A File Upload Field In A Form?
I need a way to find the full path of a file uploaded by file upload form box.
$_FILES['file1']['name'] gives just file name.

I have a file upload form and if the user submitted something wrong I want the page to go back and have the forms maintain its values (the full path to local file, e.g: "c:/docs/landscape.jpg"). Any way to do that? This is a more complex applications so using another upload tool is not an option.

---

Windows To Unix Text Doc Parsing Problem
i have been having trouble with a login script that works on my
windows machine, however when i upload it to the Unix server through
VPN, the same script won't work! It won't parse member.txt properly i
think. The password and usernames i am using are at the bottom of this
post.
Each time i go to login on the unix server, it clears the username and
password field. I have been attempting to solve the problem, but have
been baffled to see why there is such an issue. I have taken out the
HTML, and just given you the PHP script that i am running. The
original programming was done on Unix!.

My question to people on here is why does my script run on Windows and
not on Unix? How can i fix the problem?

Here is the source code i have been looking at:

<?
session_start();
if ($userid && $password)
{
//listf stores a reference to the file itself
$listf=fopen("username/member.txt","r");
#read the file
list($duserid,$dpassword)=fgetcsv($listf,1024,"|");
$duserid=trim($duserid);
$dpassword=trim($dpassword);
//keep reading until the end of the file is reached
while(!feof($listf)){
if(($userid==$duserid)&&($password==$dpassword))
{
$_SESSION['valid_user']=$userid;
break;
}
list($duserid,$dpassword)=fgetcsv($listf,1024,"|");
$duserid=trim($duserid);
$dpassword=trim($dpassword);
}
fclose($listf);
}
?>
------HTML-------
<?
$tempstr=$_SESSION['valid_user'];
if (isset($_SESSION['valid_user'])){
echo "You are logged in as: $tempstr<br>";
echo "<a href="admin_home.php">Admin Home</a><br>";
}
else{
if (isset($userid)) {
// if they've tried and failed to log in
echo "Could not log you in";
}
else{
// they have not tried to log in yet or have logged out
echo "<table border=0 width="600" cellspacing=0 cellpadding=0
border=0 valign="top" align="center">";
echo "<tr><td>You are not logged in.<br></td>";
echo "</table>";......

---

Smart Text Search - How To ?
I've got a list of names in a mysql table:

example:
Mike
John
Peter
.... (100,000) entries


How do I perform a search in this table, so that it is tolerant to mild
spelling errors ? Am I looking for a phoenetic search algo ? Or is
there a simpler algorithm to do this ? Or perhaps a PHP library for
this ?

For example if someone searched for "Jon", he should find "John" from
the table.

---

Search A Text For Linebreaks
Got some text that contain url`s. Whenever there is an occurence of the string "http://" its going to read what comes after it - until next whitespace and make a link of it. But theres a problem when the link is at the end of a sentence, then it doesn`t treat the linebreak as a whitespace and the first word on the next line is treated as a part of the link. Is there any way I can search for linebreaks in a text/string?

---

Text Search Question
I'm working on a new project with php & mysql.

I anticipate that one of the tables may grow above 40 GB in a short
time. The main content is text. Now what tools do I need to implement a
text search in this big text database ? I'd like the search to be quite
advanced, supporting boolean operations and metaphones and so on.

What should I be surveying now ?

---

Search A String Of Text...
I would like to search a string of text for "x" if "x" is found then it echo's something, else it echo's something else... Anyways I'm kind of lost as to the best way to search a string of text (for example "1,2,3,4,5,6,7,8,9,10,11,12") and find lets say "7,"...

---

Search Through A String For Various Text
My users enter text into a database field in the form of:

this file {link:afile.pdf} is big and this file {link:another.doc} is small.
I have no idea how many times they may enter {link:xxx} into a given field.
The code we worked out (see other post below) fabulously translates the text into a hyperlink.

The problem is that no matter how often the {link:xxx} is in a given field, I only ever get the first file returned; i.e:
This file afile.pdf is big and this file afile.pdf is small.
But what I should get is, obviously:
This file afile.pdf is big and this file another.doc is small.

I need to use some kind of looping function, or a foreach type function. Does anyone have any clues? Code:

---

Eregi Search And Replace For Some Text
I'm trying to do a search and replace for some text. For example, I have a word that is 'history', I want to replace every occurence of 'history' with:

<a href="history.html">history</a>

Now, that's easy, I can do that - however, as 'history' could also be 'History', I want to ensure that the format of 'history' stays the same. For example: $strRep = 'history'

$str = eregi_replace($strRep, '<a href="history.html">'.$strRep.'</a>, $my_other_string);

That's fine - however if history is capitalised (History), then I have a problem:

$str = eregi_replace($strRep, '<a href="history.html">'.$strRep.'</a>, $my_other_string);

because the above example would lowercase History to history.

---

Best Text Search For Many Gigabyte Database?
I have a so big DB, It's about 10-100GB of text.
Now I want to create a searching function but It take too long when I
use LIKE from MySQL
Anyone know the best sollution?

---

Read And Display Japanese Text From Text File
I posted a question regarding reading japanese
text from a text file.

Well, since I solved the problem, I thought I'd post my solution for
the benefit of other people with the same problem.

The plan was to make a script to read and display japanese text. I
will use it for making a japanese proverb script and for a japanese
language study script.

Method :

I wrote a simple kanji text file (saved with UTF-8 encoding)
I wrote a simple PHP script to display the file contents (saved with
UTF-8
encoding)
I specified the content-type header for the HTML page :
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

*** All files have the same encoding. ***

UTF-8 supports japanese characters.

and it works!

this is my PHP (and HTML) script :

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>PHP : Japanese Text File Read : Exercise 1</title>
</head>
<body>

<?php

$filename="japanese.txt";
//open file
$fp = fopen($filename,'r');

//loop through each line in the file
while($line=fgets($fp))
{
//output current text file line
print $line."<br>";
}
//close file handle
fclose($fp);

?>

</body>
</html>

I know it's a very simple script, for testing purposes only. It
displays the contents of the japanese text file line by line.

The key was to save all files in the same encoding (I used UTF-8) and
to specify the encoding / charset in the HTML header (<meta
http-equiv="Content-Type" content="text/html; charset=utf-8">)

---

Open, Search, And Process Other (text) Files
How could I use php to open, read, and perform operations on a file?

Say I wanted to open somefile.htm and search for instances of <p class="one">Foo</p>, strip the tags, then put 'Foo' in a variable called $foo... Is this doable?

---

Highlighting Search Text, But Not Inside Tags
I've looked at some examples of text highlighting in this forum, and they all work fairly well. But I have a problem with my html tags. I have a site where the main body text is kept in a database text field. This text can contain some simple html. In particular urls. These urls get completely messed up with the css-tags that i insert to highlight the search words. Is there a simple regex expression that can make my script skip anything between < and >?

I am currently using the example from this thread:

---

Dictionary Search Definition Replace Of Text
I'm trying to search a weather report for matches to the keys in my associative dictionary array and replace them with the $term => $definition.

following is my code. again, any advise appreciated.

REF: full php source PHP Code:

---

Help On A Multiple Keyword Search On A TEXT Field Type
Working on a php/mysql driven site which requires me to have a search facility based on multiple keywords. I read the tutorial "quick and dirty search engine". What i want to know is :

1. Is there any SQL query i could perform which could to almost the same thing ( i dont really need the accuracy ) ?

2. Any other methods of achieving it?

---

Remove Text Header In Text File
I have been having trouble removeing two header lines at the top of my txt file. You will see in my code the different things I have tried but with no luck. Below I posted my script and data. I am doing a few other things in my script as you will notice but this is the only thing I am having troble with. Code:

---

Dir Indexing
I have a folder "archives" full of folders for every month (Jan04,Feb04,...) this script is supposed to add them to an array but it doesn't. DID I DO SOMETHING WRONG?

$dir = "archives/";
$d = opendir($dir);
while(false!== ($f = readdir($d))) {
if (is_dir($f) && ($f!= ".") && ($f!= "..")) {
echo $f."<br>";
$folders[] = $f;
}
}

---

PHP Is Looking For Php.ini File In C:windows
As the installation txt suggests, I have left my php.ini file inside
the c:php directory... but running phpinfo() shows that my php.ini
file is in c:windows as shown below.

Configuration File (php.ini path) - C:WINDOWS

Well there is no php.ini file in my windows directory, so I'm not sure
where it's getting it's configuration settings from... unless I suppose
they're built into the executable somewhere if no php.ini file is
found. My questions is - why php is not locating my php.ini file that
is located in the c:php directory? I'd rather not move it to the
windows directory so I can keep everything nice and tidy like the
installation file suggests.

BTW - This is a brand new laptop I'm installing PHP on so there is (I
just double checked) no PHP.ini file in the c:windows directory. The
path is set to include c:php... and opening a DOS prompt and typing
"php" is a recognizable command regardless of my current directory.
This is the 2nd machine I have had this same problem with. I'm still
confused...

---

Document Indexing
I need to build an (databased) index from uploaded documents (.doc, .xls, .ppt, .pdf) for searching purposes, using php, of course. Does anyone know about this issue?

---

Hashing And Indexing
i am trying to count all of the words in a string, and am trying to
determine the best data structure for this.

i would like to be able to do something like

$arr_text = explode(" ", $text);

then, i want to count each unique word in $arr_text

i tried using the words as the array offsets, and of course i found
that this is illegal.

is there a php data structure which can accomplish this for me?

something like python's dictionary data structure would be perfect.

---

Google Api Indexing
I'm currently playing around with the api with php and came around some error (No Signature match) If you're familiar with the api, here goes: PHP Code:

---

PHP 5 Windows File Upload
I need to define a loaction for file for uploads. I am using Win server 2003 PHP 5. I have edited my php.ini to define a location for uploads.
upload_tmp_dir=/somedir/somesubdir/

I restarted my server and I still get no value for upload_tmp_dir.

Any ideas what's wrong?

---

File Permission In Windows Using Php
I have a doubt that how to set a directory, file permissions in windows using php.

---

PHP In Windows Media ASX File
What I am trying to do is get my Windows Media Player embedded pages to point to one .asx file instead of having to manage the stream links of some 30 odd pages individually.

My question is how would I get the html page to "submit" the info to the asx file? Is there a way for the page to automatically submit the stream info to the ASX file using the POST method?

---