PHP :: Data Mining / Data/Website Scraping

Obtain Data From Another Website And Store It?
Basically theres a website that displays the results of football matches and every few days. I want to somehow grab those results and put them in a mysql db and then show them on a part of my website. The layout of the results is always the same.

I know some basic php but I don't have any clue as to how I would go about doing this. Has anyone done this before or something similar?

Simulate Send Data With POST And Get Data From A Web Page (whith A Session)
I'd like to get text of a webpageB that I reach by putting
a login and a password in a webpageA.

I mean:
I'd like to get text of a WebPageB with a php script.
But before I can see WebPageB I have to autenticate myself
in a WebPageA writing my login and my password in a form,
the host recognise me and (by session) let me see the WebPageB.

The question is:
1)how to simulate send data (login an password) in form with a PHP script?
2)how to navigate in a Web usin the same session?

Problems With Reading Data From A Text File And Inserting The Data Into DB
Ive been having problems getting this to work for 3 days now. Basically what im trying to do is read from a text file and insert the information into mysql. The text file is in this format:

"LuCyndi@infiniti.com","carol","kellerhals","109 veeder dr","las vegas","NV","89128","702-254-2061","45"
"the_prophet@att.net","a.","cohen","6794 e. bonanza rd","las vegas","NV","89110","702-463-6569","55"

I understand how to open files to read but im having problems turning each set of data into an array so i may insert it into mysql. If someone could please help it would be appreciated.

Comparing Form Data To Mysql Data
Im interested in building a script that would take form data, a variable like an address, and compare it with a known address in a mysql table (called addresses), then if there is a match log the address information in a separate table called (addresslog), and return the visitor to a certain page. If there is no match between the address entered in the form and known addresses then the information is still logged into the addresslog database but the visitor will be sent to a different page.

I'm still new to this, and am working through the logic and the syntax, but I believe I'm close. It's still not working, I have all of the proper tables, maybe I'm way off with the coding so I thought I would seek out some expert advice. Here's the entire script with as much explanation as I could give. PHP Code:

Data Type Change Without Loosing Data
I have a problem in mysql. I stored a date in 08/14/2006 format and choose a datatype longtext now i want to store date in 2006-08-14 form is it possible to do it without loosing data in db when i convert datatype longtext to date the data lost. Any solution?

Showing Data From Data In Specific Interval
i will make a banner system to my website. there is some information in the database i get out with to my site. that is pasteing the data in random way. i have a date with day-month-year and a time .. hour:min. how do i do in my code so its is displayed in ex 1 month from that day it is acceptet. give me an example to how i could do it.

Delete Data From The Data Base
Im doing the PHP-MYSQL program to search the data in the database and display on the HTML Page. Everything is working fine till now. Now I want to know that, how to write program to delete the searched and displayed data from the database.....

I know the MySQL Syntax to delete data from database...Im not asking about MySQL programming. I want just an idea, how to do the delete process. Here is My full PHP script, Code:

Comparing Database Data And New Data
I am trying to create a PHP script that reads in a file, parses it for the data I want and compares it to the data I have in a database.

The data is points and ranking information for a Folding@Home Team. Every week someone creates a news item with points gained by each team member for that week and other info. This is done manually and takes a lot of time. I hope for my script to automate as much of it as possible.

The script was to run as follows:

Download and parse latest statistics into 2D array
Compare usernames in array to usernames held in Database
Any new usernames and their respective points/rank to be added to database

Compare points in 2D array of each user to those held in database.
Output username and points gained

I thought I had a working script until I actually tested it on real data, and realised I missed something rather obvious - the order of usernames from the parsed file is not going to be the same order as in my database - due to people moving up and down the ranks. So i can't just get a recordset from the db and run a simple loop to compare it to the parsed data.

I am not sure how to go about comparing the database information with the parsed data that works around the fact the data could be in any order. The only thing i can think of is:

foreach item in $parsed_data_array
query database for $parsed_data_array=>username
if(number of results == 0)
build new insert query for current username
else
continue

I don't know if this works [haven't tested yet] but I can't help feel it is a little DB intensive. There could be up to 200 queries being sent to the database. This can be minimized by adding extra criteria, like ignoring inactive users of course. Is this a viable solution or is there a much better way of doing this?

Retreiving Data And Maintaining Data
I am trying to add data to a table in oracle but get

Warning: ociexecute(): OCIStmtExecute: ORA-00936: missing expression in /homedir/ilex-s02/mrace/public_html/addnewemp.php on line 18

here is the PHP script i am working with, Can anyone see any faults with it? Code:

Get The Data From Those Rows That Have Data
Hi Guys,
If i were to have the following php query:

$query = "SELECT * FROM alist WHERE list_id = $id";
$result = mysql_query($query);
while($row = mysql_fetch_array($result)){
echo "<b>{$row['List_Name']}</b><BR>";
echo "{$row['address']}, {$row['state']}<BR>{$row['post_code']}<BR>Tel: {$row['phone']}<BR><BR>";
echo "{$row['Info']}<BR>";
echo '</blockquote>'

....how can i get the data from those rows that have data.
For example, some rows won't have a 'phone' value so i don't want this to display "Tel: " if there is no value associated with 'phone'.
What do i need to add to my script inorder to achieve this?

Thanx...

Posting Form Data And Also Allow To Download Form Data
i have a client that whats to post form data to a database but he also wants users of the script to download it aswell how will i got about doing this i thought about it that i could make it save it into a txt file then echo the download link is there any ideas? because to post it to a txt file it would be posting 2 places is there a way to do this? Code:

Screen Scraping Help
I need to find out how i can screen scrape the top ten virus threats from a website, store them in an txt/xml pref xml and the from there i can display them where ever on my site. I know people like NAI.com offer something you can put on your site but you have to have there logo and there layout which i dont want.

I know you can do it with asp but i hate asp so i would like to know how i can do it with php bearing in mind im not exactly hot on it.

Screen Scraping
I need to do some screen scraping from PHP. I'm planning on using

$content = file_get_contents( "http://www.somesite.com/somepage.html" );

to get the page and store it as a variable. I'll then parse it out to get what I need. But I'd rather not reinvent the wheel if there's code out there that might give me a basis to work off of. Anyone know of any? If not, does anyone have any suggestions for parseing the html to find what I need?

Web Scraping Blocked
I have been using some PHP code to get some football scores from a website for a while, but it has stopped working.

I can see the site myself, and the source code is as before.

When my PHP code tries to read it though, the page is just one line. I assume this is intentional by the site. (I'm not sure why, as the results aren't theirs, and there are plenty of other sites with them.)

Anyway, I am simply getting them from another place now, but I am wondering how they do this, and if it is avoidable.

Image Scraping
I've just been told about something called web scraping where you can extract data from remote websites, and use it on your own. I was wondering if it's possible to do this with php, with images.

i.e.

IF i have a variable for movies in my url "movies=Borat" the code would somehow search a site like google images for example and pull up an image of borat and put it on my site.

Is this possible? if so, where should i start reading into it?

Web Site Screen Scraping In PHP?
Are there any hot tips for stuffing a couple strings into a simple
website page? I want to help people that need to look up a bunch of
zip codes on this website.

http://zip4.usps.com/zip4/welcome.jsp

I expect to hit the URL with Address, city and state pre-loaded and
invoke "submit" and let a Human Being see the response and adjust the
input as necessary.

Scraping Headlines From A Text File
I've got a programming logic problem (as is often the case with me!) about exactly how to go about the following:

I've got a set of headlines and accompanying stories stored in a text file. The file looks like this:

++ Headline 1

A few lines of text

++ Headline 2

A few lines of text

++ Headline 3

A few lines of text

++ Headline 4

A few lines of text
(etc. etc.)

What I need to do is end up with an array of the first three headlines from the page, without the plus signs before, so I can display them on another page of the site.

I know I can use fopen to open the file, substr($value,0,3); to remove the plus signs, but I'm not sure of the best procedure for putting it all together.

How would you go about ths process in the simplest way? (Pseudo code would be just fine - I can look up the functions, I just need to get the logic!)

Screen Scraping Solutions - Maintain Sessions?
I have an affiliate deal with a company who have given me a number of affiliate urls to run in a frame.

Their branding is included in the site, and I want to remove this. I have their permission to do this.

The urls use sessions.

Is there a way to include these urls in my pages (with the branding stripped out), and maintain the sessions?

Im thinking i need to use CURL or sockets and run everything in an IFRAME .

any tips?

Its a linux server.

Perhaps there is software or scripts already avaible for this?

How To Upload File From Website To My Website?
I want to make upload page. I could upload file from computer to my website. But I don't know how to upload file from website to my website? How can I make it?

Getting Data From The URL.
I´m using php to build dinamyc images. The whole process is done via form. The user enters some data in text fields and them a script retrives this info and builds the image.

The next step is to provide the users with a url so they can display that image in their site. But I need that image to be updated ( if needed ) when it is called to be displayed.

My question is this, how do I get data from the url and use it on my script.

Let me give an example:
The url for the image would be something like:
http://www.myserver.com/images/fabiop.jpg

I need to retrive the username ( fabiop ) and the extension ( .jpg ) and use them in the script to generate the image on the fly to be displayed. How do I acomplished that ?

I´ve seen this working on some sites but I didn´t figure out how it was perform. My best guess was that this sites have a webservice running in the background that performs this action.

Getting Data From A Txt File
Get data from a txt file, and say that the first 8 characters are the id, the characteres 9 to 17 are the date.

Sorting Data
Im displaying data from a mysql query that displays a Player, email, id etc.. and I have links that sort the data by player, email, id etc... But Ive seen websites where you click sort and it will sort the data by that and if you click it again it will sort that data this time in descending order.

How am I able to have link that will sort the data normally and descending order?

Parse XML Data To PHP
When parsing XML data in a php file the script is told where the data is. For example:

Inserting Data Into A Row
I’ve got a table of three fields (id, name and comments) that contain 30 rows. Each row contains data in the two first fields (an id and a name). The third filed is empty.

When a user accesses my page they can select one of 30 names from a drop down menu. After they’ve selected a name the user can write a comment about the person he/she selected.

How can I insert the comment from the user into the comment field related to the name selected?

Dynamic Data
I have included a php script into my page
as <img src="plot.php">. That script just plot some time changing data. Now I have two questions that I cannot solve myself:

(1) When I press the reload button I expected
the plot reflected the changed data.
Instead of that plot remains unchanged
until I directly invoke the php script
from the browser...
Is there any method to avoid this
unwanted caching?

(2) That is a bit off topic but is there
any easy way to get a web page updating
after a certain period of time?
Once solve my first question I'd like
to offer mi visitors a automatically
changing graphic.

Truncating Data
I have a simple form, going to a simple catch page, but it doesn't work perfectly. In the one text field, it truncates the first 4 characters of the text. The rest of the string comes through fine, but it's the first four that get lost.

this is the top of the form, and the field that is givng me a problem.

Extracting Data From XML ?
The following is the sample XML file from which I want to
extract the data of which availabity ="Available" using php.

The XML file will be generated dynamically from the URL http://www.upc.nl/zipcode.php?zipcode=3565BG

<UPC-Results>
<DateTime>05-12-2005 02:08:26</DateTime>
<Errors/>
<Products>
<Product name="CaTV" availability="NotAvailable"/>
<Product name="Chello Starter" availability="NotAvailable"/>
<Product name="Chello Easy" availability="NotAvailable"/>
<Product name="Chello Light" availability="NotAvailable"/>
<Product name="Digitale telefonie FreeTime" availability="NotAvailable"/>
<Product name="Chello Classic on DSL" availability="Available"/>
<Product name="Chello Plus on DSL" availability="Available"/>
<Product name="Chello Extreme on DSL" availability="Available"/>
<Product name="Digitale telefonie Basis on DSL" availability="Available"/>
<Product name="Digitale telefonie FreeTime on DSL" availability="Available"/>
</Products>
</UPC-Results>

from the
above XML, I need to get data of Product name which has the status of
"Available"

Excerpts Of Data
I am working on a project that involves PHP and MYSQL. I am currently pulling information from a TEXT field (could be LONGTEXT). I would like to be able to create an excerpt from the data.

So...say I run this query:

SELECT review FROM bookreview WHERE id=1

That will give me a rather lengthy review. What I would like to do is display only the first X number of words or characters (preferably words), add a ... and then link it to the rest of the story.

My question is how to cut short "review" after X words or characters. I presume it can be done in either PHP or MYSQL but I have no idea how.

I am uncertain if this is important, but this query is going to loop at least several times so I need to be able to work whatever code you give me into a loop so I can pull multiple reviews.

Sort Data
When I click a title of table, I want to sort data. It same phpMyAdmin.

Select Data From Db?
I have a db that I need to select data out of. In the field it is selecting out of there is data input there from an array and then implode is used to seperate each item with a, When I run a query and wanna select data from this row I need the data from the previous form to = only one of the items in the row, but could be any of the items. how do I tell the page or the query that it could be any of many things in the row, but only has to be equal to one of them?

How To Use PHP To Retrieve Data From A PDF
I am looking for a tutorial, script about how to use PHP to retrieve data (Ex: filled customer input in a PDF file) from a PDF file and store it as a XML file.

Printing Data?
why the following code isn't printing any data from the selected database. PHP Code:

Logging Data
i am making a website for off-housing department at my university. the dept. wants to have a daily log so taht they can check on a daily basis, as to how many ppl entered the website on a paricular day. Seperate logs for each day would be preferred.

Check Data Not Already In D/b
Hwo can you do it, on an insert query so that it will check whether there is a record with the same info, say club name for example, and then if u try input same thing again it will reject it.

Data Sorting
I have a database table that has these fields: id (auto increment), date, message and what I wanted to do is sort the messages in order starting from the latest messages. I then wanted to sort the messages and only show 5 messages per page. I managed to do all this but now I also want all the messages with the SAME DATE (but different times) to be shown in the same table. for example: Code:

Storing Data
Say I have a number: number = 34043923, this represents a wordID, so the word belonging to this wordID is say: forum (whatever).

This can be stored in 4 bytes (so i am told ) BUT when I save it with fopen/fwrite it takes up 8 bytes (1 for every character) and the file can be read (when opening the file it just reads 34043923).

How can I store this data so it takes up only 4 bytes AND how can I do this storing it in a textfield of a mysql db?

How To Add Data To An Array
I have a question about how to add to an array. below is a test program for an associative array in which the value is also an array. I can work out how to fill the array when it is declared (as shown) but I have not been able to understand how to add another set of data later on. PHP Code:

Truncated Data
My data is truncated when updating to the database. I have two pages. "Edit page" and "form page" I added some debug codes in my page.

My results:- (echo $t1_client)
In edit page - t1_client = "Company ABC"
Before submitting in form page - t1_client = "Company ABC"
After submitting back to edit page - t1_client = "Company"

I cannot spot any thing missing, so I was hoping for some experts here to help me out.

Here is my form page:- (the field is $t1_client)
<select size="1" name="t1_client" >
<option value=<? if ($t1_client) echo $t1_client; else echo $jobs_cust['jobs_cname'] ?>>
<? if ($t1_client) echo $t1_client; else echo $tt_client; ?>
<? client_pull_down(); ?>
</select>

Postcode/zip Data
I need to make a 'find nearest' for a couple of different countries, does anyone have zip/postcode databases for all countries, or, if anyone just has the data for one country that would also be good.

SSL : When Data Is Not Encrypted?
If I have a PHP login page that is displayed via an SSL connection and then the form sends a POST request to a page that is not requested via the SSL connection, is the transmission of the username and password encrypted?

I thought it was as the login page is encrypted but just want someone to confirm this to give me a happy,warm feeling. Are there any security flaws with this method?

Feeding Data From Txt -> Sql Db
im making small catalogue in php. just add, view and order features. I first thought it might be easier to make the product database in .txt -file but when/if the catalogue grows its a lame solution(?) So is it hard to make a script in php which feeds data directly to sql db or form -> txt -> database.

Data Transfer From A Non-SSL To And From SSL
I would like to know how we can transfer data through a
secured channel to a non-secured site. I am designing an
e-commerce site and getting data from secured site
database(on a firewall) and display to my site.

Posting Data
I've got form with few submit buttons and one select field.
I need to post data to another file or to the same document.
Is it possible?

How Can I Insert Data In A Pdf From Php??
I have been reading this group but I can not find the simple answer
tohow i can solve what i need to do. I have a pdf template, it is, a
pdf with some data like name, surname,etc ...and i want from web
(using php) to open this pdf, fill thesedata and repeat it so many
times as number of students there are... I have read about fdf , etc,

Data Structures
Is it possible to create data structures in PHP and if so, would someone
provide a bried example?

Example:

I would like to create a Data Structure JRec which has the following fields

lID (Long)
dtDate (Date)
sName (10 Character String)

Extracting XML & RDF Data Using PHP
I have an XML file that contains metadata in the form of RDF. The file
is something like this:

<?xml version="1.0" encoding="utf-8" ?>
<xmpData>
<WavXmpInfo path="C:.....................................")
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about="" xmlns:xmpDM="http://ns.adobe.com/xmp/1.0/
DynamicMedia/" xmlns:xap="http://ns.adobe.com/xap/1.0/"
xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
xmlns:xapRights="http://ns.adobe.com/xap/1.0/rights/" xmlns:dc="http://
purl.org/dc/elements/1.1/" xmpDM:audioSampleRate="16000"
xmpDM:audioSampleType="8-bit"
xmpDM:audioChannelType="1"
..........
..........
..........
<dc:rights>
<rdf:Alt>
<rdf:li xml:lang="x-default">HHHHHHHHHHHHHH</rdf:li>
</rdf:Alt>
</dc:rights>
</rdf:Description>
</rdf:RDF>
</WavXmpInfo>
</xmpData>

I want to extract data such as audioSampleRate, audiosampleType etc,
so that i can insert them into a database. I'm having problems
accessing this data.
I'm using DOMXML. Can this type of data be extracted using DOMXML? If
not how would one go about accessing it?