Customizing The "User Agent" With CURL Screen Scraping?
I have a script which performs a screen scrape of a webpage using cURL. I was wondering, how can I customize the "User-Agent" that shows up in the server-logs whenever my tool is used so that I can track it's usage on my site.
View Complete Forum Thread with Replies
See Related Forum Messages: Follow the Links Below to View Complete Thread
FTP_login + $pass="" And $user=""
I use the following scrit in an Intranet with the user name = "" and the password = "" : <?php $ftp_server = "192.168.25.25"; $ftp_user = ""; $ftp_pass = ""; $conn_id = ftp_connect($ftp_server) or die("Couldn't connect to $ftp_server"); if (@ftp_login($conn_id, $ftp_user, $ftp_pass)) { echo "Connected at $ftp_user@$ftp_server "; } else { echo "Can't connect with $ftp_user "; } ?> I have the following message : Connected to 192.168.25.25 Warning: ftp_login(): 'USER': Invalid number of parameters in E:www est_ftp.php on line 26 I tried to : ftp_login($conn_id, "", "") but i have the same result...
Post Form With Field Containing "curl Curl"
I have a form that posts and 1 field is for a suburb, 1 of the suburbs in our area is "curl curl". When ever this is entered we get an error message "You don't have permission to access".
How Do I Do Something Like $color = ("red", "yellow", "blue")[$colornum]?
How do I take an index of a literal array, instead of a variable? That is, say I want to get the short name of a month. Can I do something like: $monthname = ("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec")[$month]; or do I always have to do: $blah = array("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"); $monthname = $blah[$month]; It seems a shame to have to create a variable specifically for this one-shot purpose.
Please Heeeelp - CURL Lib Appends "0" To All Output
I'm using the wonderful CURL lib to handle data between php-scripts in the background. However, I am experiencing a problem: Whenever the processed data is returned for output I see a Ɔ' (zero) appended to it. I seem to be unable to get rid of it. I don't even know where this zero comes from and why it is there. Any help greatly appreciated. A simple example: 1.) Script 1 handles some data (e.g. some form input) and sends it to script 2. 2.) Script 2 then processes the data and generates the output. Here is a live demo which produces the output "test output". Just click the following link and note the appended "0": http://ulices.com/curl_test/script1.php Script 1 looks like this: ================= <?php $url = "http://www.ulices.com/curl_test/script2.php"; $test_value = "test output"; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL,"$url"); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, "test_value=$test_value"); curl_exec ($ch); curl_close ($ch); ?> Script 2 looks like this: ================= <?php $test_value = $_POST[test_value]; echo $test_value; ?>
Curl "remembering Cookies"?
In which the following is said: ---- > curl_close ($ch); > $ch = curl_init(); If you don't close and re-init, you may not even have to save the cookies to a file, and just let curl keep them in memory. ---- Now l'm pretty new to curl, and was wondering how l is it possible to "let curl" keep cookies in memory without creating or writing to a file (ie COOKIEJAR)?
Mail() Won't Work; Server Says "Unable To Authenticate User..."
I am working on a sendmail script using the PHP mail() function. If I click on the HTML form page from within my hosting control panel (while I'm logged into my hosting account), fill out the form, and then click on the "submit" button the mail sends just fine. However, if I log out of my control panel and approach the form page, just like a visitor or stranger would do, the PHP mail() function will not send the mail. If I look inside the "./tmp/mailError.log" file there is the following message: "Unable to authenticate user. Message not sent." It appears that the server will not send the mail because there is not any identification for the outgoing mail. I have tried inserting a fourth variable in the PHP mail() function a couple of different ways like so:
Preventing Visitors Ability To "re-visit" A Page Via The "back Button"..
I have a rather complex registration/member sign up form.. which gets a little confused, when a visitor "backs" completely out of the form, then for whatever reason returns to the form. The form contains 'cancel' buttons, which naturally cancel the reg process and redirects them to the document just before they started. JS provides us with the "location.replace()" method, but I can't seem to find a way to coordinate the use of location.replace() with the posting of the form values. In short, I want to "replace" the first step of the form with the "second step". I think what's making this tricky, is that each step of the form are located within the "same" page (register.php). The document works like so: The first step is the "Member Agreement" when they click "agree" (submit) the system creates a new row in the temporary register table of the db. It then issues a registration cookie, which carries the row id, and session value. in addition (for those cookie impared users) we carry the row id and session value in hidden elements. So far, as long as the user continues through the process all is fine, but if they (for whatever reason) back all the way out, and click 'agree' again, the process fails. I guess I have to find a way to force a reload of the 1st step "if they've gone back" OR "replace()" the first step with the second, so as to prevent the user from going "back" to the 1st step of the register page.
Using Setcookies In My "new User Login" Program
While using setcookies in my "new user login" program, i'm facing a problem. My cookie is storing the login name in the same page but as i go to another page and insert information about that user, who has just logged in, then it doesn't take the login in the sql query.
Pagnation - Limit Number Of Pages Between "Previous" & "Next" Hyperlinks
Was hoping for some advice with a part of page navigation.... It works at the moment so there is a "Previous" & "Next" Hyperlink underneath the records. In between the Previous & Next links are the number of pages that there is of these records. e.g: Previous 1234 Next This is fine if there are only a few pages of records, but when there are hundreds of pages the numbers are listed and they take up half the web page with them. How can i make it so only limited 10 numbers of pages say appear between the "Previous" & "Next" Hyperlink and as you progress through the pages they are as do the next 10 numbers of pages? Code:
Checking If $user & $pass = " "
if !( $user=="" || $pass==""); mysql_query("INSERT INTO users (id, sn, pw) VALUES ( '$idhigh + 1', '$user', '$pass')"); It wont load up my page when I try to run it with this in my page, what am I doing wrong? It checks my text boxes to see if the user and pass equals a blank input when the person clicks on the submit button, and if it does, then it does nothing.
[regex] Stripping Out "name" And "value" Values Out Of Tags, Aaargh
Searched through the site, but couldn't find the right regex syntax I was looking for. the thing i want is, to extract with preg_match_all the given values in a tag... ex. $str = '<input type="text" name="name" value="value">' $pattern = some magic stuff I need to know preg_match_all($pattern, $str, $result); And in result I want to have the values text, name and value AND it's important that the order doesn't matter so structure of tags can be <input type =name= value=> or <input name= class= value= type=> etc, etc...
'Parse Error' Problem In "isset" And "empty"
I've been having some problems with a parse error that I can't figure out (PHP 4.3.11 on Solaris9). Sample code: <?php // getting strange parse errors on this class A { var $value; function A() { $this->value = 1; } function getValue() { return $this->value; } } $a = new A(); if (!empty($a->getValue())) { echo "success"; } else { echo "failure"; } ?> The result of this is: Parse error: parse error, unexpected '(', expecting ')' in /foo/public_html/parse_errors.php on line 15 I also get the same effect with "isset", however, testing with "is_null" or "is_int" produces the correct ("success"/"failure") result. Any ideas?
Echo "<!--#include File="top.html" -->";
I have a search script which returns a page of results, however I'd like the returned page to be in the same format as the rest of my site. To do this I've tried adding the HTML for my site to my PHP page with echo" " commands. This works for the plain HTML, but my regular pages also use SSI to do other stuff (some of them even execute other PHP scripts to generate tables of data etc). I've tried echo "<!--#include file="top.html" -->"; but it doesn't work as I get a parse error. My question is, how do I go about doing this properly? Is this possible at all? And if not have you guys got any suggestions to get around this?
Any File Size Limit In <PARAM Name="***" Value="***"> ?
I embed an applet in my php script which has a parameter providing an input file for the jar. Does this file has size limit? The code is as follows: <APPLET ARCHIVE="myapplet.jar" WIDTH=372 HEIGHT=360 VSPACE=0 HSPACE=0 ALIGN=middle> <?php echo "<PARAM NAME="data" VALUE="myfile.">";?> when "myfile" is beyond 15M, my applet cannot load sucessfully, but while this file is limited like 5M or 7M, it works well. Does anybody know is the reason caused by php script limit, applet mechanism, or by different browser?
How To Do PHP "require()" Or TCL "source" In Bash Script
I can't figure out how to explain this any better than this. In PHP we have a command "require()" that obtains a file and logically places it into another file. I cannot figure out how to do this in bash script as the requirement is necessary for a migration script to obtain the code from a .cfg file and then be able for the "parent" script to run the code it "imported" from the .cfg file, much like PHP's require() or TCL's "source". This is what I have so far and it fails: if [ -f ivc.cfg ]; then cat ivc.cfg fi Anyone really know bash script well please help, it's a barrier for me not to be able to get just this one piece working.
Screen Scraping Help
I need to find out how i can screen scrape the top ten virus threats from a website, store them in an txt/xml pref xml and the from there i can display them where ever on my site. I know people like NAI.com offer something you can put on your site but you have to have there logo and there layout which i dont want. I know you can do it with asp but i hate asp so i would like to know how i can do it with php bearing in mind im not exactly hot on it.
Screen Scraping
I need to do some screen scraping from PHP. I'm planning on using $content = file_get_contents( "http://www.somesite.com/somepage.html" ); to get the page and store it as a variable. I'll then parse it out to get what I need. But I'd rather not reinvent the wheel if there's code out there that might give me a basis to work off of. Anyone know of any? If not, does anyone have any suggestions for parseing the html to find what I need?
Best Way To Keep User "logged In"?
I'm trying to build a web app where my users can login and view personalized data (their profile, stuff specific to them) and I'm really pulling my hair out over what would be the best way to log them in and keep them "authenticated" through out different pages. So far I have two ways in mind. 1) Keep the username and password in session variables, and check it against the database on every page access. So each time a user clicks on a page, that page will re-evaluate them against the database to see if they are allowed to be viewing what they are viewing. This also allows me to grab their User ID, which is the primary key of that table, so I can use it to search other tables where that User ID is a forgein key. 2 Session Variables: user, pass Problem: I am not sure if this is overkill on the database. I have to run that query every single page the user access a new page. Is this a bad thing? And since I have to re-authenticate on each page I have to keep the user/pass in session variables. 2) Use a boolean flag. The user logs in, and i compare the user/pass to a database. If yes, I store the their User ID in a session variable and set another session variable, such as $allow=true, and use that to "allow" the user to go from page to page. Each page checking the session variable "allow" and the session variable that contains the User ID so I can query the database to get their relevant information. 2 session varialbess: allow flag, User ID Problem: If for some reason the variable name of the flag becomes known, a simple alteration of a url could allow access to the pages.
Web Site Screen Scraping In PHP?
Are there any hot tips for stuffing a couple strings into a simple website page? I want to help people that need to look up a bunch of zip codes on this website. http://zip4.usps.com/zip4/welcome.jsp I expect to hit the URL with Address, city and state pre-loaded and invoke "submit" and let a Human Being see the response and adjust the input as necessary.
Error In Page When Checking Http_get_vars("myvar") And "myvar" Was Not Passed In URL String
PHP, coming from ASP coding if (HTTP_GET_VARS['b']==1){ print "Yes"; }else{ print "No"; } when "b" is not passed via (http://www.mydomain.com?b=1 an error occurs and the "NO" is printed but an error is printed too. How can I check for the value of "b" so that no error occurs if b is not passed?? example http=www.mydomain.com $myvar=HTTP_GET_VARS['b'] causes an error because "b" does not exist (ie. its not passed in the URL string. $myvar maybe evaluated as NULL or Empty but the error is printed in the page, anyway.
Screen Scraping Solutions - Maintain Sessions?
I have an affiliate deal with a company who have given me a number of affiliate urls to run in a frame. Their branding is included in the site, and I want to remove this. I have their permission to do this. The urls use sessions. Is there a way to include these urls in my pages (with the branding stripped out), and maintain the sessions? Im thinking i need to use CURL or sockets and run everything in an IFRAME . any tips? Its a linux server. Perhaps there is software or scripts already avaible for this?
Coding "NEXT" And "PREVOIUS" Buttons
Hi I am stuck here. I am doing a form which dispalys details of client companies. My query includes this statement: .... WHERE CompanyName LIKE %$name% .... What I need is following: if a query returns more then one result I what to "flip" through result with NEXT and PREVIOUS buttons.
Mysql/php - "LIMIT" And "ORDER"
In the following fragment of code, I am querying a database that in order to display a list of films. The list is very long and I would like to offer the possbility to restrain the range of results showned by page keeping in mind that the user should be able to browse forward or backward data if applicable. I know that I have to add the LIMIT method in the MySQL query but I have a hard time figuring how not to get lost in setting it dynamically in the PHP script. Then I have a second question. Is there a more elegant / straightforward way to deal with the display order of the data (cf. $ot, $od and $oy)? Code:
Sticking "if" Or Changes Within A MySQL "where" Array?
I'm not exactly sure how to even ask the question, and I know my terminology is not good as I'm a SQL beginner, but, here goes. I need to find a way to make an if statement within an array...or, the "while" portion of a recordset. The best way I can ask is show what I mean. http://oscarguy.mechphisto.net/awardbrowse.php If you go there and select an award (like Best Picture), leave the year field alone, and select YES and submit, you'll see a lot of results. And I have HR's dividing each entry. What I need to do is group all the entries of the same year, or at least only put HR's between the blocks of years. So, I'd have all the entries for 2002, an HR, then all for 2001, an HR, etc. Here at the bottom I'll include my PHP code I'm using so far. But I have no idea where to even start looking for an answer...if it's even possible. If someone can just give me the name of the function or process or technique, then I can go Web/book searching myself...I just need a direction of where to go. PHP:------------------------------------------------------------------------ ------ $query_RS_award = "SELECT * FROM $award WHERE id != Ƈ' ORDER BY year DESC, w DESC"; //$query_RS_award = "SELECT * FROM $award WHERE year = "$year""; $RS_award = @mysql_query($query_RS_award, $connection) or die("Couldn't query: " . mysql_error()); $totalRows_RS1 = mysql_num_rows($RS_award); while ($row_RS_award = mysql_fetch_assoc($RS_award)) { $award_id = $row_RS_award['id']; $award_tblid = $row_RS_award['tblid']; $award_award = $row_RS_award['award']; $award_year = $row_RS_award['year']; $award_category = $row_RS_award['category']; $award_won = $row_RS_award['w']; $award_film = $row_RS_award['film']; $award_nominee = $row_RS_award['nominee']; $award_note = $row_RS_award['note']; if (!($award_category)) { $award_category = "n/a"; } else { $award_category = $award_category; } if ($award_won == "1") { $award_won = " - Award Winner"; } else { $award_won = ""; } $display_block_Award .= "YEAR: <span class='normalText12White'>$award_year</span><br>FILM: <span class='normalText12White'>$award_film</span><br>CATEGORY: <span class='normalText12White'>$award_category</span><br>NOMINEE: <span class='normalText12White'>$award_nominee</span><span class='normalText12BoldWhite'>$award_won</span><br><blockquote><span class='normalText12White'>$award_note</span></blockquote><hr>"; }
Cookies And "include" Or "require"
i'm having a problem with my setcookie() instructions. Although there are absolutely no html tags in front of them,for some reason PHP3 sends me back the message "oops...php3_SetCookie called after header has been sent"... I do however use two "require" calls, one to a file with a .php extension, and one to a file with an .inc extension. At the beginning of each files there's a Could the cookie failure and the include functions be related ? In my text, I have even wiped out any HTML tags such as "bold, paragraph, line break" tags, even in "echo" commands, but to no avail what am i doing wrong ?
Why Use "Stripslashes" And "Addslashes" Function
Can someone let me know why one would use strip slashes or add slashes functions. I know what they do but cannot see the exact use for them e.g. if i have a textfield and encase it in an stripslashes function, it will escape any back slashes etc...
Parse Errors With "case "action":
I want to call an action that is in a file called "configm.php" but I get a parse error on line 8. (the bit that goes 'case "add":') Code:
How To Count "Hit" And "View"
I would like to know when a user logs in, he clicks Mike's E-mail. It will count one which mean "Hit". So that I will be able to know how many times users click Mike's E-mail. Like Sitepoint. When other users read other problems. Views will count 1. I would like to know how to do it. Could you explain with script?
"echo" Or "print" ?
Can tell me someone is some big different between "echo" and "print". What You prefer to use?
Make "blabla" Into Keywords As In "php.net/blabla"
I know this must be documented, but I cant seem to find it. How would you transform the URL into values? Like how php.net does; www.php.net/date does a search for "date" in their database... I have built a simple search function on my site, and I would like to do the same.. anyone who knows how? Or who has seen an article in the subject? Or has any other tips?
Custom User Agent
I have a simple crawler as part of a website. It uses file() to copy HTML content from a web page on another website and reformat it into a new HTML page using a template. Is there a simple way of having the crawler supply a custom HTTP user agent header to identify itself to the website being crawled? I don't want to alter the php.ini file, I need to set the user agent value from PHP if possible.
Writing The User-Agent
if it's possible to masquerade PHP as a web browser when using fopen and/or fsockopen? i know in PERL you can just set the User-Agent equal to mozilla or something, but unfortunately the project is already complete in PHP.
Detecting User-Agent
I have a site that uses access-restriction to only allow registered users to view certain pages. However, I would really like Google to be able to index *every* page regardless of whether it's password protected or not. Is there a way to identify the user-agent header in PHP? That way I could identify the Googlebot, and then set a session variable that would allow the Googlebot to spider my entire site.
Is User Agent Change Possible?
I heard PHP hard coded useragent in several places. It sucks, unlike Perl which you can enter anything. It is even worse than Wget. I wonder if PHP5 allows User Agent changes now, because it seems to add many header related functions. But I still did not see one yet.
Spoof The User-agent When Submitting Form?
I have a form with some variables, which is submitted to a php file (action="script.php"). But the problem is, that php file only accepts data from a special user agent. Is there any way i can make the data submitted, pretend to be sent from at different user-agent? The form is just a html file, but maybe it would be possible to fix this using a php script? Or maybe a perl script?
[Newbie] (User Agent) Language Detection - Can You Please Take A Look?
I am not a PHP developer, so I grabbed the following code from ---------------------------------------------------- <?php if ($userlang=="") { $userlang=$HTTP_ACCEPT_LANGUAGE; $userlang=substr($userlang,0,2);} if ($userlang=="EN") { header ("location: /TEST_index_EN.htm");} elseif ($userlang=="DE") { header ("location: /TEST_index_DE.htm");} else { header ("location: /TEST_detectionfailed.htm");} ?> <html> <head></head> <body></body> </html> ---------------------------------------------------- This doesn't work, though - the browser always goes straight to "TEST_detectionfailed.htm". However, a simple <?php echo "$HTTP_ACCEPT_LANGUAGE"; ?> does work, i.e. it shows the correct Language Code. Probably an ultra-dumb newbie error ... :/
Error Using Header("", FALSE) To Show Multiple Images From MySQL Db
Many people seems to have problems when displaying multiple images from their database (mySQL db). Often you can read error statements like "Header already sent out ...". To solve this, people often propose to use include files. To keep the scipt it would be nice not having this problem, being able to sent out multiple headers to a browser without getting these error messages. In the PHP4 manual, we can read an option that can be added to the header statement, Header("xxx",FALSE). It would enable multiple headers to be sent out. For instance the following statement: while ($product = db_fetch_object($result)) { header("Content-type: pv($product->filetype)",false); echo pv($product->image)" } However, it doesn't work. Anybody has any idea why? Does anybody use this option FALSE with success?
Reading An Array Of Records From MYSQL Then "Save As" Them Back As Copy
Hi all, Ok here is what I want to do then I will show you what I have. I have some records in a MySQL DB that I want to store in an array and then immediately copy them back where the only thing that would change would be one field - the date. So here is my table: ------------------------------------- |userName | date | value | -------------------------------------- | theDano | 200005 | 123 | -------------------------------------- | theDano | 200005 | 345 | -------------------------------------- | theDano | 200004 | 678 | -------------------------------------- | joeUser | 199908 | 897 | --------------------------------------- So if I am querying for the user "theDano" on a specific date (format yyyymm) in this case 200005 (year 2000 month of May) Here is my array to query all records to fullfill this request: if ($update=="future") { if ($month!="01") { $db = mysql_connect("localhost", "dbuser"); if($db) { mysql_select_db("dbName",$db); $result = mysql_query("SELECT * FROM tableName WHERE userName = '$userName' AND date = '$date' ORDER BY value DESC ",$db); if ($result) { $num=mysql_numrows($result); $i=0; while($i < $num) { $userName = mysql_result($result,$i,"userName"); $value = mysql_result($result,$i,"value"); echo "<tr>"; echo "<td width=Ê×'>" . $userName . "</td>"; echo "<td width=ï`'><div align='right'>" . $value . "</div></td>"; echo "</tr>"; ++$i; } } } } } Now I want to take the array of records queried which was: | theDano | 200005 | 123 | -------------------------------------- | theDano | 200005 | 345 | and copy them to look like this back into the database: | theDano | 200006 | 123 | -------------------------------------- | theDano | 200006 | 345 | so you can see that the only thing changed was increasing the month by one. Would seem simple to most but I am having a tough time wirting the code to read the array and increase each date value by "1". Thank you in advance for anyone that replies. ) Dano
Bizarre Phantom Bug (Calling A PHP Script From An Apache "Action" Directive)
I am using an "Action" directive in my Apache configuration to map a certain file type (by means of AddType and AddHandler) to a PHP script. I wasn't sure if this would work, since the Apache docs say the Action directive is for mapping a handler name to a CGI script in particular. But it seemed to work at first. Then a mysterious bug arose: Sometimes it works perfectly, and other times the output is a totally blank page. It seems that if I haven't viewed a page for a while, it chooses to do one of these at random (though it is blank more often than not), and then it persists in the same behaviour until I reload it many times. (That is, usually a page appearing blank will eventually display if I reload it many times; however, only rarely will a correctly-displayed page regress to appearing blank after reloads.) Checking with `curl -i', it would appear to a client program that the request was perfectly successful: all the headers are there (with the correct content-type), it has an HTTP/1.1 200 OK line, and so on (and there's nothing unusual in the server's access or error logs), except Content-length is 0, and there is indeed no content at all after the two newlines. WTF?? Since when do computers behave non-deterministically? I should hope, if a program is not going to behave correctly, that it at least do so predictably and consistently. That would make debugging it much easier. But what am I to make of this Schrodinger's Bug? How do I even approach the debugging process? (Also, if anyone could suggest a workaround, an alternate method of having a PHP script handle files with a certain extension, then that would be just as welcome. Though I'd like to stay within mod_php, avoiding a CGI-based technique, because I'd prefer to not start a process every time these pages are requested.)
Another "Fatal Error: Call To Undefined Function: Mysql_connect()"Question
I am getting the following error when I try to view a test page in my browser. This test page just reads a table in mysql and outputs the results. Fatal error: Call to undefined function: mysql_connect() in /var/web/public/test.php on line 2 I'm using: Apache 2 MySQL 4.1.12 PHP 4.4.0 I have tried upgrading PHP to PHP5. I went back down to PHP4. I verified that mysql.so was being loaded. I tried using the OLD_PASSWORD feature in mysql. Verified that the PHP module is loaded in apache. What is weird however (at least I think) is when I run the following via telnet: php ./test.php It pulls up the page, connects successfully to the database and displays the records. No error message.
|
TRACKER
