Large Data Transfer Between 2 Processes
fist the context: I have a web server which query a mySql database. but
as the number of parralel queries increase, the server slows down too
much.
I got 2 ideas, one of which is to run N deamons which effectively
execute requests. the PHP uses message queuing(1) to queue queries
request to the daemons. but the answer may be quite large, larger than
messages the message queueing service can handle.
I thought to open back a communication link between the current daemon
and the waiting PHP to tranfer/process the result, using the message
queueing service to send back info regarding the opened link.
I thought to open a pipe between the current daemon and the PHP. is
this possible? and if so, how? if not, is an IP link to localhost be as
fast as a pipe?
View Complete Forum Thread with Replies
Related Forum Messages:
Large FTP Transfer Freezing Site For Others.
I have a website which, a few times a day (8am, 4pm and 8pm) exports around 30-40mb of images to an external server via the PHP FTP extension. The transfer process normally only takes 4-5 minutes but during this time the rest of the site freezes up for anyone else trying to view it. At the moment im not sure whether the problem is php freezing up or the fact that all the servers bandwidth is being used for the FTP transfer. What i'd like to know is: - If php is using all of its resources for the FTP transfer how do i fix it? OR - If the problem is with the limited bandwidth on the server is there a way to throttle the amount of bandwidth used during the FTP PUT operation.
View Replies !
Large File Uploads - Either The Transfer Hangs Or $HTTP_POST_FILES Is Empty
I've got a client who wants to allow his clients to upload 100MB files thru HTTP. He also wants a progress meter. I'm using the megaupload progress meter, minus the 5MB filesize limit. I've also set my apache transfer timeout to a ludicrously high number, and in php.ini I've set upload_max_filesize, max_execution_time, max_execution_time, etc to values that should allow for large file uploads. I can upload files of several MB, but w/ anything larger, either the transfer hangs or $HTTP_POST_FILES is empty for the processing script. megaupload uses some perl/cgi code to handle the post data and show the progress meter, but I looked thru it carefully and nothing in there seems to be likely to cause a problem. Can anyone give me advice on what else I should look into? This is driving me NUTS...
View Replies !
Data Transfer
In mysql there is table downup ************************ site enum('site1', 'site2'), down datetime, up datetime ************************ and there is file with such statistics about site1, site2 down and up period ************************ 2003-01-28 15:42:08 (back up) 2003-01-27 14:05:45 (down) 2003-01-25 15:54:17 2003-01-25 15:11:19 2003-01-23 09:38:19 2003-01-23 09:06:55 2003-01-10 13:42:38 2003-01-10 12:45:40 2003-01-07 01:56:44 2003-01-07 01:34:47 ************************ how to transfer these datas to the mysql table. If there is not back up the data will not be transffered and always check the highest datetime / event has to be on the top/last. should i use regular expression (preg_match).
View Replies !
Data Transfer Problem
I have an interesting problem. I have a (LARGE) set of historical data that I want to keep on a central server, as several separate files. I want a client process to be able to request the data in a specific file by specifying the file name, start date/time and end date/time. The files are in binary format, to conserve space on the server (as well as to increase processing time). The data in each file can be quite large, covering several years of data. New data will be appended to these files each day, by a (PHP) script. The server machine is likely to be a Unix machine, whereas my clients will be running on windows machines. My clients program is written in C++. My two main problems/questions are as follows: 1). Transfer method issue: What is the best (i.e. most efficient and fast way) to transfer data from the server to clients ?. I think SOAP is likely to be too slow, because of the sheer size of the data 2). Cross platform issue: How can I insure that that the (binary?) data sent from the Unix server can be correctly interpreted at the client side? 2). Security issue: How can I prevent clients from directly accessing the files (to prevent malicious or accidental corruption of the data files.?
View Replies !
PHP Data Transfer In Forms
basically I am looking for a way to transfer form data into php variables. Most of them are not a problem but I can not figure out how to transfer select values or radio button values. Can anyone help? I will be posting the code for the page I am working on after I get home from work.
View Replies !
Transfer Filteref Data
I have two databases which are identical in structure. One is called stats and the other is called statistics. I have a program in php (encoded by zend hence I cannot make any changes to it) that updates the stats database tables with data. However I need to filter this data to trim it down and insert it into the statistics database tables. I want this to happen everytime a record is added in the stats table. So i guess i will be using triggers. I am using pgadmin III. Can anyone think of a solution to this?
View Replies !
Transfer Form Data Using A URL
I've tried looking this up using the forum SEARCH feature but have not been able to find anything that helped. What I am trying to do seems like it should be very simple, but I cannot get it to work for the life of me. I have a page. On the page is a list of names (generated from the DB), next to each name is an "approve" text link that when clicked changes the status of the person in the DB. This part of the process works. I am trying to add a text field (form item) at the top of this page where the individual who is approving the people on the list can enter a batch id# (ie: 7001) such that for every "Approve" clicked, it will change the status of that users account (as it is doing) AND stamp a field for that name with the indicated batch #. The problem is, I cannot get the number entered in the Batch ID text field to transfer to the following page without using a SUBMIT button (which I don't want to use). How can I transfer the contents of the text field (ie #7001) to the next page using only the URL in place of a submit button? I am currently transferring the user_id through the URL properly, so I need to know how to append the text field contents to that url as well. From what I've found searching this forum, it seems I may need to use the $_GET function rather than $_POST, but I do not know how to do that in this case.
View Replies !
Data Transfer On Php Pages
I want to be able to pass values in the controls say textbox on a form on one page to another and back. All my pages are php files. I would also like to be able to pass some of the values through session from one page to the other.
View Replies !
How To Detect Data Transfer...
Is there a way to detect if data transfer is occuring between the client and the server? Let me explain... I have a page that streams a movie. Is it possible to use php to detect if "transfer of data" to the client has stopped, or if it never started? This is an unconventional problem, and I don't want to conclude that php can/can't do it until you pros say so.
View Replies !
Data Transfer Worked In 4.4.4
I am passing data from an embedded device using Content-Type = multipart/form-data. My data passed nicely into $ variables when the server ran with PHP 4.4.4 but now that it is upgraded to 5.2.3 none of the data is passing.
View Replies !
Data Transfer From MySQL To DB2
I want to transfer from mySQL to DB2. I have already created the tables and indexes. I need to transfer the data from mySQL database to the new DB2 database with same table structure.
View Replies !
Transfer Data From An Xls Spreadsheet Into A Table
Is there any way I can transfer data from an excel spreadsheet to my table?? Example: I have an .xls spreadsheet called hello.xls and it has 5 columns with hundreds of rows of data in it. I want to copy all this data to my table instead of copying line by line which will take me eternity. I have a table called hello which has the same 5 columns.
View Replies !
Transfer Form Data In A Different Frame
i've got a problem i have 5 frames and i want to transfer the data of a form to the other frame but the source of the frame is a different url. i want to create a button where when the user clicks on it, it will transfer the data of that form to the other from in another frame.
View Replies !
Build A Page That Takes Form Data, Processes It In A Second Page
I'm trying to build a page that takes form data, processes it in a second page, then sends the user to a third,depending on data processing. On the second page, I am trying to get the page to have some sort of "Loading" message while the backend processes the data, but I can't seem to get anything to work. Does anyone know how I can get that Loading message to display?
View Replies !
Data Transfer Problem - Ideas/solutions
I have an interesting problem. I have a (LARGE) set of historical data that I want to keep on a central server, as several separate files. I want a client process to be able to request the data in a specific file by specifying the file name, start date/time and end date/time. The files are in binary format, to conserve space on the server (as well as to increase processing time). The data in each file can be quite large, covering several years of data. New data will be appended to these files each day, by a (PHP) script. The server machine is likely to be a Unix machine, whereas my clients will be running on windows machines. My clients program is written in C++. My main problems/questions are as follows: 1). Transfer method issue: What is the best (i.e. most efficient and fast way) to transfer data from the server to clients ?. I think SOAP is likely to be too slow, because of the sheer size of the data 2). Cross platform issue: How can I insure that that the (binary?) data sent from the Unix server can be correctly interpreted at the client side? 2). Security issue: How can I prevent clients from directly accessing the files (to prevent malicious or accidental corruption of the data files.?
View Replies !
Displaying Transfer-Encoding: Chunked Data As It Arrives
I'm trying to write a script that reads from a URL which, for example's sake, sends data at 5 second intervals. As the server sends each chunk of data I want to echo that data to the page. So far I've only been able to display the data after all of it has been sent, but not as it arrives in the 5 second intervals.
View Replies !
Warning: Ftp_fput(): Opening BINARY Mode Data Connection ... But Zero Bytes Transfer
I am using the following short script to try and upload a file to an ftp server (the script is the example for ftp_fput from php.net). Usually the script executes perfectly and the file is created and filled correctly. However, occasionaly the file is created on the server, but it has a size of zero bytes and no data is in it and the following warning is recieved: Warning: ftp_fput(): Opening BINARY mode data connection for somefile.txt in ... on line 19 The server, username and password are correct, verified by maually logging in with an ftp program. The php version being run is 4.4.2.I have had safe mode turned off on both the server running the php script and the server i am ftping the file too, this seems to have made it work successfully more often. // Server, username and password settings removed for security // open some file for reading $file = 'somefile.txt' $fp = fopen($file, 'r'); // set up basic connection $conn_id = ftp_connect($ftp_server); // login with username and password $login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass); $response = ftp_chdir($conn_id, "httpdocs"); // try to upload $file if (ftp_fput($conn_id, $file, $fp, FTP_ASCII)) { echo "Successfully uploaded $file"; } else { echo "There was a problem while uploading $file"; } // close the connection and the file handler ftp_close($conn_id); fclose($fp);
View Replies !
Inserting Large Data
i have this form that we will need to fill in and the info gets entered into a DB. what is the easiest way of inserting this data into a tbl? is there anyway that i can check that the same driver has not been entered twice?
View Replies !
Large Data Quantities
I'm makigna rather big database with some information. I have one table at the moment, expected to be round about 5 more in the future. I have about 11 milion rows in my single table right now, expected to be round about 100 - 500 million. My tables are in a structured way, to further increase optimation, only two columns. My tables basically stores words/letters and similar things.. My problem is that working with these large tables/databases is not the fact that it takes allot of disk space, but the actual load time. Code:
View Replies !
Get Data In Between Large Blanks
I have a variable: $var = " some writing "; And I only want to get the "some writing" part of it. I've tired many ways to do it, but I can't seem to get any of them to work. Any suggestions?
View Replies !
Displaying Large Amount Of Data
let me know a way to display large amount of data in a table column without expanding the height, where other columns containing one or two lines? This particular table has 20 columns and one column cotains several lines of text, about 50 lines, whereas others contain one or two lines. Due to this, when displaying text, this particular column gets expanded in height.
View Replies !
Import Large MySQL Data
My webhost does not give me shell access. Normally, I use this sort of command on my local machine: c:datac:mysqlinmysql -uroot -pxxxxx db < file.sql My webhost gives me PhpMyAdmin access, but I can never use that successfully with large files. The sql file I need to import is 15MB.
View Replies !
Send Data As Large As POST
You hardly ever see people use GET to send large amounts information say, an entire forum post. Is there some reason behind that? I have a situation where GET seems to be the only method I can use. Is it OK to use GET to send about 256kb (or even 512kb) of data? it didnt say anything about GETting too much information. The data I want to send is not private/cant be malicious since I am going to be the only one using it. I just want confirmation that it is OK from someone who knows more about this than myself.
View Replies !
Displaying Large Amounts Of Data
I've run into a situation where i'm hoping php could reduce the time to create this project. First of i have linux based server, mySQL server. Now what i'm trying to do is create an online catalog which has about 300 different types of clothing. Heiarchi goes Category of clothes (t- shirts), brand (hanes), then style (long sleeve). As you can imagine there could be tons of choices. each specific item has its own page that has a picture and description on the page. What would the most practical way of doing this besides making a website for each item?
View Replies !
Manipulating Large Volumes Of Data
I'm importing CSV files into a table in a mySQL database using PHP, with the number of rows being anything in the region of a seven hundred thousand. Another PHP script reads through each row in the newly created table and applys some manipulation to the data and adds it to a "master table". Now, this is all running locally on an XP installation of Apache 2 and these scripts are taking 20 plus minutes to run in some cases. I have any other alteratives to using PHP to manipulate the data on mySQL? Is there some setting I be looking at for when large volumes of data are manipulated in mySQL/PHP? I've had to repeatedly increase the timeout figure (in php.ini) to way beyond it's default amount - this would never run on a commerical host, they'd cut my fingers off for using up so much CPU time.
View Replies !
Large Amounts Of Data From POST
If you place about 6 long paragraphs of text into a textbox in my script, and you submit it, it will just say at the bottom "Waiting for WEBSITE..." we tried it on several computers with different ISPs. If you shorten the amount you post it works. Does anyone have any clue whats going on, and how to fix it?
View Replies !
I Want My Class To Reference A Large Data Object
there are many gridPartition objects in my code, each of which represents a division of a large matrix in memory. two different gridPartition objects can refer to TWO DIFFERENT MATRICES in memory so i can't use a static variable for this because that would require that a single matrix object be common to ALL gridPartition objects. On the other hand, I don't want ot make a bazillion copies of this huge data object because one is attached to each gridPartition object. Code:
View Replies !
Importing Large Amounts Of Data Via File Upload
I was wondering how anyone here deals with move large amounts of mysql data to be uploaded to the server? Large as in 75MB CSV file. I have a client who wants to admin there database with 15,000 users. They manage all data offline and want to upload to the site for display purposes only. My server has 5MB php file upload limit. I really do not want to up this limit. What I am wondering is how does cPanel or PHP MyAdmin get around this upload limit using php scripting?
View Replies !
Error In Trying To Pass Large Amount Of Data In Textarea?
I'm trying to pass HTML codes via a form through textarea. It seems that if I put in text larger than certain size, the data I receive through the POST array becomes [first part of data]var_name = [data]. Is this a common problem in php? I'm currently using PHP 4.3.9 Here's the code: <?=$test?> <!--data already distorted --> <form action="<?=$_SELF?>" method="POST"> <textarea name="test"><?=$test?></textarea> <input type="submit" name="update"> </form>
View Replies !
PHP Processes
I have a cronjob triggering a php script that takes several minutes to accomplish. I need to prevent multiple triggering by the cronjob mulfunctioning. So I would like a clean way to know how many instances of a certain php are currently working on the server. is there a server variable for that? if not, how do I know (via php) how many concurrent accesses are there to a certain php page? for example in mysql (the "SHOW PROCESSLIST" command), but I need a similar function for the php page iself.
View Replies !
Site To Site Data Transfer
When a visitor goes to siteb i want it to send information to sitea, in this case lets choose browser type. Eventually I want to have more sites sending data back to sitea. some not managed by somebody that i know i can trust. so mysql passwords etc. cant be given out. I thought about $_GET vars, but there is no way to make that secure from bogus entries, is there? Unless i used some sort of encryption and the decrypt key was at sitea.
View Replies !
Background Processes
I'd like to know how to send an external process to the background, so that I can run multiple instances of the same process at one time. Any tips, links.. etc?
View Replies !
Listing Processes In PHP
Using PHP, is it possible to return and display a list of currently running processes on the client's machine? In fact, is it possible to run a system command on a client's machine?
View Replies !
Automated Processes
Customers post 'projects' to my site. These 'projects' are closed after 10 days. I could run a script on the default.php page that detects this, but it seems a waste of resources to run it everytime someone visits the site. Is there a way around this, for example the first visitor to the site in any, say, 3 hour period, runs the script & switches the most recently outdated projects from open to closed?
View Replies !
Max Number Of Processes
how can i determine what the maximum number of processes i should open with popen (or proc_open) is? i assume it'd depend on the hardware of the computer in question, but if that were teh case, then is there a way to get info. about it, and some general method to determine a good number of processes to open from that info?
View Replies !
Stranded PHP Processes And ProcessExplorer
Does anyone know what it means if I see PHP processes in the process tree at the base level, instead of under the IIS worker process? I do nothing but web serving off of the box. These processes that are not under the IIS worker process seem to be stranded/hung. They are never cleaned up by any garbage collection and I have to kill them manually. They also always have a fairly low memory usage, potentially indicating that they fail before loading completely. Could it be that the execution of the process fails early enough that ProcessExplorer doesn't see them as being executed by IIS?
View Replies !
Handling Long Processes
I am building web app, that will run a long processs (image processing) that can porbably take 15-30 minutes to complete) in respond to a user request (the user clicks a "process" button on a php web form). The ideal situation that I see is once the user's request was recevied, a reply that says "you request is being processed. we will send you email once finished", will be sent to the user, while the other process will take place. When it's finished, a callback function will be called that sends email to the user notifying him the process outcome. What are the options to implement that? multithreading?
View Replies !
Shared Memory Between 2 Processes?
I have two running php processes (they have infinite loops) and I need them to pass some data to eachother. Writing and reading shared memory in one process works just fine but when I try to save var in process1 and read in process2 I get error: "Variable key 555 doesn't exist". What I'm doing wrong? //PROCESS 1 (saving var) $key = 'My Key' $value = 'My Value' $app = 'lovelyapp' $key = $key . 'a' $segment_key = abs(crc32($app . $key)); $segment_size = 1024; $segment_perms = 0600; $shm_id = shm_attach($segment_key, $segment_size, $segment_perms); shm_put_var($shm_id, 555, $value); shm_detach($shm_id); // PROCESS 2 (reading var) $key = 'My Key' $app = 'lovelyapp' $key = $key . 'a' $segment_key = abs(crc32($app . $key)); $segment_size = 1024; $segment_perms = 0600; $shm_id = shm_attach($segment_key, $segment_size, $segment_perms); $value = shm_get_var($shm_id, 555); shm_detach($shm_id);
View Replies !
Insert Processes Before Form Is Submitted
I am trying to stop using Macromedia as my crutch and you guys have been a big help. I need some assistance with submitting a form. I have a form on my page and need to do an insert. I have the action posting it back to the same page I am on versus sending it to an insert page. The problem I am having is it trys to do the insert as soon as I pull up the page instead of waiting till the form is submitted. I believe I need to do something with isset, but can't really find a tutorial that goes over it really well. Code:
View Replies !
Forking Multiple Processes At A Time
i have a php script that needs to execute an external php script (call this script 2) when the user clicks a link, however I don't want the user to have to wait until script 2 finishes executing before he sees the output of script one... i know I can use the exec() command to do this, and use the & to run the command in the background.... but say i want to run 100 instances of script 2 when the user clicks the link (with different arguments)... i dont want to have all 100 processes running at the same time... what I would like to do is run 10 or so at once, and once those 10 are completed, run another 10...
View Replies !
Keeping Users Informed In Lengthy PHP Processes
Often PHP processes take a long time, such as wildcard searching a PostgreSQL database, or generating thumbnails. The PHP host page does not display until the script is complete, so you can not effectively use code such as echo "Converting file ".$Infile; exec(convert $Infile $O8utfile); for realtime feedback to the user. How have others implemented a "progress meter" in PHP processes , one thought I had was to use Javascript redirection to call a PHP page with differing parameters each time .
View Replies !
Sending Signals To Processes As A Different User Than Owner Using Php
i know, it seems more to be a problem according to unix but i got it while developing software with php: i have 2 different users on a hp-ux machine that execute php scripts. they are both in the same group. these php scripts start background processes, i can communicate with using signals (sending with posix_kill, starting background processes via pcntl_fork). so i got severel background processes owned by my 2 users. the problem is i want to send signals as one of my 2 users to all my php background processes no matter who of my 2 users is the owner. is it possible other than using external tools like sudo? i'm using: PHP 4.3.4 (cgi) (built: Aug 18 2004 15:37:46) Copyright (c) 1997-2003 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2003 Zend Technologies compiled with: ../configure' '--prefix=/usr/local/php-cgi-saprfc' '--enable-calendar' '--enable-filepro' '--with-gd' '--with-jpeg-dir=/usr/local' '--with-zlib-dir=/usr/local' '--without-mysql' '--enable-sysvsem' '--enable-sysvshm' '--with-pgsql=/usr/local/pgsql' '--enable-trans-sid' '--with-cpdflib=/usr/local' '--enable-sockets' '--with-ncurses=/usr/local' '--disable-shared' '--with-tsrm-pthreads' '--with-saprfc=/usr/local/rfcsdk' '--enable-pcntl
View Replies !
Max_execution_time Ignored, Multiple Apache Processes Spawned
I have an app that is going berzerk on its own; without any code or environmental changes of any kind, for some bizarre reason it will randomly just hang, spawn multiple Apache processes and completely ignore max_execution_time, literally running infinitely. Using PHP 5.2.0, Apache 1.3 and Win XP Is this normal "bizarre" behavior that can occur within this setup; if so, what workarounds have you found to combat this, other than the obvious (try to debug and fix code)?
View Replies !
Limiting The Number Of Spawned MySQL Child Processes
Need to know if there is any way to limit the number of spawned MySQL child processes in linux ? I have done my research at http://www.mysql.com and have not been able to find my answer. Have also looked at the source code (what i can understand of it). Using MySQL v.4.0.25-max. Using an older system and the 14 childern are just bogging the system down. I am the only one with access to the server so 14 childern is overkill.
View Replies !
Checkboxes - Loop That Takes The Values And Processes And Writes The Changes To The Database.
I generate a form page from my MySQL database. let's say the table structure is: Table A > ID, title, active the number of rows generated by the query is dynamic. If a field's value is 1, then the form output appears as <input type=checkbox name=$ID checked> $title and if not <input type=checkbox name=$ID> $title At the bottom of the form there is a submit button. The visitor can check/uncheck rows and then submit the form. I need to write a loop (possibly foreach) that takes the values and processes them and writes the changes to the database.
View Replies !
|
TRACKER
