Large Table, Slow Query Question
I have a table with ~800,000 records. I need to grab random rows from the table based on certain criteria. The problem is that average lowest subset to grab the random row is around 200k. Here is what I'm trying to do:
There are 4 columns: data,n1,n2, and n3. I need to get the value of the data column based on criteria using the n1-n3 columns.
The most common query is SELECT data FROM table WHERE n1 = ?
The problem is that n1 can be only 1 of 5 possiblities. When the table is finished being populated there will be roughly 1.5 million records and 250k for each value of n1. Of course, I have an index on each n column.
Right now with just the 800k records it can take over a second, sometimes multiple seconds to run the following in order to get a random row from that subset:
SELECT COUNT(1) AS total FROM table WHERE n1 = 3;
index = random number from 1 to total
SELECT data FROM table WHERE n1 = 3 LIMIT index,1;
How can I speed this up? I need it to take less than half a second if possible. Thank you.
View Complete Forum Thread with Replies
Related Forum Messages:
815,000+ Records, DB Table Query Too Slow
I have a database table with about 815,000 records in it, each record containing the source of individual webpages (with all the tags and multiple spacing stripped out leaving just the words). At first my match boolean queries were fine but once I got to around 100,000+ records it became too slow to use for a web-based search tool. I was able to separate the data into 10 tables and then use AJAX to run the match query on all 10 tables at the same time and that has increased performance. My question is, is it possible for me to increase the performance for the 1 table some how instead of resorting to separating the data into multiple tables? Is MySQL the right DB for this kind of setup, would MSSQL or PostgreSQL perform better for larger sets of data? I'm not sure how MySQL 5 clustering works,
View Replies !
Slow SELECT Query INNODB Table
I have a couple hundred connections doing "SELECT [Char36 Field], [LongLong Field], [Long Field] FROM [Connection Specific Table] WHERE [NonIndexed VarChar36 Field]=[Value]". Notice that the table the select statement is being called on is unique to each connection. These tables have less than 10000 records, but this statement can sometimes take over an hour to execute. It is an INNODB table.
View Replies !
Slow Join On Large Tables
I have two tables: D (500,000 recs), and DL (2,500,000 recs) D has a PK and an index on HLQ. DL has a PK and an index on ID. The following SQL: SELECT HLQ as "HLQ", count(*) FROM D, DL WHERE D.DLID=DL.ID GROUP BY HLQ produces the following explain: tabletypepossible_keyskeykey_lenrefrowsExtra DALL500000Using where; Using temporary; Using filesort DLeq_refIDID4D.DLID1Using index The query takes ~ 3:30 on a Athlon xp2200; 1GB RAM; default bufer settings. Adding the following buffer settings only slightly decrerased the time (~3:00). key_buffer=512M table_cache=256 sort_buffer=16M read_buffer_size=16M It appeasrs that the 'Using filesort' on table D is due to the Group By clause and is the problem. I have an index on HLQ. Is there any way to get MySQL to use it?
View Replies !
Large Tables, Very Slow Deletes.
I've been using MySQL for a while now and are starting to run into limitations. Either my own, or something else. I have this really large table, it stores images, it has a mediumblob field, an unsigned id integer field and a timestamp field. This table contains up to a few million rows and is constantly filled real time. It grows up to sizes between 50 and 100 gb and in the future probably even larger. This data is not supposed to be stored for all time, therefor when a certain treshhold is reached, the oldest 5000 rows are deleted to make room for newer rows. This is where my problem kicks in. Whenever I try to delete those rows, it can take for ever to complete. 500 rows I can delete without problem, 2500 becomes slow, 5000 rows takes several seconds and 10.000 rows or more makes it looks like things are frozen. Now the temporary solution I have is to delete 500 rows at time, but this means I must do that every 10 seconds just to maintain status quo. I would much rather check every 5 minutes or so, and if needed delete 15.000 rows in one go. I use MyISAM tables and tried setting the key size to 128M, no luck. I run this on a dual Opteron system with a GB of memory and WinXP-SP2 Proffesional.
View Replies !
Query On Large Table When Implementing Paging With LIMIT?
I'm working on a paging implementation but I am stuck on a couple of things: 1) I tested with a simple query which yielded this: EXPLAIN SELECT create_date FROM threads LIMIT 0,25 id=1 select_type= SIMPLE table=threads type=index possible_keys=NULL key=create_date key_len=4 ref=NULL rows=57852 extra=Using index I thought that the LIMIT clause would limit the number of rows scanned, why is it still showing 57k rows? 2) Assuming there is a simple fix to the above, now if I want to order by create_date (SELECT create_date FROM threads ORDER BY create_date LIMIT 0,25) I understand that mysql must pull the entire table to sort the data set before applying the limit. So the explain would probably look a lot like the above. In this instance, is there a more scalable solution to implement a sorted paged result set?
View Replies !
UPDATE Large Number Of Rows With CASE Is Very Slow
UPDATE table SET rating = (CASE uniq WHEN 'ACD' THEN 0.6857 WHEN 'GGYB' THEN 0.5831 WHEN 'QUCG' THEN 0.5831 WHEN 'CYP' THEN 0.5831 ... ELSE 0 END), score = (SELECT @t := 1/(1+rating))*votes_up - (1-@t)*votes_down; The table has the column `uniq` as PRIMARY KEY of type VARCHAR(10), and about 500 rows. I need to update the entire table with this query. The problem is the query runs extremely slowly, taking 4.6529 seconds to run. I realize the index is not the most efficient, but is it the main culprit?
View Replies !
Slow Query Log
my slow log is catching a slow query, however the timestamp for the query is "0". I also placed a timestamp on the query to echo out to the results page, and it is about 4 thousands of a second. Why is it showing in the slow log?
View Replies !
Slow Query Using NOT IN
I am migrating a MSSQL server to MySQL. I know the following SQL is valid for both servers, but MSSQL finishes execution of the query almost instantly, and MySQL has been running the query for the past ten minutes and still is not finished. There is basically the same amount of data in each database. Does anyone know ....
View Replies !
Slow Query
i have this query on a website/webapp that has expanded beyond all expectation. It now takes nearly 30secs to return results from the database SELECT cl_t.Client_ID, Buyer_1_Title, Buyer_1_Prename, Buyer_1_Surname, Tel_No, Mob_No, Buyer_2_Title, Buyer_2_Prename, Buyer_2_Surname, Email_Add, Price_Max, MAX(activity_t.Date) AS lastcomm FROM cl_t INNER JOIN cl_want_t ON cl_t.Client_ID = cl_want_t.Client_ID AND Agency_Code ='$agencyloggedincode' AND Deleted = 'N' LEFT JOIN activity_t ON Buy_Sell = 'B' AND Ref_No = cl_t.Client_ID WHERE cl_t.Sales_Agent_ID = $agentid GROUP BY cl_t.Client_ID ORDER BY $order The problem is the call to MAX(activity_t.Date) AS lastcomm activity_t holds all known contact with all known clients and as such is a very large table, the call to search through all of these records and return only the date of the last entry for this client is taking the time. If I remove this from the query I get results in 3 seconds. I have indexing on activity_t.Date & activity_t.Ref_No Question, is there a way of doing this quicker within the table I already have, or should I create another table that just holds the last update date for each client, and get the date from this much smaller table.
View Replies !
Why Is This Query Too Slow?
I find this query to be exceptionally slow(around 2.5 seconds), could some tell me why this is so? MySQL SELECT st.profile_views,count( DISTINCT p.ID ) news_submitted, count( DISTINCT pv.ID ) news_voted, count( DISTINCT pcom.ID ) news_commented, u.joined, u.weight FROM users u LEFT JOIN posts p ON p.submitted_user_id = u.user_id LEFT JOIN post_votes pv ON pv.user_id = u.user_id LEFT JOIN post_comments pcom ON pcom.user_id = u.user_id LEFT JOIN stats st ON st.user_id=u.user_id WHERE u.user_id='john' GROUP BY u.user_id I traced the cause to this line count( DISTINCT p.ID ) news_submitted (from LEFT JOIN posts p ON p.submitted_user_id=u.user_id) But when i execute something like this MySQL SELECT count( DISTINCT p.ID ) news_submitted FROM posts WHERE submitted_user_id='john' it is quite fast (around 0.03 seconds) So why does it slow down when i'm joining the above query with 3 other tables ? Should i use INTEGER for user_id instead of string like 'john'?
View Replies !
Slow Query->Efficient Query
I have a query that is running really slow !!!! I have joined on Key fields and indexed the tables fully but it is still solw. -------------------------------------------- select d.id, a.signed, u.Forename, u.Surname, d.paid, p.date, d.payment, p.amount, d.acctual from details d join poten a on a.id = d.id left join recieved p on d.id = p.id left join users u on a.signed = u.userid where d.paid > '01-Dec-2005' and d.authorrceived is not null and d.authorrefused is null and ((d.payment starting 'E' or d.payment starting 'e') or (d.payment starting 'Q' or d.payment starting 'q' and p.target = '500')) order by d.paid, a.signed, d.id
View Replies !
Slow Query Log Not Staying On
I'm running MySQL 4.0.16 on Windows 2003. I just added the mysqld-nt command line option to enable the slow query log, started MySQL, and the option showed up as turned on. Then later I restarted the server, and the slow query log option went back to being turned off. Is this a Windows problem in not remembering the service parameter? Has anyone else seen this?
View Replies !
View Slow Query Log
I searched here, google and MySQL docs but did not find an answer. I'm using MySQL-Front from a Windows platform to administer a remote database. The remote server doesn't have phpMyAdmin or anything like that. MySQL-Front reports 133 Slow Queries and an average of 15 queries per second. But I dont' know how get more information than that.
View Replies !
Slow Query Log Files
I have edited my.ini file to create a file called slow queries. My problem is that when the server starts up i do not get a full list of all slow queries. Is there anything else i need to change? How else can i come about in getting all queries that took a long time to execute? im using mySQL 5.0.9.
View Replies !
Query Execution Too Slow!
I have migrated from MySQL 4.1.22 to MySQL 5.0.45. Many of the queries that were working on my old server is working too slow in my new server(MySQL version 5.0.45). I am not aware why is this happening. Actually there is no error message or query failing. The query is executing, but the time taking to execute the query is too long. This problem is not occurring for all queirs, but only for some complex queires, that is queries with Left Join or Right Join, Group By etc. I have also checked the MySQL details through SSH(that is by mysql> SHOW VARIABLES;) Now one thing I found is there is lot of difference in key_buffer_size, table_cache, join_buffer_size, read_buffer_size, read_rnd_buffer_size , table_cache etc between my new server MySQL and old server MySQL. Will this be the reason for query execution taking too much time. Can any one guide me why this is happening. The striking thing is that these quires was working very much fine and faster in my old server.
View Replies !
Recycling The Slow-query.log
Am using v5.0.22. log-slow-queries=/var/lib/mysql/log/slow-query.log, but this has grown to over 4GB. Can this slow-query.log file by recycled without starting the server? Meaning can I do the following: mv slow-query.log slow-query.OLD.log, followed by a touch slow-query.log WHILE the server is running?
View Replies !
Slow Update Query
I have about 2000 update queries to do, which takes about 1 hr on 250,000 rows. My table is getting kinda slow here is the query i am using UPDATE nametable SET sectionname = replace(sectionname,'".$oldsec."','".$sec."'), categoryname = replace(categoryname,'".$oldcat."','".$cat."'), published=Ƈ' where sectionname='".$oldsec."' and categoryname='".$oldcat."' ; I am wondering if the same thing is possible with an insert... on duplicate key statement? I cant seem to get the insert statement to work, but not even sure whether it is appropriate. With this query I am basically finding and replacing some columns based upon another table (within the php script I am using)
View Replies !
Slow Query Execution
Right now I am fazing one issue which is getting hell out of me. I am explaining the issues step by step. 1.I have one site running from last 3 years with large database and there is one main table which has maximum load. 2.Now, I have redesigned the site with lots of changes within this table too. 3.The problem is that the server is same, queries are same but output in the new database is taking 50 times more execution time. 4.I also just copy paste the table, and fire the same query... Strange even that is not working... Is it something where newly added table or database is having issues in the server? I have tried all, compared the structures of the data and table but still the query result is too slow on live testing sever. While same query in main site of same server is running perfectly.
View Replies !
Slow Query Issue
I'm getting issues with slow queries being generated by a table in my database for my app, this query is consistently generating an entry in the slow query log: SQL Code: # Query_time: 2 Lock_time: 0 Rows_sent: 27 Rows_examined: 958SELECT `p_id` FROM `products` LEFT JOIN `products_images` ON (`p_id` = `pi_pid`) WHERE `pi_image_mime` IS NULL AND `p_archived` = '0' GROUP BY `p_id`; an explain query generates this: 1 SIMPLE ls_products ref p_archived p_archived 1 const 866 Using where; Using mysqlorary; Using filesort 1 SIMPLE ls_products_images ref pi_pid pi_pid 4 shop.ls_products.p_id 12 Using where; Not exists and a desc generates this: SQL Code: pi_id  int(11)   NO   PRI   NULL  AUTO_INCREMENT     pi_pid   int(11)   NO  MUL  0        pi_label  varchar(255)  NO               pi_primary_image  tinyint(1)   NO      0         pi_sort_order  int(11)  NO      0       pi_micro_blob  mediumblob   NO                pi_thumb_blob  mediumblob  NO             pi_large_blob  mediumblob  NO             pi_zoom_blob  mediumblob   NO                pi_image_mime  varchar(25)  NO               pi_date_added  datetime   NO     0000-00-00 00:00:00        pi_filename   varchar(50)   NO                pi_image_label   varchar(255)   NO
View Replies !
Query With 3.3million Rows Is Slow?
I'm not that great with MySQL...so I was hoping someone could help me out. The query I'm running is too slow...can anyone tell me what I can do to speed it up..if I can at all? I was wondering if because ZipListMatrix has 3.3 million rows that 8 seconds is all the faster it's going to be. Any help is greatly appreciated! I have already "optimized" the tables.
View Replies !
Server Slow When Exec Following Query
i try to tweak mysql server but the process still very slow to exec o following query exec time 2min for 11k row in two table update uploaded,uploaded_job set uploaded.export = 'yes' WHERE uploaded.work_order =uploaded_job.work_order and uploaded_job.claim_status IS NULL i had try indexing and optimise my database result still the same.
View Replies !
Slow Query Wile Using Indexes
i am having a problem with a aparently normal query, i am using indexes and it is still very slow, as the table grows biger and biger i am afraid the system cant hadle it... Here is the table data: ....
View Replies !
Federated Tables Slow? (like 4.5 Hrs For A Query)
I've got a problem with federated tables. I'm using MySQL 5.1 (with InnoDB as the default table type) on a Win2K server, on which I've got four federated tables pointing at four MyISAM tables on a MySQL 4.1.11 server. Of the four tables, three of them run just fine, and I can retrieve data quickly with no problems. The fourth is a sheer pig. While they have different columns, all four tables are roughly as complex as each other, all having the same features and developed by the same team. The most obvious difference, and what I suspect might be the problem, is that the first three tables have between 150 and 1,000 records, the fourth table has closer to 15,000. Still, there isn't that much lag when I'm pulling from the smaller tables, and the lag is really serious when I'm pulling from the larger one; I ran three queries last night to test, and I could pull data from the smaller tables in about 5 minutes, but the larger table took 4.5 hours- possibly because it was joined with two other tables, but the joins on the smaller tables didn't cause this kind of problem. The second obvious difference is the fact that I'm pulling from a MyISAM table into a federated table... from which I would like to store into an InnoDB table, but it ends up timing out quite a bit. Connecting to the database I've federated to isn't a problem. It responds to a PHP frontend lightning-fast. It's just my federated tables that suck so bad. I wouldn't even use federated tables, but I need to pull from the MyISAM database for storing historical records of inventory. What am I doing wrong, and what can I do to speed things up?
View Replies !
Slow Inefficent Query (FIXED)
I have this query which pulls members from a table of 1200 members details, filtering out those without and email address and those that don't want Newsletters, and also any whose email address exists in a second table (emails that have hard bounced previously) My first attempt took an age and returned about 10,000 members, clearly loads of repeated rows My 'final' attempt with a DISTINCT shoved in works perfectly but is really slow, PHPAdmin says it takes 1.1s but on the website the data doesn't appear for over 30s, all the other sundry queries on the same table work nice and quick on the same website using the same php to generate the table of results The googling I've done suggests (to me) that there's nothing wrong with the query but I suspect that there's far too much checking of fields going on from the pre DISTINCT query's results ....
View Replies !
Mysql 5.0 - Using My.cnf - Unix / Slow-query-log
I installed mysql 5.0 and need to set up slow-query-log and other logging options. Here is what I did. But I dont see it working yet. 1. cd /var/db/mysql 2. chown mysql slowquery.log 3. touch /usr/local/etc/my.cnf 4. chown mysql /usr/local/etc/my.cnf vi my.cnf [mysqld_safe] -u mysql --log-slow-queries=/var/db/mysql/slowquery.log so now when I type: mysqladmin shutdown and than mysqld_safe & my sql restarts but the log files are not being used. also - how do I know if my my.cnf is being used at all?
View Replies !
Turning On Slow Query Logging?
Background: I paid a young admin set me up on a database server. He installed the basic I needed for the server...at my request...No Cpanel...mysql and apache and some tight security w/o even a domain name to SSH into. Unfortunately, he's a busy kid, and teens sometimes don't realize that people depend on them...and well, I can't really seem to get him to do much so I gave up and figured it's a good way to force me to learn all this myself... Well anyway, now I want to turn on Slow Query logging. But before I do that, I need to know how MySQL is running. Is SQLogging turned on already? Where is it logging to? So first thing I want to look up is, when the server is rebooted, what's the command to restart mysql? No clue. How do I change the setting? And of course, the server is production, so when I make the change, it needs to be quick, it needs to be smooth, and I need to be able to roll back to the previous config if necessary. I'm running Redhat Enterprise.
View Replies !
Slow Query W/ Join & Ordering
I am trying to figure out why I have a hugely slow query (~2 seconds in my testing environment). Details are below: It involves two tables, products and vendors. Products is a huge table, so I will only include the (ostensibly!) relevant fields in its description: CREATE TABLE `products` ( `id` int(11) NOT NULL auto_increment, `vendor_id` smallint(6) NOT NULL default Ɔ', `product_code` varchar(255) NOT NULL default '', `internal_name` varchar(255) NOT NULL default '', `lastmodified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, PRIMARY KEY (`id`), UNIQUE KEY `product_code` (`product_code`), KEY `vendor_id` (`vendor_id`) ) ENGINE=MyISAM; Vendors are much more straightforward: CREATE TABLE `vendors` ( `id` smallint(6) NOT NULL auto_increment, `name` varchar(255) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM; The following query executes in no MORE than 0.01 seconds: SELECT DISTINCT p.id , p.product_code , unix_timestamp(p.lastmodified) as lastmodified , p.internal_name FROM products as p ORDER BY p.product_code ASC LIMIT 0, 30; And has the following attributes: +----+-------------+-------+-------+---------------+--------------+---------+------+-------+-----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+---------------+--------------+---------+------+-------+-----------------+ | 1 | SIMPLE | p | index | NULL | product_code | 257 | NULL | 25124 | Using temporary | +----+-------------+-------+-------+---------------+--------------+---------+------+-------+-----------------+ When I join with the vendors table, so that I can fetch the vendor's name for each product, I use the following query, which takes about 1.88 seconds: SELECT DISTINCT p.id , p.product_code , unix_timestamp(p.lastmodified) as lastmodified , p.internal_name , v.name as vendor_name FROM products as p LEFT JOIN vendors as v ON v.id=p.vendor_id ORDER BY p.product_code ASC LIMIT 0, 30; It has the following characteristics: +----+-------------+-------+--------+---------------+---------+---------+--------------------------+-------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+---------------+---------+---------+--------------------------+-------+---------------------------------+ | 1 | SIMPLE | p | ALL | NULL | NULL | NULL | NULL | 25124 | Using temporary; Using filesort | | 1 | SIMPLE | v | eq_ref | PRIMARY | PRIMARY | 2 | te_inventory.p.vendor_id | 1 | | +----+-------------+-------+--------+---------------+---------+---------+--------------------------+-------+---------------------------------+ Note the addition of the filesort. I'm unhappy enough about the temporary, which I don't really understand, but the filesort is, I'm fairly sure, killing me. Closer investigation (or maybe just common sense if you aren't a MySQL newbie like me) shows that the ORDER BY clause is responsible, for when I join without the ORDER BY, my query time goes back down to 0.01 seconds or so: mysql> explain SELECT DISTINCT p.id -> , p.product_code -> , unix_timestamp(p.lastmodified) as lastmodified -> , p.internal_name -> , v.name as vendor_name -> FROM products as p -> LEFT JOIN vendors as v ON v.id=p.vendor_id -> LIMIT 0,30; +----+-------------+-------+--------+---------------+---------+---------+--------------------------+-------+-----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+---------------+---------+---------+--------------------------+-------+-----------------+ | 1 | SIMPLE | p | ALL | NULL | NULL | NULL | NULL | 25124 | Using temporary | | 1 | SIMPLE | v | eq_ref | PRIMARY | PRIMARY | 2 | te_inventory.p.vendor_id | 1 | | +----+-------------+-------+--------+---------------+---------+---------+--------------------------+-------+-----------------+ Any clues on how I can get the execution time to go down when I am sorting? I'm also curious why MySQL is using a temporary table,
View Replies !
Help Rewriting A Slow Phpbb Query
I have a "glance" or "Recent Topics" list on my forums that have become fairly complex. I modified an already feature rich glance mod to allow users to select individual forums to exclude from showing topics in the list. As well when users are members of certain forum groups, they see topics from the group forum in the list, and they are highlighted a different colour. The main SQL query to create the list often is showing up in the MySQL Slow_query log and I'm pretty sure is the main cause for the page loading slow. I am no mysql Guru, so I thought I would seek the advice of some to improve or totaly rewrite this slow query. PHP $sql = "SELECT f.forum_id, f.forum_name, t.topic_title, t.topic_id, t.topic_last_post_id, t.topic_poster, t.topic_views, t.topic_replies, t.topic_type, p2.post_time, p2.poster_id, u.username as last_username, u2.username as author_username FROM " . FORUMS_TABLE . " f, " . POSTS_TABLE . " p, " . TOPICS_TABLE . " t, " . POSTS_TABLE . " p2, " . USERS_TABLE . " u, " . USERS_TABLE . " u2 WHERE f.forum_id NOT IN (" . $forumsignore . $glance_recent_ignore . ") AND t.forum_id = f.forum_id AND p.post_id = t.topic_first_post_id AND p2.post_id = t.topic_last_post_id AND t.topic_moved_id = 0 AND p2.poster_id = u.user_id AND t.topic_poster = u2.user_id ORDER BY t.topic_last_post_id DESC"; $sql .= ($glance_recent_offset) ? " LIMIT " . $glance_recent_offset . ", " . $glance_num_recent : " LIMIT " . $glance_num_recent; The "NOT IN" list varies per user, but here is an example: NOT IN (77,75,76,25,26,37,63,64,66,67,67,1,25,26,37,70,28,75,76,78)
View Replies !
Can't Turn On Slow Query Logging
long_query_time = 1 log-slow-queries = /var/lib/mysql/slow_queries.log Is the above syntax not correct for enabling slow query logging? All examples I've seen have the dashes in the second variable and underscores in the first. When I restart MySQL with those lines in my.cnf, it fails to start, but writes nothing to its error log. /var/lib/mysql/slow_queries.log exists, is owned by mysql, and has read/write permission.
View Replies !
Do Parallel Inserts And Selects Slow Down A Query?
I've been facing problems with a query which I think is relatively well optimized but has been performing really slow as of late as the size of the table has increased to more than 1GB. The MyISAM table is updated every few seconds, only a few rows, and a different set of rows are selected every few seconds.Wwould that cause a slowdown and if so, what is the solution?
View Replies !
Speed Up Query, Order By Column Is Too Slow!
I have a query that takes ages: SELECT tbl1.a, tbl2.b MATCH (tbl1.a) AGAINST ( 'someValue'IN BOOLEAN MODE ) AS score FROM `a` , `b` WHERE tbl2.b = 'someRestriction' ORDER BY score DESC, b.tbl1 DESC LIMIT 20 but because there are thousands of rows it takes ages to do both of the orders, however just : ORDER BY score LIMIT 20 -- is really quick and : ORDER BY b.tbl1 LIMIT 20 -- is really slow is it possible rearrange a constantly updated database so that its naturally ordered by b.tbl1 (so there will be no need to do the ORDER BY b.tbl1 query each time), or are there any other methods to speed this up (putting it all in one table instead of 2, is that significant?)
View Replies !
Slow Query Execution With Strange Issue
Right now I am fazing one issue which is getting hell out of me. I am explaining the issues step by step. 1.I have one site running from last 3 years with large database and there is one main table which has maximum load. 2.Now, I have redesigned the site with lots of changes within this table too. 3.the problem is that the server is same, queries are same but output in the new database is taking 50 times more time. I have tried all, compared the structures of the data and table but still the query result is too slow on live testing sever. While same query in main site of same server is running perfectly. Have any one fazed such and issue? Please help me out of the situation as site is ready but I m not able to put new version live.
View Replies !
VER VERY VERY Slow MySQL Query HELP URGENTLY NEEDED
I have the following MySQL query, but it is VERY VERY slow and seems to be crashing the server. There are 300,000+ records in the 'tracker' table. SELECT sites.*, SUM(if(tracker.type='view',1,0)) AS numberOfViews, SUM(if(tracker.type='click',1,0)) AS numberOfClicks, SUM(tracker.revenue) AS totalRevenue FROM sites LEFT JOIN tracker ON tracker.site_id = sites.id GROUP BY sites.id ORDER BY sites.domain_name
View Replies !
Why Does The Slow Query Log Show More Rows Than Exist?
# Time: 070528 17:14:57 # User@Host: counter[counter] @ localhost [] # Query_time: 3 Lock_time: 0 Rows_sent: 7 Rows_examined: 120647 SELECT SQL_CACHE `webpageUrl`, `webpageName`, COUNT(*) AS `count`, (COUNT(*) / (SELECT COUNT(*) FROM _1_log)) AS `pct` FROM _1_log GROUP BY `webpageUrl` ORDER BY `count` DESC LIMIT 7; mysql> select count(*) from _1_log; +----------+ | count(*) | +----------+ | 111824 | +----------+ 1 row in set (0.00 sec)
View Replies !
Large Join Query
I am at a loss. Right now I have a PHP script that needs to run a huge join query. They query joins two tables of approximately 200,000 records. I'm running mysql 3.x so I can't use a view... at this point indexes don't seem to help. I am thinking the only way to do this is with script processing
View Replies !
Mysql_fetch_array() And Large Query
I am running a SELECT query on a table of about 1.6 million records. The number of rows I get back is about 1.5million. SQL runs fine, in command line and php, but when I try to move data from 'resource' to an array with mysql_fetch_array() my browser stalls. So my question is, am I overflowing the buffer somewhere, and if anybody has a clue how to fix this.
View Replies !
Slow Execution For A Left Outer Join Query
Whats likely to be the cause of slow execution for a left outer join query? The original query joins three tables but even if I narrow it down to one it still takes a long time to execute. $query = "select distinct materials.* from materials"; $query .= " left outer join materials_products on materials.material_id = materials_products.material_id"; There's 914 rows in the materials table and 1348 row in the materials_products table Is it likely to take a long time for this amount of data or is there likely to be a problem in the table(s) set up or query?
View Replies !
Help Me Refine This Large Join Query
$sql = "SELECT f.*, p.post_time, p.post_username, u.post_subject,u.post_text,v.username FROM (( " . phpbb_forums . " f LEFT JOIN " . phpbb_posts . " p ON p.post_id = f.forum_last_post_id ) LEFT JOIN " . phpbb_posts_text . " u, phpbb_users v ON u.post_id = p.post_id and ON v.user_id=p.poster_id) ORDER BY f.cat_id, f.forum_order"; Right now its select all the posts from every forum, I just want it to select the last post from every forum and the second join on phpbb_users don't seem to be working as the user_id doesn't seem to be matching on the poster_id, any suggestions? The query also doesn't work, I want phpbb_posts to left join to two tables, posts_text and posts_topics, how to I do that?
View Replies !
Noob: Select Large Query
If I had to select large number of records, say 1 million from the mysql server. What problems would I be facing? MySQL connection timeout, network latency? Anyone has done a large simple mysql select query and what problem did you face? I am not doing any joins.
View Replies !
Retreiving Large Query Results In Chunks
I'm running queries with MySql 4.0.17 that return thousands of records. Because I need to present them in GUI, I returieve the results in chunks using LIMIT, for example - get first 100, then the range 100-2000 and so on. The problem is as follows: in the first chunk, MySQL uses one strategy to fetch the results, and in the following chunks - a different strategy. This means that records from the subsequent queries might have records that already appeared in the first query or that some records will be left out. For performance issues it is a problem to add a unique secondary sorting criteria (like id) to the query. Is there a clean way to force MySQL to relate to the first (initial) query result set?
View Replies !
Batch Query Or Large List Of IN() Arguments?
I'm receiving a potentially large list of productID numbers from an external data source. Each productID has a number of partID's associated with it and those partID's are stored in my database. I would like to create a list of products sorted by the number of parts each has. If I only have one table called "part" which stores a partID and its associated productID, which of the following methods would be more efficient assuming the application and DB are on the same server. Option A) SELECT productID, Count(*) counter FROM part WHERE productID IN(large list of productID's received, maybe 1 or >10000) GROUP BY productID ORDER BY counter DESC LIMIT ?, 10 Option B) Execute a batch query of prepared statements (SELECT COUNT(*) FROM part WHERE productID=?) for each product in the received product list then sort the returned results in the Java app that is making the query. Option C) ???
View Replies !
Time Out Message When Query Large Tables
I'm trying to get data from 6 large tables but the volume of data in each table is too large and even select * from one of them make the system stop. I have afew questions: 1-what can I do to avoid the system stop or time out message? 2- To use several tabels infomation should I use 'View' command or can I use other methods? 3-I need to create a new table and insert the result from query in it. If I use the "view" can I insert the result of the view in a table? (I use postgresql).
View Replies !
Very Slow Table Update
I have two tables. One is really a subset of the other. However, they came in different data files and I would like to pull data from one and put it into the other. However, it is VERY slow! Once the tables are setup I will only read from them and perform operations. I will never update or insert. However, I can't get things setup to that point. Code:
View Replies !
Table Gets Slow At Around 60k Records
I've a table that starts getting slow with count(val) and select ... order by when it has around 60k records. Table type is innodb Strange enough, at 50k records the response time is great, but looks like it grows exponentially after that. Its something like 50k records - 0.2 seconds for count(val) 60k records - 3 seconds 100k records - 20 seconds I realize order by and count forces mysql to go through the entire table...
View Replies !
Large Table
I have a website with over 36,000 registered users. I have a table with 5 million+ records, each of which correspond to a particular user. Obviously some users are tied to a few of these records, others are tied to thousands.I am trying to decide: -Keep everything in the one table, and allow it to grow infinitely large. -Create an individual table for each of the 36,000 users and keep only their records in their respective table.
View Replies !
Slow Table Access On Local Machine
I'm running MySQL 4.1.7-nt on my laptop, and I'm accessing it through ODBC (MyODBC-3.51.10-x86-win-32bit). I'm developing a .NET web application, but I don't think that's relevant. I already had the application running OK on my desktop, and when I created the same environment on the laptop, retrieving trivial amounts of data (5 rows) from a table is taking 6 seconds. Updates the same. IN otherwords any table access takes 5 or six seconds. (Table only has about twenty rows in it). I can connect to the database on the laptop from the desktop (i.e - application runs on desktop, retrieves data from laptop - works fine, not noticeably slower than local on desktop. If I connect the other way - application on laptop, database on desktop, it runs just as slowly as locally. The effect is the same whether I connect to 127.0.0.1 or via the machine name. Code:
View Replies !
MySQL With Large Table.
I am using MySQL for a table which will have 100M+ records, avg length of records being 130 bytes. When the number of records reach approx. 25M (and the file size close to 4GB), the rate of inserts falls drastically from 800 per second to 30-40 per second. Details: * MySQL 3.23.58 on Fedora Core 3 * Table has 4 indices. * I have got rid of 4GB file size problem with MAX_ROWS=1000000000. * File system : ext3 on single disk. ext3 could create 10G file without much trouble. So I am convinced that ext3 is not the bottleneck. * Tried using InnoDB engine but it also doesn't meet the requirements. Requirements of database: * A single table in the database with 100M+ rows, each of size 130 bytes (approx). * 500-600 inserts per second. * 200 selects and 200 updates per second. (These statements will affect only one row) * 3-5 select statements per minute which can return 10k to 500k records. * No foreign keys/ACID transaction requirements. * Fast recovery in case of crash. Questions: * Does MySQL performance drops when the table grows beyond 4G? * Which are most important server variables which need to fine tuned? Currently I am using only key_buffer = 512M.
View Replies !
Large Test Table
How do I create a very large table, for testing? Can I download one somewhere? I've tried to search google, but couldn't find anything. The contents dosn't need to mean anything, so a random function with a loop or so should do it, but if I could download a large test table somewhere with "meaning" that would be great.
View Replies !
|
TRACKER
