One Large Table Or Many Small Tables?
I'm trying to decide whether to use one large table or many small tables.
I need to gather information from various devices (about 500). Each device
has its own Id and some data.
Should I use only one table with an indexed column for the ID and another
column for the data, or should I use 500 tables each with only one column
for the data?
View Complete Forum Thread with Replies
Related Forum Messages:
One Large Table Vs. Many Small Tables
I'm working on a design using PHP & MySQL and I'd like to get some opinions on this. My design has several tables that will be referenced but I'm wondering if those tables should be broken down even more and referenced more dynamically. The reason that I wonder about this is for long term goals. I hope that eventually there will be two or three thousand records that will be used on a regular basis. These records will need to be separated into groups, but I'm not sure if I should use a field in the database table or create a new table for each group. If a few hundred records could be in each group, do you think it's better to use one large table with a field for the group ID, or a new table for each group?
View Replies !
Several Small Calls Vs One Large Call
I am working on a php interface to a mySQL db which will pull lots of rows from the database. Assuming I am selecting a hundred rows by their auto_incremented ID's, is it better to do this with a loop of 100 "SELECT ... WHERE id=" calls, or to do it with one single "SELECT ... WHERE id IN(...)" call, with IN() holding a list of the hundred items.
View Replies !
Large .sql File (break Down To Small Parts)
i have an oscommerce installation that generates a 200mb sql backup file. i want to try and rebuild the installation on my local pc (localhost) to emulate and play with, unfortunately the 200mb file is difficult to handle. phpymadmin seems to crash. i have console access on the host however. is there a way i can break down the sql file into smaller increments for easier uploading?
View Replies !
Large Table Or Many Tables?
I posted a similar question about 6~12 months ago with no definate answer, so I will give it another go, as the choice has popped up again in my programming life: If the number of entries is the same is it more efficient (better) to have a single table with many columns, or many tables with few columns? The obvious answer is a single table, because there is less closing and opening tables, but the reason why I ask is what about searching through a lot of columns being even more inefficient? The situation is I need to record data for every week of the year. Say ~50 columns per week --> 2600 columns. Or, do I split it so it's 52 tables with 50 columns each? In each of these options the user (or user ID) will be the primary key. The seemingly pointless other option is to insert new columns with every new user, and have the week-item as the primary key?
View Replies !
One Big Table Or Many Small Ones?
I am designing an interactive website consisting of a few pages and I want to track individual user progress. I was wondering if it would be better to create one big table for all values on all pages (Which may have a lot of columns, easily hundred or more), or created a separate table for each page (so I could have some 10 tables with 10 columns or more). Or does it not matter?
View Replies !
One Huge Table As Appose To Many Small Ones
i have a very large XML DB (60gb+ and growing on a daily basis) which holds complete life cycles of stock option , i used to hold it all in folders with each folder named like the option paperId so it seemed logical to me to hold it now in diffrent tables for each paper even tough the are all the same exact table , now i'm having some problems with hibernate (one of the biggest reasons i left XML) and so many tables , is it more effiecent to keeeping data like that ? ofcours i will have to change the PK from datetime to paperId+DateTime if i decide to change to one big table ... how can i calculate diffrence in retrieving data ?
View Replies !
Small Tables Or A Few Big Tables
My Web site is going to use a MySQL database extensively throughout the Web site. It is a content management system type Web site with lots of users and written articles and other things. With my database I'm trying to figure out if I should create big tables that have a lot of rows in them or if I should create a lot of tables that are very specific to its data and use a lot of table joins in my select statements. I'm mostly concerned about performance. I'm thinking that if I make a lot of tables, then select statements I do will be really fast because my tables won't contain very many rows. But I don't know if this is true or not or if it matters for performance reasons or not. In my Web site I have a lot of different kinds of data that could put in many different tables. But, without data redundancy and without using extra storage space, I can still make a few big tables, and make that work. I just don't know which way is best for performance -- speed, I want speed. Anybody know what I should do or know a book I should read or a Web site to go to?
View Replies !
Large Tables
So i've got a table with 15 million rows. id is the primary key SELECT COUNT(id) FROM listings WHERE id != 0; Takes about a minute to load... which is insanely slow.. anyone know what could be causing this?
View Replies !
Very Large Tables
I am looking at using mysql to store some particle physics data. So the idea is (for example) I would have a table with a list of events, then a table with a list of particle types (one event can have many particles) and so on. The problem is that quite often I want to calculate something for every single event, but this table might run into 10s of millions of events and many GB. Obviously it would be mad to load the whole result set into memory then iterate over it. My question is: is there any way to have the db push results to the users one (or a few) at a time in an effiecent way? I was thinking of something like an iterator that pulls the data from the disk one (or few) at a time. An alternative (but much less useful) method might be to hand the function I want to calculate to mysql, for example: select myfunc(x,y,z,a,b,c) from events where energy>3; But as far as I know there is no way of defining your own functions in SQL
View Replies !
Large Number Of Tables
I would like to ask whether it is feasible (or whether it makes sense) to run a PHP script that creates a number of tables for each user. Basically, this script allows users to sign up for a free service, and for each user that signs up, the script creates 9 tables in the database to contain that user's details. Is this a sensible way of containing user data, and what are the inherent problems this may cause especially when say are 1000 users signup?
View Replies !
Large DB Should Use Multiple Tables?
I have a social networking site similar to myspace, currently my table which links who is who's friend on my site is 2 weeks old and is at 300,000 rows but is only at 15mb in size, this is the most accessed table of all my tables. My question for performance, as this table grows should it eventually make a new table? If so here is my current select code select userid,friendid,status from friend_friend where userid='$temp[auto_id]' and status='1' Could someone give me an example of how o select this from multiple tables if I add more for this table in the future?
View Replies !
Managing Large Tables
I recently inherited a database (version 5.0.18) that has 1 table constantly growing out of hand between 10GB -30GB, therefore making is difficult/impossible to query. Also, the Archive Storage Engine is not installed. Currently, the table is manually renamed (i.e. tablename_daterange) and a new table created. ....
View Replies !
Join Tables On A Large Database 200 Meg
i am trying to work out the most efficient way to list say multipl= e categories of entries, the database is quite large about 200 meg.=20 I would like to know if using join tables is more efficient than storing th= e keys in a varchar field then within the second loop doing a where in (1,2= ,3,4) where the 1,2,3,4 are the stored category keys in the varchar field = rather than a where in (1), where 1 is the pirmaryID of the entry for insta= nce ?
View Replies !
Optimizing Mysql For Large Tables
I have been working with Mysql for about 5 years - mainly in LAMP shops. The tables have been between 20-100 thousand records size. Now I have a project where the tables are in the millions of records. This is very new to me and I am noticing that my queries are really slowwwwww! What are the options that I have to speed my queries on the mysql side with regards to the my.cnf file. I have a fair understanding of sql optimization and I understand explain. I just want to see if there is something that I can do with the server also.
View Replies !
Slow Join On Large Tables
I have two tables: D (500,000 recs), and DL (2,500,000 recs) D has a PK and an index on HLQ. DL has a PK and an index on ID. The following SQL: SELECT HLQ as "HLQ", count(*) FROM D, DL WHERE D.DLID=DL.ID GROUP BY HLQ produces the following explain: tabletypepossible_keyskeykey_lenrefrowsExtra DALL500000Using where; Using temporary; Using filesort DLeq_refIDID4D.DLID1Using index The query takes ~ 3:30 on a Athlon xp2200; 1GB RAM; default bufer settings. Adding the following buffer settings only slightly decrerased the time (~3:00). key_buffer=512M table_cache=256 sort_buffer=16M read_buffer_size=16M It appeasrs that the 'Using filesort' on table D is due to the Group By clause and is the problem. I have an index on HLQ. Is there any way to get MySQL to use it?
View Replies !
Large Tables, Very Slow Deletes.
I've been using MySQL for a while now and are starting to run into limitations. Either my own, or something else. I have this really large table, it stores images, it has a mediumblob field, an unsigned id integer field and a timestamp field. This table contains up to a few million rows and is constantly filled real time. It grows up to sizes between 50 and 100 gb and in the future probably even larger. This data is not supposed to be stored for all time, therefor when a certain treshhold is reached, the oldest 5000 rows are deleted to make room for newer rows. This is where my problem kicks in. Whenever I try to delete those rows, it can take for ever to complete. 500 rows I can delete without problem, 2500 becomes slow, 5000 rows takes several seconds and 10.000 rows or more makes it looks like things are frozen. Now the temporary solution I have is to delete 500 rows at time, but this means I must do that every 10 seconds just to maintain status quo. I would much rather check every 5 minutes or so, and if needed delete 15.000 rows in one go. I use MyISAM tables and tried setting the key size to 128M, no luck. I run this on a dual Opteron system with a GB of memory and WinXP-SP2 Proffesional.
View Replies !
Large Unused Tables Vs Performance
I've been working on a website with a rather large a database. One of the about 100 tables in there was about 1/4 of the total database size. The table had no use anymore at the time I worked the website, so it was just sitting there, doing absolutely nothing. Obviously to spare diskspace we deleted the table. But the question I have now, can unused large tables affect the perfomance of Mysql? I've searched a bit around, but honestly I couldn't find a clear answer about it right away (maybe I haven't searched enough either :) )
View Replies !
Large Number Of Tables And Columns
i'm developing an online portal, which my MYSQL database contains 100 tables.. my doubt is if any table contains 30-40 columns will it become a problem? like my html front end form has 40 fields for data insertion. and i'm not maintaing any duplicate data if my DB contains 100-150 tables will it create any problems? as i'm inserting data from EXCEL via PHP to MYSQL, please suggest me whether a healthy MYSQL table is dependent or independent on NO:OF COLUMNS(FIELDS) or not?
View Replies !
Speeding Up Large MySQL MyISAM Tables
I'm somewhat new to MySql. I've been using it for a while, but pretty much out of the box setup, and am starting to suffer heavily with my larger tables. I have a table with 5,000,000+ rows that I have to search and do joins on. Although I have an index set up for it, the joined select will still take some 400+ seconds to return, which is obviously unacceptable. This is due to enormous HD access. Perhaps someone can help me with indexes here; I was under the impression that the index for the tables are cached in memory, and therefore permitted "instant" searchability, without having to retrieve data from each of the rows of the DB. Is there a startup parameter, or something in the mysql.ini file, that must be set to allow for this? I have the index configured properly, and have made sure that the query uses there parameters in the where clause in the same order that they appear in the index. Code:
View Replies !
Speed Of InnoDB DELETEs On Large Tables
I am finding delete queries on large InnoDB tables very slow - are there ways to speed this up? I have a table with about 100 million rows: I am trying to delete just a few of these rows (the following select takes a couple of seconds): [color=blue] > SELECT count(*)[/color] -> FROM UserSnap -> WHERE LogDate<now() - INTERVAL 750 DAY; +----------+ | count(*) | +----------+ | 308969 | +----------+ [color=blue] > DELETE FROM UserSnap WHERE LogDate<now() - INTERVAL 750 DAY;[/color] That delete query takes hours to run. The structure of the table is: [color=blue] > desc UserSnap;[/color] +----------+-------------+------+-----+---------------------+-------+ | Field | Type | Null | Key | Default | Extra | +----------+-------------+------+-----+---------------------+-------+ | LogDate | datetime | | PRI | 0000-00-00 00:00:00 | | | Period | tinyint(4) | | PRI | 0 | | | UserName | varchar(50) | | PRI | | | | RateType | varchar(50) | | PRI | default | | | Rate | float | YES | | NULL | | +----------+-------------+------+-----+---------------------+-------+ Any suggestions on why this is slow, and what to do about it?
View Replies !
Loading Large Tables From Files (wikipedia)
I'm an Oracle DBA and new to MySQL. I was trying to load a 5.6 GB xml file onto mysql database using mediawiki tools. The performance decreased gradually and crashed at ~60% completion. Now I'm planning to use xml2sql tools to convert the dumps to txt format. Then I'm planning to load this using mysqlimport. I would like to run this load faster, and on Oracle, I would use the IMP parameters such as INDEXES=n and so on. What are the recommended steps when using mysql? Should I target any of the system variables? I have a 4G ram on the server and can use all of it for the load. Any pointers?
View Replies !
Time Out Message When Query Large Tables
I'm trying to get data from 6 large tables but the volume of data in each table is too large and even select * from one of them make the system stop. I have afew questions: 1-what can I do to avoid the system stop or time out message? 2- To use several tabels infomation should I use 'View' command or can I use other methods? 3-I need to create a new table and insert the result from query in it. If I use the "view" can I insert the result of the view in a table? (I use postgresql).
View Replies !
Selecting From Multiple Large Tables Quickly
My skills with MySQL typically end at "SELECT * FROM table WHERE stuff", so I've had a lot of trouble with optimizing this query. I've tried doing multiple select statements for this, but that usually comes out with an average execution time of 99s. This method has a execution time of 6s still, so I'd like to get it down. I have 4 important tables here (I'm only posting the columns which I use): `auth` UID mediumint(8) unsigned PRI NULL auto_increment username varchar(32) MUL clan_id mediumint(8) unsigned MUL 0 clan_abbrev varchar(7) MUL `clans` ID mediumint(8) unsigned PRI NULL auto_increment Abbrev varchar(7) MUL Name varchar(64) icon_id smallint(5) unsigned 0 `stats` UID int(10) unsigned PRI 0 kills mediumint(9) NULL `usernames` UID int(10) unsigned 0 ID int(10) unsigned PRI NULL auto_increment Name varchar(32) Uses mediumint(9) 0 `auth`, `stats`, and `usernames` are linked by `UID`. `clans` is linked to a row in `auth` based on it's `clan_id`. I need to search the `usernames` table for a `Name` which matches a certain text (I use "storm" in my example below). I need to return a result with matches to that search, with each row containing:The `UID` which links the tables together for that user.The matching `Name` from `usernames`The matching `username` from `auth`The `Uses` from `usernames`.The `clan_id` from `auth`.The `Abbrev` from `clans`.The `Name` from `clans`.The `icon_id` from `clans`.The `kills` from `stats`. The result should be ordered by `Uses` (from `usernames`) -- highest to lowest. The tricky part is that some users may have Ɔ' as their `clan_id` in the `auth` table, in which case the clan id, abbrev, name, and icon_id should all be blank (0 or '' based on the type). Here's what I managed to hammer out with my limited knowledge of SQL: ( SELECT usernames.UID AS `UID` , usernames.Name AS `result` , auth.username AS `playername` , usernames.Uses AS `Uses` , auth.clan_id AS `clan_id` , clans.Abbrev AS `clan_abbrev` , clans.Name AS `clan_name` , clans.icon_id AS `clan_icon` , stats.kills AS `kills` FROM `usernames` , `auth` , `clans` , `stats` WHERE usernames.Name LIKE '%storm%' AND auth.UID = usernames.UID AND clans.ID = auth.clan_id AND stats.UID = auth.UID ) UNION ( SELECT usernames.UID AS `UID` , usernames.Name AS `result` , auth.username AS `playername` , usernames.Uses AS `Uses` , 0 AS `clan_id` , '' AS `clan_abbrev` , '' AS `clan_name` , 0 AS `clan_icon` , stats.kills AS `kills` FROM `usernames` , `auth` , `stats` WHERE usernames.Name LIKE '%storm%' AND auth.UID = usernames.UID AND auth.clan_id =0 AND stats.UID = auth.UID ) ORDER BY `Uses` DESC It works great, but the average query takes around 6s (these tables have several thousand entries in each).
View Replies !
Large Table
I have a website with over 36,000 registered users. I have a table with 5 million+ records, each of which correspond to a particular user. Obviously some users are tied to a few of these records, others are tied to thousands.I am trying to decide: -Keep everything in the one table, and allow it to grow infinitely large. -Create an individual table for each of the 36,000 users and keep only their records in their respective table.
View Replies !
MySQL With Large Table.
I am using MySQL for a table which will have 100M+ records, avg length of records being 130 bytes. When the number of records reach approx. 25M (and the file size close to 4GB), the rate of inserts falls drastically from 800 per second to 30-40 per second. Details: * MySQL 3.23.58 on Fedora Core 3 * Table has 4 indices. * I have got rid of 4GB file size problem with MAX_ROWS=1000000000. * File system : ext3 on single disk. ext3 could create 10G file without much trouble. So I am convinced that ext3 is not the bottleneck. * Tried using InnoDB engine but it also doesn't meet the requirements. Requirements of database: * A single table in the database with 100M+ rows, each of size 130 bytes (approx). * 500-600 inserts per second. * 200 selects and 200 updates per second. (These statements will affect only one row) * 3-5 select statements per minute which can return 10k to 500k records. * No foreign keys/ACID transaction requirements. * Fast recovery in case of crash. Questions: * Does MySQL performance drops when the table grows beyond 4G? * Which are most important server variables which need to fine tuned? Currently I am using only key_buffer = 512M.
View Replies !
Large Test Table
How do I create a very large table, for testing? Can I download one somewhere? I've tried to search google, but couldn't find anything. The contents dosn't need to mean anything, so a random function with a loop or so should do it, but if I could download a large test table somewhere with "meaning" that would be great.
View Replies !
Breaking Up A Large Table
I'm working with a web-based project management app that stores task information for multiple users in a common 'tasks' table. The table has over 500K rows at this point and is accessed by around 1000 users. The queries are optimized and the table is indexed properly (I think) but as the db grows performance is starting to lag. I'm thinking that the quick fix would be to break up the tasks table into smaller ones by user. So ... is that a good idea? All of a sudden I'm going to have 1000 more tables. Also the app code will be adding tables on the fly as new users come on board. (The existing tables are all of type MyISAM)
View Replies !
Large Database Table
I am using MYSQL in order to develope a database table that currently has about 20 million records. The table grows on a daily basis and is expected to grow to a size of about 300 million records.I wanted to get some suggestions in order to handle such large database. What kind of indexes are best? When should splitting the table be considered? What are ways of achieving good performance of queries?
View Replies !
Copying A Large Table
I need to copy a large table without locking the database. So at the moment i have Insert into newtable select * from oldtable. This locks the database for 10 seconds which although doesn't sound like a lot of time is 2 much for what i need the database for!. I have looked at these commands FOR UPDATE. LOCK IN SHARE MODE. SQL_BUFFER_RESULT But I am not quite sure if any of these actually perform what i would like to do. The other option is to update the table line by line or in chunks using limit i suppose. But im sure there must be a better way.
View Replies !
Copy Large Table?
I want to copy just part of a very large table from one database to another database. The table contains several million of rows. So far the only thing I can think of is using MySql dump. However, that will give me all the rows in that table, right? But I just want to grab 1000 rows to do testing.
View Replies !
Large Table Structures
What are your opinions on large table structures? I am building the largest database I have ever built and want to make sure its done right.. We are creating a db for entering information from large forms... The simple forms I have just been creating as one or two tables as needed.. But now I have a form with the possibility of 100 fields.. many of them blobs... I am seperating the forms into logical sections... For example, Job Information, Contact Information, etc... Would it be better to store each logical section as a seperate table since the number of fields in the table is so high... or does it even matter at all?
View Replies !
MERGE Vs 1 Large Table
Ok, so I have a bunch of small tables I'm creating based on the data. Users submit information with a # and a source, and their review of it. My problem is the #'s between the sources could be identical so I broke the main table up into each table for each source. (only one # per-source so I'm using it as unique so I don't get more than one entry per-source) I'm curious if MERGING the tables when doing a query will be slower than doing the 1 large table. My idea for the 1 large table was #####sourceID that way they could be unique still - but it is easier and cleaner to use seperate tables. So in the end... is merge as fast as 1 large table?
View Replies !
Extracting Data From Large SINGLE-table Database To MULT-table Relational Database
I have a very large single-table database of articles that I want to convert to a multi-table, relational database. The existing single-table database contains fields for article author, article source, and article category, where several 'author', 'source', and 'category' IDs repeat dozens of times for hundreds of different articles. I want to create seperate tables for author, source, and category and populate the new tables by extracting data from the original single-table database by unique ID field. I figured out how to use INSERT and SELECT to pull data in new tables, but can't figure out how to pull only a single instance of a unique author, source, and category to create master reference tables for author/source/category.
View Replies !
Update Performance On Large Table
I have a table that is approaching 4 gigs. I have optimized as much as I can with indexes so that select statements are ok, but updating entries seems to be taking a bit of time. I have 2 questions: 1) I know about the EXPLAIN command for select queries, but is there something like that for updates? I get an error when I try to do "EXPLAIN UPDATE mytable..." Do updates use indexes the same way as selects? 2) Would breaking the table up into seperate tables and then using a merge table increase the performace? I've heard that key reads are slower on merge tables - but since I could make 4 tables of a gig each, wouldn't it be faster opening up 1 smaller file to make an update instead of a huge one?
View Replies !
Equi-join On A Large Table?
I'm running on MySQL 5.x Let's say I have the following table of data: Code: Table: test USER_ID|CLASS_ID 12 13 14 22 24 32 34 I'm trying get all the classes that these users have in common. E.g. If i ran a query to do this I would get class_id 2 and 4, because those are the only classes that all three users have in common. I think I want to do a equi-join, something like: Code: ....
View Replies !
Optimizing Join W/ Large Table
Here is a high level description of my tables I have a table called Items items are grouped by an ID, but it is NOT unique I have an index on ID for the Items table and a table called CompletedItems completed items consolidates information about the items, and here the ID is unique. I have unique index on ID for the CompletedItems table What i need is a query that will return all item id's which have not been completed. I have this query: select distinct(items.id) from items left join (completeditems) on (items.id = completeditems.id) WHERE completeditems.state is null; this gives me the data I want except there are two problems a) this query takes a long time to run. while it is running, the completed items table is locked, which causes several other processes to block while this is happening. to give you an idea of scale, the items table has about 20 million records in it, and the completed items has about 4 million.
View Replies !
Index For A Large Myisam Table (MYD ~=7GB, MYI ~= 7GB)
I have a very large table with 66,836,760 rows (in my world), MYI and MYD files each about 7GB in sizes and even though I've created indexes for all the possible columns, some queries are still slow. I think I read somewhere that for a very large table, there is a way to configure Mysql to store the index file separately just for that table to facilitate the query. How do I do that?
View Replies !
Mysqldump Of A Large Database Into A New Table?
Is there a way where by one can do a mysqldump of a large database, change the table stucture(ie add columns to the tables) in the main db and restore the old data from the mysqldump into the new table taking care not to enter any data into the new columns.
View Replies !
Large Number JOINs And With Two FK To One Table
1) I have a first table with a PK (OwnerID) and a large number of FKs 2) I want to return the 'filename' column from the tables referenced by the FKs (and wonder if there is a better/faster/more concise way to do it than the JOIN syntax below? 3) In one instance there are 'bookcaseRight' and 'bookcaseLeft' fields in the first table which both reference the same 'bookcases.bookcaseID' -- and I cannot figure the way to do this (and/or/union all give improper results) Simplified code below and Thanks in advance: --------------------------------------------------- SELECT doors.filename AS door, bookcases.filename AS bookcaseL, bookcases.filename AS bookCaseR, FROM offices JOIN doors ON offices.doorID = doors.doorID JOIN bookcases ON offices.bookcaseR = bookcases.bookcaseID {?AND?OR?Union?} offices.bookcaseL = bookcases.bookcaseID WHERE offices.ownerID = 1 ;
View Replies !
Index Large Table And Low Memory
Experienced users warned me against possible problems with *index* definition for large tables: they claim MySQL indices are stored completely in memory, which is problematic to me : I have a huge table and limited memory size (RAM). I did the math, and the entire index just won't fit into the RAM. Unfortunatelly it will be a while before management approves hardware improvements.
View Replies !
System Resets When Loading A Large Table
I have an application which is essentially 'read-only' - once the database is loaded it is only read from. So, I am trying to implement it with various indices and a table containg some large content in blob columns. I observe the max_allowed_packet setting and 'split' blobs too large among multiple rows. All is fine during testing. However, when I try a production load (which should load about 1.5G to the various tables) the system resets at or near the 1G mark leaving no logs/msgs (that I can find) telling me what went wrong. I have tried the following to no avail:
View Replies !
Faster Way To Select Range On Large Table
I've written a script to backup my database. Some of the tables are quite large and I don't want to lock up the tables for too long as the site has a lot of traffic so the script backups the tables in chunks of 10000 rows. When the script gets into the larger part of a large table it gets quite slow. Here's one of the queries which takes around 30 seconds: SELECT * FROM mail WHERE mail_id > 2410175 AND mail_id <= 2420175 I've tried using BETWEEN instead, but it doesn't seem to go much faster if at all. Any thoughts on how I could speed up this query,
View Replies !
Large Table, Slow Query Question
I have a table with ~800,000 records. I need to grab random rows from the table based on certain criteria. The problem is that average lowest subset to grab the random row is around 200k. Here is what I'm trying to do: There are 4 columns: data,n1,n2, and n3. I need to get the value of the data column based on criteria using the n1-n3 columns. The most common query is SELECT data FROM table WHERE n1 = ? The problem is that n1 can be only 1 of 5 possiblities. When the table is finished being populated there will be roughly 1.5 million records and 250k for each value of n1. Of course, I have an index on each n column. Right now with just the 800k records it can take over a second, sometimes multiple seconds to run the following in order to get a random row from that subset: SELECT COUNT(1) AS total FROM table WHERE n1 = 3; index = random number from 1 to total SELECT data FROM table WHERE n1 = 3 LIMIT index,1; How can I speed this up? I need it to take less than half a second if possible. Thank you.
View Replies !
Import Large Size CSV File Into Table
I need to import a large size CSV file, around 1 million records into a table, I found that the speed is very slow, it takes 1 minute to import 1400 records. If I import the whole file, it will takes more than 10 hours.
View Replies !
How To Download To Local Client From A Large Table
I have been trying to download a large text file (17 MB) from my large table using all sorts of scripts etc. The best so far has been using - "SELECT * INTO OUTFILE 'tmp imes.csv' FIELDS TERMINATED BY ',' LINES TERMINATED BY ' ' FROM tbl_times"; This is fine on my test server - because it is 'local'... but what I need is the syntax to get it to work in the real environment and save the file to a client machine. Ideally to give them the choice of where it will be saved.
View Replies !
|