Large Number Of Tables And Columns
i'm developing an online portal, which my MYSQL database contains 100 tables.. my doubt is if any table contains 30-40 columns will it become a problem? like my html front end form has 40 fields for data insertion. and i'm not maintaing any duplicate data if my DB contains 100-150 tables will it create any problems?
as i'm inserting data from EXCEL via PHP to MYSQL, please suggest me whether a healthy MYSQL table is dependent or independent on NO:OF COLUMNS(FIELDS) or not?
View Complete Forum Thread with Replies
Related Forum Messages:
Large Number Of Tables
I would like to ask whether it is feasible (or whether it makes sense) to run a PHP script that creates a number of tables for each user. Basically, this script allows users to sign up for a free service, and for each user that signs up, the script creates 9 tables in the database to contain that user's details. Is this a sensible way of containing user data, and what are the inherent problems this may cause especially when say are 1000 users signup?
View Replies !
Large Number Of Queries
Ive currently got a loop on a php page which takes ~7000 rows from an sql table, parses information from it and does inserts/updates/deletes depending on the information. At the moment, it does a new mysql_query for each of the inserts/updates/deletes so it ends up doing ~10k queries each time its run. Im wondering if its better to use an array to store the queries and then put them all together (";" seperated) and run all the queries at once in the same mysql_query. One thing to bear in mind is some of the data involved in the queries relies on previous data from queries in the loop, would this cause any problems with the combined query
View Replies !
Large DataBase Columns
I am developing, what i consider to be, an extremely large database, using mysql and php. im concerned that one of my sql tables will be too large. tonight, im running some tests, but i was hoping the forums could toss me some good advise. The table will be around 1500 columns long, and about 500 rows on average. each column will not be any larger than 20 varchars. i've never created anything beyond 30 columns. is this beyond what's recommended with using mysql ??? i know the queries may take a little bit to load, but is this simply a bad idea anyway, or am i just getting nervous for no reason at all.
View Replies !
Large Number JOINs And With Two FK To One Table
1) I have a first table with a PK (OwnerID) and a large number of FKs 2) I want to return the 'filename' column from the tables referenced by the FKs (and wonder if there is a better/faster/more concise way to do it than the JOIN syntax below? 3) In one instance there are 'bookcaseRight' and 'bookcaseLeft' fields in the first table which both reference the same 'bookcases.bookcaseID' -- and I cannot figure the way to do this (and/or/union all give improper results) Simplified code below and Thanks in advance: --------------------------------------------------- SELECT doors.filename AS door, bookcases.filename AS bookcaseL, bookcases.filename AS bookCaseR, FROM offices JOIN doors ON offices.doorID = doors.doorID JOIN bookcases ON offices.bookcaseR = bookcases.bookcaseID {?AND?OR?Union?} offices.bookcaseL = bookcases.bookcaseID WHERE offices.ownerID = 1 ;
View Replies !
Number Of User Accounts For Large Database
I have a large database (816 tables with 11 fields each, each field could hold up to 50 entires) and I'd like to know if each table should have its own user account (created in Cpanel and used in the config file to access the database). May sound like a simple question to some but I need to know in case we have a lot of traffic and 1 user account is not "enough" for all visitors to access the database info.
View Replies !
UPDATE Large Number Of Rows With CASE Is Very Slow
UPDATE table SET rating = (CASE uniq WHEN 'ACD' THEN 0.6857 WHEN 'GGYB' THEN 0.5831 WHEN 'QUCG' THEN 0.5831 WHEN 'CYP' THEN 0.5831 ... ELSE 0 END), score = (SELECT @t := 1/(1+rating))*votes_up - (1-@t)*votes_down; The table has the column `uniq` as PRIMARY KEY of type VARCHAR(10), and about 500 rows. I need to update the entire table with this query. The problem is the query runs extremely slowly, taking 4.6529 seconds to run. I realize the index is not the most efficient, but is it the main culprit?
View Replies !
Number Of Columns
Do you think a table having more columns affects the efficiency when inserting/selecting/deleting? I mean like 100-150 columns for one table. Is it wise to make two tables and spliting the columns or it's better to keep it this way
View Replies !
Performance :: Number Of Columns
have a table where misc data are stored. Right now this table has about 30 columns, but for sure it will be more in the near future. So I wonder how the big number (50-100) of table's columns affect for MySQL DB performance. Maybe it's better to create more tables rather than more table's columns ?
View Replies !
Selecting Columns By Number
Is it possible to SELECT a column by number rather than name? I can't find this anywhere in the manual. It's for a generalised function on a set of tables with different fieldnames where I always want to select values by the value of the first field.
View Replies !
Count Number Of Columns
I have a table in which all columns except the primary key are type boolean (tinyint(1) in mysql). I want to be able to compare two specific rows, referenced by their primary key, and count the number of columns in which they both have a "1". As a second search, returning the names of these columns would be helpful too.
View Replies !
Increasing Maximum Number Of Columns
I have a requirement for tables with thousands of columns. I know this is not typical, but these tables are both application-generated and appplication-read. We need a minimum of ~3000 columns. I thought MySQL supported this ... but we can't get past 1000 (4.1.18) Any ideas?
View Replies !
Number Of Columns - Performance Impact?
I have 5 pieces of info that I need per row in my table. I have two possibilities: 1. create 5 columns of type smallint 2. create 1 column of type varchar(30) The first possibility puts all pieces of info in separate columns, the second places everything next to each other: 122;4234;874;; My question is, will performance be any different in situation 1 or 2? Worse or bad? Are more columns a bad thing? If you count the actual size of the bytes used, solution 1 sounds better, but I don't know if the number of columns has any impact on the table. The table is not fixed width. Note that the reason why I may choose to put them in 1 column instead of 5 is that I have 3 different fields where I have 5 pieces of info on, so that would translate in either, 15 extra columns, or 3 extra columns.
View Replies !
Number Of Columns In Table ? Is There Any Limit?
My question may sound stupid but i really want to know if there is some kind of recommended limit for number of columns in a table of mysql table. For example, is it okay if i make 100 colums in one mysql table? or i should make more tables to decrease number of columns?
View Replies !
Maximum Number Of Columns And Actual Data In A Row
SUMMARY: I need to know what the actual maximum date limit possible on a row of a MYSQL database. MORE DETAILS: For example, does a MYSQL database only allow 4032 bytes of data to be stored in one row or is there no limit? Does it only allow a maximum of 50 colmums? I tried googling on this, but all the results seem to be about recommended maximums or design style. As an ex-Access developer I remeber hitting a limit with the Access DB which I think was that one row could contain 2048 bytes max(things like blobs do not contribute to this limit). I'm hoping there are no limitations with MYSQL. My reason for asking is that I have been asked to add new fields to someone elses DB. There are already a lot of columns and the short and dirty fix for me which would mean minimul PHP script changes is by adding new fields to the current table. But there are already quite a few so thought I would check.
View Replies !
Loading Textfile With Unknown Number Of Columns!
I have an odd textfile like such. Store Address Employee1 Title1 Employee2 Title2 Employee3 Title3 etc ABC 283 1st Ave Bob Cash Bill Mgr Ben Cash The number of columns depend on how many employees work at the store. I would like to load the textfile into my table which has the following fields. Store Address Employee Title ABC 283 1st Ave Bob Cash ABC 283 1st Ave Bill Mgr ABC 283 1st Ave Ben Cash
View Replies !
How To Model Variable Number/type Table Columns?
I currently model reports using a single report table with it's fixed set of columns describing the parameters of a report. Now we'd like to make the system more flexible by introducing a number of report mysqllates. Each mysqllate defines a different set of parameters a report can have, and the user gets to decide what report format they want by assigning it a mysqllate. the problem is that if I include all those possible parameters as columns in the report table it will be inefficient. No report uses all of them, so every row will only use a subset of columns to store values. I thought the solution would be to create a new table called report_field, and then have a many:many relationship between it and the report table. When a user selects a mysqllate to build a report, I'd create a row in the report table, N report field rows based on what the mysqllate dictates, and then link them. But the problem is that the parameters can be of varying data types, and I can only specify the data types of the columns in the report_field table when I create the table. I could create report_field_int, report_field_date tables etc. But that doesn't seem the right way to go. Querying becomes very difficult as you'd have to determine table names from the mysqllate field data types. I could abandon the report_field table idea and just create report_mysqllate_A, report_mysqllate_B tables etc that model exactly the number and type of fields for the particular mysqllates. But I'd like to maintain a master report table if possible. Also, down the road, it would nice to allow the user the option to design their own mysqllate. In which case I can't know ahead of time what fields I will need to assign to a report. I'm certain this must be a very common scenario, and I'm hoping that there is a much more elegant way of solving this I haven't seen yet. I tried searching online, but I found nothing. It's probably a terminology problem. I'm a db beginner so I don't know the phrases I should be looking up.
View Replies !
Large Tables
So i've got a table with 15 million rows. id is the primary key SELECT COUNT(id) FROM listings WHERE id != 0; Takes about a minute to load... which is insanely slow.. anyone know what could be causing this?
View Replies !
Very Large Tables
I am looking at using mysql to store some particle physics data. So the idea is (for example) I would have a table with a list of events, then a table with a list of particle types (one event can have many particles) and so on. The problem is that quite often I want to calculate something for every single event, but this table might run into 10s of millions of events and many GB. Obviously it would be mad to load the whole result set into memory then iterate over it. My question is: is there any way to have the db push results to the users one (or a few) at a time in an effiecent way? I was thinking of something like an iterator that pulls the data from the disk one (or few) at a time. An alternative (but much less useful) method might be to hand the function I want to calculate to mysql, for example: select myfunc(x,y,z,a,b,c) from events where energy>3; But as far as I know there is no way of defining your own functions in SQL
View Replies !
Large Table Or Many Tables?
I posted a similar question about 6~12 months ago with no definate answer, so I will give it another go, as the choice has popped up again in my programming life: If the number of entries is the same is it more efficient (better) to have a single table with many columns, or many tables with few columns? The obvious answer is a single table, because there is less closing and opening tables, but the reason why I ask is what about searching through a lot of columns being even more inefficient? The situation is I need to record data for every week of the year. Say ~50 columns per week --> 2600 columns. Or, do I split it so it's 52 tables with 50 columns each? In each of these options the user (or user ID) will be the primary key. The seemingly pointless other option is to insert new columns with every new user, and have the week-item as the primary key?
View Replies !
Large DB Should Use Multiple Tables?
I have a social networking site similar to myspace, currently my table which links who is who's friend on my site is 2 weeks old and is at 300,000 rows but is only at 15mb in size, this is the most accessed table of all my tables. My question for performance, as this table grows should it eventually make a new table? If so here is my current select code select userid,friendid,status from friend_friend where userid='$temp[auto_id]' and status='1' Could someone give me an example of how o select this from multiple tables if I add more for this table in the future?
View Replies !
Managing Large Tables
I recently inherited a database (version 5.0.18) that has 1 table constantly growing out of hand between 10GB -30GB, therefore making is difficult/impossible to query. Also, the Archive Storage Engine is not installed. Currently, the table is manually renamed (i.e. tablename_daterange) and a new table created. ....
View Replies !
Join Tables On A Large Database 200 Meg
i am trying to work out the most efficient way to list say multipl= e categories of entries, the database is quite large about 200 meg.=20 I would like to know if using join tables is more efficient than storing th= e keys in a varchar field then within the second loop doing a where in (1,2= ,3,4) where the 1,2,3,4 are the stored category keys in the varchar field = rather than a where in (1), where 1 is the pirmaryID of the entry for insta= nce ?
View Replies !
Optimizing Mysql For Large Tables
I have been working with Mysql for about 5 years - mainly in LAMP shops. The tables have been between 20-100 thousand records size. Now I have a project where the tables are in the millions of records. This is very new to me and I am noticing that my queries are really slowwwwww! What are the options that I have to speed my queries on the mysql side with regards to the my.cnf file. I have a fair understanding of sql optimization and I understand explain. I just want to see if there is something that I can do with the server also.
View Replies !
Slow Join On Large Tables
I have two tables: D (500,000 recs), and DL (2,500,000 recs) D has a PK and an index on HLQ. DL has a PK and an index on ID. The following SQL: SELECT HLQ as "HLQ", count(*) FROM D, DL WHERE D.DLID=DL.ID GROUP BY HLQ produces the following explain: tabletypepossible_keyskeykey_lenrefrowsExtra DALL500000Using where; Using temporary; Using filesort DLeq_refIDID4D.DLID1Using index The query takes ~ 3:30 on a Athlon xp2200; 1GB RAM; default bufer settings. Adding the following buffer settings only slightly decrerased the time (~3:00). key_buffer=512M table_cache=256 sort_buffer=16M read_buffer_size=16M It appeasrs that the 'Using filesort' on table D is due to the Group By clause and is the problem. I have an index on HLQ. Is there any way to get MySQL to use it?
View Replies !
One Large Table Or Many Small Tables?
I'm trying to decide whether to use one large table or many small tables. I need to gather information from various devices (about 500). Each device has its own Id and some data. Should I use only one table with an indexed column for the ID and another column for the data, or should I use 500 tables each with only one column for the data?
View Replies !
Large Tables, Very Slow Deletes.
I've been using MySQL for a while now and are starting to run into limitations. Either my own, or something else. I have this really large table, it stores images, it has a mediumblob field, an unsigned id integer field and a timestamp field. This table contains up to a few million rows and is constantly filled real time. It grows up to sizes between 50 and 100 gb and in the future probably even larger. This data is not supposed to be stored for all time, therefor when a certain treshhold is reached, the oldest 5000 rows are deleted to make room for newer rows. This is where my problem kicks in. Whenever I try to delete those rows, it can take for ever to complete. 500 rows I can delete without problem, 2500 becomes slow, 5000 rows takes several seconds and 10.000 rows or more makes it looks like things are frozen. Now the temporary solution I have is to delete 500 rows at time, but this means I must do that every 10 seconds just to maintain status quo. I would much rather check every 5 minutes or so, and if needed delete 15.000 rows in one go. I use MyISAM tables and tried setting the key size to 128M, no luck. I run this on a dual Opteron system with a GB of memory and WinXP-SP2 Proffesional.
View Replies !
Large Unused Tables Vs Performance
I've been working on a website with a rather large a database. One of the about 100 tables in there was about 1/4 of the total database size. The table had no use anymore at the time I worked the website, so it was just sitting there, doing absolutely nothing. Obviously to spare diskspace we deleted the table. But the question I have now, can unused large tables affect the perfomance of Mysql? I've searched a bit around, but honestly I couldn't find a clear answer about it right away (maybe I haven't searched enough either :) )
View Replies !
One Large Table Vs. Many Small Tables
I'm working on a design using PHP & MySQL and I'd like to get some opinions on this. My design has several tables that will be referenced but I'm wondering if those tables should be broken down even more and referenced more dynamically. The reason that I wonder about this is for long term goals. I hope that eventually there will be two or three thousand records that will be used on a regular basis. These records will need to be separated into groups, but I'm not sure if I should use a field in the database table or create a new table for each group. If a few hundred records could be in each group, do you think it's better to use one large table with a field for the group ID, or a new table for each group?
View Replies !
Speeding Up Large MySQL MyISAM Tables
I'm somewhat new to MySql. I've been using it for a while, but pretty much out of the box setup, and am starting to suffer heavily with my larger tables. I have a table with 5,000,000+ rows that I have to search and do joins on. Although I have an index set up for it, the joined select will still take some 400+ seconds to return, which is obviously unacceptable. This is due to enormous HD access. Perhaps someone can help me with indexes here; I was under the impression that the index for the tables are cached in memory, and therefore permitted "instant" searchability, without having to retrieve data from each of the rows of the DB. Is there a startup parameter, or something in the mysql.ini file, that must be set to allow for this? I have the index configured properly, and have made sure that the query uses there parameters in the where clause in the same order that they appear in the index. Code:
View Replies !
Speed Of InnoDB DELETEs On Large Tables
I am finding delete queries on large InnoDB tables very slow - are there ways to speed this up? I have a table with about 100 million rows: I am trying to delete just a few of these rows (the following select takes a couple of seconds): [color=blue] > SELECT count(*)[/color] -> FROM UserSnap -> WHERE LogDate<now() - INTERVAL 750 DAY; +----------+ | count(*) | +----------+ | 308969 | +----------+ [color=blue] > DELETE FROM UserSnap WHERE LogDate<now() - INTERVAL 750 DAY;[/color] That delete query takes hours to run. The structure of the table is: [color=blue] > desc UserSnap;[/color] +----------+-------------+------+-----+---------------------+-------+ | Field | Type | Null | Key | Default | Extra | +----------+-------------+------+-----+---------------------+-------+ | LogDate | datetime | | PRI | 0000-00-00 00:00:00 | | | Period | tinyint(4) | | PRI | 0 | | | UserName | varchar(50) | | PRI | | | | RateType | varchar(50) | | PRI | default | | | Rate | float | YES | | NULL | | +----------+-------------+------+-----+---------------------+-------+ Any suggestions on why this is slow, and what to do about it?
View Replies !
Loading Large Tables From Files (wikipedia)
I'm an Oracle DBA and new to MySQL. I was trying to load a 5.6 GB xml file onto mysql database using mediawiki tools. The performance decreased gradually and crashed at ~60% completion. Now I'm planning to use xml2sql tools to convert the dumps to txt format. Then I'm planning to load this using mysqlimport. I would like to run this load faster, and on Oracle, I would use the IMP parameters such as INDEXES=n and so on. What are the recommended steps when using mysql? Should I target any of the system variables? I have a 4G ram on the server and can use all of it for the load. Any pointers?
View Replies !
Time Out Message When Query Large Tables
I'm trying to get data from 6 large tables but the volume of data in each table is too large and even select * from one of them make the system stop. I have afew questions: 1-what can I do to avoid the system stop or time out message? 2- To use several tabels infomation should I use 'View' command or can I use other methods? 3-I need to create a new table and insert the result from query in it. If I use the "view" can I insert the result of the view in a table? (I use postgresql).
View Replies !
Selecting From Multiple Large Tables Quickly
My skills with MySQL typically end at "SELECT * FROM table WHERE stuff", so I've had a lot of trouble with optimizing this query. I've tried doing multiple select statements for this, but that usually comes out with an average execution time of 99s. This method has a execution time of 6s still, so I'd like to get it down. I have 4 important tables here (I'm only posting the columns which I use): `auth` UID mediumint(8) unsigned PRI NULL auto_increment username varchar(32) MUL clan_id mediumint(8) unsigned MUL 0 clan_abbrev varchar(7) MUL `clans` ID mediumint(8) unsigned PRI NULL auto_increment Abbrev varchar(7) MUL Name varchar(64) icon_id smallint(5) unsigned 0 `stats` UID int(10) unsigned PRI 0 kills mediumint(9) NULL `usernames` UID int(10) unsigned 0 ID int(10) unsigned PRI NULL auto_increment Name varchar(32) Uses mediumint(9) 0 `auth`, `stats`, and `usernames` are linked by `UID`. `clans` is linked to a row in `auth` based on it's `clan_id`. I need to search the `usernames` table for a `Name` which matches a certain text (I use "storm" in my example below). I need to return a result with matches to that search, with each row containing:The `UID` which links the tables together for that user.The matching `Name` from `usernames`The matching `username` from `auth`The `Uses` from `usernames`.The `clan_id` from `auth`.The `Abbrev` from `clans`.The `Name` from `clans`.The `icon_id` from `clans`.The `kills` from `stats`. The result should be ordered by `Uses` (from `usernames`) -- highest to lowest. The tricky part is that some users may have Ɔ' as their `clan_id` in the `auth` table, in which case the clan id, abbrev, name, and icon_id should all be blank (0 or '' based on the type). Here's what I managed to hammer out with my limited knowledge of SQL: ( SELECT usernames.UID AS `UID` , usernames.Name AS `result` , auth.username AS `playername` , usernames.Uses AS `Uses` , auth.clan_id AS `clan_id` , clans.Abbrev AS `clan_abbrev` , clans.Name AS `clan_name` , clans.icon_id AS `clan_icon` , stats.kills AS `kills` FROM `usernames` , `auth` , `clans` , `stats` WHERE usernames.Name LIKE '%storm%' AND auth.UID = usernames.UID AND clans.ID = auth.clan_id AND stats.UID = auth.UID ) UNION ( SELECT usernames.UID AS `UID` , usernames.Name AS `result` , auth.username AS `playername` , usernames.Uses AS `Uses` , 0 AS `clan_id` , '' AS `clan_abbrev` , '' AS `clan_name` , 0 AS `clan_icon` , stats.kills AS `kills` FROM `usernames` , `auth` , `stats` WHERE usernames.Name LIKE '%storm%' AND auth.UID = usernames.UID AND auth.clan_id =0 AND stats.UID = auth.UID ) ORDER BY `Uses` DESC It works great, but the average query takes around 6s (these tables have several thousand entries in each).
View Replies !
Number Of Tables
just a quick question, some database designs i've seen seem to want to compress everything into as little tables as possible, whereas others can go into a hundred rows for a system that doesn't seem to be that much more complex. A wordpress database is fairly tiny at only a handful of tables, whereas zen cart has about 100+ from memory, and yet I don't see either one as significantly more complicated than the other (although I don't know much about Zen Cart). I do understand the concept of keeping data in logical divisions, but I wonder how far one should push that towards keeping the number of tables to a minimum - no doubt purely just from a human management standpoint....
View Replies !
Managing A Very Big Number Of Tables
I have to manage 2^19 (approximately 524 000) tables, should I divide them into several databases to make it easy for my operating system (ext3 file system) and for my MYSQL server ? And what storage engine is the best in my case ? The second question : with the "min_rows" option can I avoid fragmented tables on my hard disk in my case ?
View Replies !
Optimal Number Of Tables
I know there are various ways to configure a database setup and therefore affect performance tweaks, but in general is there a limit to the number of tables you can have in a database after which performance would be impacted?
View Replies !
|
TRACKER
