Efficient Way Of Mass Indexing?
I have been working on a program that will populate and index a database. The populating doesn't take too long, but the indexing does. My question is: Is there a better approach to indexing this table than the one I'm using right now?
I'm doing this through QtSql. headers is a QString array of 48 column names I want to index.
for (int i = 0; i < headers.size(); ++i) {
q.exec("ALTER TABLE mysqltable ADD INDEX(" + headers[i] + ");");
}
example run for 20000 rows:
viper,david $ time ./main -r 20000
Database Connection Established
0.520u 0.060s 0:21.27 2.7% 0+0k 0+0io 0pf+0w
View Complete Forum Thread with Replies
Related Forum Messages:
Mass Insertion
I have a user event manager where they would like to be able to insert recurring events. This is to say that the user specifies that an event would recur (for example) every week at the same time, and then the script would insert all these seperate events. Is there anyway to use a single query to do something that inserts a certain amount of data into the proper records and sets the dates correctly? Or do I need to recursively contruct a query to insert all the records at the code level?
View Replies !
Mass Installation
I need to install the same database on 40 accounts, all with the same database name, and same table in it... any way of doing this? I've got access to WHM.
View Replies !
Mass SQL DELETE
managed to get put in charge of a forum that uses it extensivly. Currently, the forum is getting bogged down by the 1.5 million posts so I have archived and wish to delete them. I have tried a lot of different variations on DELETE, all of which time out. I am left SELECTing the TOP 20000 and then highlighting and deleting them. I would like to find a quicker, more efficiant way of deleting 1 million rows from a single table (tblThread) that won't timeout.
View Replies !
Mass Inserting
I'm trying to build a query for a mass insert it is like insert into table values (, , , , , ... ,) , (, , , , , ... ,), (, , , , , ... ,), (, , , , , ... ,), (, , , , , ... ,); however, I notice I can't get more than 1000 inserts this way, what is the maximum amount of inserts I can make using this query?
View Replies !
Column Alias On Mass
Is possible to retrieve all columns and alias them all at once. Eg, normally you would write select * from products which would return id | name | price ----------------------- 1 | Test | 14.00 but I want to be able to say select p.* from products p, so that it returns the columns as such p.id | p.name | p.price Obviously I can do this manually as such select p.id, p.name, p.price from products p .... But that would take a lot of big queries as some of my tables have 50 columns.
View Replies !
Mass Updates To A Table
I've got a web-based form that will allow people to update a database that is displayed as a table. Let's say that there are several columns across titled Red, Blue, Green... and several rows titled 01, 02, 03... In each cell, they can fill in some simple text data. The only way that I can think of to update the database with this information is using two imbedded loops in Perl: pseudocode: for($color=1;$color<=10;++$color) { for($number=1;$number<=10;++$number) { update table xyz set value=$value[$color][$number] where color=$color and number=$number; This would mean that it would have to execute 100 update statements. Perhaps this is no big deal and I shouldn't worry about it, but since I'm relatively new to sql, I'm guessing that there's probably a more streamlined method to do this. BTW - I've seen a lot of people asking questions that were vaguely similar to mine in this group, and they've been told to go to the php group, even though I didn't see any php-specific code in their questions. I don't use php and don't have it installed on my server.
View Replies !
Mass Left Join
I would like to join a lot of tables but they all share the same structure. All of them have the same id column. I found a solution, but I'm almost sure there is a more efficient way. My solution is like this: select id,c1,c2,c3,c4,...,c29 from t1 left join t2 using (id) left join t3 using (id) left join t4 using (id) ...
View Replies !
Mass Update Tool
we have so many mass upload tools, load from the file, load from other database built in. How about mass update tools? Say I have a table have 3000 products in it. I want to update the price of these 3000 products with a new table (or excel sheet) of new price by matching the product code (key).Is there a tool for that?
View Replies !
Sending Out Mass Email
I have about 10,000 members with emails stored in a database and would like to be able to send emails out to each of them. I'm not sure at all how it would be done though. I tried a for loop using MySQL queries but none of it worked. Does anyone have any suggestions on how to do this? I would love to send out a newsletter to my users.
View Replies !
Help With Mass Update Query
I have a field in my table (1000 records) called company_number. I've just been informed that all records that start with "1" should actually start with "0". So, for a record where compnay_number = 1000400 the number should actually be 01000400. Can anyone suggest how I would approach updating each record via one SQL statement? SELECT * FROM `myTable` WHERE `company_number` LIKE '%100%' ... then what?
View Replies !
Weekly Mass Data Relocation
I've got a table located on my website and localhost. The table maintains a size of about 200MB. The table holds a running 4 weeks of data (50MB/week). I have a winapp that updates my localhost db each week and currently, I run a backup on the table using the MySql Administrator... then I run restore on my website's mysql server. This deletes the whole table and reinserts each row. This isn't needed and is a long process. Is there a way to select a range of rows from the localhost table and "migrate" those rows to my website's mysql server without following the process I just described?
View Replies !
Mass Converting Database Entries
I have an phpbb board and some old MOD changed some entries that it shouldnt. I need to change 1300 entries to default value. How can i do that without having to change one by one manually?
View Replies !
Mass Replacing Info In Tables
fairly new to mysql and wondering is there any way to mass replace info in one of my fields. I have a town based member website and they enter their town which goes into the town field of my member table. Problem is the town has officially changed it's name and I am looking for a way to just mass replace everyone's town name in the field. For example all users are from Billington and I want to mass replace every members town field with New Billington the new town name. I understand that I can open up each members profile and type in the new town but I have about 1000 members and was looking for an easier way.
View Replies !
Mass Replace Text In String
I want to replace a few letters in a string throughout my database column and using: UPDATE products SET pID = REPLACE (pID, "BSC", "BRC") Did not do the job. The Strings are variations of 33-BSCE7_14_16 all having BSC that needs to be changed to BRC Is there a way to use something like "*BSC*" to have it found and replaced?
View Replies !
Export Table Data For Mass Labels
My goal is generate large lists of mailing labels that are based on: Name Address City State Zipcode I created the table and fields, implemented a submission form, and incorporated a link for an admin to download the data into Excel but thats as far as I got. Is there a better or more thorough approach? I believe that in the end, I want to download this into Word so I can do a mail merge.
View Replies !
Want To Speed Up Mass INSERTS When There Is Processing Of The Insert_id()...
I am importing data from a large plaintext data set and it is going very slowly. I need to acquire and process the insert_id from each record so I cannot use INSERT DELAYED or LOAD DATA INFILE, for which I could speed things up by increasing the bulk_insert_buffer size. I'm not sure if there is really anything that can be done, but I thought I'd ask. The looping process is like this. 1. insert row into table 1 2. get insert id 3. one or more inserts on table 2 using the insert_id Repeat 2000 times or so per page load, because PHP just dies if the file isn't processed in small result sets. It is partly slow because table 1 has several indexes and table 2 has one. The indexes cannot just be added at the end either. In the end there will be +30million rows in table 1 and probably more than 100million in table 2 so if I tried to add the indexes at the end php would die off long before the script could complete.
View Replies !
What's The Most Efficient Way?
I've done a database scheme and wonder what the most efficient way of storing certain entries would be. The site will have news, reviews and tutorials and these will be under the same categories. So I'm wondering what the best way to do design the database scheme would be. Have one table like I've done now in the "content" table with the posibilty to differentiate the entries with the "post_type" field or have three different tables? Below is the table "content". Field Type Null Default post_id int(10) No post_date datetime No 0000-00-00 00:00:00 post_text text No post_title text No post_cat_id int(4) No 0 post_description text No post_name varchar(200) No post_type int(4) No 0
View Replies !
How Efficient Is In() , And How Much Is The Most That You Should Put In An In()?
I am just wondering as to the efficiency of the in() function in MySQL. select field1,field2,...,fieldn from table where id in(1,2,3,4,5,6, ... , n); assuming: id is an indexed field. table is a VERY big table (100 000+ - 1 million+ records) what do you think is the largest number of values you could pass to the in() function without completly flattening your server?
View Replies !
Most Efficient YES/NO...
I) Table_A has fields Data_1, Data_2, Data_3, and Data_4; I need to determine whether value 'X' is present (at least once) in any of the Data_N fields. All I need is a YES/NO answer: YES, 'X' is present (at least once) in one of the Data_N fields of Table_A [Record number(s) not required!], or NO, 'X' is not present in any of the Data_N fields in any of the records in Table_A. II) Similar scenario, but this time Table_A has fields RecID, Data_1, Data_2, Data_3, and Data_4. I need to determine the RecID (none, one, or more) of every record that has value 'X' in any of the Data_N fields. I'm looking for the query that will be fastest and most elegant to implement.
View Replies !
Quickest And Most Efficient
i run an online game. I want to give each user a ranking, based on how high thier score is.If for example, i had 20,000 players, i dont want to have to update them all one by one, it may strain the server and take a long time. Is there another way i can assign a rank number (rank 1 has the top score and so on)
View Replies !
Efficient UPDATE
I have a table with the following structure; CREATE TABLE my_table ( id_1 int(11) NOT NULL , id_2 int(10) NOT NULL , stauts tinyint(1) NOT NULL DEFAULT 0 , PRIMARY KEY (id_1) ) Engine =InnoDB'; The table currently has arround 100,000 entries. When I try to run variations of the following statement it is taking around 4 seconds per query; UPDATE IGNORE my_table SET id_1 = 74240, id_2 = 5 I need it to be running a lot faster than 4 seconds per query as I need to update upwards of 100,000 records a day! My server is fairly beefy, a 3 gig dual core opeteron and is generaly running below 1.0 load.
View Replies !
Efficient Tables.
I am starting out on a project where I need to store GPS information. The data consists largely of a series of "Points" each consisting of a longitude,latitude and elevation. On a typical "route" there could be hundreds of points. My question is how can I efficiently store this information. It does not sound sensible to normalise this and add hundreds of rows to a table for each "route". Sample data is along the lines of : - <trkpt lat="54.016942977905273" lon="-1.4903640747070313"> <ele>82.330078125</ele> <time>2006-09-03T07:35:41Z</time> </trkpt> - <trkpt lat="54.016938870772719" lon="-1.490332055836916"> <ele>0</ele> <time>2006-09-03T07:35:42Z</time> </trkpt>
View Replies !
Efficient Storage Of IP Address
I am establishing a database for the purpose of logging access to my secure webserver and am wanting to make the database as efficient as I can because it will be doing a lot of work when the site goes live. What is the most efficient way in a MySQL table to store remote IP addresses? What data type should I use? Should I just go with a basic VARCHAR(15) to allow for 4 sets of 3 digits with 3 decimal separators, or is there a better way?
View Replies !
Efficient Placement Of Fields
is there any noticable efficiency is ordering the position of the field types?. Like say i place the join keys in a table at the end, and varchars and text at the top of the table does it really matter? Also when i do a query like select * from table where id IN (1), where 1 is usually a primary key int is that quicker than doing where id=1 or is there no difference and is it handling the int as a string or as an int?
View Replies !
Efficient Database Structure
I have been developing a new website and i need help in deciding the best database structure for it. The site is basically a dating website with various modules like blogs, videos, comments, friends, photos etc. I have created a member table that stores all the basic profile info and created seperate tables for friends, photos, messages, blogs etc and MemberID as foreign key. Now on profile page, i want to display all info related to member like his profile info, his photos, friend list, messages etc and i have to execute 7-10 short queries on profile page for this. Also, i think Joins will not be much helpful as there is one to many relationship e.g there can be more than 1 photo for a member and i am saving each photo in a seperate record. Similar is the case for other tables?
View Replies !
Creating An Efficient Database
i'm currently writing a web based catalogue system in php using a mysql database. the catalogue has a number of products in it from different brands. i would like to know if it is more efficient to have each brand in a separate table then a "master" table just listing the brand name and corresponding table or all the products in one large table and a "master" table listing each brand in the large table. the large product table would of course have a field to state which brand the product was from. the efficiency would be based on users being able to access the database via html browsers using php and also search the database.
View Replies !
How To Make Index Most Efficient
I have a large table (> 3,000,000 records). Each records contain a primary key like 'id' and a lots of attributes like 'age', 'department'. I want to build some indices to accelerate my query. I read the document which says that too many indices may slow down the INSERT and UPDATE operations. So is there any rule on how to set indices in such table? If I create an index for every field, would that be a very bad idea? If I create an index on each of two fields but not on their combination, will the indices contribute to queries on the combination?
View Replies !
Efficient Way To Count Rows
I'm trying to get the number of rows in a table with a very large number of records in it (~9 million). When I run a select count(*) for some criteria (where name='something', etc) it takes around 6-8 secs for the query to return the value. I tried by using SQL_CALC_FOUND_ROWS with a very small LIMIT but then the query was taking even longer. I'm using InnoDB, with query caching enabled. I could look at the information_schema and get the approximate row count but whenever I use a where clause it'll be way off mark.
View Replies !
Is This INSERT With SELECT As Efficient As Possible?
I'm writing a pretty complex web app and will be repeating many times over a query very similar to below and need to know if there is a more efficient way to do it. If anyone has input, I'd be happy to hear: INSERT INTO table (somecolumn) VALUES ((SELECT id FROM other_table WHERE foo = 'bar'))
View Replies !
Is This An Efficient MySQL Setup?
I would like to offload the MySQL server from my dedicated box in order to speed up page loads. I don't have an additional dedicated server so my only option is to get a VPS. But is offloading the MySQL server to a VPS, albeit a modest one, even worthwhile? I understand this is a very broad question because I am not providing any details but that being the case I am expecting a broad answer
View Replies !
Efficient Way To Left Join?
What is the most efficient way to do a table join, but even if there is no matching foreign key, still return the table on the left. SELECT col, col2, COUNT(col3) FROM tbl1 t1 LEFT JOIN tbl2 t2 ON t1.id = t2.id GROUP BY t1.id But this only works if there is a matching foreign key on t2, I would like all t1 rows to return regardless of whether they exist on t2 or not.
View Replies !
Most Efficient Way To Extract Limited Data
I am currently using the following code, is it the most efficient way to extract and sort the 6 items from the database. The database currently holds over 2,500,000 rows and I want to extract the data as efficiently and quickly as possible. Code:
View Replies !
Most Efficient, Stable Version Of MYSQL?
We're still using mysql 3.23.56, so it's time to upgrade! It has never once crashed on us, so we've been content with it so far. However, some of the inefficient query optimizations make us want to upgrade. Does anyone know what the current most efficient and stable version of MYSQL is? We don't require views or cursors or stored procedures or any of that... Just whatever's the most reliable and best optimized for fast query execution of fairly simple SQL. I know that 3.23 has some issues with using indices correctly during optimization (most notably in ORDER BY ... DESC). Is that problem fixed in 4?
View Replies !
Efficient Ways To Retrieve Specific Rows.
My site used forum software that I wrote myself (in Perl) which, up until now, has used flat files. This worked fine, however lately I've been wanting to do more stuff with user accounts, and had been eying MySQL for over a year. Finally I've decided to start off small by converting the forum's account system to a MySQL database (and convert the rest later after I'm comfortable). So far everything is working fine, and I've figured out how to create the table, insert records, modify records, and so on. However I had a question on what was the most efficient way to retrieve information about a user as I read through the flat file containing messages. As each message is read I want to find that user's relevant information, build it into my output, then continue on to the next message. Now here's my problem. Since I'm reading a flat file in a way that was once trying to be somewhat memory efficient (on really slow server way back when, trying to avoid arrays and hashes) I'm finding that I'm having to send separate questions to MySQL as each message comes up. Say from 1 to 20 very simple questions to complete printing the page. Also, sometimes the questions might be repetitive - since I'm not storing any of the results in memory, if a user appears twice I ask about it twice. Would I really be better off trying to find a way to consolidate all distinct users into a single question? Or is MySQL efficient enough that this isn't really much of a concern?
View Replies !
Efficient String Compare To A Group Of Values
(Post edited to change "WHERE field = options" to "WHERE field IN options"). I have a website using PHP and MySQL 5.0; the database holds personal names, addresses and various information. Right now I'm building a query which checks whether a person lives in a certain group of UK counties. I've got arrays set up in PHP which hold the groups of counties, and I've dynamically created a "group" from each array. So, in PHP, $county_array = array('Kent','Surrey','London','Essex','...'); $county_group gets set to "('kent','surrey','london','essex',...)" Each group may hold perhaps 20 individual counties. Now, for my database query, I use: $query = "SELECT ... WHERE lower(county) IN $county_group"; Is the "WHERE field IN (a|b|c|d|e|...)" a slow comparison in MySQL? Is there a more efficient way to do it? Is this something that stored procedures can help me with?
View Replies !
The Most Efficient Manner To Update Multiple Rows
I have a table that holds information about people The table is quite large (300,000 rows) I need to write code that update many of the rows in the table Here is what I need to do Two of the fields that I store about each person are: location and country The location is some free text which the user can type in, while the country is the ISO country code, like "US", "ES" and so on I need go over all the records in the database, and for those that have a non-empty location field, to extract the value and to try to guess what country the user is from and to update the Country field accordingly My problem is that I might find that most of the records need updating, and this might lead to 100,00+ update statements I cannot use LOAD DATA as this is product database Is there a way to update multiple rows (each with a different value and condition in a single query?
View Replies !
Efficient SELECT From Multiple Tables With Same Formatting?
Let's say I have 2 tables with the exact same formatting (field/column names, etc) -- the only difference is the name of the tables.. one is called "table1" the other is called "table2" If I wanted to query both of the together is this legal and good coding practice? "SELECT id, headline, permalink, body FROM table2007_11, table2007_12 WHERE id=23" understandly I get this error: Column 'id' in field list is ambiguous" is there a better way to do this than printing out each table name for each field individually? (ie. id. table2007_11, id. table2007_12, etc)?
View Replies !
Efficient Country/Province City Setup
I currently have a program that does a global query of all of the data. I want to break it down so as to drill the user down to thier local area. for example: Country Province Region City As you can see, a country can have many provinces, a province many regions and a region many cities or a province many cities. What is the most effecinet way to do this in MySql? I'm OK with doing single tables. ie: INSERT INTO Province VALUES (NULL, 'Ontario', 'ON'); but I'm unclear if that's the way it should be done. I'm using PHP on the client side.
View Replies !
Most Efficient Way Of Storing Data From Multiple Accounts
As part of a system I am putting together I need to allow users to create thier own accounts on my servers. Each user can create their own account, and then have their users register for it. Each account needs it data seperate from the others, a member registered for one account should not be able to view another account and a username registered with one account should still be available to the other accounts. The ways I have been looking at are: 1. Create a new database for every account created so that all users are kept in seperate databases. 2. Have one table for users, one for topics, one for posts etc and then associate each row within this table with the relevent account. So for example a user could register with the forum with the ID 4, so their user entry would be Userid: 234 Username: xxxx Password: xxxx Forumid: 4 Then when a new member registers with any account I simply check that there is not another user with the same account ID and username. Indexes on relevent fields in this system could help speed up huge tables. I expect to quickly have 20,000 plus accounts (and in theory it could go up to hundreds of thousands). I guess my question is which of these methods is better from a speed point of view once we get a large number of accounts and users. Also, are there restrictions on the number of fields in a table that could cause problems? Using MySQL by the way, on an Apache server.
View Replies !
Most Efficient Date Field Type For Selects?
I have searched this forum seeking the answer to this question, to no avail. I am making an events diary that will have a lot of entries. Entries will be entered in all kinds of order, including multiple events spanning a year or so. Even multiple events of the same type will be individual entries. (e.g. weekly chess club meeting every weds at 7pm will be 52 entries) My question is : Which is the best format to store dates? Native mysql date type Integer timestamp A.n.other? I suppose a worst case scenario is imagine I want to generate a calendar for July (31 days) and I want to show how many events there are for each day. 1st July (13 events) 2nd July (22 events) and so on ... it is possible that any given day could have up to 100 entries. What in your opinion is the best column type to use and ss there any kind of Indexing I can use to speed up select by dates? Or any other tip or trick you'd be prepared to share with me?
View Replies !
|
TRACKER
