Count Distinct In Large Table With 30M Records?

Jul 7, 2015

I have a task to count distinct records in a big table with roughly 30M records, performance is an important factor. Query is to be written to calculate weekly stats, weekly record number could be as high as 1M.

The actual result is like:
ID Policy
350235744Credit Cards
350235744PCI
350235744PCI Audit

So the final number for this particular Policy is 3

I can write the query like:

select count(distinct Incident_id) policy_name
from Reporting_DailyDlpDetail
Where (year(INSERT_DETECT_TS)=2015) and (month(INSERT_DETECT_TS) =6) and (day(INSERT_DETECT_TS) between 2 and 9)
This returns 526254 and costs 11 seconds to complete

or a query like:
Select distinct Incident_id, policy_name
from Reporting_DailyDlpDetail
Where (year(INSERT_DETECT_TS)=2015) and (month(INSERT_DETECT_TS) =6) and (day(INSERT_DETECT_TS) between 2 and 9)
This returns 749687 and costs roughly 1 minute to complete.

Result is different from the two queries, I believe the later gives correct number. How can I count the distinct based on a combo?Considering the size of data, what is the best and most efficient way to run the stats calculation against over 30 different scenarios (different policies and alert types) and not timeout?

View 9 Replies


ADVERTISEMENT

Count Distinct Records

May 15, 2008

how can i count in sql the number or records taht would be returned if i did

select distinct site,date from allrecords

View 1 Replies View Related

Transact SQL :: Optimize Count Distinct For Multiple Groups Of Records?

May 21, 2015

I have a CTE returning a recordset which contains a column SRC.  SRC is a number which I use later to get counts and sums for the records in a distinct list. 

declare@startdate date = '2014-04-01'
declare@enddate date = '2014-05-01'
; with SM as
(
SELECT --ROW_NUMBER() OVER (PARTITION BY u.SRC ORDER BY u.SRC) As Row,
u.SRC,

[Code] ....

-- If Referral start date is between our requested dates

ref.Referral_Start_Date between @startdate and @enddate

OR

-- Include referrals which started before our requested date, but are still active during our date range.

(ref.Referral_Start_Date < @startdate and (ref.Referral_End_Date > @startdate OR ref.Referral_End_Date IS NULL ))
)
INNER JOIN c_sdt s on s.Service_Delivery_Type_Id = u.Service_Delivery_Type_Id
AND s.Service_Delivery_Unit_Id = 200
)
SELECT
count(distinct (case SRC when 91 then client_number else 0 end)) As Eligable_91,

[code]....

View 5 Replies View Related

Distinct Row Count In A Table.

Aug 30, 2006

Hi,

I want a count of distinct rows in a table through a single query -- is it possible?

eg.

table-

create table ch1 (a int, b int, c int, d int)

insert ch1 values (1,1,1,1)
insert ch1 values (2,2,2,2)
insert ch1 values (1,1,1,1)
insert ch1 values (2,2,2,2)
insert ch1 values (1,3,4,5)

Here distinct row count in a table is 3 which I want to achieve thro a query.

if I do

select count(distinct a) from ch1 it works fine and gives me output as 2.

but this is not working

select count(distinct a,b,c,d) from ch1 - any workaround to find the distinct row count in a table??

Please reply.

Cheers!
Ram.

View 7 Replies View Related

Transact SQL :: How To Get Count From A Large Table

Jun 1, 2015

I have a table with a couple hundred billion records (sql server 2005). When I do a select count(*) from tblx -- it takes this side of forever. Is it possible to count partitions and then add them up to make it faster?  

How I could improve the performance for count(*) of this huge table.  Note:  if the partition idea sounds viable -- what would that look like? 

View 11 Replies View Related

Transact SQL :: How To Get Distinct Count Of All Columns Of A Table

Dec 3, 2015

I have a table called Employee which have 6 columns. This table are having 1000 records. Now I want to know the distinct value count of all these 6 columns as well as normal count. like this:

ColumnName DistinctCount NormalCount
Id 1000 1000
Name 1000 1000
Phone 600 600
PINCode 200 1000
City 400 1000
Gender 2 1000

View 5 Replies View Related

How To Get Distinct Records In Table

Dec 26, 2013

For examble:

In a table name like

1.Test
2.Test.
3.Test..
4.Test...

How will get the name "Test" using distinct keyword?

View 1 Replies View Related

How To Get Distinct Records In Table

Dec 25, 2013

For examble:

In a table name like

1.Test
2.Test.
3.Test..
4.Test...

How will get the name "Test" using distinct keyword?

View 2 Replies View Related

T-SQL (SS2K8) :: How To Get Distinct Records From Table

Apr 14, 2014

Suppose I have 2 table

1)Main
2)History

Main table maintain all the records having columns MAIN_SKU,DEDUCTIBLE_AMT,model_id,catagory,ModifiedDate

IF DEDUCTIBLE_AMT is changes it will place entry in history table ,columns are same with history_id

i want to display distinct main_sku from history table(all columns) with last DEDUCTIBLE_AMT changed from history table

table structure
main table
MAIN_SKUDEDUCTIBLE_amtmodel_idcatagory
1100100phone
2150101phone
3200109smartphone
4100202smartphone
History table
History_idMAIN_SKUDEDUCTIBLE_amtmodel_idcatagoryModifiedDate
11150100phone4/14/2014
21200101phone4/13/2014
34109202smartphone4/14/2014
44101202smartphone4/13/2014
52200101phone4/13/2014
63100109smartphone4/12/2014

View 3 Replies View Related

How To Copy Distinct Records From One Table To Another Using SQL 2005

Oct 24, 2007

I have a table on an offline database having let say 2000 rows and same table on an online database with let say 3000 rows. Now I want to copy just those records (1000) from online database that are not in the offline database table. Also, both tables have same name, same schema etc.

View 3 Replies View Related

Complexity Problem With Large Amount Of Records In A Link Table

Feb 1, 2008

A friend reminded me of a problem we tried to solve a few years ago and were unsuccessful.  Below is a copy of the email he sent me.  We would very much appreciate any ideas from the community.  Thanks!Lets start with a simple schema where you have 4
tables:

View 3 Replies View Related

SQL Server 2012 :: How To Quickly Update / Insert 3M Records In Large Table

Mar 28, 2015

Our system runs a SQL Server 2012 DB, it has a table (table_a) which has over 10M records. Our system have to receive data file from previous system daily which contains approximate 3M updated or new records for table_a. My job is to update table_a with the new data.

The initial solution is:

1 Create a table (table_b) which structur is as the same as table_a

2 Use BCP to import updated records into table_b

3 Remove outdated data in table_a:
delete from table_a inner join table_b on table_a.key_fileds = table_b.key_fields

4 Append updated or new data into table_a:
insert into table_a select * from table_b

As the test result, this solution is very inefficient. Step 3 costs several hours, e.g. How can I improve it?

View 9 Replies View Related

Group / Union Statement - Pull Unique Records From A Large Table

Sep 22, 2014

I am trying to use SQL to pull unique records from a large table. The table consists of people with in and out dates. Some people have duplicate entries with the same IN and OUT dates, others have duplicate IN dates but sometimes are missing an OUT date, and some don’t have an IN date but have an OUT date.

What I need to do is pull a report of all Unique Names with Unique IN and OUT dates (and not pull duplicate IN and OUT dates based on the Name).

I have tried 2 statements:

#1:
SELECT DISTINCT tblTable1.Name, tblTable1.INDate
FROM tblTable1
WHERE (((tblTable1.Priority)="high") AND ((tblTable1.ReportDate)>#12/27/2013#))
GROUP BY tblTable1.Name, tblTable1.INDate
ORDER BY tblTable1.Name;

#2:
SELECT DISTINCT tblTable1.Name, tblTable1.INDate
FROM tblTable1
WHERE (((tblTable1.Priority)="high") AND ((tblTable1.ReportDate)>#12/27/2013#))
UNION SELECT DISTINCT tblTable1.Name, tblTable1.INDate
FROM tblTable1
WHERE (((tblTable1.Priority)="high") AND ((tblTable1.ReportDate)>#12/27/2013#));

Both of these work great… until I the OUT date. Once it starts to pull the outdate, it also pulls all those who have a duplicate IN date but the OUT date is missing.

Example:

NameINOUT
John Smith1/1/20141/2/2014
John Smith1/1/2014(blank)

I am very new to SQL and I am pretty sure I am missing something very simple… Is there a statement that can filter to ensure no duplicates appear on the query?

View 1 Replies View Related

Display Count Of Records From The Same Table &> 1

Dec 27, 2007

How would I list the users in the users table that have duplicate IDs or count of IDs > 1?The UserName field is unique.
State   UserName     First          Last         ID         City         CountTX       Kkeaton         Kathryn     Keaton     1001     Dallas      2TX       KakiKeaton    Kathryn     Keaton     1001     Dallas      2I think I have to use a subselect? If I use group by then it won't show both records. It shows only one of them.Thanks
Craig

View 4 Replies View Related

Count For Varchar Field - How To Get Distinct Count

Jul 3, 2013

I am trying to get count on a varchar field, but it is not giving me distinct count. How can I do that? This is what I have....

Select Distinct
sum(isnull(cast([Total Count] as float),0))

from T_Status_Report
where Type = 'LastMonth' and OrderVal = '1'

View 9 Replies View Related

TSQL - Count The Number Of Records In A Table

Sep 24, 2007

Hi guys,
Is there any function to get the total number of records in a specific table?
(SQL SERVER 2000).
Thanks in advance,
Aldo.

View 3 Replies View Related

Simple Count Function - Show All Records For SSN In A Table

Jul 9, 2013

I want my query to list all SSNS that have more than one record in the table. I have this query:

Code:

SELECT SSN, name4, count(*) from [1099_PER]
group by SSN, name4
having count(SSN) > 1

It does retrieve the right SSNS and tells me how many times the SSN occurs in the table. However, I want my query results to display their full records.

For example

SSN NAME4 COUNT
123445555 WALTER - 4

I want the query to show me all four records for this SSN. I thought removing the count field would do this, but it still gives me only one instance of each SSN.

View 6 Replies View Related

SQL 2012 :: Count Records From Each Separate Data Table Corresponding To Feed

Nov 5, 2014

I have 1 table that is just a list of feeds. A, B, C, D etc (15 rows in total) and each feed has other information attached to it such as Full name, location etc etc. I am only interested in the Feed column.

Each feed then has a corresponding data table which contains a list of records. Eg Feed A data is contained in TableA, Feed B data is contains in TableB and so on.

Basically what I need is a combined table that shows the list of Feeds in the 1st Column ( So A, B, C, D…..) and then a second column which counts the records from each separate data table corresponding to that feed.

So the end result would look something like this:

Feed------No of Records
A----------4 (from TableA)
B----------7 (from TableB)
C----------8 (from TableC)
D----------1 (from TableD)

Possible?

View 2 Replies View Related

MDX: Count Records In Table A That Are Not In Table B

May 23, 2007

Hello all,



I would like to use MDX to identify "contracts" which are in a table A and not in a table B, and count them to have a total per "company".

It should be something like:



Contract / Flag contracts not in table B (1=yes, 0=not)

---------------------------------------------------------------

Contract 1 1

Contract 2 0

Contract 3 1

Contract 4 1

Contract 5 0

Contract 6 0

TOTAL 3



So far I know how to distinguish at contract level which records belong to table A and not table B, but I do not have the total for the company.



In SQL it should be something like that:

SELECT Company,count(*)

FROM tableA

left join tableB

on tableA.Contract = tableB.Contract

WHERE tableB.Contract is null

GROUP BY Company



Thanks,

Guillaume

View 3 Replies View Related

Using Count With Distinct

Oct 23, 2001

hi!
i am trying to get a count of 3 distinct values , one of them being a datetime - am running into errors - any suggestions are appreciated.


select count(distinct individualid + intakeseq + exitdate)from #temp

i am trying to do a distinct on individualid + intakeseq + exitdate,
& then get their count.

Thanks!

View 3 Replies View Related

Count From Distinct

Mar 14, 2007

Hello

for mS SQL 2000

with

SELECT DISTINCT Name, Region
FROM Groups
GROUP BY Name, Region

I get 254 rows

but how can I get the COUNT only ?

something like

SELECT COUNT(SELECT DISTINCT Name, Region
FROM Groups
GROUP BY Name, Region) AS CPT FROM Groups

i must get 254

thank you for helpings

View 4 Replies View Related

Count Distinct

Mar 23, 2007

hi all,

sometimes, when tables are poorly joined, we have exactly same rows as a result.. so we use distinct.. but how can we count(*) the distinct rows?


select distinct d.docrefid from tbljobDocuments d
left join tbljobitems j on j.docrefid=d.docrefid
where d.docrefid='DC01C06027' and j.itemid='39J1667-H86700'

will return me

docrefid
DC01C06027

but when i try to count(*) only the distinct record, it count all..

select distinct d.docrefid, count(*) from tbljobDocuments d
left join tbljobitems j on j.docrefid=d.docrefid
where d.docrefid='DC01C06027' and j.itemid='39J1667-H86700'
group by d.docrefid

will return me

DocrefId Count
DC01C060272

when i expect

DocrefId Count
DC01C060271




~~~Focus on problem, not solution~~~

View 3 Replies View Related

DISTINCT AND COUNT

Nov 9, 2007

Hi,

I have write down following query ..

select a.patientid, a.providerid
from PatientDrugList a
JOIN Admin..Loadinstance b on a.intLoadInstanceId = b.intLoadInstanceId
where a.MailReasonTypeId = -1
and b.SubClientId = 22
and b.StatusTypeId > 0



Now how could I get distinct patientid count and distinct providerid count


Reply soon

View 2 Replies View Related

Distinct Count

Jul 20, 2006

Hi



I have a table which stores the shift information for employees. The table contains 10 columns as Employeename,Employeeno,month,year,shifttimings etc. If an employee works a day in a particular shift, then a row will be inserted in to the above table for that employee.

Now at the end of the month i wanted to calculate the shift details for each employee for a particular month of a given year like employeename,employeeno, noofdays(countof shiftdays).

Can some body help?



Thanks in Advance!

Santhosh



View 4 Replies View Related

SELECTing A DISTINCT COUNT?

Nov 30, 2003

I have an SQL statement that looks like the following:

SELECT [Docs-Entities].entityID, entityName, COUNT([Docs-DocsEntities].filename) AS numDocs
FROM [Docs-Entities] LEFT JOIN [Docs-DocsEntities]
ON [Docs-Entities].entityID = [Docs-DocsEntities].entityID
GROUP BY [Docs-Entities].entityID, entityName

but the problem is that numDocs (the COUNT) is not returning a distinct count. In the DocEntities table, a particular document can actually have multiple entries with the same entityID so that produces inflated numbers for numDocs. But when I do a SELECT DISTINCT on a particular entityID, the results are less and don't match the numDocs number because I only need to list the document one time. This is not a huge issue, but it looks bad on my site.

Is there a way that I can make COUNT count distinctly?

Thanks for the help and I hope I worded that cllearly...

View 2 Replies View Related

COUNT(DISTINCT) With ROLLUP

Aug 17, 2005

I'm struggling to fin a way to use DISTINCT keyword with ROLLUP (or Cube).For example,SELECT employee_city, employee_country, COUNT(DISTINCT employee_name)FROM employeeGROUP BY employee_city, employee_country WITH ROLLUPthat query does not work.Is there a workaround?Thx.

View 2 Replies View Related

Distinct Count(fieldname)

Jan 28, 1999

hi, I have a table that contain(custname,ordno,ordchange,ordlocation,reaso n)
I want to determine the following:
how many orders per customer,how many orders changed per customer,how many order location per customer, what reason for change per customer,

here what I wrote as a query, tell me if I am right. I thank you for your help

select custname, COUNT(DISTINCT ordno) ,
COUNT(DISTINCT ordchange) ,
COUNT(DISTINCT ordlocation) ,
COUNT( reason)
from customers a, orders b
where a.custname =b.custname
group by custname

View 1 Replies View Related

SELECT COUNT Of DISTINCT

Jul 19, 2006

helloi want to do only one query for :SELECT DISTINCT Name FROM UsersSELECT COUNT(Name) AS Names FROM Users WHERE (Name LIKE 'xxx')something like :SELECT Name, COUNT(Name) AS Names FROM Users WHERE Name IN (SELECT DISTINCT Name FROM Users)i must get :Joe 23julie 17.....thank you

View 3 Replies View Related

Distinct Count Query

Dec 11, 2007

I have a table with following fields
tdate
custcode
prodcode


table is filled with full year data and i want following result

I want count of distinct custcode in every past three months.

for example
Result like this

month tjan tfeb tmar tapr tmay ..... tdec

prod1
prod2
.
.
prod5

And data under tmar should be count of distinct custcode of (jan,feb and mar) for corresponding prod code is required.
Under tapr, count of distinct custcode of (feb,mar and apr) for corresponding prod code is required.

Can any1 help me please.

I am using MS SQL 2005 and above table is a big table (approx 10 million records)

Sham

View 13 Replies View Related

I Get An Error When I Use COUNT(DISTINCT Colum_name)

Oct 14, 2007

hello,
i have a working stored procedure SELECT CommentID, UserName, PictureID, DateCommented, COUNT(CommentID) OVER (PARTITION BY PictureID) AS 'NrOfComments'
FROM Comments WHERE PictureID = @PictureID
witch returns among others the number of comments for a picture
i need to select the number of distinct users who commented that user too, so i added this at SELECT statement, COUNT(DISTINCT UserName) AS 'Expr1'
i get that error:
"Colum 'Comments.CommentID' is invalig in the select list beacuse it is not contained in either an aggregate function or the GROUP BY clause."
what should i do, to select the number of distinct users who commented on a specific picture?
here are the table rows, if you need that
CommentID --- UserName --- PictureID --- DateCommented
please help me, thank you

View 5 Replies View Related

Distinct Count, But Rewritten With Sum(case..)

Mar 28, 2007

It doesn't seem possibly, but maybe?
Is there a way to have an expression be used, but also benefit from using distinct on a column?

I'm looking for something like:
sum(case when dtEntered > '1-1-2006' then 1 else 0 end)
but also encorporating somehow a distinct count on UserName. So a username showing twice would only count once, and this would only be counted if the record's dtEntered date was greater than Jan 1, 2006.

The reason I'm writing the statement that way is because there are 5 columns which aggregate data by different time periods.

If it's not possible, I will just end up joining to the table multiple times, putting the date filter in the where clause.

View 3 Replies View Related

Ms Sql Select Count Distinct Probrem

Jul 12, 2004

I am trying to get the below query to work and can't seem to get past this error. Not sure but everywhere I look I think my syntax is right. I am probably missing something obvious but I'm just back from vacation and trying to get in the swing of things. Any help would be appreciated.


Select Distinct(chadcd) as Adjustor, Count (Distinct chclno) as NumberOfCases, Count (Distinct chclno,chwkno)as NumberofClauses
from clmhdr
where chpddt = '20040630'
Group By chadcd
Order by Adjustor

Server: Msg 170, Level 15, State 1, Line 1
Line 1: Incorrect syntax near ',chwkno'.

View 1 Replies View Related

Using Count To Find # Of Distinct Entries

Apr 17, 2008

I need a query to return the number of distinct call_num(s) between two timestamps.

For example search between 2008-04-17 05:00:00.000 and
2008-04-19 05:00:00.000

Calls
----------
id_tag(unique id) | status | call_num | entry_id(timestamp)

1 HOLD 0123456789 2008-04-17 05:07:00.080
2 ONSCENE 012345679 2008-04-17 05:10:00.012
3 ENROUTE 321654987 2008-04-19 04:00:00.000

so the sample answer would be : 2

View 2 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved