SQL重复记录查询(转载) ,及对大数据量的反复数据优化

SQL重复记录查询(转载) ,及对大数据量的重复数据优化

1、查找表中多余的重复记录，重复记录是根据单个字段（peopleId）来判断
select * from people
where peopleId in (select peopleId from people group by peopleId having count(peopleId) > 1)

例二：select * from testtablewhere numeber in (select number from people group by number having count(number) > 1 )可以查出testtable表中number相同的记录

2、删除表中多余的重复记录，重复记录是根据单个字段（peopleId）来判断，只留有rowid最小的记录
delete from people
where peopleId in (select peopleId from people group by peopleId having count(peopleId) > 1)
and rowid not in (select min(rowid) from people group by peopleId having count(peopleId )>1)

3、查找表中多余的重复记录（多个字段）
select * from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)

4、删除表中多余的重复记录（多个字段），只留有rowid最小的记录
delete from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)
and rowid not in (select min(rowid) from vitae group by peopleId,seq having count(*)>1)


5、查找表中多余的重复记录（多个字段），不包含rowid最小的记录
select * from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)
and rowid not in (select min(rowid) from vitae group by peopleId,seq having count(*)>1)

(二)
比方说
在A表中存在一个字段“name”，
而且不同记录之间的“name”值有可能会相同，
现在就是需要查询出在该表中的各记录之间，“name”值存在重复的项；
Select Name,Count(*) From A Group By Name Having Count(*) > 1

如果还查性别也相同大则如下:
Select Name,sex,Count(*) From A Group By Name,sex Having Count(*) > 1


(三)
方法一

declare @max integer,@id integer

declare cur_rows cursor local for select 主字段,count(*) from 表名 group by 主字段 having count(*) >； 1

open cur_rows

fetch cur_rows into @id,@max

while @@fetch_status=0

begin

select @max = @max -1

set rowcount @max

delete from 表名 where 主字段 = @id

fetch cur_rows into @id,@max

end

close cur_rows

set rowcount 0

方法二

　　有两个意义上的重复记录，一是完全重复的记录，也即所有字段均重复的记录，二是部分关键字段重复的记录，比如Name字段重复，而其他字段不一定重复或都重复可以忽略。

1、对于第一种重复，比较容易解决，使用

select distinct * from tableName

就可以得到无重复记录的结果集。

如果该表需要删除重复的记录（重复记录保留1条），可以按以下方法删除

select distinct * into #Tmp from tableName

drop table tableName

select * into tableName from #Tmp

drop table #Tmp

发生这种重复的原因是表设计不周产生的，增加唯一索引列即可解决。

2、这类重复问题通常要求保留重复记录中的第一条记录，操作方法如下

假设有重复的字段为Name,Address，要求得到这两个字段唯一的结果集

select identity(int,1,1) as autoID, * into #Tmp from tableName

select min(autoID) as autoID into #Tmp2 from #Tmp group by Name,autoID

select * from #Tmp where autoID in(select autoID from #tmp2)

最后一个select即得到了Name，Address不重复的结果集（但多了一个autoID字段，实际写时可以写在select子句中省去此列）

(四)
查询重复

select * from tablename where id in (

select id from tablename

group by id

having count(id) > 1

) ============================分割线,以下是优化SQl==============================================需要从表A中取得所有字段b相同的数据
如下，该SQL用于查询出数据库中某表的所有重复记录!

比如表A

—————————————————————— ID USER     PASS —————————————————————— 1 test1        a 2 test2        b 3 test3        c 4 test3        cc 5 test3        ccc 6 test4      d 7 test5      d 8 test5      e —————————————————————— 

?
用什么方法找出其中USER重复的数据，即

—————————————————————— ID USER     PASS —————————————————————— 3    test3     c 4    test3     cc 5    test3     ccc 7    test5     d 8    test5     e 

?

这只是一个例子，如果表中的数据有几百万条，如何提高查询的效率

SQL语句写了很多种了（如上面所转载的），但速度都很慢。
例如如下SQL，在表中数据量很少时执行速度还是很快的，但是当数据量打到50W时，我至今没有等到它的返回结果....

select id,user,pass from A where user in ( select user  from A   group by user having count(user)>1); 

?
方法一：再ORACLE的环境下
可以考虑用EXISTS条件语句代替IN语句 （速度有所提升）

SELECT A1.ID, A1.USER, A1.PASS    FROM A A1   WHERE EXISTS   (       SELECT 1       FROM A A2      WHERE A1.USER = A2.USER      AND A1.ID <> A2.ID   )  ----------------------SELECT A1.ID, A1.USER, A1.PASS FROM A A1 WHERE EXISTS (     SELECT 1     FROM A A2     WHERE A1.USER = A2.USER     AND A1.ID <> A2.ID ) 

?



方法二：如果是BATCH或者SP，可以考虑作一个如下的中间表B

CREATE TABLE B   AS    SELECT     USER, COUNT(*) AS USER_COUNT   FROM A   GROUP BY USER    HAVING COUNT(*) > 1   -------------------------------------ALTER TABLE B    ADD PRIMARY KEY (USER)   ------------------------------------CREATE TABLE B AS SELECT     USER, COUNT(*) AS USER_COUNT FROM A GROUP BY USER HAVING COUNT(*) > 1 ---------------------------------------ALTER TABLE B   ADD PRIMARY KEY (USER) 

?

然后用以下SQL进行检索

SELECT A.*   FROM A   INNER JOIN B ON  A.USER = B.USER  

这条执行的效率怎样？

select user             from A             group by user having count(user)>1  select user           from A           group by user having count(user)>1 

?
还有重复的几率有多大，如果超过一半的话，优化死也优化不到哪里去

方法三：可以用left join试试（mysql测试通过），把重复记录做为主表left join 表?

select  A.id, A.user, A.pass    from         (select user                from A               group by user having count(user)>1          ) AS B left join A on B.user=A.user  

?
这几种方法中，根据不同的环境进行优化了，如果重复记录比较大的情况下，推荐使用临时表，但有时增加临时表又需要进行维护，第三种方法left join也是很好的选择了

查看更多 下一篇