草庐IT

ElasticSearch单字段查询去重详解

IT之一小佬 2023-08-27 原文

1、SQL去重

在SQL中，用dinstinct语句进行去重：

  • 获取去重后的结果：SELECT DISTINCT name, sex FROM person;
  • 统计去重后的数量：SELECT COUNT(DISTINCT name, sex) FROM person;

2、ES数据构建

2.1 创建索引

from elasticsearch import Elasticsearch

# 连接es
es = Elasticsearch(hosts=["192.168.124.49:9200"], sniffer_timeout=60, timeout=30)

body = {
    "mappings": {
        "properties": {
            "id": {
                "type": "integer"
            },
            "name": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            },
            "age": {
                "type": "integer"
            },
            "gender": {
                "type": "keyword"
            },
            "email": {
                "type": "text"
            },
            "province": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            },
            "address": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            },
            "state": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            }
        }
    }
}

# 创建 index
es.indices.create(index="person_info", body=body)

2.2 查看索引

2.3 使用kibana批量生成数据

POST person_info/_bulk
{"index": {"_index": "person_info"}}
{"id": 1, "name": "刘一", "age": 25, "gender": "男", "email": "111@qq.com", "provience": "北京", "address": "北京市朝阳区", "status": "正常"}
{"index": {"_index": "person_info"}}
{"id": 1, "name": "陈二", "age": 26, "gender": "女", "email": "111@qq.com", "provience": "山东", "address": "山东省青岛市", "status": "正常"}
{"index": {"_index": "person_info"}}
{"id": 1, "name": "张三", "age": 27, "gender": "男", "email": "111@qq.com", "provience": "北京", "address": "北京市朝阳区", "status": "正常"}
{"index": {"_index": "person_info"}}
{"id": 1, "name": "李四", "age": 28, "gender": "男", "email": "111@qq.com", "provience": "山东", "address": "山东省济南市", "status": "正常"}
{"index": {"_index": "person_info"}}
{"id": 1, "name": "王五", "age": 25, "gender": "男", "email": "111@qq.com", "provience": "北京", "address": "北京市朝阳区", "status": "正常"}
{"index": {"_index": "person_info"}}
{"id": 1, "name": "刘一", "age": 26, "gender": "男", "email": "111@qq.com", "provience": "山东", "address": "山东省青岛市", "status": "正常"}
{"index": {"_index": "person_info"}}
{"id": 1, "name": "陈二", "age": 26, "gender": "女", "email": "111@qq.com", "provience": "北京", "address": "北京市朝阳区", "status": "正常"}

2.4 查看生成的数据

3、ES获取去重结果

3.1 collapse折叠功能(ES5.3之后支持)

  • 推荐。原因：性能高，占内存小

注意：去重的字段不能是text类型。如果xxxfield的mapping要有keyword，且通过xxxfield.keyword去重。

注意：如果去重字段是其他可以直接去重的类型，比如：数字类型、keyword、日期等，则直接用字段名就可以。即：如果本处xxxfield是keyword，则xxxfield.keyword处写成xxxfield就行。

查询province为北京的信息：

GET person_info/_search
{
  "query": {
    "match": {
      "provience.keyword": "北京"
    }
  }
}

运行结果：

{
  "took" : 15,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 0.5753642,
    "hits" : [
      {
        "_index" : "person_info",
        "_type" : "_doc",
        "_id" : "hFHKl4YBPv2uoOpTcHMg",
        "_score" : 0.5753642,
        "_source" : {
          "id" : 1,
          "name" : "刘一",
          "age" : 25,
          "gender" : "男",
          "email" : "111@qq.com",
          "provience" : "北京",
          "address" : "北京市朝阳区",
          "status" : "正常"
        }
      },
      {
        "_index" : "person_info",
        "_type" : "_doc",
        "_id" : "hlHKl4YBPv2uoOpTcHMi",
        "_score" : 0.5753642,
        "_source" : {
          "id" : 1,
          "name" : "张三",
          "age" : 27,
          "gender" : "男",
          "email" : "111@qq.com",
          "provience" : "北京",
          "address" : "北京市朝阳区",
          "status" : "正常"
        }
      },
      {
        "_index" : "person_info",
        "_type" : "_doc",
        "_id" : "iFHKl4YBPv2uoOpTcHMi",
        "_score" : 0.5753642,
        "_source" : {
          "id" : 1,
          "name" : "王五",
          "age" : 25,
          "gender" : "男",
          "email" : "111@qq.com",
          "provience" : "北京",
          "address" : "北京市朝阳区",
          "status" : "正常"
        }
      },
      {
        "_index" : "person_info",
        "_type" : "_doc",
        "_id" : "ilHKl4YBPv2uoOpTcHMi",
        "_score" : 0.5753642,
        "_source" : {
          "id" : 1,
          "name" : "陈二",
          "age" : 26,
          "gender" : "女",
          "email" : "111@qq.com",
          "provience" : "北京",
          "address" : "北京市朝阳区",
          "status" : "正常"
        }
      }
    ]
  }
}

查询province为北京，且根据年龄去重的信息：

# collapse获取去重结果
GET person_info/_search
{
  "query": {
    "match": {
      "provience.keyword": "北京"
    }
  },
  "collapse": {
    "field": "age"
  }
}

运行结果：

{
  "took" : 14,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "person_info",
        "_type" : "_doc",
        "_id" : "hFHKl4YBPv2uoOpTcHMg",
        "_score" : 0.5753642,
        "_source" : {
          "id" : 1,
          "name" : "刘一",
          "age" : 25,
          "gender" : "男",
          "email" : "111@qq.com",
          "provience" : "北京",
          "address" : "北京市朝阳区",
          "status" : "正常"
        },
        "fields" : {
          "age" : [
            25
          ]
        }
      },
      {
        "_index" : "person_info",
        "_type" : "_doc",
        "_id" : "hlHKl4YBPv2uoOpTcHMi",
        "_score" : 0.5753642,
        "_source" : {
          "id" : 1,
          "name" : "张三",
          "age" : 27,
          "gender" : "男",
          "email" : "111@qq.com",
          "provience" : "北京",
          "address" : "北京市朝阳区",
          "status" : "正常"
        },
        "fields" : {
          "age" : [
            27
          ]
        }
      },
      {
        "_index" : "person_info",
        "_type" : "_doc",
        "_id" : "ilHKl4YBPv2uoOpTcHMi",
        "_score" : 0.5753642,
        "_source" : {
          "id" : 1,
          "name" : "陈二",
          "age" : 26,
          "gender" : "女",
          "email" : "111@qq.com",
          "provience" : "北京",
          "address" : "北京市朝阳区",
          "status" : "正常"
        },
        "fields" : {
          "age" : [
            26
          ]
        }
      }
    ]
  }
}

3.2 字段聚合+top_hits聚合

  • 不推荐。原因：性能差，占内存大

查询province为北京，且根据年龄去重的信息：

# 聚合获取去重结果
GET person_info/_search
{
  "query": {
    "match": {
      "provience.keyword": "北京"
    }
  },
  "size": 0,
  "aggs": {
    "age_aggs": {
      "terms": {
        "field": "age",
        "size": 10
      },
      "aggs": {
        "age_top": {
          "top_hits": {
            "sort": [{
              "age": {
                "order": "desc"
              }
            }], 
            "size": 1
          }
        }
      }
    }
  }
}

运行结果：

{
  "took" : 230,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "age_aggs" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : 25,
          "doc_count" : 2,
          "age_top" : {
            "hits" : {
              "total" : {
                "value" : 2,
                "relation" : "eq"
              },
              "max_score" : null,
              "hits" : [
                {
                  "_index" : "person_info",
                  "_type" : "_doc",
                  "_id" : "hFHKl4YBPv2uoOpTcHMg",
                  "_score" : null,
                  "_source" : {
                    "id" : 1,
                    "name" : "刘一",
                    "age" : 25,
                    "gender" : "男",
                    "email" : "111@qq.com",
                    "provience" : "北京",
                    "address" : "北京市朝阳区",
                    "status" : "正常"
                  },
                  "sort" : [
                    25
                  ]
                }
              ]
            }
          }
        },
        {
          "key" : 26,
          "doc_count" : 1,
          "age_top" : {
            "hits" : {
              "total" : {
                "value" : 1,
                "relation" : "eq"
              },
              "max_score" : null,
              "hits" : [
                {
                  "_index" : "person_info",
                  "_type" : "_doc",
                  "_id" : "ilHKl4YBPv2uoOpTcHMi",
                  "_score" : null,
                  "_source" : {
                    "id" : 1,
                    "name" : "陈二",
                    "age" : 26,
                    "gender" : "女",
                    "email" : "111@qq.com",
                    "provience" : "北京",
                    "address" : "北京市朝阳区",
                    "status" : "正常"
                  },
                  "sort" : [
                    26
                  ]
                }
              ]
            }
          }
        },
        {
          "key" : 27,
          "doc_count" : 1,
          "age_top" : {
            "hits" : {
              "total" : {
                "value" : 1,
                "relation" : "eq"
              },
              "max_score" : null,
              "hits" : [
                {
                  "_index" : "person_info",
                  "_type" : "_doc",
                  "_id" : "hlHKl4YBPv2uoOpTcHMi",
                  "_score" : null,
                  "_source" : {
                    "id" : 1,
                    "name" : "张三",
                    "age" : 27,
                    "gender" : "男",
                    "email" : "111@qq.com",
                    "provience" : "北京",
                    "address" : "北京市朝阳区",
                    "status" : "正常"
                  },
                  "sort" : [
                    27
                  ]
                }
              ]
            }
          }
        }
      ]
    }
  }
}

4、ES统计去重后的数量

  • 聚合+cardinality聚合函数

查询province为北京，且根据年龄去重的数量：

# 聚合获取去重数量
GET person_info/_search
{
  "query": {
    "match": {
      "provience.keyword": "北京"
    }
  },
  "size": 0,
  "aggs": {
    "age_aggs": {
      "cardinality": {
        "field": "age"
      }
    }
  }
}

运行结果：

{
  "took" : 68,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "age_aggs" : {
      "value" : 3
    }
  }
}

参考博文：

ElasticSearch--去重查询/根据字段去重--方法/实例_IT利刃出鞘的博客-CSDN博客_elasticsearch统计去重后的数量准确值

单字ElasticSearch34person_infoproviencecardinalitycollapseaggregationtop_hits

有关ElasticSearch单字段查询去重详解的更多相关文章

  1. ruby - ECONNRESET (Whois::ConnectionError) - 尝试在 Ruby 中查询 Whois 时出错 - 2

    我正在用Ruby编写一个简单的程序来检查域列表是否被占用。基本上它循环遍历列表，并使用以下函数进行检查。require'rubygems'require'whois'defcheck_domain(domain)c=Whois::Client.newc.query("google.com").available?end程序不断出错(即使我在google.com中进行硬编码)，并打印以下消息。鉴于该程序非常简单，我已经没有什么想法了-有什么建议吗？/Library/Ruby/Gems/1.8/gems/whois-2.0.2/lib/whois/server/adapters/base.

  2. ruby-on-rails - 在 Rails 和 ActiveRecord 中查询时忽略某些字段 - 2

    我知道我可以指定某些字段来使用pluck查询数据库。ids=Item.where('due_at但是我想知道，是否有一种方法可以指定我想避免从数据库查询的某些字段。某种反拔？posts=Post.where(published:true).do_not_lookup(:enormous_field) 最佳答案 Model#attribute_names应该返回列/属性数组。您可以排除其中一些并传递给pluck或select方法。像这样:posts=Post.where(published:true).select(Post.attr

  3. sql - 查询忽略时间戳日期的时间范围 - 2

    我正在尝试查询我的Rails数据库(Postgres)中的购买表，我想查询时间范围。例如，我想知道在所有日期的下午2点到3点之间进行了多少次购买。此表中有一个created_at列，但我不知道如何在不搜索特定日期的情况下完成此操作。我试过:Purchases.where("created_atBETWEEN?and?",Time.now-1.hour,Time.now)但这最终只会搜索今天与那些时间的日期。 最佳答案 您需要使用PostgreSQL'sdate_part/extractfunction从created_at中提取小时

  4. ruby-on-rails - solr 清理查询 - 2

    我在Rails上使用带有ruby​​的solr。一切正常，我只需要知道是否有任何现有代码来清理用户输入，比如以?开头的查询。或* 最佳答案 我不知道执行此操作的任何代码，但理论上可以通过查看parsingcodeinLucene来完成并搜索thrownewParseException(只有16个匹配!)。在实践中，我认为您最好只捕获代码中的任何solr异常并显示“无效查询”消息或类似信息。编辑:这里有几个“sanitizer”:http://pivotallabs.com/users/zach/blog/articles/937-s

  5. ruby-on-rails - Rails 3 在一个查询中包含多个表 - 2

    我正在为锦标赛开发一个Rails应用程序。我在这个查询中使用了三个模型:classPlayertruehas_and_belongs_to_many:tournamentsclassTournament:destroyclassPlayerMatch"Player",:foreign_key=>"player_one"belongs_to:player_two,:class_name=>"Player",:foreign_key=>"player_two"在tournaments_controller的显示操作中，我调用以下查询:Tournament.where(:id=>params

  6. ruby-on-rails - Sunspot:如何对具有不同值的多个字段进行全文查询？ - 2

    我想用sunspot重现以下原始solr查询q=exact_term_text:fooORterm_textv:foo*ORalternate_text:bar*但我无法通过标准的太阳黑子界面理解这是否可能以及如何实现，因为看起来:fulltext方法似乎不接受多个文本/搜索字段参数我不知道将什么参数作为第一个参数传递给fulltext，就好像我通过了"foo"或"bar"结果不匹配如果我传递一个空参数，我得到一个q=*:*范围过滤器(例如with(:term).starting_with('foo*')(顾名思义)作为过滤器查询应用，因此不参与评分。似乎可以手动编写字符串(或者可能使

  7. ruby-on-rails - 在不重新查询数据库的情况下重新排序 Rails 中的事件记录？ - 2

    例如，假设我有一个名为Products的模型，并且在ProductsController中，我有以下代码用于product_listView以显示已排序的产品。@products=Product.order(params[:order_by])让我们想象一下，在product_listView中，用户可以使用下拉菜单按价格、评级、重量等进行排序。数据库中的产品不会经常更改。我很难理解的是，每次用户选择新的order_by过滤器时，rails是否必须查询，或者rails是否能够以某种方式缓存事件记录以在服务器端重新排序？有没有一种方法可以编写它，以便在用户排序时rails不会重新查询结果

  8. ruby-on-rails - 带句点(或句号)的 Rails 查询字符串。 - 2

    我目前正在尝试了解RoR。我将两个字符串传递到我的Controller中。一个是随机的十六进制字符串，另一个是电子邮件。该项目用于对数据库进行简单的电子邮件验证。我遇到的问题是当我输入如下内容来测试我的页面时:http://signup.testsite.local/confirm/da2fdbb49cf32c6848b0aba0f80fb78c/bob.villa@gmailcom我在:email的参数散列中得到的全部是'bob'。我在gmail和com之间留下了.，因为那样会导致匹配根本不起作用。我的路由匹配如下:match"confirm/:code/:email"=>"conf

  9. ruby - Rails Elasticsearch 聚合 - 2

    不知何故，我似乎无法获得包含我的聚合的响应...使用curl它按预期工作:HBZUMB01$curl-XPOST"http://localhost:9200/contents/_search"-d'{"size":0,"aggs":{"sport_count":{"value_count":{"field":"dwid"}}}}'我收到回复:{"took":4,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":90,"max_score":0.0,"hits":[]},"a

  10. ruby - 如何将编码的查询值添加到 URL？ - 2

    我正在寻找一种方便实用的方法来将编码值添加到Ruby中的URL查询字符串。目前，我有:require'open-uri'u=URI::HTTP.new("http",nil,"mydomain.example",nil,nil,"/tv",nil,"show="+URI::encode("Rosie&Jim"),nil)pu.to_s#=>"http://mydomain.example/tv?show=Rosie%20&%20Jim"这不是我要找的，因为我需要得到“http://mydomain.example/tv?show=Rosie%20%26%20Jim”，这样show=值就

随机推荐