将 2000 万个实体推送到 Java map 对象中的最佳方法是什么？

1. 如果不使用多线程，大约需要 40 秒。
2. 使用 ForkJoinPool 大约需要 25 秒，我创建了 2 个任务，每个任务都推送 1000 万个实体

我相信这两项任务都在两个不同的核心中运行。 问题:当我创建 1 个推送 1000 万条数据的任务时，需要大约 9 秒，然后当运行 2 个任务，其中每个任务推送 1000 万条数据时，为什么需要大约 26 秒？我做错了什么吗？

插入 20 M 数据需要不到 10 秒的时间是否有不同的解决方案？

最佳答案

在没有看到您的代码的情况下，造成这些不良性能结果的最可能原因是垃圾回收 Activity 。为了演示它，我编写了以下程序:

import java.lang.management.ManagementFactory;
import java.util.*;
import java.util.concurrent.*;

public class TestMap {
  // we assume NB_ENTITIES is divisible by NB_TASKS
  static final int NB_ENTITIES = 20_000_000, NB_TASKS = 2;
  static Map<String, String> map = new ConcurrentHashMap<>();

  public static void main(String[] args) {
    try {
      System.out.printf("running with nb entities = %,d, nb tasks = %,d, VM args = %s%n", NB_ENTITIES, NB_TASKS, ManagementFactory.getRuntimeMXBean().getInputArguments());
      ExecutorService executor = Executors.newFixedThreadPool(NB_TASKS);
      int entitiesPerTask = NB_ENTITIES / NB_TASKS;
      List<Future<?>> futures = new ArrayList<>(NB_TASKS);
      long startTime = System.nanoTime();
      for (int i=0; i<NB_TASKS; i++) {
        MyTask task = new MyTask(i * entitiesPerTask, (i + 1) * entitiesPerTask - 1);
        futures.add(executor.submit(task));
      }
      for (Future<?> f: futures) {
        f.get();
      }
      long elapsed = System.nanoTime() - startTime;
      executor.shutdownNow();
      System.gc();
      Runtime rt = Runtime.getRuntime();
      long usedMemory = rt.maxMemory() - rt.freeMemory();
      System.out.printf("processing completed in %,d ms, usedMemory after GC = %,d bytes%n", elapsed/1_000_000L, usedMemory);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

  static class MyTask implements Runnable {
    private final int startIdx, endIdx;

    public MyTask(final int startIdx, final int endIdx) {
      this.startIdx = startIdx;
      this.endIdx = endIdx;
    }

    @Override
    public void run() {
      long startTime = System.nanoTime();
      for (int i=startIdx; i<=endIdx; i++) {
        map.put("sambit:rout:" + i, "C:\\Images\\Provision_Images");
      }
      long elapsed = System.nanoTime() - startTime;
      System.out.printf("task[%,d - %,d], completed in %,d ms%n", startIdx, endIdx, elapsed/1_000_000L);
    }
  }
}

在处理结束时，此代码通过执行 System.gc() 紧跟 Runtime.maxMemory() - Runtime.freeMemory( )。这表明具有 2000 万个条目的 map 大约需要不到 2.2 GB，这是相当大的。我用 1 个和 2 个线程运行它，对于 -Xmx 和 -Xms JVM 参数的各种值，这里是结果输出(只是为了清楚:2560m = 2.5g):

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms2560m, -Xmx2560m]
task[0 - 19,999,999], completed in 11,781 ms
processing completed in 11,782 ms, usedMemory after GC = 2,379,068,760 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms2560m, -Xmx2560m]
task[0 - 9,999,999], completed in 8,269 ms
task[10,000,000 - 19,999,999], completed in 12,385 ms
processing completed in 12,386 ms, usedMemory after GC = 2,379,069,480 bytes

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms3g, -Xmx3g]
task[0 - 19,999,999], completed in 12,525 ms
processing completed in 12,527 ms, usedMemory after GC = 2,398,339,944 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms3g, -Xmx3g]
task[0 - 9,999,999], completed in 12,220 ms
task[10,000,000 - 19,999,999], completed in 12,264 ms
processing completed in 12,265 ms, usedMemory after GC = 2,382,777,776 bytes

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms4g, -Xmx4g]
task[0 - 19,999,999], completed in 7,363 ms
processing completed in 7,364 ms, usedMemory after GC = 2,402,467,040 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms4g, -Xmx4g]
task[0 - 9,999,999], completed in 5,466 ms
task[10,000,000 - 19,999,999], completed in 5,511 ms
processing completed in 5,512 ms, usedMemory after GC = 2,381,821,576 bytes

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms8g, -Xmx8g]
task[0 - 19,999,999], completed in 7,778 ms
processing completed in 7,779 ms, usedMemory after GC = 2,438,159,312 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms8g, -Xmx8g]
task[0 - 9,999,999], completed in 5,739 ms
task[10,000,000 - 19,999,999], completed in 5,784 ms
processing completed in 5,785 ms, usedMemory after GC = 2,396,478,680 bytes

这些结果可以总结在下表中:

--------------------------------
heap      | exec time (ms) for: 
size (gb) | 1 thread | 2 threads
--------------------------------
2.5       |    11782 |     12386
3.0       |    12527 |     12265
4.0       |     7364 |      5512
8.0       |     7779 |      5785
--------------------------------

我还观察到，对于 2.5g 和 3g 堆大小，由于 GC Activity ，CPU Activity 很高，在整个处理时间内达到 100% 的峰值，而对于 4g 和 8g，仅观察到由于 System.gc() 调用，在最后。

总结:

1. 如果您的堆大小不合适，垃圾回收将扼杀您希望获得的任何性能提升。您应该使其足够大，以避免长时间 GC 暂停的副作用。

2. 您还必须意识到，使用诸如 ConcurrentHashMap 之类的并发集合会产生显着的性能开销。为了说明这一点，我稍微修改了代码，以便每个任务都使用自己的 HashMap，然后在最后将所有映射聚合(使用 Map.putAll())在第一个任务的 map 。处理时间下降到3200ms左右

关于java - 并发 : Java Map，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/45372992/