问题描述

之前在ubuntu16上对Clickhouse21.12.1.1的基础上改了一些代码，近期把操作系统升级到unbutu 22.04后，发现编译不通过，具体报错为：

[ 22%] Linking CXX shared library libclickhouse_common_iod.so
/home/herry/dev/c++/ClickHouse/src/Common/ThreadFuzzer.cpp:293: error: undefined reference to '__pthread_mutex_lock'
/home/herry/dev/c++/ClickHouse/src/Common/ThreadFuzzer.cpp:293: error: undefined reference to '__pthread_mutex_unlock'

分析问题

没有链接pthread？

因为连接libckickhouse_common_iod.so时找不到__pthread_mutex_lock，首先考虑的是没有连接pthread，在src/CMakeLists.txt中添加
target_link_libraries(clickhouse_common_io PUBLIC pthread)
编译仍然提示同样的错误
用make VERBOSE=1可以看到libclickhouse_common_iod.so的链接过程中，已经有-lpthread了，看起来不是这个问题了。

查看pthread的符号表

在ubuntu16中

herry@herrypc:/lib/x86_64-linux-gnu$ nm -D libpthread.so.0  |grep pthread_mutex_lock
000000000000b650 T __pthread_mutex_lock@@GLIBC_2.2.5
000000000000b650 W pthread_mutex_lock@@GLIBC_2.2.5

可以看到pthread_mutex_lock和__pthread_mutex_lock都在libpthread.so.0中

在ubuntu22中

herry@inc:/lib/x86_64-linux-gnu$ nm -D libpthread.so.0  |grep pthread_mutex_lock
herry@inc:/lib/x86_64-linux-gnu$ 

herry@inc:/lib/x86_64-linux-gnu$ readelf -s libpthread.so.0 |grep pthread_mutex_
herry@inc:/lib/x86_64-linux-gnu$ 

发现libpthread.so.0中已经没有pthread_mutex_lock和__pthread_mutex_lock了。

glibc2.34 removed libpthread

参考文章[glibc2.34 removed libpthread
(https://developers.redhat.com/articles/2021/12/17/why-glibc-234-removed-libpthread#)。
从glibc2.34开始，glibc已经把pthread的功能基本都去掉了，剩下的libpthread.so基本上是个空的，只是为了让编译适配早期版本。
所以虽然我们有-lpthread，但是在其中仍然找不到__pthread_mutex_lock/unlock符号。

查看glibc版本中的内容

glibc版本号查看

方法1：

herry@inc:/lib/x86_64-linux-gnu$ ldd --version 
ldd (Ubuntu GLIBC 2.35-0ubuntu3) 2.35
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

方法2：

root@inc:/usr/lib/x86_64-linux-gnu# chmod +x libc.so.6
root@inc:/usr/lib/x86_64-linux-gnu# /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Ubuntu GLIBC 2.35-0ubuntu3) stable release version 2.35.
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 11.2.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.

两个方法都看出在ubuntu22.04上的glibc版本已经升级到2.35。

查看ubuntu22.04上的libc的符号列表
root@inc:/usr/lib/x86_64-linux-gnu# nm -D /lib/x86_64-linux-gnu/libc.so.6 |grep pthread_mutex_lock
0000000000097f70 T __pthread_mutex_lock@GLIBC_2.2.5
0000000000097f70 T pthread_mutex_lock@@GLIBC_2.2.5
果然发现了__pthread_mutex_lock符号。

解决问题

在clickhouse的问题列表中，找到了Fix undefined __pthread_mutex_lock/unlock for glibc，果然在新版本中，已经对该问题就是修复。

增加了针对glibc不同版本的适配代码。

/// Starting from glibc 2.34 there are no internal symbols without version,
/// so not __pthread_mutex_lock but __pthread_mutex_lock@2.2.5
#if defined(OS_LINUX) and !defined(USE_MUSL)
    /// You can get version from glibc/sysdeps/unix/sysv/linux/$ARCH/$BITS_OR_BYTE_ORDER/libc.abilist
    #if defined(__amd64__)
    #    define GLIBC_SYMVER "GLIBC_2.2.5"
    #elif defined(__aarch64__)
    #    define GLIBC_SYMVER "GLIBC_2.17"
    #elif defined(__riscv) && (__riscv_xlen == 64)
    #    define GLIBC_SYMVER "GLIBC_2.27"
    #elif (defined(__PPC64__) || defined(__powerpc64__)) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
    #    define GLIBC_SYMVER "GLIBC_2.17"
    #else
    #    error Your platform is not supported.
    #endif

    #define GLIBC_COMPAT_SYMBOL(func) __asm__(".symver " #func "," #func "@" GLIBC_SYMVER);

    GLIBC_COMPAT_SYMBOL(__pthread_mutex_unlock)
    GLIBC_COMPAT_SYMBOL(__pthread_mutex_lock)
#endif

把这段代码拷贝到我的工作代码中，编译，果然能够能够成功了。