造轮子的快乐！一个极快的纯 C 异步日志库

262 天前
 weidaizi

项目地址: https://github.com/MuggleWei/haclog

十一放假之前，看了一下 Nanolog 的论文，感觉思路非常赞，可惜 Nanolog 并不跨平台，而且为了性能和吞吐量，其日志输出并不直接可读。既然已经有了大体思路，接下来当然是自己撸一个啦！而且因为我也比较多 C 的库需要使用，所以就撸一个纯 C 的版本~
长假当中就断断续续写了一些，这几天在家的时间没事也写几段，今天大致到了较为满意的程度. 下面是在 Mac M1 2021 和 ThinkPad X1C 2019 上的 benchmark 结果 (测试程序可以直接在工程目录中找到： haclog gbenchmark)，欢迎各位老板们使用和提意见

Mac M1 2021

Run on (8 X 24 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB
  L1 Instruction 128 KiB
  L2 Unified 4096 KiB (x8)
Load Average: 8.88, 4.53, 2.84
-----------------------------------------------------------------------------------------------------------
Benchmark                                                                 Time             CPU   Iterations
-----------------------------------------------------------------------------------------------------------
ConstIterFixture/run/min_time:3.000/threads:1                          66.3 ns         66.2 ns     60803739
ConstIterFixture/run/min_time:3.000/threads:2                          34.0 ns         67.8 ns     58196532
ConstIterFixture/run/min_time:3.000/threads:4                          24.7 ns         96.0 ns     46114032
ConstIterFixture/run/min_time:3.000/threads:7                          30.0 ns          146 ns     25301248
ConstIterFixture/run/iterations:6000/repeats:5/threads:1                120 ns          120 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1               74.8 ns         74.8 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1               65.9 ns         66.0 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1               87.8 ns         87.8 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1               79.8 ns         80.0 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_mean          85.7 ns         85.7 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_median        79.8 ns         80.0 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_stddev        20.8 ns         20.7 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_cv           24.29 %         24.19 %             5
ConstIterFixture/run/iterations:6000/repeats:5/threads:4               25.6 ns          102 ns        24000
ConstIterFixture/run/iterations:6000/repeats:5/threads:4               23.0 ns         91.8 ns        24000
ConstIterFixture/run/iterations:6000/repeats:5/threads:4               28.5 ns          114 ns        24000
ConstIterFixture/run/iterations:6000/repeats:5/threads:4               27.8 ns          111 ns        24000
ConstIterFixture/run/iterations:6000/repeats:5/threads:4               28.6 ns          114 ns        24000
ConstIterFixture/run/iterations:6000/repeats:5/threads:4_mean          26.7 ns          107 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:4_median        27.8 ns          111 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:4_stddev        2.42 ns         9.71 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:4_cv            9.05 %          9.09 %             5
ConstIterFixture/run/iterations:6000/repeats:5/threads:8               16.7 ns          132 ns        48000
ConstIterFixture/run/iterations:6000/repeats:5/threads:8               17.5 ns          130 ns        48000
ConstIterFixture/run/iterations:6000/repeats:5/threads:8               17.5 ns          135 ns        48000
ConstIterFixture/run/iterations:6000/repeats:5/threads:8               16.8 ns          134 ns        48000
ConstIterFixture/run/iterations:6000/repeats:5/threads:8               16.9 ns          133 ns        48000
ConstIterFixture/run/iterations:6000/repeats:5/threads:8_mean          17.1 ns          133 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:8_median        16.9 ns          133 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:8_stddev       0.396 ns         1.64 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:8_cv            2.32 %          1.24 %             5
ConstIterFixture/run/iterations:6000/repeats:5/threads:16              7.82 ns          125 ns        96000
ConstIterFixture/run/iterations:6000/repeats:5/threads:16              8.64 ns          136 ns        96000
ConstIterFixture/run/iterations:6000/repeats:5/threads:16              13.2 ns          204 ns        96000
ConstIterFixture/run/iterations:6000/repeats:5/threads:16              7.25 ns          114 ns        96000
ConstIterFixture/run/iterations:6000/repeats:5/threads:16              11.0 ns          169 ns        96000
ConstIterFixture/run/iterations:6000/repeats:5/threads:16_mean         9.57 ns          149 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:16_median       8.64 ns          136 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:16_stddev       2.47 ns         36.7 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:16_cv          25.83 %         24.56 %             5

Thinkpad X1C 2019

Run on (12 X 1780.93 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 256 KiB (x6)
  L3 Unified 12288 KiB (x1)
Load Average: 0.09, 0.03, 0.08
-----------------------------------------------------------------------------------------------------------
Benchmark                                                                 Time             CPU   Iterations
-----------------------------------------------------------------------------------------------------------
ConstIterFixture/run/min_time:3.000/threads:1                          63.8 ns         63.7 ns     65480842
ConstIterFixture/run/min_time:3.000/threads:2                          35.3 ns         70.6 ns     57795198
ConstIterFixture/run/min_time:3.000/threads:6                          28.4 ns          170 ns     19480434
ConstIterFixture/run/min_time:3.000/threads:11                         2491 ns        27350 ns       370425
ConstIterFixture/run/iterations:6000/repeats:5/threads:1                144 ns          144 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1                145 ns          144 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1                144 ns          144 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1                137 ns          136 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1                128 ns          128 ns         6000
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_mean           140 ns          139 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_median         144 ns          144 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_stddev        7.14 ns         7.04 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:1_cv            5.11 %          5.06 %             5
ConstIterFixture/run/iterations:6000/repeats:5/threads:6               40.1 ns          241 ns        36000
ConstIterFixture/run/iterations:6000/repeats:5/threads:6               38.5 ns          231 ns        36000
ConstIterFixture/run/iterations:6000/repeats:5/threads:6               41.1 ns          246 ns        36000
ConstIterFixture/run/iterations:6000/repeats:5/threads:6               39.6 ns          238 ns        36000
ConstIterFixture/run/iterations:6000/repeats:5/threads:6               38.1 ns          228 ns        36000
ConstIterFixture/run/iterations:6000/repeats:5/threads:6_mean          39.5 ns          237 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:6_median        39.6 ns          238 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:6_stddev        1.20 ns         7.48 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:6_cv            3.03 %          3.16 %             5
ConstIterFixture/run/iterations:6000/repeats:5/threads:12              37.3 ns          446 ns        72000
ConstIterFixture/run/iterations:6000/repeats:5/threads:12              36.2 ns          434 ns        72000
ConstIterFixture/run/iterations:6000/repeats:5/threads:12              35.5 ns          426 ns        72000
ConstIterFixture/run/iterations:6000/repeats:5/threads:12              34.7 ns          416 ns        72000
ConstIterFixture/run/iterations:6000/repeats:5/threads:12              37.6 ns          450 ns        72000
ConstIterFixture/run/iterations:6000/repeats:5/threads:12_mean         36.3 ns          434 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:12_median       36.2 ns          434 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:12_stddev       1.21 ns         13.8 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:12_cv           3.34 %          3.19 %             5
ConstIterFixture/run/iterations:6000/repeats:5/threads:24              12.8 ns          307 ns       144000
ConstIterFixture/run/iterations:6000/repeats:5/threads:24              11.9 ns          286 ns       144000
ConstIterFixture/run/iterations:6000/repeats:5/threads:24              11.6 ns          277 ns       144000
ConstIterFixture/run/iterations:6000/repeats:5/threads:24              13.6 ns          326 ns       144000
ConstIterFixture/run/iterations:6000/repeats:5/threads:24              12.5 ns          300 ns       144000
ConstIterFixture/run/iterations:6000/repeats:5/threads:24_mean         12.5 ns          299 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:24_median       12.5 ns          300 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:24_stddev      0.783 ns         18.8 ns            5
ConstIterFixture/run/iterations:6000/repeats:5/threads:24_cv           6.27 %          6.28 %             5
3617 次点击
所在节点    程序员
29 条回复
icyalala
262 天前
我也喜欢 C 。
对于性能对比，建议画个直方图 📊 在 Readme 里，这对于宣传性能来说真的很重要。
weidaizi
262 天前
@icyalala 👍 感谢提醒！上面也有位朋友提到了可以横向对比一下，我之后准备另外开一个库，跑 C/C++ 的不同日志库的 benchmark 看看，这样也可以发现还有什么库是特别快的
bashbot
261 天前
挺好的轮子，之前也手撸过一个。
想问下 多队列环形缓冲区 能否保证多线程下日志的时间序？

之前为了保证时间序，写了一个可扩充的多生产单消费环形缓冲区。
但是在数据竞争上有点漏洞，导致多线程竞争时有偶发的丢日志，后来只能加锁搞定，性能下降了好多。
另外，运行时基于标记和级别的日志过滤，都没人用吗，看到好多日志库都没这些功能。
junmoxiao
261 天前
厉害呀大佬。可以加到 xmake-repo 的仓库里
weidaizi
261 天前
@bashbot 啊哈，只能保证同一线程的顺序，不能保证多线程的顺序；不过想了一下即使是单缓冲区，其实也是不能完全保证，除非是把加时间戳的代码段都包含在某个锁中，这个就太废了; 回到日志本身，因为带有时间戳，所以即使不同线程在日志文件中乱序，而且乱序也只是在一个消费者循环中，问题也不太大，得看使用场景权衡取舍吧
weidaizi
261 天前
@junmoxiao 谢谢！我平时不太用 xmake ，我之后去探索一下要如何加，哈~
Masoud2023
261 天前
昨天没事的时候简单的扫了一眼这个论文

我能不能简单的把这个理解为一个异步的带压缩的结构化 logger...
weidaizi
261 天前
@Masoud2023 是的，Nanolog 编译会生成一个 decompressor ，如果要以文本形式展示，得通过这程序解压一下日志~
weidaizi
257 天前
@hitmanx
@icyalala
感谢两位上周的建议，这几天我写了一个 C/C++ Benchmark 项目, 用于横向对比一些知名的或以低延时为目标的 C/C++ 日志库, 并图表化显示
地址: https://github.com/MuggleWei/cc_log_benchmark

这是一个专为移动设备优化的页面（即为了让你能够在 Google 搜索结果里秒开这个页面），如果你希望参与 V2EX 社区的讨论，你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/981311

V2EX 是创意工作者们的社区，是一个分享自己正在做的有趣事物、交流想法，可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX