Commit graph

19 commits

Author SHA1 Message Date
Yorhel
0e6967498f Support reading zstd-compressed json
Oddly enough this approach is slightly slower than just doing
`zstdcat x | ncdu -f-`, need to investigate.
2024-10-26 15:49:42 +02:00
Yorhel
8ad61e87c1 Stick with zstd-4 + 64k block, add --compress-level, fix 32bit build
And do dynamic buffer allocation for bin_export, removing 128k of
.rodata that I accidentally introduced earlier and reducing memory use
for parallel scans.

Static binaries now also include the minimal version of zstd, current
sizes for x86_64 are:

  582k ncdu-2.5
  601k ncdu-new-nocompress
  765k ncdu-new-zstd

That's not great, but also not awful. Even zlib or LZ4 would've resulted
in a 700k binary.
2024-08-03 13:16:44 +02:00
Yorhel
ebaa9b6a89 Add (temporary) compression support for the new export format
This is mainly for testing and benchmarking, I plan to choose a single
block size and compression library before release, to avoid bloating the
ncdu binary too much.

Currently this links against the system-provided zstd, zlib and lz4.
ncdubinexp.pl doesn't support compressed files yet.

Early benchmarks of `ncdu -f firefox-128.0.json` (407k files) with
different block sizes and compression options:

            bin8k        bin16k       bin32k       bin64k       bin128k      bin256k      bin512k      json
  algo      size  time   size  time   size  time   size  time   size  time   size  time   size  time   size  time

  none      16800  128   16760  126   16739  125   16728  124   16724  125   16722  124   16721  124   24835  127
  lz4        7844  143    7379  141    7033  140    6779  140    6689  138    6626  139    6597  139    5850  179

  zlib-1     6017  377    5681  310    5471  273    5345  262    5289  259    5257  256    5242  255    4415  164
  zlib-2     5843  386    5496  319    5273  284    5136  276    5072  271    5037  270    5020  268    4164  168
  zlib-3     5718  396    5361  339    5130  316    4977  321    4903  318    4862  324    4842  319    3976  196
  zlib-4     5536  424    5153  372    4903  341    4743  339    4665  338    4625  340    4606  336    3798  212
  zlib-5     5393  464    4993  419    4731  406    4561  414    4478  422    4434  426    4414  420    3583  261
  zlib-6     5322  516    4902  495    4628  507    4450  535    4364  558    4318  566    4297  564    3484  352
  zlib-7     5311  552    4881  559    4599  601    4417  656    4329  679    4282  696    4260  685    3393  473
  zlib-8     5305  588    4864  704    4568 1000    4374 1310    4280 1470    4230 1530    4206 1550    3315 1060
  zlib-9     5305  589    4864  704    4568 1030    4374 1360    4280 1510    4230 1600    4206 1620    3312 1230

  zstd-1     5845  177    5426  169    5215  165    5030  160    4921  156    4774  157    4788  153    3856  126
  zstd-2     5830  178    5424  170    5152  164    4963  161    4837  160    4595  162    4614  158    3820  134
  zstd-3     5683  187    5252  177    5017  172    4814  168    4674  169    4522  169    4446  170    3664  145
  zstd-4     5492  235    5056  230    4966  173    4765  170    4628  169    4368  222    4437  170    3656  145
  zstd-5     5430  270    4988  266    4815  234    4616  229    4485  224    4288  241    4258  223    3366  189
  zstd-6     5375  323    4928  322    4694  282    4481  279    4334  276    4231  275    4125  271    3234  235
  zstd-7     5322  400    4866  420    4678  319    4464  314    4315  312    4155  300    4078  295    3173  269
  zstd-8     5314  454    4848  689    4636  344    4420  346    4270  345    4137  350    4060  342    3082  330
  zstd-9     5320  567    4854  615    4596  392    4379  398    4228  401    4095  408    4060  345    3057  385
  zstd-10    5319  588    4852  662    4568  458    4350  466    4198  478    4066  491    4024  395    3005  489
  zstd-11    5310  975    4857 1040    4543  643    4318  688    4164  743    4030  803    3999  476    2967  627
  zstd-12    5171 1300    4692 1390    4539  699    4313  765    4154  854    4018  939    3999  478    2967  655
  zstd-13    5128 1760    4652 1880    4556 1070    4341 1130    4184 1230    3945 1490    3980  705    2932 1090
  zstd-14    5118 2040    4641 2180    4366 1540    4141 1620    3977 1780    3854 1810    3961  805    2893 1330

  mzstd-1    5845  206    5426  195    5215  188    5030  180    4921  176    4774  175    4788  172
  mzstd-2    5830  207    5424  196    5152  186    4963  183    4837  181    4765  178    4614  176
  mzstd-3    5830  207    5424  196    5150  187    4960  183    4831  180    4796  181    4626  180
  mzstd-4    5830  206    5427  196    5161  188    4987  185    4879  182    4714  180    4622  179
  mzstd-5    5430  347    4988  338    5161  189    4987  185    4879  181    4711  180    4620  180
  mzstd-6    5384  366    4939  359    4694  390    4481  391    4334  383    4231  399    4125  394
  mzstd-7    5328  413    4873  421    4694  390    4481  390    4334  385    4155  442    4078  435
  mzstd-8    5319  447    4854  577    4649  417    4434  421    4286  419    4155  440    4078  436
  mzstd-9    5349  386    4900  385    4606  469    4390  478    4241  478    4110  506    4078  436
  mzstd-10   5319  448    4853  597    4576  539    4360  560    4210  563    4079  597    4039  502
  mzstd-11   5430  349    4988  339    4606  468    4390  478    4241  478    4110  506    4013  590
  mzstd-12   5384  366    4939  361    4576  540    4360  556    4210  559    4079  597    4013  589
  mzstd-13   5349  387    4900  388    4694  390    4481  392    4334  386    4155  439    4078  436
  mzstd-14   5328  414    4873  420    4649  417    4434  424    4286  420    4155  444    4039  500

I'll need to do benchmarks on other directories, with hardlink support
and in extended mode as well to get more varied samples.

Another consideration in choosing a compression library is the size of
its implementation:

  zlib: 100k
  lz4:  106k
  zstd: 732k (regular), 165k (ZSTD_LIB_MINIFY, "mzstd" above)
2024-07-31 12:55:43 +02:00
Yorhel
a4484f27f3 Build: remove preferred_optimize_mode
Fixes #238
2024-04-25 14:15:46 +02:00
Yorhel
d60bcb2113 Copyright: remove year & use alias
Tired of bumping files every year and slowly moving stuff to my alias.
2024-04-20 15:49:51 +02:00
Yorhel
29bbab64b3 Update Zig requirement in README + set preferred build mode
+ minor irrelevant build system changes.
2024-04-20 15:40:53 +02:00
Eric Joldasov
5944b738d0
build.zig: update to Zig 0.12.0-dev.3643+10ff81c26
* LazyPath now stores `Build` owner inside in
 https://github.com/ziglang/zig/pull/19623 and
 https://github.com/ziglang/zig/pull/19597 .

Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
2024-04-12 12:52:22 +05:00
Eric Joldasov
c41e3f5828
build.zig: update to Zig 0.12.0-dev.2018+9a56228c2
* ZBS was reorganized around `Module` struct:
 https://www.github.com/ziglang/zig/pull/18160 .
 * Changes for ReleaseSafe: error return tracing is now off by default.

Signed-off-by: Eric Joldasov <bratishkaerik@landless-city.net>
2024-03-20 23:02:48 +05:00
Eric Joldasov
115de253a8
replace ncurses_refs.c workaround with pure Zig workaround
Signed-off-by: Eric Joldasov <bratishkaerik@getgoogleoff.me>
2023-11-19 14:37:52 +06:00
Eric Joldasov
ab6dc5be75
Update to Zig 0.11.0
Signed-off-by: Eric Joldasov <bratishkaerik@getgoogleoff.me>
2023-08-04 14:41:49 +06:00
Eric Joldasov
0d99781c67
build.zig: add option for building PIE
Might be useful for package maintainers.

Signed-off-by: Eric Joldasov <bratishkaerik@getgoogleoff.me>
2023-04-09 21:41:06 +06:00
Carlo Cabrera
07a13d9c73
Set headerpad_max_install_names on Darwin
This is useful for building binary distributions because it allows
references to library dependendencies on the build machine to be
rewritten appropriately upon installation on the user's machine.

Zig also does this in their `build.zig`:

    b52be973df/build.zig (L551-L554)
2023-02-22 13:51:08 +08:00
Yorhel
ec233ff33a Version 2.2.2 + copyright year bump 2023-01-19 08:00:27 +01:00
Yorhel
01f1e9188a Version 2.0.1 + copyright year bump 2022-01-01 16:01:47 +01:00
Yorhel
c8636b8982 Add REUSE-compliant copyright headers 2021-07-18 11:50:50 +02:00
Yorhel
e9c8d12c0f Store Ext before Entry
Which is slightly simpler and should provide a minor performance
improvement.
2021-07-16 19:13:04 +02:00
Yorhel
a54c10bffb More UI stuff: nice string handling/shortening + Zig bug workaround
libc locale-dependent APIs are pure madness, but I can't avoid them as
long as I use ncurses. libtickit seems like a much saner alternative (at
first glance), but no popular application seems to use it. :(
2021-05-05 08:03:27 +02:00
Yorhel
826c2fc067 Link to ncurses + some rudimentary TUI frameworky stuff
I tried playing with zbox (pure Zig termbox-like lib) for a bit, but I
don't think I want to have to deal with the terminal support issues that
will inevitably come with it. I already stumbled upon one myself: it
doesn't properly put the terminal in a sensible state after cleanup in
tmux. As much as I dislike ncurses, it /is/ ubiquitous and tends to kind
of work.
2021-05-03 08:01:18 +02:00
Yorhel
0783d35793 WIP: Experimenting with a rewrite to Zig & a new data model
The new data model is supposed to solve a few problems with ncdu 1.x's
'struct dir':
- Reduce memory overhead,
- Fix extremely slow counting of hard links in some scenarios
  (issue #121)
- Add support for counting 'shared' data with other directories
  (issue #36)

Quick memory usage comparison of my root directory with ~3.5 million
files (normal / extended mode):

  ncdu 1.15.1:     379M / 451M
  new (unaligned): 145M / 178M
  new (aligned):   155M / 200M

There's still a /lot/ of to-do's left before this is usable, however,
and there's a bunch of issues I haven't really decided on yet, such as
which TUI library to use.

Backporting this data model to the C version of ncdu is also possible,
but somewhat painful. Let's first see how far I get with Zig.
2021-04-29 12:48:52 +02:00