As aluded to in the previous commit. This approach keeps track of hard
links information much the same way as ncdu 1.16, with the main
difference being that the actual /counting/ of hard link sizes is
deferred until the scan is complete, thus allowing the use of a more
efficient algorithm and amortizing the counting costs.
As an additional benefit, the links listing in the information window
now doesn't need a full scan through the in-memory tree anymore.
A few memory usage benchmarks:
1.16 2.0-beta1 this commit
root: 429 162 164
backup: 3969 1686 1601
many links: 155 194 106
many links2*: 155 602 106
(I'm surprised my backup dir had enough hard links for this to be an
improvement)
(* this is the same as the "many links" benchmarks, but with a few
parent directories added to increase the tree depth. 2.0-beta1 doesn't
like that at all)
Performance-wise, refresh and delete operations can still be improved a
bit.
While this simplifies the code a bit, it's a regression in the sense
that it increases memory use.
This commit is yak shaving for another hard link counting approach I'd
like to try out, which should be a *LOT* less memory hungry compared to
the current approach. Even though it does, indeed, add an extra cost of
these parent node pointers.
I had planned to checkout out async functions here so I could avoid
recursing onto the stack alltogether, but it's still unclear to me how
to safely call into libc from async functions so let's wait for all that
to get fleshed out a bit more.
Sticking to "compiletime-known" error types will essentially just bring
in *every* possible error anyway, so might as well take advantage of
@errorName.
This complicated the scan code more than I had anticipated and has a
few inherent bugs with respect to calculating shared hardlink sizes.
Still, the merge approach avoids creating a full copy of the subtree, so
that's another memory usage related win compared to the C version.
On the other hand, it does leak memory if nodes can't be reused.
Not quite as well tested as I should have, so I'm sure there's bugs.
Two differences compared to the C version:
- You can now select individual paths in the listing, pressing enter
will open the selected path in the browser window.
- Creating this listing is much slower and requires, in the worst case,
a full traversal through the in-memory tree. I've tested this without
the same-dev and shared-parent optimizations (i.e. worst case) on an
import with 30M files and performance was still quite acceptable - the
listing completed in a second - so I didn't bother adding a loading
indicator. On slower systems and even larger trees this may be a
little annoying, though.
(also, calling nonl() apparently breaks detection of the return key,
neither \n nor KEY_ENTER are emitted for some reason)
The good news is: apart from this little thing, everything seems to just
work(tm) on FreeBSD. Think I had more trouble with C because of minor
header file differences.
I had used them as a HashSet with mutable keys already in order to avoid
padding problems. This is not always necessary anymore now that Zig's
new HashMap uses separate arrays for keys and values, but I still need
the HashSet trick for the link_count nodes table, as the key itself
would otherwise have padding.
Under the assumption that there are no external references to files
mentioned in the dump, i.e. a file's nlink count matches the number of
times the file occurs in the dump.
This machinery could also be used for regular scans, when you want to
scan an individual directory without caring about external hard links.
Maybe that should be the default, even? Not sure...
In a similar way to the C version of ncdu: by wrapping malloc(). It's
simpler to handle allocation failures at the source to allow for easy
retries, pushing the retries up the stack will complicate code somewhat
more. Likewise, this is a best-effort approach to handling OOM,
allocation failures in ncurses aren't handled and display glitches may
occur when we get an OOM inside a drawing function.
This is a somewhat un-Zig-like way of handling errors and adds
scary-looking 'catch unreachable's all over the code, but that's okay.
Performance is looking great, but the code is rather ugly and
potentially buggy. Also doesn't handle hard links without an "nlink"
field yet.
Error handling of the import code is different from what I've been doing
until now. That's intentional, I'll change error handling of other
pieces to call ui.die() directly rather than propagating error enums.
The approach is less testable but conceptually simpler, it's perfectly
fine for a tiny application like ncdu.
I plan to add more display options, but ran out of keys to bind.
Probably going for a quick-select menu thingy so that we can keep the
old key bindings for people accustomed to it.
The graph width algorithm is slightly different, but I think this one's
a minor improvement.
Now we're getting somewhere. This works surprisingly well, too. Existing
ncdu behavior is to remember which entry was previously selected but not
which entry was displayed at the top, so the view would be slightly
different when switching directories. This new approach remembers both
the entry and the offset.
I initially wanted to keep a directory's block count and size as a
separate field so that exporting an in-memory tree to a JSON dump would
be easier to do, but that doesn't seem like a common operation to
optimize for. We'll probably need the algorithms to subtract sub-items
from directory counts anyway, so such an export can still be
implemented, albeit slower.
libc locale-dependent APIs are pure madness, but I can't avoid them as
long as I use ncurses. libtickit seems like a much saner alternative (at
first glance), but no popular application seems to use it. :(
Eaiser to implement now that we're linking against libc.
But exclude pattern matching is extremely slow, so that should really be
rewritten with a custom fnmatch implementation. It's exactly as slow as
in ncdu 1.x as well, I'm surprised nobody's complained about it yet.
And while I'm at it, supporting .gitignore-style patterns would be
pretty neat, too.
I tried playing with zbox (pure Zig termbox-like lib) for a bit, but I
don't think I want to have to deal with the terminal support issues that
will inevitably come with it. I already stumbled upon one myself: it
doesn't properly put the terminal in a sensible state after cleanup in
tmux. As much as I dislike ncurses, it /is/ ubiquitous and tends to kind
of work.
The new data model is supposed to solve a few problems with ncdu 1.x's
'struct dir':
- Reduce memory overhead,
- Fix extremely slow counting of hard links in some scenarios
(issue #121)
- Add support for counting 'shared' data with other directories
(issue #36)
Quick memory usage comparison of my root directory with ~3.5 million
files (normal / extended mode):
ncdu 1.15.1: 379M / 451M
new (unaligned): 145M / 178M
new (aligned): 155M / 200M
There's still a /lot/ of to-do's left before this is usable, however,
and there's a bunch of issues I haven't really decided on yet, such as
which TUI library to use.
Backporting this data model to the C version of ncdu is also possible,
but somewhat painful. Let's first see how far I get with Zig.
Reduces memory by a tiny bit. Arguably we never needed tombstones
because entries are never removed, so there shouldn't be any performance
hit there. We don't even need a 'used' flag either, considering that can
be represented by a NULL value, but I'm not really up for
implementing/modifying my own hash table.
ref: https://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/
This is a best-effort approach to save ncdu state when memory is low.
There's likely allocation in libraries that isn't being checked
(ncurses, printf).
Fixes#132 (it actually doesn't, that needs a 64bit static binary too,
but I'll get to that)
This allocation is currently leaked, but as long as we don't allocate
new ones for each refresh, that shouldn't be much of an issue.
(cherry picked from commit 9dc2d32a8f)
This adds an 'm' command to show the latest modified time of all files
in a directory. The 'M' command allows for ascending and descending
mtime sorting. These are only enabled with the -e flag and overload
the dir_ext mtime field.
I had taken care to not sort empty directories during dirlist_open(),
but forgot that manual user actions can still cause dirlist_set_sort()
to be called, which does not handle empty directories.
Reported by Alex Wilson.
'char' may be unsigned on some architectures, which will cause the
"overflow check" on decrement to fail.
This would at most result in a confusing UI issue where no confirmation
option appears to be selected.
Unfortunately, there wasn't a single bit free in struct dir.flags, so I
had to increase its size to 16 bit. This commit is just the initial
preparation, there's still a few things to do:
- Add "extended information" cli flag to enable/disable this
functionality.
- Export and import extended information when requested
- Do something with the data.
I also did a few memory measurements on a file list with 12769842 items:
before this commit: 1.239 GiB
without extended info: 1.318 GiB
with extended info: 1.698 GiB
It's surprising what adding a single byte to a struct can do to the
memory usage. :(
Fixes https://dev.yorhel.nl/ncdu/bug/103
I don't think a stack overflow as a result of recursion is exploitable
on a modern system. It should just result in an unfortunate write to a
page that is not writable, followed by a crash.
I've decided not to use ls-like file name coloring for now, instead just
coloring the difference between a (regular) file and a dir.
Still looking for a good color scheme for light backgrounds.
This should fix https://dev.yorhel.nl/ncdu/bug/99 - with the downside
that this requires a C99 compiler.
I also replaced all occurrences of static allocation of struct dir with
use dynamic allocation, because I wasn't really sure if static
allocation of flexible structs is allowed. In the case of dirlist.c the
dynamic allocation is likely required anyway, because it does store a
few bytes in the name field.
TODO:
- Add (ls-like) colors to the actual file names
-> Implement full $LS_COLORS handling or something simple and custom?
- Test on a white/black terminal, and provide an alternate color scheme
if necessary.
- Make colors opt-in?
Check if the environment variable NCDU_SHELL is defined before the SHELL
variable is checked. This makes it possible to specify a program to
execute when 'b' is pressed. Setting SHELL to for example "mc" (Midnight
Commander) didn't work because mc already uses SHELL to execute
commands.
The check for the system() exit status is slightly problematic, because
bash returns the status code of the last command it executed. I've set
it to only check for status code 127 now (command not found) in order to
at least provide a message when the $SHELL command can't be found. This
error can still be triggered when executing a nonexistant command within
the shell and then exiting.
Key 'b' in the browse window spawns a shell in the current directoy.
We first check the $SHELL environment variable of the user for the preferred
shell interpreter. If it's not set, we fall back to the compile time
configured default shell (usually /bin/bash).
Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com>
Turns out that being able to open an empty directory actually has its
uses:
- If you delete the last file in a directory, you now won't be directed
to the parent directory anymore. This allows keeping 'd' pressed
without worrying that you'll delete stuff outside of the current dir.
(This is the primary motivation for doing this)
- You can now scan and later refresh an empty directory, as suggested by
#2 in http://dev.yorhel.nl/ncdu/bug/15
Tiny bug fix: The size of an excluded directory entry itself should not
be counted, either. This is consistent with what you'd expect: A cache
directory with thousands of files can easily take up several megabytes
for the dir entry - but from the perspective of a backup system that
recognizes cache dirs - the dir is empty, and therefore shouldn't take
any extra space at all.
Use a macro instead of the global constant `cachedir_tag_signature`.
Use `memcmp` instead of `strncmp`.
Add `has_cachedir_tag` to exclude.h.
(See http://dev.yorhel.nl/ncdu/bug/30)
I realized that I used addparentstats() with negative values when
removing stuff, so it had to be done this way (without rewriting
everything). It's a simple solution, anyway.
This mostly avoids the issue of getting negative sizes. It's still
possible to get a negative size after refresh or deletion, I'll get to
that in a bit.
They should now be able to cope with file sizes in the full (positive)
range of a signed integer, i.e. 0 bytes to 8 EiB minus one byte. The
size calculation of directories, however, may still overflow and cause
negative integers to be passed around. That should be fixed.
It's kinda annoying how you need to confirm the messages. I'd prefer
having some status bar where messages are automatically removed after a
short timeout or something. But that's more work, and for the few
cases where feedback is necessary this'll work fine, too.
Some measurements importing a gzip-compressed file (zcat .. | ncdu -f -)
containing a bit under 6 million items and a few choices of how often to
call input_handle():
Called on every item:
real 0m13.745s
user 0m12.576s
sys 0m4.566s
Called on every 8 items:
real 0m7.932s
user 0m9.636s
sys 0m1.623s
Called on every 16 items:
real 0m7.559s
user 0m9.553s
sys 0m1.323s
Called on every 32 items:
real 0m7.279s
user 0m9.353s
sys 0m1.277s
Called on every 64 items:
real 0m7.166s
user 0m9.389s
sys 0m1.117s
Called on every 256 items:
real 0m7.073s
user 0m9.439s
sys 0m1.027s
32 seemed like a good compromise.
- errors in item() didn't propagate properly
- empty [] and {} values weren't allowed
- fractional numbers weren't allowed
- parsing of escaped characters didn't ensure that enough data was in
the buffer
- E() didn't propagate errors properly in all cases
I'll do some more testing later on, but the current code seems to be
working quite well already.
This is a quick hack to fix http://dev.yorhel.nl/ncdu/bug/18.
path_real() is both rather unreadable, fragile, hard to maintain and a
bit inefficient. It wouldn't be too surprising if I introduced a new bug
with this commit...
*makes a note to do a proper rewrite of that code later on*
!WARNING! The export option is experimental, and the file format is not
final. I make no promise that a future version of ncdu will be able to
read the current format. There's also quite a few TODO's left.
There used to be four bytes of padding in the struct on systems with
32bit pointers. Moving the pointers down so that they are in between the
64bit and 32bit fields means that there'll never be any padding.
There may, however, be some extra padding at the end of the struct to
make the size a multiple of 8. Since we use the name field as a sort of
"flexible array member", we don't particularly care about that padding
and just want to allocate enough memory to hold the struct and the name
field. offsetof() allows us to do that without relying on compiler
support for flexible array members.
POSIX defines ino_t to be of an unsigned integer type, and searching
around the net didn't tell me of any definitions conflicting that. So
every ino_t can be represented in an uint64_t. (Assuming that is the
largest integer type in use for an inode number, but I'm sure that
assumption will hold for a while)
(dev_t, on the other hand, is a bit messier. Still figuring out what to
do with that.)
2 billion files should be enough for everyone. You probably won't have
enough memory to scan such a filesystem. int is a better choice than
long, as sizeof(int) is 4 on pretty much any system where ncdu runs.
This allows scanning stuff without initializing ncurses. Not too useful
at this point since ncdu will switch to an ncurses environment when it's
done anyway, but this will become more useful when the export-to-file
feature has been implemented.
The architecture is explained in dir.h. The reasons for these changes is
two-fold:
- calc.c was too complex, it simply did too many things. 399ccdeb is a
nice example of that: Should have been an easy fix, but it introduced
a segfault (fixed in 0b49021a), and added a small memory leak.
- This architecture features a pluggable input/output system, which
should make a file export/import feature relatively simple.
The current commit does not feature any user interface, so there's no
feedback yet when scanning a directory. I'll get to that in a bit.
I've also not tested the new scanning code very well yet, so I might
have introduced some bugs.
This fixes a bug where ncdu would stop scanning a directory if the
terminal window has been resized to a small enough space that the
warning would show up.
This should be a *significant* performance increase when scanning a
directory that has many hard links.
I used the khash library written by Attractive Chaos[1]. This library
fits perfectly into ncdu's "use as little memory as possible but still
try to be very fast"-policy. It's API is somewhat quircky in use, but I
guess that is to blame to the lack of generic programming support in C.
Blog: http://attractivechaos.wordpress.com/
Lib: https://github.com/attractivechaos/klib/blob/master/khash.h
This used to be the default before 1.5, but for some reason the default
changed in 1.5 and 1.6. Changing it back now, because the graph really
is useful, and there's still enough space for the filename even in
smaller terminals.
This solution is far cleaner. Thanks to Ben North for pointing me to the
*-width-specifier that has apparently been built into the printf-family
functions for, well, quite a while, it seems.
The memory for this format is now statically allocated as well. I
was under the impression its size would depend on wincols, but this is
the format we're talking about, the string does not have to hold the
actual line contents. I must have been sleeping again...
Oh well, this is a slight performance improvement, although it doesn't
seem the be the cause of the browing slowness when running under
valgrind. (Obviously running ncdu with valgrind is supposed to be
slower, but the current performance is rather bad...)
Rather than storing a pointer to another memory allocation in the
struct. This saves some memory and improves performance by significantly
decreasing the number of calls to [c|m]alloc() and free().
Here is the new multi-page listing functionality I promised in
5db9c2aea1.
It may look very easy, but getting this to work right wasn't,
unfortunately.
This optimizes a few actions (though not all), and makes the code easier
to understand and expand.
The behaviour of the browser has changed a bit with regards to
multi-page listings. Personally I don't like this change much, so I'd
probably fix that later on.
The displayed directory sizes are now fully correct, although in its
current state it's not all that intuitive because:
directory size != sum(size of all files and subdirectories)
This should probably be fixed later on by splitting the sizes into a
shared and non-shared part.
Also, the sizes displayed after a recalculation or deletion are
incorrect, I'll fix this later on.
