Tiny bug fix: The size of an excluded directory entry itself should not
be counted, either. This is consistent with what you'd expect: A cache
directory with thousands of files can easily take up several megabytes
for the dir entry - but from the perspective of a backup system that
recognizes cache dirs - the dir is empty, and therefore shouldn't take
any extra space at all.
Use a macro instead of the global constant `cachedir_tag_signature`.
Use `memcmp` instead of `strncmp`.
Add `has_cachedir_tag` to exclude.h.
(See http://dev.yorhel.nl/ncdu/bug/30)
I realized that I used addparentstats() with negative values when
removing stuff, so it had to be done this way (without rewriting
everything). It's a simple solution, anyway.
This mostly avoids the issue of getting negative sizes. It's still
possible to get a negative size after refresh or deletion, I'll get to
that in a bit.
They should now be able to cope with file sizes in the full (positive)
range of a signed integer, i.e. 0 bytes to 8 EiB minus one byte. The
size calculation of directories, however, may still overflow and cause
negative integers to be passed around. That should be fixed.
It's kinda annoying how you need to confirm the messages. I'd prefer
having some status bar where messages are automatically removed after a
short timeout or something. But that's more work, and for the few
cases where feedback is necessary this'll work fine, too.
Some measurements importing a gzip-compressed file (zcat .. | ncdu -f -)
containing a bit under 6 million items and a few choices of how often to
call input_handle():
Called on every item:
real 0m13.745s
user 0m12.576s
sys 0m4.566s
Called on every 8 items:
real 0m7.932s
user 0m9.636s
sys 0m1.623s
Called on every 16 items:
real 0m7.559s
user 0m9.553s
sys 0m1.323s
Called on every 32 items:
real 0m7.279s
user 0m9.353s
sys 0m1.277s
Called on every 64 items:
real 0m7.166s
user 0m9.389s
sys 0m1.117s
Called on every 256 items:
real 0m7.073s
user 0m9.439s
sys 0m1.027s
32 seemed like a good compromise.
- errors in item() didn't propagate properly
- empty [] and {} values weren't allowed
- fractional numbers weren't allowed
- parsing of escaped characters didn't ensure that enough data was in
the buffer
- E() didn't propagate errors properly in all cases
I'll do some more testing later on, but the current code seems to be
working quite well already.
This is a quick hack to fix http://dev.yorhel.nl/ncdu/bug/18.
path_real() is both rather unreadable, fragile, hard to maintain and a
bit inefficient. It wouldn't be too surprising if I introduced a new bug
with this commit...
*makes a note to do a proper rewrite of that code later on*
!WARNING! The export option is experimental, and the file format is not
final. I make no promise that a future version of ncdu will be able to
read the current format. There's also quite a few TODO's left.
There used to be four bytes of padding in the struct on systems with
32bit pointers. Moving the pointers down so that they are in between the
64bit and 32bit fields means that there'll never be any padding.
There may, however, be some extra padding at the end of the struct to
make the size a multiple of 8. Since we use the name field as a sort of
"flexible array member", we don't particularly care about that padding
and just want to allocate enough memory to hold the struct and the name
field. offsetof() allows us to do that without relying on compiler
support for flexible array members.
POSIX defines ino_t to be of an unsigned integer type, and searching
around the net didn't tell me of any definitions conflicting that. So
every ino_t can be represented in an uint64_t. (Assuming that is the
largest integer type in use for an inode number, but I'm sure that
assumption will hold for a while)
(dev_t, on the other hand, is a bit messier. Still figuring out what to
do with that.)
2 billion files should be enough for everyone. You probably won't have
enough memory to scan such a filesystem. int is a better choice than
long, as sizeof(int) is 4 on pretty much any system where ncdu runs.
This allows scanning stuff without initializing ncurses. Not too useful
at this point since ncdu will switch to an ncurses environment when it's
done anyway, but this will become more useful when the export-to-file
feature has been implemented.
The architecture is explained in dir.h. The reasons for these changes is
two-fold:
- calc.c was too complex, it simply did too many things. 399ccdeb is a
nice example of that: Should have been an easy fix, but it introduced
a segfault (fixed in 0b49021a), and added a small memory leak.
- This architecture features a pluggable input/output system, which
should make a file export/import feature relatively simple.
The current commit does not feature any user interface, so there's no
feedback yet when scanning a directory. I'll get to that in a bit.
I've also not tested the new scanning code very well yet, so I might
have introduced some bugs.
This fixes a bug where ncdu would stop scanning a directory if the
terminal window has been resized to a small enough space that the
warning would show up.
This should be a *significant* performance increase when scanning a
directory that has many hard links.
I used the khash library written by Attractive Chaos[1]. This library
fits perfectly into ncdu's "use as little memory as possible but still
try to be very fast"-policy. It's API is somewhat quircky in use, but I
guess that is to blame to the lack of generic programming support in C.
Blog: http://attractivechaos.wordpress.com/
Lib: https://github.com/attractivechaos/klib/blob/master/khash.h
This used to be the default before 1.5, but for some reason the default
changed in 1.5 and 1.6. Changing it back now, because the graph really
is useful, and there's still enough space for the filename even in
smaller terminals.
This solution is far cleaner. Thanks to Ben North for pointing me to the
*-width-specifier that has apparently been built into the printf-family
functions for, well, quite a while, it seems.
The memory for this format is now statically allocated as well. I
was under the impression its size would depend on wincols, but this is
the format we're talking about, the string does not have to hold the
actual line contents. I must have been sleeping again...
Oh well, this is a slight performance improvement, although it doesn't
seem the be the cause of the browing slowness when running under
valgrind. (Obviously running ncdu with valgrind is supposed to be
slower, but the current performance is rather bad...)
Rather than storing a pointer to another memory allocation in the
struct. This saves some memory and improves performance by significantly
decreasing the number of calls to [c|m]alloc() and free().
Here is the new multi-page listing functionality I promised in
5db9c2aea1.
It may look very easy, but getting this to work right wasn't,
unfortunately.
This optimizes a few actions (though not all), and makes the code easier
to understand and expand.
The behaviour of the browser has changed a bit with regards to
multi-page listings. Personally I don't like this change much, so I'd
probably fix that later on.
The displayed directory sizes are now fully correct, although in its
current state it's not all that intuitive because:
directory size != sum(size of all files and subdirectories)
This should probably be fixed later on by splitting the sizes into a
shared and non-shared part.
Also, the sizes displayed after a recalculation or deletion are
incorrect, I'll fix this later on.
The directory sizes are now incorrect as hard links will be counted
twice again (as if there wasn't any detection in the first place), but
this will get fixed by adding a shared size field.
This method of keeping track of hard links is a lot faster and allows
adding an interface which lists the found links.
When interrupinting the calculation process by pressing 'q' while
it's looping through a directory, or when a directory could be openend
but not chdir()'ed into, closedir() wasn't called.
Setting FF_BSEL after calling browse_init() causes two items to be
selected, as browse_init() makes sure something will be selected,
while calc_process() assumes nothing is, because the previously
selected item had just been deleted.
Hard link detection is now done in a separate pass on the in-memory tree,
and duplicates can be 'removed' and 're-added' on the fly. When making any
changes in the tree, all hard links are re-added before the operation and
removed again afterwards.
While this guarantees that all hard link information is correct, it does
have a few drawbacks. I can currently think of two:
1. It's not the most efficient way to do it, and may be quite slow on
large trees. Will have to do some benchmarks later to see whether
it is anything to be concerned about.
2. The first encountered item is considered as 'counted' and all items
encountered after that are considered as 'duplicate'. Because the
order in which we traverse the tree doesn't always have to be the
same, the items that will be considered as 'duplicate' can vary with
each deletion or re-calculation. This might cause confusion for
people who aren't aware of how hard links work.
Reproducable with the following steps:
- Make sure the first item in the list is a directory starting with a dot
- Make sure the next item is a normal directory
- Select first item
- Press 'h' to hide it, next item will properly be selected
- Open selected directory (right arrow), and see how the first (hidden)
directory was opened.
It's amazing how I've even found this bug, considering that really is the
only way to reproduce it...
