Why directory with large amounts of entries does not shrink in size after entries are removed?
Close vote notice: While the linked duplicate asks same question, the comments and the only answer do not provide an answer which draws from authoritative source. The answer on my question by Thomas Dickey, does in fact provide the answer I've been seeking. Thus the proposed duplicate isn't helpful, however Thomas's answer here is proper.
According to the answer by G-Man on Superuser question and personal account in this PerlDuck's comment, directories that have large amount of entries get to a size over 4096 bytes (which can be seen with ls -l output), but once the entries are removed - the number never goes down.
The question is "why" ? Is it due to how ext4 filesystem configured to retain directory metadata ? Obviously removing the directory and recreating it isn't a solution, since it deletes original inode and creates a new one. What can be done to decrease the number manually ?
directory ext4 inode
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
add a comment |
Close vote notice: While the linked duplicate asks same question, the comments and the only answer do not provide an answer which draws from authoritative source. The answer on my question by Thomas Dickey, does in fact provide the answer I've been seeking. Thus the proposed duplicate isn't helpful, however Thomas's answer here is proper.
According to the answer by G-Man on Superuser question and personal account in this PerlDuck's comment, directories that have large amount of entries get to a size over 4096 bytes (which can be seen with ls -l output), but once the entries are removed - the number never goes down.
The question is "why" ? Is it due to how ext4 filesystem configured to retain directory metadata ? Obviously removing the directory and recreating it isn't a solution, since it deletes original inode and creates a new one. What can be done to decrease the number manually ?
directory ext4 inode
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
not really - this question was specific enough to quote an authoritative source, unlike the suggested duplicate
– Thomas Dickey
13 hours ago
1
@Kusalananda I've edited the answer to include a rationale for why the linked duplicate isn't appropriate. I would suggest linking the other way around, ans Thomas's answer here does in fact answer the why of it for both mine and the other question
– Sergiy Kolodyazhnyy
13 hours ago
add a comment |
Close vote notice: While the linked duplicate asks same question, the comments and the only answer do not provide an answer which draws from authoritative source. The answer on my question by Thomas Dickey, does in fact provide the answer I've been seeking. Thus the proposed duplicate isn't helpful, however Thomas's answer here is proper.
According to the answer by G-Man on Superuser question and personal account in this PerlDuck's comment, directories that have large amount of entries get to a size over 4096 bytes (which can be seen with ls -l output), but once the entries are removed - the number never goes down.
The question is "why" ? Is it due to how ext4 filesystem configured to retain directory metadata ? Obviously removing the directory and recreating it isn't a solution, since it deletes original inode and creates a new one. What can be done to decrease the number manually ?
directory ext4 inode
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
Close vote notice: While the linked duplicate asks same question, the comments and the only answer do not provide an answer which draws from authoritative source. The answer on my question by Thomas Dickey, does in fact provide the answer I've been seeking. Thus the proposed duplicate isn't helpful, however Thomas's answer here is proper.
According to the answer by G-Man on Superuser question and personal account in this PerlDuck's comment, directories that have large amount of entries get to a size over 4096 bytes (which can be seen with ls -l output), but once the entries are removed - the number never goes down.
The question is "why" ? Is it due to how ext4 filesystem configured to retain directory metadata ? Obviously removing the directory and recreating it isn't a solution, since it deletes original inode and creates a new one. What can be done to decrease the number manually ?
directory ext4 inode
directory ext4 inode
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
edited 13 hours ago
Sergiy Kolodyazhnyy
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
asked 14 hours ago
Sergiy KolodyazhnyySergiy Kolodyazhnyy
8,56912254
8,56912254
not really - this question was specific enough to quote an authoritative source, unlike the suggested duplicate
– Thomas Dickey
13 hours ago
1
@Kusalananda I've edited the answer to include a rationale for why the linked duplicate isn't appropriate. I would suggest linking the other way around, ans Thomas's answer here does in fact answer the why of it for both mine and the other question
– Sergiy Kolodyazhnyy
13 hours ago
add a comment |
not really - this question was specific enough to quote an authoritative source, unlike the suggested duplicate
– Thomas Dickey
13 hours ago
1
@Kusalananda I've edited the answer to include a rationale for why the linked duplicate isn't appropriate. I would suggest linking the other way around, ans Thomas's answer here does in fact answer the why of it for both mine and the other question
– Sergiy Kolodyazhnyy
13 hours ago
not really - this question was specific enough to quote an authoritative source, unlike the suggested duplicate
– Thomas Dickey
13 hours ago
not really - this question was specific enough to quote an authoritative source, unlike the suggested duplicate
– Thomas Dickey
13 hours ago
1
1
@Kusalananda I've edited the answer to include a rationale for why the linked duplicate isn't appropriate. I would suggest linking the other way around, ans Thomas's answer here does in fact answer the why of it for both mine and the other question
– Sergiy Kolodyazhnyy
13 hours ago
@Kusalananda I've edited the answer to include a rationale for why the linked duplicate isn't appropriate. I would suggest linking the other way around, ans Thomas's answer here does in fact answer the why of it for both mine and the other question
– Sergiy Kolodyazhnyy
13 hours ago
add a comment |
1 Answer
1
active
oldest
votes
Quoting a developer (in a linux kernel thread ext3/ext4 directories don't shrink after deleting lots of files):
On Thu, May 14, 2009 at 08:45:38PM -0400, Timo Sirainen wrote:
>
> I was rather thinking something that I could run while the system was
> fully operational. Otherwise just moving the files to a temp directory +
> rmdir() + rename() would have been fine too.
>
> I just tested that xfs, jfs and reiserfs all shrink the directories
> immediately. Is it more difficult to implement for ext* or has no one
> else found this to be a problem?
It's probably fairest to say no one has thought it worth the effort.
It would require some fancy games to swap out block locations in the
extent trees (life would be easier with non-extent-using inodes), and
in the case of htree, we would have to keep track of the index block
so we could remove it from the htree index. So it's all doable, if a
bit tricky in terms of the technical details; it's just that the
people who could do it have been busy enough with other things.
It's hasn't been considered high priority because most of the time
directories don't go from holding thousands of files down to a small
handful.
- Ted
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495176%2fwhy-directory-with-large-amounts-of-entries-does-not-shrink-in-size-after-entrie%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Quoting a developer (in a linux kernel thread ext3/ext4 directories don't shrink after deleting lots of files):
On Thu, May 14, 2009 at 08:45:38PM -0400, Timo Sirainen wrote:
>
> I was rather thinking something that I could run while the system was
> fully operational. Otherwise just moving the files to a temp directory +
> rmdir() + rename() would have been fine too.
>
> I just tested that xfs, jfs and reiserfs all shrink the directories
> immediately. Is it more difficult to implement for ext* or has no one
> else found this to be a problem?
It's probably fairest to say no one has thought it worth the effort.
It would require some fancy games to swap out block locations in the
extent trees (life would be easier with non-extent-using inodes), and
in the case of htree, we would have to keep track of the index block
so we could remove it from the htree index. So it's all doable, if a
bit tricky in terms of the technical details; it's just that the
people who could do it have been busy enough with other things.
It's hasn't been considered high priority because most of the time
directories don't go from holding thousands of files down to a small
handful.
- Ted
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
add a comment |
Quoting a developer (in a linux kernel thread ext3/ext4 directories don't shrink after deleting lots of files):
On Thu, May 14, 2009 at 08:45:38PM -0400, Timo Sirainen wrote:
>
> I was rather thinking something that I could run while the system was
> fully operational. Otherwise just moving the files to a temp directory +
> rmdir() + rename() would have been fine too.
>
> I just tested that xfs, jfs and reiserfs all shrink the directories
> immediately. Is it more difficult to implement for ext* or has no one
> else found this to be a problem?
It's probably fairest to say no one has thought it worth the effort.
It would require some fancy games to swap out block locations in the
extent trees (life would be easier with non-extent-using inodes), and
in the case of htree, we would have to keep track of the index block
so we could remove it from the htree index. So it's all doable, if a
bit tricky in terms of the technical details; it's just that the
people who could do it have been busy enough with other things.
It's hasn't been considered high priority because most of the time
directories don't go from holding thousands of files down to a small
handful.
- Ted
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
add a comment |
Quoting a developer (in a linux kernel thread ext3/ext4 directories don't shrink after deleting lots of files):
On Thu, May 14, 2009 at 08:45:38PM -0400, Timo Sirainen wrote:
>
> I was rather thinking something that I could run while the system was
> fully operational. Otherwise just moving the files to a temp directory +
> rmdir() + rename() would have been fine too.
>
> I just tested that xfs, jfs and reiserfs all shrink the directories
> immediately. Is it more difficult to implement for ext* or has no one
> else found this to be a problem?
It's probably fairest to say no one has thought it worth the effort.
It would require some fancy games to swap out block locations in the
extent trees (life would be easier with non-extent-using inodes), and
in the case of htree, we would have to keep track of the index block
so we could remove it from the htree index. So it's all doable, if a
bit tricky in terms of the technical details; it's just that the
people who could do it have been busy enough with other things.
It's hasn't been considered high priority because most of the time
directories don't go from holding thousands of files down to a small
handful.
- Ted
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
Quoting a developer (in a linux kernel thread ext3/ext4 directories don't shrink after deleting lots of files):
On Thu, May 14, 2009 at 08:45:38PM -0400, Timo Sirainen wrote:
>
> I was rather thinking something that I could run while the system was
> fully operational. Otherwise just moving the files to a temp directory +
> rmdir() + rename() would have been fine too.
>
> I just tested that xfs, jfs and reiserfs all shrink the directories
> immediately. Is it more difficult to implement for ext* or has no one
> else found this to be a problem?
It's probably fairest to say no one has thought it worth the effort.
It would require some fancy games to swap out block locations in the
extent trees (life would be easier with non-extent-using inodes), and
in the case of htree, we would have to keep track of the index block
so we could remove it from the htree index. So it's all doable, if a
bit tricky in terms of the technical details; it's just that the
people who could do it have been busy enough with other things.
It's hasn't been considered high priority because most of the time
directories don't go from holding thousands of files down to a small
handful.
- Ted
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
answered 13 hours ago
Thomas DickeyThomas Dickey
52.4k595166
52.4k595166
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495176%2fwhy-directory-with-large-amounts-of-entries-does-not-shrink-in-size-after-entrie%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
not really - this question was specific enough to quote an authoritative source, unlike the suggested duplicate
– Thomas Dickey
13 hours ago
1
@Kusalananda I've edited the answer to include a rationale for why the linked duplicate isn't appropriate. I would suggest linking the other way around, ans Thomas's answer here does in fact answer the why of it for both mine and the other question
– Sergiy Kolodyazhnyy
13 hours ago