Initial rsync backup of large data slow to RAID 5
I'm using rsync to copy about 28TB of images to a 36TB RAID 5.
The source has SSDs and the destination has 6 8TB 7200 SATA3 512e drives in a RAID 5 configuration.
Servers are connected over a 10G fiber connection. They are the only two machines on the switch.
Source is CentOS 6.8 destination is Ubuntu 18.04.
I understand that the HDDs are not going to get the full 6GB/s write speed but I am currently only getting 65MB/s when I was expecting somewhere in at least the 200MB/s range.
The speed started at about 72MB/s then gradually increased to 83MB/s before falling to and maintaining 65MB/s over the course of about an hour. Currently the transfer is on course for 5 days.
This seems extremely slow. I was hoping for any suggestions on speeding this up or an explanation as to why it is so slow.
Command run:
rsync -a --info=progress2 user@sourceserver:/images/library/ /images/library
rsync
asked 9 mins ago
LoomLoom
1
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I'm using rsync to copy about 28TB of images to a 36TB RAID 5.
The source has SSDs and the destination has 6 8TB 7200 SATA3 512e drives in a RAID 5 configuration.
Servers are connected over a 10G fiber connection. They are the only two machines on the switch.
Source is CentOS 6.8 destination is Ubuntu 18.04.
I understand that the HDDs are not going to get the full 6GB/s write speed but I am currently only getting 65MB/s when I was expecting somewhere in at least the 200MB/s range.
The speed started at about 72MB/s then gradually increased to 83MB/s before falling to and maintaining 65MB/s over the course of about an hour. Currently the transfer is on course for 5 days.
This seems extremely slow. I was hoping for any suggestions on speeding this up or an explanation as to why it is so slow.
Command run:
rsync -a --info=progress2 user@sourceserver:/images/library/ /images/library
rsync
asked 9 mins ago
LoomLoom
1
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I'm using rsync to copy about 28TB of images to a 36TB RAID 5.
The source has SSDs and the destination has 6 8TB 7200 SATA3 512e drives in a RAID 5 configuration.
Servers are connected over a 10G fiber connection. They are the only two machines on the switch.
Source is CentOS 6.8 destination is Ubuntu 18.04.
I understand that the HDDs are not going to get the full 6GB/s write speed but I am currently only getting 65MB/s when I was expecting somewhere in at least the 200MB/s range.
The speed started at about 72MB/s then gradually increased to 83MB/s before falling to and maintaining 65MB/s over the course of about an hour. Currently the transfer is on course for 5 days.
This seems extremely slow. I was hoping for any suggestions on speeding this up or an explanation as to why it is so slow.
Command run:
rsync -a --info=progress2 user@sourceserver:/images/library/ /images/library
rsync
asked 9 mins ago
LoomLoom
1
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I'm using rsync to copy about 28TB of images to a 36TB RAID 5.
The source has SSDs and the destination has 6 8TB 7200 SATA3 512e drives in a RAID 5 configuration.
Servers are connected over a 10G fiber connection. They are the only two machines on the switch.
Source is CentOS 6.8 destination is Ubuntu 18.04.
I understand that the HDDs are not going to get the full 6GB/s write speed but I am currently only getting 65MB/s when I was expecting somewhere in at least the 200MB/s range.
The speed started at about 72MB/s then gradually increased to 83MB/s before falling to and maintaining 65MB/s over the course of about an hour. Currently the transfer is on course for 5 days.
This seems extremely slow. I was hoping for any suggestions on speeding this up or an explanation as to why it is so slow.
Command run:
rsync -a --info=progress2 user@sourceserver:/images/library/ /images/library
rsync
rsync
asked 9 mins ago
LoomLoom
1
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 mins ago
LoomLoom
1
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 mins ago
LoomLoom
1
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 9 mins ago
LoomLoom
1
asked 9 mins ago
LoomLoom
1
1
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Loom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Loom is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504398%2finitial-rsync-backup-of-large-data-slow-to-raid-5%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Loom is a new contributor. Be nice, and check out our Code of Conduct.
Loom is a new contributor. Be nice, and check out our Code of Conduct.
Loom is a new contributor. Be nice, and check out our Code of Conduct.
Loom is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504398%2finitial-rsync-backup-of-large-data-slow-to-raid-5%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
