Cdtjkyj

Sorry for the long question, but I want to be as detailed as possible and am new to this sort of thing so I'm unsure which bits are relevant. Here is my situation:

I had a nice Ubuntu 18.04 system up and running with many different internal SATA drives and rock solid for several months, and all were happy in the land. Then one day, there was a simple power outage. No problem, we've been through them before. Once the power came back on, the server didn't respond to SSH; sometimes I have to go down to the basement and hit the button on the box to turn it back on. I did that, and again after a few minutes, I still can't access via SSH.

So now I'm down in the basement with a little monitor and keyboard hooked up to the box and I can see that the Ubuntu logo is up there quite a while after reboot so I hit the ⬆ key and watch the boot loader thingy and it seems where its timing out is with something like:

A start job is running for dev-disk-byx2duuid-914d3b77x2d06c4x2d4514x2d8feex2d1fc6eb81bbd9.device (51s / 1min 30s)

There are actually two of these errors at first, both seeming to time out after reaching 1min 30s. The other start job that times out appears to reference UUID=a158e6ec-1433-454c-9cd2-10f7306fde82. So I search around and read about how the UUIDs in the job that is timing out correspond to some of the UUIDs I specify in /etc/fstab to automatically mount my internal hard drives upon boot.

After 1:30, I hit enter to get to a root command prompt and run vim /etc/fstab and comment out the lines that correspond to my 2 errors so now the file looks like this:

# /etc/fstab: static file system information.

#

# Use 'blkid' to print the universally unique identifier for a

# device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).

#

# <file system> <mount point>   <type>  <options>       <dump>  <pass>

# / was on /dev/sda2 during installation

UUID=6f90b496-401e-475f-add0-3c6d3bcae7a0 /               ext4    errors=remount-ro 0       1

# /boot/efi was on /dev/sda1 during installation

UUID=BFE0-55D4  /boot/efi       vfat    umask=0077      0       1

/swapfile                                 none            swap    sw              0       0

UUID=bc731122-7e4e-47d9-b6a5-db1f703f96a8       /media/Tre      ext4    defaults        0       0

UUID=6c7b175d-b80b-4069-bbbe-f82aeb302200       /media/Sam      ext4    defaults        0       0

#UUID=a158e6ec-1433-454c-9cd2-10f7306fde82       /media/Hex      ext4    defaults        0       0

#UUID=914d3b77-06c4-4514-8fee-1fc6eb81bbd9       /media/Wes      ext4    defaults        0       0

After saving the file, I run reboot and Ubuntu reboots pretty quickly. It doesn't hand and I get to the login screen. I crawl out of the deep dark basement server closet and SSH back in from the couch.

I use blkid and discover the UUID on the drive I call Wes appeared to be different than what I had in /etc/fstab, so I took a backup and edited that UUID to the one from blkid and uncommented that line. Another reboot and now I've got Wes back. So now I am just missing the big 6TB drive I call Hex and my /etc/fstab looks like:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>

# / was on /dev/sda2 during installation

UUID=6f90b496-401e-475f-add0-3c6d3bcae7a0 /               ext4    errors=remount-ro 0       1

# /boot/efi was on /dev/sda1 during installation

UUID=BFE0-55D4  /boot/efi       vfat    umask=0077      0       1

/swapfile                                 none            swap    sw              0       0

UUID=bc731122-7e4e-47d9-b6a5-db1f703f96a8       /media/Tre      ext4    defaults        0       0

UUID=6c7b175d-b80b-4069-bbbe-f82aeb302200       /media/Sam      ext4    defaults        0       0

#UUID=a158e6ec-1433-454c-9cd2-10f7306fde82      /media/Hex      ext4    defaults        0       0

UUID=bc731122-7e4e-47d9-b6a5-db1f703f96a8       /media/Wes      ext4    defaults        0       0

If I uncomment the line for Hex, I get the endless loop of waiting 1:30 for that same job to time out. If I use journalctl -xe to view the logs and go to what I think is where things go wrong, I see a red error like:

zen systemd[1]: Timed out waiting for device dev-disk-byx2duuid-a158e6ecx2d1433x2d454cx2d9cd2x2d10f7306fde82.device

which would seem to correspond to the same Hex drive.

Fearing the drive was dead, I pulled the box from the closet and opened it up and removed that particular hard disk drive. I took it to my desk and put it in a SATA to USB interface and supplied power. The drive began spinning and I hooked it up to my Mac laptop. When I opened Disk Utility, I could see the HDD, but it said it was only 1.8TB and I could not see the partitions.

I figured that part might be right as I seem to remember having to do something special to format that 6TB drive and maybe the Mac just can't see it? Anyways, I'm encouraged by seeing the drive at all and decide to go put it back in the server.

I read more and search around more and I'm in this loop. I can comment out the /etc/fstab entry and get into the system via SSH, or I can uncomment it and the reboot hangs. If I cd into /media/Hex I can see a few of the top level folders and files, but the vast majority of the file structure appears gone, or invisible to me.

Thank you to anyone who has ideas for me on how to get Hex back to normal!