LUKS BTRFS RAID1c3

Hello, I need help.

I have a BTRFS encrypted LUKS in RAID1c3 OMV6.

1.0101 4.0201 7.0301

2.0102 5.0202 8.0302

3.0103 6.0203 9.0303

3x18TB 3x12TB 3x12TB

The disks are always WD, Seagate, and Toshiba.

First, I created an 18TB RAID1c3, uploaded data, and added another group of 12TB, uploaded data again, and finally the last 12TB.

The system ran for several weeks without a restart, but today after a restart, all disks 4, 5, 6 (group 020X) are unreadable.

sudo cryptsetup luksOpen --test-passphrase /dev/sdk sdk-crypt

The device /dev/sdk is not a valid LUKS device.

These are disks SDK, SDL, and SDC.

Do you have any idea what might have happened suddenly? And why exactly the same group of disks? And most importantly, how could I recover it? At least one of them could be reconnected.

Thank you very much for your advice.

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 10,9T 0 disk

└─sda-crypt 253:4 0 10,9T 0 crypt

sdb 8:16 0 3,6T 0 disk

└─sdb-crypt 253:16 0 3,6T 0 crypt

sdc 8:32 0 10,9T 0 disk

sdd 8:48 0 5,5T 0 disk

└─sdd-crypt 253:13 0 5,5T 0 crypt

sde 8:64 0 5,5T 0 disk

└─sde-crypt 253:8 0 5,5T 0 crypt

sdf 8:80 0 3,6T 0 disk

└─sdf-crypt 253:12 0 3,6T 0 crypt

sdg 8:96 0 5,5T 0 disk

└─sdg-crypt 253:15 0 5,5T 0 crypt

sdh 8:112 0 16,4T 0 disk

└─sdh-crypt 253:10 0 16,4T 0 crypt

sdi 8:128 0 3,6T 0 disk

└─sdi-crypt 253:7 0 3,6T 0 crypt /srv/dev-disk-by-uuid-b105c050-574b-4070-952b-68300159c431

sdj 8:144 0 16,4T 0 disk

└─sdj-crypt 253:5 0 16,4T 0 crypt

sdk 8:160 0 10,9T 0 disk

sdl 8:176 0 10,9T 0 disk

sdm 8:192 0 16,4T 0 disk

└─sdm-crypt 253:2 0 16,4T 0 crypt

sdn 8:208 0 2,7T 0 disk

└─sdn-crypt 253:14 0 2,7T 0 crypt

sdo 8:224 0 10,9T 0 disk

└─sdo-crypt 253:3 0 10,9T 0 crypt

sdp 8:240 0 2,7T 0 disk

└─sdp-crypt 253:11 0 2,7T 0 crypt

sdq 65:0 0 931,5G 0 disk

└─md1 9:1 0 2,1T 0 raid0 /srv/dev-disk-by-uuid-d5cfecd9-41a1-4c97-a453-58a5a5064f99

sdr 65:16 0 2,7T 0 disk

└─sdr-crypt 253:6 0 2,7T 0 crypt /srv/dev-disk-by-uuid-f2f2eeee-2c5f-4289-b597-b18000c1ae05

sds 65:32 0 10,9T 0 disk

└─sds-crypt 253:9 0 10,9T 0 crypt

sdt 65:48 0 931,5G 0 disk

└─md1 9:1 0 2,1T 0 raid0 /srv/dev-disk-by-uuid-d5cfecd9-41a1-4c97-a453-58a5a5064f99

sdu 65:64 0 279,5G 0 disk

└─md1 9:1 0 2,1T 0 raid0 /srv/dev-disk-by-uuid-d5cfecd9-41a1-4c97-a453-58a5a5064f99

sdv 65:80 0 2,7T 0 disk /srv/dev-disk-by-uuid-3c469e3d-3e09-45fe-b0eb-76cd0287938f

sdw 65:96 0 2,7T 0 disk /srv/dev-disk-by-uuid-e35e7ea6-c7b6-4a8e-9904-159ef0610737

sdx 65:112 0 2,7T 0 disk /srv/dev-disk-by-uuid-6b32b6e6-7ad6-4e26-a497-ed2d98b211d4

sdy 65:128 0 3,6T 0 disk /srv/dev-disk-by-uuid-ecd2369c-99b3-4647-bb78-a43e62dc6232

sdz 65:144 0 3,6T 0 disk /srv/dev-disk-by-uuid-ba4491f0-3e1a-4314-9fc3-da6a1f324d8d

sdaa 65:160 0 3,6T 0 disk /srv/dev-disk-by-uuid-b943a5ee-755c-4b6c-8d6d-6a0854406598

sdab 65:176 0 3,6T 0 disk /srv/dev-disk-by-uuid-1e3ccde1-51a1-46bd-809e-c9ad59086d2e

sdac 65:192 0 5,5T 0 disk /srv/dev-disk-by-uuid-a9c6acc6-d611-4514-9141-5ca680793136

sdad 65:208 0 3,6T 0 disk

└─sdad-crypt 253:0 0 3,6T 0 crypt

sdae 65:224 0 3,6T 0 disk

└─sdae-crypt 253:1 0 3,6T 0 crypt

sdaf 65:240 0 119,2G 0 disk

└─sdaf1 65:241 0 119,2G 0 part /srv/dev-disk-by-uuid-2f4e25b2-df03-42bd-8c82-fe4dcc201344

sdag 66:0 0 232,9G 0 disk

└─sdag1 66:1 0 232,9G 0 part /srv/dev-disk-by-uuid-6dba7e59-210e-4cd6-bde4-9df6894bf05d

sdah 66:16 0 3,6T 0 disk /srv/dev-disk-by-uuid-4ec6b146-32e4-4d0f-b1b1-886d0e239711

nvme2n1 259:0 0 931,5G 0 disk

nvme0n1 259:1 0 931,5G 0 disk /srv/dev-disk-by-uuid-4d5fbc50-c661-471a-b311-ca1d8cbf9c07

nvme1n1 259:2 0 465,8G 0 disk

├─nvme1n1p1 259:3 0 512M 0 part /boot/efi

├─nvme1n1p2 259:4 0 464,3G 0 part /

└─nvme1n1p3 259:5 0 976M 0 part [SWAP]

Edit:

sudo btrfsck /dev/sdk

Opening filesystem to check...

No valid Btrfs found on /dev/sdk

ERROR: cannot open file system

I found something on forums, but this looks wrong.

root@r-omv:~# sudo hexdump -C -n 512 /dev/sdg

00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

*

00000200

root@r-omv:~# sudo hexdump -C -n 512 /dev/sdi

00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

*

00000200

root@r-omv:~# sudo hexdump -C -n 512 /dev/sdl

00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

root@r-omv:~# sudo hexdump -C /dev/sdi | grep LUKS

2b25014a0 4c 55 4b 53 29 77 c7 02 5b 23 ed 11 7d 79 36 76 |LUKS)w..[#..}y6v|

5b9005820 d1 30 42 fe a4 a6 0e 15 4c 55 4b 53 88 2e c0 29 |.0B.....LUKS...)|

61becfe30 b4 40 e7 5d 22 39 ab c5 6b 38 4c 55 4b 53 f5 66 |.@.]"9..k8LUKS.f|

6a1c00ec0 94 dc 45 ac 76 d7 c6 70 ed 66 b0 e5 4c 55 4b 53 |..E.v..p.f..LUKS|

82b3a7450 b1 b5 c6 cd ba bd 07 dd 9f b4 4c 55 4b 53 15 66 |..........LUKS.f|

842518090 0a 76 4c 55 4b 53 0d 82 25 ca fe 53 8a a6 13 c1 |.vLUKS..%..S....|

8bbf0cda0 7a 12 17 5b bf aa ec 35 ce f4 4c 55 4b 53 4c c6 |z..[...5..LUKSL.|

cca255890 7d ad de 8e cb 4a 4c 55 4b 53 ff 16 68 9d 29 85 |}....JLUKS..h.).|

d39a10ee0 e9 6b 25 b9 7e 88 4c 55 4b 53 1c 95 f9 e7 3f fe |.k%.~.LUKS....?.|

deaffbf50 64 39 4c 55 4b 53 15 c5 c5 66 6d 54 da ff 15 96 |d9LUKS...fmT....|

Nevertheless, this gives me hope. (After a restart, the disks have different names.)

It would help me a lot if someone could tell me how to get at least one of the three drives working.

I am trying to find the headers on the disk.

It would also help me to explain how it happens that three disks are unreadable after a restart out of nowhere.

Thank you...

root@r-omv:~# sudo hexdump -C /dev/sdi | grep LUKS

2b25014a0 4c 55 4b 53 29 77 c7 02 5b 23 ed 11 7d 79 36 76 |LUKS)w..[#..}y6v|

5b9005820 d1 30 42 fe a4 a6 0e 15 4c 55 4b 53 88 2e c0 29 |.0B.....LUKS...)|

61becfe30 b4 40 e7 5d 22 39 ab c5 6b 38 4c 55 4b 53 f5 66 |.@.]"9..k8LUKS.f|

6a1c00ec0 94 dc 45 ac 76 d7 c6 70 ed 66 b0 e5 4c 55 4b 53 |..E.v..p.f..LUKS|

82b3a7450 b1 b5 c6 cd ba bd 07 dd 9f b4 4c 55 4b 53 15 66 |..........LUKS.f|

842518090 0a 76 4c 55 4b 53 0d 82 25 ca fe 53 8a a6 13 c1 |.vLUKS..%..S....|

8bbf0cda0 7a 12 17 5b bf aa ec 35 ce f4 4c 55 4b 53 4c c6 |z..[...5..LUKSL.|

cca255890 7d ad de 8e cb 4a 4c 55 4b 53 ff 16 68 9d 29 85 |}....JLUKS..h.).|

d39a10ee0 e9 6b 25 b9 7e 88 4c 55 4b 53 1c 95 f9 e7 3f fe |.k%.~.LUKS....?.|

deaffbf50 64 39 4c 55 4b 53 15 c5 c5 66 6d 54 da ff 15 96 |d9LUKS...fmT....|

10d2b7d460 26 66 fd 20 e7 4c 55 4b 53 be 22 25 13 11 3a f4 |&f. .LUKS."%..:.|

141dd330b0 e4 c3 4c cd ac ae f2 ca da 6a 52 4c 55 4b 53 f0 |..L......jRLUKS.|

1485004050 bf 61 dc 37 6f 7f 58 7b cf 4c 55 4b 53 6a 8b aa |.a.7o.X{.LUKSj..|

17464ae980 fc 47 5c 27 b9 0a 6e 7c 4c 55 4b 53 a1 87 5c a0 |.G\'..n|LUKS..\.|

17ddc60e80 07 a1 d0 92 5b 45 4c 55 4b 53 26 3b 66 c3 ab c1 |....[ELUKS&;f...|

1853a1dd50 37 8d 5f 4c 55 4b 53 2e d4 47 b3 3a 44 ce ce f7 |7._LUKS..G.:D...|

198d0c3f50 c4 82 aa c9 25 e4 de 95 5a d0 4c 55 4b 53 69 1b |....%...Z.LUKSi.|

1cd9c38310 da bd b1 7d 4c 55 4b 53 7c 7e cd 46 ee 35 07 ec |...}LUKS|~.F.5..|

2034061050 e1 5e 8d 83 3f e9 34 88 4c 55 4b 53 2c 06 ef a6 |.^..?.4.LUKS,...|

22414290c0 4c 55 4b 53 51 70 73 28 01 c6 e4 9e dd 2c 9b ef |LUKSQps(.....,..|

22877beb80 32 23 76 b8 10 49 50 81 4c 55 4b 53 ee 98 b6 09 |2#v..IP.LUKS....|

22d04a9d50 76 cf 95 71 9d c7 9b 42 88 1c 1b b2 4c 55 4b 53 |v..q...B....LUKS|

24c6be09b0 30 f1 22 19 13 4c 55 4b 53 79 90 07 40 6d 1f 86 |0."..LUKSy..@m..|

257b7fa2e0 82 4c 55 4b 53 a1 ef bc 74 df 4f 23 59 b2 1a 7f |.LUKS...t.O#Y...|

25c95724b0 4c 55 4b 53 dc 0b b7 30 a7 16 f5 88 fc 7e 46 08 |LUKS...0.....~F.|

28a2c8d740 79 04 30 cf 11 f8 4c 55 4b 53 64 9f f2 12 96 5e |y.0...LUKSd....^|

299c3cd440 d3 d1 62 53 38 d9 4c 55 4b 53 35 46 1e 72 c6 7d |..bS8.LUKS5F.r.}|

2b1404d1e0 5f d8 3f 79 7c 4c 55 4b 53 6d b8 58 61 2b 91 ab |_.?y|LUKSm.Xa+..|

2c5d918630 b6 12 2b ec 86 53 8d 30 de b0 2b 4c 55 4b 53 22 |..+..S.0..+LUKS"|

2c9639d040 7d ff e0 be 7f ec 4c 55 4b 53 dd 40 44 0b da 05 |}.....LUKS.@D...|

2cce7ac0e0 e0 4c 55 4b 53 6f 88 47 15 82 5f 73 00 c7 36 7a |.LUKSo.G.._s..6z|

33f1d0a2b0 24 c1 1d 9f 4d 55 36 34 0c 4c 55 4b 53 1f 61 db |$...MU64.LUKS.a.|

3531816fa0 05 9a 71 3a 4c 55 4b 53 64 17 ac 2a cf c8 d3 e5 |..q:LUKSd..*....|

36775fb4c0 a8 96 a8 05 b7 1e 94 36 06 4c 55 4b 53 b3 c6 da |.......6.LUKS...|

36bc88baf0 a5 83 db 9b 4d 06 96 4c 55 4b 53 16 25 e6 da 71 |....M..LUKS.%..q|

380b552960 d8 b0 78 6a 13 ca 4c 55 4b 53 48 46 9b cc eb 37 |..xj..LUKSHF...7|

38d1b10560 c0 46 d5 7e 37 4c 55 4b 53 5e b1 39 4f 4b 74 15 |.F.~7LUKS^.9OKt.|

3b1451c7d0 f2 0d 90 a3 f0 4c 55 4b 53 15 30 e4 89 e2 0a a3 |.....LUKS.0.....|

3d12c7f930 32 e5 83 11 81 a2 4c 55 4b 53 ae 92 f5 c9 b5 ee |2.....LUKS......|

3d98d3be00 15 31 c6 4c 55 4b 53 b2 c0 e7 a0 c3 c6 44 ab 5b |.1.LUKS......D.[|

3f73adbea0 ed 41 db 4c 55 4b 53 60 3c 0b a4 c6 dc 94 df d2 |.A.LUKS`<.......|

417c7a2250 fb df 4c 55 4b 53 97 45 30 38 b0 7f 2c f8 5e af |..LUKS.E08..,.^.|

42e339fe90 e6 b1 4f 41 35 d8 e3 4c 55 4b 53 1e 15 0d 00 1a |..OA5..LUKS.....|

43911fe480 48 52 0c 48 9a d6 84 dc cc 4c 55 4b 53 81 7c 17 |HR.H.....LUKS.|.|

4438505800 5e 4c 55 4b 53 bc 6b 13 54 78 3b c9 c0 bc 79 58 |^LUKS.k.Tx;...yX|

44c198cfe0 3b c3 4c 55 4b 53 36 ee 45 61 f0 7a bc e6 46 94 |;.LUKS6.Ea.z..F.|

44fab188f0 64 52 4c 55 4b 53 a6 bb d3 59 8a 6a d1 f2 a9 a8 |dRLUKS...Y.j....|

4520678140 96 0e 47 35 b1 c6 d0 24 3a 4b 4c 55 4b 53 f4 0e |..G5...$:KLUKS..|

4870a98a70 20 7d 92 ea c0 69 aa 4c 55 4b 53 2d c7 92 bf a5 | }...i.LUKS-....|

48bc4f5830 37 a7 de c6 4c 55 4b 53 36 25 8f e4 f7 40 95 63 |7...LUKS6%...@.c|

48c6969820 4c 55 4b 53 68 27 6f ea b8 85 82 68 6e a4 37 dd |LUKSh'o....hn.7.|

4a3ae7b2d0 20 e2 d4 31 4c 55 4b 53 c6 59 c6 92 8c 98 cb 6c | ..1LUKS.Y.....l|

4bceab0e30 4c 55 4b 53 0c 63 a6 a2 ce e0 24 ff 2b 8a 5e 55 |LUKS.c....$.+.^U|

4bf8170a80 c4 d3 4c 55 4b 53 31 99 64 27 f8 00 4c 9e ef 34 |..LUKS1.d'..L..4|

4c9c498c80 cd 1f da 5b 50 e5 9c 6d 1b e4 97 66 4c 55 4b 53 |...[P..m...fLUKS|

4da6552730 b9 94 9b 1e 64 7b e7 97 3b 78 f8 4c 55 4b 53 92 |....d{..;x.LUKS.|

4e70193a50 62 b5 54 19 06 3f 6c 98 30 b5 4c 55 4b 53 b1 1d |b.T..?l.0.LUKS..|

4fced3c290 d9 4c 05 b0 4c 55 4b 53 79 24 da 4a fb 3a 1f 8e |.L..LUKSy$.J.:..|

50a6e4bf70 f7 04 bd b0 e2 d1 58 4c 55 4b 53 e8 3a a4 b0 2b |......XLUKS.:..+|

50a7ec9500 b9 0b e6 df 4c 55 4b 53 91 db 32 68 c2 0f 63 4f |....LUKS..2h..cO|

50e48c87a0 00 9e 43 5e 09 ce 01 f1 4c 55 4b 53 13 cd dc 44 |..C^....LUKS...D|

5164049970 4c 55 4b 53 31 7c 94 5e e0 40 bc 15 ee bb 67 d9 |LUKS1|.^.@....g.|

5213a1f130 4e d5 e2 78 95 ca 64 79 3b 4c 55 4b 53 36 ad 49 |N..x..dy;LUKS6.I|

5344a8c880 99 c7 b3 37 43 8e b3 0b 0e 4c 55 4b 53 fc f1 16 |...7C....LUKS...|

53f479faa0 d7 80 3a 3b e7 dd db 39 15 4c 55 4b 53 bf f8 2d |..:;...9.LUKS..-|

542701b840 ba b9 e8 df 55 bd 7a c1 35 79 4c 55 4b 53 41 a5 |....U.z.5yLUKSA.|

560df586d0 d0 a8 da 39 95 90 90 0f 1b 78 e2 4c 55 4b 53 b8 |...9.....x.LUKS.|

57d75b6190 ab 4e 18 4c 55 4b 53 9f 24 da 12 70 e7 2e 45 cd |.N.LUKS.$..p..E.|

5819ca1ed0 a4 ff 38 25 23 4c 55 4b 53 0b cd 0a cf 08 89 c7 |..8%#LUKS.......|

5bc5d2e750 8b 3f 1c 0d b6 79 4c 55 4b 53 a6 98 11 04 84 fd |.?...yLUKS......|

5d57be8070 4f a4 89 82 dc c7 f7 ee 9b 5d 4c 55 4b 53 af 19 |O........]LUKS..|

5e470c4500 7d 08 cc 4c 55 4b 53 8e 8a bb 6d b8 00 92 c7 cf |}..LUKS...m.....|

5f3336a1d0 f6 28 fe 4c 55 4b 53 97 cb 9f a8 de 5f 73 70 c0 |.(.LUKS....._sp.|

5f7862c650 2f 9b 90 52 4c 55 4b 53 4d 21 d9 96 2f 6d 12 d2 |/..RLUKSM!../m..|

5feba1ca90 9e f1 ab 15 3d 79 13 89 63 b5 d5 e7 4c 55 4b 53 |....=y..c...LUKS|

64d8d394a0 e8 5d b0 24 4c 55 4b 53 ce 81 1c 20 80 40 ed 91 |.].$LUKS... .@..|

669e887ad0 4c 55 4b 53 01 19 7f d2 2b ad e5 64 db 5e ba 86 |LUKS....+..d.^..|

66bce37340 89 d3 ad c4 f4 65 aa ba 4e 4c 55 4b 53 b5 11 43 |.....e..NLUKS..C|

6718bccc70 45 a9 60 ee f0 5b 4c 55 4b 53 59 66 9f 09 d1 0f |E.`..[LUKSYf....|

6737544020 64 1f 28 39 13 76 06 3d bc 4c 55 4b 53 83 74 c3 |d.(9.v.=.LUKS.t.|

67bf5de940 e9 9c f3 4c 55 4b 53 c0 2f 4e 8c be ef 5c 1e 53 |...LUKS./N...\.S|

680484b990 cc 5d 4a 83 63 da 71 4c 55 4b 53 fc b1 59 8a a5 |.]J.c.qLUKS..Y..|

6b1a013640 2c 7d 4c 55 4b 53 a9 79 c3 91 3d b3 36 2b a8 79 |,}LUKS.y..=.6+.y|

6b722f9320 00 4c 55 4b 53 91 71 f3 69 2d 02 37 c3 24 3a 94 |.LUKS.q.i-.7.$:.|

6bcdde4d30 38 35 21 65 13 e2 03 2d 0d 4c 55 4b 53 c7 00 b3 |85!e...-.LUKS...|

6c1f5f6630 5d 20 ff f2 80 50 a9 04 0d 51 47 4c 55 4b 53 37 |] ...P...QGLUKS7|

6c26844450 52 58 46 86 9c 14 9d f3 4f 27 4c 55 4b 53 e3 be |RXF.....O'LUKS..|

6e7ae15e80 42 e4 0b e1 4c 55 4b 53 ce 09 f9 f9 09 a1 f7 df |B...LUKS........|

6eea9dd6e0 a3 40 fa 4d c0 4c 55 4b 53 10 a2 86 43 01 b1 2b |.@.M.LUKS...C..+|

700933ae50 67 8c 4f af 4c 55 4b 53 ce 6f 68 d1 6f f3 a2 7f |g.O.LUKS.oh.o...|

70c3b33490 97 8c 50 4c 55 4b 53 b7 ac bb 66 bf 3b f3 8a 86 |..PLUKS...f.;...|

721b0a0a40 3c 4c 55 4b 53 2f 51 fe 40 70 01 d9 91 3d b6 8a |<LUKS/Q.@p...=..|

73d4c81620 02 86 0d 68 6b b9 9b c4 4c 55 4b 53 2f 7d c5 6c |...hk...LUKS/}.l|